Rule-based filtering algorithm for textual document

Textual document is usually in unstructured form and high dimensional data.The exploration of hidden information from the unstructured text is useful to find interesting patterns and valuable knowledge.However, not all terms in the text are relevant and can lead to misclassification. Improper filtration might cause terms that have similar meaning to be removed.Thus, to reduce the high-dimensionality of text, this study proposed a filtering algorithm that is able to filter the important terms from the pre-processed text and applied term weighting scheme to solve synonym problem which will help the selection of relevant term.The proposed filtering algorithm utilizes a keyword library that contained special terms which is developed to ensure that important terms are not eliminated during filtration process.The performance of the proposed filtering algorithm is compared with rough set attribute reduction (RSAR) and information retrieval (IR) approaches.From the experiment, the proposed filtering algorithm has outperformed both RSAR and IR in terms of extracted relevant terms.