Fine Granularity Pipeline Indexing Algorithm for Multi-core Platform

The scale and growth rate of text collection bring new challenges for index construction. To tackle this problem, an efficient indexing algorithm, the Fine Granularity Pipeline (FGP), is proposed to improve the indexing performance for multi-core platform. Compared to the Simple Pipeline (SP) algorithm, compressing and parsing are divided in the FGP to get more balanced pipeline stages. Evaluations for three collections from Terabyte track in the TREC 2011 over Intel Woodcrest platform showed that the performance improvements of the FGP were over 45% and 17% compared to indexing in Indri 2.4 and the SP with three cores.