Indexing with stop words

View as MarkdownMarkdown

Set up your AI agent

Give your coding agent live access to the GroupDocs documentation and API reference through the GroupDocs Docs MCP server — no install, no API key. Learn more

claude mcp add --transport http groupdocs-docs https://docs.groupdocs.com/mcp

Also process documents locally

The GroupDocs.Search MCP server lets your agent work with files on your machine; they are never uploaded. It runs in Docker: replace /path/to/documents with the folder it may use.

claude mcp add groupdocs-search -- docker run --rm -i -v /path/to/documents:/data ghcr.io/groupdocs-search/search-net-mcp:latest
Folders, licensing and more →

Stop words are frequently used words that do not carry a semantic meaning and can be removed from an index to reduce its size.

You can enable or disable the use of stop words by setting a value of the UseStopWords property of the IndexSettings class. The default value is true, meaning that stop words are filtered during indexing and not added to an index.

A list of stop words to use during indexing can be specified in the stop word dictionary. By default, the stop word dictionary is filled with the most widely used pronouns, prepositions and conjunctions of English and Russian. The list of stop words of a new index can be replaced or supplemented before documents are added to it, and it is saved in the index. Stop words are kept in lower case, and the lookups in the stop word dictionary ignore letter case. For information on managing the stop word dictionary, see the Stop word dictionary page in the Managing dictionaries section.

Since version 26.10, the stop word dictionary and the UseStopWords setting can be changed only while the index contains no documents and is not performing an operation; otherwise, InvalidOperationException is thrown. Therefore, configure them after creating a new index and before adding the first document, as the example below does. To use other stop words with documents that are already indexed, create a new index and index the documents again. See the Index format and compatibility page for details.

When stop words are used, a stop word of a search query does not narrow the result of the other words of the query: for example, the query “the AND cat” finds the documents that contain the word “cat”. Since version 26.11, this also applies to a stop word excluded with NOT and to a part of a query that consists of stop words only: the queries “cat AND NOT the”, “the AND the AND cat” and “cat AND (the OR a)” find the documents that contain the word “cat” as well.

If you need to keep all text information extracted from documents, and you are not afraid of a significant increase in the size of the index, then an example of indexing without stop words can be found below.

C#

string indexFolder = @"c:\MyIndex\";
string documentsFolder = @"c:\MyDocuments\";
 
// Creating index settings with the use of stop words disabled
IndexSettings settings = new IndexSettings();
settings.UseStopWords = false;
 
// Creating a new index in the specified folder
// The UseStopWords setting can be changed only while the index contains no documents
Index index = new Index(indexFolder, settings, true);
 
// Indexing documents from the specified folder
index.Add(documentsFolder);
 
// Searching in the index
// Now in the index it is possible to search for the stop word 'on'
SearchResult result = index.Search("on");

More resources

GitHub examples

You may easily run the code from documentation articles and see the features in action in our GitHub examples:

Free online document search App

Along with full featured .NET library we provide simple, but powerful free Apps.

You are welcome to search over your PDF, DOC, DOCX, PPT, PPTX, XLS, XLSX and more with our free online Free Online Document Search App.