Migration Notes Leave feedback

GroupDocs.Search for Python via .NET 26.10 is the first release of GroupDocs.Search for Python. It is built on version 26.10 of GroupDocs.Search, which introduces a new index format and changes in behavior. This page describes what to take into account if your Python application uses indexes created by previous versions of GroupDocs.Search, for example by an application built on GroupDocs.Search for .NET, or if you port code written for previous versions:

• Migrating to version 26.10 — the new index format and the behavior changes that may require changes in your code.

Version 26.10 introduces a new format for storing an index on disk. Compared with previous versions:

• the index size on disk grows in proportion to the indexed data, and a loaded index requires about half as much RAM;
• an index is opened 2 to 4 times faster, and search in an index with many segments reads only the segments that contain the query terms;
• index data are protected with checksums, and all changes to an index are applied atomically: if an operation is interrupted by an error, a power failure or termination of the process, the index remains in its previous consistent state;
• deleting documents no longer rewrites the index, and searches and index-modifying operations no longer block each other.

The full rules of compatibility between the new format and indexes of previous versions are described on the Index format and compatibility page.

Version 26.10 opens only indexes of the new format. The Index constructors and the load_into_memory_completely method raise RuntimeError for an index created by a previous version, including versions 26.7 and 26.8. Such an index must be updated with the IndexUpdater class first:

• An index created by versions 22.7–26.8 is converted to the new format without re-indexing; the source documents are not required.
• An index created by versions 18.5–22.6 is re-indexed: the documents must be available at the paths from which they were indexed.
• An index created by a version earlier than 18.5 cannot be updated; create a new index and add the documents to it.

The updated index is written to another folder; the source index is not modified. Conversely, previous versions cannot open an index of version 26.10.

The following example demonstrates how to update an index of a previous version before opening it.

from groupdocs.search import Index, IndexUpdater
from groupdocs.search.common import VersionUpdateResult

index_folder = "./old-index"  # The folder of an index created by a previous version
updated_index_folder = "./updated-index"  # A new empty folder for the updated index

updater = IndexUpdater()
if updater.can_update_version(index_folder):
    # The source index is not changed; the content of the target folder is deleted before the update
    result = updater.update_version(index_folder, updated_index_folder)
    if result == VersionUpdateResult.UPDATED:
        with Index(updated_index_folder) as index:
            search_result = index.search("Einstein")
            print("Documents found:", search_result.document_count)

Documents found: 2

See Update index version and Index format and compatibility for the rules of the target folder, the values of VersionUpdateResult and the exceptions.

• Upgrade all applications together. All applications that use the same index folder, including applications built on GroupDocs.Search for .NET and all nodes of a search network, must be upgraded to version 26.10 at the same time. Update the index of each shard or create the shards anew before starting the nodes.
• Update indexes into new folders. Update each index with IndexUpdater into a new folder that is not the source folder, does not contain it and is not inside it; then switch the application to the new folder. See Update index version.
• Configure term dictionaries before adding documents. The alphabet, the tokenization dictionary, the character replacements, the stop words and the use_stop_words and use_character_replacements settings can be changed only while the index contains no documents. See Settings of an updated index.
• Merge updated indexes into compatible indexes. An index updated from a previous version has the replace_unpaired_surrogates and skip_blended_only_terms settings disabled. To merge it into a new index, create the new index with both settings set to False. See Merging compatibility.
• Check the number of search threads. Search now uses as many threads as there are processor cores by default. To restore single-threaded search with a deterministic order of results, set search_threads to NumberOfThreads.ONE. See Search index settings.
• Optimize the index to free disk space after deletions. Deleted documents are excluded from search immediately but are physically removed only by optimize. See Optimize index.
• Iterate over chunks by the token. The number of chunks of a search by chunks is not known in advance; continue while next_chunk_search_token is not None. See Search by chunks.
• Empty the backup folder of save_to once. save_to does not overwrite an index of a previous version. If an application regularly saves its index to the same folder, empty that folder once after the upgrade. See Saving an index to another folder.
• Check documents interrupted by cancellation. A document whose indexing was interrupted by cancellation is marked with indexed_with_error and is indexed again by the next update. See Indexing options.

The following changes in behavior compared with previous versions of GroupDocs.Search matter when you port code written for them or work with indexes they created.

The alphabet, tokenization_dictionary, character_replacements and stop_word_dictionary dictionaries, as well as the use_stop_words and use_character_replacements settings, can be changed only while the index contains no documents and is not performing an operation. Otherwise, a change raises RuntimeError; a call that does not change the content is allowed. Previously such a change was accepted, and documents indexed before and after it were processed by different rules. The other dictionaries (synonyms, homophones, aliases, the spelling corrector and document passwords) can be changed at any time.

Code that changes these dictionaries of an existing index must be moved to the place where the index is created. The following example demonstrates how to configure the dictionaries of a new index before adding documents.

from groupdocs.search import Index
from groupdocs.search.dictionaries import CharacterType

# Creating a new index; an existing index in the folder is overwritten
index = Index("./index", True)

# Configuring the term dictionaries while the index contains no documents
index.dictionaries.alphabet.set_range(["-"], CharacterType.BLENDED)
index.dictionaries.stop_word_dictionary.add_range(["and", "or", "the"])

# Indexing documents; after this, the dictionaries above cannot be changed
index.add("./documents")
print("Indexed documents:", len(index.get_indexed_documents()))

Indexed documents: 4

The merge method raises RuntimeError before making any changes if the merged indexes divide text into terms differently: they have different alphabets, tokenization dictionaries, stop words or character replacements, or different values of the use_stop_words, use_character_replacements, replace_unpaired_surrogates or skip_blended_only_terms settings. The exception is raised also when MergeOptions.is_async is True. In addition, the term dictionaries of the current index are no longer changed by merging, and a merge is not performed if a merged index is performing an operation or has failed; the error_occurred event reports it.

In the following example, the other index uses an additional stop word, so the indexes divide text into terms differently.

from groupdocs.search import Index
from groupdocs.search.options import MergeOptions

index = Index("./index")
index.add("./documents")

other_index = Index("./other-index", True)
other_index.dictionaries.stop_word_dictionary.add_range(["physics"])
other_index.add("./documents")

try:
    index.merge(other_index, MergeOptions())
except RuntimeError as exception:
    # The indexes divide text into terms differently; the message names the differences
    print(exception)

The indexes cannot be merged because they divide the text of documents into terms differently. The index 'other-index' differs from the current index in: StopWordDictionary.
[.NET System.InvalidOperationException]
   at .(IndexRepository , MergeOptions ,  , OperationType )
   at .(IndexRepository , MergeOptions ,  , OperationType )
   at .(Index , MergeOptions )
   at GroupDocs.Search.Index.Merge(Index index, MergeOptions options)
   at InvokeStub_Index.Merge(Obje
[TRUNCATED]

See Merge indexes and Merging compatibility.

With the default value of IndexSettings.search_threads, search now uses as many threads as there are processor cores; previously only one thread was used. As a result, the order of the terms found by regular expression, numeric range, fuzzy search and spelling correction may differ from run to run, and so may the order of search results that depends on it. The property can be changed at any time through the Index.index_settings property, and the new value is saved in the index; a value that is not a member of NumberOfThreads raises ValueError.

The following example demonstrates how to restore single-threaded search with a deterministic order of results.

from groupdocs.search import Index
from groupdocs.search.options import NumberOfThreads

index = Index("./index")
index.add("./documents")
index.index_settings.search_threads = NumberOfThreads.ONE  # The value is saved in the index

result = index.search("^relativ.*")
print("Documents found:", result.document_count)

Documents found: 2

The new replace_unpaired_surrogates and skip_blended_only_terms settings are enabled by default in a new index and are fixed when the index is created: unpaired surrogate characters are replaced with the Unicode replacement character (U+FFFD) instead of being removed, and terms that consist only of blended characters are not indexed. To keep the behavior of previous versions, set both settings to False before creating an index, as shown on the Index format and compatibility page. An index updated from a previous version has both settings disabled.

Other changes in text processing:

• Terms longer than 80 characters are not indexed.
• Letter case is converted in the same way on .NET Framework and .NET, so search results no longer depend on the platform.
• The lookup methods of the stop word, synonym and homophone dictionaries ignore letter case; words are kept in lower case.

A chunk no longer corresponds to an index segment: only the segments that may contain matches are searched, and the number of chunks is not known in advance. Code that relies on the number of segments must iterate over the chunks while next_chunk_search_token is not None.

from groupdocs.search import Index
from groupdocs.search.options import SearchOptions

index = Index("./index")
index.add("./documents")
options = SearchOptions()
options.is_chunk_search = True

result = index.search("Einstein", options)
document_count = result.document_count
while result.next_chunk_search_token is not None:
    result = index.search_next(result.next_chunk_search_token)
    document_count += result.document_count
print("Documents found:", document_count)

Documents found: 2

See Search by chunks.

Deleting documents (delete, or update for deleted files) no longer rewrites the index. Deleted documents are excluded from search immediately, but their data remain on disk until the index is optimized with the optimize method. Optimizing a very large index may leave several segments. See Delete indexed documents and Optimize index.

Searches and index-modifying operations (indexing, update, deletion, merging, optimization) no longer block each other. Each search works with a snapshot of the index taken when the search starts, so a search started during indexing does not find the documents being added. See Search thread safety.

The save_to method raises RuntimeError and does not change the target folder if the folder contains an index of another version or a damaged index, or if it is the folder of the index itself. An index of the current version in the target folder is replaced. An application that saves its index to the same folder, for example as a backup, has to empty that folder once after the upgrade.

If an indexing operation is cancelled, a document whose indexing was interrupted gets DocumentInfo.indexed_with_error = True and is indexed again by the next update. Documents added from streams, structures or extracted data cannot be re-read by the update and have to be added again.