Separate data extraction Leave feedback

The feature described in this article makes it possible to separate the operations of extracting data from a document and adding the extracted data to the index. The extracted data can be easily serialized and deserialized as needed: the serialize method returns bytes, which can be stored or sent to another process or server, and ExtractedData.deserialize restores the data from them.

This feature can be useful because the operation of extracting text and other data from documents can be very long for some formats. At the same time, adding data to the index is a rather lightweight operation. Therefore, separating these operations significantly reduces the time during which the index is busy performing operations, and adding more data extractors increases the overall indexing performance. Data extractors can even work on separate servers, completely freeing the indexing server from extracting operations.

The Extractor class is used to extract data from documents. Upon completion of the operation, the extract method returns an instance of the ExtractedData class, which is used directly to add to the index.

The Extractor class has the following members in addition to the extract method:

• The settings property returns an ExtractorSettings object. Its index_type property specifies the type of the index for which data are extracted (the default value is IndexType.NORMAL_INDEX); set it to the type of the index to which the data will be added. Its temp_folder property specifies the folder for temporary files.
• The error_occurred, password_required and image_preparing events, which work like the index events with the same names.

The ExtractionOptions class contains the same extraction options as the IndexingOptions class inherited from BaseOptions (see Indexing options), as well as the custom_extractor property, which specifies a custom text extractor for the document.

The following example demonstrates how to perform separate data extraction and indexing.

from groupdocs.search import Extractor, Index
from groupdocs.search.common import Document, ExtractedData
from groupdocs.search.options import ExtractionOptions, IndexingOptions

document_path = "./sample.pdf"

# Extracting data from a document
extractor = Extractor()
extractor.error_occurred += lambda sender, args: print(args.message)
document = Document.create_from_file(document_path)
extraction_options = ExtractionOptions()
extraction_options.use_raw_text_extraction = False
extracted_data = extractor.extract(document, extraction_options)

# Serializing the data
array = extracted_data.serialize()

# Deserializing the data
deserialized_data = ExtractedData.deserialize(array)

# Creating an index
index = Index("./index")

if deserialized_data.is_extraction_successful:
    # Indexing the data
    data = [
        deserialized_data
    ]
    index.add(data, IndexingOptions())

    # Searching in the index
    result = index.search("Einstein")
    print("Documents found:", result.document_count)

Documents found: 1

Note that when indexed documents change and need to be updated in the index, the same code must be executed. That is, data must be extracted separately from the updated documents and then the extracted data must be added to the index. The update method does not update documents added from extracted data.

You may find more code examples in our GitHub repository:

• GroupDocs.Search for Python via .NET examples

Along with the full-featured library, we provide a free online document search app. You are welcome to search over your PDF, DOC, DOCX, PPT, PPTX, XLS, XLSX and more with our Free Online Document Search App.