Components of Export pipelines

An export pipeline consists of the following components chained together:

1. Custom Format Templating Purge, a cleaning mechanism that prepares the pipeline for export.
2. Custom Format Templating, which prepares the desired export format.
3. REST API Export, which exports the prepared data to a REST API and stores the response.
4. Data Value Extractor, which extracts important data from the API response and stores it in the annotation object, such as the downstream document ID and HTTP status codes.
5. Export Evaluator, which determines whether the export succeeded or failed.
6. SFTP Export, which uploads the prepared data to SFTP or S3 file storage.

How to use Export pipelines

Export pipeline components are typically connected by the standard run-after extension-chaining mechanism. The following examples demonstrate several extension chains.

Simple SFTP export pipeline

1. Custom Format Templating prepares the extracted data in the desired format.
2. SFTP Export stores the data on an SFTP server or in S3.

Configure the export_rules of your SFTP extension as follows to accept the Custom Format Templating output:

{
  "path_to_directory": "/Path_To_Your_Directory",
  "export_object_configurations": [
    {
      "type": "custom_format",
      "filename_template": "{export_order_name}.json",
      "export_reference_key": "your_reference_key"
    }
  ]
}

Simple API export pipeline

1. Optionally, Pipeline Cleaning removes previous export data. This is useful for debugging.
2. Custom Format Templating prepares the extracted data in the desired format.
3. REST API Export calls an external API service and sends the prepared data. The extension also stores returned values, including the status code.
4. Data Value Extractor stores required information, such as the status code, in the document.
5. Export Evaluator uses a condition, such as a status code of 200 or 201, to determine whether the export succeeded.

Complex API export pipeline