Duplicate Handling Technical Documentation

Technical reference for the Duplicate Handling extension's configuration options, including how to define and chain multiple detection configurations using the configurations attribute.

Detailed documentation of the duplicate handling extension configuration options.

It is possible to define a list of configurations. All of them are encapsulated in the configurations attribute:

{  
  "configurations": [  
    {...},  
    {...},  
    {...}         
  ]  
}

Configuration parameters

Each configuration can use the following attributes.

Attribute Type Required Description
queues list false

IDs of queues where the duplicate detection should be performed. You can assign the extension to multiple queues and specify numerous actions for different queues in one instance.

If not present, the duplicate detection will be performed on all the queues to which the extension is assigned unless excluded_queues is set.
excluded_queues list false IDs of queues where the duplicate detection won't be performed. This parameter cannot be set along with the queue_ids.
trigger_events list false

List of Rossum events that trigger this configuration. Corresponds to selected hook events.

If not specified, the duplicate detection will be performed for all allowed trigger_events.

Possible values:

  • annotation_status

  • annotation_content

trigger_actions list false

List of Rossum actions that trigger this configuration. Corresponds to selected hook actions.

If not specified, the duplicate detection will be performed for all actions relevant for a selected trigger_event.

Possible values:

  • changed

  • initialize

  • updated

  • started

  • confirm

  • export

  • user_update
statuses list false
List of annotation status-couples in the format 
\{previous status}.\{current status\}
or 
\{previous status}->\{current
                                        status\}
defining when the processing of logic should start.

If not present, duplicate detection will be performed for all allowed status changes.
logic list false List of logic objects. More details are in the section below.

Logic parameters

Detailed specification of the logic object attributes.

Attribute Type Required Default value Description
rules list false [] List of rules based on which the duplicates will be detected. More details are in the section below.
scope object false

The scope of the duplicate detection logic is where the extension will look for duplicates. The section below provides more details.

If not specified, the default scope will be used.
timestamp object false \{\} How long in the past the extension should look for duplicates. More details are in the section below.
matching_flow list false

Definition of how to combine duplicates detected by individual rules. It can be a list of IDs or logical and operations between rule ids.

Individual list elements behave like logical or operations.
search_query object false

[Beta] Alternatively, you can also define a search_query instead of the combination of rules, matching_flow, scope, and timestamp. More details are in the section below.

If defined, such configuration will be preferred.
actions list true List of actions to be performed if duplicates were detected.

Rule parameters

Detailed specification of the rule object attributes.

Attribute Type Required Description
id int true ID of the rule object. This can be referred to in the matching_flow field. IDs must be unique.
attribute enum true

Type of the rule.

Possible values:

  • field

  • relation

  • filename
field_schema_id string Required for field rule. The schema_id of the field that should be used to compare for duplicates.

Scope parameters

Detailed specification of the scope object attributes.

Attribute Type Required Default value Description
object enum false queue

Scope, where the extension should look for duplicates.

It is recommended to use the queue scope and define as few queues as possible. Detecting duplicates in the whole workspace or organization can be very expensive.

Possible values:

  • queue

  • workspace

  • organization

ids list false

The object's IDs (queue/workspace) should be considered during detection.

Default is the context of the incoming annotation.
statuses list false

The statuses of annotations that should be considered during detection. Default is all.

Possible values:

  • created

  • importing

  • failed_import

  • split

  • to_review

  • reviewing

  • in_workflow

  • confirmed

  • rejected

  • exporting

  • exported

  • failed_export

  • postponed

  • deleted

  • purged

Timestamp parameters

Detailed specification of the timestamp object attributes. If not specified, it searches without any limit.

Attribute Type Required Description
action enum false

What annotation date condition should be considered when checking the timespan.

Possible values:

  • arrived_at_before

  • arrived_at_after

  • assigned_at_before

  • assigned_at_after

  • confirmed_at_before

  • confirmed_at_after

  • modified_at_before

  • modified_at_after

  • exported_at_before

  • exported_at_after

timespan int false The time span in days for the selected date condition

Search query parameters

[Beta] Search query for the duplicate detection. Uses more powerful searching based on the /search endpoint.

If this object is defined, it will be preferred for duplicate detection. matching_flow, rules, scope and timestamp objects will be ignored.

It is possible to reference some various values using the following syntax easily:

  • annotation and document object attributes using the dot . notation: {annotation.queue} or {document.created_at}

  • datapoint values using the @ operator: @{invoice_id}

  • datetime features

Attribute Type Required Default value Description
query object false {} A subset of MongoDB Query Language. See Rossum api docs for specifics and more details.
query_string object false {} Object with configuration for full-text search. See Rossum api docs for specifics and more details.

Actions parameters

All actions have one common required attribute - type - specifying the type of action that will be performed. Other attributes vary based on the type. Possible action types:

  • apply_label

  • fill_field

  • forward_annotation

  • mark_duplicate

  • show_message

  • stop_automation

Apply label action

Apply a label to the annotation that triggered the hook.

Attribute Type Required Description
type enum true The type of action that should be performed. Possible value: apply_label.
label string true name of the label that should be applied

Fill field action

Fill a field with a defined value.

The following placeholders can be used in the value_to_fill attribute:

  • duplicate_ids - list of IDs of detected duplicate annotations

  • now - current datetime object. Additional operations can follow.

  • timedelta - timedelta object. Allows for operations like + or - together with now.

  • annotation - nested object containing attributes of the annotation that triggered the hook.

  • %ANNOTATION_ID% - expression to fill list of detected duplicates

Example: 
Detected {duplicate_ids | length} duplicates.<br>
                    Date of detection: {now.strftime('%m/%d/%Y')}.

Detailed specification of the fill_field action attributes:

Attribute Type Required Default value Description
type string true The type of action that should be performed. Possible value: fill_field
fill_only_if_empty bool false false If true, the field will only be filled if it was originally empty.
field_to_fill string true The schema ID of the datapoint in Rossum schema to fill with custom value if duplicate is detected.
value_to_fill string true The custom value to be filled to the datapoint in field_to_fill. Various placeholders can be used to create the value.

Forward annotation action

Forward the annotation to a different queue and/or different status.

Detailed specification of the forward_annotation action attributes:

Attribute Type Required Description
type string true The type of action that should be performed. Possible value: forward_annotation.
target_queue inst false ID of the target queue where the duplicate annotation should be placed. If not specified, the current annotation queue will be used.
target_status string false The target status of the annotation after it is forwarded to another queue. By default, the status is kept the same. If not specified, the current annotation status will be used.

Mark duplicate action

Create a new relation object. Ignored if the relation already exists.

Attribute Type Required Description
type string true The type of action that should be performed. Possible value: mark_duplicate

Show message action

Show a message to the user in the UI. The type and content of the message can be customized.

Following placeholders can be used in the `message`` attribute:

  • duplicate_ids - list of IDs of detected duplicate annotations

  • now - current datetime object. Additional operations can follow.

  • timedelta - timedelta object. Allows for operations like + or - together with now.

  • annotation - nested object containing attributes of the annotation that triggered the hook.

  • %ANNOTATION_ID% - expression to fill list of detected duplicates

Detailed specification of the show_message action attributes:

Attribute Type Required Default value Description
type string true The type of action that should be performed. Possible value: show_message.
message string true Content of the message that will be displayed. Various placeholders can be used to create the message.
message_type enum true info Possible values: error, info, warning.

Stop automation action

Stop the automation by returning an error message. Only applicable for annotation_content.initialize event.

Attribute Type Required Description
type string true The type of action that should be performed. Possible value: stop_automation

Configuration examples

Simple detection with error message as an action

This configuration is triggered only on the initialize, started or updated actions of annotation_content event. It will detect duplicates based on the invoice_id field. If any other annotation in the same queue has the same invoice_id, an error will be shown in the annotation screen.

{
    "configurations": [
        {
            "trigger_events": ["annotation_content"],
            "trigger_actions": ["initialize", "started", "updated"],
            "logic": [
                {
                    "rules": [
                        {
                            "attribute": "field",
                            "field_schema_id": "invoice_id"
                        }
                    ],
                    "actions": [
                        {
                            "type": "show_message",
                            "message_type": "error",
                            "message": "Detected {duplicate_ids | length} duplicates"
                        }
                    ]
                }
            ]
        }
    ]
}

Complex detection logic with multiple actions

This configuration is triggered only on the changed action of annotation_status event and only if the status is changed from importing to to_review or from to_review to postponed.

It will detect duplicates based on a combination of several rules. The detection will be scoped on the queue with ID 12345 and only annotations in selected statuses will be considered. The detection will also search only among annotations that arrived at Rossum in the last 60 days. If duplicates are detected, multiple actions will be performed in the order defined in the configuration.

{
    "configurations": [
        {
            "trigger_events": ["annotation_status"],
            "trigger_actions": ["changed"],
            "statuses": ["importing->to_review", "to_review->postponed"],
            "logic": [
                {
                    "matching_flow": ["1and2and3", "4"],
                    "rules": [
                        {
                            "id": 1,
                            "attribute": "field",
                            "field_schema_id": "invoice_id"
                        },
                        {
                            "id": 2,
                            "attribute": "field",
                            "field_schema_id": "sender_name"
                        },
                        {
                            "id": 3,
                            "attribute": "filename"
                        },
                        {
                            "id": 4,
                            "attribute": "relation"
                        }
                    ],
                    "scope": {
                        "object": "queue",
                        "ids": [12345],
                        "statuses": ["confirmed", "exported", "deleted"]
                    },
                    "timestamp": {
                        "action": "arrived_at_after",
                        "timespan": 60
                    },
                    "actions": [
                        {
                            "type": "fill_field",
                            "field_to_fill": "duplicate",
                            "value_to_fill": "Duplicate of %ANNOTATION_ID%"
                        },
                        {
                            "type": "forward_annotation",
                            "target_queue": 123456,
                            "target_status": "postponed"
                        },
                        {
                            "type": "mark_duplicate",
                            "message": "Marked as duplicate"
                        },
                        {
                            "type": "show_message",
                            "message_type": "error",
                            "message": "Detected {duplicate_ids | length} duplicates"
                        }
                    ]
                }
            ]
        }
    ]
}

Detection based on Search Query

Such configuration is triggered only on the initialize, started or updated actions of the annotation_content event. It will detect duplicates based on the invoice_id field among annotations in the same queue and created in the last 200 days (
{(now -
                    timedelta(days=200)).isoformat()}
).
{
    "configurations": [
        {
            "trigger_events": ["annotation_content"],
            "trigger_actions": ["initialize", "started", "updated"],
            "logic": [
                {
                    "search_query": {
                        "query": {
                            "$and": [
                                {
                                    "field.invoice_id.string": {
                                        "$eq": "@{invoice_id}"
                                    }
                                },
                                {
                                    "queue": {
                                        "$in": ["{annotation.queue}"]
                                    }
                                },
                                {
                                    "created_at": {
                                        "$gt": "{(now - timedelta(days=200)).isoformat()}"
                                    }
                                }
                            ]
                        }
                    },
                    "actions": [
                        {
                            "type": "show_message",
                            "message_type": "error",
                            "message": "Duplicates detected: %ANNOTATION_ID%"
                        }
                    ]
                }
            ]
        }
    ]
}

Detection based on Search Query — Gotchas

Normalized vs. raw datapoint values (the most common pitfall)

Rossum content datapoints can contain both:

  • value (raw, as seen on the document, e.g. "1 500.00", "15.1.2024")

  • normalized_value (machine-normalized, e.g. "1500.00", "2024-01-15")

In our setup:

  • json-templating @{schema_id} resolves to normalized_valuewhen available, otherwise value.

  • The search index behind field.<schema_id>.string matches the rawvalue formatting.

Result:

  • field.amount_due.string == "@{amount_due}" can become "1 500.00" == "1500.00" → no match.

  • field.date_issue.string == "@{date_issue}" can become "15.1.2024" == "2024-01-15" → no match.

Fix: Use typed fields for normalized matching:

  • Amounts: use .number

  • Dates: use .date

Example:

{
  "search_query": {
    "query": {
      "$and": [
        {"field.invoice_id.string": {"$eq": "@{invoice_id}"}},
        {"field.amount_due.number": {"$eq": "@{amount_due}"}},
        {"field.date_issue.date": {"$eq": "@{date_issue}"}},
        {"queue": {"$in": ["{annotation.queue}"]}}
      ]
    }
  }
}