Database Client Options | emtelligent Documentation

The database client provides many options for configuring how it processes documents and saves them to a database; this document describes all of them, grouped by the command they belong to.

All these descriptions can also be found using emtellipro-db-client --help and emtellipro-db-client COMMAND --help (where COMMAND is one of the commands supported by the client).

The database client’s options are grouped into two parts: the global options and the commands options. The global options apply to the client overall and are shared by all the commands, and the command options are specific to the given command.

When running the database client, you’ll format the command as follows:

$ emtellipro-db-client [GLOBAL-OPTIONS] COMMAND [OPTIONS]

Where COMMAND may be something like create-db or process. The [GLOBAL-OPTIONS] will contain options about configuring the client (such as which database to be used, and how logging will be handled) and [OPTIONS] will be options for the command being run (e.g. the process command has options for connecting to the NLP API, but these would be irrelevant for create-db).

On top of the regular command-line options, the database client also allows passing in a configuration file using the emtellipro-db-client --config global option. This configuration file allows the user to store commonly used options in one place to be re-used by multiple processing jobs.

Configuration file

The configuration file is in the TOML¹ format. Our limited use of the TOML format will look a lot like INI files, although TOML is better defined.

A simple configuration file will look something like the following:

[emtellipro-db-client]
# GLOBAL-OPTIONS will go here
log-file = 'emtellipro.log'
[general]
# options shared by multiple commands can go in here, to avoid duplication
database = "URL of your database"
access-key = "your access key"
shared-secret = "your secret key"
server = "emtelliPro server URL"
[process]
# command-specific OPTIONS will go here
max-retries = 10

The global options all go in the [emtellipro-db-client] section, and command specific options will go into a section named the same as the command.

The [emtellipro-db-client] section is required, even if empty, but no other sections are required. Any options not set in the configuration file will simply use the defaults, or be overridden by passing command-line options.

Defaults and overriding options

Most options have defaults which will be used if not overridden by the configuration file or command-line options. The priorities of the option values are the following:

The defaults for each option are used if not otherwise set.
The options in the configuration file override the defaults (for the options specified in the configuration file).
The options passed as command-line options override the options set in the configuration file and the defaults in (1).

As a simple example, the default for log-level is info. If the configuration file specifies log-level = "error", then only error messages will be logged. If you then also specify --log-level "debug", then the logging level will be set to “debug”, and all debugging messages will be printed.

This hierarchy allows you to (1) only set the options you care about, (2) place commonly-used options in a configuration file, and (3) set options specific to the given job on the command line without having to edit the configuration file for each job.

Option descriptions

All options are described below, with these columns:

Config option name
This contains the config-file option name. This will be the exact same as the command-line option, without the leading hyphens.

Type
This is the type of argument this option takes. Some options will take a string as argument, some will take a number, and some are boolean options.

Boolean options (also called “flags”) will usually have a complementing --no-* version which is available simply to be more explicit when running the client; omitting the flag is the exact same as specifying the --no-* option, so specifying --no-* acts as a reminder that the option is disabled.

Boolean options in the configuration file are specified using the true or false values.

Required
Required options must be passed in.

Default
This is the default value for the given option. Options that have defaults are optional.

Descriptions of all the options can also be found using the --help option. This section may sometimes go into more detail than is found in the help message.

Global options

These options apply to the database client as a whole, and are not command-specific. When passed on a command line, they immediately follow emtellipro-db-client, and precede the command name. In the configuration file, they go in the [emtellipro-db-client] section.

-c, —config

config
n/a

type
string: path to a file

Specifies the path to the configuration file, if used. The configuration file is described above.

—log-file

config
log-file

type
string: path to a file

default
errors.log

File path for where to store logging output.

—log-level

config
log-level

type
string: one of error, warning, info, or debug

default
info

The minimum level of log messages to send to the log file.

Shared options

These options are shared by multiple commands. They can be set in the [general] section of the config file, or on each specific command.

-o, —output, —database

config
database, output

type
string: a URL

required
yes

shared with
create-db, migrate, store-result, process

The database to connect to, formatted as a URL containing all necessary connection settings. For most database systems, a SQLAlchemy-compatible URL format is necessary, but storing to files on disk is also supported using special prefixes, such as: jsonl://PATH, json://PATH, csv://PATH, and raw://PATH which allow storing results in jsonl, json, csv and unprocessed result files (respectively) in the provided directory PATH.

The following prefixes are supported:

Database URL:
e.g. postgresql://user:password@localhost:15432/sdk_db

If you pass a database URL, the output data will be stored in that database.

Note that if you’re using Snowflake the format is documented in Snowflake’s SQLAlchemy driver documentation. This differs from other URL formats Snowflake describes on other pages (for use with other tools), but as we pass the URL unchanged to Snowflake’s driver, only the format describe on that page will work.

csv, csv+table, table+csv:
e.g.

csv+table://relative/path/
csv:///absolute/path/to/directory/ (note that the prefix is csv:// so this path starts with the third /)

The output path is a directory in which CSV files will be stored, matching the schema of the regular database tables. There is a CSV file for each database table, with a filename corresponding to the table name. These files can be imported directly into a Snowflake database.

json, json+table, table+json:
e.g. json+table://path/to/json/

The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above.

Each file will contain a list filled with JSON objects, where the keys match the table columns.

jsonl, jsonl+table, table+jsonl:
e.g. jsonl+table://path/to/jsonl/

This is similar to the JSON format above, the only difference being that each line in the file is a complete JSON object (instead of the entire file being a list of objects, the objects are written one per line).

raw, raw+json, json+raw:
e.g. raw+json://path/to/raw-results/

The output path is a directory in which the raw JSON results from the NLP API are stored. The processed documents will be split into multiple tasks, and the results are stored in files with the filename being the task ID.

For each result JSON <task_id>.json, there are also corresponding <task_id>.input and <task_id>.map files, containing serialized input documents and mapping from input filenames to document IDs, respectively.

raw+jsonl, jsonl+raw:
e.g. raw+jsonl://path/to/results.jsonl

The output path is a file (not a directory like the other formats). This file will contain JSONL data, where each line contains the raw JSON result from the NLP API. The output file will be appended to.

There will also be a corresponding <filename>.map file containing a mapping from input filenames to document IDs.

—schema

config
schema

type
string

default
legacy

shared with
create-db, migrate, store-result, process

The name of the database schema to use. Run emtellipro-db-client process --help (or similar for any of the other commands that accept this option) to see the list of available schemas. Additional schemas may be registered via the emtellipro.plugins.schemas entry point.

—access-key

config
access-key

type
string

required
yes

shared with
get-user, process, cancel

The access key used when authenticating with the NLP API. Can also be set via the EMTELLIPRO_ACCESS_KEY environment variable.

—shared-secret, —secret-key

config
shared-secret

type
string

required
yes

shared with
get-user, process, cancel

The shared secret used when authenticating with the NLP API. Can also be set via the EMTELLIPRO_SHARED_SECRET environment variable. --secret-key is accepted as an alias on the command line.

—api-key

config
api-key

type
string

required
yes

shared with
get-user, process, cancel

The API key to use when authenticating to the NLP API. This is mutually exclusive with the --access-key and --shared-secret options. Can also be set via the EMTELLIPRO_API_KEY environment variable.

—server

config
server

type
string: a URL

required
yes

shared with
get-user, process, cancel

The URL of the NLP API server you’d like to send the reports to. Can also be set via the EMTELLIPRO_SERVER environment variable.

—snowflake-private-key-path

config
snowflake-private-key-path

type
string: path to a file

default
None

shared with
create-db, migrate, store-result, process

When connecting to a Snowflake database to store results, one can either use password-based authentication by setting the password in process --output, or by setting this option and using key-pair authentication.

-S, —save-opt

config
save-opt

type
string

multiple
true

New in version 7.0.

This can be passed multiple times to set the save options for the schema you’re using. Options that take arguments can be set like -S job_id=1234, and boolean options can be just passed in by name like -S store_json.

In the config file, there should be a section called:

[process]
[process.save_options.legacy]     # for the legacy schema
job_id = 1234
store_json = true

For the legacy schema, the available options are described in the SaveOptions class. During interactive CLI use, you may wish to run the list-options command to see the available options: “list-options” command.

The following values will be injected into the save options by the client when running (since they’re dynamic and can’t be set ahead of time):

task_id with the task ID received from the NLP API
engine_version with the version string returned by the NLP API with the result JSON
sql_query with the value of --sql-query, if present.

If the schema you’re using doesn’t include those attributes in its save options, they’ll be ignored.

For the legacy schema in version 6 and earlier, the save options were separate options like --store-json instead of --save-opt store_json.

See the migration guide for how to use the new save options if upgrading to v7: Save options replace the ``--store-\*flags.

For the available options when using the legacy schema see the API documentation: SaveOptions.

”process” command

The process command reads in documents from the input paths, sends them to the NLP API to be processed, and stores the results in the target output (either database of files).

Command-line usage

$ emtellipro-db-client [GLOBAL-OPTIONS] process [OPTIONS] INPUT_PATH...

Each INPUT_PATH passed in as argument may be:

a file path containing document(s) to be loaded
a directory containing files (subdirectories are searched recursively)
the URL of a database (only one input path may be provided for this case).

See Input Sources for more details on how to use the different input sources.

Options

These options are for the process command, and immediately follow the process command on the command-line; in the configuration file they go in the [process] section.

Program: process

-o, —output, —database

config
database, output

type
string: a URL

required
yes

shared with
create-db, migrate, store-result, process

The following prefixes are supported:

Database URL:
e.g. postgresql://user:password@localhost:15432/sdk_db

If you pass a database URL, the output data will be stored in that database.

csv, csv+table, table+csv:
e.g.

csv+table://relative/path/
csv:///absolute/path/to/directory/ (note that the prefix is csv:// so this path starts with the third /)

json, json+table, table+json:
e.g. json+table://path/to/json/

The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above.

Each file will contain a list filled with JSON objects, where the keys match the table columns.

jsonl, jsonl+table, table+jsonl:
e.g. jsonl+table://path/to/jsonl/

raw, raw+json, json+raw:
e.g. raw+json://path/to/raw-results/

raw+jsonl, jsonl+raw:
e.g. raw+jsonl://path/to/results.jsonl

There will also be a corresponding <filename>.map file containing a mapping from input filenames to document IDs.

—schema

config
schema

type
string

default
legacy

shared with
create-db, migrate, store-result, process

—access-key

config
access-key

type
string

required
yes

shared with
get-user, process, cancel

The access key used when authenticating with the NLP API. Can also be set via the EMTELLIPRO_ACCESS_KEY environment variable.

—shared-secret, —secret-key

config
shared-secret

type
string

required
yes

shared with
get-user, process, cancel

The shared secret used when authenticating with the NLP API. Can also be set via the EMTELLIPRO_SHARED_SECRET environment variable. --secret-key is accepted as an alias on the command line.

—api-key

config
api-key

type
string

required
yes

shared with
get-user, process, cancel

—server

config
server

type
string: a URL

required
yes

shared with
get-user, process, cancel

The URL of the NLP API server you’d like to send the reports to. Can also be set via the EMTELLIPRO_SERVER environment variable.

—snowflake-private-key-path

config
snowflake-private-key-path

type
string: path to a file

default
None

shared with
create-db, migrate, store-result, process

-s, —state <state>

config
state

type
string: path to a file

required
no

The path to a file in which to store the current processing state. This is used for restarting processing if the client has been stopped early (by crashing or hitting CTRL-C). Simply re-run the same command with the same state file and same options to continue where it left off.

If processing has completed and the client is re-run with the same state file, it will still “continue where it left off”, meaning it will show progress of 100% and exit again.

The state file should be different for each distinct processing submission. This can be accomplished by setting the state file name to a unique name for each submission, or deleting the state file between submissions.

Omitting the state file path will result in a temporary state file being created, which will be deleted when processing has completed successfully. The path to the state file will be printed when the process command is executed.

—category

config
category

type
string

default
Clinical

The document category specified when processing reports. If the input documents come from JSON and SQL, the per-document category set on the input documents will override this option.

This can also be a ‘auto’, to have the NLP API infer the category.

—subcategory

config
subcategory

type
string

default
generic

The document subcategory specified when processing reports. If the input documents come from JSON and SQL, the per-document subcategory set on the input documents will override this option.

When having the NLP API infer the category (by setting it to ‘auto’), this subcategory option will be ignored.

—document-type

config
document-type

type
string

default
plain

The document type parameter sent to the engine. See the NLP API documentation for available values.

—section-label

config
section-label

type
string

default
None

The section label used for all documents, if you’d like to use the NLP API’s process_with_section_labels feature. Setting this option will tell the NLP API to assume the entire input document is part of a single section (with this section label).

—features

config
features

type
string

default
all features

The NLP API features to request when processing the documents. This is a comma-separated list of feature names with no spaces. In the configuration file you may use either a single comma-separated string, or a list of strings (as per TOML format). See NLP API documentation for what features are available.

Available features:

-j, —jobs

config
jobs

type
integer

default
1

How many submit/process/store jobs to run in parallel. Each job will take a batch of documents and process them with the NLP API, and store them to output.

—file-encoding

config
file-encoding

type
string

default
auto

The file encoding to use when reading input files. If not set, it will be autodetected. If you know the file encoding ahead of time, it’s a good idea to set it here to avoid the overhead of autodetection.

-S, —save-opt

config
save-opt

type
string

multiple
true

New in version 7.0.

In the config file, there should be a section called:

[process]
[process.save_options.legacy]     # for the legacy schema
job_id = 1234
store_json = true

The following values will be injected into the save options by the client when running (since they’re dynamic and can’t be set ahead of time):

task_id with the task ID received from the NLP API
engine_version with the version string returned by the NLP API with the result JSON
sql_query with the value of --sql-query, if present.

If the schema you’re using doesn’t include those attributes in its save options, they’ll be ignored.

For the legacy schema in version 6 and earlier, the save options were separate options like --store-json instead of --save-opt store_json.

See the migration guide for how to use the new save options if upgrading to v7: Save options replace the ``--store-\*flags.

For the available options when using the legacy schema see the API documentation: SaveOptions.

—sql-query

config
sql-query

type
string

optional

If the input path is a database URL, this option must be set to tell the client what query to use to retrieve documents from the input database.

The query must return at least id and text columns; any extra fields are stored in the documentmetadata table. If category and subcategory are included, they will be stored as original_category and original_subcategory.

This can be combined with --text-type if the text column in the SQL query contains non-plaintext data.

The query will be processed in a transaction that executes a COMMIT at the end of loading, so the SQL query is able to make modifications to the source database.

—sql-count-query

config
sql-count-query

type
string

required
no

Optional query that returns the number of rows the query in --sql-query will return. If not provided, a default SELECT COUNT query is generated based on --sql-query.

—sql-limit

config
sql-limit

type
integer

required
no

Optional limit on how many rows to read from --sql-query. It’s assumed the query will return at least this many rows, but if it returns fewer the progress bars will not reach 100%.

—text-type

config
text-type

type
string

default
text

This value specifies what type of data is stored in the ‘text’ column when reading input from a database using --sql-query.

This may be any of:

ccd
The text column must contain an XML document. The XML will be parsed as a CCD document, and the metadata will also be included, same as if it were parsed from a file.

pdf
The text column must contain raw PDF bytes.

text
This is the default; the text column must contain plaintext.

—quiet

config
quiet

type
boolean / integer

default
false / 0

Make the client quieter. Passing this once disables the progress bar, but sends error messages to stderr; passing it twice hides the error messages (they’ll still be present in the log file). In the config file, you can set a number to indicate how quiet you wish the client to be.

—batch-size

config
batch-size

type
integer

default
100

Maximum number of documents to submit at once. If set to -1, the number of documents submitted in each shard is automatically determined based on document size (to submit the maximum number of documents that fit within the NLP API server’s submission size limit).

This option is useful for restricting the shard size further than what is technically allowed by the server; this can be helpful for lowering memory usage.

—skip-database-checks / —no-skip-database-checks

config
skip-database-checks

type
boolean

default
false

Skip checks for whether the database schema is valid. By default the client will check on startup whether all the tables and columns in the target database have the correct schema; this avoids potential errors when trying to save to a database that needs to be migrated (or has not been created), but there’s overhead to doing this. If you’ve already confirmed the database schema is correct, you should disable the checks.

—max-retries

config
max-retries

type
integer

default
5

Maximum number of times to retry failed API requests. Failures may be due to network issues, so this should be a number greater than 0.

—retry-failed

config
retry-failed

type
bool

default
false

Retry documents that received a processing status of ‘error’ from the NLP API.

—store-failed / —no-store-failed

config
store-failed

type
boolean

default
false

Store details of failed documents in the documents table. You can find these by querying for documents.processing_status = 'error'.

—doc-id-filepath

config
doc-id-filepath

type
boolean

default
false

Use the absolute file path as the document ID for NLP API input documents. By default random UUIDs are generated to anonymize input document locations when submitting documents to the NLP API, but this option can be useful for debugging.

For JSON files this will include the document ID from the file appended to the file path.

—poll-freq

config
poll-freq

type
integer

default
1

Number of seconds to wait between API calls to check processing status. Polling too frequently can put extra load on the NLP API, so for non-interactive uses of the client, it’s helpful to set this value higher. EmtelliPro will respond as soon as processing is completed, so setting this value higher than 1 second will not lead to slower processing, but it will cause the progress bar to be updated less often.

—filetype

config
filetype

type
string

default
detected from file extension

Assume all input files have the given file type, disabling automatic detection of file type based on file extension. See the --help option for available options.

Statistics explanation

At the end of execution, the client will provide some statistics about the processing times of the documents.

Documents processed: 137
  success: 137
Average time per document per stage:
  loading     0.005s/doc  (0.7s over 137 docs)
  task-setup  0.000s/doc  (0.0s over 137 docs)
  processing  1.196s/doc  (163.8s over 137 docs)
  saving      0.016s/doc  (2.2s over 137 docs)
Mean batch size: 9.79
Mean concurrency: 10.0 jobs
Throughput:
  loading     0.001s/doc (0.4%)
  task-setup  0.000s/doc (0.0%)
  processing  0.181s/doc (98.3%)
  saving      0.002s/doc (1.3%)
Overall throughput: 5.44 doc/s
Run statistics:
  Start time: 2026-08-15 00:35:19.002561 UTC
  Completed: 2026-08-15 00:35:44.243688 UTC
  Cumulative run time: 0:00:25.241127

The different statistics are as follows:

Documents processed
Includes total count, as well as count per processing status.

Average time per document per stage
This is the average time taken in each stage while loading/processing/saving each batch, when treating each batch independently. This tells you how long each document independently took in each stage (i.e. if you use -j1 and -j2 you should see similar numbers).

Mean batch size
The average batch size across all batches. This will likely be slightly lower than the requested --batch-size since the last batch may not be completely filled.

Mean concurrency
The average value of --jobs across all runs. If the process is stopped and restarted with a different number of jobs, this value can change.

Throughput
The mean time taken per document per stage when taking concurrency into account. For example, if two documents are processed in parallel, the processing time should appear halved per document than if they were processed one at a time.

This differs from the “average time” reported above which does not take concurrency into account.

Overall throughput
The number of documents per second that were processed (i.e. that went through all the stages listed in Throughput).

Run statistics
The overall statistics across all runs using the same state file. The start time is the time of the first processing run, and the cumulative run time across all runs.

”get-user” command

The get-user command retrieves the current user’s details. This is useful for checking that the authentication keys and server URL are configured correctly.

Command-line usage

$ emtellipro-db-client [GLOBAL-OPTIONS] get-user

If you run it, you’ll see output such as the following:

Testing emtelliPro connection...
Successful connection to https://emtellpro/
Connected as USER@example.com

Options

These options are used by the get-user command.

See the “Shared options” section above for options this command shares with others:

Program: get-user

—server

config
server

type
string: a URL

required
yes

shared with
get-user, process, cancel

The URL of the NLP API server you’d like to send the reports to. Can also be set via the EMTELLIPRO_SERVER environment variable.

—access-key

config
access-key

type
string

required
yes

shared with
get-user, process, cancel

The access key used when authenticating with the NLP API. Can also be set via the EMTELLIPRO_ACCESS_KEY environment variable.

—shared-secret, —secret-key

config
shared-secret

type
string

required
yes

shared with
get-user, process, cancel

The shared secret used when authenticating with the NLP API. Can also be set via the EMTELLIPRO_SHARED_SECRET environment variable. --secret-key is accepted as an alias on the command line.

—api-key

config
api-key

type
string

required
yes

shared with
get-user, process, cancel

”create-db” command

The create-db command will create the tables or files necessary for the process command to use afterwards.

Command-line usage

$ emtellipro-db-client [GLOBAL-OPTIONS] create-db [OPTIONS]

The target output is specified using --output. If the output is a database, the tables will be created. For file-based outputs, the directories and files will be created (when relevant, the files will contain headers).

Options

These options are used by the create-db command.

See the “Shared options” section above for options this command shares with others:

Program: create-db

-o, —output, —database

config
database, output

type
string: a URL

required
yes

shared with
create-db, migrate, store-result, process

The following prefixes are supported:

Database URL:
e.g. postgresql://user:password@localhost:15432/sdk_db

If you pass a database URL, the output data will be stored in that database.

csv, csv+table, table+csv:
e.g.

csv+table://relative/path/
csv:///absolute/path/to/directory/ (note that the prefix is csv:// so this path starts with the third /)

json, json+table, table+json:
e.g. json+table://path/to/json/

The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above.

Each file will contain a list filled with JSON objects, where the keys match the table columns.

jsonl, jsonl+table, table+jsonl:
e.g. jsonl+table://path/to/jsonl/

raw, raw+json, json+raw:
e.g. raw+json://path/to/raw-results/

raw+jsonl, jsonl+raw:
e.g. raw+jsonl://path/to/results.jsonl

There will also be a corresponding <filename>.map file containing a mapping from input filenames to document IDs.

—schema

config
schema

type
string

default
legacy

shared with
create-db, migrate, store-result, process

—snowflake-private-key-path

config
snowflake-private-key-path

type
string: path to a file

default
None

shared with
create-db, migrate, store-result, process

”migrate” command

The migrate command migrates the schema in the output database to the newest schema.

Command-line usage

$ emtellipro-db-client [GLOBAL-OPTIONS] migrate [OPTIONS]

The output target may only be a database in this case; file-based outputs are not supported for migrations.

Options

These options are used by the migrate command.

Program: migrate

-o, —output, —database

config
database, output

type
string: a URL

required
yes

shared with
create-db, migrate, store-result, process

The following prefixes are supported:

Database URL:
e.g. postgresql://user:password@localhost:15432/sdk_db

If you pass a database URL, the output data will be stored in that database.

csv, csv+table, table+csv:
e.g.

csv+table://relative/path/
csv:///absolute/path/to/directory/ (note that the prefix is csv:// so this path starts with the third /)

json, json+table, table+json:
e.g. json+table://path/to/json/

The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above.

Each file will contain a list filled with JSON objects, where the keys match the table columns.

jsonl, jsonl+table, table+jsonl:
e.g. jsonl+table://path/to/jsonl/

raw, raw+json, json+raw:
e.g. raw+json://path/to/raw-results/

raw+jsonl, jsonl+raw:
e.g. raw+jsonl://path/to/results.jsonl

There will also be a corresponding <filename>.map file containing a mapping from input filenames to document IDs.

—schema

config
schema

type
string

default
legacy

shared with
create-db, migrate, store-result, process

—snowflake-private-key-path

config
snowflake-private-key-path

type
string: path to a file

default
None

shared with
create-db, migrate, store-result, process

”store-result” command

The store-result command stores NLP API results to a database.

Command-line usage

$ emtellipro-db-client [GLOBAL-OPTIONS] store-result [OPTIONS] PATH...

Each input PATH may either be a JSON file containing NLP API results, or a directory containing multiple such JSON files. The JSON files must have been created by the process command using --output raw+json://PATH.

Example usage may be:

$ emtellipro-db-client -c config.toml \
    process \
    --output raw+json://./output_dir/ \
    example-data/*
$ emtellipro-db-client -c config.toml \
    store-result \
    --output postgresql://... \
    ./output_dir

Storing results will always store the document text when returned by the NLP API. This is necessary when processing PDFs and when using the infer-document-structure processing feature, where the returned text is different than what was provided as input.

Options

These options are used by the store-result command.

Program: store-result

-o, —output, —database

config
database, output

type
string: a URL

required
yes

shared with
create-db, migrate, store-result, process

The following prefixes are supported:

Database URL:
e.g. postgresql://user:password@localhost:15432/sdk_db

If you pass a database URL, the output data will be stored in that database.

csv, csv+table, table+csv:
e.g.

csv+table://relative/path/
csv:///absolute/path/to/directory/ (note that the prefix is csv:// so this path starts with the third /)

json, json+table, table+json:
e.g. json+table://path/to/json/

The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above.

Each file will contain a list filled with JSON objects, where the keys match the table columns.

jsonl, jsonl+table, table+jsonl:
e.g. jsonl+table://path/to/jsonl/

raw, raw+json, json+raw:
e.g. raw+json://path/to/raw-results/

raw+jsonl, jsonl+raw:
e.g. raw+jsonl://path/to/results.jsonl

There will also be a corresponding <filename>.map file containing a mapping from input filenames to document IDs.

—schema

config
schema

type
string

default
legacy

shared with
create-db, migrate, store-result, process

—snowflake-private-key-path

config
snowflake-private-key-path

type
string: path to a file

default
None

shared with
create-db, migrate, store-result, process

-S, —save-opt

config
save-opt

type
string

multiple
true

New in version 7.0.

In the config file, there should be a section called:

[process]
[process.save_options.legacy]     # for the legacy schema
job_id = 1234
store_json = true

The following values will be injected into the save options by the client when running (since they’re dynamic and can’t be set ahead of time):

task_id with the task ID received from the NLP API
engine_version with the version string returned by the NLP API with the result JSON
sql_query with the value of --sql-query, if present.

If the schema you’re using doesn’t include those attributes in its save options, they’ll be ignored.

For the legacy schema in version 6 and earlier, the save options were separate options like --store-json instead of --save-opt store_json.

See the migration guide for how to use the new save options if upgrading to v7: Save options replace the ``--store-\*flags.

For the available options when using the legacy schema see the API documentation: SaveOptions.

”cancel” command

The cancel command cancels a given task, either from a state file (produced by the process command), or a task ID.

Command-line usage

$ emtellipro-db-client [GLOBAL-OPTIONS] cancel [OPTIONS]

Options

These options are used by the cancel command.

Program: cancel

—server

config
server

type
string: a URL

required
yes

shared with
get-user, process, cancel

The URL of the NLP API server you’d like to send the reports to. Can also be set via the EMTELLIPRO_SERVER environment variable.

—access-key

config
access-key

type
string

required
yes

shared with
get-user, process, cancel

The access key used when authenticating with the NLP API. Can also be set via the EMTELLIPRO_ACCESS_KEY environment variable.

—shared-secret, —secret-key

config
shared-secret

type
string

required
yes

shared with
get-user, process, cancel

The shared secret used when authenticating with the NLP API. Can also be set via the EMTELLIPRO_SHARED_SECRET environment variable. --secret-key is accepted as an alias on the command line.

—api-key

config
api-key

type
string

required
yes

shared with
get-user, process, cancel

—state

config
state

type
string: path to a file

required
no

The path to a file in which the current processing state is stored. This is used to find the latest task ID and cancel that.

-t, —task

config
task

type
string

required
no

The task ID to cancel, if the state file isn’t passed.

”list-options” command

The list-options command lists save options available for the selected schema.

Usage

$ emtellipro-db-client [GLOBAL-OPTIONS] list-options --schema SCHEMA

Options

These options are used by the list-options command.

Program: list-options

—schema

config
schema

type
string

default
legacy

required
no

This is the schema for which the save options are listed.

”debug” command

The debug command produces output that is useful for debugging issues. The only argument it takes is the state file which was created using the process command.

Usage

$ emtellipro-db-client [GLOBAL-OPTIONS] debug [--json] [--gantt PATH] STATEFILE

The STATEFILE argument is the path to the state file produced by the process command.

See the Statistics Explanation section for the meaning of all the reported statistics.

Options

These options are used by the debug command.

Program: debug

—json

config
json

type
boolean

default
false

required
no

If set, the output of the debug command will be as a JSON object. The output structure may change from version to version, so this should only be used for debugging and not relied upon to be parseable using the same code long term.

—gantt

config
gantt

type
string (file path)

required
no

If set, the debug command additionally writes a Gantt chart of the per-batch stage timings (loading, processing, and saving) to the given file path. This does not replace the regular debug output, which is still printed as usual. The image format is inferred from the file extension (e.g. .png, .svg, .pdf).

This option requires matplotlib, which is not installed by default. Install it via the cli-extras extra:

$ pip install emtellipro[cli-extras]

The output of the debug command is only to be used for debugging purposes, so it should not be relied upon to be consistent between releases.

Example output

With the --json flag, you may get some output such as this:

{
    "parameters": {
        "process": {
            "poll_freq": 1,
            "features": null,
            "document_type": "plain"
        },
        "store": {
            "max_save_shard_size": 50,
            "job_id": "f0216bfb-3f80-4894-96b0-1b25258ecb4a",
            "job_timestamp": "2025-03-05T19:22:28.632813",
            "store_text": false,
            "store_json": false,
            "store_failed": true,
            "store_sentences": false,
            "store_sections": false,
            "store_headings": false,
            "store_pages": false,
            "store_source_data": false,
            "sql_query": null
        }
    },
    "progress": {
        "loaded": 2,
        "processed": 2,
        "stored": 2
    },
    "failed": [
        {
            "filepath": "path/to/some_broken_doc.txt",
            "source_id": null,
            "failure": "processing"
        }
    ]
}

The TOML format is a full-featured configuration language, but for our purposes we’re only using simple key-value options which will look fairly similar to INI files. This document shows an example using the [table] style, but dotted keys are also supported. Full documentation of TOML can be found here: https://toml.io/en/ ↩

Configuration file

Defaults and overriding options

Option descriptions

Global options

Shared options

”process” command

Options

Statistics explanation

”get-user” command

Options

”create-db” command

Options

”migrate” command

Options

”store-result” command

Options

”cancel” command

Options

”list-options” command

Options

”debug” command

Options

Example output

Footnotes