> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.emtelligent.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.emtelligent.com/_mcp/server. # Database Client Options The database client provides many options for configuring how it processes documents and saves them to a database; this document describes all of them, grouped by the command they belong to. All these descriptions can also be found using `emtellipro-db-client --help` and `emtellipro-db-client COMMAND --help` (where `COMMAND` is one of the commands supported by the client). The database client's options are grouped into two parts: the global options and the commands options. The global options apply to the client overall and are shared by all the commands, and the command options are specific to the given command. When running the database client, you'll format the command as follows: ```bash $ emtellipro-db-client [GLOBAL-OPTIONS] COMMAND [OPTIONS] ``` Where `COMMAND` may be something like `create-db` or `process`. The `[GLOBAL-OPTIONS]` will contain options about configuring the client (such as which database to be used, and how logging will be handled) and `[OPTIONS]` will be options for the command being run (e.g. the `process` command has options for connecting to the NLP API, but these would be irrelevant for `create-db`). On top of the regular command-line options, the database client also allows passing in a configuration file using the `emtellipro-db-client --config` global option. This configuration file allows the user to store commonly used options in one place to be re-used by multiple processing jobs. ## Configuration file The configuration file is in the TOML[^1] format. Our limited use of the TOML format will look a lot like INI files, although TOML is better defined. A simple configuration file will look something like the following: ```toml [emtellipro-db-client] # GLOBAL-OPTIONS will go here log-file = 'emtellipro.log' [general] # options shared by multiple commands can go in here, to avoid duplication database = "URL of your database" access-key = "your access key" shared-secret = "your secret key" server = "emtelliPro server URL" [process] # command-specific OPTIONS will go here max-retries = 10 ``` The global options all go in the `[emtellipro-db-client]` section, and command specific options will go into a section named the same as the command. The `[emtellipro-db-client]` section is required, even if empty, but no other sections are required. Any options not set in the configuration file will simply use the defaults, or be overridden by passing command-line options. ## Defaults and overriding options Most options have defaults which will be used if not overridden by the configuration file or command-line options. The priorities of the option values are the following: 1. The defaults for each option are used if not otherwise set. 2. The options in the configuration file override the defaults (for the options specified in the configuration file). 3. The options passed as command-line options override the options set in the configuration file and the defaults in (1). As a simple example, the default for `log-level` is `info`. If the configuration file specifies `log-level = "error"`, then only error messages will be logged. If you then *also* specify `--log-level "debug"`, then the logging level will be set to "debug", and all debugging messages will be printed. This hierarchy allows you to (1) only set the options you care about, (2) place commonly-used options in a configuration file, and (3) set options specific to the given job on the command line without having to edit the configuration file for each job. ## Option descriptions All options are described below, with these columns: Config option name\ This contains the config-file option name. This will be the exact same as the command-line option, without the leading hyphens. Type\ This is the type of argument this option takes. Some options will take a string as argument, some will take a number, and some are boolean options. Boolean options (also called "flags") will usually have a complementing `--no-*` version which is available simply to be more explicit when running the client; omitting the flag is the exact same as specifying the `--no-*` option, so specifying `--no-*` acts as a reminder that the option is disabled. Boolean options in the configuration file are specified using the `true` or `false` values. Required\ Required options must be passed in. Default\ This is the default value for the given option. Options that have defaults are optional. Descriptions of all the options can also be found using the `--help` option. This section may sometimes go into more detail than is found in the help message. ### Global options These options apply to the database client as a whole, and are not command-specific. When passed on a command line, they immediately follow `emtellipro-db-client`, and precede the command name. In the configuration file, they go in the `[emtellipro-db-client]` section. **-c, --config** config\ n/a type\ string: path to a file Specifies the path to the configuration file, if used. The configuration file is described above. **--log-file** config\ `log-file` type\ string: path to a file default\ errors.log File path for where to store logging output. **--log-level** config\ `log-level` type\ string: one of *error*, *warning*, *info*, or *debug* default\ info The minimum level of log messages to send to the log file. ### Shared options These options are shared by multiple commands. They can be set in the `[general]` section of the config file, or on each specific command. **-o, --output, --database** config\ `database`, `output` type\ string: a URL required\ yes shared with\ `create-db`, `migrate`, `store-result`, `process` The database to connect to, formatted as a URL containing all necessary connection settings. For most database systems, a SQLAlchemy-compatible URL format is necessary, but storing to files on disk is also supported using special prefixes, such as: `jsonl://PATH`, `json://PATH`, `csv://PATH`, and `raw://PATH` which allow storing results in `jsonl`, `json`, `csv` and unprocessed result files (respectively) in the provided directory `PATH`. The following prefixes are supported: Database URL:\ e.g. `postgresql://user:password@localhost:15432/sdk_db` If you pass a database URL, the output data will be stored in that database. Note that if you're using Snowflake the format is documented in [Snowflake's SQLAlchemy driver documentation](https://docs.snowflake.com/en/developer-guide/python-connector/sqlalchemy#connection-parameters). This differs from other URL formats Snowflake describes on other pages (for use with other tools), but as we pass the URL unchanged to Snowflake's driver, *only* the format describe on that page will work. csv, csv+table, table+csv:\ e.g. * `csv+table://relative/path/` * `csv:///absolute/path/to/directory/` (note that the prefix is `csv://` so this path starts with the third `/`) The output path is a directory in which CSV files will be stored, matching the schema of the regular database tables. There is a CSV file for each database table, with a filename corresponding to the table name. These files can be imported directly into a Snowflake database. json, json+table, table+json:\ e.g. `json+table://path/to/json/` The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above. Each file will contain a list filled with JSON objects, where the keys match the table columns. jsonl, jsonl+table, table+jsonl:\ e.g. `jsonl+table://path/to/jsonl/` This is similar to the JSON format above, the only difference being that each line in the file is a complete JSON object (instead of the entire file being a list of objects, the objects are written one per line). raw, raw+json, json+raw:\ e.g. `raw+json://path/to/raw-results/` The output path is a directory in which the raw JSON results from the NLP API are stored. The processed documents will be split into multiple tasks, and the results are stored in files with the filename being the task ID. For each result JSON `.json`, there are also corresponding `.input` and `.map` files, containing serialized input documents and mapping from input filenames to document IDs, respectively. raw+jsonl, jsonl+raw:\ e.g. `raw+jsonl://path/to/results.jsonl` The output path is a **file** (not a directory like the other formats). This file will contain JSONL data, where each line contains the raw JSON result from the NLP API. The output file will be appended to. There will also be a corresponding `.map` file containing a mapping from input filenames to document IDs. **--schema** config\ `schema` type\ string default\ legacy shared with\ `create-db`, `migrate`, `store-result`, `process` The name of the database schema to use. Run `emtellipro-db-client process --help` (or similar for any of the other commands that accept this option) to see the list of available schemas. Additional schemas may be registered via the `emtellipro.plugins.schemas` entry point. **--access-key** config\ `access-key` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The access key used when authenticating with the NLP API. Can also be set via the `EMTELLIPRO_ACCESS_KEY` environment variable. **--shared-secret, --secret-key** config\ `shared-secret` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The shared secret used when authenticating with the NLP API. Can also be set via the `EMTELLIPRO_SHARED_SECRET` environment variable. `--secret-key` is accepted as an alias on the command line. **--api-key** config\ `api-key` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The API key to use when authenticating to the NLP API. This is mutually exclusive with the `--access-key` and `--shared-secret` options. Can also be set via the `EMTELLIPRO_API_KEY` environment variable. **--server** config\ `server` type\ string: a URL required\ yes shared with\ `get-user`, `process`, `cancel` The URL of the NLP API server you'd like to send the reports to. Can also be set via the `EMTELLIPRO_SERVER` environment variable. **--snowflake-private-key-path** config\ `snowflake-private-key-path` type\ string: path to a file default\ None shared with\ `create-db`, `migrate`, `store-result`, `process` When connecting to a Snowflake database to store results, one can either use password-based authentication by setting the password in `process --output`, or by setting this option and using key-pair authentication. **-S, --save-opt** config\ `save-opt` type\ string multiple\ true *New in version 7.0.* This can be passed multiple times to set the save options for the schema you're using. Options that take arguments can be set like `-S job_id=1234`, and boolean options can be just passed in by name like `-S store_json`. In the config file, there should be a section called: ```toml [process] [process.save_options.legacy] # for the legacy schema job_id = 1234 store_json = true ``` For the legacy schema, the available options are described in the `SaveOptions` class. During interactive CLI use, you may wish to run the `list-options` command to see the available options: ["list-options" command](#list-options-command). The following values will be injected into the save options by the client when running (since they're dynamic and can't be set ahead of time): * `task_id` with the task ID received from the NLP API * `engine_version` with the version string returned by the NLP API with the result JSON * `sql_query` with the value of `--sql-query`, if present. If the schema you're using doesn't include those attributes in its save options, they'll be ignored. > **Tip** > > For the legacy schema in version 6 and earlier, the save options were separate options like `--store-json` instead of `--save-opt store_json`. > > See the migration guide for how to use the new save options if upgrading to v7: `Save options replace the ``--store-\*`[flags](/nlp/python/features/migrating-to-v7#save-options-replace-the---store--flags). > **Info** > > For the available options when using the legacy schema see the API documentation: `SaveOptions`. ### "process" command The `process` command reads in documents from the input paths, sends them to the NLP API to be processed, and stores the results in the target output (either database of files). **`Command-line usage`** ```console title="Command-line usage" $ emtellipro-db-client [GLOBAL-OPTIONS] process [OPTIONS] INPUT_PATH... ``` Each `INPUT_PATH` passed in as argument may be: * a file path containing document(s) to be loaded * a directory containing files (subdirectories are searched recursively) * the URL of a database (only one input path may be provided for this case). See [Input Sources](/nlp/python/database-client/overview#input-sources) for more details on how to use the different input sources. #### Options These options are for the `process` command, and immediately follow the `process` command on the command-line; in the configuration file they go in the `[process]` section. **Program:** `process` **-o, --output, --database** config\ `database`, `output` type\ string: a URL required\ yes shared with\ `create-db`, `migrate`, `store-result`, `process` The database to connect to, formatted as a URL containing all necessary connection settings. For most database systems, a SQLAlchemy-compatible URL format is necessary, but storing to files on disk is also supported using special prefixes, such as: `jsonl://PATH`, `json://PATH`, `csv://PATH`, and `raw://PATH` which allow storing results in `jsonl`, `json`, `csv` and unprocessed result files (respectively) in the provided directory `PATH`. The following prefixes are supported: Database URL:\ e.g. `postgresql://user:password@localhost:15432/sdk_db` If you pass a database URL, the output data will be stored in that database. Note that if you're using Snowflake the format is documented in [Snowflake's SQLAlchemy driver documentation](https://docs.snowflake.com/en/developer-guide/python-connector/sqlalchemy#connection-parameters). This differs from other URL formats Snowflake describes on other pages (for use with other tools), but as we pass the URL unchanged to Snowflake's driver, *only* the format describe on that page will work. csv, csv+table, table+csv:\ e.g. * `csv+table://relative/path/` * `csv:///absolute/path/to/directory/` (note that the prefix is `csv://` so this path starts with the third `/`) The output path is a directory in which CSV files will be stored, matching the schema of the regular database tables. There is a CSV file for each database table, with a filename corresponding to the table name. These files can be imported directly into a Snowflake database. json, json+table, table+json:\ e.g. `json+table://path/to/json/` The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above. Each file will contain a list filled with JSON objects, where the keys match the table columns. jsonl, jsonl+table, table+jsonl:\ e.g. `jsonl+table://path/to/jsonl/` This is similar to the JSON format above, the only difference being that each line in the file is a complete JSON object (instead of the entire file being a list of objects, the objects are written one per line). raw, raw+json, json+raw:\ e.g. `raw+json://path/to/raw-results/` The output path is a directory in which the raw JSON results from the NLP API are stored. The processed documents will be split into multiple tasks, and the results are stored in files with the filename being the task ID. For each result JSON `.json`, there are also corresponding `.input` and `.map` files, containing serialized input documents and mapping from input filenames to document IDs, respectively. raw+jsonl, jsonl+raw:\ e.g. `raw+jsonl://path/to/results.jsonl` The output path is a **file** (not a directory like the other formats). This file will contain JSONL data, where each line contains the raw JSON result from the NLP API. The output file will be appended to. There will also be a corresponding `.map` file containing a mapping from input filenames to document IDs. **--schema** config\ `schema` type\ string default\ legacy shared with\ `create-db`, `migrate`, `store-result`, `process` The name of the database schema to use. Run `emtellipro-db-client process --help` (or similar for any of the other commands that accept this option) to see the list of available schemas. Additional schemas may be registered via the `emtellipro.plugins.schemas` entry point. **--access-key** config\ `access-key` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The access key used when authenticating with the NLP API. Can also be set via the `EMTELLIPRO_ACCESS_KEY` environment variable. **--shared-secret, --secret-key** config\ `shared-secret` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The shared secret used when authenticating with the NLP API. Can also be set via the `EMTELLIPRO_SHARED_SECRET` environment variable. `--secret-key` is accepted as an alias on the command line. **--api-key** config\ `api-key` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The API key to use when authenticating to the NLP API. This is mutually exclusive with the `--access-key` and `--shared-secret` options. Can also be set via the `EMTELLIPRO_API_KEY` environment variable. **--server** config\ `server` type\ string: a URL required\ yes shared with\ `get-user`, `process`, `cancel` The URL of the NLP API server you'd like to send the reports to. Can also be set via the `EMTELLIPRO_SERVER` environment variable. **--snowflake-private-key-path** config\ `snowflake-private-key-path` type\ string: path to a file default\ None shared with\ `create-db`, `migrate`, `store-result`, `process` When connecting to a Snowflake database to store results, one can either use password-based authentication by setting the password in `process --output`, or by setting this option and using key-pair authentication. **-s, --state \** config\ `state` type\ string: path to a file required\ no The path to a file in which to store the current processing state. This is used for restarting processing if the client has been stopped early (by crashing or hitting CTRL-C). Simply re-run the same command with the same state file and same options to continue where it left off. If processing has completed and the client is re-run with the same state file, it will still "continue where it left off", meaning it will show progress of 100% and exit again. The state file should be different for each distinct processing submission. This can be accomplished by setting the state file name to a unique name for each submission, or deleting the state file between submissions. Omitting the state file path will result in a temporary state file being created, which will be deleted when processing has completed successfully. The path to the state file will be printed when the `process` command is executed. **--category** config\ `category` type\ string default\ Clinical The document category specified when processing reports. If the input documents come from JSON and SQL, the per-document category set on the input documents will override this option. This can also be a 'auto', to have the NLP API infer the category. **--subcategory** config\ `subcategory` type\ string default\ generic The document subcategory specified when processing reports. If the input documents come from JSON and SQL, the per-document subcategory set on the input documents will override this option. When having the NLP API infer the category (by setting it to 'auto'), this subcategory option will be ignored. **--document-type** config\ `document-type` type\ string default\ plain The document type parameter sent to the engine. See the NLP API documentation for available values. **--section-label** config\ `section-label` type\ string default\ None The section label used for all documents, if you'd like to use the NLP API's `process_with_section_labels` feature. Setting this option will tell the NLP API to assume the entire input document is part of a single section (with this section label). **--features** config\ `features` type\ string default\ all features The NLP API features to request when processing the documents. This is a comma-separated list of feature names with no spaces. In the configuration file you may use either a single comma-separated string, or a list of strings (as per TOML format). See NLP API documentation for what features are available. Available features: **-j, --jobs** config\ `jobs` type\ integer default\ 1 How many submit/process/store jobs to run in parallel. Each job will take a batch of documents and process them with the NLP API, and store them to output. **--file-encoding** config\ `file-encoding` type\ string default\ auto The file encoding to use when reading input files. If not set, it will be autodetected. If you know the file encoding ahead of time, it's a good idea to set it here to avoid the overhead of autodetection. **-S, --save-opt** config\ `save-opt` type\ string multiple\ true *New in version 7.0.* This can be passed multiple times to set the save options for the schema you're using. Options that take arguments can be set like `-S job_id=1234`, and boolean options can be just passed in by name like `-S store_json`. In the config file, there should be a section called: ```toml [process] [process.save_options.legacy] # for the legacy schema job_id = 1234 store_json = true ``` For the legacy schema, the available options are described in the `SaveOptions` class. During interactive CLI use, you may wish to run the `list-options` command to see the available options: ["list-options" command](#list-options-command). The following values will be injected into the save options by the client when running (since they're dynamic and can't be set ahead of time): * `task_id` with the task ID received from the NLP API * `engine_version` with the version string returned by the NLP API with the result JSON * `sql_query` with the value of `--sql-query`, if present. If the schema you're using doesn't include those attributes in its save options, they'll be ignored. > **Tip** > > For the legacy schema in version 6 and earlier, the save options were separate options like `--store-json` instead of `--save-opt store_json`. > > See the migration guide for how to use the new save options if upgrading to v7: `Save options replace the ``--store-\*`[flags](/nlp/python/features/migrating-to-v7#save-options-replace-the---store--flags). > **Info** > > For the available options when using the legacy schema see the API documentation: `SaveOptions`. **--sql-query** config\ `sql-query` type\ string optional If the input path is a database URL, this option must be set to tell the client what query to use to retrieve documents from the input database. The query must return at least `id` and `text` columns; any extra fields are stored in the `documentmetadata` table. If `category` and `subcategory` are included, they will be stored as `original_category` and `original_subcategory`. This can be combined with `--text-type` if the `text` column in the SQL query contains non-plaintext data. The query will be processed in a transaction that executes a COMMIT at the end of loading, so the SQL query is able to make modifications to the source database. **--sql-count-query** config\ `sql-count-query` type\ string required\ no Optional query that returns the number of rows the query in `--sql-query` will return. If not provided, a default `SELECT COUNT` query is generated based on `--sql-query`. **--sql-limit** config\ `sql-limit` type\ integer required\ no Optional limit on how many rows to read from `--sql-query`. It's assumed the query will return at least this many rows, but if it returns fewer the progress bars will not reach 100%. **--text-type** config\ `text-type` type\ string default\ text This value specifies what type of data is stored in the 'text' column when reading input from a database using `--sql-query`. This may be any of: ccd\ The text column must contain an XML document. The XML will be parsed as a CCD document, and the metadata will also be included, same as if it were parsed from a file. pdf\ The text column must contain raw PDF bytes. text\ This is the default; the text column must contain plaintext. **--quiet** config\ `quiet` type\ boolean / integer default\ false / 0 Make the client quieter. Passing this once disables the progress bar, but sends error messages to stderr; passing it twice hides the error messages (they'll still be present in the log file). In the config file, you can set a number to indicate how quiet you wish the client to be. **--batch-size** config\ `batch-size` type\ integer default\ 100 Maximum number of documents to submit at once. If set to -1, the number of documents submitted in each shard is automatically determined based on document size (to submit the maximum number of documents that fit within the NLP API server's submission size limit). This option is useful for restricting the shard size further than what is technically allowed by the server; this can be helpful for lowering memory usage. **--skip-database-checks / --no-skip-database-checks** config\ `skip-database-checks` type\ boolean default\ false Skip checks for whether the database schema is valid. By default the client will check on startup whether all the tables and columns in the target database have the correct schema; this avoids potential errors when trying to save to a database that needs to be migrated (or has not been created), but there's overhead to doing this. If you've already confirmed the database schema is correct, you should disable the checks. **--max-retries** config\ `max-retries` type\ integer default\ 5 Maximum number of times to retry failed API requests. Failures may be due to network issues, so this should be a number greater than 0. **--retry-failed** config\ `retry-failed` type\ bool default\ false Retry documents that received a processing status of 'error' from the NLP API. **--store-failed / --no-store-failed** config\ `store-failed` type\ boolean default\ false Store details of failed documents in the `documents` table. You can find these by querying for `documents.processing_status = 'error'`. **--doc-id-filepath** config\ `doc-id-filepath` type\ boolean default\ false Use the absolute file path as the document ID for NLP API input documents. By default random UUIDs are generated to anonymize input document locations when submitting documents to the NLP API, but this option can be useful for debugging. For JSON files this will include the document ID from the file appended to the file path. **--poll-freq** config\ `poll-freq` type\ integer default\ 1 Number of seconds to wait between API calls to check processing status. Polling too frequently can put extra load on the NLP API, so for non-interactive uses of the client, it's helpful to set this value higher. EmtelliPro will respond as soon as processing is completed, so setting this value higher than 1 second will not lead to slower processing, but it will cause the progress bar to be updated less often. **--filetype** config\ `filetype` type\ string default\ detected from file extension Assume all input files have the given file type, disabling automatic detection of file type based on file extension. See the `--help` option for available options. #### Statistics explanation At the end of execution, the client will provide some statistics about the processing times of the documents. ```text Documents processed: 137 success: 137 Average time per document per stage: loading 0.005s/doc (0.7s over 137 docs) task-setup 0.000s/doc (0.0s over 137 docs) processing 1.196s/doc (163.8s over 137 docs) saving 0.016s/doc (2.2s over 137 docs) Mean batch size: 9.79 Mean concurrency: 10.0 jobs Throughput: loading 0.001s/doc (0.4%) task-setup 0.000s/doc (0.0%) processing 0.181s/doc (98.3%) saving 0.002s/doc (1.3%) Overall throughput: 5.44 doc/s Run statistics: Start time: 2026-08-15 00:35:19.002561 UTC Completed: 2026-08-15 00:35:44.243688 UTC Cumulative run time: 0:00:25.241127 ``` The different statistics are as follows: Documents processed\ Includes total count, as well as count per processing status. Average time per document per stage\ This is the average time taken in each stage while loading/processing/saving each batch, when treating each batch independently. This tells you how long each document *independently* took in each stage (i.e. if you use `-j1` and `-j2` you should see similar numbers). Mean batch size\ The average batch size across all batches. This will likely be slightly lower than the requested `--batch-size` since the last batch may not be completely filled. Mean concurrency\ The average value of `--jobs` across all runs. If the process is stopped and restarted with a different number of jobs, this value can change. Throughput\ The mean time taken per document per stage when taking concurrency into account. For example, if two documents are processed in parallel, the processing time should appear halved per document than if they were processed one at a time. This differs from the "average time" reported above which does not take concurrency into account. Overall throughput\ The number of documents per second that were processed (i.e. that went through all the stages listed in Throughput). Run statistics\ The overall statistics across all runs using the same state file. The start time is the time of the first processing run, and the cumulative run time across all runs. ### "get-user" command The `get-user` command retrieves the current user's details. This is useful for checking that the authentication keys and server URL are configured correctly. **`Command-line usage`** ```console title="Command-line usage" $ emtellipro-db-client [GLOBAL-OPTIONS] get-user ``` If you run it, you'll see output such as the following: ```bash Testing emtelliPro connection... Successful connection to https://emtellpro/ Connected as USER@example.com ``` #### Options These options are used by the `get-user` command. See the "[Shared options](#shared-options)" section above for options this command shares with others: **Program:** `get-user` **--server** config\ `server` type\ string: a URL required\ yes shared with\ `get-user`, `process`, `cancel` The URL of the NLP API server you'd like to send the reports to. Can also be set via the `EMTELLIPRO_SERVER` environment variable. **--access-key** config\ `access-key` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The access key used when authenticating with the NLP API. Can also be set via the `EMTELLIPRO_ACCESS_KEY` environment variable. **--shared-secret, --secret-key** config\ `shared-secret` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The shared secret used when authenticating with the NLP API. Can also be set via the `EMTELLIPRO_SHARED_SECRET` environment variable. `--secret-key` is accepted as an alias on the command line. **--api-key** config\ `api-key` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The API key to use when authenticating to the NLP API. This is mutually exclusive with the `--access-key` and `--shared-secret` options. Can also be set via the `EMTELLIPRO_API_KEY` environment variable. ### "create-db" command The `create-db` command will create the tables or files necessary for the `process` command to use afterwards. **`Command-line usage`** ```console title="Command-line usage" $ emtellipro-db-client [GLOBAL-OPTIONS] create-db [OPTIONS] ``` The target output is specified using `--output`. If the output is a database, the tables will be created. For file-based outputs, the directories and files will be created (when relevant, the files will contain headers). #### Options These options are used by the `create-db` command. See the "[Shared options](#shared-options)" section above for options this command shares with others: **Program:** `create-db` **-o, --output, --database** config\ `database`, `output` type\ string: a URL required\ yes shared with\ `create-db`, `migrate`, `store-result`, `process` The database to connect to, formatted as a URL containing all necessary connection settings. For most database systems, a SQLAlchemy-compatible URL format is necessary, but storing to files on disk is also supported using special prefixes, such as: `jsonl://PATH`, `json://PATH`, `csv://PATH`, and `raw://PATH` which allow storing results in `jsonl`, `json`, `csv` and unprocessed result files (respectively) in the provided directory `PATH`. The following prefixes are supported: Database URL:\ e.g. `postgresql://user:password@localhost:15432/sdk_db` If you pass a database URL, the output data will be stored in that database. Note that if you're using Snowflake the format is documented in [Snowflake's SQLAlchemy driver documentation](https://docs.snowflake.com/en/developer-guide/python-connector/sqlalchemy#connection-parameters). This differs from other URL formats Snowflake describes on other pages (for use with other tools), but as we pass the URL unchanged to Snowflake's driver, *only* the format describe on that page will work. csv, csv+table, table+csv:\ e.g. * `csv+table://relative/path/` * `csv:///absolute/path/to/directory/` (note that the prefix is `csv://` so this path starts with the third `/`) The output path is a directory in which CSV files will be stored, matching the schema of the regular database tables. There is a CSV file for each database table, with a filename corresponding to the table name. These files can be imported directly into a Snowflake database. json, json+table, table+json:\ e.g. `json+table://path/to/json/` The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above. Each file will contain a list filled with JSON objects, where the keys match the table columns. jsonl, jsonl+table, table+jsonl:\ e.g. `jsonl+table://path/to/jsonl/` This is similar to the JSON format above, the only difference being that each line in the file is a complete JSON object (instead of the entire file being a list of objects, the objects are written one per line). raw, raw+json, json+raw:\ e.g. `raw+json://path/to/raw-results/` The output path is a directory in which the raw JSON results from the NLP API are stored. The processed documents will be split into multiple tasks, and the results are stored in files with the filename being the task ID. For each result JSON `.json`, there are also corresponding `.input` and `.map` files, containing serialized input documents and mapping from input filenames to document IDs, respectively. raw+jsonl, jsonl+raw:\ e.g. `raw+jsonl://path/to/results.jsonl` The output path is a **file** (not a directory like the other formats). This file will contain JSONL data, where each line contains the raw JSON result from the NLP API. The output file will be appended to. There will also be a corresponding `.map` file containing a mapping from input filenames to document IDs. **--schema** config\ `schema` type\ string default\ legacy shared with\ `create-db`, `migrate`, `store-result`, `process` The name of the database schema to use. Run `emtellipro-db-client process --help` (or similar for any of the other commands that accept this option) to see the list of available schemas. Additional schemas may be registered via the `emtellipro.plugins.schemas` entry point. **--snowflake-private-key-path** config\ `snowflake-private-key-path` type\ string: path to a file default\ None shared with\ `create-db`, `migrate`, `store-result`, `process` When connecting to a Snowflake database to store results, one can either use password-based authentication by setting the password in `process --output`, or by setting this option and using key-pair authentication. ### "migrate" command The `migrate` command migrates the schema in the output database to the newest schema. **`Command-line usage`** ```console title="Command-line usage" $ emtellipro-db-client [GLOBAL-OPTIONS] migrate [OPTIONS] ``` The output target may only be a database in this case; file-based outputs are not supported for migrations. #### Options These options are used by the `migrate` command. **Program:** `migrate` **-o, --output, --database** config\ `database`, `output` type\ string: a URL required\ yes shared with\ `create-db`, `migrate`, `store-result`, `process` The database to connect to, formatted as a URL containing all necessary connection settings. For most database systems, a SQLAlchemy-compatible URL format is necessary, but storing to files on disk is also supported using special prefixes, such as: `jsonl://PATH`, `json://PATH`, `csv://PATH`, and `raw://PATH` which allow storing results in `jsonl`, `json`, `csv` and unprocessed result files (respectively) in the provided directory `PATH`. The following prefixes are supported: Database URL:\ e.g. `postgresql://user:password@localhost:15432/sdk_db` If you pass a database URL, the output data will be stored in that database. Note that if you're using Snowflake the format is documented in [Snowflake's SQLAlchemy driver documentation](https://docs.snowflake.com/en/developer-guide/python-connector/sqlalchemy#connection-parameters). This differs from other URL formats Snowflake describes on other pages (for use with other tools), but as we pass the URL unchanged to Snowflake's driver, *only* the format describe on that page will work. csv, csv+table, table+csv:\ e.g. * `csv+table://relative/path/` * `csv:///absolute/path/to/directory/` (note that the prefix is `csv://` so this path starts with the third `/`) The output path is a directory in which CSV files will be stored, matching the schema of the regular database tables. There is a CSV file for each database table, with a filename corresponding to the table name. These files can be imported directly into a Snowflake database. json, json+table, table+json:\ e.g. `json+table://path/to/json/` The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above. Each file will contain a list filled with JSON objects, where the keys match the table columns. jsonl, jsonl+table, table+jsonl:\ e.g. `jsonl+table://path/to/jsonl/` This is similar to the JSON format above, the only difference being that each line in the file is a complete JSON object (instead of the entire file being a list of objects, the objects are written one per line). raw, raw+json, json+raw:\ e.g. `raw+json://path/to/raw-results/` The output path is a directory in which the raw JSON results from the NLP API are stored. The processed documents will be split into multiple tasks, and the results are stored in files with the filename being the task ID. For each result JSON `.json`, there are also corresponding `.input` and `.map` files, containing serialized input documents and mapping from input filenames to document IDs, respectively. raw+jsonl, jsonl+raw:\ e.g. `raw+jsonl://path/to/results.jsonl` The output path is a **file** (not a directory like the other formats). This file will contain JSONL data, where each line contains the raw JSON result from the NLP API. The output file will be appended to. There will also be a corresponding `.map` file containing a mapping from input filenames to document IDs. **--schema** config\ `schema` type\ string default\ legacy shared with\ `create-db`, `migrate`, `store-result`, `process` The name of the database schema to use. Run `emtellipro-db-client process --help` (or similar for any of the other commands that accept this option) to see the list of available schemas. Additional schemas may be registered via the `emtellipro.plugins.schemas` entry point. **--snowflake-private-key-path** config\ `snowflake-private-key-path` type\ string: path to a file default\ None shared with\ `create-db`, `migrate`, `store-result`, `process` When connecting to a Snowflake database to store results, one can either use password-based authentication by setting the password in `process --output`, or by setting this option and using key-pair authentication. ### "store-result" command The `store-result` command stores NLP API results to a database. **`Command-line usage`** ```console title="Command-line usage" $ emtellipro-db-client [GLOBAL-OPTIONS] store-result [OPTIONS] PATH... ``` Each input `PATH` may either be a JSON file containing NLP API results, or a directory containing multiple such JSON files. The JSON files must have been created by the `process` command using `--output raw+json://PATH`. Example usage may be: ```console $ emtellipro-db-client -c config.toml \ process \ --output raw+json://./output_dir/ \ example-data/* $ emtellipro-db-client -c config.toml \ store-result \ --output postgresql://... \ ./output_dir ``` > **Note** > > Storing results will always store the document text when returned by the NLP API. This is necessary when processing PDFs and when using the `infer-document-structure` processing feature, where the returned text is different than what was provided as input. #### Options These options are used by the `store-result` command. **Program:** `store-result` **-o, --output, --database** config\ `database`, `output` type\ string: a URL required\ yes shared with\ `create-db`, `migrate`, `store-result`, `process` The database to connect to, formatted as a URL containing all necessary connection settings. For most database systems, a SQLAlchemy-compatible URL format is necessary, but storing to files on disk is also supported using special prefixes, such as: `jsonl://PATH`, `json://PATH`, `csv://PATH`, and `raw://PATH` which allow storing results in `jsonl`, `json`, `csv` and unprocessed result files (respectively) in the provided directory `PATH`. The following prefixes are supported: Database URL:\ e.g. `postgresql://user:password@localhost:15432/sdk_db` If you pass a database URL, the output data will be stored in that database. Note that if you're using Snowflake the format is documented in [Snowflake's SQLAlchemy driver documentation](https://docs.snowflake.com/en/developer-guide/python-connector/sqlalchemy#connection-parameters). This differs from other URL formats Snowflake describes on other pages (for use with other tools), but as we pass the URL unchanged to Snowflake's driver, *only* the format describe on that page will work. csv, csv+table, table+csv:\ e.g. * `csv+table://relative/path/` * `csv:///absolute/path/to/directory/` (note that the prefix is `csv://` so this path starts with the third `/`) The output path is a directory in which CSV files will be stored, matching the schema of the regular database tables. There is a CSV file for each database table, with a filename corresponding to the table name. These files can be imported directly into a Snowflake database. json, json+table, table+json:\ e.g. `json+table://path/to/json/` The output path is a directory in which JSON files will be stored; the JSON files will contain the same data as the CSV files described above. Each file will contain a list filled with JSON objects, where the keys match the table columns. jsonl, jsonl+table, table+jsonl:\ e.g. `jsonl+table://path/to/jsonl/` This is similar to the JSON format above, the only difference being that each line in the file is a complete JSON object (instead of the entire file being a list of objects, the objects are written one per line). raw, raw+json, json+raw:\ e.g. `raw+json://path/to/raw-results/` The output path is a directory in which the raw JSON results from the NLP API are stored. The processed documents will be split into multiple tasks, and the results are stored in files with the filename being the task ID. For each result JSON `.json`, there are also corresponding `.input` and `.map` files, containing serialized input documents and mapping from input filenames to document IDs, respectively. raw+jsonl, jsonl+raw:\ e.g. `raw+jsonl://path/to/results.jsonl` The output path is a **file** (not a directory like the other formats). This file will contain JSONL data, where each line contains the raw JSON result from the NLP API. The output file will be appended to. There will also be a corresponding `.map` file containing a mapping from input filenames to document IDs. **--schema** config\ `schema` type\ string default\ legacy shared with\ `create-db`, `migrate`, `store-result`, `process` The name of the database schema to use. Run `emtellipro-db-client process --help` (or similar for any of the other commands that accept this option) to see the list of available schemas. Additional schemas may be registered via the `emtellipro.plugins.schemas` entry point. **--snowflake-private-key-path** config\ `snowflake-private-key-path` type\ string: path to a file default\ None shared with\ `create-db`, `migrate`, `store-result`, `process` When connecting to a Snowflake database to store results, one can either use password-based authentication by setting the password in `process --output`, or by setting this option and using key-pair authentication. **-S, --save-opt** config\ `save-opt` type\ string multiple\ true *New in version 7.0.* This can be passed multiple times to set the save options for the schema you're using. Options that take arguments can be set like `-S job_id=1234`, and boolean options can be just passed in by name like `-S store_json`. In the config file, there should be a section called: ```toml [process] [process.save_options.legacy] # for the legacy schema job_id = 1234 store_json = true ``` For the legacy schema, the available options are described in the `SaveOptions` class. During interactive CLI use, you may wish to run the `list-options` command to see the available options: ["list-options" command](#list-options-command). The following values will be injected into the save options by the client when running (since they're dynamic and can't be set ahead of time): * `task_id` with the task ID received from the NLP API * `engine_version` with the version string returned by the NLP API with the result JSON * `sql_query` with the value of `--sql-query`, if present. If the schema you're using doesn't include those attributes in its save options, they'll be ignored. > **Tip** > > For the legacy schema in version 6 and earlier, the save options were separate options like `--store-json` instead of `--save-opt store_json`. > > See the migration guide for how to use the new save options if upgrading to v7: `Save options replace the ``--store-\*`[flags](/nlp/python/features/migrating-to-v7#save-options-replace-the---store--flags). > **Info** > > For the available options when using the legacy schema see the API documentation: `SaveOptions`. ### "cancel" command The `cancel` command cancels a given task, either from a state file (produced by the `process` command), or a task ID. **`Command-line usage`** ```console title="Command-line usage" $ emtellipro-db-client [GLOBAL-OPTIONS] cancel [OPTIONS] ``` #### Options These options are used by the `cancel` command. **Program:** `cancel` **--server** config\ `server` type\ string: a URL required\ yes shared with\ `get-user`, `process`, `cancel` The URL of the NLP API server you'd like to send the reports to. Can also be set via the `EMTELLIPRO_SERVER` environment variable. **--access-key** config\ `access-key` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The access key used when authenticating with the NLP API. Can also be set via the `EMTELLIPRO_ACCESS_KEY` environment variable. **--shared-secret, --secret-key** config\ `shared-secret` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The shared secret used when authenticating with the NLP API. Can also be set via the `EMTELLIPRO_SHARED_SECRET` environment variable. `--secret-key` is accepted as an alias on the command line. **--api-key** config\ `api-key` type\ string required\ yes shared with\ `get-user`, `process`, `cancel` The API key to use when authenticating to the NLP API. This is mutually exclusive with the `--access-key` and `--shared-secret` options. Can also be set via the `EMTELLIPRO_API_KEY` environment variable. **--state** config\ `state` type\ string: path to a file required\ no The path to a file in which the current processing state is stored. This is used to find the latest task ID and cancel that. **-t, --task** config\ `task` type\ string required\ no The task ID to cancel, if the state file isn't passed. ### "list-options" command The `list-options` command lists save options available for the selected schema. **`Usage`** ```console title="Usage" $ emtellipro-db-client [GLOBAL-OPTIONS] list-options --schema SCHEMA ``` #### Options These options are used by the `list-options` command. **Program:** `list-options` **--schema** config\ `schema` type\ string default\ legacy required\ no This is the schema for which the save options are listed. ### "debug" command The `debug` command produces output that is useful for debugging issues. The only argument it takes is the state file which was created using the `process` command. **`Usage`** ```console title="Usage" $ emtellipro-db-client [GLOBAL-OPTIONS] debug [--json] [--gantt PATH] STATEFILE ``` The `STATEFILE` argument is the path to the state file produced by the `process` command. See the [Statistics Explanation](#statistics-explanation) section for the meaning of all the reported statistics. #### Options These options are used by the `debug` command. **Program:** `debug` **--json** config\ `json` type\ boolean default\ false required\ no If set, the output of the `debug` command will be as a JSON object. The output structure may change from version to version, so this should only be used for debugging and not relied upon to be parseable using the same code long term. **--gantt** config\ `gantt` type\ string (file path) required\ no If set, the `debug` command additionally writes a Gantt chart of the per-batch stage timings (loading, processing, and saving) to the given file path. This does not replace the regular debug output, which is still printed as usual. The image format is inferred from the file extension (e.g. `.png`, `.svg`, `.pdf`). This option requires [matplotlib](https://matplotlib.org/), which is not installed by default. Install it via the `cli-extras` extra: ```console $ pip install emtellipro[cli-extras] ``` > **Warning** > > The output of the `debug` command is only to be used for debugging purposes, so it should not be relied upon to be consistent between releases. #### Example output With the `--json` flag, you may get some output such as this: ```javascript { "parameters": { "process": { "poll_freq": 1, "features": null, "document_type": "plain" }, "store": { "max_save_shard_size": 50, "job_id": "f0216bfb-3f80-4894-96b0-1b25258ecb4a", "job_timestamp": "2025-03-05T19:22:28.632813", "store_text": false, "store_json": false, "store_failed": true, "store_sentences": false, "store_sections": false, "store_headings": false, "store_pages": false, "store_source_data": false, "sql_query": null } }, "progress": { "loaded": 2, "processed": 2, "stored": 2 }, "failed": [ { "filepath": "path/to/some_broken_doc.txt", "source_id": null, "failure": "processing" } ] } ``` [^1]: The TOML format is a full-featured configuration language, but for our purposes we're only using simple key-value options which will look fairly similar to INI files. This document shows an example using the `[table]` style, but dotted keys are also supported. Full documentation of TOML can be found here: [https://toml.io/en/](https://toml.io/en/)