Migrating to v7 | emtelligent Documentation

New in version 7.0.

Version 7.0 is a major release of the SDK. The headline changes are:

The database client (emtellipro-db-client) is now multi-threaded and can submit, process, and save documents in parallel.
The database schema is now configurable. The schema the SDK has always used is still the default (now called the legacy schema), and additional schemas can be installed as plugins.
The --store-* and --job-id command-line options have been replaced by per-schema save options, set with process --save-opt.
The JSON input format has been redesigned and is now fully documented. The old format still works, but is deprecated.

This page summarizes what you need to change when upgrading. Most workflows need only small adjustments to command-line options; the database tables themselves are unchanged.

Database client changes

Parallel processing

The client now processes documents using multiple worker jobs. Use the new -j/--jobs option to control how many run in parallel:

$ emtellipro-db-client process -j 4 example-data/

The default is 1, which behaves like previous versions. See Multithreaded database client for full details.

Configurable schemas

How results are laid out in the database is now controlled by a schema, selected with the new --schema option. The schema used by all previous versions of the SDK is now named legacy and remains the default, so if you don’t pass --schema nothing changes: you get the same tables and columns as before, and existing databases can still be migrated with the migrate command.

It is also possible to write your own schema and install it as a plugin; see Pluggable database schemas.

Save options replace the `--store-*` flags

Because each schema defines its own storage options, the fixed set of --store-* flags (and --job-id) has been replaced by the generic -S/--save-opt option, which can be passed multiple times:

$ emtellipro-db-client process \
    --save-opt job_id=test_run_1 \
    --save-opt store_json \
    example-data/

Use this table to translate your existing commands (these are the save options for the default legacy schema; see SaveOptions for the full list):

Old option	New equivalent
`--job-id <id>`	`--save-opt job_id=<id>`
`--store-reports`	`--save-opt store_text`
`--store-json`	`--save-opt store_json`
`--store-sections`	`--save-opt store_sections`
`--store-sentences`	`--save-opt store_sentences`
`--store-sections-and-sentences`	`--save-opt store_sections` `--save-opt store_sentences`
`--store-headings`	`--save-opt store_headings`
`--store-pages`	`--save-opt store_pages`
`--store-pdf`	`--save-opt store_source_data`

In the configuration file, save options move from the [process] section into a section keyed by schema name:

config.toml

[process.save-options.legacy]
# save options specific to the 'legacy' schema
job_id = "test_run_1"
store_json = true
store_text = true

Not all options are relevant to set; for example task_id, engine_version, and sql_query will be set by the client based on the processing jobs it sends to the NLP API and the --sql-query option, respectively.

Other renamed and removed options

Old option	Replacement
`--max-submit-shard-size`	`--batch-size`
`-r, --recursive`	Removed (directories are now always searched recursively).
`--max-save-shard-size`	Removed (saving uses the same batch size as submitting).
`--retry-forever`	Removed (use `--max-retries`).
`cancel -t/--task-id`	`cancel -t/--task`

Remember to update configuration files as well as scripts, since the config keys match the option names.

New options

process --sql-count-query and process --sql-limit give finer control when reading input documents from a database.
debug --gantt writes a Gantt chart of per-batch stage timings to an image file (requires installing the cli-extras extra: pip install emtellipro[cli-extras]).

State files

State files created by older versions are not compatible with v7. Finish any in-progress runs with your current version before upgrading, and start v7 runs with a fresh state file.

The new state file no longer contains a copy of every input document; it instead stores unique identifiers for every input document, and records how long each document spent in each processing stage, which the run summary and the debug command can report. This keeps the file size smaller while still supporting restarting processing where it left off (in case of errors or early exits).

New JSON input format

The JSON/JSONL input format has been redesigned to be more structured, and is now thoroughly documented in Input File Formats Supported By The SDK, including a downloadable JSON Schema you can validate your files against.

The main changes to the JSON object keys:

Old key	New equivalent
`text`	`contents` (with `type` set to `text/plain`, the default)
`pdf`	`contents` (with `type` set to `application/pdf`)
`filepath`	`path` (with `type` set explicitly, or detected from the file extension)
`pdf_path`	`source` object with `path` and `data`
`structured_metadata`	`metadata` object
metadata keys (top-level)	`metadata` object

For example, this old-format document:

{
  "id": "doc-1",
  "text": "Patient presents with...",
  "category": "clinical_report",
  "structured_metadata": {"site": "ABC"}
}

becomes:

{
  "id": "doc-1",
  "contents": "Patient presents with...",
  "category": "clinical_report",
  "metadata": {"site": "ABC"}
}

The old format is still accepted, so existing files keep working, but it is deprecated.

Python API changes

If you only use the command-line client you can skip this section.

Database and save options

Database now accepts a schema argument (a schema name or schema object); the default is the legacy schema, which behaves the same as previous versions:

from emtellipro.db.database import Database
db = Database(uri='sqlite:///out.db', schema='legacy')

db.save() method still accepts save options as keyword arguments (job_id, store_json, and so on), and now also accepts an options argument taking a schema’s save-options instance directly, e.g. SaveOptions.

The schema’s save options class is also available as an attribute, so you can write somewhat generic code against a schema (the accepted parameters will differ, though).

opts = db.schema.save_options(job_id='test')

Moved modules

The database ORM models moved to the legacy schema package.

# before
from emtellipro.db import models
# after
from emtellipro.db.schemas.legacy import models

The table and column definitions themselves are unchanged. The row types module moved likewise, from emtellipro.db.rowtypes to emtellipro.db.schemas.legacy.rowtypes.

The models can be accessed through the schema object, though, so the module doesn’t have to be imported:

from emtellipro.db.schemas import load_schema
legacy_schema = load_schema('legacy')
# this is going to be emtellipro.db.schemas.legacy.models
models = legacy_schema.models

All schema objects are guaranteed to have a models attribute, so one can write generic code against that instead of finding the right module to import.

HTTP library

The SDK no longer uses requests for HTTP; it now uses httpx2 (with tenacity handling retries). This is transparent for normal use of the Emtellipro client, and the exceptions raised by the SDK are unchanged.

As the HTTP library has changed, any exceptions that might have been previously caught from requests must be changed to the equivalent httpx2 exception (where relevant).

If you used the emtellipro.auth.EmtelliproAuth class to sign your own requests sessions, it has been removed. Its replacement is sign_request, which signs an httpx2.Request.