> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.emtelligent.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.emtelligent.com/_mcp/server.

# Multithreaded database client

*New in version 7.0.*

The database client (`emtellipro-db-client`) has been rebuilt to process documents using multiple worker threads. Each worker repeatedly takes a batch of documents, submits it to the NLP API, retrieves the results, and stores them to the output database. This lets a single run submit, process, and save in parallel rather than one stage at a time.

For most users the command-line interface works the same way as before, but several options have been added, renamed, or removed, and the state file format has changed. The sections below describe what changed.

## Running jobs in parallel

The new `-j/--jobs` option controls how many submit/process/store jobs run in parallel:

```console
$ emtellipro-db-client process -j 4 example-data/
```

Each job processes its own batch of documents end-to-end. The default is `1`, which preserves the previous single-threaded behaviour.

## New options

These are all for the `process` command.

`-j, --jobs`\
Number of parallel submit/process/store jobs to run. Default: `1`.

`--sql-count-query`\
When reading input from a database, an optional query returning the number of rows that `--sql-query` will produce. If omitted, a `SELECT COUNT` query is generated automatically. This is used to drive the progress bars.

`--sql-limit`\
An optional limit on how many rows to read from `--sql-query`.

## Renamed options

The 'submit shard size' option was renamed for clarity. Update any scripts or config files that use the old names.

| Old name                  | New name       |
| ------------------------- | -------------- |
| `--max-submit-shard-size` | `--batch-size` |

## Removed options

The following options have been removed:

`-r, --recursive`\
Directories passed as input paths are now always searched recursively, so this flag is no longer needed.

`--max-save-shard-size`\
Saving is now handled by the worker jobs and no longer needs a separate shard size. The saving shard size is the same as `--batch-size`.

`--retry-forever`\
Removed. Use `--max-retries` to control retry behaviour.

The `--job-id` option is no longer accepted by the `process` and `store-result` commands. To set a job ID when processing, use `process --save-opt` instead:

```console
$ emtellipro-db-client process --save-opt job_id=<uuid> example-data/
```

## State file changes

The state file (set with `-s/--state`) has changed. Previously it stored a compressed copy of every input document so a run could be resumed. It now stores only a unique key per document along with its processing status, which keeps the file much smaller while still allowing a run to be restarted from the same state file.

In addition, the state file now records how long each document spent in each processing stage (loading, processing, and saving). This means the run summary can report finer-grained timing information, for example average time per document per stage:

```text
Documents processed: 1200
  saved: 1200
Average time per document per stage:
  loading      0.012s/doc  (14.4s over 1200 docs)
  processing   0.305s/doc  (366.0s over 1200 docs)
  saving       0.041s/doc  (49.2s over 1200 docs)
```

Because the format changed, state files created by older versions of the client are not compatible with this version.