> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.emtelligent.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.emtelligent.com/_mcp/server.

# Pluggable database schemas

*New in version 7.0.*

The database client no longer hard-codes a single set of tables. The way annotated documents are stored is now defined by a *schema*, and the SDK ships with a built-in default schema while also supporting additional schemas that you can write yourself and install as plugins.

This makes it possible to store NLP API results in whatever table layout suits your needs, instead of being limited to the tables the SDK provides.

## Selecting a schema

A schema is chosen with the `--schema` option on the database client. The built-in [legacy schema](#the-legacy-schema) is used by default, so existing workflows continue to behave exactly as before:

```console
$ emtellipro-db-client --schema legacy create-db
$ emtellipro-db-client --schema legacy process example-data/
```

The Python API accepts the schema name (or a schema object) when constructing a [Database](/nlp/python/reference/db-database#emtelliprodbdatabasedatabase):

```python
from emtellipro.db.database import Database

db = Database(uri='sqlite:///out.db', schema='legacy')
```

Schema names may be written with hyphens or underscores; they are normalized internally, so `my-schema` and `my_schema` refer to the same schema.

## The legacy schema

The schema that the SDK has always used is now packaged as the `legacy` schema (`emtellipro.db.schemas.legacy`). If you do not pass `--schema` you get the legacy schema, and nothing about its behaviour has changed.

## The schema manifest

Every schema is described by a single object: an instance of `SchemaManifest`. This is the entry point the SDK loads and the glue that ties the pieces of a schema together. A manifest declares:

`name`\
The schema's name. This must match the name it is registered under (see [Registering a schema as a plugin](#registering-a-schema-as-a-plugin)).

`models`\
The module containing the SQLAlchemy ORM models. The manifest exposes the underlying `metadata` and `tables` from this module.

`save`\
The callable used to write documents to the database (see [The save function](#the-save-function)).

`migrations`\
The Alembic migrations module, or `None` if the schema does not support migrations. File-based outputs never run migrations, regardless of this value.

A minimal manifest looks like this:

```python
from emtellipro.db.schemas import SchemaManifest

from . import models, save, _migrations

schema = SchemaManifest(
    name='sample-plugin-schema',
    models=models,
    save=save.save,
    migrations=_migrations,   # optional
)
```

If you need to override how a part of the schema behaves (for example, the Alembic configuration), subclass `SchemaManifest` rather than instantiating it directly.

## Registering a schema as a plugin

Schemas other than the built-ins are discovered through Python entry points in the `emtellipro.plugins.schemas` group. Each entry point maps a schema name to the location of its manifest object. In a plugin's `pyproject.toml`:

**`pyproject.toml`**

```toml title="pyproject.toml"
[project.entry-points."emtellipro.plugins.schemas"]
# the entry-point name must match the manifest's `name`
sample-plugin-schema = "schema_plugin:schema"
```

Once the plugin is installed into the same environment as the SDK, its schema name appears as a valid choice for `--schema` automatically. The SDK loads schemas lazily so installed plugins do not slow down unrelated commands.

A complete, runnable example lives in `examples/schema_plugin/` in the SDK source tree.

## The save function

The heart of a schema is its `save` function. The SDK calls it with the batch of documents to store, an open database connection, the save options, and a cache object:

```python
def save(docs, connection, options: SaveOptions, cache=None):
    ...
    return cache
```

`docs`\
A list of `SaveDocument` objects. Each pairs the `annotated` document with its (optional) `input` document, and exposes a `text` property that returns whichever text is available. The annotated document is what carries the found entities, relations, and other results you will read from.

`connection`\
The database connection to write to. Use the insert helpers described below rather than issuing inserts by hand.

`options`\
An instance of this schema's save-options class (see [Save options](#save-options)). Note that the type annotation of this parameter is used to discover the class that can be instantiated and passed here automatically.

`cache`\
An object the SDK threads through successive `save` calls that belong to the same job, so values computed once can be reused. Your function returns the cache (creating one if it was passed `None`), and the SDK passes that same object back on the next call. A typical use is inserting a "job" row once, caching the returned ID, and referencing it as a foreign key in later calls. You decide its shape and when to reset it.

### Insert helpers

Different databases impose different limits on how many rows or parameters can go into one statement. The `insert` module handles this batching for every supported backend (SQLite, PostgreSQL, MSSQL, Snowflake, DuckDB, and the file-based engines). Pass all of your rows to `exec_insert` and let it chunk them appropriately:

```python
from emtellipro.db.schemas.insert import exec_insert

exec_insert(connection, models.FoundEntity, values)
```

There are companion helpers for inserts that need to return generated IDs (`exec_returning_id` and `exec_insert_default_returning_id`).

## Models

Schema tables are ordinary SQLAlchemy ORM models. To keep naming conventions (index, constraint, and foreign-key names) consistent with the rest of the SDK, build your declarative base with `create_base` instead of SQLAlchemy's `declarative_base` directly:

```python
from sqlalchemy import Column, Integer, UnicodeText
from sqlalchemy.orm import DeclarativeBase
from emtellipro.db.schemas.util import create_metadata

class Base(DeclarativeBase):
    # this ensures a constraint naming convention is always used
    metadata = create_metadata()

class FoundEntity(Base):
    id = Column(Integer, primary_key=True, autoincrement=True)
    start = Column(Integer)
    end = Column(Integer)
    text = Column(UnicodeText)
```

The base derives table names from the class name automatically. The `util` module also provides a `uuid7` helper for time-ordered UUID primary keys.

## Save options

Each schema defines its own *save options* -- a dataclass of options that control how documents are stored (for example, a minimum confidence threshold for keeping found entities):

```python
import dataclasses

@dataclasses.dataclass
class SaveOptions:
    #: Drop found entities below this concept confidence.
    confidence_min: float = 0
```

The SDK discovers this class from the type annotation on your `save` function's `options` parameter, exposed as `schema.save_options` (it can also be included when calling `SchemaManifest()` if you wish to avoid the type annotation). When saving through the Python API you can pass an instance directly, or pass keyword arguments that the SDK uses to build one:

```python
db.save(input_docs, annotated_docs, confidence_min=0.5)
```

On the command line, save options are read from the configuration file, keyed by schema name, so each schema can carry its own settings:

**`config.toml`**

```toml title="config.toml"
[process.save-options.sample-plugin-schema]
confidence-min = 0.5
```

> **Warning**
>
> Note that the `save-options` key is hyphenated, as are the actual save options in the example. In practice, save options themselves may be hyphenated *or* snake_case, and will be normalized to snake_case when passed to the `schema.save_options()` class.

## Migrations

A schema may optionally ship Alembic migrations by pointing its manifest's `migrations` attribute at a migrations module. When present, the `migrate` command can bring an existing database up to the latest revision. When `migrations` is `None` — or when the output is a file-based format — the `migrate` command reports that migrations are not supported and does nothing.

The migrations module follows the standard Alembic layout (an `env.py`, a `script.py.mako`, a `versions/` package, and a `resources/alembic.ini`). The example plugin and the legacy schema both include a working setup you can copy from.

## Putting it together

To write your own schema you generally provide, in one package:

1. **models** — SQLAlchemy ORM models built on `create_base`.
2. **a save function** — reads `SaveDocument` objects and writes rows using the `insert` helpers.
3. **save options** — a dataclass annotated on the save function's `options` parameter.
4. **migrations** *(optional)* — an Alembic module for schema upgrades.
5. **a manifest** — a `SchemaManifest` tying the above together, exported and registered under the `emtellipro.plugins.schemas` entry-point group.

See `examples/schema_plugin/` for a complete template, and the built-in `legacy` schema for a full-featured reference.