Pluggable database schemas

New in version 7.0.

The database client no longer hard-codes a single set of tables. The way annotated documents are stored is now defined by a schema, and the SDK ships with a built-in default schema while also supporting additional schemas that you can write yourself and install as plugins.

This makes it possible to store NLP API results in whatever table layout suits your needs, instead of being limited to the tables the SDK provides.

Selecting a schema

A schema is chosen with the --schema option on the database client. The built-in legacy schema is used by default, so existing workflows continue to behave exactly as before:

$ emtellipro-db-client --schema legacy create-db
$ emtellipro-db-client --schema legacy process example-data/

The Python API accepts the schema name (or a schema object) when constructing a Database:

from emtellipro.db.database import Database
db = Database(uri='sqlite:///out.db', schema='legacy')

Schema names may be written with hyphens or underscores; they are normalized internally, so my-schema and my_schema refer to the same schema.

The legacy schema

The schema that the SDK has always used is now packaged as the legacy schema (emtellipro.db.schemas.legacy). If you do not pass --schema you get the legacy schema, and nothing about its behaviour has changed.

The schema manifest

Every schema is described by a single object: an instance of SchemaManifest. This is the entry point the SDK loads and the glue that ties the pieces of a schema together. A manifest declares:

name
The schema’s name. This must match the name it is registered under (see Registering a schema as a plugin).

models
The module containing the SQLAlchemy ORM models. The manifest exposes the underlying metadata and tables from this module.

save
The callable used to write documents to the database (see The save function).

migrations
The Alembic migrations module, or None if the schema does not support migrations. File-based outputs never run migrations, regardless of this value.

A minimal manifest looks like this:

from emtellipro.db.schemas import SchemaManifest
from . import models, save, _migrations
schema = SchemaManifest(
    name='sample-plugin-schema',
    models=models,
    save=save.save,
    migrations=_migrations,   # optional
)

If you need to override how a part of the schema behaves (for example, the Alembic configuration), subclass SchemaManifest rather than instantiating it directly.

Registering a schema as a plugin

Schemas other than the built-ins are discovered through Python entry points in the emtellipro.plugins.schemas group. Each entry point maps a schema name to the location of its manifest object. In a plugin’s pyproject.toml:

[project.entry-points."emtellipro.plugins.schemas"]
# the entry-point name must match the manifest's `name`
sample-plugin-schema = "schema_plugin:schema"

Once the plugin is installed into the same environment as the SDK, its schema name appears as a valid choice for --schema automatically. The SDK loads schemas lazily so installed plugins do not slow down unrelated commands.

A complete, runnable example lives in examples/schema_plugin/ in the SDK source tree.

The save function

The heart of a schema is its save function. The SDK calls it with the batch of documents to store, an open database connection, the save options, and a cache object:

def save(docs, connection, options: SaveOptions, cache=None):
    ...
    return cache

docs
A list of SaveDocument objects. Each pairs the annotated document with its (optional) input document, and exposes a text property that returns whichever text is available. The annotated document is what carries the found entities, relations, and other results you will read from.

connection
The database connection to write to. Use the insert helpers described below rather than issuing inserts by hand.

options
An instance of this schema’s save-options class (see Save options). Note that the type annotation of this parameter is used to discover the class that can be instantiated and passed here automatically.

cache
An object the SDK threads through successive save calls that belong to the same job, so values computed once can be reused. Your function returns the cache (creating one if it was passed None), and the SDK passes that same object back on the next call. A typical use is inserting a “job” row once, caching the returned ID, and referencing it as a foreign key in later calls. You decide its shape and when to reset it.

Insert helpers

Different databases impose different limits on how many rows or parameters can go into one statement. The insert module handles this batching for every supported backend (SQLite, PostgreSQL, MSSQL, Snowflake, DuckDB, and the file-based engines). Pass all of your rows to exec_insert and let it chunk them appropriately:

from emtellipro.db.schemas.insert import exec_insert
exec_insert(connection, models.FoundEntity, values)

There are companion helpers for inserts that need to return generated IDs (exec_returning_id and exec_insert_default_returning_id).

Models

Schema tables are ordinary SQLAlchemy ORM models. To keep naming conventions (index, constraint, and foreign-key names) consistent with the rest of the SDK, build your declarative base with create_base instead of SQLAlchemy’s declarative_base directly:

from sqlalchemy import Column, Integer, UnicodeText
from sqlalchemy.orm import DeclarativeBase
from emtellipro.db.schemas.util import create_metadata
class Base(DeclarativeBase):
    # this ensures a constraint naming convention is always used
    metadata = create_metadata()
class FoundEntity(Base):
    id = Column(Integer, primary_key=True, autoincrement=True)
    start = Column(Integer)
    end = Column(Integer)
    text = Column(UnicodeText)

The base derives table names from the class name automatically. The util module also provides a uuid7 helper for time-ordered UUID primary keys.

Save options

Each schema defines its own save options — a dataclass of options that control how documents are stored (for example, a minimum confidence threshold for keeping found entities):

import dataclasses
@dataclasses.dataclass
class SaveOptions:
    #: Drop found entities below this concept confidence.
    confidence_min: float = 0

The SDK discovers this class from the type annotation on your save function’s options parameter, exposed as schema.save_options (it can also be included when calling SchemaManifest() if you wish to avoid the type annotation). When saving through the Python API you can pass an instance directly, or pass keyword arguments that the SDK uses to build one:

db.save(input_docs, annotated_docs, confidence_min=0.5)

On the command line, save options are read from the configuration file, keyed by schema name, so each schema can carry its own settings:

[process.save-options.sample-plugin-schema]
confidence-min = 0.5

Migrations

A schema may optionally ship Alembic migrations by pointing its manifest’s migrations attribute at a migrations module. When present, the migrate command can bring an existing database up to the latest revision. When migrations is None — or when the output is a file-based format — the migrate command reports that migrations are not supported and does nothing.

The migrations module follows the standard Alembic layout (an env.py, a script.py.mako, a versions/ package, and a resources/alembic.ini). The example plugin and the legacy schema both include a working setup you can copy from.

Putting it together

To write your own schema you generally provide, in one package:

1. models — SQLAlchemy ORM models built on create_base.
2. a save function — reads SaveDocument objects and writes rows using the insert helpers.
3. save options — a dataclass annotated on the save function’s options parameter.
4. migrations (optional) — an Alembic module for schema upgrades.
5. a manifest — a SchemaManifest tying the above together, exported and registered under the emtellipro.plugins.schemas entry-point group.

See examples/schema_plugin/ for a complete template, and the built-in legacy schema for a full-featured reference.