Python SDKUsing the SDK in PythonReference

db.database

emtellipro.db.database

The database instance can be used for saving documents to a database.

Module Contents

Classes

NameDescription
MigrationTableStateThe state of the migration tables, if supported. Whether they’re supported depends on both the schema and the database type; a file-based “database” will not support migrations even if the schema does.
DatabaseTestResultThe result of testing the database.
DatabaseTransactionRepresents a single database transaction. This is created using db.begin().
DatabaseThe Database can be used for saving documents to a database. This may be a real DBMS or files on disk.

API

emtellipro.db.database.MigrationTableState

class emtellipro.db.database.MigrationTableState

Bases: object

The state of the migration tables, if supported. Whether they’re supported depends on both the schema and the database type; a file-based “database” will not support migrations even if the schema does.

Initialization
__init__(state, revision)
MigrationTableState.state
state: typing.Literal['empty', 'absent', 'current', 'previous']

The alembic table can be:

  • ‘empty’ when it’s present, but contains no rows;
  • ‘absent’ when it’s not present (unrelated to the fact that maybe all other tables are also absent);
  • ‘current’ when it contains the revision of the latest migration script; or
  • ‘previous’ when it contains a revision ID that’s older than the latest migration.
MigrationTableState.revision
revision: str | None

The revision ID found in the alembic table, if it’s present.

emtellipro.db.database.DatabaseTestResult

class emtellipro.db.database.DatabaseTestResult

Bases: object

The result of testing the database.

This is truthy if the database is ready to use to save documents (this means all the tables are in a consistent state). If the database is empty and it’s safe to create tables, bool(result) will be False.

Initialization
__init__(*, migration=None, columns_found, columns_required)
DatabaseTestResult.migration
migration: emtellipro.db.database.MigrationTableState | None

Type: MigrationTableState | None

This contains info about whether the migration-related tables are in a consistent state. For schemas where database migrations are not supported, this will be None. This will also be None for file-based “databases”, which don’t support migrations regardless of schema.

Value: None

DatabaseTestResult.columns_found
columns_found: set[str]

The columns found in the database, each one prefixed with the table name like "{table_name}.{column_name}".

DatabaseTestResult.columns_required
columns_required: set[str]

These are the columns that are required by the latest schema to be present.

DatabaseTestResult.__bool__
__bool__()
DatabaseTestResult.tables_found
property tables_found() -> set[str]

The table names found in the database.

Return type: set[str]

DatabaseTestResult.tables_required
property tables_required() -> set[str]

The table names required by the schema.

Return type: set[str]

DatabaseTestResult.can_create
property can_create

Whether it’s safe to run the Database.init() method.

This checks if the tables that would be created do not already exist.

DatabaseTestResult.can_migrate
property can_migrate

Whether it’s safe to run the Database.migrate() method, according to the contents of the alembic table (if migrations are supported).

Note that this does not check if the tables are consistent with an earlier schema, so it may be possible that a migration will fail if the schema was modified manually at some point.

emtellipro.db.database.DatabaseTransaction

class emtellipro.db.database.DatabaseTransaction(engine_connection, schema)

Bases: object

Represents a single database transaction. This is created using db.begin().

This should not be instantiated by user code.

Initialization

Parameters:

  • engine_connection: The connection to the engine that’s within a transaction.
  • schema: The schema to use when saving.
DatabaseTransaction.bind
property bind

The underlying Engine to which this transaction is bound.

DatabaseTransaction.connection
property connection
DatabaseTransaction.dialect
property dialect

The underlying engine connection’s dialect.

DatabaseTransaction.get_bind
get_bind()

Return the underlying Engine to which this transaction is bound.

DatabaseTransaction.execute
execute(
    *args, **kwargs
)

Calls out to the underlying Connection.execute() method.

DatabaseTransaction.save
save(
    input_docs,
    annotated_docs,
    cache = None,
    options = None,
    **save_options
)

Save the documents to the database.

Parameters:

input_docs

A list of the input documents that were processed.

Note that if you wish to store document metadata in the documentmetadata or documentstructuredmetadata tables, that’ll need to be included in the input documents.

See the metadata arugment of InputDocument for more details.

annotated_docs

A list of annotated documents from the results of processing the input_docs. Note that the IDs of the input docs and annotated docs must match: each annotated document’s ID must be found in the list of input documents.

When storing to a raw file engine (i.e. raw://...), this must be the Result object instead.

cache

A cache used by the underlying schema save function to track values that should be shared across calls. A potentially new object of the same type will be returned by this function.

If all the calls to save are related in some way, the same cache should be shared by all the calls.

This is useful for cases where (e.g.) there are multiple calls within a processing job, and you want them all to be saved with the same processing job ID, but the job ID is only determined by inserting to a table. Then the ID is cached and the same cache is shared by subsequent save calls so they all store references to the same ID.

options

The SaveOptions instance for use with the schema’s save function. See Database.schema.save_options for the class.

**save_options

If options is None, then these keyword arguments (if present) will be used to instantiate a SaveOptions instance.

Returns:

The cache instance.

emtellipro.db.database.Database

class emtellipro.db.database.Database(uri, schema='legacy', engine=None, file_mode='a', **kwargs)

Bases: object

The Database can be used for saving documents to a database. This may be a real DBMS or files on disk.

Example usage:

db = Database(...)
db.init()       # create tables
# OR
db.migrate()    # if tables already exist and only need be migrated
with db.begin() as conn:
    conn.save(input_docs=..., annotated_docs=...)
Initialization

Parameters:

  • uri: The URL for the database connection. If you’re connecting to a database server, see the SQLAlchemy documentation for full documentation on supported URL formats. This may also be a file-based URL, such as CSV/JSON. The supported formats are the same as for the --output command-line option, therefore see the documentation there for examples.
  • schema: The name of the schema to use with this database, or the Schema object itself.
  • engine: An Engine instance to use instead of connecting using the URI. This is for internal use.
  • file_mode: When using a CSV or JSONL output, this can be set to 'w' to enable write mode on the output files; this means each transaction will overwrite the previous one. It’s important to note that if you leave file_mode='a', you must call the init method to create the files (and write headers where necessary). Otherwise the files will only be created when there’s data to write to them; depending on the processing features enabled this might not be the case. When seting file mode to ‘w’, all the files are unconditionally created when the begin context manager is used.
  • **kwargs: Any extra arguments to be used when instanting the underlying database engine. In the case of file-based databases (e.g. csv, json, jsonl), any extra arguments are ignored.
  • If you’re using Snowflake, you can pass: : snowflake_private_key_path=“path/to/key”
  • If the private key is encrypted, you may also pass the passphrase: : snowflake_private_key_passphrase=“PASSWORD” If you don’t pass in the passphrase, it’ll be attempted to be read from the SNOWFLAKE_PRIVATE_KEY_PASSPHRASE or SNOWSQL_PRIVATE_KEY_PASSPHRASE environment variables. Any extra arguments are passed to SQLAlchemy’s create_engine() function.
Database.name
property name

The name of the underlying engine’s dialect.

Database.test
test() -> emtellipro.db.database.DatabaseTestResult

Attempt to connect to the database and validate it for use with the schema.

Returns:

The validation results when checking the database.

Raises:

Return type: DatabaseTestResult

Database.init
init()

Create all the database tables that are needed.

If the database is already initialized with the current schema, this function will do nothing; if the schema version marker is missing, however, DatabaseSchemaConflict will be raised.

Raises:

  • DatabaseSchemaConflict: The database contains a schema that conflicts with the schema defined in this library.
Database.create_indexes
create_indexes()

Create the appropriate full text search indexes for the database.

Regular indexes useful for joining tables efficiently are always created by default when the tables are created.

Raises:

  • NotImplementedError: The database type is not supported for index. This includes Snowflake and file-based “databases”.
  • MissingFullTextSupport: The database doesn’t support full text indexes.
Database.check_schema
check_schema()

Check whether the schema in the database is compatible with the current SDK release.

“Compatible” means that all the tables and columns that are required for saving annotated documents are present.

Raises:

  • exc.DatabaseSchemaMissingColumnsError: This is raised when there are missing columns in the database.

    This will also include information about the missing columns and the schema version.

    If the schema version is a specific version, then it may be possible to migrate to the latest version.

    If the schema version number is ‘missing’, then it’s not possible to automatically determine which schema version it was for a migration to the latest schema (or if the database is compatible with any older schema version).

    If the schema version is “latest” and there are missing columns, this may indicate that the columns were improperly dropped by some other tool or individual.

  • exc.DatabaseMissingVersionError: Raised when the schemna version number stored in alembic_version.version_num is missing, but all the expected columns are present.

    This indicates that it may be possible to manually add the latest schema version number to that table.

  • exc.DatabaseSchemaValidationError: May be raised as a catch-all error.

  • NotImplementedError: For file-based output formats, where schema checking is not possible.

Database.migrate
migrate()

Migrate the database to the latest schema version.

create_indexes should be called after migration to rebuild the indexes.

Database.drop
drop()

Will drop all tables managed by the SDK.

If the database is file-based (e.g. CSV/JSON), the entire directory will be removed.

Database.clear
clear()

Will DELETE all data rows from the database. This includes only the tables this library manages; any other tables will be left untouched.

This only works for real databases connected through SQLAlchemy. For file-based databases, you should delete the files instead.

Database.begin
begin()

Begin a transaction, returning a context manager. The context manager must be used for storing documents and results in the database.

Example usage:

db = Database(...)
with db.begin() as conn:
    conn.save(input_docs=..., annotated_docs=...)