> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.emtelligent.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.emtelligent.com/_mcp/server. # Changelog All notable changes to the NLP API Python SDK and associated files will be documented in this file. This SDK follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html), although version identifiers adhere to standard Python packaging guidelines specified in [PEP 440](https://www.python.org/dev/peps/pep-0440/). ## 7.2.0 (2026-08-18) ### Added * The database client's `process` and `debug` commands now report throughput numbers alongside the average time per document numbers. The 'time per document' values tell you on average how long the NLP API took to process each document, while the throughput numbers take into account the concurrency in the client (using the `process --jobs` option). * There's a new `emtellipro-db-client list-options` command that lists the available save options for the requested schema. See ["list-options" command](/nlp/python/database-client/options#list-options-command) for usage. * The [User](/nlp/python/reference/data#emtelliprodatauser) now has a `permissions` attribute which includes the user permissions returned by the NLP API. This information is also included in the output of the `emtellipro-db-client get-user` command. ### Changed * The default server set as [DEFAULT_SERVER](/nlp/python/reference/engine#emtelliprodefault_server) is now set to `https://api.us.emtelligent.com`. * When using the database client, if both API key and HMAC access keys are set, then the client will show an error rather than prioritizing the API key. Only setting one or the other is valid. ### Fixed * When processing CCD documents containing structured metadata, the database client would return an error. It now processes them correctly and stores the JSON representation of the metadata in the `documentstructuredmetadata` table as expected. ### Docs * The documentation for `process --save-opt` now includes a links to the migration guide for the options that were present in version 6 and earlier. ## 7.1.0 (2026-08-12) ### Added * The database client now includes run time statistics at the end of processing. This will include the cumulative run time (across multiple restarts with the same state file), as well as the current execution time (producing a similar time to what the `time` command would produce). Note that these times do not include Python startup times, nor module import times, so the "execution time" reported will be slightly lower than what the `time` command would report. ### Changed * The error message presented by the database client when retrying has been exhausted will now include more detail about the last exception that cased the error. ### Docs * The behaviour of the legacy schema when storing relation concept links, Snomed ICD-10 CM concepts, and document text is now documented in a single place: [How the legacy schema works](/nlp/python/database-client/legacy-schema) ## 7.0.1 (2026-07-31) ### Fixed * Ensure only the 'legacy' database schema is listed in list of built-in schemas. ## 7.0.0 (2026-07-30) For migration information see: [Migrating to v7](/nlp/python/features/migrating-to-v7). ### Added * Support for multiple schemas has been added, with plugin support. See [Pluggable database schemas](/nlp/python/features/pluggable-schemas). * There's now a `debug --gantt` option that produces a Gantt chart of the different processing stages. * There's now an `process --retry-failed` flag which will retry any documents that were returned with processing status 'error' by the NLP API. * The "legacy" database schema now accepts a `snowflake_cache_dir` save option which caches database writes to that directory before copying to Snowflake using `COPY INTO`. This can improve saving performance on larger batches where executing `INSERT` statements would be slower. See `snowflake_cache_dir` * The `document-infer-structure` feature is now supported. * When submitting tasks to the NLP API and checking the status, one of the possible task states is "task setup". When the task is detected to still be in the "task setup" state, the time between when the submission started and the returned status being "task setup" is now recorded and is presented at the end of execution. However, if the first returned task state is "processing", then task setup time is assumed to be 0, as it's not possible to monitor true task setup time from the client side. Note that if the `--poll-freq` option is set to 1 or higher, it's unlikely to detect the task setup time since it's usually quite short. * If the `emtellipro-db-client` state file is not compatible with the current version of the SDK, an error message will be presented. ### Changed * The JSON input format for the SDK has now changed to be more structured and thoroughly documented. The old format is still supported, but is deprecated. See documentation for input formats: [Input File Formats Supported By The SDK](/nlp/python/input-formats). * The database client's save options are now set using the `process --save-opt` option, which can be passed multiple times to set different save options. This change was required due to the schema being dynamically set. * Dependency on `requests` has now been replaced with `httpx2` and `tenacity`. * The database client is now multi-threaded and has some changed options. See [Multithreaded database client](/nlp/python/features/multithreaded-db-client) for details. * When creating found entities based on relation concept links, the new found entities did not contain any type name information; now the type names will be parsed out of the corresponding concepts' descriptions. * `type_name` may now contain None values (they were previously filtered out). * It's now possible to pass a floating point number to `process --poll-freq`; the integer portion will be used for the NLP API (as it only supports integers), and any left-over is waited on client-side. This effectively means that one can poll more frequently than once a second by passing something like `--poll-freq 0.5`, however that puts more load on the NLP API so it's not recommended. * The end-to-end notebook example has been updated to import the models from `emtellipro.db.schemas.legacy`. ### Fixed * The database client previously would say it found 0 files when the input paths specified as arguments did not exist, but now that's been fixed to provide an error message specifying the path doesn't exist. * The `insert_or_ignore` parameter for [duckdb_caching_connection](/nlp/python/reference/db-schemas-insert#emtelliprodbschemasinsertduckdb_caching_connection) was incorrectly assumed to only include 'concept', but now any table name is supported. * Specifying processing features as a list in the config file caused an error; now it's fully supported. ## 6.17.0 (2026-06-11) Database migration is required for this version. ### Changed * The data type for the `sectionlocation.name` column no longer has a 255 char limit, and matches the type of `foundentity.section_name`. ## 6.16.0 (2026-05-14) ### Added * Automatic detection of categories and subcategories is now possible using the `--category auto` option, or setting the category/subcategory to `'auto'` or `None` when instantiating [InputDocument](/nlp/python/reference/data#emtelliprodatainputdocument), or as an argument to any function that accepts such a category/subcategory parameter. For more details see: [Inferring the category](/nlp/python/features/infer-category) * An example was added for processing documents from a Snowflake database with the NLP API and storing it in a Snowflake database, designed for use in a Databricks notebook. See `examples/end-to-end-notebook/` for details. ### Fixed * Avoid crash CCD files with empty tables or table rows. ### Changed * The `lxml` dependency version now requires `~= 6.0` to support Python 3.14. ## 6.15.0 (2026-04-09) Database migration is necessary for this version. ### Added * The `documentmetadata` table now contains a `encounterdate` column, with the same type as `chartdate`. ### Fixed * There was a bug when merging entities to create synthetic found entities based on relations that led to an AttributeError. ### Deprecated * Support for HMAC authentication is deprecated and will be removed in the next major release. ## 6.14.0 (2026-03-06) ### Added * DuckDB is now supported for use with the [Database](/nlp/python/reference/db-database#emtelliprodbdatabasedatabase) class, and the database client. * The [Database](/nlp/python/reference/db-database#emtelliprodbdatabasedatabase) class's `file_mode` argument now supports `'w'` for JSONL outputs (i.e. URLs beginning with `jsonl://`, `jsonl+table://`, `table+jsonl://`, which all mean the same thing). * Support for the 'experiencer' attribute on found entity has been added. For more details: [Found Entity Experiencer](/nlp/python/features/found-entity-experiencer) * Support for the 'factuality' attribute on found entity has been added. For more details: [Found Entity Factuality](/nlp/python/features/found-entity-factuality) * 3 new processing features are now supported: * `entity-experiencer` * `entity-factuality` * `entity-contextual-validity` * There are new performance improvements in the database client, so loading files should be significantly faster. ### Fixed * Attempts to create indexes in DuckDB will raise the appropriate NotImplementedError. * The `emtellipro-db-client debug` subcommand now has better error handling when there are issues with the state file. * When connecting to an non-existent database in Snowflake, you may have gotten an exception from the Snowflake driver at an unexpected place; now the SDK will check that the database exists when first connecting to it (this is the behaviour of all other database types, only Snowflake allows you to connect to a non-existent database and will only return an error when trying to query tables within it). * The database client no longer throws an UnboundLocalError exception when retrying failed; it now presents an error message. ### Changed * If using Snowflake, the dependency on `snowflake-connector-python` is now `>= 3.18` because earlier versions appear not to have correctly specified their dependency on `cryptography`. This only applicable if you installed the SDK using the `snowflake` extras. * Python 3.10 is now the minimum Python version required, as 3.9 is now past its EOL. * DuckDB uses UUIDs for the ID columns, not sequential integers. ### Docs * The optional parameters for [save](/nlp/python/reference/db-database#databasetransactionsave) now specify which database columns they affect. ## 6.13.0 (2025-10-23) ### Added * There's now a `check_schema` method that can check whether the schema in the database is compatible with the current SDK version. "Compatible" just means all the expected columns are present and the schema version in `alembic_version.version_num` is correct. If the schema's not compatible, an appropriate exception will be raised. ### Fixed * When attempting to run multiple database clients with the same state file, they'd interfere with each other and cause "database is locked" errors. Now the state file is locked for the entire runtime of the client, so running a second client will immediately error out. * When running many database clients in parallel and storing results in a Postgres database, it was possible to get deadlocks. This has now been fixed by getting an EXCLUSIVE lock on the `concept` table in each transaction; this will allow SELECT statements to execute while inserts are happening. ### Changed * An extra validation step is added to ensure that empty type names are not inserted into `foundentitytype.type_name` column. ## 6.11.0 (2025-10-15) ### Added * The [Emtellipro](/nlp/python/reference/engine#emtelliproengineemtellipro) class now accepts extra headers to include in all requests made using that instance. ### Changed * There is now an extra validation step when saving relation concept links to ensure that concept IDs are not 'null' or "NULL"; these concept links are saved by creating synthetic found entities (see [Relation Concept Links](/nlp/python/features/relation-concept-links) for how those synthetic found entities are created). This is only relevant when saving to a database/file, not when parsing NLP API JSON files in Python. * As an extra validation step, all concept IDs that are 'null' or "NULL" are not stored in the database. ### Docs * Updated snomed concept in [End-to-end Report Processing and Data Extraction Example](/nlp/python/database-client/end-to-end-example) to include entity type. ## 6.10.0 (2025-07-23) Database migration is necessary for this version. ### Added * There is now support for authenticating using API keys (as an alternative to HMAC-based authentication). This means that instead of passing in an access key and shared secret, you now pass in a single API key. For more details: [API Keys](/nlp/python/features/api-key) * There is now support for the new `concept_links` feature for relations. This is available as a new `.concepts` attribute on all [Relation](/nlp/python/reference/data#emtelliprodatarelation) subclasses, which is similar to the same attribute on [FoundEntity](/nlp/python/reference/data#emtelliprodatafoundentity) objects. This new data is stored in the database by creating new found entities that merge the found entities in the arguments of the relations. The new found entities are discontinuous since they combine the spans of each individual found entity. For more details: [Relation Concept Links](/nlp/python/features/relation-concept-links) * The model classes in `models` now contain more relationship definitions. Specifically, the following classes have attributes that link them to more concrete objects `Entity`, `Relation`, and `Location` (with attributes `.entity`, `.relation`, and `.location`, respectively). For example accessing the `.location` attribute would get you the appropriate `*Location` object, thus avoiding having to join the tables yourself. * The `clear` method has been added which deletes all the rows for all tables in the database which the SDK is responsible for (determined by what's in SQLAlchemy's metadata for the Base model in `models`). This excludes the `alembic_version` table. * There's a new attribute: `state` that contains the value of the "state" field in the task status returned by the NLP API. * Subcategory "generic" is now valid for category "gastroenterology". * The `foundentity` table now contains two new columns: text\ The concatenation of the text for each span, separated by a single space character. For most single-span entities this will be the same text as the text in the `foundentityspan` table, but some entities are discontinuous so this'll contain all the discontinuous components in one string (this will happen for the found entities created from relations' concept links). start\ The start offset of the found entity; this is effectively the minimum of the associated `foundentityspan.start` values. ### Changed * If you hit `Control-c` while the SDK client is processing, you now get prompted on whether you want to cancel the last task that was submitted. * The default category when using the command-line client has been changed from 'Radiology' to 'Clinical' (see `process --category`). This only affects command-line usage. There are no changes to the Python API (as the Python API always required the category to be explicitly set by the caller). * The retrying behaviour of the database client has been reworked so that only specific errors are retried rather than retrying everything by default. Now only errors that are clearly transitory (such as connection errors) are retried. Everything else is assumed to be a permanent error. * The category and subcategory values are now validated up front rather than submitted as is. * When processing documents using the database client which come from a SQL database (i.e. when using `process --sql-query`), the database client started a transaction but didn't call COMMIT at the end. The database client will now automatically call COMMIT after loading all the documents from the source database. ### Fixed * The `database` option in the client's config file was able to be called both `output` and `database` in the `[emtellipro]` section, but only `database` in other sections. It's now possible to use `output` and `database` interchangeably everywhere in the config file. * When using the database client to store to a `raw+json://` or `raw+jsonl://` output path, and the entire processing task has failed (on the engine side), you'd get an exception raised because there was no JSON to store for the failed task. Now an empty JSON object will be stored for those cases, ensuring that the number of output files match the number of tasks created (whether the tasks succeeded or failed). * Fixed crash where the NLP API returned extra attributes on relations and the parsing code didn't handle them. * [Flattener](/nlp/python/reference/flatten#emtelliproflattenflattener) class now correctly includes spans when asked. * When installing the `[snowflake]` optional extras dependencies (when installing the wheel file), the `cryptography` dependency was not directly specified; it's now included since the SDK explicitly uses it for Snowflake private key files. * Validation of category and subcategory was missing when loading documents from JSON files, and so the error was only noticed when submitting to the NLP API. The category/subcategory is now validated when input JSON files are parsed. ### Docs * The license for this SDK is no longer included in the `.zip` or `.tar` file. The current license is included separately on the download page. * Added documentation for how document metadata can be handled when using the SDK in Python. See the `metadata` argument of [InputDocument](/nlp/python/reference/data#emtelliprodatainputdocument) for more details. * The help message for `process --features` now lists all the available features. * The supported categories and subcategories when submitting to the NLP API are documented as [CATEGORIES](/nlp/python/reference/engine#emtelliproenginecategories). * Documented that the URL format for `Database` is the same as `--output`, while also clarifying that the URL format for Snowflake is the one documented in Snowflake's SQLAlchemy driver. * Clarified that the maximum submission size is 500MB, not 500MB per input document. * The link to the CCD metadata schema on the [Structured entries as metadata](/nlp/python/ccd/metadata) page is now fixed. ## 6.9.0 (2025-03-12) ### Added * [heading](/nlp/python/reference/data#sectionlocationheading) has been added, which contains the heading linked to the section (if any). This was previously just found by looking at the heading locations and checking the corresponding [section](/nlp/python/reference/data#headinglocationsection) attribute. * Added indexes on the following `documentmetadata` columns on all databases except for Snowflake: * `chartdate` * `document_id` * `original_category` * `original_subcategory` * `source_document_id` * `subject_id` * `subject_id_2` * `subject_name` * Added support for relation confidence scores. These will all be floating point numbers or None. Each relation table now contains a `confidence` column: * `experiencerrelation.confidence` * `followuprelation.confidence` * `measurementrelation.confidence` * `imagelinkrelation.confidence` * `qualifierrelation.confidence` * `medicationrelation.confidence` * `temporalityrelation.confidence` * `reportedeventrelation.confidence` * `anatomicsiterelation.confidence` Each relation data class now contains a `.confidence` attribute: * [confidence](/nlp/python/reference/data#experiencerrelationconfidence) * [confidence](/nlp/python/reference/data#followuprelationconfidence) * [confidence](/nlp/python/reference/data#measurementrelationconfidence) * [confidence](/nlp/python/reference/data#imagelinkrelationconfidence) * [confidence](/nlp/python/reference/data#qualifierrelationconfidence) * [confidence](/nlp/python/reference/data#temporalityrelationconfidence) * [confidence](/nlp/python/reference/data#reportedeventrelationconfidence) * [confidence](/nlp/python/reference/data#anatomicsiterelationconfidence) See more detail: [Relation Confidence](/nlp/python/features/relation-confidence). * Quickstart documentation now uses the database client rather than the deprecated simple client. * The database client now lists documents that failed processing to stderr at the end of processing. * The `emtellipro-db-client debug` command now lists files and document source IDs that failed processing. See ["debug" command](/nlp/python/database-client/options#debug-command) for more details. * The [Database Client Options](/nlp/python/database-client/options) document contains more details about the arguments each database client command accepts. * The documentation for the command line options for the database client has been improved so each option can be linked to, and shared options are duplicated for each command that shares it. * The Snowflake documentation page has been updated with new tables. [Using the Database Client with Snowflake](/nlp/python/database-client/snowflake) ### Changed * When attempting to use the database client's `migrate` command on an empty database, the user is presented with a more clear error message about it. ### Fixed * Client: restarting processing when documents fully completed processing would raise an error. Now it will exit without error. * The documentation for objects in `engine` did not include the return types for all the methods; they now include better descriptions of what they return. * When using the JSON file input format for the database client with empty strings for `subject_dob` or `chartdate`, there was previously an error when storing the metadata to the database because empty strings are not valid for datetime columns; the empty strings are now cast to NULL. * Added error handling to database client for a known bug where the NLP API may not return all failed documents in the result output. These "missing" documents are treated as failed for the purposes of `--store-failed`. * There was a bug when using the database client to process a PDF file, which would lead to an error message about invalid path type. * When storing files containing source data (e.g. PDFs) to the database using the Python API you'd get an error about invalid data type. The source file path is now converted to a string before storing to the database. * The keyword arguments documented for `save` should be optional, but the code required that some were passed in. They're all optional now, as documented. * The `--store-failed` flag was not storing failed documents correctly; now it does. * When an entire task failed processing, the resulting exception caused the database client to retry the request, even though the failure was permanent. Failed tasks are now logged and if `--store-failed` is used the documents within the task will all be treated as failed. * The database client failed to print to stdout the number of documents that failed processing. Now the failed document count is printed at the end of processing. * The relationship definition between the `SectionLocation` and `PageLocation` SQLAlchemy models is now fixed so the SQLAlchemy warning is gone. ## 6.8.0 (2025-01-22) ### Added * The `Concept` class now contains `map_rule`, `map_advice`, `map_group`, `map_priority`, and `map_category` attributes which may be present for the `snomed_icd10_cm` ontology. ### Changed * The log level for the case where a submitted document was not returned by the NLP API has been increased from WARNING to ERROR. * The minimum required Python version is now 3.9 since Python 3.8 has passed its EOL date. * Concepts in the `snomed_icd10_cm` ontology may be skipped or may have their concept IDs modified based on the new `map_rule` and `map_advice` attributes when being stored to a database. If the rule contains a conditional, the new concept ID will have format: `concept_id | map_rule | map_group | map_priority`. ### Fixed * If attempting to use `--filetype json` flag with a non JSON file (including JSONL), a better error message will be presented than a `JSONDecodeError`. Note that JSONL and JSON filetypes are not compatible, so for JSONL files `--filetype jsonl` would be appropriate. The `--filetype` option is optional, though, and by default filetype detection is done by looking at the file extension. ## 6.7.0 (2024-12-16) Database migration is necessary for this version. ### Added * There are 3 new columns in the `documentmetadata` table: `clinical_domain`, `source_document_id_2`, and `source_document_id_3`. ### Changed * Log CCD "unable to resolve metadata item" message as warning instead of error. ### Fixed * The `file_mode` parameter introduced in 6.6.0 had a bug when `file_mode='a'` which didn't write the headers to the files. * The `--no-store-pages` and `--no-store-headings` flags were incorrectly worded. This is now fixed. * If the NLP API returned unsupported fields in its output, that would lead to an exception in the SDK. This is now fixed to ignore unsupported fields. ## 6.6.0 (2024-11-15) Database migration is necessary for this version. ### Added * Added `file_mode` parameter to `Database`. This is only supported for CSV database URIs. * Added support for the `concept_confidences` attribute in found entities. This is enabled using the `entity-concept-link-confidence` feature flag. The confidences can be found in [concept_confidences](/nlp/python/reference/data#foundentityconcept_confidences), as well as the `foundentityconcept.confidence` database table column. See more detail: [Concept confidence](/nlp/python/features/concept-confidence). * Added support for heading locations. This is found in the `'heading'` key of `locations`. The data for this location will be stored in the new `headinglocation`, `headinglocationspan` tables. Storing text of the page is now controlled using the `--store-headings` CLI flag. See more detail: [Heading locations](/nlp/python/features/heading-location). * Added support for page locations. This is found in the `'page'` key of `locations`. There is also the new attributes `pages`. The data for this location will be stored in the new `pagelocation`, `pagelocationspan`, and `pagesection` tables. Storing text of the page is controlled using the `--store-pages` CLI option. See more detail: [Page locations](/nlp/python/features/page-location). * Added support for the 'heading_status' attribute of found entities. This is available as [heading_status](/nlp/python/reference/db-schemas-legacy-models#foundentityheading_status), and is found in the database in the new `foundentity.heading_status` column. See more detail: [Entity heading status](/nlp/python/features/heading-status). * Added support for the new 'heading' location type. ## 6.5.2 (2024-10-11) ### Fixed * JSON input file validation is now improved to check that all metadata values are strings or 'null'; conversion to strings is not done automatically because it may be ambiguous, so it's left to the user to convert any non-string object to a string. * When the database client's `--sql-query` parameter has a syntax error or the input database URL is invalid, better error messages are now presented. ## 6.5.1 (2024-10-09) ### Added * The database client's `debug` command now accepts a `--json` flag which tells it to produce its output as a JSON object. The exact structure is not documented however since it's intended just for debugging and may change from version to version. ### Fixed * There was an intermittent error when input documents are in Snowflake where the connection to Snowflake would be closed by the Python interpreter before SQLAlchemy cleanly reset it right at the end of the database client's execution; this had no impact on processing or storage. ## 6.5.0 (2024-09-16) Database migration is necessary for this version. ### Added * When using JSON/JSONL input documents with the database client, it's now possible to specify both the plaintext contents of the document and the PDF document source from which it came. See [Full format definition](/nlp/python/database-client/overview#full-format-definition) for more details (specifically the `pdf` and `pdf_path` keys). This functionality can be used when you've extracted the text from the PDF and wish to use that text for processing, but still keep a track of the original PDF it came from. There are 3 new database columns to support this in the `document` table: `source_path` (if the `pdf_path` JSON key is set), `source_type` (only `application/pdf` is supported now), and `source_size` (the number of bytes in the source PDF). There is also a new database table: `documentsourcechunk`. This is used for storing the source data, split into chunks that fit into the maximum allowed for the `documentsourcechunk.data` column (BYTEA for PostgreSQL, LONGBLOB in MySQL/MariaDB, and VARBINARY(max) in SQL Server). There is also a new `process --store-pdf` option to enable storage of the source data into `documentsourcechunk`. This is only supported for PostgreSQL, MySQL, MariaDB, and SQL Server. No storage will happen for any other output formats. ### Fixed * When processing large numbers of CCD documents using the database client, the state file would grow much larger than the input documents due to large amounts of metadata read from the CCD; the metadata is now compressed so the state file will be smaller than the input documents. * Storing data containing datetimes in an SQLite database would fail if the datetimes were strings, because the SQLite driver requires `datetime` objects. Now the strings are parsed before insertion. Parsing the datetime strings only happens in SQLite. For all other database types, the strings in the input metadata are passed along unchanged to the database. ## 6.4.0 (2024-08-28) ### Added * Expanded documentation and added more examples. ## 6.3.2 (2024-08-26) ### Fixed * Since 6.0.0 the database client did not insert anything into the `documentmetadata` table if there was no data to insert. Now it will insert a row with just the `document_id`, which was the behaviour in 5.x. ## 6.3.1 (2024-08-21) ### Fixed * When the metadata for [InputDocument](/nlp/python/reference/data#emtelliprodatainputdocument) contained a datetime object, this was not serialized correctly and would raise an exception. Serialized datetimes is now fixed. * If there was an error while ingesting files, and the database client was restarted, it would exit early saying processing was already completed (with 0 documents). Now it correctly checks whether the file ingesting successfully completed on restart. ## 6.3.0 (2024-07-30) ### Added * The client now supports `raw+jsonl://PATH.JSONL`, where `PATH` is the path to a JSONL file. This will append the results to that file, along with `PATH.map` file that contains a mapping between document IDs and input file paths. This format matches the output format of the simple client that was available in v5. * The simple client from version 5 has been brought back, but it remains deprecated; it's available as `emtellipro-client` as before. * The `emtellipro-db-client store-result` command now accepts directories containing JSON files as arguments (previously it only accepted the JSON files themselves). * The `-o, --output, --database` option can now be passed directly to `process`, `create-db`, `migrate`, and `store-results`, rather than as a global option before. The new position is encouraged and passing it globally is deprecated. The same applies to the `--snowflake-private-key-path` option. This change clarifies which commands require a database/output option, since commands like `get-user` don't use a database. Example: ```bash # before emtellipro-db-client -o postgresql://... create-db # now emtellipro-db-client create-db -o postgresql://... ``` * The command-line help messages for the database client have now been expanded and generally improved. * The `emtellipro-db-client debug` command now provides progress information, and nicer formatting of the parameters. ### Changed * The state file used for `emtellipro-db-client process` is now an optional parameter. If it's not passed, a state file will be generated and its location will be printed out and logged. If a state file is provided, its progress statistics are now printed out. * Dropped support for SQLAlchemy 1.4 as indicated in the 6.0.0 release notes, given that `snowflake-sqlalchemy >= 1.6` now supports SQLAlchemy 2.0. > **Warning** > > If you're using Snowflake, you must ensure that you have `snowflake-sqlalchemy >= 1.6` installed, because earlier versions will raise an exception when used with SQLAlchemy 2.0. The easiest way is to use the `pip install -U` flag when installing it. > > You may also install the SDK using the the `[snowflake]` extras which will install the correct `snowflake-sqlalchemy` version: ```bash pip install emtellipro-6.3.0-py3-none-any.whl[snowflake] ``` * The database client will now insert up to 16,384 rows at once into Snowflake database. ### Fixed * When running the database client in a location where `$PWD/errors.log` was not writable, and no alternative log file was provided using `--log-file`, the client would crash. Now it provides an error message stating the log file path is not writable. ## 6.2.0 (2024-07-04) ### Added * The flatten JSON script from `examples/` has now been added to the core SDK. It's available as `emtellipro-flatten-json` after installing the SDK. Documentation for usage is found in [Flatten JSON](/nlp/python/flatten-json). * There's now a new Python module that provides similar functionality to the `emtellipro-flatten-json` CLI tool; example usage is found in `flatten`. * The `emtellipro-db-client` command now takes `-o, --output` as an alias for `--database`. ## 6.1.0 (2024-06-13) ### Changed * Updates to CCD LOINC to section name mapping. * Default LOINC code for CCD sections, to use in the case where a section has one or more LOINC codes from XML elements but none are in our mapping. * Improvements to documentation for CCD section normalization, especially for LOINC code handling. ## 6.0.0 (2024-05-24) ### Added * The `documentmetadata` table now contains 3 new columns `subject_id_2`, `subject_address`, and `document_summary`. * [AnnotatedDocument](/nlp/python/reference/data#emtelliprodataannotateddocument) now takes a `text` parameter which contains text of the document; this is then used for filling in the `.text` attributes of contained objects when the text was not contained in the result JSON object. * CCD files can now be loaded from a database by using `--sql-query QUERY --text-type ccd`. * CCD handling now normalizes section titles based on LOINC codes if present, if there is no OID to normalize with. * Added example of result JSON parsing to the Python notebook in `examples`. * Improved example of result JSON parsing in the Python notebook in `examples`. * [FoundEntity](/nlp/python/reference/data#emtelliprodatafoundentity) now has [start](/nlp/python/reference/data#foundentitystart) and [stop](/nlp/python/reference/data#foundentitystop) attributes, which provide easy access to the overarching character span. * [Span](/nlp/python/reference/data#emtelliprodataspan) now has a [slice](/nlp/python/reference/data#spanslice) attribute which returns an equivalent `slice` object. * Support for anatomic site relations has been added. This includes * `anatomicsite-relations` feature option * 'anatomic-site' key in `relations`, which contains [AnatomicSiteRelation](/nlp/python/reference/data#emtelliprodataanatomicsiterelation) objects * `anatomicsiterelation` table The new relation contains `site` and `situated_entity` attributes. * Added a new `processingdetails.sql_query` database column. When the database client reads documents from a database, the SQL query used will be stored in this column. * The database client now accepts both `--store-sentences` and `--store-sections` flags that can be used instead of `--store-sections-and-sentences`. * When reading documents from a database, the client will now store any extra columns in the SQL query in the `documentstructuredmetadata` table. The extra columns are any columns other than 'id', 'text', or ones that match the `documentmetadata` table columns. ### Changed * Updated to lxml 5 for CCD parsing. * Support both SQLAlchemy 1.4 and 2.0; support for 1.4 will be dropped in the next minor release when the Snowflake SQLAlchemy driver is updated. * The SDK has been rearchitected to make it easier to use by people interacting with it in Python to write their own code for loading files, submitting them for processing and saving them to a database. #### Major changes * The [Emtellipro](/nlp/python/reference/engine#emtelliproengineemtellipro)` class now takes the access key and shared secret as a tuple to be passed as an argument to ``auth`\`. This allows for different authentication methods in the future. * Support for plugins has been removed. * There is now a [Database](/nlp/python/reference/db-database#emtelliprodbdatabasedatabase) class (also available as `emtellipro.db.Database`). This provides a method for end users to interact with the database, creating tables, running migrations, and store processing results. * There is now a [readfile](/nlp/python/reference/load#emtelliproloadreadfile) function that provides a simple method for reading a file; it loads a single path that can potentially contain multiple documents. * The [InputDocument](/nlp/python/reference/data#emtelliprodatainputdocument) class now takes a `metadata` argument containing arbitrary key/value pairs that can be later stored in the database. * The [InputDocument](/nlp/python/reference/data#emtelliprodatainputdocument) class now takes a single `data` argument which must be either bytes or a string, replacing the multiple arguments for plaintext vs PDF documents and the file path. Loading files will now happen outside the document. * [locations](/nlp/python/reference/data#foundentitylocations) is now a dictionary, to match `locations`. To easily access the first sentence in the sentences list, you can also use [sentence](/nlp/python/reference/data#foundentitysentence). * The `emtellipro_db` package is replaced by the `db` module. * There is now a single SDK client, which is the old database client. It provides all the functionality that was previously available in the simple client. * The SDK client now requires a `--state, -s` option that stores its state in a local file, allowing for re-starting processing. * The client's `--max-retries` and new `--retry-forever` options will retry processing and storing when errors occur; the `--retry-forever` may lead to an infinite loop, however, so it's important to check the log file to see why the errors are occurring. * The client has a `store-result` subcommand which takes paths to JSON files produced by using the `raw://` file output option for the `process` command and stores it to a database. * The client now supports reading the NLP API server, access key, and shared secret options from the `EMTELLIPRO_SERVER`, `EMTELLIPRO_ACCESS_KEY`, and `EMTELLIPRO_SHARED_SECRET` environment variables, respectively. These will override the matching options from the config file, but if you explicitly pass those options to the client (e.g. using `--server` option), the explicitly passed options will take precedence over the environment variables and the config file. * There's a new API for reading CCD files in `ccd`. * Using SQL server as the target database will now require using Microsoft's official drivers. * The database client's `--bulk-insert` and `--fast-postgres` options have now been removed. The bulk insert mode is the only method for inserting rows into target databases. ### Fixed * The package will now work if it's somehow imported without being installed (for example if it's converted to a zip file and imported directly from the zip file). ## 5.30.1 (2024-03-22) ### Changed * Change Sphinx documentation theme to furo. ## 5.30.0 (2024-03-04) ### Changed * Have CCD metadata pick up `` as a code element if there is no other code element at the same level. * Re-trying requests will now re-compute the authentication header before sending the new request. ## 5.29.0 (2024-01-18) ### Added * Log messages mentioning errors in JSON files will now also include the document ID in the JSON file alongside the filepath. ## 5.28.0 (2023-10-23) ### Added * JSON input files can now contain a `"filepath"` item which points to a file containing the actual data for the document. This can be either a `.txt` or `.pdf` file. The path should not be in the same directory as the JSON file, or else there's a risk the document will be processed twice. ### Changed * Updated JSON example in documentation to show all the possible metadata options. * New CCD config option `report-subsection-sections` in config section `[file_handlers.ccd]`, to control what sections the report subsection feature is applied in. See new CCD config docs for details. ## 5.27.0 (2023-08-22) ### Added * When the database client detects that it's running outside a virtual environment it will print a warning to let the user know. * `APIError` exceptions (and subclasses) now contain a `task_id` attribute when available. * The JSON files used as input for the database client can now contain structured metadata. This metadata must be stored as a mapping in a `"structured_metadata"` key and will be saved to the `documentstructuredmetadata` table; if this key is missing, then no structured metadata will be stored, as before. * The database client will now include version numbers when listing installed plugins. * The CCD plugin has now been merged into the SDK and the database client now natively supports CCD files. The separate plugin is now ignored if it's installed. ### Deprecated * The database client's `--bulk-insert` and `--fast-postgres` options are deprecated; the bulk-insert mode will be the only insert method moving forward, since it supports all database types and is faster. ## 5.26.0 (2023-03-17) Database migration is required for this version. ### Added * The `radplaybook-ontology` the NLP API feature is now supported. * There's now support for the `date_time` argument of medication relations. This is stored in `emtellipro.data.MedicationRelation.date_times` (as a list), and in the `medicationrelationdatetime` table. * The database client is now able to save the unprocessed results from the NLP API to disk instead of in a database. You can use this by setting `--database raw://SOME_DIRECTORY_PATH` for the `process` command; the `create-db` command is not necessary because the `process` command will automatically create the target directory if missing. * Support has been added for the `guidance` attribute of found entities. This is now found in the SDK as `FoundEntity.guidance` (where it is stored as a string), and it's stored in the database in the new `foundentity.guidance` column. The new feature is called `entity-guidance`. This change will require a database migration. * Support is now available for the `ontology_versions` returned by the NLP API for each processed document. These are available as `AnnotatedDocument.ontology_versions` and in the `ontologyversions` table. The database migration will insert rows in the `ontologyversions` table based on the `documentconcepts` table, but with `release_version = NULL` since previous ontology version data is not available. ## 5.25.0 (2022-10-17) ### Added * You can now use `-q` as an alias for `--quiet` for the database client. * Error log messages are now printed to STDERR when `emtellipro-db-client process --quiet` is used; to hide the error messages you can pass `--quiet` (or `-q`) twice. In the config file, this can now be controlled by setting `quiet = 2` (using the old `true` and `false` values still works, but integer values are also supported). * In the `emtellipro.data` module, there are now `FoundEntity.attributes`, `Relation.attributes`, and `Relation.arguments` instance attributes which tell you the names of the instance attributes which are relation attributes and relation arguments, respectively; each `Relation` subclass have these attributes. This provides some support for introspection to allow writing generic relation-handling code. ### Changed * The JSON file loader in the database client now validates input JSON documents and provides descriptive error messages. * Document that Python 3.7 is now the minimum supported version, since Python 3.6 is no longer officially supported. * The `emtellipro_flatten_json` example uses the new data model metadata to look up attributes and arguments. New found entity attributes, new relation types, and new relation attributes and arguments will be handled automatically as long as their values fall into the general formats already supported. There are some changes to table names and column order. Relation tables are now produced only if non-empty. * When cancelling tasks, the HTTP `PATCH` method is now used instead of `GET`; both methods are still supported by the API, but `GET` is deprecated. ### Fixed * Filenames with extra dots in them would previously not have the file extension parsed correctly, and so would not be read ## 5.24.0 (2022-07-04) ### Added * When running `emtellipro-db-client process --help`, the defaults for all options are now printed. * The database client now takes a `-c / --config` option which allows you to pass in a configuration file containing options. All command-line options are supported (simply remove the `--` from the beginning of an option to get the config file version). * The `emtellipro-db-client process` command now accepts a `--filetype` option which tells it to treat all input files as if they had the given file type (thus disabling detecting file type based on file extension). The associated config file option is called `filetype`. See `process --help` for available file types (since it depends on installed plugins). ### Changed * API requests will now automatically time out after 90 seconds. ## 5.23.0 (2022-05-17) Database migration is required for this version. ### Added * The SDK's version is stored in the new `processingdetails.sdk_version` column. * The `emtellipro-db-client process --doc-id-filepath` option now supports JSON and JSONL files. * Support for the "reported event" relation in `examples/emtellipro_flatten_json/`. * Support for the `snomedicd10cm-ontology` feature has now been added. * There is now support for the "reported event" relation. This is accessible using `emtellipro.data.ReportedEventRelation`, and in the database in the `reportedeventrelation` table, with arguments in the `reportedeventrelationtoentity`, `reportedeventrelationfromentity`, `reportedeventrelationmodifier`, `reportedeventrelationtimeexpression` tables. The feature name is `reportedevent-relations`. * Example `examples/emtellipro_flatten_json/` which reads NLP API JSON result files and uses the SDK's data model to convert a flatter format. * The database client now stores the engine version returned by the NLP API in the new `processingdetails.engine_version` column; if not returned, this will be NULL. * The `documentmetadata` table now has 2 new columns: `subject_gender`, and `requestor`, which can be populated by providing the same field names in the input documents' metadata. ### Fixed * Some database errors were leading to the database client to exit with code 0 instead of 5 (even though the error was printed and logged). ## 5.22.0 (2022-03-03) ### Added * The database client now accepts a `process --doc-id-filepath` flag which tells it to use the filepath of the input document as the document ID when submitting to the server; this is currently only supported for `.txt` documents. * The database client now allows configuration of the logging path and level through the `--log-file PATH` and `--log-level LEVEL` options, which go in the same position as the `--database` option (i.e. in between `emtellipro-db-client` and the subcommand); see `emtellipro-db-client --help` for details. * The database client now takes a `process --polling-frequency SECONDS` option which adjusts how often it polls the NLP API for status updates on the job processing status; by default it waits for 1 second between status checks, but for jobs with very large documents it might make sense to increase that value. ### Fixed * Previously it was not possible to run multiple database clients in parallel using the `--no-bulk-insert` option due to uniqueness constraints on the `concept` table; this was not an issue with `--bulk-insert`, and now `--no-bulk-insert` has been fixed to allow multiple clients to insert into the `concept` table simultaneously. ## 5.21.0 (2022-01-27) Database migration is required for this version. ### Added * in addition to CSV and JSONL output from the database client, it now also supports JSON output. You can use this by setting `--database json://SOME_DIRECTORY_PATH` for the `process` and `create-db` commands. * the simple client now takes a `--max-retries` option in the same position as the `--server` option, and the database client now takes the same option for the `process` command. This option specifies the number of times to retry failed API requests, and is enabled default, with a default value of 5, using exponential backoff between retries. There is a maximum wait time between retries of 120 seconds, so setting `--max-retries` to large values will eventually result in multiple retries at 120 second intervals. The intention behind this option is that API requests may fail due to network issues, so this will allow continued processing in the case of transient network errors. * the database client now accepts a `process --store-failed` option which will store details about documents that failed processing (i.e. ones where `processing_status` returned by the NLP API is `"error"`). * the database now contains a `document.processing_status` column which contains the processing status returned by the NLP API for the document. This will usually be `"success"` unless `--store-failed` is passed to the database client's `process` command, in which case it may be `"error"`. * the database client has improved error handling and reporting about errors when submitting reports for processing ### Fixed * the database client would raise an error when saving temporality relations using `--no-bulk-insert` due to an incorrect relationship definition. This affects version 5.20.0 and later. ## 5.20.0 (2022-01-17) Database migration is required for this version. ### Added * Temporality relations are now supported. They're found in `emtellipro.data.TemporalityRelation`, and in the database in the `temporalityrelation` table, with the modifiers in the `temporalityrelationmodifier` table. ## 5.19.0 (2021-12-08) Database migration is required for this version. ### Added * The `documentmetadata` table now has 3 new columns: `author_name`, `subject_name`, and `subject_dob`. ### Fixed * The `migrate` command for the database client would raise an exception when trying to migrate CSV or JSON files (which is not supported). It now prints an simple error message instead. * There was performance regression introduced in 5.17.0 when saving to SQL Server, which caused significantly reduced storage speed. This was improved in 5.18.0 and has now been completely fixed. ## 5.18.0 (2021-11-26) ### Added * Support has been added for the `question_status` attribute of found entities. This is now found in the SDK as `FoundEntity.question_status` (where it is stored as a string), and it's stored in the database in the new `foundentity.question_status` column. The new feature is called `entity-question-status`. This change will require a database migration. ### Fixed * there are now some performance improvements when saving to SQL Server, especially when using Microsoft's ODBC driver instead of FreeTDS. * creating an `InputDocument` with an empty string as the text raised a `ValueError`, but now empty strings are allowed. ## 5.17.1 (2021-10-13) ### Fixed * There was a bug introduced in 5.17.0 when exporting to CSV which caused a crash; this has now been fixed. * The `--features` option was not parsed properly in 5.17.0; this has now been fixed. ## 5.17.0 (2021-10-08) ### Added * The database client's `process` command now accepts a `--max-save-shard-size` which specifies the number of reports to store to the database at once; this is ideally used for lowering the default value from 50, since large numbers can cause database errors. * You can now press CTRL-C when the database client is running and it will finish saving to the database what it has so far, and will exit cleanly. Pressing CTRL-C a second time will cause it to exit immediately. ### Changed * The database client is now multi-threaded, so it can read files, submit them for processing, and store results to the database in parallel. * The database client's `--max-submit-shard-size` default is now 100, to avoid having the database thread waiting too long for results. ## 5.16.0 (2021-09-23) ### Added * Support for key-pair authentication for Snowflake has been added. All commands now take `--snowflake-private-key-path` as an option alongside `--database` which is the path to the private key used for the connecting to Snowflake. If using this option, omit the password in the connection URL. To be consistent with SnowSQL, the passphrase for this key file can be passed using `SNOWFLAKE_PRIVATE_KEY_PASSPHRASE` or `SNOWSQL_PRIVATE_KEY_PASSPHRASE` in the environment, or if those environment variables are not set, the client will prompt for the passphrase if necessary. Both encrypted and unencrypted key files are supported. ## 5.15.0 (2021-09-20) ### Added * There is now a plugin architecture based on signals. * Improved error handling when running `create-db` on an existing database, and checking whether the database schema version stored in the `alembic_version` table has been accidentally deleted. * If the `alembic_version` table has been accidentally cleared, the `create-db` command will now allow the user to re-insert the database schema version in that table. * The simple client now has a `cancel` command that allows cancelling one or more tasks. See `emtellipro-client cancel --help` for available options. * The simple client will now print the last unfinished task ID when it receives a SIGINT (e.g. from CTRL-C) while a job is in progress ### Fixed * There was a memory leak in the SDK which has now been fixed, so running the simple client or database client will now use a stable amount of memory. ## 5.14.1 (2021-08-09) ### Fixed * The last version of the SDK introduced a bug where JSON documents weren't counted properly for the progress bar, and JSON document IDs were not properly attached to filenames, leading to an error message. This is now fixed. ## 5.14.0 (2021-07-29) ### Added * The database client now supports saving output to CSV or JSONL files. Simply use `--database csv://PATH_TO_DIRECTORY` or `--database jsonl://PATH_TO_DIRECTORY` for the `create-db` and `process` commands, instead of the regular database URLs. The directory will be populated with files named after each table in the regular database schema, and all ID columns will use UUIDs in standard format (36 characters, with the hyphens). ### Changed * The database schema when saving to Snowflake now uses UUIDs defined as `CHAR(36)` for all the ID columns (and associated foreign keys). These are generated locally by the database client and will lead to significantly faster insertions for Snowflake users. This will require running the `migrate` command to update existing Snowflake databases. Other databases will continue using auto-incrementing integer IDs. * When using `--max-submit-shard-size` with the database client, it will now only load the specified number of input documents at once (previously all input documents were loaded unconditionally, and this option only limited how many were *submitted* in each shard). This option can now be used to reduce memory usage when there are many input documents. ## 5.13.0 (2021-07-15) ### Added * The database client's `process` command now accepts a `--skip-database-checks` option which skips checks for database consistency (i.e. checking all the tables exist and have the correct columns). ### Fixed * When reading input documents from JSON files and storing results using `--bulk-insert` the database client failed to store document metadata into the `documentmetadata` table if the set of metadata keys weren't the same across all JSON files. This has now been fixed and JSON files can have differing metadata keys. ## 5.12.0 (2021-06-24) ### Added * The database client will now use `INSERT IGNORE` in MySQL and `INSERT OR IGNORE` in SQLite to speed up inserts into the `concepts` table when using the `--bulk-insert` option * The database client now accepts a `--max-submit-shard-size` option which allows restricting the number of documents submitted at once to the API. If unset, the client will continue determining shard size based on document size, so this is mostly useful for restricting shard size further than normal. ### Fixed * On certain reports, when using the `--bulk-insert` option the database client would attempt to insert duplicate concepts in the `concepts` table which would fail due to the uniqueness constraints on the table. Inserted concepts are now de-duplicated before insertion to fix this issue. ## 5.11.0 (2021-06-18) ### Added * There is now support for the "duration" and "indication" arguments for medication relations. Like all the other medication relation arguments, these will also be stored as lists. They can be found in the `MedicationRelation.duration` and `MedicationRelation.indication` attributes and `medicationrelationduration` and `medicationrelationindication` tables, respectively; the new tables will require a database migration. ## 5.10.1 (2021-06-08) ### Changed * Stopped Snowflake's SQLAlchemy driver from logging INSERT queries to `errors.log` * Improved performance of Snowflake insertions ### Fixed * Specified SQLAlchemy version must be at least 1.4 ## 5.10.0 (2021-06-08) ### Added * Snowflake databases are now supported for storing data using the database client * Bulk inserts (the `--fast-postgres`) option is now available for all databases, and is called `--bulk-insert`); this will lead to some speed improvements especially in the cases of databases which support returning primary keys on inserted rows without an extra query (such as PostgreSQL and SQL Server). Postgres still has the best support in this case since it allows inserting missing entries into the `concepts` table in a single query. * Regular storing of documents one at a time (without `--bulk-insert`) also has received some speed improvements by grouping inserts more efficiently. ## 5.9.0 (2021-05-13) ### Added * The `foundentity` and `assumedentity` tables now contain a `document_id` column with a foreign key referencing `document.id`; this should make it simpler to query found entities and assumed entities in a document without having to join to the `entity` table (although the `entity` table is unchanged so older queries will continue working). This will require a database schema migration using the `migrate` command. * Added `AnnotatedDocument.processing_status` attribute which stores the response of the engine for the "processing_status" JSON attribute. This is used for checking for failed reports in the database client (in the case that the entire task isn't marked as failed). ### Changed * Modified the type of the `document.json_representation` and `document.text` columns in MySQL from TEXT to LONGTEXT, since depending on the document size, TEXT might not be able to hold all of the data. * When you enable `--fast-postgres` with an unsupported database (i.e. anything other than postgres), you now get a clear error message. * Minimum required Python version is now 3.6; all previous versions are now end-of-life and are no longer receiving security updates. ## 5.8.0 (2021-03-09) ### Added * If SQL Server has full-text search support installed, then the database client will now create a FULLTEXT index on the `sectionlocation.text` and `sentencelocation.text` columns. Note that you will need to run the `migrate` command to create these indexes. This matches the existing support for similar indexes in MySQL and Postgres. If full-text support is not installed, the `migrate` and `create-db` commands will indicate that no full-text indexes were created. ## 5.7.0 (2021-02-25) ### Added * Support for the `snomedicd10-ontology` feature has now been added. * The new `known_ambiguity` attribute of found entities is now stored in the database as the `foundentity.known_ambiguity` column, and is available in the `FoundEntity.known_ambiguity` attribute in the SDK. You will need to run the database client's `migrate` command when updating. ### Fixed * The issue with the extra COMMIT being sent to SQL Server when outside of a transaction has now been fixed. ## 5.6.0 (2021-02-04) ### Added * PDFs stored in JSON files are now supported; this requires the PDF to be Base64-encoded and stored under a `pdf` key in the JSON; the `text` option must be omitted in that case, or else the JSON will be considered to contain plaintext data instead. ### Fixed * The database client was storing document text when using the `--fast-postgres` option even when it shouldn't have been. Ensure you're using `--store-reports` if you'd like to store report text. If the input document is a PDF, the document text will be stored regardless, to ensure the spans make sense. ## 5.5.0 (2020-12-05) ### Added * Both the simple client and the database client now support the `--quiet` flag when processing reports, which will hide the progress bars. * Both clients now support the `timeout` parameter on `/status` calls to the engine, potentially speeding up processing of single documents. ### Fixed * The clients were making some unnecessary status calls at the end of processing, and that's now improved. Calling `ResultFuture.result()` and `ResultFuture.raw_result()` may now raise `TaskNotFoundError` when attempting to retrieve a result for an invalid task ID. ## 5.4.3 (2020-11-27) ### Fixed * Auth was signing all headers, and now only signs those required. This will avoid auth failures in some network configurations. ## 5.4.2 (2020-11-05) ### Fixed * There was a typo in an index name which was confusing (`ix_sectionlocationlocation_text`, has too many locations) ## 5.4.1 (2020-11-05) ### Fixed * There was a misalignment of sentence locations and sentence spans when storing sentences in the database when using the `--fast-postgres` option; this has occurred since 5.3.0. This has now been fixed, and checks have been added to ensure this sort of issue will not recur with other similar tables. ## 5.4.0 (2020-11-03) ### Added * Added two new Postgres GIN indexes on the `sentencelocation.text` and `sectionlocation.text` columns. ### Changed * The Postgres B-tree indices on `sentencelocation.text` and `sectionlocation.text` were removed because there is a maximum size for these indices and sometimes the text in those columns exceeds the maximum size supported by the index. ## 5.3.0 (2020-10-19) ### Added * When using the database client to ingest documents from a database, you can now provide the `--text-is-pdf` option to tell it the 'text' column contains raw PDF bytes, rather than plaintext. * documentation has been added for the new PDF ingest from database column feature * the database client now has a `process --store-sections-and-sentences` option which will store section and sentence text in `sectionlocation.text` and `sentencelocation.text` columns, respectively. In the case that sentences or sections are discontinuous, their different parts will simply joined by using single spaces. * There are now indices on the `sectionlocation.text` and `sentencelocation.text` columns. On MySQL these will be FULLTEXT indices, and on Postgres, they're B-tree indices using the `text_pattern_ops` operator class. ### Fixed * The database storing progress bar was usually not very smooth when using the fast-postgres option. This was because the batches were too big and usually most docs fit into one batch. Now there's a batch of size 50 when storing results which leads to a smoother progress bar. ## 5.2.1 (2020-07-27) ### Fixed * There was a corner case where the database client detected that some documents failed to be processed, but didn't log which ones. ## 5.2.0 (2020-07-03) ### Added * There's now support for arbitrary key-value metadata which can be returned by file handler plugins. This data is stored in the `documentstructuredmetadata` table where the keys and values are both unicode texts; you may want to cast the values to more useful types for queries. ## 5.1.0 (2020-06-09) ### Added * DB client support for plugins that can add handlers for new file types. * More general DB client handling of file paths, especially to handle more than one JSON or JSONL file at once and to apply the recursive option to JSON, JSONL, or plugin-added file types. ## 5.0.0 (2020-05-01) ### Added * Added support for `umlsloinc-ontology` as an available feature. * The necessity and modifier arguments for medications have been added, and are stored in the `medicationrelationnecessity` and `medicationrelationmodifier` tables, respectively. ### Changed * `ResultFuture.done()` now returns a `ResultFutureStatus` object (which is truthy), and this object must be used now for checking progress and general status; `ResultFuture.progress()` has now been removed (use `ResultFutureStatus.progress` for this) ## 4.9.0 (2020-04-24) ### Added * The database client can now store the JSON it received from the server for each document into a new `document.json_representation` column. This will require a migration using the `migrate` subcommand. Storing JSON can be enabled using the `process --store-json` command-line option. ## 4.8.0 (2020-04-04) ### Added * All HTTP client error response codes (4xx) from the NLP API server are now handled using an `HTTPClientError` exception if there isn't a more specific exception for them. The clients will continue to exit with code 4 for these errors (unless there's a more specific exit code for them). * There is now support for `process_with_section_labels`. Both clients now accept a `--section-label` option which will be used for all documents being read in. This is passed verbatim to the server. Support for this option has also been added using the `section_label` parameter of `InputDocument` objects. The section label is now stored in a new `document.section_label` column in the database. **NOTE**: this change requires you to use the `migrate` command to add the column to your exiting database. ## 4.7.0 (2020-03-23) ### Added * Added support for medication relations. It's controlled by the `medication-relations` feature. New tables are `medicationrelation`, `medicationrelationdosage`, `medicationrelationfrequency`, `medicationrelationmode`, `medicationrelationquantity`, and `medicationrelationroute`. It can also be accessed using `AnnotatedDocument.relations['medication']`. ## 4.6.0 (2020-03-18) ### Added * Support for the `entity-measurement-unit` feature. This means there's a new table where it gets stored: `foundentitymeasurementunit`, and the `FoundEntity` object now has a `FoundEntity.measurement_unit` attribute. * There's now an `emtellipro-db-client migrate` sub-command which runs migration scripts against the database to upgrade it to the current schema, copying data as necessary; this currently only supports databases going back to v4.3.0, and going forward every new version of the database client will allow migrating the database. Please ensure you have a backup of the database just in case. ## 4.5.0 (2020-03-13) ### Added * The database client now has a `--version` option ### Changed * The API server option `--server` on the two clients is now a required option with no default; this is to avoid accidentally sending data to the wrong jurisdiction. * Improved progress bars so they are displayed more reliably. ### Fixed * It's now possible to use `--help` on all subcommands without setting a value for `--database` on `emtellipro-db-client`. ## 4.4.0 (2020-02-20) ### Added * The database client now checks if the target database contains all necessary columns before submitting documents, and exits early with an error message if any are missing. There is also an new exit code (5) for this and other database-related errors. ### Changed * There's now a `foundentitytype` table which contains the ontology and type name pairs for each found entity. This replaces the `foundentity.type_name_*` columns, and allows for any future ontologies to be added without changing any table's schema. * The `FoundEntity.type_name` dictionary will now contain whatever was returned by the API, and no type name will ever be None. If it wasn't returned by the API, it won't be present in that dictionary. ## 4.3.0 (2020-02-04) ### Added * Added support to the database client for different exit codes based on the error. * Improved error handling and logging of exceptions, so now almost every exception ends up being logged to error.log, including details about which documents failed. See the `process --help` command option for details. ## 4.2.3 (2020-01-28) ### Fixed * When attempting to store chartdate in an SQLite database, there was an error about the sqlite driver only accepting Python datetime objects for storing in a DateTime column. Now when storing dates in an SQLite database, the dates are parsed using the [dateutil](https://dateutil.readthedocs.io/en/stable/) library. For all other databases the dates are left as strings. * If a submission fails, the database client no longer crashes with an exception. It now tries to split each submission into individual documents to narrow down the failed document, and logs the failed document to `errors.log`. ## 4.2.2 (2020-01-21) ### Fixed * There was an exception being raised when there weren't any documents found. Now the client simply prints a message stating that zero documents were found. ## 4.2.1 (2020-01-16) ### Fixed * The database client now stores the JSON/JSONL filename in the `document.filename` column * There was an exception being raised when there weren't any entities found in a document. This is now fixed. ## 4.2.0 (2020-01-15) ### Added * The database client now accepts JSON input in a single JSON or JSONL file. ## 4.1.0 (2020-01-09) ### Changed * Removed the `--fetchall` option from the database client, as it is no longer necessary for connecting to SQL Server. This also removes the dependency on `records`. ## 4.0.0 (2020-01-05) ### Changed * Backwards incompatible: To ensure consistent naming of relations and locations, all relation and location types are now *singular*. This means that `AnnotatedDocument.locations.keys()` is now `['section', 'sentence']` rather than `['sections', 'sentences']`. And for relations, `AnnotatedDocument.relations['experiencers']` is now `AnnotatedDocument.relations['experiencer']`. All other relations were already singular, so `experiencers` was the odd one out. If using the database client or `emtellipro_db` package, you can now find enums for all the type names in `emtellipro_db.rowtypes`. * The `source_document_id` column in the `documentmetadata` table is now a unicode string of length 255, rather than an integer. ### Fixed * Inconsistent `type_` values in the database when using `--fast-postgress` have now been fixed. They're now all singular and match the names used without `--fast-postgres`. ### Added * There's now an `emtellipro_db.rowtypes` module that contains enums for all the different type names stored in `type_` columns in the tables that have them. ## 3.1.1 (2019-12-25) ### Added * A new option `--fetchall` to the Database client that is useful when processing documents taken from a database and the connection suffers from timeouts. This option will let you fetch all the documents in one query at the expense of a larger memory footprint. This option is currently required for retrieving documents from Microsoft SQL Server databases for processing. ## 3.1.0 (2019-12-18) ### Added * A new database table for storing document metadata, called `documentmetadata`. It will be populated with extra fields provided using `--sql-query` for `emtellipro-db-client`. * The database client now accepts a `--fast-postgres/--no-fast-postgres` option which enables/disables a new batched insertion mode for saving to the database which uses PostgreSQL-specific SQL extensions. This should result in up to a 10x performance improvement. * `emtellipro.data.InputDocument` now takes a `file_obj` parameter, allowing callers to submit PDF files that don't come from the file-system. ## 3.0.1 (2019-11-14) ### Added * Updated documentation for the `emtellipro-db-client` Database Client * Other minor documentation updates/fixes ## 3.0.0 (2019-10-31) ### Changed * Moved `examples/database.py` into its own module, and it's now automatically installed when the complete package is installed. The database client is now callable using `emtellipro-db-client` or `python3 -m emtellipro_db` * Ported advanced client to use SQLAlchemy instead of peewee as the ORM, and also changed the database schema slightly to more easily allow for future additions ## 2.8.0 (2019-10-01) ### Added * Added support for sections to the API. They can be accessed through `AnnotatedDocument.locations['sections']`. ### Fixed * Since the API response may omit the category and subcategory attributes, the SDK now handles that case correctly by allowing those attributes to be None in `AnnotatedDocument`, and NULL in the database columns used by the advanced client. Previously, missing attributes would result in a KeyError. ## 2.7.0 (2019-09-10) ### Added * Added support to the advanced client for ingesting documents for processing from a database ## 2.6.0 (2019-08-26) ### Added * Added support for the `umlshgnc-ontology` feature * Added job IDs and job timestamps to advanced client ## 2.5.1 (2019-07-17) ### Fixed * There was a typo in the mime-type for plaintext documents that caused an error in some situations on CentOS. The mime-type has now been fixed. ## 2.5.0 (2019-07-09) ### Added * Added support for new features `umlsnci-ontology` and `medcin-ontology`. ## 2.4.0 (2019-06-13) ### Added * There is now a new exception `emtellipro.exceptions.TaskFailedError` that is raised when trying to retrieve results of a task that failed. * The command-line clients now print some simple stats about how long it took to process the submitted documents. ## 2.3.0 (2019-04-04) ### Added * Added support for qualifier relations to the SDK and `database.py` example. The relevant new class is `emtellipro.data.QualifierRelation` which is usable through `annotated_document.relations['qualifier']`. ### Fixed * There was a bug which caused an empty list of features to be treated as requesting all features (only None means "all features"). This has been fixed and an empty list of features correctly requests no features. ## 2.2.0 (2019-03-08) ### Added * Added a `debug` command to the example client which prints out internal debugging information from the state file. ### Changed * The example client's `--access-key` and `--secret-key` options now need to be set on the specific commands which use it (such as `submit`) instead of at the top level. This allows `submit --help` to work as expected and allows for commands which don't need those options. ## 2.1.0 (2019-02-21) ### Added * The type names dictionary for found entities now contains a "radlex" key. This has also been added to the relevant database table. ## 2.0.2 (2019-02-19) ### Fixed * A bug was fixed where one couldn't specifically request the followup relations. ## 2.0.1 (2019-02-14) ### Fixed * A bug was fixed where one couldn't specifically request the measurement and imagelink relations. ## 2.0.0 (2019-02-11) ### Added * There is now support for two new relation types: measurements and image-links. They're implemented similarly to the previous relation types. ### Changed * The example client now shards reports, and produces a JSONL formatted output file. * The example client now allows continuing an unfinished submission using the `continue` subcommand. * The `Emtellipro.submit()` method now splits up the submitted documents so they fit in the maximum request size limit; this means that it now returns an iterable of `ResultFuture` instances, instead of just one. ## 1.6.0 (2019-01-16) ### Added * The example `database.py` script now saves the text of found entities in a `foundentitytext` table. * The example client now saves a mapping file along with the output file which contains the mapping from document ID to file path (space-separated). * The example client's `submit` command learned `--doc-id-filepath` which sets the document ID to the filepath (for easier matching of files to annotations). If this flag isn't set, the new mapping file can be used instead. * The example client's `submit` command learned `--recursive` for recursively looking for files in any directories provided as input. ## 1.5.1 (2019-01-10) ### Fixed * The progress bar in the example client was incorrectly computed, and got to 100% too quickly. This has now been fixed so the progress displayed accurately represents the progress provided by the NLP API server. ## 1.5.0 (2018-12-24) ### Changed * The document type can now be specified per `InputDocument` instance by passing the `type_` parameter to `__init__()`. If not specified, the `Emtellipro.submit()` method will set it, so previous ways of setting the document type for all documents will continue to work. * `utils.read_files()` used a default encoding of UTF-8 for all files, but in some cases not all files will have the same encoding. It now autodetects the encoding for each file, unless explicitly told the encoding to use for all files. ## 1.4.1 (2018-11-23) ### Changed * Use Radiology/generic for default category/subcategory in `database.py` and example client. * Set default document type as 'plain' in `database.py` ## 1.4.0 (2018-11-20) ### Changed * Use new multi-file format for submitting documents to NLP API server instead of older JSON-based one. There are no user-facing changes as a result of this. * The computation of document progress was changed to match changes to the API's response for `/status` * Added `sample_Discharge_summary_report.txt` to `example-data` folder. * Added `generic` report subtype as default ## 1.3.0 (2018-11-15) ### Added * `examples/database.py` gained a `--store-reports` flag on its `process` command which will enable storing of the original report text in the `text` column of the `document` table. * A `foundentitylocation` table was added to `examples/database.py` which maps found entities to locations (e.g. sentences) where they were found. These are the same locations as available in `emtellipro.data.AnnotatedDocument.locations` ### Fixed * `emtellipro.data.Span` objects are now hashable, fixing an issue with their use as dictionary keys in `FoundEntity.parts` and `SentenceLocation.parts` ## 1.2.0 (2018-11-9) * Initial post-Beta release