Changelog

All notable changes to the NLP API Python SDK and associated files will be documented in this file. This SDK follows Semantic Versioning, although version identifiers adhere to standard Python packaging guidelines specified in PEP 440.

7.2.0 (2026-08-18)

Added

• The database client’s process and debug commands now report throughput numbers alongside the average time per document numbers. The ‘time per document’ values tell you on average how long the NLP API took to process each document, while the throughput numbers take into account the concurrency in the client (using the process --jobs option).

• There’s a new emtellipro-db-client list-options command that lists the available save options for the requested schema. See “list-options” command for usage.

• The User now has a permissions attribute which includes the user permissions returned by the NLP API.

  This information is also included in the output of the emtellipro-db-client get-user command.

Changed

• The default server set as DEFAULT_SERVER is now set to https://api.us.emtelligent.com.
• When using the database client, if both API key and HMAC access keys are set, then the client will show an error rather than prioritizing the API key. Only setting one or the other is valid.

Fixed

• When processing CCD documents containing structured metadata, the database client would return an error. It now processes them correctly and stores the JSON representation of the metadata in the documentstructuredmetadata table as expected.

Docs

• The documentation for process --save-opt now includes a links to the migration guide for the options that were present in version 6 and earlier.

7.1.0 (2026-08-12)

Added

• The database client now includes run time statistics at the end of processing. This will include the cumulative run time (across multiple restarts with the same state file), as well as the current execution time (producing a similar time to what the time command would produce).

  Note that these times do not include Python startup times, nor module import times, so the “execution time” reported will be slightly lower than what the time command would report.

Changed

• The error message presented by the database client when retrying has been exhausted will now include more detail about the last exception that cased the error.

Docs

• The behaviour of the legacy schema when storing relation concept links, Snomed ICD-10 CM concepts, and document text is now documented in a single place: How the legacy schema works

7.0.1 (2026-07-31)

Fixed

• Ensure only the ‘legacy’ database schema is listed in list of built-in schemas.

7.0.0 (2026-07-30)

For migration information see: Migrating to v7.

Added

• Support for multiple schemas has been added, with plugin support. See Pluggable database schemas.

• There’s now a debug --gantt option that produces a Gantt chart of the different processing stages.

• There’s now an process --retry-failed flag which will retry any documents that were returned with processing status ‘error’ by the NLP API.

• The “legacy” database schema now accepts a snowflake_cache_dir save option which caches database writes to that directory before copying to Snowflake using COPY INTO. This can improve saving performance on larger batches where executing INSERT statements would be slower.

  See snowflake_cache_dir

• The document-infer-structure feature is now supported.

• When submitting tasks to the NLP API and checking the status, one of the possible task states is “task setup”. When the task is detected to still be in the “task setup” state, the time between when the submission started and the returned status being “task setup” is now recorded and is presented at the end of execution.

  However, if the first returned task state is “processing”, then task setup time is assumed to be 0, as it’s not possible to monitor true task setup time from the client side.

  Note that if the --poll-freq option is set to 1 or higher, it’s unlikely to detect the task setup time since it’s usually quite short.

• If the emtellipro-db-client state file is not compatible with the current version of the SDK, an error message will be presented.

Changed

• The JSON input format for the SDK has now changed to be more structured and thoroughly documented.

  The old format is still supported, but is deprecated. See documentation for input formats: Input File Formats Supported By The SDK.

• The database client’s save options are now set using the process --save-opt option, which can be passed multiple times to set different save options. This change was required due to the schema being dynamically set.

• Dependency on requests has now been replaced with httpx2 and tenacity.

• The database client is now multi-threaded and has some changed options. See Multithreaded database client for details.

• When creating found entities based on relation concept links, the new found entities did not contain any type name information; now the type names will be parsed out of the corresponding concepts’ descriptions.

• type_name may now contain None values (they were previously filtered out).

• It’s now possible to pass a floating point number to process --poll-freq; the integer portion will be used for the NLP API (as it only supports integers), and any left-over is waited on client-side.

  This effectively means that one can poll more frequently than once a second by passing something like --poll-freq 0.5, however that puts more load on the NLP API so it’s not recommended.

• The end-to-end notebook example has been updated to import the models from emtellipro.db.schemas.legacy.

Fixed

• The database client previously would say it found 0 files when the input paths specified as arguments did not exist, but now that’s been fixed to provide an error message specifying the path doesn’t exist.
• The insert_or_ignore parameter for duckdb_caching_connection was incorrectly assumed to only include ‘concept’, but now any table name is supported.
• Specifying processing features as a list in the config file caused an error; now it’s fully supported.

6.17.0 (2026-06-11)

Database migration is required for this version.

Changed

• The data type for the sectionlocation.name column no longer has a 255 char limit, and matches the type of foundentity.section_name.

6.16.0 (2026-05-14)

Added

• Automatic detection of categories and subcategories is now possible using the --category auto option, or setting the category/subcategory to 'auto' or None when instantiating InputDocument, or as an argument to any function that accepts such a category/subcategory parameter.

  For more details see: Inferring the category

• An example was added for processing documents from a Snowflake database with the NLP API and storing it in a Snowflake database, designed for use in a Databricks notebook. See examples/end-to-end-notebook/ for details.

Fixed

• Avoid crash CCD files with empty tables or table rows.

Changed

• The lxml dependency version now requires ~= 6.0 to support Python 3.14.

6.15.0 (2026-04-09)

Database migration is necessary for this version.

Added

• The documentmetadata table now contains a encounterdate column, with the same type as chartdate.

Fixed

• There was a bug when merging entities to create synthetic found entities based on relations that led to an AttributeError.

Deprecated

• Support for HMAC authentication is deprecated and will be removed in the next major release.

6.14.0 (2026-03-06)

Added

• DuckDB is now supported for use with the Database class, and the database client.

• The Database class’s file_mode argument now supports 'w' for JSONL outputs (i.e. URLs beginning with jsonl://, jsonl+table://, table+jsonl://, which all mean the same thing).

• Support for the ‘experiencer’ attribute on found entity has been added.

  For more details: Found Entity Experiencer

• Support for the ‘factuality’ attribute on found entity has been added.

  For more details: Found Entity Factuality

• 3 new processing features are now supported:

  • entity-experiencer
  • entity-factuality
  • entity-contextual-validity

• There are new performance improvements in the database client, so loading files should be significantly faster.

Fixed

• Attempts to create indexes in DuckDB will raise the appropriate NotImplementedError.
• The emtellipro-db-client debug subcommand now has better error handling when there are issues with the state file.
• When connecting to an non-existent database in Snowflake, you may have gotten an exception from the Snowflake driver at an unexpected place; now the SDK will check that the database exists when first connecting to it (this is the behaviour of all other database types, only Snowflake allows you to connect to a non-existent database and will only return an error when trying to query tables within it).
• The database client no longer throws an UnboundLocalError exception when retrying failed; it now presents an error message.

Changed

• If using Snowflake, the dependency on snowflake-connector-python is now >= 3.18 because earlier versions appear not to have correctly specified their dependency on cryptography. This only applicable if you installed the SDK using the snowflake extras.
• Python 3.10 is now the minimum Python version required, as 3.9 is now past its EOL.
• DuckDB uses UUIDs for the ID columns, not sequential integers.

Docs

• The optional parameters for save now specify which database columns they affect.

6.13.0 (2025-10-23)

Added

• There’s now a check_schema method that can check whether the schema in the database is compatible with the current SDK version. “Compatible” just means all the expected columns are present and the schema version in alembic_version.version_num is correct.

  If the schema’s not compatible, an appropriate exception will be raised.

Fixed

• When attempting to run multiple database clients with the same state file, they’d interfere with each other and cause “database is locked” errors. Now the state file is locked for the entire runtime of the client, so running a second client will immediately error out.
• When running many database clients in parallel and storing results in a Postgres database, it was possible to get deadlocks. This has now been fixed by getting an EXCLUSIVE lock on the concept table in each transaction; this will allow SELECT statements to execute while inserts are happening.

Changed

• An extra validation step is added to ensure that empty type names are not inserted into foundentitytype.type_name column.

6.11.0 (2025-10-15)

Added

• The Emtellipro class now accepts extra headers to include in all requests made using that instance.

Changed

• There is now an extra validation step when saving relation concept links to ensure that concept IDs are not ‘null’ or “NULL”; these concept links are saved by creating synthetic found entities (see Relation Concept Links for how those synthetic found entities are created). This is only relevant when saving to a database/file, not when parsing NLP API JSON files in Python.
• As an extra validation step, all concept IDs that are ‘null’ or “NULL” are not stored in the database.

Docs

• Updated snomed concept in End-to-end Report Processing and Data Extraction Example to include entity type.

6.10.0 (2025-07-23)

Database migration is necessary for this version.

Added

• There is now support for authenticating using API keys (as an alternative to HMAC-based authentication). This means that instead of passing in an access key and shared secret, you now pass in a single API key.

  For more details: API Keys

• There is now support for the new concept_links feature for relations. This is available as a new .concepts attribute on all Relation subclasses, which is similar to the same attribute on FoundEntity objects.

  This new data is stored in the database by creating new found entities that merge the found entities in the arguments of the relations. The new found entities are discontinuous since they combine the spans of each individual found entity.

  For more details: Relation Concept Links

• The model classes in models now contain more relationship definitions. Specifically, the following classes have attributes that link them to more concrete objects Entity, Relation, and Location (with attributes .entity, .relation, and .location, respectively).

  For example accessing the .location attribute would get you the appropriate *Location object, thus avoiding having to join the tables yourself.

• The clear method has been added which deletes all the rows for all tables in the database which the SDK is responsible for (determined by what’s in SQLAlchemy’s metadata for the Base model in models). This excludes the alembic_version table.

• There’s a new attribute: state that contains the value of the “state” field in the task status returned by the NLP API.

• Subcategory “generic” is now valid for category “gastroenterology”.

• The foundentity table now contains two new columns:

  text
  The concatenation of the text for each span, separated by a single space character. For most single-span entities this will be the same text as the text in the foundentityspan table, but some entities are discontinuous so this’ll contain all the discontinuous components in one string (this will happen for the found entities created from relations’ concept links).

  start
  The start offset of the found entity; this is effectively the minimum of the associated foundentityspan.start values.

Changed

• If you hit Control-c while the SDK client is processing, you now get prompted on whether you want to cancel the last task that was submitted.

• The default category when using the command-line client has been changed from ‘Radiology’ to ‘Clinical’ (see process --category).

  This only affects command-line usage. There are no changes to the Python API (as the Python API always required the category to be explicitly set by the caller).

• The retrying behaviour of the database client has been reworked so that only specific errors are retried rather than retrying everything by default. Now only errors that are clearly transitory (such as connection errors) are retried. Everything else is assumed to be a permanent error.

• The category and subcategory values are now validated up front rather than submitted as is.

• When processing documents using the database client which come from a SQL database (i.e. when using process --sql-query), the database client started a transaction but didn’t call COMMIT at the end. The database client will now automatically call COMMIT after loading all the documents from the source database.

Fixed

• The database option in the client’s config file was able to be called both output and database in the [emtellipro] section, but only database in other sections.

  It’s now possible to use output and database interchangeably everywhere in the config file.

• When using the database client to store to a raw+json:// or raw+jsonl:// output path, and the entire processing task has failed (on the engine side), you’d get an exception raised because there was no JSON to store for the failed task. Now an empty JSON object will be stored for those cases, ensuring that the number of output files match the number of tasks created (whether the tasks succeeded or failed).

• Fixed crash where the NLP API returned extra attributes on relations and the parsing code didn’t handle them.

• Flattener class now correctly includes spans when asked.

• When installing the [snowflake] optional extras dependencies (when installing the wheel file), the cryptography dependency was not directly specified; it’s now included since the SDK explicitly uses it for Snowflake private key files.

• Validation of category and subcategory was missing when loading documents from JSON files, and so the error was only noticed when submitting to the NLP API. The category/subcategory is now validated when input JSON files are parsed.

Docs

• The license for this SDK is no longer included in the .zip or .tar file. The current license is included separately on the download page.
• Added documentation for how document metadata can be handled when using the SDK in Python. See the metadata argument of InputDocument for more details.
• The help message for process --features now lists all the available features.
• The supported categories and subcategories when submitting to the NLP API are documented as CATEGORIES.
• Documented that the URL format for Database is the same as --output, while also clarifying that the URL format for Snowflake is the one documented in Snowflake’s SQLAlchemy driver.
• Clarified that the maximum submission size is 500MB, not 500MB per input document.
• The link to the CCD metadata schema on the Structured entries as metadata page is now fixed.

6.9.0 (2025-03-12)

Added

• heading has been added, which contains the heading linked to the section (if any). This was previously just found by looking at the heading locations and checking the corresponding section attribute.

• Added indexes on the following documentmetadata columns on all databases except for Snowflake:

  • chartdate
  • document_id
  • original_category
  • original_subcategory
  • source_document_id
  • subject_id
  • subject_id_2
  • subject_name

• Added support for relation confidence scores. These will all be floating point numbers or None.

  Each relation table now contains a confidence column:

  • experiencerrelation.confidence
  • followuprelation.confidence
  • measurementrelation.confidence
  • imagelinkrelation.confidence
  • qualifierrelation.confidence
  • medicationrelation.confidence
  • temporalityrelation.confidence
  • reportedeventrelation.confidence
  • anatomicsiterelation.confidence

  Each relation data class now contains a .confidence attribute:

  • confidence
  • confidence
  • confidence
  • confidence
  • confidence
  • confidence
  • confidence
  • confidence

  See more detail: Relation Confidence.

• Quickstart documentation now uses the database client rather than the deprecated simple client.

• The database client now lists documents that failed processing to stderr at the end of processing.

• The emtellipro-db-client debug command now lists files and document source IDs that failed processing. See “debug” command for more details.

• The Database Client Options document contains more details about the arguments each database client command accepts.

• The documentation for the command line options for the database client has been improved so each option can be linked to, and shared options are duplicated for each command that shares it.

• The Snowflake documentation page has been updated with new tables. Using the Database Client with Snowflake

Changed

• When attempting to use the database client’s migrate command on an empty database, the user is presented with a more clear error message about it.

Fixed

• Client: restarting processing when documents fully completed processing would raise an error. Now it will exit without error.
• The documentation for objects in engine did not include the return types for all the methods; they now include better descriptions of what they return.
• When using the JSON file input format for the database client with empty strings for subject_dob or chartdate, there was previously an error when storing the metadata to the database because empty strings are not valid for datetime columns; the empty strings are now cast to NULL.
• Added error handling to database client for a known bug where the NLP API may not return all failed documents in the result output. These “missing” documents are treated as failed for the purposes of --store-failed.
• There was a bug when using the database client to process a PDF file, which would lead to an error message about invalid path type.
• When storing files containing source data (e.g. PDFs) to the database using the Python API you’d get an error about invalid data type. The source file path is now converted to a string before storing to the database.
• The keyword arguments documented for save should be optional, but the code required that some were passed in. They’re all optional now, as documented.
• The --store-failed flag was not storing failed documents correctly; now it does.
• When an entire task failed processing, the resulting exception caused the database client to retry the request, even though the failure was permanent. Failed tasks are now logged and if --store-failed is used the documents within the task will all be treated as failed.
• The database client failed to print to stdout the number of documents that failed processing. Now the failed document count is printed at the end of processing.
• The relationship definition between the SectionLocation and PageLocation SQLAlchemy models is now fixed so the SQLAlchemy warning is gone.

6.8.0 (2025-01-22)

Added

• The Concept class now contains map_rule, map_advice, map_group, map_priority, and map_category attributes which may be present for the snomed_icd10_cm ontology.

Changed

• The log level for the case where a submitted document was not returned by the NLP API has been increased from WARNING to ERROR.
• The minimum required Python version is now 3.9 since Python 3.8 has passed its EOL date.
• Concepts in the snomed_icd10_cm ontology may be skipped or may have their concept IDs modified based on the new map_rule and map_advice attributes when being stored to a database. If the rule contains a conditional, the new concept ID will have format: concept_id | map_rule | map_group | map_priority.

Fixed

• If attempting to use --filetype json flag with a non JSON file (including JSONL), a better error message will be presented than a JSONDecodeError.

  Note that JSONL and JSON filetypes are not compatible, so for JSONL files --filetype jsonl would be appropriate.

  The --filetype option is optional, though, and by default filetype detection is done by looking at the file extension.

6.7.0 (2024-12-16)

Database migration is necessary for this version.

Added

• There are 3 new columns in the documentmetadata table: clinical_domain, source_document_id_2, and source_document_id_3.

Changed

• Log CCD “unable to resolve metadata item” message as warning instead of error.

Fixed

• The file_mode parameter introduced in 6.6.0 had a bug when file_mode='a' which didn’t write the headers to the files.
• The --no-store-pages and --no-store-headings flags were incorrectly worded. This is now fixed.
• If the NLP API returned unsupported fields in its output, that would lead to an exception in the SDK. This is now fixed to ignore unsupported fields.

6.6.0 (2024-11-15)

Database migration is necessary for this version.

Added

• Added file_mode parameter to Database. This is only supported for CSV database URIs.

• Added support for the concept_confidences attribute in found entities. This is enabled using the entity-concept-link-confidence feature flag.

  The confidences can be found in concept_confidences, as well as the foundentityconcept.confidence database table column.

  See more detail: Concept confidence.

• Added support for heading locations. This is found in the 'heading' key of locations.

  The data for this location will be stored in the new headinglocation, headinglocationspan tables.

  Storing text of the page is now controlled using the --store-headings CLI flag.

  See more detail: Heading locations.

• Added support for page locations. This is found in the 'page' key of locations. There is also the new attributes pages.

  The data for this location will be stored in the new pagelocation, pagelocationspan, and pagesection tables.

  Storing text of the page is controlled using the --store-pages CLI option.

  See more detail: Page locations.

• Added support for the ‘heading_status’ attribute of found entities. This is available as heading_status, and is found in the database in the new foundentity.heading_status column.

  See more detail: Entity heading status.

• Added support for the new ‘heading’ location type.

6.5.2 (2024-10-11)

Fixed

• JSON input file validation is now improved to check that all metadata values are strings or ‘null’; conversion to strings is not done automatically because it may be ambiguous, so it’s left to the user to convert any non-string object to a string.
• When the database client’s --sql-query parameter has a syntax error or the input database URL is invalid, better error messages are now presented.

6.5.1 (2024-10-09)

Added

• The database client’s debug command now accepts a --json flag which tells it to produce its output as a JSON object. The exact structure is not documented however since it’s intended just for debugging and may change from version to version.

Fixed

• There was an intermittent error when input documents are in Snowflake where the connection to Snowflake would be closed by the Python interpreter before SQLAlchemy cleanly reset it right at the end of the database client’s execution; this had no impact on processing or storage.

6.5.0 (2024-09-16)

Database migration is necessary for this version.

Added

• When using JSON/JSONL input documents with the database client, it’s now possible to specify both the plaintext contents of the document and the PDF document source from which it came. See Full format definition for more details (specifically the pdf and pdf_path keys).

  This functionality can be used when you’ve extracted the text from the PDF and wish to use that text for processing, but still keep a track of the original PDF it came from.

  There are 3 new database columns to support this in the document table: source_path (if the pdf_path JSON key is set), source_type (only application/pdf is supported now), and source_size (the number of bytes in the source PDF).

  There is also a new database table: documentsourcechunk. This is used for storing the source data, split into chunks that fit into the maximum allowed for the documentsourcechunk.data column (BYTEA for PostgreSQL, LONGBLOB in MySQL/MariaDB, and VARBINARY(max) in SQL Server).

  There is also a new process --store-pdf option to enable storage of the source data into documentsourcechunk. This is only supported for PostgreSQL, MySQL, MariaDB, and SQL Server. No storage will happen for any other output formats.

Fixed

• When processing large numbers of CCD documents using the database client, the state file would grow much larger than the input documents due to large amounts of metadata read from the CCD; the metadata is now compressed so the state file will be smaller than the input documents.

• Storing data containing datetimes in an SQLite database would fail if the datetimes were strings, because the SQLite driver requires datetime objects. Now the strings are parsed before insertion.

  Parsing the datetime strings only happens in SQLite. For all other database types, the strings in the input metadata are passed along unchanged to the database.

6.4.0 (2024-08-28)

Added

• Expanded documentation and added more examples.

6.3.2 (2024-08-26)

Fixed

• Since 6.0.0 the database client did not insert anything into the documentmetadata table if there was no data to insert. Now it will insert a row with just the document_id, which was the behaviour in 5.x.

6.3.1 (2024-08-21)

Fixed

• When the metadata for InputDocument contained a datetime object, this was not serialized correctly and would raise an exception. Serialized datetimes is now fixed.
• If there was an error while ingesting files, and the database client was restarted, it would exit early saying processing was already completed (with 0 documents). Now it correctly checks whether the file ingesting successfully completed on restart.

6.3.0 (2024-07-30)

Added

• The client now supports raw+jsonl://PATH.JSONL, where PATH is the path to a JSONL file. This will append the results to that file, along with PATH.map file that contains a mapping between document IDs and input file paths.

  This format matches the output format of the simple client that was available in v5.

• The simple client from version 5 has been brought back, but it remains deprecated; it’s available as emtellipro-client as before.

• The emtellipro-db-client store-result command now accepts directories containing JSON files as arguments (previously it only accepted the JSON files themselves).

• The -o, --output, --database option can now be passed directly to process, create-db, migrate, and store-results, rather than as a global option before. The new position is encouraged and passing it globally is deprecated.

  The same applies to the --snowflake-private-key-path option.

  This change clarifies which commands require a database/output option, since commands like get-user don’t use a database.

  Example:

# before
emtellipro-db-client -o postgresql://... create-db
# now
emtellipro-db-client create-db -o postgresql://...

• The command-line help messages for the database client have now been expanded and generally improved.
• The emtellipro-db-client debug command now provides progress information, and nicer formatting of the parameters.

Changed

• The state file used for emtellipro-db-client process is now an optional parameter. If it’s not passed, a state file will be generated and its location will be printed out and logged.

  If a state file is provided, its progress statistics are now printed out.

• Dropped support for SQLAlchemy 1.4 as indicated in the 6.0.0 release notes, given that snowflake-sqlalchemy >= 1.6 now supports SQLAlchemy 2.0.

pip install emtellipro-6.3.0-py3-none-any.whl[snowflake]

• The database client will now insert up to 16,384 rows at once into Snowflake database.

Fixed

• When running the database client in a location where $PWD/errors.log was not writable, and no alternative log file was provided using --log-file, the client would crash. Now it provides an error message stating the log file path is not writable.

6.2.0 (2024-07-04)

Added

• The flatten JSON script from examples/ has now been added to the core SDK. It’s available as emtellipro-flatten-json after installing the SDK.

  Documentation for usage is found in Flatten JSON.

• There’s now a new Python module that provides similar functionality to the emtellipro-flatten-json CLI tool; example usage is found in flatten.

• The emtellipro-db-client command now takes -o, --output as an alias for --database.

6.1.0 (2024-06-13)

Changed

• Updates to CCD LOINC to section name mapping.
• Default LOINC code for CCD sections, to use in the case where a section has one or more LOINC codes from XML elements but none are in our mapping.
• Improvements to documentation for CCD section normalization, especially for LOINC code handling.

6.0.0 (2024-05-24)

Added

• The documentmetadata table now contains 3 new columns subject_id_2, subject_address, and document_summary.

• AnnotatedDocument now takes a text parameter which contains text of the document; this is then used for filling in the .text attributes of contained objects when the text was not contained in the result JSON object.

• CCD files can now be loaded from a database by using --sql-query QUERY --text-type ccd.

• CCD handling now normalizes section titles based on LOINC codes if present, if there is no OID to normalize with.

• Added example of result JSON parsing to the Python notebook in examples.

• Improved example of result JSON parsing in the Python notebook in examples.

• FoundEntity now has start and stop attributes, which provide easy access to the overarching character span.

• Span now has a slice attribute which returns an equivalent slice object.

• Support for anatomic site relations has been added. This includes

  • anatomicsite-relations feature option
  • ‘anatomic-site’ key in relations, which contains AnatomicSiteRelation objects
  • anatomicsiterelation table

  The new relation contains site and situated_entity attributes.

• Added a new processingdetails.sql_query database column. When the database client reads documents from a database, the SQL query used will be stored in this column.

• The database client now accepts both --store-sentences and --store-sections flags that can be used instead of --store-sections-and-sentences.

• When reading documents from a database, the client will now store any extra columns in the SQL query in the documentstructuredmetadata table. The extra columns are any columns other than ‘id’, ‘text’, or ones that match the documentmetadata table columns.

Changed

• Updated to lxml 5 for CCD parsing.
• Support both SQLAlchemy 1.4 and 2.0; support for 1.4 will be dropped in the next minor release when the Snowflake SQLAlchemy driver is updated.
• The SDK has been rearchitected to make it easier to use by people interacting with it in Python to write their own code for loading files, submitting them for processing and saving them to a database.

Major changes

• The Emtellipro class now takes the access key and shared secret as a tuple to be passed as an argument to ``auth`. This allows for different authentication methods in the future.
• Support for plugins has been removed.
• There is now a Database class (also available as emtellipro.db.Database). This provides a method for end users to interact with the database, creating tables, running migrations, and store processing results.
• There is now a readfile function that provides a simple method for reading a file; it loads a single path that can potentially contain multiple documents.
• The InputDocument class now takes a metadata argument containing arbitrary key/value pairs that can be later stored in the database.
• The InputDocument class now takes a single data argument which must be either bytes or a string, replacing the multiple arguments for plaintext vs PDF documents and the file path. Loading files will now happen outside the document.
• locations is now a dictionary, to match locations. To easily access the first sentence in the sentences list, you can also use sentence.
• The emtellipro_db package is replaced by the db module.
• There is now a single SDK client, which is the old database client. It provides all the functionality that was previously available in the simple client.
• The SDK client now requires a --state, -s option that stores its state in a local file, allowing for re-starting processing.
• The client’s --max-retries and new --retry-forever options will retry processing and storing when errors occur; the --retry-forever may lead to an infinite loop, however, so it’s important to check the log file to see why the errors are occurring.
• The client has a store-result subcommand which takes paths to JSON files produced by using the raw:// file output option for the process command and stores it to a database.
• The client now supports reading the NLP API server, access key, and shared secret options from the EMTELLIPRO_SERVER, EMTELLIPRO_ACCESS_KEY, and EMTELLIPRO_SHARED_SECRET environment variables, respectively. These will override the matching options from the config file, but if you explicitly pass those options to the client (e.g. using --server option), the explicitly passed options will take precedence over the environment variables and the config file.
• There’s a new API for reading CCD files in ccd.
• Using SQL server as the target database will now require using Microsoft’s official drivers.
• The database client’s --bulk-insert and --fast-postgres options have now been removed. The bulk insert mode is the only method for inserting rows into target databases.

Fixed

• The package will now work if it’s somehow imported without being installed (for example if it’s converted to a zip file and imported directly from the zip file).

5.30.1 (2024-03-22)

Changed

• Change Sphinx documentation theme to furo.

5.30.0 (2024-03-04)

Changed

• Have CCD metadata pick up <effectiveTime> as a code element if there is no other code element at the same level.
• Re-trying requests will now re-compute the authentication header before sending the new request.

5.29.0 (2024-01-18)

Added

• Log messages mentioning errors in JSON files will now also include the document ID in the JSON file alongside the filepath.

5.28.0 (2023-10-23)

Added

• JSON input files can now contain a "filepath" item which points to a file containing the actual data for the document. This can be either a .txt or .pdf file. The path should not be in the same directory as the JSON file, or else there’s a risk the document will be processed twice.

Changed

• Updated JSON example in documentation to show all the possible metadata options.
• New CCD config option report-subsection-sections in config section [file_handlers.ccd], to control what sections the report subsection feature is applied in. See new CCD config docs for details.

5.27.0 (2023-08-22)

Added

• When the database client detects that it’s running outside a virtual environment it will print a warning to let the user know.
• APIError exceptions (and subclasses) now contain a task_id attribute when available.
• The JSON files used as input for the database client can now contain structured metadata. This metadata must be stored as a mapping in a "structured_metadata" key and will be saved to the documentstructuredmetadata table; if this key is missing, then no structured metadata will be stored, as before.
• The database client will now include version numbers when listing installed plugins.
• The CCD plugin has now been merged into the SDK and the database client now natively supports CCD files. The separate plugin is now ignored if it’s installed.

Deprecated

• The database client’s --bulk-insert and --fast-postgres options are deprecated; the bulk-insert mode will be the only insert method moving forward, since it supports all database types and is faster.

5.26.0 (2023-03-17)

Database migration is required for this version.

Added

• The radplaybook-ontology the NLP API feature is now supported.
• There’s now support for the date_time argument of medication relations. This is stored in emtellipro.data.MedicationRelation.date_times (as a list), and in the medicationrelationdatetime table.
• The database client is now able to save the unprocessed results from the NLP API to disk instead of in a database. You can use this by setting --database raw://SOME_DIRECTORY_PATH for the process command; the create-db command is not necessary because the process command will automatically create the target directory if missing.
• Support has been added for the guidance attribute of found entities. This is now found in the SDK as FoundEntity.guidance (where it is stored as a string), and it’s stored in the database in the new foundentity.guidance column. The new feature is called entity-guidance. This change will require a database migration.
• Support is now available for the ontology_versions returned by the NLP API for each processed document. These are available as AnnotatedDocument.ontology_versions and in the ontologyversions table. The database migration will insert rows in the ontologyversions table based on the documentconcepts table, but with release_version = NULL since previous ontology version data is not available.

5.25.0 (2022-10-17)

Added

• You can now use -q as an alias for --quiet for the database client.
• Error log messages are now printed to STDERR when emtellipro-db-client process --quiet is used; to hide the error messages you can pass --quiet (or -q) twice. In the config file, this can now be controlled by setting quiet = 2 (using the old true and false values still works, but integer values are also supported).
• In the emtellipro.data module, there are now FoundEntity.attributes, Relation.attributes, and Relation.arguments instance attributes which tell you the names of the instance attributes which are relation attributes and relation arguments, respectively; each Relation subclass have these attributes. This provides some support for introspection to allow writing generic relation-handling code.

Changed

• The JSON file loader in the database client now validates input JSON documents and provides descriptive error messages.
• Document that Python 3.7 is now the minimum supported version, since Python 3.6 is no longer officially supported.
• The emtellipro_flatten_json example uses the new data model metadata to look up attributes and arguments. New found entity attributes, new relation types, and new relation attributes and arguments will be handled automatically as long as their values fall into the general formats already supported. There are some changes to table names and column order. Relation tables are now produced only if non-empty.
• When cancelling tasks, the HTTP PATCH method is now used instead of GET; both methods are still supported by the API, but GET is deprecated.

Fixed

• Filenames with extra dots in them would previously not have the file extension parsed correctly, and so would not be read

5.24.0 (2022-07-04)

Added

• When running emtellipro-db-client process --help, the defaults for all options are now printed.
• The database client now takes a -c / --config option which allows you to pass in a configuration file containing options. All command-line options are supported (simply remove the -- from the beginning of an option to get the config file version).
• The emtellipro-db-client process command now accepts a --filetype option which tells it to treat all input files as if they had the given file type (thus disabling detecting file type based on file extension). The associated config file option is called filetype. See process --help for available file types (since it depends on installed plugins).

Changed

• API requests will now automatically time out after 90 seconds.

5.23.0 (2022-05-17)

Database migration is required for this version.

Added

• The SDK’s version is stored in the new processingdetails.sdk_version column.
• The emtellipro-db-client process --doc-id-filepath option now supports JSON and JSONL files.
• Support for the “reported event” relation in examples/emtellipro_flatten_json/.
• Support for the snomedicd10cm-ontology feature has now been added.
• There is now support for the “reported event” relation. This is accessible using emtellipro.data.ReportedEventRelation, and in the database in the reportedeventrelation table, with arguments in the reportedeventrelationtoentity, reportedeventrelationfromentity, reportedeventrelationmodifier, reportedeventrelationtimeexpression tables. The feature name is reportedevent-relations.
• Example examples/emtellipro_flatten_json/ which reads NLP API JSON result files and uses the SDK’s data model to convert a flatter format.
• The database client now stores the engine version returned by the NLP API in the new processingdetails.engine_version column; if not returned, this will be NULL.
• The documentmetadata table now has 2 new columns: subject_gender, and requestor, which can be populated by providing the same field names in the input documents’ metadata.

Fixed

• Some database errors were leading to the database client to exit with code 0 instead of 5 (even though the error was printed and logged).

5.22.0 (2022-03-03)

Added

• The database client now accepts a process --doc-id-filepath flag which tells it to use the filepath of the input document as the document ID when submitting to the server; this is currently only supported for .txt documents.
• The database client now allows configuration of the logging path and level through the --log-file PATH and --log-level LEVEL options, which go in the same position as the --database option (i.e. in between emtellipro-db-client and the subcommand); see emtellipro-db-client --help for details.
• The database client now takes a process --polling-frequency SECONDS option which adjusts how often it polls the NLP API for status updates on the job processing status; by default it waits for 1 second between status checks, but for jobs with very large documents it might make sense to increase that value.

Fixed

• Previously it was not possible to run multiple database clients in parallel using the --no-bulk-insert option due to uniqueness constraints on the concept table; this was not an issue with --bulk-insert, and now --no-bulk-insert has been fixed to allow multiple clients to insert into the concept table simultaneously.

5.21.0 (2022-01-27)

Database migration is required for this version.

Added

• in addition to CSV and JSONL output from the database client, it now also supports JSON output. You can use this by setting --database json://SOME_DIRECTORY_PATH for the process and create-db commands.
• the simple client now takes a --max-retries option in the same position as the --server option, and the database client now takes the same option for the process command. This option specifies the number of times to retry failed API requests, and is enabled default, with a default value of 5, using exponential backoff between retries. There is a maximum wait time between retries of 120 seconds, so setting --max-retries to large values will eventually result in multiple retries at 120 second intervals. The intention behind this option is that API requests may fail due to network issues, so this will allow continued processing in the case of transient network errors.
• the database client now accepts a process --store-failed option which will store details about documents that failed processing (i.e. ones where processing_status returned by the NLP API is "error").
• the database now contains a document.processing_status column which contains the processing status returned by the NLP API for the document. This will usually be "success" unless --store-failed is passed to the database client’s process command, in which case it may be "error".
• the database client has improved error handling and reporting about errors when submitting reports for processing

Fixed

• the database client would raise an error when saving temporality relations using --no-bulk-insert due to an incorrect relationship definition. This affects version 5.20.0 and later.

5.20.0 (2022-01-17)

Database migration is required for this version.

Added

• Temporality relations are now supported. They’re found in emtellipro.data.TemporalityRelation, and in the database in the temporalityrelation table, with the modifiers in the temporalityrelationmodifier table.

5.19.0 (2021-12-08)

Database migration is required for this version.

Added

• The documentmetadata table now has 3 new columns: author_name, subject_name, and subject_dob.

Fixed

• The migrate command for the database client would raise an exception when trying to migrate CSV or JSON files (which is not supported). It now prints an simple error message instead.
• There was performance regression introduced in 5.17.0 when saving to SQL Server, which caused significantly reduced storage speed. This was improved in 5.18.0 and has now been completely fixed.

5.18.0 (2021-11-26)

Added

• Support has been added for the question_status attribute of found entities. This is now found in the SDK as FoundEntity.question_status (where it is stored as a string), and it’s stored in the database in the new foundentity.question_status column. The new feature is called entity-question-status. This change will require a database migration.

Fixed

• there are now some performance improvements when saving to SQL Server, especially when using Microsoft’s ODBC driver instead of FreeTDS.
• creating an InputDocument with an empty string as the text raised a ValueError, but now empty strings are allowed.

5.17.1 (2021-10-13)

Fixed

• There was a bug introduced in 5.17.0 when exporting to CSV which caused a crash; this has now been fixed.
• The --features option was not parsed properly in 5.17.0; this has now been fixed.

5.17.0 (2021-10-08)

Added

• The database client’s process command now accepts a --max-save-shard-size which specifies the number of reports to store to the database at once; this is ideally used for lowering the default value from 50, since large numbers can cause database errors.
• You can now press CTRL-C when the database client is running and it will finish saving to the database what it has so far, and will exit cleanly. Pressing CTRL-C a second time will cause it to exit immediately.

Changed

• The database client is now multi-threaded, so it can read files, submit them for processing, and store results to the database in parallel.
• The database client’s --max-submit-shard-size default is now 100, to avoid having the database thread waiting too long for results.

5.16.0 (2021-09-23)

Added

• Support for key-pair authentication for Snowflake has been added. All commands now take --snowflake-private-key-path as an option alongside --database which is the path to the private key used for the connecting to Snowflake. If using this option, omit the password in the connection URL. To be consistent with SnowSQL, the passphrase for this key file can be passed using SNOWFLAKE_PRIVATE_KEY_PASSPHRASE or SNOWSQL_PRIVATE_KEY_PASSPHRASE in the environment, or if those environment variables are not set, the client will prompt for the passphrase if necessary. Both encrypted and unencrypted key files are supported.

5.15.0 (2021-09-20)

Added

• There is now a plugin architecture based on signals.
• Improved error handling when running create-db on an existing database, and checking whether the database schema version stored in the alembic_version table has been accidentally deleted.
• If the alembic_version table has been accidentally cleared, the create-db command will now allow the user to re-insert the database schema version in that table.
• The simple client now has a cancel command that allows cancelling one or more tasks. See emtellipro-client cancel --help for available options.
• The simple client will now print the last unfinished task ID when it receives a SIGINT (e.g. from CTRL-C) while a job is in progress

Fixed

• There was a memory leak in the SDK which has now been fixed, so running the simple client or database client will now use a stable amount of memory.

5.14.1 (2021-08-09)

Fixed

• The last version of the SDK introduced a bug where JSON documents weren’t counted properly for the progress bar, and JSON document IDs were not properly attached to filenames, leading to an error message. This is now fixed.

5.14.0 (2021-07-29)

Added

• The database client now supports saving output to CSV or JSONL files. Simply use --database csv://PATH_TO_DIRECTORY or --database jsonl://PATH_TO_DIRECTORY for the create-db and process commands, instead of the regular database URLs. The directory will be populated with files named after each table in the regular database schema, and all ID columns will use UUIDs in standard format (36 characters, with the hyphens).

Changed

• The database schema when saving to Snowflake now uses UUIDs defined as CHAR(36) for all the ID columns (and associated foreign keys). These are generated locally by the database client and will lead to significantly faster insertions for Snowflake users. This will require running the migrate command to update existing Snowflake databases. Other databases will continue using auto-incrementing integer IDs.
• When using --max-submit-shard-size with the database client, it will now only load the specified number of input documents at once (previously all input documents were loaded unconditionally, and this option only limited how many were submitted in each shard). This option can now be used to reduce memory usage when there are many input documents.

5.13.0 (2021-07-15)

Added

• The database client’s process command now accepts a --skip-database-checks option which skips checks for database consistency (i.e. checking all the tables exist and have the correct columns).

Fixed

• When reading input documents from JSON files and storing results using --bulk-insert the database client failed to store document metadata into the documentmetadata table if the set of metadata keys weren’t the same across all JSON files. This has now been fixed and JSON files can have differing metadata keys.

5.12.0 (2021-06-24)

Added

• The database client will now use INSERT IGNORE in MySQL and INSERT OR IGNORE in SQLite to speed up inserts into the concepts table when using the --bulk-insert option
• The database client now accepts a --max-submit-shard-size option which allows restricting the number of documents submitted at once to the API. If unset, the client will continue determining shard size based on document size, so this is mostly useful for restricting shard size further than normal.

Fixed

• On certain reports, when using the --bulk-insert option the database client would attempt to insert duplicate concepts in the concepts table which would fail due to the uniqueness constraints on the table. Inserted concepts are now de-duplicated before insertion to fix this issue.

5.11.0 (2021-06-18)

Added

• There is now support for the “duration” and “indication” arguments for medication relations. Like all the other medication relation arguments, these will also be stored as lists. They can be found in the MedicationRelation.duration and MedicationRelation.indication attributes and medicationrelationduration and medicationrelationindication tables, respectively; the new tables will require a database migration.

5.10.1 (2021-06-08)

Changed

• Stopped Snowflake’s SQLAlchemy driver from logging INSERT queries to errors.log
• Improved performance of Snowflake insertions

Fixed

• Specified SQLAlchemy version must be at least 1.4

5.10.0 (2021-06-08)

Added

• Snowflake databases are now supported for storing data using the database client
• Bulk inserts (the --fast-postgres) option is now available for all databases, and is called --bulk-insert); this will lead to some speed improvements especially in the cases of databases which support returning primary keys on inserted rows without an extra query (such as PostgreSQL and SQL Server). Postgres still has the best support in this case since it allows inserting missing entries into the concepts table in a single query.
• Regular storing of documents one at a time (without --bulk-insert) also has received some speed improvements by grouping inserts more efficiently.

5.9.0 (2021-05-13)

Added

• The foundentity and assumedentity tables now contain a document_id column with a foreign key referencing document.id; this should make it simpler to query found entities and assumed entities in a document without having to join to the entity table (although the entity table is unchanged so older queries will continue working). This will require a database schema migration using the migrate command.
• Added AnnotatedDocument.processing_status attribute which stores the response of the engine for the “processing_status” JSON attribute. This is used for checking for failed reports in the database client (in the case that the entire task isn’t marked as failed).

Changed

• Modified the type of the document.json_representation and document.text columns in MySQL from TEXT to LONGTEXT, since depending on the document size, TEXT might not be able to hold all of the data.
• When you enable --fast-postgres with an unsupported database (i.e. anything other than postgres), you now get a clear error message.
• Minimum required Python version is now 3.6; all previous versions are now end-of-life and are no longer receiving security updates.

5.8.0 (2021-03-09)

Added

• If SQL Server has full-text search support installed, then the database client will now create a FULLTEXT index on the sectionlocation.text and sentencelocation.text columns. Note that you will need to run the migrate command to create these indexes.

  This matches the existing support for similar indexes in MySQL and Postgres. If full-text support is not installed, the migrate and create-db commands will indicate that no full-text indexes were created.

5.7.0 (2021-02-25)

Added

• Support for the snomedicd10-ontology feature has now been added.
• The new known_ambiguity attribute of found entities is now stored in the database as the foundentity.known_ambiguity column, and is available in the FoundEntity.known_ambiguity attribute in the SDK. You will need to run the database client’s migrate command when updating.

Fixed

• The issue with the extra COMMIT being sent to SQL Server when outside of a transaction has now been fixed.

5.6.0 (2021-02-04)

Added

• PDFs stored in JSON files are now supported; this requires the PDF to be Base64-encoded and stored under a pdf key in the JSON; the text option must be omitted in that case, or else the JSON will be considered to contain plaintext data instead.

Fixed

• The database client was storing document text when using the --fast-postgres option even when it shouldn’t have been. Ensure you’re using --store-reports if you’d like to store report text. If the input document is a PDF, the document text will be stored regardless, to ensure the spans make sense.

5.5.0 (2020-12-05)

Added

• Both the simple client and the database client now support the --quiet flag when processing reports, which will hide the progress bars.
• Both clients now support the timeout parameter on /status calls to the engine, potentially speeding up processing of single documents.

Fixed

• The clients were making some unnecessary status calls at the end of processing, and that’s now improved. Calling ResultFuture.result() and ResultFuture.raw_result() may now raise TaskNotFoundError when attempting to retrieve a result for an invalid task ID.

5.4.3 (2020-11-27)

Fixed

• Auth was signing all headers, and now only signs those required. This will avoid auth failures in some network configurations.

5.4.2 (2020-11-05)

Fixed

• There was a typo in an index name which was confusing (ix_sectionlocationlocation_text, has too many locations)

5.4.1 (2020-11-05)

Fixed

• There was a misalignment of sentence locations and sentence spans when storing sentences in the database when using the --fast-postgres option; this has occurred since 5.3.0. This has now been fixed, and checks have been added to ensure this sort of issue will not recur with other similar tables.

5.4.0 (2020-11-03)

Added

• Added two new Postgres GIN indexes on the sentencelocation.text and sectionlocation.text columns.

Changed

• The Postgres B-tree indices on sentencelocation.text and sectionlocation.text were removed because there is a maximum size for these indices and sometimes the text in those columns exceeds the maximum size supported by the index.

5.3.0 (2020-10-19)

Added

• When using the database client to ingest documents from a database, you can now provide the --text-is-pdf option to tell it the ‘text’ column contains raw PDF bytes, rather than plaintext.
• documentation has been added for the new PDF ingest from database column feature
• the database client now has a process --store-sections-and-sentences option which will store section and sentence text in sectionlocation.text and sentencelocation.text columns, respectively. In the case that sentences or sections are discontinuous, their different parts will simply joined by using single spaces.
• There are now indices on the sectionlocation.text and sentencelocation.text columns. On MySQL these will be FULLTEXT indices, and on Postgres, they’re B-tree indices using the text_pattern_ops operator class.

Fixed

• The database storing progress bar was usually not very smooth when using the fast-postgres option. This was because the batches were too big and usually most docs fit into one batch. Now there’s a batch of size 50 when storing results which leads to a smoother progress bar.

5.2.1 (2020-07-27)

Fixed

• There was a corner case where the database client detected that some documents failed to be processed, but didn’t log which ones.

5.2.0 (2020-07-03)

Added

• There’s now support for arbitrary key-value metadata which can be returned by file handler plugins. This data is stored in the documentstructuredmetadata table where the keys and values are both unicode texts; you may want to cast the values to more useful types for queries.

5.1.0 (2020-06-09)

Added

• DB client support for plugins that can add handlers for new file types.
• More general DB client handling of file paths, especially to handle more than one JSON or JSONL file at once and to apply the recursive option to JSON, JSONL, or plugin-added file types.

5.0.0 (2020-05-01)

Added

• Added support for umlsloinc-ontology as an available feature.
• The necessity and modifier arguments for medications have been added, and are stored in the medicationrelationnecessity and medicationrelationmodifier tables, respectively.

Changed

• ResultFuture.done() now returns a ResultFutureStatus object (which is truthy), and this object must be used now for checking progress and general status; ResultFuture.progress() has now been removed (use ResultFutureStatus.progress for this)

4.9.0 (2020-04-24)

Added

• The database client can now store the JSON it received from the server for each document into a new document.json_representation column. This will require a migration using the migrate subcommand. Storing JSON can be enabled using the process --store-json command-line option.

4.8.0 (2020-04-04)

Added

• All HTTP client error response codes (4xx) from the NLP API server are now handled using an HTTPClientError exception if there isn’t a more specific exception for them. The clients will continue to exit with code 4 for these errors (unless there’s a more specific exit code for them).
• There is now support for process_with_section_labels. Both clients now accept a --section-label option which will be used for all documents being read in. This is passed verbatim to the server. Support for this option has also been added using the section_label parameter of InputDocument objects. The section label is now stored in a new document.section_label column in the database. NOTE: this change requires you to use the migrate command to add the column to your exiting database.

4.7.0 (2020-03-23)

Added

• Added support for medication relations. It’s controlled by the medication-relations feature. New tables are medicationrelation, medicationrelationdosage, medicationrelationfrequency, medicationrelationmode, medicationrelationquantity, and medicationrelationroute. It can also be accessed using AnnotatedDocument.relations['medication'].

4.6.0 (2020-03-18)

Added

• Support for the entity-measurement-unit feature. This means there’s a new table where it gets stored: foundentitymeasurementunit, and the FoundEntity object now has a FoundEntity.measurement_unit attribute.
• There’s now an emtellipro-db-client migrate sub-command which runs migration scripts against the database to upgrade it to the current schema, copying data as necessary; this currently only supports databases going back to v4.3.0, and going forward every new version of the database client will allow migrating the database. Please ensure you have a backup of the database just in case.

4.5.0 (2020-03-13)

Added

• The database client now has a --version option

Changed

• The API server option --server on the two clients is now a required option with no default; this is to avoid accidentally sending data to the wrong jurisdiction.
• Improved progress bars so they are displayed more reliably.

Fixed

• It’s now possible to use --help on all subcommands without setting a value for --database on emtellipro-db-client.

4.4.0 (2020-02-20)

Added

• The database client now checks if the target database contains all necessary columns before submitting documents, and exits early with an error message if any are missing. There is also an new exit code (5) for this and other database-related errors.

Changed

• There’s now a foundentitytype table which contains the ontology and type name pairs for each found entity. This replaces the foundentity.type_name_* columns, and allows for any future ontologies to be added without changing any table’s schema.
• The FoundEntity.type_name dictionary will now contain whatever was returned by the API, and no type name will ever be None. If it wasn’t returned by the API, it won’t be present in that dictionary.

4.3.0 (2020-02-04)

Added

• Added support to the database client for different exit codes based on the error.
• Improved error handling and logging of exceptions, so now almost every exception ends up being logged to error.log, including details about which documents failed. See the process --help command option for details.

4.2.3 (2020-01-28)

Fixed

• When attempting to store chartdate in an SQLite database, there was an error about the sqlite driver only accepting Python datetime objects for storing in a DateTime column. Now when storing dates in an SQLite database, the dates are parsed using the dateutil library. For all other databases the dates are left as strings.
• If a submission fails, the database client no longer crashes with an exception. It now tries to split each submission into individual documents to narrow down the failed document, and logs the failed document to errors.log.

4.2.2 (2020-01-21)

Fixed

• There was an exception being raised when there weren’t any documents found. Now the client simply prints a message stating that zero documents were found.

4.2.1 (2020-01-16)

Fixed

• The database client now stores the JSON/JSONL filename in the document.filename column
• There was an exception being raised when there weren’t any entities found in a document. This is now fixed.

4.2.0 (2020-01-15)

Added

• The database client now accepts JSON input in a single JSON or JSONL file.

4.1.0 (2020-01-09)

Changed

• Removed the --fetchall option from the database client, as it is no longer necessary for connecting to SQL Server. This also removes the dependency on records.

4.0.0 (2020-01-05)

Changed

• Backwards incompatible: To ensure consistent naming of relations and locations, all relation and location types are now singular. This means that AnnotatedDocument.locations.keys() is now ['section', 'sentence'] rather than ['sections', 'sentences']. And for relations, AnnotatedDocument.relations['experiencers'] is now AnnotatedDocument.relations['experiencer']. All other relations were already singular, so experiencers was the odd one out.

  If using the database client or emtellipro_db package, you can now find enums for all the type names in emtellipro_db.rowtypes.

• The source_document_id column in the documentmetadata table is now a unicode string of length 255, rather than an integer.

Fixed

• Inconsistent type_ values in the database when using --fast-postgress have now been fixed. They’re now all singular and match the names used without --fast-postgres.

Added

• There’s now an emtellipro_db.rowtypes module that contains enums for all the different type names stored in type_ columns in the tables that have them.

3.1.1 (2019-12-25)

Added

• A new option --fetchall to the Database client that is useful when processing documents taken from a database and the connection suffers from timeouts. This option will let you fetch all the documents in one query at the expense of a larger memory footprint. This option is currently required for retrieving documents from Microsoft SQL Server databases for processing.

3.1.0 (2019-12-18)

Added

• A new database table for storing document metadata, called documentmetadata. It will be populated with extra fields provided using --sql-query for emtellipro-db-client.
• The database client now accepts a --fast-postgres/--no-fast-postgres option which enables/disables a new batched insertion mode for saving to the database which uses PostgreSQL-specific SQL extensions. This should result in up to a 10x performance improvement.
• emtellipro.data.InputDocument now takes a file_obj parameter, allowing callers to submit PDF files that don’t come from the file-system.

3.0.1 (2019-11-14)

Added

• Updated documentation for the emtellipro-db-client Database Client
• Other minor documentation updates/fixes

3.0.0 (2019-10-31)

Changed

• Moved examples/database.py into its own module, and it’s now automatically installed when the complete package is installed. The database client is now callable using emtellipro-db-client or python3 -m emtellipro_db
• Ported advanced client to use SQLAlchemy instead of peewee as the ORM, and also changed the database schema slightly to more easily allow for future additions

2.8.0 (2019-10-01)

Added

• Added support for sections to the API. They can be accessed through AnnotatedDocument.locations['sections'].

Fixed

• Since the API response may omit the category and subcategory attributes, the SDK now handles that case correctly by allowing those attributes to be None in AnnotatedDocument, and NULL in the database columns used by the advanced client. Previously, missing attributes would result in a KeyError.

2.7.0 (2019-09-10)

Added

• Added support to the advanced client for ingesting documents for processing from a database

2.6.0 (2019-08-26)

Added

• Added support for the umlshgnc-ontology feature
• Added job IDs and job timestamps to advanced client

2.5.1 (2019-07-17)

Fixed

• There was a typo in the mime-type for plaintext documents that caused an error in some situations on CentOS. The mime-type has now been fixed.

2.5.0 (2019-07-09)

Added

• Added support for new features umlsnci-ontology and medcin-ontology.

2.4.0 (2019-06-13)

Added

• There is now a new exception emtellipro.exceptions.TaskFailedError that is raised when trying to retrieve results of a task that failed.
• The command-line clients now print some simple stats about how long it took to process the submitted documents.

2.3.0 (2019-04-04)

Added

• Added support for qualifier relations to the SDK and database.py example. The relevant new class is emtellipro.data.QualifierRelation which is usable through annotated_document.relations['qualifier'].

Fixed

• There was a bug which caused an empty list of features to be treated as requesting all features (only None means “all features”). This has been fixed and an empty list of features correctly requests no features.

2.2.0 (2019-03-08)

Added

• Added a debug command to the example client which prints out internal debugging information from the state file.

Changed

• The example client’s --access-key and --secret-key options now need to be set on the specific commands which use it (such as submit) instead of at the top level. This allows submit --help to work as expected and allows for commands which don’t need those options.

2.1.0 (2019-02-21)

Added

• The type names dictionary for found entities now contains a “radlex” key. This has also been added to the relevant database table.

2.0.2 (2019-02-19)

Fixed

• A bug was fixed where one couldn’t specifically request the followup relations.

2.0.1 (2019-02-14)

Fixed

• A bug was fixed where one couldn’t specifically request the measurement and imagelink relations.

2.0.0 (2019-02-11)

Added

• There is now support for two new relation types: measurements and image-links. They’re implemented similarly to the previous relation types.

Changed

• The example client now shards reports, and produces a JSONL formatted output file.
• The example client now allows continuing an unfinished submission using the continue subcommand.
• The Emtellipro.submit() method now splits up the submitted documents so they fit in the maximum request size limit; this means that it now returns an iterable of ResultFuture instances, instead of just one.

1.6.0 (2019-01-16)

Added

• The example database.py script now saves the text of found entities in a foundentitytext table.
• The example client now saves a mapping file along with the output file which contains the mapping from document ID to file path (space-separated).
• The example client’s submit command learned --doc-id-filepath which sets the document ID to the filepath (for easier matching of files to annotations). If this flag isn’t set, the new mapping file can be used instead.
• The example client’s submit command learned --recursive for recursively looking for files in any directories provided as input.

1.5.1 (2019-01-10)

Fixed

• The progress bar in the example client was incorrectly computed, and got to 100% too quickly. This has now been fixed so the progress displayed accurately represents the progress provided by the NLP API server.

1.5.0 (2018-12-24)

Changed

• The document type can now be specified per InputDocument instance by passing the type_ parameter to __init__(). If not specified, the Emtellipro.submit() method will set it, so previous ways of setting the document type for all documents will continue to work.
• utils.read_files() used a default encoding of UTF-8 for all files, but in some cases not all files will have the same encoding. It now autodetects the encoding for each file, unless explicitly told the encoding to use for all files.

1.4.1 (2018-11-23)

Changed

• Use Radiology/generic for default category/subcategory in database.py and example client.
• Set default document type as ‘plain’ in database.py

1.4.0 (2018-11-20)

Changed

• Use new multi-file format for submitting documents to NLP API server instead of older JSON-based one. There are no user-facing changes as a result of this.
• The computation of document progress was changed to match changes to the API’s response for /status
• Added sample_Discharge_summary_report.txt to example-data folder.
• Added generic report subtype as default

1.3.0 (2018-11-15)

Added

• examples/database.py gained a --store-reports flag on its process command which will enable storing of the original report text in the text column of the document table.
• A foundentitylocation table was added to examples/database.py which maps found entities to locations (e.g. sentences) where they were found. These are the same locations as available in emtellipro.data.AnnotatedDocument.locations

Fixed

• emtellipro.data.Span objects are now hashable, fixing an issue with their use as dictionary keys in FoundEntity.parts and SentenceLocation.parts

1.2.0 (2018-11-9)

• Initial post-Beta release