Release NotesDetailed release notes

Release notes for NLP API 2.28

Contents

Summary

New Features

Question Status attribute

The new question_status attribute equals is_question in cases where the concept has been raised as a question to resolve. For example, rule out appendicitis where appendicitis will be is_question. In other cases, the attribute will have the value not_question.

The question_status attribute is only applied to documents with the Radiology category and only in the Indication section.

This new feature is available in the result output by asking for the entity-question-status processing feature when submitting to the NLP API.

For the following input processed as a Radiology/generic by the NLP API:

INDICATION:
r/o appendicitis.

in output json result format v2.19.0 or later the NLP API adds a new attribute question_status to the list of attributes. See the output for the entity appendicitis below. The focus in the example below is on one entity and the new question_status attribute. The example output below is truncated to the relevant parts (the ... indicate the ellipsis of unnecessary details).

{
  "documents": [
    {
    ...
    "entities": {
      "found": [
      ... 
      {
        "label": "E3",
        "spans": [
          {
            "start": 16,
            "end": 28
          }
        ],
        "section_name": "INDICATION",
        "attributes": {
          "question_status": "is_question",
          ...
        },
        ...
        "text": [
          "appendicitis"
        ]
      },
      ...
      ]
     }
     ...
    ]
   }
  "format": "emtellipro-json-2",
  "version": "2.19.0"
}

Temporality relation (beta)

Beta version of a new temporality relation indicating whether the subject entity is in scope of a temporal_entity. Currently the only supported category of temporality is historical.

This relation indicates that the subject of this relation (e.g. an entity like heart disease) is within the scope of a historical temporal_entity (e.g. like history of) in a sentence history of heart disease.

In some cases, the historical temporal_entity is not a FoundEntity but rather an AssumedEntity by virtue of occurring within a section that is categorized as historical e.g. Family History:.

There is also a modifier argument which is currently always empty in this beta version of the relation. It will contain useful modifiers in future releases of the NLP API.

The polarity attribute for this relation indicates whether the temporality status itself is negated or asserted.

This new feature is available in the result output by asking for the temporality-relations processing feature when submitting to the NLP API.

For the following input processed as a Clinical/generic by the NLP API:

history of heart disease.

in output json result format v2.19.0 or later the NLP API adds a new relation temporalities to the list of relations. See the output for the above sentence below. The focus in the example below is on one relation, namely temporalities. The example output below is truncated to the relevant parts (the ... indicate the ellipsis of unnecessary details).

{
  "documents": [
    {
      ...
      "relations": {
        ...
        "temporalities": [
          {
            "label": "RT0",
            "attributes": {
              "polarity": "asserted",
              "category": "historical"
            },
            "args": {
              "subject": {
                "ref": "E1",
                "text": [
                  "heart disease"
                ]
              },
              "modifiers": [],
              "temporal_entity": {
                "ref": "E0",
                "text": [
                  "history of"
                ]
              }
            }
          }
        ]
      },
      ...
  "format": "emtellipro-json-2",
  "version": "2.19.0"
}

Ontology updates

  • RadLex ontology updated to 4.1.
  • RxNorm update RxNorm Full Monthly Release January 3, 2022. Added Indicator, Reagent, or Diagnostic Aid as a valid type of drug for medication relations.
  • More flexible lemmatization options for each supported ontology.
  • Updated MEDCIN ontology database to new version with section-based disambiguation.
  • Added section based disambiguation for MEDCIN concepts.
  • Produce longer matches of mentions to MEDCIN concepts than previously.

Changes

  • Disable remote as qualifier relation trigger since it will be a trigger for historical temporality relations.
  • Changed all measurement entities in SNOMED ontology to concept 246445000 Measure (attribute).

Bug Fixes

  • Bug fix for some non-determinism issues leading to some missing output.
  • Bug fix to avoid crashes when there are no sections or no sentences (both of which occur when the input is empty or all whitespace).
  • Bug fix for sentence detection model which was creating spurious boundaries in very long documents.
  • Bug fix for rare cases where some plurals were being assigned the concept ID for the singular form.

Accuracy

Relations

  • Enable medication relations for Calcium.
  • Improved accuracy of medication relations.
  • Improvements to assumed family experiencers in experiencer relations.
  • Improved qualifier relations, including new triggers for duration/time qualifier, normal as value for status qualifiers.
  • Fix false positive qualifier relation with old.
  • Detect a larger set of SNOMED person concepts and add them as valid experiencer subjects.
  • Added new date, time and temporal expressions like 54 year old, 1AM, 5:33PM, 21 days, 4 to 6 hours, etc.
  • Fix false positive for 2C/3C/4C (number of chambers) in ejection fraction measurement relations when they are in parentheses. Also match C measurement units for temperature and temp measurement subjects.
  • Eliminate relation triggers that appear inside an entity span, e.g. injection of technetium 99m tilmanocept was previously creating a measurement relation between injection and 99m even though there is an entity technetium 99m tilmanocept found in this input.
  • Improved measurement relations, including identification of pairs of measurement values for FEV1 and FVC, added FEV1/FVC ratios, potassium K, mg/kg/min as measurement unit, packs per day, g / dl and percent as measurement units, 80's and 80?s as measurement values.
  • Added LVEF unitless measurements, e.g. LVEF for 2C/4C was 75.
  • Fix false positive trigger 2c/4c for ejection fraction measurements.
  • Do not identify dates as measurement values.
  • Add ambiguous term in as measurement unit only for height in measurement relations.
  • Add at stress as a post-modifier of a measurement trigger for cases like 42% at rest and 48% at stress.

Section parsing

  • Change back section label from immunizations to immunization.
  • Improvements to section parsing with new headings added for all document types.
  • New additions and updates for CCD headings including: fixed level of CCD Immunizations heading, ignore CCD headings Indication and Author, Problem List mapped to Problems
  • Improvements to cardiology/echocardiogram headings.
  • Added new Radiology level 1 section headings.
  • New CRITICAL RESULTS COMMUNICATION heading.
  • New wet read and pfi radiology headings.
  • Add new REFERENCE EXAMINATION and REFERENCE STUDY variant headings.
  • New Social heading for clinical.

Polarity

  • Polarity (negation) improvements, including negatives include, no evidence by, no ... to indicate ..., free of.
  • Ensure never smoked is an asserted entity.

Misc

  • Annotation updates for SNOMED, MEDCIN, RxNorm including new biomarker concepts.
  • Fix after fall being picked up as temporal entity.
  • Allow British spellings when normalizing medical terms.
  • Improved uncertainty accuracy, including with possible.
  • Improve date finding to avoid false positives with dosages like 10-20-300.
  • Improved speed and accuracy of sentence boundary detection, including the handling of ICD code, f.u as abbreviation, decimals and hyphens in CCDs, fix for smoking-cessation false positive, long sequences of hyphens and underscores from flat ascii tables, handle “fill in the blanks” cases with underscores.

Infrastructure

  • We improved the stability and throughput of our API server to handle the case where a single user wants to run hundreds of simultaneous clients connecting to the NLP API service.
  • Improvements in the NLP API Admin UI including the ability to download and sort the logs.
  • Documentation web site is completely auto-built with versions for each sub-component like the NLP API description, the JSON result description, etc.
  • IAM command-line client has added as a new download in our Downloads page.
  • IAM support for custom JWT endpoints like the Visual Client.
  • For on-premises installations of the NLP API the audit logs have been available on the NLP API admin UI since v2.27. The audit logs can be now downloaded using the shell script provided below:
#!/bin/sh
# Make sure you have httpie and jq installed on your system:
# sudo apt install httpie jq
#USER_NAME="name"  # set as environment variable
#PASSWORD="pass"   # set as environment variable
IAM_URL="https://<your-emtellipro-url>:1222" # Warning: this has changed in recent releases
EMTELLIPRO_ADMIN_URL="https://<your-emtellipro-url>:5051"
echo “Getting auth token...”
auth_token=$(http POST $IAM_URL/iam/v1/login user_name=$USER_NAME password=$PASSWORD | jq --raw-output .auth_token)
echo “Downloading audit log...”
http --timeout 120 GET $EMTELLIPRO_ADMIN_URL/admin/download-audit-log COOKIE:auth_token=$auth_token --download
  • Extensive customized memory and time profiling of the NLP API was conducted.
  • Memory and time usage and time complexity improvements optimized to handle larger documents. Fix for potential memory leak in some rare cases.
  • Enabled support for longer documents without timing out.

Python SDK updates

The new NLP API Python SDK is version 5.22.0. Refer to the changelog for the Python SDK for a full explanation of the updates made since the last public release which was version 5.17.1 released with the NLP API v2.27. You will need to use at least version 5.7.0 of the Python SDK or later to send requests to the NLP API v2.28.

To use existing NLP API databases with the new Python SDK you must do a data migration using the Python database client migrate subcommand:

emtellipro-db-client --database postgresql://username:password@hostname:port/example_db migrate

Question Status in Output Database

When using the emtellipro-db-client Python database client the table for foundentity contains the question_status attribute value.

The updated foundentity table looks like this with an additional column for question_status:

 id | polarity | section_name | uncertainty | known_ambiguity | question_status | document_id
----+----------+--------------+-------------+-----------------+-----------------+-------------
  1 | asserted | INTRO        | certain     | unknown         | not_question    |           1
  2 | asserted | INTRO        | certain     | unknown         | not_question    |           1
  3 | asserted | INTRO        | certain     | unambiguous     | not_question    |           1
  4 | asserted | INTRO        | certain     | unambiguous     | not_question    |           1
  5 | asserted | INTRO        | certain     | unknown         | not_question    |           1
 ...

Temporality relation in Output Database

The new temporality relation appears in the temporalityrelation table and associated temporalityrelationmodifier table when using the emtellipro-db-client Python database client.

The temporalityrelation table looks like this:

 id | polarity |  category  | temporal_entity_id | subject_id
----+----------+------------+--------------------+------------
  1 | asserted | historical |                  2 |          1
  2 | asserted | historical |                 18 |         17
  3 | asserted | historical |                135 |        134
  4 | asserted | historical |                138 |        139
  5 | asserted | historical |                154 |        155
 ...

The temporalityrelationmodifier table looks like this:

 id | temporality_id | entity_id
----+----------------+-----------
...

Java SDK updates

The latest NLP API Java SDK is version 5.8.0. Refer to the changelog for the Java SDK for a full explanation of the updates made since the last public release. You will need to use version 5.8.0 or later to take advantage of the new features described above.