> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.emtelligent.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.emtelligent.com/_mcp/server. # Release notes for NLP API 2.29 ## Contents * [Brief Summary](#brief-summary) * [New Features](#new-features) * [Changes](#changes) * [Python SDK updates](#python-sdk-updates) * [Java SDK updates](#java-sdk-updates) ## Brief Summary ### Release versions * NLP API version 2 Release 29. * Python SDK and client v5.24.0. * Java SDK and client v5.9.0. * Python SDK CCD reader plugin v1.9.0. * [NLP API specification v2.16.0](/nlp/api): added support for new processing features `snomedicd10cm-ontology` ontology and `reportedevent-relations`. * JSON result output schema v2.21.0 for [emtellipro-json-2](/nlp/reference/result-format): added new output for `snomed_icd10_cm` concepts and `reportedevents` relations and the NLP API `engine-version` used to process the submitted documents. * [visualclient](https://visualclient.emtelligent.com) version `vc-v2022.07.04-ef5da396` added support for new processing features `snomed_icd10_cm` ontology and `Reported Event` relations. ### Highlights * Added ICD-10-CM ontology. * Added new Reported Event relation with category Critical Results Communication for Radiology reports (beta). * Improved accuracy and coverage of measurement relations. * Improved coverage for historical events in temporality relations. * SNOMED CT database updated to version `2022-04-11` US Edition. * UMLS updated to version `2021AB`. ## New Features ### ICD-10-CM Ontology Support The ICD-10 Clinical Modification, ICD-10-CM, is a modification of the ICD-10, and is primarily used as a source for diagnostic codes in the United States of America. In this release, the NLP API now supports ICD-10-CM as a new ontology, `snomedicd10cm-ontology`, which will output ICD-10-CM codes derived from SNOMED-based named entity recognition. Please note that in many cases, this will output different ICD-10 codes from the NLP API's currently supported `snomedicd10-ontology` feature. In the output json result format `v2.21.0` or later the NLP API adds a new ontology `snomed_icd10_cm`. For the following input processed as a Clinical/generic by the NLP API, and using the `snomedicd10cm-ontology` feature: ```text This has the appearance of breast cancer. ``` A link to the concept will appear for each found entity that has an ICD-10-CM code. The link will appear in the list of concept_links for each found entity (in the list found in the `entities` object). The example output below is truncated to the relevant parts (the `...` indicate the removal of unnecessary details). Note that NLP API's `snomedicd10-ontology` for `breast cancer` is C50.9 (Malignant neoplasm of breast of unspecified site) while for the `snomedicd10cm-ontology`, two codes are produced, C50.919 and C50.929 ("Malignant neoplasm of unspecified site of unspecified female breast" and "Malignant neoplasm of unspecified site of unspecified male breast"). The two codes come with a `description` that contains information to help decide which code to choose based on the gender of the patient. ```javascript { "documents": [ { ... "entities": { "found": [ { ... "concept_links": [ ... "Csnomed_icd100", "Csnomed_icd10_cm0", "Csnomed_icd10_cm1", ], "entity_type": { ... "snomed_icd10_cm": "disorder", "snomed_icd10": "disorder", ... }, "text": [ "breast cancer" ] } ], } ... "concepts": { ... "snomed_icd10": [ { "label": "Csnomed_icd100", "concept_id": "C50.9", "description": "Malignant neoplasm of breast of unspecified site (Neoplastic Process)" }, ... ], "snomed_icd10_cm": [ { "label": "Csnomed_icd10_cm0", "concept_id": "C50.919", "description": "Malignant neoplasm of unspecified site of unspecified female breast (Neoplastic Process)" }, { "label": "Csnomed_icd10_cm1", "concept_id": "C50.929", "description": "Malignant neoplasm of unspecified site of unspecified male breast (Neoplastic Process)" } ] } ... } ], "format": "emtellipro-json-2", "version": "2.21.0", } ``` ### Reported Event relation (beta) Beta version of a new `reportedevent` relation indicating whether the `subject` entity is the topic of communication from one group of entities to another. Currently the only supported category of `reportedevent` is `critical_results`. This relation indicates that the `subject` (e.g. the "findings" of the report) has been communicated by a group of entities (could be a singleton) in the list argument `from_entities` to another group of entities (also could be a singleton) in the list argument `to_entities`. The mode of communication is provided in the argument `methods` while the `time_expression` argument captures when the communication took place. Currently the only category produced by the NLP API is the category of critical_results which indicates that the communication was sent to a healthcare profession and that the sentence occurs in the following sections: `FINDINGS`, `IMPRESSION`, `CALLED REPORT`, `CRITICAL RESULTS COMMUNICATION`, `COMMENTS` and only from reports submitted with an NLP API category of `Radiology` with any subcategory. This new feature is available in the result output by asking for the `reportedevent-relations` processing feature when submitting to the NLP API. For the following input processed as a `Radiology/generic` by the NLP API: ```text CRITICAL RESULTS COMMUNICATION: The above findings and recommendations were discussed in person with Dr. Smith on 2149-4-2 . ``` The NLP API adds a new relation `reportedevents` to the list of `relations` in output JSON result format `v2.21.0` or later. The output JSON for the above sentence is shown below. The focus in the example below is on one relation, namely `reportedevents`. The example output below is truncated to the relevant parts (the `...` indicate the removal of unnecessary details). ```javascript { "documents": [ { ... "relations": { ... "reportedevents": [ { "label": "RR0", "attributes": { "category": "critical_results", "polarity": "asserted" }, "args": { "subject": { "ref": "E4", "text": [ "findings" ] }, "to_entities": [ { "ref": "E5", "text": [ "Dr. Smith" ] } ], "from_entities": [], "methods": [ { "ref": "E6", "text": [ "in person" ] } ], "time_expressions": [ { "ref": "E7", "text": [ "2149-4-2" ] } ] } }, { "label": "RR1", "attributes": { "category": "critical_results", "polarity": "asserted" }, "args": { "subject": { "ref": "E8", "text": [ "recommendations" ] }, "to_entities": [ { "ref": "E5", "text": [ "Dr. Smith" ] } ], "from_entities": [], "methods": [ { "ref": "E6", "text": [ "in person" ] } ], "time_expressions": [ { "ref": "E7", "text": [ "2149-4-2" ] } ] } } ] }, ... "format": "emtellipro-json-2", "version": "2.21.0" } ``` ## Changes ### Ontology updates * UMLS updated to version `2021AB`. * SNOMED CT database updated to version `2022-04-11` US Edition. * ICD-10-CM ontology added. * Removed mapping advice for ICD-10 ontology and replaced it with UMLS Preferred Term and sementic group, e.g. `Malignant neoplasm of breast of unspecified site (Neoplastic Process)`. ### Accuracy Improvements #### Entities * Numerous additions and updates to entity linking for multiple ontologies. * Addition of less common units of measurement for matching of entire measurement value. * Updates to latest version of SNOMED and UMLS. * Identification of new measurements with the measurement units `mL/hr`, `pack year`, `fL` (femtoliter), `centimeters per second`, `x10^3/uL`, `10E6/uL` and variants, `L/min/m2`, `mg/24 hr`, `mg/dL`, `cc/hr` and `{M/uL}`. * Include `<`, `<=`, `>=` and `>` in measurement values. * Add support for ranges of measurement values, e.g. `120-159`. * Fixed false positive of `HR` (heart rate) as part of measurement value. It is now correctly identified as measurement subject. * Fixed false positive with `Mg` (magnesium) incorrectly identified as milligrams. * Fix for time expressions that sometimes blocked reported event relations. * Give priority to entities with a definitionStatusId of `900000000000073002 (Sufficiently defined by necessary conditions definition status)`. #### Relations * Remove `man`, `woman` and similar ambiguous terms as experiencers in experiencer relations. * Recognize proper names that start with `Dr.`, `Mr.`, `Ms.`, `Mx.` and `Mrs.` and other honorifics for experiencer and reported event relations.. * New temporal relation model with improved consistency and accuracy. * New measurement relation model with significantly increased accuracy. * Add `,` as a possible scope breaker in measurement relations. * Reported event relation is enabled in Radiology reports (only) in the sections: `FINDINGS`, `IMPRESSION`, `CALLED REPORT`, `CRITICAL RESULTS COMMUNICATION`, `COMMENTS`. * Fix overlapping span bug between arguments in measurement relations. * Fix false positive for measurement values where you have two separated numbers followed by a unit, e.g. `32 96%`. * Qualifier relations updated to include `stage 0` etc. as qualifiers. * Enable matching of qualifier values in qualifier relations to `assessment scale` and `observable entity` entities. #### Polarity * Fix negation detection failure with `there is no X to suggest Y` pattern. * Fix negation issue with negated temporality triggers. * Fixed scope of `question of` to be rightwards in negation detection. * Fix negation error in `no acute distress`. * Assert some entities which properly contain negation terms. #### Misc * Fix lemmatization bug that was causing incorrect concept ID assignment for mentions like `fainting`. * Support for `CALLED REPORT` radiology heading. * Several accuracy improvements for the uncertainty attribute on found entities. * More robust handling of dirty input data with OCR output that appears as Turkish text. * Improve uncertainty accuracy for `is concerning for` triggers. ### NLP API Improvements * Added `engine-version` to the JSON result output and the engine version is also stored in the `processingdetails` table in the NLP API output database. * For on-premises deployments, the admin UI has a new look with improved search and paging support for log views. * Added support for more workers in the API server which has improved the throughput of the NLP API engine. ## Python SDK updates The latest NLP API Python SDK is version `5.24.0`. The last NLP API release `v2.28` was released with Python SDK version `5.22.0`. Refer to the [changelog for the Python SDK](/nlp/python/changelog) for a full explanation of the updates made for `5.23.0` and `5.24.0`. You will need to use at least version `5.7.0` of the Python SDK or later to send requests to the NLP API v2.29 but if you want to use the new features explained above you will need to use at least version `5.23.0`. We recommend using the latest version of the Python SDK `5.24.0` with the NLP API `v2.29`. > **Warning** > > To use existing NLP API databases with the new Python SDK you must do a data migration using the Python database client `migrate` subcommand: ```bash emtellipro-db-client --database postgresql://username:password@hostname:port/example_db migrate ``` ### ICD-10-CM Support in Output Database When using the `emtellipro-db-client` Python database client the `foundentity` table contains the found entities and the `id` column can be used to find the concept mapping in the `foundentityconcept` table: ```bash # select * from foundentity where id=443 ; id | polarity | section_name | uncertainty | known_ambiguity | question_status | document_id -----+----------+--------------------------------+-------------+-----------------+-----------------+------------- 443 | asserted | CRITICAL RESULTS COMMUNICATION | certain | unambiguous | not_question | 1 # select * from foundentity where id=496 ; id | polarity | section_name | uncertainty | known_ambiguity | question_status | document_id -----+----------+--------------------------------+-------------+-----------------+-----------------+------------- 496 | asserted | CRITICAL RESULTS COMMUNICATION | certain | unambiguous | not_question | 1 # select * from foundentityconcept where concept_ontology='snomed_icd10_cm'; found_entity_id | concept_id | concept_ontology -----------------+------------+------------------ 443 | K76.9 | snomed_icd10_cm 496 | I82.90 | snomed_icd10_cm ... ``` You can look up the description of the `concept_id` from the above table in the `concept` table: ```bash # select * from concept where concept_id='K76.9'; concept_id | ontology | description ------------+-----------------+--------------------------------------- K76.9 | snomed_icd10 | Liver disease, unspecified (Disease or Syndrome) K76.9 | snomed_icd10_cm | Liver disease, unspecified (Disease or Syndrome) # select * from concept where concept_id='I82.90'; concept_id | ontology | description ------------+-----------------+--------------------------------------------------------------- I82.90 | snomed_icd10_cm | Acute embolism and thrombosis of unspecified vein (Disease or Syndrome) ``` ### Reported Event relation in Output Database When using the `emtellipro-db-client` Python database client the new reported event relation appears in the `reportedeventrelation` table and associated `reportedeventrelationtoentity`, `reportedeventrelationfromentity`, `reportedeventrelationmethod` and `reportedeventrelationtimeexpression` tables which correspond to the various arguments of the reported event relation. The attributes of `polarity` and `category` and the identifier of the found entity that is the subject of the relation is provided in `subject_id` which is the link to the `foundentity` table which lets you obtain all the information about the subject: #### Relation tables The ER diagram for the Reported Event relation is shown below: ![ER diagram for reportedevent relation](/_fern-files/emtelligent-prod.docs.buildwithfern.com/3a21ad5edccbe029b7d46f2ffab718483125141e485f5e5477a58ab2a870b483/assets/nlp/reported_event_erd.png) > ER Diagram for the Reported Event relation. The `reportedeventrelation` table looks like this: ```bash # select * from reportedeventrelation; id | polarity | category | subject_id ----+----------+------------------+------------ 1 | asserted | critical_results | 5 2 | asserted | critical_results | 9 3 | asserted | critical_results | 12 4 | asserted | critical_results | 15 5 | asserted | critical_results | 16 ... ``` The `reportedeventrelationtoentity` table below has the `reported_event_id` which links back to the `reportedeventrelation` table and `entity_id` which is the link to the `foundentity` table: ```bash # select * from reportedeventrelationtoentity; reported_event_id | entity_id -------------------+----------- ... ``` The `reportedeventrelationfromentity` table below has the `reported_event_id` which links back to the `reportedeventrelation` table and `entity_id` which is the link to the `foundentity` table: ```bash # select * from reportedeventrelationfromentity; reported_event_id | entity_id -------------------+----------- ... ``` The `reportedeventrelationmethod` table below has the `reported_event_id` which links back to the `reportedeventrelation` table and `entity_id` which is the link to the `foundentity` table: ```bash # select * from reportedeventrelationmethod; reported_event_id | entity_id -------------------+----------- ... ``` The `reportedeventrelationtimeexpression` table below has the `reported_event_id` which links back to the `reportedeventrelation` table and `entity_id` which is the link to the `foundentity` table: ```bash # select * from reportedeventrelationtimeexpression; reported_event_id | entity_id -------------------+----------- ... ``` #### SQL query A sample SQL query that provides a list of the reported event relations in a database produced by the database client is provided below: ```text WITH critical_res AS ( SELECT ev.*, rel.document_id FROM relation rel JOIN reportedeventrelation ev ON rel.relation_id = ev.id AND rel.type_='reportedevent' WHERE ev.category = 'critical_results' ), to_persons AS ( SELECT ev.reported_event_id, string_agg(fes.text, '::') as to_entities FROM reportedeventrelationtoentity ev JOIN entity e ON ev.entity_id = e.id AND e.type_='found' JOIN foundentity fe ON e.entity_id = fe.id JOIN foundentityspan fes ON fe.id = fes.found_entity_id GROUP BY ev.reported_event_id ), from_persons AS ( SELECT ev.reported_event_id, string_agg(fes.text, '::') as from_entities FROM reportedeventrelationfromentity ev JOIN entity e ON ev.entity_id = e.id AND e.type_='found' JOIN foundentity fe ON e.entity_id = fe.id JOIN foundentityspan fes ON fe.id = fes.found_entity_id GROUP BY ev.reported_event_id ), _methods AS ( SELECT ev.reported_event_id, string_agg(fes.text, '::') as methods FROM reportedeventrelationmethod ev JOIN entity e ON ev.entity_id = e.id AND e.type_='found' JOIN foundentity fe ON e.entity_id = fe.id JOIN foundentityspan fes ON fe.id = fes.found_entity_id GROUP BY ev.reported_event_id ), time_exprs AS ( SELECT ev.reported_event_id, string_agg(fes.text, '::') as time_expressions FROM reportedeventrelationtimeexpression ev JOIN entity e ON ev.entity_id = e.id AND e.type_='found' JOIN foundentity fe ON e.entity_id = fe.id JOIN foundentityspan fes ON fe.id = fes.found_entity_id GROUP BY ev.reported_event_id ) SELECT sub_fes.text as subject, sub_c.description as concept, sub_c.concept_id, tp.to_entities, frp.from_entities, m.methods, te.time_expressions, cr.polarity as event_polarity, sl.text as sentence, d.filename FROM critical_res cr -- get subject JOIN entity sub_e ON cr.subject_id = sub_e.id AND sub_e.type_='found' JOIN foundentity sub_fe ON sub_e.entity_id = sub_fe.id JOIN foundentityspan sub_fes ON sub_fe.id = sub_fes.found_entity_id JOIN foundentityconcept sub_fec on sub_fe.id = sub_fec.found_entity_id AND sub_fec.concept_ontology = 'snomed' JOIN concept sub_c on sub_fec.concept_id = sub_c.concept_id and sub_fec.concept_ontology = sub_c.ontology -- get critical result communication details LEFT JOIN to_persons tp ON cr.id = tp.reported_event_id LEFT JOIN from_persons frp ON cr.id = frp.reported_event_id LEFT JOIN _methods m ON cr.id = m.reported_event_id LEFT JOIN time_exprs te ON cr.id = te.reported_event_id -- get document info JOIN document d ON cr.document_id = d.id JOIN foundentitylocation fel ON sub_fe.id = fel.found_entity_id JOIN location l on fel.location_id = l.id AND l.type_='sentence' JOIN sentencelocation sl ON l.location_id = sl.id ORDER BY cr.document_id, sub_fes.start; ``` Which produces a table of results with the following columns. ```text subject | concept | concept_id | to_entities | from_entities | methods | time_expressions | event_polarity | sentence | filename ---------+---------+------------+-------------+---------------+---------+------------------+----------------+----------+---------- ``` ## Java SDK updates The latest NLP API Java SDK is version `5.9.0`. Refer to the [changelog for the Java SDK](https://legacy-docs.emtelligent.com/emtellipro-java-sdk/v5.15.0/) for a full explanation of the updates made since the last public release. You will need to use version `5.9.0` or later to take advantage of the new features described above.