Understanding the NLP API Output
The NLP API engine takes unstructured medical text as its input and returns structured medical data as its output. It is accessed via its API. There are two sets of knowledge that a user needs to know in order to understand what the NLP API engine is doing, and what users can do with the output:
- What types of structured data the engine outputs
- How the API output is structured
N.B. Please note that upon submitting a request to the NLP API, the user can specify (as input options) the types of output that they want. These input options are detailed fully in the API documentation. The sections below demonstrate what the output can be with certain input options and with a small sample file. Processing different types of text with different input options will produce different output.
Structured Data Output
In converting medical unstructured text into structured text — at its most basic level — NLP API identifies entities in the text, and information about those entities, and relationships between entities.
An example of an entity might be the name of a medical condition, such as heart attack. In the sentence:
There are two entities: patient and heart attack.
In this same sentence, there is a relationship between the entity patient and heart attack because it is explicitly stated that the patient (a person) had a heart attack (a medical condition).
While not visible in the sentence itself, the NLP API would read this sentence and extract the following information about the entity heart attack:
- That the entity ‘heart attack’ is a synonym of a SNOMED-CT concept, with a concept ID of: 22298006
- That the SNOMED-CT fully-specified name for this concept is ‘Myocardial infarction (disorder)’
- That SNOMED-CT considers a ‘heart attack’ to have a semantic type of ‘disorder’
- That the entity ‘heart attack’ has a UMLS semantic type of ‘Disease or Syndrome’
- That the term’s polarity was asserted, i.e. that the patient had a heart attack, and that there was no negation term found to suggest that the patient did not have a heart attack.
- That there was no uncertainty about the statement, i.e. there was no uncertainty words used in describing what happened, e.g. “The patient might have had a heart attack.”
- That the sentence that the term ‘heart attack’ was found in read, “The patient had a heart attack.”
- That the term ‘heart attack’ was found in the “Indication” section
- The start and end character positions of the term ‘heart attack’ and of the sentence it was found in
These are not all of the types of information that NLP API extracts; the apidocs contains the most up-to-date listing of all of the feature, relationships, and entity types that NLP API can extract; this is just intended to introduce the reader to the basics of the NLP API’s data extraction abilities.
How the API Output is Structured
The structured data that the API returns to users is highly structured and has internal links and can be quite complicated-appearing at first glance. If we take the following raw text input (a document fragment, but it’s illustrative):
Once we process it as Radiology/CT document using the NLP API we get the following JSON output:
Let’s break down the output into sections of the hierarchy so that we can go over everything.
Output at a Glance
Every element of the documents array has the same shape. Four of its members describe the document itself —id, attributes (the report category and subcategory), processing_status, and ontology_versions —and a text member carries the document text when it is needed (see Documents). The remaining four members carry the extracted data and cross-reference each other by label:
- entities — the
foundentities, each covering one or more spans of text, and theassumedentities that some relations introduce. - concepts — the ontology concepts that entities and relations were linked to, grouped by ontology.
- relations — the relations that hold between entities, grouped by relation type.
- locations — the
sentences,sections,headings, andpagesof the document.
Each found entity carries an attributes object. heading_status (inside_heading or outside_heading) is always present; the remaining attributes are present only when the corresponding processing feature was requested:
polarity—assertedornegated: whether the entity is within the scope of a negation.uncertainty—certainoruncertain: whether the entity is within the scope of a hedge, e.g. “might be”.question_status—is_questionornot_question: whether the entity is within the scope of a clinical question, e.g. “rule out”.guidance—is_guidanceornot_guidance: whether the entity is part of a statement advising someone to do something.factuality—factual,hypothetical, orgeneric: whether the mention is a statement of fact about the patient, a hypothetical, or general knowledge.experiencer— who experiences the entity, as a string such as"patient","mother", or"provider".known_ambiguity—ambiguous,unambiguous, orunknown: whether the text of the entity can refer to more than one medical concept.measurement_unit— the unit heuristically extracted from a measurement entity, e.g.["MG"], ornullwhen the measurement has no unit.
A found entity may also carry a concept_confidences object, which scores the entity’s links to concepts, and a text member giving the text of each of its spans.
Nine types of relation can be extracted, each in its own array under relations:
- experiencers — a person experiences a condition, e.g. the patient, or the patient’s mother.
- anatomicsites — the anatomic location of a disorder, finding, or other clinical entity.
- qualifiers — a qualifier value modifies an entity; the kind of modification is given by the
qualifier_typeattribute. - measurements — the measured value of a morphological abnormality or anatomical site.
- medications — the parts of a medication prescription: the drug plus its dosage, route, quantity, mode, frequency, duration, necessity, modifiers, indications, and date/time.
- temporalities — an entity falls within the scope of a temporal expression; the
categoryattribute ishistoricalorspecific. - followups — a follow-up procedure is requested, optionally with a timeframe and a reason.
- imagelinks — a reference to an image or image slice, and to the findings in that image.
- reportedevents — a communication between two groups of entities; the
categoryattribute iscritical_results.
Result Format
As new features are added to the NLP API, its output will necessarily change to include new types of structured data in its output. The name of the NLP API output format is provided by the format field of the output, e.g. "format": "emtellipro-json-2". The version field gives the version of that format used for this result, e.g. "version": "2.37.0", and the engine-version field records the version of the engine that produced it, e.g. "engine-version": "2.37".
NLP API result format version numbering is described in Understanding NLP API Component Versions.
Documents
Here, we have a JSON array of the documents/reports/notes that were submitted to the engine, e.g.:
The main data element here is the id which is the identifier that you, the client, submitted to the engine as the unique identifier that you wanted to use to identify all of the data that originated in this document. You can submit as many documents you wish, but each should have a unique identifier to identify the report.
For the purposes of highlighting discovered entities and sentences in the original text — useful in application creation, when submitting unique identifiers, a useful unique identifier should be used that can track the annotations back to the original medical document. Customers can limit PHI exposure by using non-PHI report unique identifiers such as database ids rather than source-referenceable unique identifiers like accession numbers.
Each document also reports a processing_status of success or error. If it is error, check that the report meets the input requirements (UTF-8 encoding being the most common problem) and then contact Emtelligent.
A document may additionally contain a text member holding the text of the whole report. It is present when the text processing feature is enabled, when the input document did not supply plain text itself (for example PDF input), or when a processing feature rewrote the text of the document. In the last two cases the returned text — and not the text that was submitted — is what all of the spans in the rest of the output are offsets into.
Locations
This section identifies locations within a document that are of clinical relevance. There are four kinds of location, each in its own array: sentences, sections, headings, and pages. Sentences and sections are produced whenever a processing feature generates entities from text; page locations are produced only when the input carries page information. Entities and locations cross-reference each other: the locations array of an entity lists the labels of every location containing it.
The locations example from this section looks like:
The sentences array has a single element here — the heading and the line beneath it are segmented as one sentence — and the sentence span (start and end character position) of the sentence is contained in the spans array under each sentence label. If the same sentence occurred in more than one place in this report (relatively common in medical reports), then the spans element would contain more than one span. As well, it is possible to have sentences that span multiple pages in a report, so multiple spans can be required to describe a single sentence.
Every element of the output that can be referred to elsewhere carries a label made up of a prefix and a number: S for a sentence, SEC for a section, H for a heading, E for an entity, C followed by an ontology name for a concept, and a per-relation-type prefix such as REX for an experiencer relation. Labels are how the parts of the output refer to one another.
These labels are opaque identifiers. They are unique within a document, but the numbering is not guaranteed to start at any particular value, to be contiguous, or to run in the order in which the elements appear in the text — in this example the only sentence is labelled S2 and the only section SEC-1 — and the same input processed with a different set of processing features may attach a given label to a different element. The arrays that carry these elements are likewise not guaranteed to be in document order. Use a label only as a key to look up the element it names; do not parse or sort by the number it contains, and sort by spans if you need the elements in reading order.
When the text processing feature is enabled, each location also carries a text member giving the text of its spans, which is useful for building applications, so the end user can see what the source of the entities in the sentence was.
Sections nest: the level of a section is 0 for an outer-most section and increases for each level of sub-section, and parent points at the section that contains it (or is null at the outer-most level). The heading member points at the heading location that named the section, and the corresponding heading points back at the section. Not every section has a heading — text at the start of a document before any heading is grouped into an intro section, and sections supplied through single section labels in the input have no heading in the text — and in those cases heading is null.
When the input has pages, as it does for PDF input, each page location gives its page_index in the page order starting from zero, and each section location gains a pages member listing the pages it appears on.
Entities
Let’s look at this very important section next. From processing this reports, 4 separate entities were discovered: indication, patient, heart, and heart attack. Let’s address why we found both heart and heart attack. It’s because of the different ontologies that we are using. RadLex doesn’t contain the entity heart attack, but it does contain the entity heart. SNOMED-CT contains the entity heart, but in this case, the entity heart attack is the best match as it is the subject of the sentence.
We’re going to look at just a single entity here - heart attack:
Here, we describe a number of discrete attributes about the entity heart attack:
- That it was found in the INDICATION: section of the report
- That it had a character span from character position 30 to character position 42 in the report, possibly useful in application building
- That it was found in sentence S2 (from Section 2: Locations above)
- That it has a UMLS semantic type of Disease or Syndrome and a SNOMED-CT semantic type of disorder
- That there was no uncertainty about the patient having a heart attack
- That the heart attack happened, i.e. it was not negated by saying something like, “The patient did not have a heart attack”
- That the entity is not inside a section heading, but in the body text under one
- That the mention is a statement of fact about the patient rather than a hypothetical or a piece of general medical knowledge
- That the person who experienced the heart attack is the patient
Which attributes appear depends on which processing features were requested; the full set is listed in Output at a Glance above. Two of them, factuality and experiencer, are worth singling out because they summarize context that would otherwise have to be reconstructed from several attributes at once. In the sentence “Please call your doctor if you have severe pain”, the entity severe pain is not something the patient has, and its factuality would be hypothetical rather than the factual seen here; a statement of general medical knowledge would instead be generic. The experiencer attribute names who the entity happened to, as a plain string, so that an application can restrict a search to the patient’s own findings without having to follow experiencer relations.
Other attributes that this request did not ask for would appear alongside these. If the input text had read “r/o heart attack”, for example, and the corresponding processing feature had been requested, the entity would also carry "question_status": "is_question"; and when the text processing feature is used the entity carries a text member giving the text of each of its spans — here, "text": ["heart attack"] — which is again useful in debugging.
We’ve also give it two distinct references - a label of E0 and its concept links.
When the entity-concept-link-confidence processing feature is enabled, an entity also has a concept_confidences member that scores the entity’s links to concepts. It is keyed by the same concept labels that appear in concept_links, though a confidence is not necessarily available for every one of them:
Here the entity links to two concepts but only the SNOMED-CT link is scored. An entity whose links are all unscored still carries the member, as an empty object — for example the heart entity above, whose only link is to RadLex, has "concept_confidences": {}. Treat a missing key as “no score available” rather than as a score of zero.
The entity_type field is to explain semantic types (or other types of post-coordination) that are useful for application development. If a term like ‘plasmacytoma’ was mentioned (a type of tumor seen in multiple myeloma, a type of cancer), this would have the entity types of ‘disorder’ for SNOMED-CT and ‘Neoplastic process’ for UMLS. Thus an application could be built to show all patients with cancers, and the patient with a plasmacytoma would be listed as the NLP API has already returned to you that a plasmacytoma is a type of neoplastic process.
For this input, the assumed array at the end of the entities section is empty, because it was not necessary to make any assumptions about hidden meaning in the text. For example, if the sentence read, “History of heart attack.”, we would have to assume it was about the patient’s family if it was in a family history section of a report, otherwise we would have to assume it was about the patient if it was in some other section of the report.
Document Attributes
This section refers to the document attributes which we know about, which is useful in determining the context of the report. In this case, we have the following attributes about the document:
Here, we see that the attribute subcategory for the report was specified to have the value “CT”. The method member records where the value came from: manual when it was supplied in the input, and automatic when the NLP API detected it. Both category and subcategory are optional in the input, and when they are not supplied the NLP API detects the report kind and reports the detected values here with a method of automatic. Automatic detection is not enabled in every installation of the NLP API; where it is disabled, the report kind supplied in the input is used. A report kind that is supplied in the input is always used as given.
Relations
This section is for describing relations between entities in the text. In the example that we looked at in Section 3 above, the NLP API has extracted a relation of type experiencer - that between patient and heart attack:
The relations element could contain information such as a drug’s dosage, or the anatomical site of a disease process (e.g. ‘Right lower lobe pneumonia’ describes the disorder pneumonia occurring in the body site right lower lobe of lung). In this simple example, we are extracting the experiencer of the disease process heart attack. The nine relation types that can appear here, one array each, are listed in Output at a Glance above; an array is present when the processing feature that produces that relation type was requested.
Every relation has an args object whose members are the entities that take part in it, an attributes object holding whatever qualifications apply to that relation type, and a concept_links array. Relation concept links work like the ones on entities, but they are for the concept denoted by the relation as a whole rather than by either of its arguments. For example, in “Family history of cardiovascular disease” the experiencer relation as a whole links to the SNOMED-CT concept 266894000, Family history: Cardiovascular disease (situation), which neither argument denotes on its own. The attributes object of a relation may also carry a confidence score, which is null when a score is not available for that relation.
The attributes section for this experiencer relation contains a polarity element, because of the linguistic necessity in assigning a polarity to a sentence where a person does or does not have a thing (such as a medical condition). To better explain, if the sentence read, “The patient’s mother had a heart attack but his father did not.”, we would have to have separate polarities for each experiencer, and this data structure allows us to do so. In building an application to properly use this data, the polarity in the entities section described above in Section 3 should be ignored because there would be potentially more useful polarity information in the relations section. But getting back to the example of “The patient had a heart attack”, this is a simple declarative sentence and the polarities from the experiencer relation and entity section are concordant due to the information contained in the sentence.
Since relations have two (or more) entities, each entity in the relation is assigned an entity reference E2 and E0 respectively. These references can be used to look up entity information found in the entities section. As well, each relation has a relation label (here, REX0) for internal reference purposes. When the text processing feature is used, each argument also carries a text member giving the text of the entity it refers to.
Concepts
To understand the intent of this section, the reader should think about the use of synonyms in medical reports. There are many, many medical synonyms that may all refer to the same core concept. For example, the core concept of the proper medical term, myocardial infarction could have the following medical synonyms:
- Infarction of heart
- Cardiac infarction
- Heart attack
- MI
- Myocardial infarct
And in a medical report, a concept may be used in several places. In a hospitalization, a patient may have multiple myocardial infarction events (if they are extremely unwell), and so the NLP API normalizes the output to prevent having to re-state the core concept multiple times in its output.
The concepts are grouped by the ontology they came from, and each ontology is present only when the processing feature that enables it was requested. Here is part of the concepts section from the example above, showing the concepts found in the SNOMED-CT and RadLex ontologies:
Here, we can see the unique concept IDs for the identified concepts (entities) as well as the source ontology from which they came. The term heart was identified in the RadLex ontology only, whereas heart attack was identified in SNOMED-CT and in ICD-10-CM, the other ontologies that were enabled for this request. The same real-world concept therefore appears once per ontology, under that ontology’s own identifier.
The NLP API also returns the fully-specified name for the term taken from the ontology (e.g. the colloquial term ‘heart attack’ has a preferred/fully-specified name of ‘Myocardial infarction’, and SNOMED-CT fully-specified names also contain a semantic type, in this case, ‘(disorder)’). We assign internal relationship labels to the concepts so that these concepts can be mapped back to the entity that described them. In this case, in the entities section of the report, the entity with concept link Csnomed0 referred to the SNOMED-CT concept of 22298006, and the internal relation of Csnomed0 can be used to link the concepts and entities sections together.
A few ontologies add fields of their own beyond label, concept_id, and description. MEDCIN concepts carry a result member, a MEDCIN-specific polarity classification with the values positive (abnormal), negative (normal), and unspecified (un-entered). ICD-10-CM concepts carry the mapping information that SNOMED-CT publishes for the code: map_rule, map_advice, map_group, map_priority, and map_category. These are needed because a single SNOMED-CT concept can map to more than one ICD-10-CM code, and the map rules and groups say how to choose between them.
Ontology Versions
The NLP API performs named entity recognition and maps terms it recognizes to medical concepts in biomedical ontologies. This section of the output provides information about the ontologies used to generate the NLP API output. It lists the ontology version of every ontology used to process the source document. Other details about ontologies may also be provided such as their release_type, packaging_date, and language. The ontologies that can appear here — and, with the same names, in the concepts section — are snomed, radlex, rxnorm, umls_nci, umls_hgnc, umls_loinc, medcin, snomed_icd10, and snomed_icd10_cm.
JSONlines
The NLP API JSON output from the Python client uses the JSONlines format. For single report submissions, just a single line of JSON output will be written, but for large submissions, the output may contain multiple individual JSON lines, each is a valid JSON object as per the emtellipro JSON Output Specification, and JSONlines specification.
Summary
This document will evolve as new text extraction features are added to the NLP API. The Engine API documentation is the ultimate final source for what the engine can output, and so users are referred to the API documentation that they are using to get a complete list of feature options that are available.