How the legacy schema works

The NLP API SDK currently includes a single schema called ‘legacy’, which tries to store the output from the NLP API as faithfully as possible. However, it has certain quirks that are useful to know about to understand how to find the items you’re interested in.

Relation concept links

Relations in the NLP API’s results have links to concepts. However, the schema only supports found entity concepts using the foundentityconcept table.

To support relation concept links, new concept links are stored in the database by creating a new discontinuous found entity for each relation that has a concept link.

For each relation with at least one concept link, we create a new found entity that merges the arguments of the relation together. The attributes are merged together, and the spans and locations are the union of the two individual found entities.

Thus, if a sentence contains “mild mitral regurgitation” with a relation between “mild” and “mitral regurgitation” (which are the only 2 found entities), the schema saving code will create a new found entity for the relation containing two spans for the 2 base found entities.

This new discontinuous entity, will result in two rows in the two rows in the foundentityspan table.

SELECT found_entity_id, concept_ontology, concept_id, start, "end", text
FROM foundentityconcept
JOIN foundentityspan
USING (found_entity_id)
WHERE concept_id = '838451005';

And if we run that, we should see the following result:

 found_entity_id | concept_ontology | concept_id | start | end |         text
-----------------+------------------+------------+-------+-----+----------------------
           13574 | snomed           | 838451005  |    16 |  20 | mild
           13574 | snomed           | 838451005  |    29 |  49 | mitral regurgitation
  (2 rows)

Since the new discontinuous found entity is created from two existing found entities, you will now see 3 found entities in the database:

 found_entity_id | concept_ontology | concept_id | start | end |         text
-----------------+------------------+------------+-------+-----+----------------------
           13565 | snomed           | 255604002  |    16 |  20 | mild
           13567 | snomed           | 48724000   |    29 |  49 | mitral regurgitation
           13574 | snomed           | 838451005  |    16 |  20 | mild
           13574 | snomed           | 838451005  |    29 |  49 | mitral regurgitation

Found entities 13565 and 13567 are the original two found entities that are combined to create 13574; the original two found entities retain their individual concept links while the new found entity has a new concept link that was copied from the relation that generated it.

Snomed ICD-10 CM

The Snomed ICD-10 CM ontology returned by the NLP API contains extra mapping information in the form of map_rule, map_advice, map_group, map_priority, and map_category fields. Rather than storing these fields directly, they’re used to determine which concepts to store, with synthetic concept IDs.

The schema does not contain extra columns for these mapping fields; they’re just used to determine what to store in the concept_id column.

The rules for how to handle these concepts is as follows:

• if map_rule is TRUE and the map_advice is “ALWAYS <concept id>”, then the concept will be stored with the concept ID unchanged in the concept_id column.
• if the map_rule contains “IFA” or “OTHERWISE TRUE”, then the concept is stored with the concept_id column having the following format: concept_id | map_rule | map_group | map_priority
• otherwise, the concept is not stored.

Storing document text

There are two cases where the document text is stored: when the store_text save option is enabled, and when the NLP API returned document text in the annotated result. In all cases the NLP API-returned text is preferred.

The NLP API will return document text in its output in the following cases:

• the text processing feature is enabled,
• PDF document was provided as input, or
• the infer-document-structure feature is enabled.

Note that for PDF documents, and when document structure is inferred, the returned text is important to store because the spans for the found entities will only be valid on the returned text; without it, the spans are useless.