Entity heading status

The heading_status attribute of found entities is used for determining if a found entity is within a heading.

Example NLP API result file:

Specifically, the following snippet from the file contains the new heading_status fields.

"entities": {
  "found": [
    {
      "label": "E0",
      ...,
      "section_name": "ALLERGIES",
      "attributes": {
        "heading_status": "outside_heading",
        ...,
      },
      "text": [
        "75 MG"
      ]
      ...,
    },
    {
      "label": "E1",
      ...,
      "section_name": "ALLERGIES",
      "attributes": {
        "heading_status": "inside_heading",
        ...,
      },
      "text": [
        "Allergies"
      ]
      ...,
    },
...

Changes

Python API

There is a new attribute: heading_status. When returned by the NLP API this will have values "inside_heading" or "outside_heading" (it may also be None if not returned by the NLP API).

Example usage:

>>> import json
>>> import pathlib
>>> from emtellipro.data import AnnotatedDocument
>>> json_data = pathlib.Path('headings.json').read_text()
>>> doc_text = pathlib.Path('headings.txt').read_text()
>>> doc_dict = json.loads(json_data)['documents'][0]
>>> ann_doc = AnnotatedDocument(doc_dict, text=doc_text)
>>> for found_entity in ann_doc.found_entities:
...     print(found_entity.text, found_entity.heading_status)
['75 MG'] outside_heading
['Allergies'] inside_heading
['LYRICA'] outside_heading
['CAPSULE'] outside_heading
['DEPRESSION'] outside_heading

Schema changes

The heading_status attribute is stored in the foundentity table in a new column.

The schema of that table now looks like (for PostgreSQL):

                                        Table "public.foundentity"
     Column      |          Type          | Collation | Nullable |                 Default
-----------------+------------------------+-----------+----------+-----------------------------------------
 id              | integer                |           | not null | nextval('foundentity_id_seq'::regclass)
 polarity        | character varying(255) |           |          |
 section_name    | text                   |           |          |
 uncertainty     | character varying(255) |           |          |
 known_ambiguity | character varying(255) |           |          |
 question_status | character varying(255) |           |          |
 guidance        | character varying(255) |           |          |
 heading_status  | character varying(15)  |           |          |
 document_id     | integer                |           | not null |
Indexes:
    "pk_foundentity" PRIMARY KEY, btree (id)
Foreign-key constraints:
    "fk_foundentity_document_id_document" FOREIGN KEY (document_id) REFERENCES document(id)
Referenced by:
    TABLE "foundentityconcept" CONSTRAINT "fk_foundentityconcept_found_entity_id_foundentity" FOREIGN KEY (found_entity_id) REFERENCES foundentity(id)
    TABLE "foundentitylocation" CONSTRAINT "fk_foundentitylocation_found_entity_id_foundentity" FOREIGN KEY (found_entity_id) REFERENCES foundentity(id)
    TABLE "foundentitymeasurementunit" CONSTRAINT "fk_foundentitymeasurementunit_found_entity_id_foundentity" FOREIGN KEY (found_entity_id) REFERENCES foundentity(id)
    TABLE "foundentityspan" CONSTRAINT "fk_foundentityspan_found_entity_id_foundentity" FOREIGN KEY (found_entity_id) REFERENCES foundentity(id)
    TABLE "foundentitytype" CONSTRAINT "fk_foundentitytype_found_entity_id_foundentity" FOREIGN KEY (found_entity_id) REFERENCES foundentity(id)

The following is an example of what the table contents would look like:

testdb=# select * from foundentity limit 5;
 id | polarity | section_name | uncertainty | known_ambiguity | question_status |   guidance   | heading_status  | document_id
----+----------+--------------+-------------+-----------------+-----------------+--------------+-----------------+-------------
  1 | asserted | ALLERGIES    | certain     | unknown         | not_question    | not_guidance | outside_heading |           1
  2 | asserted | ALLERGIES    | certain     | unknown         | not_question    | not_guidance | inside_heading  |           1
  3 | asserted | ALLERGIES    | certain     | unambiguous     | not_question    | not_guidance | outside_heading |           1
  4 | asserted | ALLERGIES    | certain     | unambiguous     | not_question    | not_guidance | outside_heading |           1
  5 | asserted | ALLERGIES    | certain     | unknown         | not_question    | not_guidance | outside_heading |           1
(5 rows)