Page locations | emtelligent Documentation

When the input contains pages, the NLP API returns new page locations and links sections to pages.

Example NLP API result file:

Specifically, the following two snippets show the new page location, and how they’re linked to the sections.

title="The pages are their own field in "locations""

"locations": {
    ...,
    "pages": [
        {
            "label": "P1",
            "spans": [
                {
                    "start": 0,
                    "end": 1484
                }
            ],
            "page_index": 0,
            "text": [...]
        }
    ]

The sections contain a reference to pages.

"locations": {
    "sections": [
        {
            "label": "SEC-0",
            "spans": [
                {
                    "start": 0,
                    "end": 130
                }
            ],
            "name": "INTRO",
            "level": 0,
            "parent": null,
            "heading": null,
            "pages": [
                "P1"
            ],
            "text": [
                "Fax: (555) 555-1212\n\nFake Imaging Labs\n\nFake City, FP FOF OFO\n\nwag ake\nPh: (555) 555-1212 | 7\n\nCT SCAN OF THE ABDOMEN AND PELVIS\n\n"
            ]
        },

Changes

Python API

The new page locations are represented using the new class PageLocation. These can be found in attr:emtellipro.data.AnnotatedDocument.locations under the "page" key.

Example usage:

import json
import pathlib
from emtellipro.data import AnnotatedDocument
json_data = pathlib.Path(
    'sample_ct_imagebased_report.json',
).read_text()
doc_dict = json.loads(json_data)['documents'][0]
ann_doc = AnnotatedDocument(doc_dict)
print("Sections")
for section in ann_doc.locations['section']:
    print(section.name, [p.page_index for p in section.pages])
print()
print("Pages")
for page in ann_doc.locations['page']:
    print(page.spans, page.page_index)

Output of above code.

Sections
INTRO [0]
INDICATION [0]
TECHNIQUE [0]
COMPARISON [0]
FINDINGS [0]
IMPRESSION [0]
Pages
[Span(start=0, end=1484)] 0

Schema changes

The new page locations are supported using:

new pagelocation table (contains the page location)
new pagelocationspan table (contains the spans for the pages)
new pagesection table (links pages and sections)

Page location

Schema of the new pagelocation table (in PostgreSQL):

                              Table "public.pagelocation"
   Column   |  Type   | Collation | Nullable |                 Default
------------+---------+-----------+----------+------------------------------------------
 id         | integer |           | not null | nextval('pagelocation_id_seq'::regclass)
 text       | text    |           |          |
 page_index | integer |           |          |
Indexes:
    "pk_pagelocation" PRIMARY KEY, btree (id)
Referenced by:
    TABLE "pagelocationspan" CONSTRAINT "fk_pagelocationspan_page_location_id_pagelocation" FOREIGN KEY (page_location_id) REFERENCES pagelocation(id)
    TABLE "pagesection" CONSTRAINT "fk_pagesection_page_id_pagelocation" FOREIGN KEY (page_id) REFERENCES pagelocation(id)

Example of the data that is in that table:

testdb=# select * from pagelocation limit 5;
 id | text | page_index
----+------+------------
  1 |      |          0
(1 row)

Linking to the document is done through the same location table as usual, but with type_ = 'page':

testdb=# select * from location where type_ = 'page';
 id | type_ | document_id | location_id
----+-------+-------------+-------------
 45 | page  |           1 |           1
(1 row)

Page location span

Schema of new pagelocationspan table (in PostgreSQL):

                                 Table "public.pagelocationspan"
      Column      |  Type   | Collation | Nullable |                   Default
------------------+---------+-----------+----------+----------------------------------------------
 id               | integer |           | not null | nextval('pagelocationspan_id_seq'::regclass)
 start            | integer |           |          |
 end              | integer |           |          |
 page_location_id | integer |           |          |
Indexes:
    "pk_pagelocationspan" PRIMARY KEY, btree (id)
Foreign-key constraints:
    "fk_pagelocationspan_page_location_id_pagelocation" FOREIGN KEY (page_location_id) REFERENCES pagelocation(id)

The two tables can be joined on pagelocation.id = pagelocationspan.page_location_id:

testdb=# select * from pagelocation join pagelocationspan on pagelocation.id = pagelocationspan.page_location_id;
 id | text | page_index | id | start | end  | page_location_id
----+------+------------+----+-------+------+------------------
  1 |      |          0 |  1 |     0 | 1484 |                1
(1 row)

Section page

The link between the page and section is done using the new pagesection table:

              Table "public.pagesection"
   Column   |  Type   | Collation | Nullable | Default
------------+---------+-----------+----------+---------
 page_id    | integer |           | not null |
 section_id | integer |           | not null |
Indexes:
    "pk_pagesection" PRIMARY KEY, btree (page_id, section_id)
Foreign-key constraints:
    "fk_pagesection_page_id_pagelocation" FOREIGN KEY (page_id) REFERENCES pagelocation(id)
    "fk_pagesection_section_id_sectionlocation" FOREIGN KEY (section_id) REFERENCES sectionlocation(id)