Imagelink Relations

In this section, we explain the imagelink relation data model and how it is implemented in the NLP API JSON output and in the Output Database schema. We also provide sample queries to illustrate the aspects of the data schema.

After reading this page, we hope you will:

1. Understand what imagelink relations are.
2. Know how to retrieve image references and related findings for imagelink relations.
3. Know how to find imagelink relations in documents.

The content in this section builds on the concepts and data models explained in earlier sections (see Entity-level Tables, and Document-level Tables).

What are Imagelink Relations?

Imagelink relations are radiology-specific relations. They are found in sentences where the dictating radiologist explicitly mentions the Series and Image where a finding was observed. Imagelink relations is an association between two types of arguments: image findings and references.

Consider the following sentence:

2.3 cm lesion in the liver (series 2, image 128)

From the above sentence, the NLP API extracts the arguments that comprise an imagelink relation as shown:

In addition to the example above, the NLP API recognizes many variations in how images can be referenced in reports. The following are examples of image references recognized by the NLP API:

image 128 of series 2
(2:128)
(2:120, 2:128)
series 2, image 128

Note, the list above is not exhaustive.

An imagelink relation will typically have only one image reference. However, it can have many image finding entities. When using the SNOMED ontology, findings are entities with semantic type such as body structure, finding, disorder or morphologic abnormality. You may also use other semantic type categorizations provided by other ontologies enabled during processing such as UMLS, MEDCIN and RadLex.

It should be noted that imagelink relations are only extracted from FINDINGS and IMPRESSIONS sections of radiology reports.

JSON Data Model

The JSON below shows the NLP API’s JSON result output for a medication relation extracted from this statement:

2.3 cm lesion in the liver (series 2, image 128)

{
    "relations": {
        "imagelinks": [
            {
                "label": "RIM0",
                "attributes": {
                    "confidence": 1
                },
                "args": {
                    "image_findings": [
                        {
                            "ref": "E1",
                            "text": [
                                "lesion"
                            ]
                        },
                        {
                            "ref": "E3",
                            "text": [
                                "liver"
                            ]
                        }
                    ],
                    "references": [
                        {
                            "ref": "E2",
                            "text": [
                                "(series 2, image 128)"
                            ]
                        }
                    ]
                },
                "concept_links": []
            }
        ]
    }
}

From the JSON output above, we see that:

1. The imagelinks array is an element of the relations object. Each element of the imagelinks array represents an imagelink relation instance.
2. An imagelink relation instance has two types of named arguments: references and image_findings.
3. The references argument is an entity that represents an image reference.
4. The image_findings arguments are entities that relate to an observed finding or structure.
5. There can be a one-to-many relation between references and image_findings, but at least one image reference must be present.

ImageLink Relation Tables

In the Output Database schema, imagelink relations are stored in the imagelinkrelation table. Its arguments are represented by the imagelinkrelationimagefinding and imagelinkrelationreference tables.

The following ER diagram shows the relationship between these tables:

From the ER diagram above, a few key points should be noted:

1. The primary key of the imagelinkrelationimagefinding table is a foreign key reference to the imagelinkrelation table’s primary key.
2. Similarly, the primary key of the imagelinkrelationreference table is a foreign key reference to the imagelinkrelation table’s primary key.
3. The imagelinkrelationimagefinding table entity_id column is a foreign key reference to the entity table. This foreign key constraint implements a one-to-many relation between imagelink relation and finding entities.
4. The imagelinkrelationreference table entity_id column is a foreign key reference to the entity table. This foreign key constraint implements a one-to-many relation between imagelink relation and image reference entities.
5. The entity table has a foreign key reference to the foundentity table which stores many entity attributes. By joining the foundentity with other entity-related tables, you can retrieve information associated with the concepts, entity types and locations for an entity. See Entity-level Tables.

Note: While the NLP API’s database schema supports a one-to-many relation between imagelink relations and image references, in practice, we can consider the relation between imagelink relations and image references to be one-to-one.

Sample Queries

In this section, we provide sample SQL queries that illustrate how to retrieve data for imagelink relations. SQL queries will be discussed in order as follows:

• Query: Retrieve the annotated text for image references and findings
• Query: Count of image references by finding concept
• Query: The average number of image references per Radiology CT report
• Query: Retrieve sentences containing imagelinks

Note: Because imagelink relations are specific to Radiology reports, please use an Output Database that contains output from processing Radiology reports.

Query: Retrieve the annotated text for image references and findings

This query finds image references related to umbilical hernia: a partial or full collapse of the lung. It retrieves the terms that NLP API identified as the image reference, and the finding of umbilical hernia.

To find information about imagelink relations we start at the imagelinkrelation table and perform a series of joins. The query involves the following steps:

1. First, construct an intermediate table containing the image term and a comma-delimited list of finding terms for all imagelink relations, using a CTE, as follows:
   1. Get the finding terms by performing an inner join between the imagelinkrelation and imagelinkrelationimagefinding tables to get the entity_id. Then, perform a series of inner joins with entity-related tables to retrieve the text column from the foundentity table.
   2. Get the image terms by performing an inner join between the imagelinkrelation and imagelinkrelationreference tables to get the entity_id. Again, perform the same series of inner joins to retrieve the annotated text from the foundentity table.
   3. Use the string_agg Window function to aggregate the finding terms associated with a specific image reference into a comma-delimited list by partitioning on the imagelinkrelation table primary key.
2. In the outer query, use a WHERE clause to filter for the string umbilical hernia in the comma-delimited list of findings.

WITH image_findings_table AS (SELECT imagelinkrelation.id AS imagerelation_id, image_fe.text AS image_reference_term,
                                string_agg(finding_fe.text, ', ')
                                OVER (PARTITION BY imagelinkrelation.id ) AS image_finding_terms
                            FROM imagelinkrelation
                            -- Joins to retrieve entity text for 'Image FINDING' foundentity                                            
                                    JOIN imagelinkrelationimagefinding image_finding
                                    ON image_finding.imagelink_id = imagelinkrelation.id
                                    JOIN entity finding_e
                                    ON finding_e.id = image_finding.entity_id AND finding_e.type_='found'
                                    JOIN foundentity finding_fe
                                    ON finding_fe.id = finding_e.entity_id
                            -- Joins to retrieve entity text for 'Image REFERENCE' 
                                    JOIN imagelinkrelationreference image_ref
                                    ON image_ref.imagelink_id = imagelinkrelation.id
                                    JOIN entity image_e 
                                    ON image_e.id = image_ref.entity_id AND image_e.type_='found'
                                    JOIN foundentity image_fe
                                    ON image_fe.id = image_e.entity_id
                        ORDER BY imagelinkrelation.id)
SELECT *
FROM image_findings_table
WHERE image_finding_terms LIKE '%%umbilical hernia%%' -- 
GROUP BY image_findings_table.imagerelation_id, image_reference_term, image_finding_terms;

Query: Count of image references by finding concept

This query finds the frequencies of image references grouped by findings. Since each imagelink relation is one image reference, we can count imagelink relations to get the number of image references. Then, instead of grouping by finding terms, we will group the imagelink relations by the finding’s SNOMED CT concept.

This query involves the following steps:

1. First, start with imagelinkrelation table and perform the series of table joins to retrieve the SNOMED CT concept from the concept table.
2. Then, perform a count distinct on the imagelink relation ID to count the number of imagelink relations for each concept, grouping by concept ID and description.

Note, the DISTINCT keyword is required because it is unlikely, but possible, for an imagelink relation to have multiple findings with the same concept ID.

SELECT finding_c.concept_id, finding_c.description, count(DISTINCT imagelinkrelation.id) as num_image_refs
  FROM imagelinkrelation
           JOIN imagelinkrelationimagefinding image_finding
           ON image_finding.imagelink_id = imagelinkrelation.id
           JOIN entity finding_e
           ON finding_e.id = image_finding.entity_id AND finding_e.type_='found'
           JOIN foundentity finding_fe
           ON finding_fe.id = finding_e.entity_id
           JOIN foundentityconcept finding_fec
           ON finding_fec.found_entity_id = finding_fe.id
        -- Join with concept table to retrieve concept id and description
           JOIN concept finding_c
           ON finding_c.concept_id = finding_fec.concept_id
              AND finding_c.ontology = finding_fec.concept_ontology --Ensure
 WHERE finding_c.ontology = 'snomed'
 GROUP BY finding_c.concept_id, finding_c.description
 ORDER BY 1
 LIMIT 5;

Query: The average number of image references per Radiology CT report

This query counts the imagelink occurrences by document in a CTE, and then gets the average number of imagelink occurrences across all documents in the database.

  WITH image_counts_by_doc AS (SELECT d.id, d.category, d.subcategory,
                                      count(imagelinkrelation.id) AS image_reference_count
                                 FROM imagelinkrelation
                                          JOIN imagelinkrelationreference image_ref
                                          ON image_ref.imagelink_id = imagelinkrelation.id
                                          JOIN entity finding_e
                                          ON finding_e.id = image_ref.entity_id AND finding_e.type_='found'
                                          JOIN document d
                                          ON d.id = finding_e.document_id
                                GROUP BY d.id)
SELECT round(AVG(image_reference_count)) AS average
  FROM image_counts_by_doc
 WHERE category = 'Radiology'
   AND subcategory = 'CT';

Query: Retrieve sentences containing imagelinks

This query shows how to use the location, document and sentencelocation tables to retrieve sentences containing imagelink relations from documents. The query involves the following steps:

1. Starting with the imagelinkrelation table, perform a series of JOINs as shown in the SQL code below.
2. Note that when joining the foundentitylocation with the location table, in addition to joining on the foreign key fields, we need to check that the location type is sentence and not section as in the SQL code below.
3. Then, in the SELECT statement, use the text column from the sentencelocation table to retrieve the sentence containing each imagelink relation.

For a more complete explanation of how to work with the location, sentencelocation, and document tables see Document-level Tables.

SELECT image_doc.id AS doc_id, imagelinkrelation.id AS imagelink_id,
    image_fe.text,
    sl.text AS sentence
FROM imagelinkrelation
        JOIN imagelinkrelationreference image_ref ON image_ref.imagelink_id =
                                                        imagelinkrelation.id
        JOIN entity image_e ON image_e.id = image_ref.entity_id AND
                                image_e.type_='found'
        JOIN foundentity image_fe ON image_fe.id = image_e.entity_id
        JOIN foundentitylocation image_fel ON image_fel.found_entity_id = image_fe.id
        JOIN location image_loc ON image_loc.id = image_fel.location_id AND
                                    image_loc.type_ = 'sentence' -- IMPORTANT
        JOIN sentencelocation sl ON image_loc.location_id = sl.id
        JOIN document image_doc ON image_doc.id = image_loc.document_id