Simple Query App Tutorial

Medical NLP doesn’t exist just for itself - you use NLP software on unstructured medical text for a reason - e.g. to know how many of your patients have cancer, or to be able to ask your phone if Mrs. Smith has any allergies, or to have better search tools in your EMR. And because there are so many different uses of NLP, everyone is going to do the integration a little differently. But that’s ok - the basics are that you need to:

1. Take some unstructured medical text (e.g. a clinic note from a doctor’s office visit)
2. Process the medical text using an NLP engine
3. Take the output from the engine and put it into some kind of a data store
4. Deploy some kind of an application using this data.

This guide is to walk you through some of the basics of the steps above to show you how you can do this. We’re going to pretend that we have an EMR that we want to add a ‘smart search’ feature for medical imaging reports to - we’re going to make our EMR a search page where a doctor can search for a disease term and whether it’s present or absent or not.

In this example, our EMR back-end uses MariaDB in which it has a bunch of patient records stored as text. Now for a production environment, you’re going to want to deploy things a little differently than we do here - but what we’re doing here is showing you all of the steps in a way that illustrates how to build this ‘smart search’ application using the NLP API.

Step 1: Get our Reports Ready

Here, we’re going to just use a simple example which will work well with the Advanced Client from the Python SDK - we’re going to just process a simple text report. We’re going to use the sample_ct_report.txt file included with the Python SDK, and there’s not much else to do so let’s move on to Step 2.

Step 2: Create the Output Tables in our Database

This part is a bit more complicated. Here, we’re going to start off by adding some new tables to our emtellipro_example database into which we can store the structured data that we’ve extracted from the processing run. The data that NLP API will output is stored in the API specification, but a short list of what we need to store is:

1. Identified entities (e.g. “acute appendicitis”)

2. Ontology information about those entities:

   • The ontology name
   • Concept IDs
   • Semantic type information - e.g. “symptom” or “disease”

3. Assertions about those entities:

   • Polarity (did the report author say that the entity was present or absent)
   • Uncertainty (did the reprort author indicate that uncertainty about an entity was present, e.g. “possible appendicitis”)

4. Relationships those entities have, such as:

   • The experiencer of the entity, e.g. “The patient’s mother had a hip fracture”

5. Positional information about where the entity was found in the original report, including the sentence position and absolute positon of the text itself.

6. Other useful information, such as:

   • The original report_id where the entity was found
   • The sentence in which the entity was found

In keeping with good database normalization rules, we need to create several new tables and relations between these tables to properly store this information. The tables that we’re going to create are:

1. The entity table, which will store:
   • A unique identifier for the entity
   • Whether the entity was found (via an implicit mention) or assumed
   • A foreign key to the document identifier from which the entity originated
2. The concept table that will store:
   • The fully-specified name of the concept
   • The original ontology concept_id
   • The ontology name from which the concept was identified
3. The document table that will store both a pointer to the original document and the original document itself, along with a unique id for the document.
4. A series of tables about found and assumed entities that describes:
   • The start and end spans of the entity in the original document
   • The concept ID and the ontology from which the concept originated
   • Features about the entity such as the semantic type, polarity, uncertainty, and the section of the report that the entity was identified in
   • The sentence in which the entity was identified in
5. Tables for information on relations in the document such as the relation and experiencerrelation tables that describe relations contained in the document.
6. Tables for information about the sentences in the report such as the sentencelocationspan table that describes the start and end position of all the sentences in the document.

Set up your username and password for your database as environment variables:

username=YOUR-MYSQL-USERNAME
password=YOUR-MYSQL-PASSWORD

In order to create these database tables, we’re going to use the Python SDK’s advanced client (emtellipro-db-client) which can build the database tables we’ll need for storing the NLP API output in a properly normalized fashion:

emtellipro-db-client --database mysql+pymysql://$username:$password@127.0.0.1:3306/emtellipro_example create-db

Step 3: Processing and Storage

Here, we’re going to take the example CT scan report provided with the Python SDK, and we’re going to process it with the NLP API, then store it in the tables that we created in Step 2.:

We recommend that you store the public and private keys in a file for use later. In this example, we ask you to store the key files in your home directory (in practice, they can be stored anywhere).

Store your the API keys as follows:

1. Create a file named access.ak in your home directory.
2. In access.ak, add a line containing only the public key.
3. Create a file named secret.ak in your home directory.
4. In secret.ak, Add a line containing only the private key.

We’re going to use the advanced client (emtellipro-db-client) from the Python SDK to process our JSON file of reports:

emtellipro-db-client \
    --database mysql+pymysql://$username:$password@127.0.0.1:3306/emtellipro_example \
    process \
    --access-key $(cat ~/access.ak) \
    --secret-key $(cat ~/secret.ak) \
    --server https://nlp.emtelligent.com \
    --store-reports \
    example-data/sample_ct_report.txt

If this all works correctly, we should see a message showing us that the processing run completed and the output data was inserted into the new tables in our database that we created in Step 2.

Step 4: Application Deployment

For the final step, we’re going to create and deploy a PHP application that is going to allow a user to find any SNOMED concepts from our processed report that:

• Contain a part of a SNOMED term
• Allow the user to select whether the concept is asserted or negated
• Allow the user to determine if there is uncertainty used in describing this concept

We’re going to make some assumptions here as well:

• The user has a LAMP stack up and running on their server (we’re going to use our emtellipro_example MariaDB database as the ‘M’ in LAMP here)
• The webserver root is in /var/www/html
• apache2 as well as PHP and its proper MySQL extensions must be installed and working properly.

First of all, let’s create an HTML form that will allow our user to search for a SNOMED concept ID:

<!DOCTYPE html>
    <body>
        <h2>Find SNOMED Concepts</h2>
        <h4>Enter a SNOMED Concept Description to search reports for:</h4>
        <FORM enctype="multipart/form-data" action="results.php" method="GET">
            <table border=1>
                <tr><td><b>Enter a Concept Description:</b> <input type="text" name="description" value="lesion"><br /></td></tr>
                <tr><td><b>Polarity:</b><br />
                    <input type="radio" name="polarity" value="asserted" checked>Asserted<br />
                    <input type="radio" name="polarity" value="negated">Negated<br /></td></tr>
                <tr><td><b>Uncertainty:</b><br />
                    <input type="radio" name="uncertainty" value="certain" checked> Certain<br />
                    <input type="radio" name="uncertainty" value="uncertain"> Uncertain<br /></td></tr>
            </table>
              <input type="submit" value="Search" />
        </form>
    </body>
</html>

This is going to create a form that looks like this:

Our next step is to create the PHP script that will search the database for any reports that contain the SNOMED concept that the user has searched for with the polarity and uncertainty assertions that the user has chosen (present/absent and certain/uncertain):

<?php
//*************************************************************************************
// Declare our variables
//*************************************************************************************
// Database Settings
$database_host = "10.100.1.75";
$database_username = "emtellipro";
$database_password = "emtelligent";
$database_name = "emtellipro_example";
$error_message = "Database error; please contact the site administrator.";
// Formatting tags
$bold_start = "<b>";
$bold_end = "</b>";
// Variables the HTML form should have passed to us:
$description = $_GET['description'];
$polarity = $_GET['polarity'];
$uncertainty = $_GET['uncertainty'];
//*************************************************************************************
// main()
//*************************************************************************************
// Connect to the Database
$db = mysqli_connect("$database_host", "$database_username", "$database_password", "$database_name") or die($error_message);
// The SQL Query we're going to use:
$sql_query = "SELECT
    fes.id,
    c.description,
    fes.start AS entity_start,
    fes.end AS entity_end,
    fes.found_entity_id,
    fes.text as anno_text,
    fec.concept_id,
    fec.concept_ontology,
    fe.type_name_umls,
    fe.polarity,
    fe.uncertainty,
    fe.type_name_snomed,
    fe.section_name,
    fel.location_id,
    sls.start AS sentence_start,
    sls.end AS sentence_end,
    e.document_id,
    SUBSTRING(d.text, sls.start, (sls.end-sls.start+1)) AS sentence
FROM
    foundentityspan fes
JOIN
    foundentityconcept fec ON fes.found_entity_id=fec.found_entity_id
JOIN
    foundentity fe ON fe.id=fec.found_entity_id
JOIN
    concept c ON fec.concept_id=c.concept_id
JOIN
    entity e ON fe.id=e.entity_id
JOIN
    foundentitylocation fel ON fe.id=fel.found_entity_id
JOIN
    location l ON fel.location_id=l.id AND (l.type_='sentence')
JOIN
    sentencelocationspan sls ON sls.sentence_location_id=l.location_id
JOIN
    document d ON d.id=e.document_id
WHERE
    fec.concept_ontology='snomed'
AND
    fe.polarity='$polarity'
AND
    fe.uncertainty='$uncertainty'
AND
    description LIKE CONCAT('%', '$description', '%')
GROUP BY
    anno_text, sentence
ORDER BY
    fes.start;
";
// Now, let's set up the output table:
echo "<!DOCTYPE html>\n";
echo "<html>\n";
echo "<body>\n";
echo "<h3>Search Results</h3>\n";
echo "<table border=1>\n";
echo "\t<tr><th>Document ID</th><th>Found Term</th><th>SNOMED Match</th><th>Type</th><th>Concept ID</th><th>Polarity</th><th>Uncertainty</th><th>Section Name</th><th>Source Sentence</th></tr>\n";
// And let's run the SQL query and print the data
$result = mysqli_query($db, $sql_query) or die($error_message);
while ($row = mysqli_fetch_array($result))  {
    $rel_pos_start = $row['entity_start'] - $row['sentence_start']+1;
    $rel_pos_end = $row['entity_end'] - $row['sentence_start']+1;
    echo "\t<tr>";
    echo "<td>" . $row['document_id'] . "</td>";
    echo "<td>" . $row['anno_text'] . "</td>";
    echo "<td>" . $row['description'] . "</td>";
    echo "<td>" . $row['type_name_umls'] . "</td>";
    echo "<td>" . $row['concept_id'] . "</td>";
    echo "<td>" . $row['polarity'] . "</td>";
    echo "<td>" . $row['uncertainty'] . "</td>";
    echo "<td>" . $row['section_name'] . "</td>";
    echo "<td>" . substr_replace(substr_replace($row['sentence'], $bold_end, $rel_pos_end, 0), $bold_start, $rel_pos_start, 0) . "</td>";
    echo "</tr>\n";
}
// And close the table and page:
echo "</table>\n";
echo "</body>\n";
echo "</html>\n";
?>

Assuming we have found a report in our output table that contains the concept and assertions we’re looking for, this will output a table that looks like:

Summary

And that’s pretty much it - you’ve created your first application using the NLP API! Just to recap, we:

• Exported some raw reports from our EMR database as JSON (useful for the Java SDK)
• Added some new tables to our database to store structured medical data
• Processed these reports with the NLP API and stored the results in our new tables
• Created a web-based application to search for SNOMED concepts in our reports with assertions about those concepts