Quick overview of the SDK

Ready to get started building your own client? This page gives a quick overview to using the main Emtellipro interface for submitting documents for processing and retrieving the results.

See the Building your own client page for more in-depth examples you can copy-paste and run.

Prerequisites

Set up the environment variables used in this guide for access to the NLP API server. You will need to set the following environment variables for the NLP API server and your access and secret keys. If you do not have access to the appropriate values for these variables please contact us.

export EMTELLIPRO_SERVER=''
export EMTELLIPRO_ACCESS_KEY=''
export EMTELLIPRO_SHARED_SECRET=''

Testing the connection

First, ensure you’ve installed the package properly and have connectivity working: Installation

The next step is ensuring that your API keys are set up and working —to make sure you can use your API keys and can connect to the server, the quickest way is to use the provided example client to try and retrieve your user information:

emtellipro-db-client \
    get-user \
    --access-key $EMTELLIPRO_ACCESS_KEY \
    --shared-secret $EMTELLIPRO_SHARED_SECRET \
    --server $EMTELLIPRO_SERVER

If successful and your access is set up, this command will print out your username with a response like:

User with username: test_user

Retrieve your user info

As a first step in building our client, let’s replicate the client example above and try a code example where we connect to the API to test our access keys and retrieve our username. First, we’ll begin by importing the Emtellipro class and instantiating it with our access key and secret key.

>>> import os
>>> import emtellipro
>>> ACCESS_KEY = os.environ['EMTELLIPRO_ACCESS_KEY']
>>> SHARED_SECRET = os.environ['EMTELLIPRO_SHARED_SECRET']
>>> SERVER = os.environ['EMTELLIPRO_SERVER']
>>> client = emtellipro.Emtellipro(
...     SERVER,
...     auth=(ACCESS_KEY, SHARED_SECRET),
... )

Next we’ll retrieve the user info:

>>> user = client.user
>>> print(user)
User with username: user@example.com
>>> print(user.username)
user@example.com

Great - let’s go on to the next step where we’ll build a client for processing some medical text.

Submit a document for processing

Now that we’ve confirmed we can connect and authenticate against the NLP API, let’s submit a simple document for processing. Note that although we’re submitting a single document the API handles a list of documents, so we’ll submit a list containing a single document.

>>> from emtellipro.data import InputDocument
>>> doc = InputDocument(
...     id='12',
...     category='Radiology',
...     subcategory='CT',
...     data='The raw text of the document',
... )
>>> result_future = client.submit([doc])

The ResultFuture instance returned by the submit call keeps track of the task ID returned by the API. It allows you to check on result of the processing using done.

>>> bool(result_future.done())
True

Now that we know that processing of the submitted document is completed, we can retrieve the results. Calling result returns an instance of Result which contains the result from the engine in a raw form, as well as parsed into AnnotatedDocument instances.

>>> result = result_future.result()
>>> result.task_id
'9ff30842202c49bd97c93260a2911117'
>>> result.annotated_docs[0].id
12

Each element in annotated_docs is an instance of AnnotatedDocument, which we can work with by reading its attributes, or if we’re simply looking to dump the raw results returned by the API, we can instead call raw_result.

>>> raw_data = result_future.raw_result()
>>> with open('data.json', 'w') as f:
...     f.write(raw_data)

The regular Result class already provides access to the raw result, but it will also parse the results, so if you’re simply looking to store the unprocessed output from the NLP API calling raw_result will use less memory.

Submit many files for processing

If you’re submitting multiple documents for processing in one batch, it’s important to keep the batch size under the maximum limits of the engine. To help with this, there is batch_documents, which will take in an iterable of input documents and yield batches that will fit within the engine’s limits.

This example also shows how to read files from disk.

>>> import pathlib
>>> from emtellipro.load import readfile
>>> from emtellipro.utils import batch_documents
>>> datapath = pathlib.Path('./documents')
>>> all_docs = []
>>> for filepath in datapath.glob('*.txt'):
...     loaded_file = readfile(
...         filepath,
...         category='clinical',
...         subcategory='generic',
...     )
...     all_docs.extend(loaded_file.docs)
...
>>> results = []
>>> for batch in batch_documents(all_docs):
...     future = client.submit(batch)
...     while not future.done():
...         pass
...     results.append(future.result())

The readfile function returns a LoadedFile which contains the documents included in the file. This will be a single document for text files and PDFs, but other file types may contain multiple documents (such as JSON files).

Storing results in a database

If you with to replicate functionality similar to the database client, you can use Database.

It’s simply instantiated using the same URI you’d use with the database client.

>>> from emtellipro.db import Database
>>> storedb = Database('postgresql://...')

If this is a fresh database, we’ll first need to create the tables by calling init.

>>> storedb.init()

However, if the tables were already created and you only need to upgrade the schema to the latest version, run migrate instead.

>>> storedb.migrate()

Then to store, we first have to begin a transaction and use it to store the results we got above.

>>> input_docs = all_docs
>>> annotated_docs = []
>>> for result in results:
...      annotated_docs.extend(result.annotated_docs)
...
>>> with storedb.begin() as conn:
...     conn.save(input_docs, annotated_docs)
...

Next steps

Now that you have a high level overview of what’s available in the SDK, take a look at the Building your own client page for more in-depth information about building your own client.