Input File Formats Supported By The SDK
The SDK supports multiple file formats from which documents can be read for processing using the NLP API. These file formats are specific to the SDK and are not sent directly to the NLP API.
The readfile function provides the main entry-point for loading files, and the emtellipro-db-client uses that function.
For each of the formats described in this page, one or more file extensions are listed. The file extension is what the file loading code expects files in that format to have as a suffix; it can also be used to override the automatic filetype detection.
Whenever specifying a filetype for loading files, the leading . is optional, and the filetype is case insensitive.
i.e. the following are all equivalent: .txt, TXT, .Txt, .TXT
The descriptions on this page will standardize to lowercase with a leading period for simplicity.
Plaintext
File extension
.txt
The plaintext format is the simplest input format. All it must contain is plain text that will be submitted largely as-is to the NLP API.
The file encoding should be UTF-8, but it can be other encodings. An attempt will be made to load the file as UTF-8, but if that fails, the encoding will be auto-detected using chardet. If the encoding is known, it’s best to use the process --file-encoding option or the file_encoding parameter for readfile because auto-detection is slower and can be inaccurate.
File extension
.pdf
PDF files are similarly handled to plaintext files, in that each PDF is considered to contain a single document that will be submitted as-is to the NLP API. No other processing attempt is made, other than to read the file as bytes and submit it unchanged.
HL7 CCD
File extensions
.ccd.xml, .xml, .ccd
The HL7 CCD format is an XML based format and loading it is configurable. The regular readfile function supports loading it as with the other file types, but a more full-featured parsing module is provided as ccd.
See CCD Features for more details on loading CCD files.
JSON and JSONL
File extensions
.json for JSON, and .jsonl for JSONL
The JSON input format allows for processing plaintext files or PDFs with associated metadata. The JSON document can contain the plaintext/PDF directly inside it or as references to external files.
The difference between JSON and JSONL: Both formats are documented together because the only difference is that JSON contains a list of objects, and JSONL contains one object per line. The objects they both contain must have the same format, it’s just the container that differs.
The specification in JSON Schema format is available here.
JSON files contain a list of JSON documents, with identifier, contents, metadata, and optional source of the document (to track the provenance of the document contents).
JSON Document
Each JSON document within a JSON/JSONL file represents a single document to submit to the NLP API.
It contains the following field:
id
(optional) A string identifier for this document. If present, this must be unique within the file as it is used to log/track failing documents.
If missing, the index of the document within the file will be used.
This ID will not be submitted to the NLP API, unless doc_id_filepath is enabled (in which case the filepath + this ID will be submitted as the document ID for debugging purposes).
category
(optional) A string specifying the document category to use for this document. If unset, the default category will be used.
For category auto-detection, set this to 'auto'. If this is set to null, then the default category will be used (which can itself be set to 'auto' using process --category or readfile’s category parameter).
subcategory
(optional) A string specifying the document subcategory. See category for overriding details.
section_label
(optional) A string specifying the section this document came from; this is sent unchanged to the NLP API.
metadata
(optional) An object with arbitrary keys. The values must be either strings, integers, null, or lists of strings/integers.
The metadata is loaded as-is into the InputDocument objects, and is not sent to the NLP API. It is useful when storing the NLP API output to a database, in which case the database saving code will store metadata into appropriate tables and columns.
See the database schema documentation for how the metadata is used in your schema of choice.
contents
(optional if filepath is set) The contents of the document to submit to the NLP API. This may be either plaintext, or base64 encoded PDF data; the format is specified using the type field.
path
(optional if contents is set) The path to the contents of the document to submit to the NLP API. The format of this document is specified using the type field.
type
(optional) The mime-type of the document contents. The supported types are
application/pdftext/plain
source
(optional) An object containing the source of this document. This is loaded as an InputSource object. This is useful in cases where the document contents came from an image or PDF (using OCR), and you wish to store the original image/PDF, but want to use the extracted text for processing using the NLP API.
See Document Source for details.
Document Source
The document source describes the source of the JSON Document.
It contains 3 fields:
type
(required) The MIME type of the source data. This can be any MIME type, as it is not interpreted by the SDK.
path
(optional) The path of the data source.
data
(required) The base64-encoded contents of the data source.