Advanced Usage

Client Commands

To see the available commands and options on the client

emtellisplit-client --help

emtellisplit-client has a base option —logfile, when specified will create a logfile in the same directory as being run, capturing all print outputs.

emtellisplit-client --logfile <subcommand>

All of the other functionalities of the client are under subcommands. All subcommands also have a help option

emtellisplit-client <subcommand> --help

The client provides some utilities to interact with your cloud object storage(s) in Prep specific ways.

Uploading files into the input bucket

emtellisplit-client upload-file --object-key <cloud_file_name> \
   --file-path <local_source_file_path>

Downloading completed jobs from the output bucket. The output directory must already exist as the subcommand will create a job prefixed directory inside the given directory. Downloading does not delete from the bucket.

emtellisplit-client download-jobs --output <local_output_directory> <job_id>

Processing commands

The processing-related subcommands are:

• process
• process2
• bulk-process
• bulk-process2

These commands have a shared option --merge-jsonl to merge the outputted jsonl’s of a job into a combined jsonl for emtelliPro submission, but this option only does something when —output is also specified.

To submit a cloud object inside your input bucket to the API server.

emtellisplit-client process --output <local_output_directory> --merge-jsonl <cloud_file_name>

Processing parameters

Inside a client config, you can configure the processing steps, and options for those steps for the server to use when processing your files.

The split step is used for splitting a document containing multiple logical subdocuments into its subdocuments.

The ocr step is used for extracting a text document from a non-text document.

The ep step can be used after both split and ocr to create a structured JSON document that can be used as input to emtelliPro.

The fields in the config relevant to processing are shown below.

[processing]
operations = ['split', 'ocr', 'ep']
[processing.split]
model = "concerto2"
splitter_version = "v2"
sections = true
[processing.ocr]
enable_collation = true
[processing.ep]
[processing.ep.ep_rollup]
subject_id = "Derek"
subject_dob = "1996-02-30"

Processing multiple files with different parameters

Because different files may require different processing parameters, and changing the configuration every time may become tedious, there exists a bulk-processing command bulk-process.

This command also takes in the toml configuration, but instead of accepting object keys as arguments, accepts a csv file where each row specifies the object key and any processing parameters which are desired to be overridden.

For the processing parameters that are to be overridden, as some of them are nested inside objects in the toml, specifying a nested field requires using special notation for the csv column name. This notation uses periods '.' to separate keys that are nested inside objects, similar to toml syntax. For example, ep.ep_rollup.subject_name as a column name indicates that for that entry, override or set the field equivalent to config['processing']['ep']['ep_rollup']['subject_name'].

Note: since only processing options can be overridden, client assumes that the csv column name does not contain processing when using the above notation.

Note: a field can only be overwritten if the previous objects are found inside the config. If the fields of processing, ep, and/or ep_rollup are missing from the config, there will be an error when trying to use ep.ep_rollup.subject_name.

Note: boolean fields accept JSON or Python style boolean. Lists of strings are comma separated.

Note: an empty string as value will use the default specified by the toml.

Sample csv

"document", "operations", "ep.ep_rollup.subject_name"
"doc_1.pdf","split,ocr,ep","""Jane, Doe"""
"doc_2.pdf","ocr",
"doc_3.pdf",,"""Joe, Doe"""

Another sample csv

"document", "splits_json", "operations", "ep.subdoc_num_string", "ep.ep_rollup.subject_name"
"doc_1.pdf","splits.json","ocr,ep","0001","""Jane, Doe"""
"doc_2.pdf","splits.json","ocr,ep","0002",
"doc_3.pdf","splits.json","ocr,ep","0003","""Joe, Doe"""

Processing from local files

The processing subcommands process and bulk-process expect input objects to already be uploaded to your cloud storage. There exist variants of these processing subcommands (process2 and bulk-process2) which instead take local file paths, and upload them to your cloud storage using the fully specified file path.

Example

emtellisplit-client process2 --output <local_output_directory> --merge-jsonl ./my_input_files/*.pdf

Resuming the client

To resume waiting (e.g. for when the client disconnects from the server) on one or multiple jobs that have already been submitted

emtellisplit-client resume <statefile>

Retry failed processing jobs

To retry files for jobs that have failed processing, the following command can be run on a statefile

emtellisplit-client retry <statefile>