> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.emtelligent.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.emtelligent.com/_mcp/server. # Getting Started Tutorial This tutorial provides a step-by-step guide to using the Extracton API. We have provided sample files you can use to follow along. Before you start, ensure that you have: * Input and output buckets with the correct permissions * Access and Secret Access keys for these buckets * (Optional) STS Role ARNs for generating STS credentials * An emtelligent-provided API Key * Pipeline name * Sample files for the tutorial (Click to download: tutorial.zip ). **Quick Start Option:** Download tutorial.py to run the complete end-to-end workflow automatically. Simply configure your credentials and execute the script to upload files, submit a job, monitor progress, and collect outputs. **Manual Step-by-Step Option:** Follow the detailed instructions below to understand each API call individually. For more information see [Getting Set Up](/extract/workflow/setup). The key steps in the data flow is outlined in this image: ![Extraction Submission Flow](/_fern-img/a35e13fbdfa2ae9015f61079a9c516d66e57fe84dd46d7c2719f89908143b574.webp) Notice that key actions must be initiated by the customer to transition the job its stages through from start to finish. The actions you must take are: 1. Upload Files to the Input Bucket 2. Post a job request 3. Get job status 4. Collect outputs The next sections below provide detailed explanations of each step, including code snippets and curl commands. > **Warning** > > In the following sections, we use hard-coded API keys/passwords for readability. When implementing this in your own project, it is strongly recommended that you replace these with securely-managed secrets (as per your organizations security policy). ## Upload Files to Input Bucket Sample manifest and PDF files\ tutorial.zip 1. Download the sample files and save them to a local directory e.g. `~/tutorial` 2. Upload the files to your input bucket and note the manifest file's S3 key * **IMPORTANT**: Ensure the document values in your manifest match the exact S3 keys where your PDF files are stored **`upload-example.py`** ```python title="upload-example.py" import boto3 from pathlib import Path # Fill in the credentials to access your s3 bucket credentials = { 'aws_access_key': "access_key", 'aws_secret_key': "secret_key", 'aws_region': "region" } # emtelligent-provided input bucket name input_bucket_name = "your-input-bucket-name" file_path = "~/tutorial" manifest_path = f"{file_path}/manifest.csv" s3_prefix = "tutorial" manifest_s3_key = f"{s3_prefix}/manifest.csv" s3_client = boto3.client( "s3", aws_access_key_id=credentials["aws_access_key"], aws_secret_access_key=credentials["aws_secret_key"], region_name=credentials["aws_region"], ) # Upload manifest s3_client.upload_file(manifest_path, input_bucket_name, manifest_s3_key) # Upload all PDF files from the directory # Ensure the document values in your manifest match the exact S3 keys where your PDF files are stored for pdf_file in Path(file_path).glob("*.pdf"): s3_key = f"{s3_prefix}/{pdf_file.name}" s3_client.upload_file(str(pdf_file), input_bucket_name, s3_key) ``` ### Generate STS Credentials A secure way to provide emtelligent with credentials we need to access files in your input bucket is to use AWS Security Token Service (STS). We recommend that you use STS to generate temporary, limited-privilege security credentials. However, you may also choose use regular AWS credentials: the same secret and access keys you used to upload files to your input. The following explains how to issue a STS credentials using AWS S3 `boto3` sdk, follow these steps: * Get STS Role ARN for input bucket * Assume the role and get the STS credentials **`assume-role-example.py`** ```python title="assume-role-example.py" input_bkt_role_arn = '' # This must be provided by your IT department. sts_client = boto3.client( 'sts', aws_access_key_id=credentials['aws_access_key'], aws_secret_access_key=credentials['aws_secret_key'], region_name=credentials['aws_region'] ) # For testing purposes we set the expiry to 3600s for this tutorial. Increase as needed up to maximum of # 12 hours expiry_s = 3600 # 60min # Assume the role response = sts_client.assume_role( RoleArn=input_bkt_role_arn, RoleSessionName='tutorial-temp-session', DurationSeconds=expiry_s ) sts_creds = response['Credentials'] print(json.dumps({ "AccessKeyId": sts_creds["AccessKeyId"], "SecretAccessKey": sts_creds["SecretAccessKey"], "SessionToken": sts_creds["SessionToken"], "Expiration": str(sts_creds["Expiration"]) }, indent=2)) ``` ## Query for Pipeline Parameters > **Note** > > This step is optional. You can look up the supported parameters in this documentation. For example, for the Core Clinical pipelines, the supported extraction types are documented [here](/extract/extractions/overview). > > However, a discovery endpoint is provided so you can programmatically to find the parameters that a pipeline supports. How to discover a pipeline's parameters: 1. Get the pipeline name for which you have access(e.g. `core_clinical` or as provided by emtelligent support during onboarding) 2. Call GET `/api/v1/pipelines/:pipeline_name/pipeline_params` with: * API key in the `X-API-Key` header * `pipeline_name` in the path parameters ```bash curl -X 'GET' \ 'https://extraction-api.emtelligent.com/api/v1/pipelines//pipeline_params' \ -H 'accept: application/json' \ -H 'X-API-Key: ' ``` **`get-pipeline-params-example.py`** ```python title="get-pipeline-params-example.py" import requests BASE_URL = "https://extraction-api.emtelligent.com/api/v1" # Replace with your emtelligent issued API Key and the provided pipeline name API_KEY = "" pipeline_name = "" def get_pipeline_params(pipeline_name: str) -> dict: response = requests.get( f"{BASE_URL}/pipelines/{pipeline_name}/pipeline_params", headers={ "accept": "application/json", "X-API-Key": API_KEY, }, ) response.raise_for_status() return response.json() params = get_pipeline_params(pipeline_name) print(params) ``` The response will include the pipeline parameter names and values as required by your pipeline configuration, for example: ```json { "required_fields": { "pipeline_parameters": { "extraction_type": { "name": "extraction_type", "schema": { "type": "array", "examples": [ "labs", "vitals", "problems", "procedures", "medication_rxnorm", "family_history", "social_history" ] }, "default": [ "labs", "vitals", "problems", "procedures", "medication_rxnorm", "family_history", "social_history" ], "description": "customer_required: Determines which clinical extractions to perform" } } } } ``` ## Post a Job Request Now we will start a job using the API. Submit a job by calling POST on this API route `/api/v1/jobs`. Provide the following in the request body: * API key in the `X-API-Key` header * AWS STS credentials in the request body * File manifest S3 key * input bucket AWS region * Pipeline name and extraction parameters Below is an example of request body, and how to post the job request using CURL or Python. ### Sample Request body ```json { "aws_region": "region", "manifest_key": "tutorial/manifest.csv", "pipeline_name": "pipeline_name", "user_credentials": { "aws_access_key_id": "key", "aws_secret_access_key": "secret", "expiry": "expiry", "session_token": "token" }, "pipeline_parameters": { "extraction_type": ["labs", "vitals"] } } ``` ### Sample CURL Command ```bash curl -X 'POST' \ 'https://extraction-api.emtelligent.com/api/v1/jobs' \ -H 'accept: application/json' \ -H 'X-API-Key: ' \ -H 'Content-Type: application/json' \ -d '{ "aws_region": "region", "manifest_key": "tutorial/manifest.csv", "pipeline_name": "pipeline_name", "user_credentials": { "aws_access_key_id": "key", "aws_secret_access_key": "secret", "expiry": "expiry", "session_token": "token" }, "pipeline_parameters": { "extraction_type": ["labs", "vitals"] } }' ``` ### Sample Python Function **`submit-job-example.py`** ```python title="submit-job-example.py" BASE_URL = "https://extraction-api.emtelligent.com/api/v1" # Replace with your emtelligent issued API Key and the provided pipeline name API_KEY = "" pipeline_name = "" def submit_job( pipeline_name: str, manifest_key: str, aws_region: str, user_credentials: dict, pipeline_parameters: dict, ) -> dict: response = requests.post( f"{BASE_URL}/jobs", headers={ "accept": "application/json", "X-API-Key": API_KEY, "Content-Type": "application/json", }, json={ "pipeline_name": pipeline_name, "manifest_key": manifest_key, "aws_region": aws_region, "user_credentials": user_credentials, "pipeline_parameters": pipeline_parameters, }, ) response.raise_for_status() return response.json() # For this tutorial, let's extract lab results and vital signs pipeline_params = {"extraction_type": ["labs", "vitals"]} result = submit_job( pipeline_name="pipeline_name", manifest_key="tutorial/manifest.csv", aws_region=credentials["aws_region"], user_credentials={ "aws_access_key_id": sts_creds["AccessKeyId"], "aws_secret_access_key": sts_creds["SecretAccessKey"], "expiry": str(sts_creds["Expiration"]), "session_token": sts_creds["SessionToken"], }, pipeline_parameters=pipeline_params, ) print(result) # save job ID for future steps job_id = result.get("job_id") ``` ### Sample Response ```json { "job_id": "4a8580c4-41a1-4de4-8bff-590ee0522131", "job_name": "pipeline_name_aa9f84d4-b31f-4e05-92d2-8b7274e895fd", "job_status": "queued", "job_details": { "queued": { "pipeline_name": "pipeline_name", "pipeline_params": { "extraction_type": ["labs", "vitals"] } }, "extract": { "queued_at": "2025-08-29T19:45:16.785248Z" } } } ``` ## Get Job Status A pipeline can take a while to complete depending on the number of documents in the manifest and the extractions to be performed. You need to check the status of the job by calling the GET `/api/v1/jobs/{job_id}/status` endpoint, so that you can start downloading the output as soon as the pipeline finishes and the job status is transitioned to `completed`. > **Note** > > You will not be able to collect the outputs via the POST `/api/v1/jobs/:job_id/collect` until the job is completed ### Sample CURL Command ```bash curl -X 'GET' \ 'https://extraction-api.emtelligent.com/api/v1/jobs/4a8580c4-41a1-4de4-8bff-590ee0522131/status' \ -H 'accept: application/json' \ -H 'X-API-Key: ' ``` ### Sample Python Function **`get-job-status-example.py`** ```python title="get-job-status-example.py" import time BASE_URL = "https://extraction-api.emtelligent.com/api/v1" # Replace with your emtelligent issued API Key and the provided pipeline name API_KEY = "" pipeline_name = "" def get_job_status(job_id: str) -> dict: response = requests.get( f"{BASE_URL}/jobs/{job_id}/status", headers={ "accept": "application/json", "X-API-Key": API_KEY, }, ) response.raise_for_status() return response.json() FAILED_STATUSES = {"transfer_failed", "failed", "cancelled"} # These settings assume you are submitting a small job for the tutorial # Choose appropriate polling intervals for larger jobs MAX_WAIT_TIME_S = 3600 # 1 hour max wait time INITIAL_POLL_INTERVAL_S = 5 MAX_POLL_INTERVAL_S = 60 BACKOFF_FACTOR = 1.5 start_time = time.time() poll_interval = INITIAL_POLL_INTERVAL_S while True: status_response = get_job_status(job_id) status = status_response.get("job_status") print(f"Job {job_id} status: {status}") if status == "completed": break if status in FAILED_STATUSES or status == "paused": raise RuntimeError(f"Job {job_id} ended with status: {status}") # Check if we've exceeded max wait time if time.time() - start_time > MAX_WAIT_TIME_S: raise TimeoutError(f"Job {job_id} did not complete within {MAX_WAIT_TIME_S}s") # Sleep with exponential backoff time.sleep(poll_interval) poll_interval = min(poll_interval * BACKOFF_FACTOR, MAX_POLL_INTERVAL_S) ``` ### Sample Response ```json { "job_id": "4a8580c4-41a1-4de4-8bff-590ee0522131", "job_name": "job_name", "job_status": "running", "job_details": { "queued": { "pipeline_name": "pipeline_name", "pipeline_params": { "extraction_type": ["labs", "vitals"] } }, "extract": { "queued_at": "2025-08-29T22:41:16.816604Z" } } } ``` ## Collect Outputs After the job is complete, you can call POST `/api/v1/jobs/:job_id/collect` to transfer final outputs to output bucket. You will need to provide the following: - an API key in the `X-API-Key` header - STS credentials for output bucket in the request body - optionally, S3 prefix in the output bucket where the outputs should be saved ### Sample Request body ```json { "aws_region": "region", "output_prefix": "tutorial_output/", "user_credentials": { "aws_access_key_id": "key", "aws_secret_access_key": "secret", "expiry": "expiry", "session_token": "token" } } ``` ### Sample CURL command ```bash curl -X 'POST' \ 'https://extraction-api.emtelligent.com/api/v1/jobs/4a8580c4-41a1-4de4-8bff-590ee0522131/collect' \ -H 'accept: application/json' \ -H 'X-API-Key: ' \ -H 'Content-Type: application/json' \ -d '{ "aws_region": "region", "output_prefix": "tutorial_output", "user_credentials": { "aws_access_key_id": "key", "aws_secret_access_key": "secret", "expiry": "expiry", "session_token": "token" } }' ``` ### Sample Python function **`collect-job-example.py`** ```python title="collect-job-example.py" BASE_URL = "https://extraction-api.emtelligent.com/api/v1" # Replace with your emtelligent issued API Key and the provided pipeline name API_KEY = "" pipeline_name = "" def collect_job( job_id: str, aws_region: str, output_prefix: str, user_credentials: dict, ) -> dict: response = requests.post( f"{BASE_URL}/jobs/{job_id}/collect", headers={ "accept": "application/json", "X-API-Key": API_KEY, "Content-Type": "application/json", }, json={ "aws_region": aws_region, "output_prefix": output_prefix, "user_credentials": user_credentials, }, ) response.raise_for_status() return response.json() result = collect_job( job_id=job_id, aws_region="region", output_prefix="tutorial_output", user_credentials={ "aws_access_key_id": sts_creds["AccessKeyId"], "aws_secret_access_key": sts_creds["SecretAccessKey"], "expiry": sts_creds["Expiration"], "session_token": sts_creds["SessionToken"], }, ) start_time = time.time() poll_interval = INITIAL_POLL_INTERVAL_S while True: status_response = get_job_status(job_id) status = status_response.get("job_status") print(f"Job {job_id} status: {status}") if status == "transfer_completed": break if status in FAILED_STATUSES or status == "paused": raise RuntimeError(f"Job {job_id} transfer ended with status: {status}") # Check if we've exceeded max wait time if time.time() - start_time > MAX_WAIT_TIME_S: raise TimeoutError(f"Job {job_id} transfer did not complete within {MAX_WAIT_TIME_S}s") # Sleep with exponential backoff time.sleep(poll_interval) poll_interval = min(poll_interval * BACKOFF_FACTOR, MAX_POLL_INTERVAL_S) ``` The final outputs can be found in your output bucket at the prefix you specified: **`list-output-example.py`** ```python title="list-output-example.py" output_bucket_name = "your-output-bucket-name" output_prefix = "tutorial_output" # Generate and use STS credentials for output bucket from assume_role (see the example for the input bucket) s3_client = boto3.client( 's3', aws_access_key_id=sts_creds["AccessKeyId"], aws_secret_access_key=sts_creds["SecretAccessKey"], aws_session_token=sts_creds["SessionToken"], region_name=credentials['aws_region'] ) response = s3_client.list_objects_v2(Bucket=output_bucket_name, Prefix=output_prefix) for obj in response.get("Contents", []): print(obj["Key"]) ``` ## Error Handling and Troubleshooting Tips These are the common errors you may encounter: * **UNAUTHORIZED:** The provided AWS credentials are invalid or have expired * **INVALID_INPUT:** The pipeline name does not match any existing pipelines * **INVALID_DATA:** Input data validation against the manifest has failed; one or more data contract requirements were not met > **Note** > > When you received **INVALID_INPUT** or **INVALID_DATA** error, the job is failed preemptively. More information is provided in the response, under job details. Please correct the issue and submit a new job.