Output Reference

Depending on the pipeline that is run for clinical extractions, the extracted output is provided in CSV and JSONL formats.

In the table that follows, you will find a list of files that may be produced.

Standard Output Files

Job Completion Report

The job_completion_report.jsonl file provides a comprehensive summary of document processing and extraction results for each document in your manifest. The report is a JSONL (JSON Lines) file where each line is a valid JSON object representing one document from your manifest CSV.

File Structure

Each line in the job_completion_report.jsonl corresponds to one row in your input manifest CSV file. The rows appear in the same order as they were listed in the manifest (excluding the header row).

The manifest_index field provides a zero-based index that maps back to the original manifest row:

{
  "document": "eptrial_pdf/patient_11110001_bundle.pdf",
  "subject_id": "11110001",
  "subject_dob": "1966-08-01 04:10:48",
  "subject_gender": "male",
  "manifest_index": 0,
  "processing_summary": {
    "status": "ok",
    "pct_complete": 100.0,
    "info": "",
    "page_total": 17
  },
  "extraction_summary": {
    "labs": {"extracted": 16},
    "vitals": {"extracted": 13},
    "medication_rxnorm": {"extracted": 19},
    "problems": {"extracted": 30},
    "procedures": {"extracted": 29},
    "family_history": {"extracted": 1},
    "social_history": {"extracted": 1}
  }
}

Field Descriptions

Manifest Metadata Fields

Any recognized metadata included in the manifest.csv will be added to the job completion summary. For a list of accepted manifest columns, see File Manifest Schema.

Processing Summary

Processing refers to document processing steps, document splitting, and medical NLP processing. The processing_summary object provides information about this stage of the pipeline:

• status (string): Processing status for this document
  • "ok" - Processing completed successfully for all subdocuments
  • "failed" - Document could not be split, or all subdocuments failed processing
  • "skipped" - Document was skipped due to failure during the splitting process
• pct_complete (number): Percentage of subdocuments successfully processed (0-100)
  • 100.0 indicates all subdocuments processed successfully
  • Values less than 100 indicate partial processing failures
• info (string): Detailed information about processing issues
  • Empty string when processing was successful
  • Examples of failure messages:
    • "Document could not be split" - Failure occurred during splitting step
    • "2 subdocs could not be processed: pages(5,6,7,8), pages(1,2,3)" - Specific subdocuments failed to process
• page_total (number): Total number of pages in the source document

Extraction Summary

Extraction refers to retrieving clinical data from the PDF that occurs during document processing. The extraction_summary object contains extraction counts for each extraction type that was executed.

The extraction types present in this object depend on which extractions were configured to run via the extraction_type pipeline parameters.

Each Extraction Type Contains

• extracted (number): The total number of extracted items/records for this extraction type for this specific document
  • This count represents the number of rows written to the corresponding extraction output CSV file
  • A value of 0 means no items were extracted for this document

Example Interpretations

Example 1: Successful Processing

{
  "document": "patient_001.pdf",
  "subject_id": "001",
  "manifest_index": 0,
  "processing_summary": {
    "status": "ok",
    "pct_complete": 100.0,
    "info": "",
    "page_total": 15
  },
  "extraction_summary": {
    "labs": {"extracted": 8},
    "vitals": {"extracted": 5}
  }
}

Interpretation: The document was processed successfully. All 15 pages were processed. The extraction found 8 lab results and 5 vital sign measurements.

Example 2: Partial Processing Failure

{
  "document": "patient_002.pdf",
  "subject_id": "002",
  "manifest_index": 1,
  "processing_summary": {
    "status": "failed",
    "pct_complete": 95.0,
    "info": "1 subdocs out of 20 could not be processed: pages(82,83,84,85,86)",
    "page_total": 120
  },
  "extraction_summary": {
    "labs": {"extracted": 24},
    "vitals": {"extracted": 18}
  }
}

Interpretation: The document had 120 pages split into 20 subdocuments. One subdocument (pages 82-86) failed during processing, while 19 subdocuments succeeded (19/20 = 95.0%). Despite the partial failure, significant data was extracted: 24 lab results and 18 vital signs from the successfully processed subdocuments.

Job Details

The job_details.json file provides technical metadata about the job run, including timing information, page counts, and error details if the job failed.

File Structure

Top-Level Fields

• queued (object): Information about how the job was queued and what pipeline was executed
• extract (object): Information about the extraction/processing phase execution
• billable_doc_count (number): Total number of billable documents in the manifest (only computed on successful runs)
• manifest_key (string): The S3 key path to the input manifest CSV file

Field Descriptions

Queued Object

Contains information about the pipeline that was executed:

• pipeline_name (string): Name of the pipeline that was executed
• pipeline_params (object): Parameters that were passed to the pipeline
  • extraction_type (array): List of extraction types that were configured to run
  • May contain other pipeline-specific parameters

Extract Object

Contains information about the extraction/processing phase:

For Successful Runs:

• end_time (string): ISO 8601 timestamp indicating when the pipeline completed

For Failed Runs:

• end_time (string): ISO 8601 timestamp indicating when the pipeline failed

• exception (object): Details about the error that caused the failure

  • exception_name (string): The type/class name of the exception
  • exception_desc (string): Human-readable description of the error

  See Troubleshooting Errors for detailed information about each exception type and how to resolve them.

Examples

Example 1: Successful Job Run

{
  "queued": {
    "pipeline_name": "core_clinical",
    "pipeline_params": {
      "extraction_type": []
    }
  },
  "extract": {
    "end_time": "2024-04-28T18:07:08.927597Z"
  },
  "billable_doc_count": 10,
  "manifest_key": "eptrial_pdf/manifest.csv"
}

Interpretation:

• The core_clinical pipeline was executed
• The job completed successfully at 2024-04-28T18:07:08.927597Z
• A total of 10 billable documents were processed
• The manifest was located at eptrial_pdf/manifest.csv
• Individual page counts are provided for each document

Example 2: Failed Job Run (Validation Error)

{
  "queued": {
    "pipeline_name": "core_clinical",
    "pipeline_params": {
      "extraction_type": []
    }
  },
  "extract": {
    "end_time": "2024-03-12T19:50:58.587014Z",
    "exception": {
      "exception_desc": "Manifest file size (50.0MB) exceeds maximum limit (20.0MB)",
      "exception_name": "InvalidManifestException"
    }
  }
}

Interpretation:

• The core_clinical pipeline was executed
• The job failed at 2024-03-12T19:50:58.587014Z during manifest validation
• The failure was caused by an InvalidManifestException
• The manifest file was too large (50.0MB vs 20.0MB limit)
• No billable_doc_count field present because processing never started

For guidance on resolving this and other errors, see Troubleshooting Errors.

Differences from job_completion_report.jsonl

The job_details.json and job_completion_report.jsonl files serve different purposes:

job_details.json (this file):

• Single JSON object with job-level metadata
• Focuses on execution timing and infrastructure details
• Contains page counts and manifest location
• Provides exception details for failed jobs

job_completion_report.jsonl:

• JSONL file with one entry per manifest row/document
• Focuses on document processing results
• Contains extraction counts and processing status per document
• Includes patient/subject metadata
• Detailed, per-document information

Notes

• The extraction_type field will be an empty array if the user did not specify any extraction parameters. In this case, all extraction types are performed by default.

Document Metadata

The document metadata output files provide detailed information about each subdocument extracted during the document splitting stage, including the full OCR text, metadata, and document classification.

Key Concept: Each source PDF document in your manifest is split into multiple subdocuments (e.g., individual reports, lab results, discharge summaries). The document metadata output contains one entry for each subdocument, not one entry per source PDF.

File Formats

Two files are generated with identical content in different formats:

JSONL Format

{timestamp}_document_metadata_output.jsonl

Structure: JSON Lines format where each line is a complete JSON object representing one subdocument.

CSV Format

{timestamp}_document_metadata_output.csv

Structure: Standard comma-delimited CSV with headers.

Note: In the CSV format, newlines and tabs in the text field are replaced with spaces for proper CSV formatting.

Field Descriptions

Each record (line in JSONL or row in CSV) represents one subdocument and contains the following fields:

Core Identification Fields

• filename (string): The original PDF filename from the manifest
  • Example: "patient_11110003_bundle.pdf"
  • This is the source document that was split into subdocuments
• manifest_index (integer): Zero-based index of the source document in the manifest
  • Example: 2 (corresponds to the 3rd row in the manifest CSV, excluding header)
  • Matches the manifest_index field in job_completion_report.jsonl
• subdoc_id (string): Unique subdocument identifier
  • Format: "{filename}-{subdoc_index}"
  • Example: "patient_11110003_bundle.pdf-0002"
  • Uniquely identifies a subdocument that was split from the source PDF document

Content Fields

• text (string or null): The full OCR text extracted from the subdocument
  • Contains all text content from the subdocument pages
  • In JSONL format: preserves original formatting (newlines, tabs, spacing)
  • In CSV format: newlines and tabs replaced with spaces
  • May be null if OCR failed or no text was extracted

Metadata Fields

These fields provide additional context about the subdocument. They may be null if the information was not available or could not be extracted.

Important - Manifest Precedence: If any of these metadata fields (chartdate, original_category, institution) were provided in your input manifest CSV, those manifest values will take precedence over values extracted from the document. This allows you to provide authoritative metadata that overrides automatic extraction.

• chartdate (string or null): The clinical date associated with the subdocument
  • Format: "YYYY-MM-DD" (ISO 8601 date format)
  • Example: "2022-02-22"
  • Represents when the clinical event occurred (e.g., lab draw date, imaging date, visit date)
• original_category (string or null): The high-level clinical category classification
  • Example: "Cardiology", "Laboratory", "Radiology"
  • Indicates the general type of clinical document
• description (string or null): A more specific description of the document type
  • Example: "ECG Report", "Echocardiogram", "Chest X-Ray"
  • Provides finer-grained classification than original_category
  • Assigned by the document classification model during document splitting
• institution (string or null): The healthcare institution or facility name
  • Example: "City Hospital", "General Heart Hospital"

Notes

Field Nullability

All metadata fields (text, chartdate, original_category, description, institution) may be null if:

• The information could not be extracted from the document
• OCR failed for that subdocument
• The classification model could not determine the value
• If a field is null in JSONL, it may appear as an empty cell in CSV

Text Formatting Differences

• JSONL format: Preserves original whitespace, newlines (\n), and tabs (\t)
• CSV format: Newlines and tabs replaced with single spaces for CSV compatibility