> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.emtelligent.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.emtelligent.com/_mcp/server. # Output Reference Depending on the pipeline that is run for clinical extractions, the extracted output is provided in CSV and JSONL formats. In the table that follows, you will find a list of files that may be produced. | File Name | Format | Description | | -------------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------- | | `job_completion_report.jsonl` | JSONL | [Job Completion Report](#job-completion-report): Summary of processing status and extraction counts per document | | `job_details.json` | JSON | [Job Details](#job-details): Information about a specific run of a pipeline | | `document_metadata.jsonl` | JSONL & CSV | [Document Metadata](#document-metadata): Detailed information about each subdocument including OCR text and metadata | | `{date-time}_{extraction-type}_output.csv` | CSV | See [Clinical Extractions](/extract/extractions/overview) for more details on their respective outputs | | `{date-time}_{extraction-type}_output.jsonl` | JSONL | See [Clinical Extractions](/extract/extractions/overview) for more details on their respective outputs | Standard Output Files ## Job Completion Report The `job_completion_report.jsonl` file provides a comprehensive summary of document processing and extraction results for each document in your manifest. The report is a **JSONL** (JSON Lines) file where each line is a valid JSON object representing one document from your manifest CSV. ### File Structure Each line in the `job_completion_report.jsonl` corresponds to one row in your input manifest CSV file. The rows appear in the same order as they were listed in the manifest (excluding the header row). The `manifest_index` field provides a zero-based index that maps back to the original manifest row: ```json { "document": "eptrial_pdf/patient_11110001_bundle.pdf", "subject_id": "11110001", "subject_dob": "1966-08-01 04:10:48", "subject_gender": "male", "manifest_index": 0, "processing_summary": { "status": "ok", "pct_complete": 100.0, "info": "", "page_total": 17 }, "extraction_summary": { "labs": {"extracted": 16}, "vitals": {"extracted": 13}, "medication_rxnorm": {"extracted": 19}, "problems": {"extracted": 30}, "procedures": {"extracted": 29}, "family_history": {"extracted": 1}, "social_history": {"extracted": 1} } } ``` ### Field Descriptions #### Manifest Metadata Fields Any recognized metadata included in the manifest.csv will be added to the job completion summary. For a list of accepted manifest columns, see [File Manifest Schema](/extract/reference/input#file-manifest-schema). > **Note** > > Unknown columns from your manifest CSV will not appear in the report ### Processing Summary Processing refers to document processing steps, document splitting, and medical NLP processing. The `processing_summary` object provides information about this stage of the pipeline: * **status** (string): Processing status for this document * `"ok"` - Processing completed successfully for all subdocuments * `"failed"` - Document could not be split, or all subdocuments failed processing * `"skipped"` - Document was skipped due to failure during the splitting process * **pct_complete** (number): Percentage of subdocuments successfully processed (0-100) * `100.0` indicates all subdocuments processed successfully * Values less than 100 indicate partial processing failures * **info** (string): Detailed information about processing issues * Empty string when processing was successful * Examples of failure messages: * `"Document could not be split"` - Failure occurred during splitting step * `"2 subdocs could not be processed: pages(5,6,7,8), pages(1,2,3)"` - Specific subdocuments failed to process * **page_total** (number): Total number of pages in the source document ### Extraction Summary Extraction refers to retrieving clinical data from the PDF that occurs during document processing. The `extraction_summary` object contains extraction counts for each extraction type that was executed. The extraction types present in this object depend on which extractions were configured to run via the `extraction_type` pipeline parameters. #### Each Extraction Type Contains * **extracted** (number): The total number of extracted items/records for this extraction type for this specific document * This count represents the number of rows written to the corresponding extraction output CSV file * A value of `0` means no items were extracted for this document ### Example Interpretations #### Example 1: Successful Processing ```json { "document": "patient_001.pdf", "subject_id": "001", "manifest_index": 0, "processing_summary": { "status": "ok", "pct_complete": 100.0, "info": "", "page_total": 15 }, "extraction_summary": { "labs": {"extracted": 8}, "vitals": {"extracted": 5} } } ``` **Interpretation**: The document was processed successfully. All 15 pages were processed. The extraction found 8 lab results and 5 vital sign measurements. #### Example 2: Partial Processing Failure ```json { "document": "patient_002.pdf", "subject_id": "002", "manifest_index": 1, "processing_summary": { "status": "failed", "pct_complete": 95.0, "info": "1 subdocs out of 20 could not be processed: pages(82,83,84,85,86)", "page_total": 120 }, "extraction_summary": { "labs": {"extracted": 24}, "vitals": {"extracted": 18} } } ``` **Interpretation**: The document had 120 pages split into 20 subdocuments. One subdocument (pages 82-86) failed during processing, while 19 subdocuments succeeded (19/20 = 95.0%). Despite the partial failure, significant data was extracted: 24 lab results and 18 vital signs from the successfully processed subdocuments. > **Note** > > The `pct_complete` field represents the percentage of subdocuments that were successfully processed, calculated as: (number of succeeded subdocuments / total subdocuments) \* 100. ## Job Details The `job_details.json` file provides technical metadata about the job run, including timing information, page counts, and error details if the job failed. ### File Structure #### Top-Level Fields * **queued** (object): Information about how the job was queued and what pipeline was executed * **extract** (object): Information about the extraction/processing phase execution * **billable_doc_count** (number): Total number of billable documents in the manifest (only computed on successful runs) * **manifest_key** (string): The S3 key path to the input manifest CSV file ### Field Descriptions #### Queued Object Contains information about the pipeline that was executed: * **pipeline_name** (string): Name of the pipeline that was executed * **pipeline_params** (object): Parameters that were passed to the pipeline * **extraction_type** (array): List of extraction types that were configured to run * May contain other pipeline-specific parameters #### Extract Object Contains information about the extraction/processing phase: **For Successful Runs:** * **end_time** (string): ISO 8601 timestamp indicating when the pipeline completed **For Failed Runs:** * **end_time** (string): ISO 8601 timestamp indicating when the pipeline failed * **exception** (object): Details about the error that caused the failure * **exception_name** (string): The type/class name of the exception * **exception_desc** (string): Human-readable description of the error See [Troubleshooting Errors](/extract/reference/troubleshooting) for detailed information about each exception type and how to resolve them. ### Examples #### Example 1: Successful Job Run ```json { "queued": { "pipeline_name": "core_clinical", "pipeline_params": { "extraction_type": [] } }, "extract": { "end_time": "2024-04-28T18:07:08.927597Z" }, "billable_doc_count": 10, "manifest_key": "eptrial_pdf/manifest.csv" } ``` **Interpretation**: * The `core_clinical` pipeline was executed * The job completed successfully at `2024-04-28T18:07:08.927597Z` * A total of 10 billable documents were processed * The manifest was located at `eptrial_pdf/manifest.csv` * Individual page counts are provided for each document #### Example 2: Failed Job Run (Validation Error) ```json { "queued": { "pipeline_name": "core_clinical", "pipeline_params": { "extraction_type": [] } }, "extract": { "end_time": "2024-03-12T19:50:58.587014Z", "exception": { "exception_desc": "Manifest file size (50.0MB) exceeds maximum limit (20.0MB)", "exception_name": "InvalidManifestException" } } } ``` **Interpretation**: * The `core_clinical` pipeline was executed * The job failed at `2024-03-12T19:50:58.587014Z` during manifest validation * The failure was caused by an `InvalidManifestException` * The manifest file was too large (50.0MB vs 20.0MB limit) * No `billable_doc_count` field present because processing never started For guidance on resolving this and other errors, see [Troubleshooting Errors](/extract/reference/troubleshooting). ### Differences from job_completion_report.jsonl The `job_details.json` and `job_completion_report.jsonl` files serve different purposes: **job_details.json** (this file): * Single JSON object with job-level metadata * Focuses on execution timing and infrastructure details * Contains page counts and manifest location * Provides exception details for failed jobs **job_completion_report.jsonl**: * JSONL file with one entry per manifest row/document * Focuses on document processing results * Contains extraction counts and processing status per document * Includes patient/subject metadata * Detailed, per-document information ### Notes * The `extraction_type` field will be an empty array if the user did not specify any extraction parameters. In this case, all extraction types are performed by default. ## Document Metadata The document metadata output files provide detailed information about each subdocument extracted during the document splitting stage, including the full OCR text, metadata, and document classification. **Key Concept**: Each source PDF document in your manifest is split into multiple subdocuments (e.g., individual reports, lab results, discharge summaries). The document metadata output contains one entry for each subdocument, not one entry per source PDF. ### File Formats Two files are generated with identical content in different formats: #### JSONL Format ```json {timestamp}_document_metadata_output.jsonl ``` **Structure**: JSON Lines format where each line is a complete JSON object representing one subdocument. #### CSV Format ```json {timestamp}_document_metadata_output.csv ``` **Structure**: Standard comma-delimited CSV with headers. **Note**: In the CSV format, newlines and tabs in the `text` field are replaced with spaces for proper CSV formatting. ### Field Descriptions Each record (line in JSONL or row in CSV) represents one subdocument and contains the following fields: #### Core Identification Fields * **filename** (string): The original PDF filename from the manifest * Example: `"patient_11110003_bundle.pdf"` * This is the source document that was split into subdocuments * **manifest_index** (integer): Zero-based index of the source document in the manifest * Example: `2` (corresponds to the 3rd row in the manifest CSV, excluding header) * Matches the `manifest_index` field in `job_completion_report.jsonl` * **subdoc_id** (string): Unique subdocument identifier * Format: `"{filename}-{subdoc_index}"` * Example: `"patient_11110003_bundle.pdf-0002"` * Uniquely identifies a subdocument that was split from the source PDF document #### Content Fields * **text** (string or null): The full OCR text extracted from the subdocument * Contains all text content from the subdocument pages * In JSONL format: preserves original formatting (newlines, tabs, spacing) * In CSV format: newlines and tabs replaced with spaces * May be `null` if OCR failed or no text was extracted #### Metadata Fields These fields provide additional context about the subdocument. They may be `null` if the information was not available or could not be extracted. **Important - Manifest Precedence**: If any of these metadata fields (`chartdate`, `original_category`, `institution`) were provided in your input manifest CSV, those manifest values will take precedence over values extracted from the document. This allows you to provide authoritative metadata that overrides automatic extraction. * **chartdate** (string or null): The clinical date associated with the subdocument * Format: `"YYYY-MM-DD"` (ISO 8601 date format) * Example: `"2022-02-22"` * Represents when the clinical event occurred (e.g., lab draw date, imaging date, visit date) * **original_category** (string or null): The high-level clinical category classification * Example: `"Cardiology"`, `"Laboratory"`, `"Radiology"` * Indicates the general type of clinical document * **description** (string or null): A more specific description of the document type * Example: `"ECG Report"`, `"Echocardiogram"`, `"Chest X-Ray"` * Provides finer-grained classification than `original_category` * Assigned by the document classification model during document splitting * **institution** (string or null): The healthcare institution or facility name * Example: `"City Hospital"`, `"General Heart Hospital"` ### Notes #### Field Nullability All metadata fields (`text`, `chartdate`, `original_category`, `description`, `institution`) may be `null` if: * The information could not be extracted from the document * OCR failed for that subdocument * The classification model could not determine the value * If a field is `null` in JSONL, it may appear as an empty cell in CSV #### Text Formatting Differences * **JSONL format**: Preserves original whitespace, newlines (`\n`), and tabs (`\t`) * **CSV format**: Newlines and tabs replaced with single spaces for CSV compatibility