Workflow

Preparing Files for Processing

A file manifest (.csv file) is the entry point for the document processing pipelines. It lists all the files to be processed, providing a path to where they are located within the input bucket. It is also a way for you to pass through metadata that you want to associate with the extracted output. This means you do not need to implement downstream processing steps to merge clinical extraction output with chartdate and other critical file or patient-based metadata.

About this tutorial

This tutorial explains how to create a properly formatted manifest file, and how to upload it to your S3 bucket along with the documents it references. More details about the format and required columns can be found in File Manifest Schema.

We will walk through completing these 4 steps:

  1. Create the csv file
  2. Add a header row
  3. Add rows for each document you want to process
  4. Upload to S3

Step 1: Create the CSV File

Create a new file named manifest.csv using a text editor or spreadsheet application.

Any file name is acceptable (e.g. manifest_batch_1.csv, manifest_dataset_1.csv); however, it must be valid CSV and you must provide the correct name in the job request.

Step 2: Add the Header Row

The first row must contain column names. At minimum, these two columns are required:

  • document, and
  • subject_id

You may add any optional columns. Let’s add subject_name, subject_dob and subject_gender, so that your header row now contains:

document,subject_id,subject_name,subject_dob,subject_gender

Step 3: Add Document Rows

For each PDF document, add a row with the corresponding metadata:

document,subject_id,subject_name,subject_dob,subject_gender
eptrial_pdf/patient_11110001_bundle.pdf,11110001,"Shui Chan","1966-08-01 04:10:48","male"
eptrial_pdf/patient_11110002_bundle.pdf,11110002,"Arabella Gabrielson","1960-05-23 15:04:07","female"

Important: The document column must specify S3 object keys (paths) to your PDF documents. These are the S3 object keys relative to the root of your input bucket. For example, if your files are organized as follows:

s3://my-input-bucket/
├── eptrial_pdf/manifest.csv
├── eptrial_pdf/patient_11110001_bundle.pdf
└── eptrial_pdf/patient_11110002_bundle.pdf

Then, the object keys must be as follows:

eptrial_pdf/patient_11110001_bundle.pdf
eptrial_pdf/patient_11110002_bundle.pdf

Step 4: Upload to S3

Now, we will use the AWS CLI to upload the files your PDF documents and the corresponding manifest.csv to the designated S3 input bucket.

Before uploading, review the Requirements Checklist below to ensure your manifest is properly formatted.

During onboarding, you were provided with a bucket name to use for the input bucket. Make sure to use that one. The Extract API will only read from this bucket.

Using AWS CLI:

# Upload all PDF files and manifest from local directory to S3
aws s3 cp eptrial_pdf/ s3://my-input-bucket/eptrial_pdf/ --recursive

Using AWS CLI with a specific profile:

# If you have multiple AWS profiles configured
aws s3 cp eptrial_pdf/ s3://my-input-bucket/eptrial_pdf/ --recursive --profile my-profile

The examples above demonstrate the use of AWS CLI commands. You may use other methods to upload files to S3, including the AWS Console, AWS SDK, or other S3-compatible tools. For additional AWS CLI options, refer to the AWS CLI S3 documentation.

Validate S3 paths

Ensure the paths in the document column match the actual S3 keys.

Verify your upload:

# List files to confirm they were uploaded correctly
aws s3 ls s3://my-input-bucket/eptrial_pdf/ --recursive

Example S3 bucket contents:

s3://my-input-bucket/
└── eptrial_pdf/
    ├── manifest.csv
    ├── patient_11110001_bundle.pdf
    └── patient_11110002_bundle.pdf

Requirements Checklist

Before uploading your manifest, ensure these requirements are met:

  1. All documents and manifest in the same S3 bucket with matching paths.

    Documents must be accessible from the same S3 bucket where manifest.csv is stored.

  2. All required columns populated.

    Both document and subject_id columns must have values for every row.

  3. Consistent date formats.

    Use ISO Timestamp format for all date fields.

  4. Properly quoted text fields.

    Fields containing commas, quotes, or special characters must be enclosed in double quotes.

Common Pitfalls

PitfallHow to Avoid
Including full S3 URIs or absolute paths in the document columnUse relative paths like folder/file.pdf, not s3://bucket-name/folder/file.pdf or /folder/file.pdf.
Editing CSV in Excel and corrupting date formatsExcel may auto-format dates incorrectly. Use a text editor or ensure dates remain in ISO format (YYYY-MM-DD).
Including spaces or special characters in S3 keys without encodingAvoid spaces in filenames. Use underscores or hyphens: patient_001.pdf instead of patient 001.pdf.
Mixing up the manifest location with document locationsDocuments must be in the same bucket as the manifest. If your manifest is at s3://my-input-bucket/data/manifest.csv, documents should be at s3://my-input-bucket/data/file.pdf (referenced as data/file.pdf).
Inconsistent date formats across rowsPick one format and stick with it. All dates should use the same ISO timestamp pattern throughout the manifest.

Best Practices

  1. Use consistent naming conventions for your document S3 keys
  2. Include chartdate when available for better temporal tracking
  3. Keep subject_ids unique
  4. Avoid spaces in filenames. Use underscores or hyphens: patient_001.pdf instead of patient 001.pdf.
  5. Avoid PHI in filenames. Replace PII (personal identifiers) with generated IDs

Troubleshooting Manifest Validation Errors

For guidance on troubleshooting validation errors, refer to the Troubleshooting Input Validation Errors section.