Supported data types

Ingestion is the single pipeline that turns your uploaded files into datapoints. Whatever you bring (a CSV of labels, a folder of JSON, a bundle of PDFs, or a long Markdown report), ingestion reads it, normalizes it into one canonical format, and produces datapoints from that canonical form. Every supported file type is just a different on-ramp to the same pipeline, not a separate, one-off behavior.

The canonical format

Under the hood, ingestion treats every source as a temporary table:

• Rows: one row per item in the source (a CSV line, a JSON file, a Markdown section).
• Columns: the fields inside each item (a CSV header, a JSON key, a captured value).

When a project uses more than one source, joins line those tables up into one wider canonical table before anything else happens, and [ingestion.fields] selects which columns are kept and what they are named. Once ingestion finishes, the temporary tables are discarded and only the canonical rows remain.

The rule that ties everything together:

One canonical row = one datapoint.

Validators never see your original files. They see the datapoints produced from the canonical table, and every downstream section of the spec (evidence, rubric, consensus) operates on that shape alone. This is why the choice of file type matters less than it first appears: CSV, JSON, file_collection, and Markdown all resolve to the same rows-and-columns form, and the sections below just describe how each one gets there.

Supported file types

Ingestion recognizes four source types. Each is a different way of feeding the canonical format, and each has its own rule for how a source becomes canonical rows:

CSV (.csv)

The most direct path into the canonical format: a CSV is already rows and columns. One row per item, with an optional header row supplying the column names. Columns and formatting are detected automatically, and each row becomes one canonical row unchanged.

JSON (.json)

Each JSON file becomes one canonical row, with columns read from its object keys, including nested keys, addressed with a dotted path such as rubric.severity. If a file instead wraps an array of items under a key, ingestion unnests that array into one row per element, so a single file can expand into many canonical rows.

Binary or opaque files (.jpg, .pdf, ...)

The file_collection upload type ingests a folder of binary files, such as images and PDFs. File contents are not parsed, so each file becomes exactly one canonical row whose column is a reference to the file itself. It is the same rows-and-columns shape as any other source, just with the file as the value rather than parsed text.

Markdown files (.md)

Markdown reaches the canonical format the same way every other source does. It is simply split first. A long document is divided into separate sections wherever a chosen heading appears, and each section becomes one canonical row, exactly like a CSV line or a JSON file. The section splits are not a special case: they are just the step that turns one .md document into the rows the pipeline expects. The section's text lands in a body column, and a regex pulls any repeated header parts (an id, a title) into their own columns. See markdown_split for the full field reference.

Limits

If your data comes in a format ingestion does not yet understand, contact the Sapien team.

See also