Supported data types
Ingestion is where input files become datapoints. When a project is defined, its data folder is read according to the spec (poq.toml), and every validator sees only what ingestion produces. This page describes the file types the ingestion process accepts and what each one turns into.
Supported file types
Ingestion recognizes four source types. Each covers one or more underlying file types and has its own rule for what becomes a datapoint:
| Source type | File types | What becomes a datapoint |
|---|---|---|
| CSV | .csv | Each row |
| JSON | .json | Each file, or each element of an unnested array |
| File collection | .jpg, .pdf, and other binary types | Each file, unparsed |
| Markdown | .md | Each section, split on a heading |
CSV (.csv)
One row per item, with an optional header row. Columns and formatting are detected automatically.
| Case | Supported? |
|---|---|
A single file such as labels.csv, with a header row and one row per item | Yes |
Combining multiple .csv files into one source | No — each CSV source reads exactly one file |
| Unusual or inconsistent formatting, such as odd quoting or mixed encodings | No — rows must parse deterministically |
JSON (.json)
Each JSON file becomes one datapoint, with fields read from its object keys — including nested keys, addressed with a dotted path such as rubric.severity. If a file instead wraps an array of items under a key, ingestion unnests that array into one datapoint per element.
| Case | Supported? |
|---|---|
One datapoint per file, for example doc-summary-001.json, doc-summary-002.json | Yes |
| A file holding an array of items — unnested into one datapoint per element | Yes |
An array nested under a key, such as report.doc-summaries | Yes |
Fields read from nested objects, such as rubric.severity | Yes |
| More than one array unnested per source, or an array nested inside another array | No — flatten the inner array (for example, to a CSV) first |
Binary or opaque files (.jpg, .pdf, ...)
The file_collection upload type ingests a folder of binary files, such as images and PDFs. File contents are not parsed, so one file is always one datapoint.
| Case | Supported? |
|---|---|
A folder of images, PDFs, videos, or other binary types, matched by a pattern such as images/*.jpg | Yes |
Mixed file types in one pattern, such as images/*.{jpg,png} | Yes |
| Pointing at a single file | No — this source type is for folders containing many files |
Markdown files (.md)
A long document divided into separate sections wherever a chosen heading appears.
| Case | Supported? |
|---|---|
One or more text or report files, divided wherever a heading such as ## Doc-summary 1 appears | Yes |
| Pulling a few extra labeled facts out of each section | Yes |
| Dividing sections into smaller pieces again | No — only one level of division is supported |
Limits
| Limit | Value |
|---|---|
| Maximum file size | 5 GB |
| Datapoints per project | 1,000,000 |
| File paths | Must stay inside the project's own folder |
If your data comes in a format ingestion does not yet understand, contact the Sapien team.
See also
| Page | Description |
|---|---|
| Define your spec | How poq.md and poq.toml work together |
poq.toml Specification | Field-level reference for [[ingestion.sources]], [[ingestion.joins]], and [ingestion.fields] |
| Quickstart | Creating a first project |