Most text datasets for AI teams are delivered either as JSON Lines, with one JSON record per line, or as Apache Parquet, a compressed columnar format with typed fields. CSV is best kept for small, flat tables such as manifests, and large collections of images, audio or video are often packaged as WebDataset tar shards. The right choice depends on the shape of the content, its size and how the recipient will read it.

Delivery format matters more than it looks. A well-prepared archive that arrives as a single enormous file, or as millions of loose files with no index, can cost the recipient days before they read a word of it. Format is the last step in making an archive AI-ready, and the one the recipient meets first.

This guide compares the main formats, shows how to design a record for a document, and covers file sizing, compression, checksums and the conventions the Hugging Face Hub documents for large datasets.

Key takeaways

  • JSON Lines suits text records with nested metadata at small to moderate scale, while Parquet is the better default for large datasets because it is compressed, typed and fast to filter.
  • CSV has no single formal specification, so reserve it for flat tables such as manifests and keep long text out of it.
  • WebDataset tar shards suit large image, audio and video collections that will be streamed into training, ideally alongside a Parquet index of the metadata.
  • Design one record per meaningful unit, with a stable ID, the text, descriptive metadata and provenance, and keep each field's type identical across every record.
  • Split data into files of a few hundred megabytes to around a gigabyte, and ship a manifest with record counts and SHA-256 checksums for every file.

The main formats at a glance

Format Structure Strengths Limitations Typical use
JSON Lines (.jsonl) One JSON value per line Nested fields, streamable, readable in any text editor Larger on disk; no enforced schema; not recommended beyond several gigabytes Text records with rich metadata; fine-tuning and evaluation sets
CSV / TSV Flat rows and columns Opens almost anywhere No types or nesting; long text with quotes and line breaks is fragile Manifests and simple metadata tables
Apache Parquet (.parquet) Columnar, typed, compressed Efficient storage; read only the columns needed; skip irrelevant row groups Not human-readable; needs a library Large text corpora and metadata at scale; the Hub's recommended default
WebDataset (.tar shards) Tar archives of files grouped by shared name Fast sequential streaming of media in native formats Poor for random access, filtering and analytics Large image, audio and video collections
Raw files plus an index Original files in folders with a metadata table Easy access to individual files Many small files are slow to move and list Small or medium media collections

JSON Lines and CSV

JSON Lines

The JSON Lines convention has three rules: the file is UTF-8 encoded without a byte order mark, each line is a valid JSON value, and lines are separated by a newline character. A newline after the last record is strongly recommended. The usual extension is .jsonl, and the format's documentation recommends stream compressors such as gzip or bzip2, giving .jsonl.gz or .jsonl.bz2. A media type, application/jsonl, has been suggested but is not yet standardised.

Because every line is a complete record, JSON Lines files can be streamed, split, concatenated and appended without parsing the whole file, and nested metadata sits naturally inside each record. Hugging Face's guidance is to prefer JSON Lines over CSV for nested data, while noting that both are not recommended for data larger than several gigabytes. Here is a single illustrative record for a newspaper article; in a real file it occupies exactly one line:

{"id": "examplepaper-1987-03-14-p007-a02", "title": "Harbour plan approved after late-night vote", "text": "Councillors approved the harbour redevelopment plan by a single vote...", "published": "1987-03-14", "language": "en", "section": "News", "labels": {"topics": ["local government", "transport"], "topic_confidence": [0.94, 0.81]}, "source": {"file": "scans/1987/03/14/page-007.tif", "page": 7, "edition": "final"}, "processing": {"ocr_tool": "tesseract 5.3.0", "pipeline_version": "2026.11", "human_reviewed": false}, "duplicate_cluster": null, "split": "train"}

The commonest mistake is pretty-printing: a record spread over several indented lines is valid JSON but breaks JSON Lines, because each line must be a complete value.

CSV and TSV

CSV looks simple, but RFC 4180, published in 2005, is an informational document rather than a standard. It describes the format "that seems to be followed by most implementations", because no formal specification existed, and it notes considerable differences between implementations. Under RFC 4180, records end with a carriage return and line feed, a header row is optional, fields containing commas, double quotes or line breaks should be enclosed in double quotes, and a double quote inside such a field is escaped by doubling it.

Those rules are exactly where long text goes wrong. Article bodies contain commas, quotation marks and line breaks, and one writer or reader that handles quoting differently will shift every subsequent column. Opening a CSV in a spreadsheet and saving it again can also silently reformat dates and strip leading zeros from identifiers. Tab-separated files avoid the comma problem but not the others. Use CSV or TSV for flat, short-valued tables, such as a file manifest or a list of titles and date ranges, and keep text in JSON Lines or Parquet.

Apache Parquet

Parquet is a columnar format. As Hugging Face's dataset viewer documentation explains, a Parquet file is divided into row groups; within each row group, data is stored column by column, each column is compressed separately, and statistics such as minimum and maximum values and null counts are recorded. That structure means a reader can load only the columns it needs, skip whole row groups whose statistics rule them out, and move far less data than a row-based file would require. Each Parquet file holds a single table.

For most types of dataset, the Hugging Face Hub recommends Parquet because of its efficient compression, rich typing and wide support across tools. For a content archive, a typical pattern is to store text and metadata together in Parquet, so a recipient can, for example, select every article from one decade and language without reading any of the text of the others.

Row group size is the main tuning choice, and the guidance depends on how the files will be read. The Apache Parquet documentation recommends large row groups of 512 MB to 1 GB, written for Hadoop (HDFS) set-ups where a whole row group should fit in one storage block, and 8 KB data pages. Hugging Face's dataset viewer documentation recommends row groups of 100 to 300 MB uncompressed, because most data libraries stream Parquet one row group at a time and smaller groups use less memory; if a dataset's row groups are too big, the viewer generates new Parquet files. If the recipient's environment is unknown, the more moderate figure is the safer choice.

WebDataset for multimodal archives

For scans, audio and video, WebDataset packages data as standard tar archives. All consecutive files in a tar that share the same name prefix form one example, so a page might travel as page-000123.jpg, page-000123.json with its metadata and page-000123.txt with its OCR text. Media stays in its native formats, such as JPEG, PNG, TIFF, MP3, WAV, FLAC and MP4.

A large WebDataset is split into many shards. Hugging Face notes that each shard is often around 1 GB, while the full dataset can run to terabytes. Reading contiguous chunks from a tar can be orders of magnitude faster than reading separate files one by one, which is why the format suits streaming into training. Datasets in this format are generally already shuffled, and readers can reshuffle by shuffling the list of shards and using a buffer.

The trade-off is that tar shards are poor for looking up individual items or filtering by metadata. Hugging Face recommends Parquet for analytics, filtering and metadata parsing even for large image and audio datasets, so a good delivery for a broadcast or photographic archive pairs WebDataset shards with a Parquet table of metadata keyed by the same IDs. Our guide to audio and video archives covers the transcription side.

Designing the document record

Whatever the file format, the record design does most of the work. Group fields by purpose:

Group Example fields Notes
Identity id, record_version Stable and unique; never reused or renumbered between releases
Content title, text, structured blocks Plain text with any structure documented, not mixed markup
Descriptive metadata published, language, section, authors, labels Controlled values; one date format; consistent types
Provenance Source file, page or time range, tools and versions, review status, confidence Enough to trace every record back to its original
Relations duplicate_cluster, canonical flag, parent_id Supports deduplication and document hierarchies
Delivery split Keep all members of a duplicate cluster in the same split

A few rules prevent most downstream problems:

  • One record per meaningful unit. Usually an article, chapter, section or episode, not a page or a whole book.
  • One type per field, everywhere. If published is a YYYY-MM-DD string in one record, it must be in all of them. Mixed types cause schema errors when files are combined, especially in Parquet.
  • Decide what missing means. Choose between null and an absent key, apply it consistently, and document it.
  • Keep machine output separate. Store model-generated labels and their confidence scores in their own fields, apart from metadata taken from the source.
  • Deliver full documents unless asked otherwise. Recipients often have their own approach to chunking long documents; if you deliver chunks, include the parent document ID and position.

Sizing, compression, checksums and Hub conventions

File sizes and names

Aim for files of a few hundred megabytes to around a gigabyte. The Hugging Face dataset viewer converts large datasets into Parquet shards of about 500 MB, and WebDataset shards are often around 1 GB. The Hub's repository guidance recommends fewer than 100,000 files per repository and fewer than 10,000 per folder, suggests merging many small JSON files into JSON Lines, and recommends splitting any file into chunks under 200 GB, with 500 GB as the hard limit for a single file. Name shards with a zero-padded index and the total count, in the pattern train-00000-of-00014.parquet, keep shards roughly equal in size, and never split a record across two files.

Compression

The JSON Lines documentation recommends gzip or bzip2, and the Hub supports files compressed with ZIP, GZIP, ZSTD, BZ2, LZ4 and LZMA. Parquet compresses each column inside the file, so wrapping a Parquet file in another compressor adds little and stops tools from reading individual columns directly.

Checksums and manifests

Ship a manifest listing every file with its path, record count, size in bytes and SHA-256 checksum, so the recipient can confirm that every file arrived intact and complete. Machine-readable metadata can carry the same information: MLCommons' Croissant format describes each file with its content URL, encoding format, content size and SHA-256 checksum, and recommends semantic versioning (MAJOR.MINOR.PATCH) backed by checksums. Document the files, fields and versions in the dataset's datasheet as well; our guide to documenting a dataset explains what to include.

Hub conventions

If a dataset is going onto the Hugging Face Hub, the Hub natively supports Parquet, CSV and TSV, JSON and JSON Lines, Arrow, text, image, audio, video and PDF files, WebDataset and Lance. For large datasets it asks for a dataset card, file formats that are well integrated with its ecosystem, such as Parquet and WebDataset, and no custom loading scripts, which in its experience limit reuse.

Common mistakes

  • Pretty-printed records in a .jsonl file. Each record must sit on one line.
  • Inconsistent types. A field that is sometimes a number and sometimes a string breaks schema inference.
  • One giant file, or millions of tiny ones. Both make transfer, listing and parallel reading slow.
  • Long text in CSV. Quoting differences between tools corrupt columns.
  • No manifest or checksums. Without them, a truncated download looks like a complete one.
  • Identifiers that change between releases. Recipients cannot apply corrections or withdrawals if IDs move.
  • Encoding drift. Files that are not UTF-8, or that start with a byte order mark, fail in strict readers.

How Druid Learning helps

Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. Through Data Partnerships, it prepares an archive, raw or structured, and connects content owners with AI companies. Archive Optimization turns source material in formats including OCR scans, PDF, XML, IDML, INDD, SVG, EPUB, WAV, MP3 and MP4 into an indexed, labelled dataset. The platform has processed more than 10 million assets and handles content in more than 140 languages.

Frequently asked questions

What is the difference between JSON and JSON Lines?

A JSON file normally holds one value, such as a single large array of records, which usually has to be parsed in full. A JSON Lines file holds one complete JSON value on each line, separated by newlines. That makes it possible to stream, split, append and process records one at a time, which makes JSON Lines well suited to large collections of records and to logs.

Is Parquet better than JSON Lines for AI training data?

For large datasets, usually yes. Parquet is compressed, typed and columnar, so tools can read only the fields they need and skip irrelevant data, and the Hugging Face Hub recommends it for most datasets. JSON Lines remains a good choice for smaller sets, deeply nested records or files people will inspect by eye, but it is not recommended beyond several gigabytes.

How large should each dataset file be?

A practical range is a few hundred megabytes to around a gigabyte per file. Hugging Face converts large datasets into Parquet shards of about 500 MB, and WebDataset shards are often around 1 GB. Files in that range transfer reliably, can be read in parallel and restart cleanly after a failed download. Keep shards similar in size and never split a single record across files.

Why include checksums with a dataset?

A checksum such as SHA-256 is a short fingerprint of a file's exact contents. Listing one for every file in a manifest lets the recipient confirm that each file arrived complete and unaltered, catches truncated or corrupted transfers, and ties a specific release to specific files. Metadata formats such as MLCommons' Croissant include per-file checksums for exactly this purpose.

Should audio and video archives be delivered as WebDataset?

For large collections that will be streamed into training, WebDataset tar shards are a good fit, because they keep media in native formats and read sequentially at high speed. Pair them with a Parquet table holding the metadata and transcripts keyed by the same IDs, since tar shards are poor for filtering or looking up individual items. Smaller collections can simply ship as raw files plus an index.

Sources