NDJSON, JSON Lines and plain JSON

NDJSON and JSON Lines are two names for the same format. Here is what it is, why it exists, and when you want it instead of one big array.

The format

One JSON value per line. No commas between them, no brackets around the whole thing.

{"id":1,"name":"ada","ok":true}
{"id":2,"name":"grace","ok":false}
{"id":3,"name":"edsger","ok":true}

The same data as ordinary JSON:

[
  {"id":1,"name":"ada","ok":true},
  {"id":2,"name":"grace","ok":false},
  {"id":3,"name":"edsger","ok":true}
]

The file extensions you will see are .ndjson, .jsonl and quite often just .json. The names NDJSON and JSON Lines came from two different specs written around the same time. They describe the same thing and nobody has ever needed to tell them apart in practice.

Why bother

You can read it a line at a time

This is the whole point. A normal JSON array is one value, so a parser has to reach the closing bracket before it can hand you anything. NDJSON gives you a complete record at every newline. You can process a hundred gigabyte file with a few megabytes of memory, and you can start doing useful work before the file has finished arriving.

You can append to it

Adding a record is a write at the end of the file. With an array you would have to find the closing bracket, back up over it, add a comma, and put it back. That is why log pipelines, event streams and export jobs all end up producing NDJSON: appending is the operation they do constantly.

It survives being cut in half

Every standard line tool works on it. head, tail, grep, split, wc -l, and every one of them leaves you with a valid file. Truncate an array in the middle and you have garbage.

wc -l events.ndjson          # how many records
head -1000 events.ndjson     # a sample, still valid
grep '"level":"error"' events.ndjson
split -l 100000 events.ndjson chunk-

When you want the array instead

When something needs the whole thing as one value. An HTTP API returning a response body, a config file, anything a person will read and edit by hand. NDJSON is a stream format. If the data is not a stream, the extra brackets are not costing you anything.

Converting between them

# array to NDJSON
jq -c '.[]' array.json > lines.ndjson

# NDJSON to array
jq -s '.' lines.ndjson > array.json

-c keeps each record on one line, which is the part that matters. -s slurps the whole input into an array, so it does need to hold everything at once.

Things that catch people out

A trailing newline is fine. An empty final line is normal and every reasonable reader skips it. Do not go hunting for a bug there.
Records do not have to match. Nothing says line 2 has the same shape as line 1. Handy for mixed event logs, annoying when you assumed a schema.
Pretty printing breaks it. A record spread over several lines is no longer one JSON value per line, and the file is no longer NDJSON. This is the single most common way people produce a file that nothing will read.
One bad line poisons a naive reader. If you are writing the reader, catch the parse error per line and report the line number. Failing the whole file because of one bad record is rarely what anyone wants.

Try it

jsontree.app detects newline delimited JSON on its own and reads it line by line, so you do not have to convert anything first. If a line fails to parse it tells you which one.

Open a file