What is missing from CSV? A peek into the SFlat format

CSV (Comma-Separated Values, as in RFC 4180) is one of those file formats that seems almost perfect until you need to agree on what the data actually means. For example, a row in a CSV file might contain values such as:

00123,true,2026-10-06,

If you investigate the line, you will quickly find yourself asking questions like: is 00123 a number or an identifier? Is true really a Boolean? Is the date always in ISO format? Does the final empty field mean an empty string or perhaps a null value?

Usually, the answers live somewhere else: in application code, documentation, an email, or an agreement between the systems producing and consuming the file.

That is the problem I've been experimenting with in a file format called SFlat (Structured Flat Data).

The idea is not to replace CSV everywhere. CSV is simple, widespread, and often exactly the right tool. Instead, I wanted to explore a narrower question: what would a flat, text-based interchange format look like if the schema and validation rules were part of the file itself?

Or put another way: flat data does not necessarily have to mean schema-less data. A small SFlat file can look like this:

#!sflat 0.2
Profile: text
Layout: row
Body-Encoding: text
Records: 2
Schema: Name:str(50):req;Age:int32:opt;Joined:date:req

--Alice,35,2024-01-15
--Bob,\N,2024-03-02

Even without knowing much about the format, you can already see some useful information:

• the file contains two records
• the first column Name is a required string
• Age is an optional 32-bit integer
• Joined is a required date
• \N explicitly means a null value.

This is the main design goal: reduce guessing. I also wanted the format to remain simple enough that you can open it in Notepad and understand the basic idea without first reading 80 pages of specification. The ability to email the file, view it with any device and see the file is not truncated are also pluses.

These things matters to me. A text-based interchange formats are useful partly because a developer can open the file and inspect what went wrong. SFlat tries to keep that property while adding more structure.

One problem with CSV is type interpretation. The file itself does not normally say whether a column contains strings, integers, dates, Booleans, or something else. SFlat puts that information into the schema:

Schema: Id:int32:req;Name:str(100):req;Price:decimal:opt

Another problem is missing data.

In CSV, these two values can easily become ambiguous:

Alice,
Bob,""

Does the first row contain null? Does the second contain an empty string? That depends on the conventions of the application reading the file.

SFlat makes nullability part of the schema and gives null a defined representation. There is also basic structural validation. For example:

Records: 1000

If only 997 records arrive, the consumer immediately knows something is wrong. Optional integrity information can go further and describe the number of body bytes or a SHA-256 hash of the serialized data.

This is useful when files move between systems and you want to know not only that the syntax is valid, but that the complete expected payload arrived unchanged.

Another design goal is deterministic serialization. If two implementations write the same logical SFlat data, the goal is that the canonical representation should also be the same. This matters for things such as:

• hashing
• digital signatures
• testing
• diffs
• reproducible data exchange.

SFlat also has a Payload profile for cases where a flat record contains large text or binary content.

That is useful because normal delimited formats become awkward when one field suddenly contains megabytes of text or binary data.

None of this means SFlat should be used everywhere. For example, if the data is naturally hierarchical, JSON is usually a better fit.

If you are processing huge analytical datasets, formats such as Parquet are designed for that job. Similarly, if you just need to send a quick list of names and email addresses between two tools, CSV may still be the simplest answer.

SFlat is aimed at the space where the data is fundamentally flat, but the contract around that data matters.

Some examples could be:

• data exchanged between business systems
• imports and exports with a defined schema
• batch integrations
• archived flat datasets that should remain self-describing
• files where validation should happen before application-specific processing begins.

I think one of the interesting questions in file format design is how much structure is enough. Add too little and every application invents its own conventions. But if you add too much, the format becomes difficult to implement, inspect, or explain.

SFlat is my attempt to find a useful point somewhere between those extremes. It is currently an experimental, pre-1.0 format, so the specification can still change. That is intentional: I'd rather learn from implementations and real use cases before declaring the format finished.

If you work with flat-file integrations, I'd be interested in which problems you regularly run into with CSV and similar formats. You can read more at sflat.dev.

Happy hacking!