Variant RFC by AdamGS · Pull Request #15 · vortex-data/rfcs

AdamGS · 2026-02-25T21:08:17Z

RFC for a logical Variant type.

gatesn · 2026-02-25T21:11:14Z

proposals/0015-variant-type.md

+
+In addition to a new canonical encoding, we'll need a few more pieces to make variant columns useful:
+
+1. A set of new expressions, which extract children of variant arrays with a combination of path (similarly to `GetExpr`) and a dtype.


Maybe also worth mentioning expressions that convert to/from other variant-like data, e.g. JSON as a DType::Utf8 can be parsed into a DType::Variant.

I wonder if our JSON extension type has storage type DType::UTf8? or storage type DType::Variant...?

In my mind JSON type is “string verified as JSON”, like a PG column.
So far my impression is that there’s no consistent naming, and any choice we make will end up conflicting with something

So we would just also implement the variant expressions over a JSON extension type array?

accepted/0015-variant-type.md

a10y · 2026-02-26T02:01:50Z

proposals/0015-variant-type.md

+
+Different systems have different variations of this idea, but at its core its a type that can hold nested data with either a flexible or no schema. In addition to this "catch all" column, most system include the concept of "shredding", extracting a key with a specific type out of this column, and storing it in a dense way. This design can make commonly access subfields perform like first-class columns, while keeping the overall schema flexible.
+
+I propose a new dtype - `DType::Variant`. The variant type is always nullable, and its canonical encoding is just an array with a single child array, which is encoded in some specialized variant type.


how should we do execute_arrow for these, using the Parquet Variant? Or union?

added some thoughts on this point, it might require ~~a pretty big change~~ some changes/extending our arrow exporting logic

gatesn · 2026-02-26T15:57:04Z

I think this all makes sense, but I think it should explicitly call out what changes you want to make to DType enum / Scalar enum / Canonical enum / etc.

connortsui20

For something as complicated as the design space of variant, I think it would be worth putting together a few diagrams (could literally just be some text trees) that show the different kinds of variant and shredding designs as well as some concrete examples.

I'm happy to add this myself as well!

connortsui20 · 2026-02-27T19:07:13Z

accepted/0015-variant-type.md

+Vortex currently requires a strict schema, but real world data is often only semi-structured and deeply hierarchical. Logs, traces and user-generated data often take the form of many sparse fields.
+
+This proposal introduces a new dtype - `Variant`, which can capture data with row-level schema, while storing it in a columnar form that can compress well while being available for efficient analysis in a columnar format.
+


I know this is being pedantic but I think it would be good to have a motivation section here. Do we care about supporting all possible variant types, or just JSON, for example? And it might be good to say that we want this because other formats support this and people like that

When you say all types what do you mean?

connortsui20 · 2026-02-27T19:10:45Z

accepted/0015-variant-type.md

+
+Different systems have different variations of this idea, but at its core its a type that can hold nested data with either a flexible or no schema.
+
+Variant types are usually stored in two ways - values that aren't accessed often in some system-specific binary encoding, and some number of "shredded" columns, where a specific key is extracted from the variant and stored in a dense format with a specific type, allowing for much more performant access. This design can make commonly accessed subfields perform like first-class columns, while keeping the overall schema flexible. Shredding policies differ by system, and can be pre-determined or inferred from the data itself or from usage patterns.


I think "accessed" is the wrong word here? You could motivate this by giving an example of json data where a majority has the same type (string) but sometimes there happens to be a different type (int)

There is also the case here you have a int and a float? Or even a i8 and a u64 which is of course the same for json but not for a vortex scalar?

JSON->Variant conversion is a complex topic that is out of scope here IMO, JSON tiles deals with it and other systems have different behaviors, and I don't want to make a decision early that will lock us out

joseph-isaacs · 2026-03-03T10:50:55Z

accepted/0015-variant-type.md

+
+```rust
+enum Variant {
+	Value(Scalar),


This is a vortex scalar?

I think this is more of a "actual type"

joseph-isaacs · 2026-03-03T10:52:09Z

accepted/0015-variant-type.md

+
+We'll start with a rough description of the variant type, as many different systems define in different ways (see the [Prior Art](#prior-art) section at the bottom of the page).
+
+The variant type can be commonly described as the following rust type:


Are you proposing that the variant type is opaque or visable?

the array is opaque

accepted/0015-variant-type.md

joseph-isaacs · 2026-03-03T10:56:33Z

accepted/0015-variant-type.md

+
+I suggest we add a new expression - `get_variant_element(path, dtype)` (name TBD) which will support flexible paths and allow extracting children from variants. I use the `path` argument in this document loosely, but a subset of JSONPath might be appropriate here, see the [prior art](#prior-art) section to see how other systems handle it.
+
+Every variant encoding will need to be able to dispatch these behaviors, returning arrays of the expected type.


That are the ctors of a variant?

not sure I understand the question

How do I create a Variant array?

I'm honestly not sure whats the right API here, one option @gatesn and I discussed is that you can have an extension type (json_utf8) that holds strings and on write gets parsed and figures out the shredding. Another is to have a builder.

You can't create a Variant array. Or rather, you can but it just has a single child with DType::Variant.

You must construct a concrete implementation of a variant array, e.g. ParquetVariant.

accepted/0015-variant-type.md

joseph-isaacs · 2026-03-03T11:03:19Z

proposals/0015-variant-type.md

+## Unresolved Questions
+
+- Do we want a JSON extension type that automatically compresses as variant?
+- How do variant expressions operate over different variant encodings?


can we expand on this?

I guess we don't push arithmetic operations down? Missed those code paths.

I guess these might be reduce rules? Allowing variant encodings to implement their own specific logic for each of the variant-specific functions. Am I making sense? I don't think I fully understand how all the pieces connect.

So the iterative execution could cover this. But if you're extracting a column from the binary variant (i.e. non-shredded), then ideally you would fuse the extraction with the projection expression. Maybe this is too tricky to figure out at the moment and actually we could get most of the benefits from some sort of pipelined execution (run over 2k elements at a time for instance).

accepted/0015-variant-type.md

Signed-off-by: Adam Gutglick <adam@spiraldb.com>

gatesn

Let's ship it and start implementing. We may run into issues but these can be reflected in the developer guide.

gatesn reviewed Feb 25, 2026

View reviewed changes

a10y reviewed Feb 26, 2026

View reviewed changes

AdamGS force-pushed the adamg/variant branch from c5d2aff to e444968 Compare February 26, 2026 13:16

AdamGS changed the title ~~WIP: Variant RFC~~ Variant RFC Feb 26, 2026

connortsui20 reviewed Feb 27, 2026

View reviewed changes