[Parquet][C++] Add experimental VECTOR repetition level for Arrow FixedSizeList

[rok]

This PR prototypes a new experimental Parquet repetition type, VECTOR, mainly for Arrow's FixedSizeList<T, N> as proposed in Option B here.

Today this is written through Parquet LIST, which means we pay repetition/definition-level costs for an inner shape that is already fixed. VECTOR stores the fixed length in the Parquet schema as vector_length and avoids the inner repetition level.

Writing FixedSizeList<T, N> as VECTOR is opt-in via ArrowWriterProperties::Builder::enable_experimental_vector_encoding() defaulting to LIST otherwise.

Implemented in this prototype

• Parquet schema/thrift addition of VECTOR
• Arrow schema conversion:
  • FixedSizeList -> VECTOR when explicitly enabled
  • VECTOR -> FixedSizeList
• required and nullable rows in VECTOR
• nullable primitive elements in VECTOR
• primitive type vectors
• limited composability proof of concept: FixedSizeList<struct<x: float, y: int32>, N>
• roundtrip/schema/path tests
• read/write/roundtrip benchmarks

Deferred for now:

• dictionary / non-PLAIN encodings
• statistics
• page index
• general nested VECTOR children (non-fixed will not be possible)

Compatibility

We expect non-VECTOR-aware readers to fail when encountering VECTOR repetition level.

Benchmark snapshot

Both VECTOR and LIST use identical WriterProperties (dictionary, statistics and page-index disabled, PLAIN encoding) — the only intended difference is enable_experimental_vector_encoding.

Numbers below are from a local release build on m4 mac, so directional only.

Required FixedSizeList<float, N>: FixedSizeList encoded as LIST vs VECTOR

All with FixedSizeList<float, {80,768,10k}>.

Vector length	LIST write	VECTOR write	Write speedup	LIST read	VECTOR read	Read speedup	LIST roundtrip	VECTOR roundtrip	Roundtrip speedup
80	3.11M rows/s	21.36M rows/s	6.9x	7.36M rows/s	123.31M rows/s	17x	2.13M rows/s	17.68M rows/s	8.3x
768	339.52k rows/s	2.32M rows/s	6.8x	876.86k rows/s	13.06M rows/s	15x	247.32k rows/s	1.86M rows/s	7.5x
10,000	27.41k rows/s	192.99k rows/s	7.0x	66.99k rows/s	795.37k rows/s	12x	19.53k rows/s	165.72k rows/s	8.5x

Realistic embedding row: (int64 id, timestamp ts, int32 category, float score, string label, FSL<float, N> embedding)

Vector length	LIST write	VECTOR write	Write speedup	LIST read	VECTOR read	Read speedup	LIST roundtrip	VECTOR roundtrip	Roundtrip speedup
768	347.58k rows/s	2.37M rows/s	6.8x	864.73k rows/s	11.87M rows/s	14x	248.52k rows/s	1.79M rows/s	7.2x

[adamreeve]

Would it be valid at some point to have a sechema like vector<vector<int>>, and then the effective length would be the multiple of the two vector lengths?

[adamreeve]

Should this and other similar tests be re-using the MakeVectorFixedSizeListTable helper above?

[adamreeve]

Rather than support vector repetition directly on a primitive, should we enforce that it should always use a three-level schema and require use of a new VECTOR or FixedSizeList logical type? The Parquet format allows using a single node with repetition::repeated to represent a list for backwards compatibility but encourages use of the three-level schema. We don't have to worry about backwards compatibility here though.

[adamreeve]

What if we have a table with an Arrow fixed-size list of string and a fixed-size list of int? We should allow using the normal list encoding for the string list and vector for the int list rather than requiring that all fixed size lists can use the vector encoding if enable_experimental_vector_encoding is used.

[adamreeve]

This change is a bit surprising to me, I would have thought that the vector path would still use the same code as in the else branch where records_to_read is the total number of elements (rows * vector length), which would match the behaviour of variable size list data.