committed 12:18AM - 10 Sep 26 UTC
Read Apache Avro object container files on every source that goes through
the sh…ared file readers: `file://events/day.avro`, `s3://bucket/day.data#avro`,
and the rest. Avro is decoded with Polars, already a core dependency, so the
format ships with the base install and adds nothing to the dependency set that
GH-316 is working to narrow.
Logical types survive the trip as themselves: `date` as a date, `time-micros`
as a time, `timestamp-micros` and `timestamp-millis` as UTC-aware datetimes,
`decimal` (carried in `bytes` or in `fixed`) as a decimal. `#columns=` and
`#chunksize=` behave as they do on the other columnar readers; the reader is
whole-file, because the library behind it has no lazy Avro scan.
Polars maps Avro onto Arrow, and it panics rather than raising in two places
that an ordinary file reaches. A `map` field has no Arrow mapping and aborts
during the read; a `timestamp-micros` past year 9999 decodes into the frame
quite happily and aborts on the way out to Python objects. Both raise
`PanicException`, which inherits `BaseException` and is therefore invisible to
the `except Exception` in dlt's own extract step, so either would end a load in
a Rust backtrace naming neither the file nor the field. Both are now converted
into an error naming both, with the location redacted the way every other
reader error is. A `null` field and a multi-branch union raise catchably
already and are pinned so a change in that behaviour surfaces here.
An Avro container is a sequence of self-describing blocks with no trailing
index or record count, so a tail cut exactly at a block boundary leaves a
shorter valid file that nothing distinguishes from the original: the surviving
blocks load and nothing is reported. Feather, ORC and Parquet each carry a
footer that a truncation destroys, so they raise on the same damage. A cut one
byte further leaves a partial record count, which is detectable corruption and
is accepted as end of file anyway; two bytes further it raises, so the silent
window is one byte wide. That second case is a limit of the reader rather than
of the format, and the pages say which is which. All three are measured and
pinned; closing the second means carrying a second container parser.
Avro is a read format. Polars can write it, but its writer mis-frames a list
column holding an empty list before a non-empty one: the list items go out
without their block headers and the record body overruns the block it
declares. `{"tags": []}` followed by `{"tags": ["x"]}` is ordinary JSON that
reaches a writer as exactly that frame, and depending on the sibling columns
the result either fails to read or reads back with wrong values and no error.
It behaves the same on polars 1.34.0, 1.39.3, 1.43.1 and 2.0.0rc1, so it is a
long-standing defect rather than a regression to wait out. Two `xfail(strict)`
tests hold the shape, decoding the container by hand so the claim rests on the
bytes rather than on the same library's reader; when it is fixed upstream they
turn red and say to register the writer.
Fixtures are hand-built for the same reason: a file written by the library
under test can only contain what that library can write, so a suite built from
round trips is structurally blind to the reader's worst input.