committed 09:33PM - 20 Jun 26 UTC
- Accept a bunch of decoding options per `KafkaDecodingOptions`
- Provide a bunc…h of output formatting options per `KafkaEvent`
- Tie both elements together using `KafkaEventProcessor`
The machinery is effectively the same like before, but provides a few
more options to allow type decoding for Kafka event's key/value slots,
a selection mechanism to limit the output to specific fields only, and
a small projection mechanism to optionally drill down into a specific
field.
In combination, those decoding options allow users to relay
JSON-encoded Kafka event values directly into a destination table,
without any metadata wrappings.
The output formatter provides three different variants out of the box.
More variants can be added in the future, as other users or use cases
may have different requirements in the same area.
Most importantly, the decoding unit is very compact, so relevant tasks
don't need a corresponding transformation unit down the pipeline, to
keep the whole ensemble lean, in the very spirit of omniload.
For performance reasons, the codec also started using the `orjson`
package.