You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Remove hard-coded internal amount to buffer
This allows users to specify a flush interval, and as long as they have
enough memory, it will buffer as much as possible internally!
Correctly handle writing larger batch sizes for the parquet sink
Inside Apache Arrow (Rust) there's a 1024 row batch size default which
was clipping the amount of data being decoded when flushing massive
parquet buffers
Make failing to infer the schema non-fatal
if a schem cannot be inferred it's important to log the error, but the
process crashing entirely is not idea
Add support for defining schemas to be used by the sinks
Right now the Kafka sink does not support the use of the defined
schemas, but these allow for defining valid/acceptable schemas up front
for data written to specific topics.
What this will _not_ do however is any form of type coercion! Make sure
the schemas are the right types for the data coming in!