Protobuf default recursion depth limit of 32 - fails disk buffering, codec, and Vector source/sink #19315
Labels
domain: codecs
Anything related to Vector's codecs (encoding/decoding)
type: enhancement
A value-adding code change that enhances its existing functionality.
Comments
|
We have been able to reproduce this locally with a trivial configuration on v0.35.1 sources/files.yaml: transforms/debug.yaml: sinks/blackhole.yaml Our test data is the same JSON object as above. The relevant output is captured below. As a work around we have set |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A note for the community
Problem
Vector's protobuf decoder has an implied by-default recursion depth limit of 32. This will cause events that are nested deeper than this limit to fail delivery at any point in a pipeline that uses protobuf encoding - whether protobuf/native codec, using the Vector native source/sink, or using a sink with disk buffering enabled. Depending on the configured retry behavior, an event of this type may cause infinite retries, effectively stopping the pipeline as a poison event.
This is directly related to the default recursion limit of 100-deep in the Prost protobuf crate, and is resolved by enabling the
no-recursion-limitprost crate feature.The error message differs between the usage, whether from a vector source/sink, or from disk buffering.
Vector sink:
2023-12-05T19:13:00.920742Z WARN sink{component_kind="sink" component_id=vector_out component_type=vector}:request{request_id=2}: vector::sinks::util::retries: Retrying after error. error=Request failed: status: Internal, message: "failed to decode Protobuf message: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: ValueMap.fields: Value.kind: Log.fields: EventWrapper.event: PushEventsRequest.events: recursion limit reached", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 05 Dec 2023 19:13:00 GMT", "content-length": "0"} } internal_log_rate_limit=true(note the smoking gun error message of "recursion limit reached")
Any disk-buffered sink:
2023-12-05T19:17:32.250361Z ERROR sink{component_kind="sink" component_id=sender component_type=http}: vector_buffers::internal_events: Error encountered during buffer read. error=failed to decoded record: InvalidProtobufPayload error_code="decode_failed" error_type="reader_failed" stage="processing" internal_log_rate_limit=trueConfiguration
Version
vector 0.35.0-custom-ca1b6d14a (x86_64-unknown-linux-gnu debug=full)
Debug Output
No response
Example Data
Any trivial 32+-level-deep JSON structure is usable, such as below.
{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":{"hi":false}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}Additional Context
No response
References
#18130
The text was updated successfully, but these errors were encountered: