Enhance Kafka exporter to respect max message size

[yurishkuro]

This is a copy of the upstream issue open-telemetry/opentelemetry-collector-contrib#36982 that we encountered, creating a ticket here to track and for better discoverability.

Problem is described here: #6437 (comment)

[Riteek712]

@yurishkuro I would like to contribute to this issue can you please assign me this?

[yurishkuro]

consider it assigned

[Riteek712]

Hi @yurishkuro , I am just informing you that I'm working on this issue, I came down with a cold last week, I am sorry for the delay.

[ADI-ROXX]

@yurishkuro This issue seems to be in limbo. So I would also like to work on this.

@Riteek712 Can we contribute together on this issue?

[Riteek712]

@ADI-ROXX sure, can you mention your email for correspondence?

[ADI-ROXX]

@yurishkuro Can you provide some commands to duplicate the problem on our own system?

[shivanshuraj1333]

hey folks, I'm working on this as it is needed for us, the splitting logic for traces is not straight forward for spans, had to do binary search over spans.

Here's a PR open-telemetry/opentelemetry-collector-contrib#37176, I have added unit tests.
Would open it up for review once I have manually tested it with a kafka cluster that the splitting is happening as expected.

[ADI-ROXX]

Okay @shivanshuraj1333 👍