Do the SCL changes in 3.7.2 break the kafka C support from the incubator?

[mooney6023]

Running into this on gentoo. Was using the 3.7.1-r1 build on gentoo with the incubator version of kafka.

3.7.2 seems to break this.

Gentoo maintainer disables the syslog-ng java support in the syslog-ng ebuild I'm looking for a path forward.

[ihrwein]

Thanks for reporting this issue, @mooney6023 !

Could you show me some error messages? I don't remember any SCL changes that affects the Kafka destination in incubator.

[ihrwein]

@mooney6023 I understand your problem now. We created an SCL file for the Java based kafka destination which uses the name kafka. This name collides with the name of the kafka destination in incubator.

The situation is complicated, because the destination in incubator is older than the Java based destination, which resides in the syslog-ng git repository...

Our idea, is to rename the kafka destination in incubator to kafka-c. @vincentbernat is the original author, I'd ask his opinion about the topic.

[vincentbernat]

The original author is @pyr. I just added some bugs. Yes, the destination can be renamed in incubator.

[ihrwein]

@pyr I'm sorry about the authorship.

[pyr]

@ihrwein no worries. I'm fine with the change.

[ihrwein]

@pyr @vincentbernat Thanks for the quick response! We will do the renaming.

[ihrwein]

We have to solve #147 first.

[mooney6023]

Thanks gents,

I worked around the problem by removing the SCL kafka files after
installing 3.7.2

yanked
/usr/share/include/scl/kafka
/usr/share/include/scl/kafka/plugin.conf

then all was good. So yea, caught the name conflict.

I'm hoping that Gentoo will get a grails build into portage, as that is
what is stopping java support in the gentoo syslog-ng packaging.

In the meantime we push about 2 million 0.5kb sized records an hour through
the C based kafka destination per a server with no issues.

Sincerely,
Michael Mooney
Devadmin
mmooney@simpli.fi
M: 512.560.6577

http://www.simpli.fi/
Unstructured Data. Unlimited Potential.
1407 Texas Street, Suite 202 | Fort Worth, TX 76102
1.800.840.0768 | www.simpli.fi

On Tue, Jan 5, 2016 at 7:04 AM, Tibor Benke notifications@github.com
wrote:

We have to solve #147
#147 first.

—
Reply to this email directly or view it on GitHub
#146 (comment)
.

[szabist]

@mooney6023 Can you please share some details about your use case of syslog-ng working together with kafka? I mean what sort of logs you publish from syslog-ng to kafka? How is your setup looking like? What is your experience thus far with these two technologies working together? Siince kafka destination is relatively new in syslog-ng and there is not much usage experience publicly shared thus far, learning anything about your experience would really be interesting. One possibility if you have time and you are interested in is to post about the topic at https://syslog-ng.org/blogs/ Thanks in advance.