Log delivery support to Message Queue

[databeast]

Logs collected on CPE sensor should be pushed into a message queue stream for deliver to upstream analytics.

Logs will need to be collected by syslog/eventlog collector on host, then pushed into message queue system.

[databeast]

I've been looking at various aspects of this, including writing a logstash connector to push things into Kafka, or just having rsyslog push everything directly into kafka with no intermedia store.

In generaly, the idea of taking log data and pushing them either individually or grouped, into Kafka MQ items seems reasonably trivial.

[RevREB]

Except for the kafka dependancy on Zookeeper.
IF you build the Elasticseach environment right with a Nginx proxy and certs it can be made simpler with just log stash and the HTTP send module

[databeast]

fyi: we want kafka for a whole bunch of stuff, but early on however the primary benefit is removing IP address config from individual components (ie. just send everything to the local kafka instance, let it worry about off-host connectivity)

[RevREB]

Well, do we have a "system" map yet.
something to show how all the bits are connecting?