Test sources

You can add a test source to your docker-compose.yml config file. Just put listed configuration under services directive alongside with dc and agent containers.

MongoDB

mongo:
  image: mongo
  container_name: mongo-source
  volumes:
    - /data/db

With this config no username and password are required to log in. The connection string for the agent will be mongodb://mongo:27017

To import a CSV file use mongoimport command

docker cp /path/to/file.csv mongo-source:/
docker exec -i mongo-source mongoimport --db test --collection test_collection --type csv --headerline --file /file.csv

Database test and collection test_collection are automatically created

Kafka

zookeeper:
  image: wurstmeister/zookeeper

kafka:
  image: wurstmeister/kafka
  container_name: kafka-source
  environment:
    KAFKA_LISTENERS: PLAINTEXT://:29092,PLAINTEXT_HOST://:9092
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    KAFKA_CREATE_TOPICS: "test:1:1"
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

Topic test will be automatically created. The broker connection URL for the agent will be kafka:29092

To stream a JSON file use these commands:

docker cp /path/to/file.json kafka-source:/home/test_data.json
docker exec -i kafka-source bash -c "kafka-console-producer.sh --broker-list localhost:9092 --topic test < /home/test_data.json"

InfluxDB

influx:
  image: influxdb
  container_name: influx-source
  volumes:
    - /var/lib/influxdb
  environment:
      INFLUXDB_DB: test

Influx API URL for the agent will be http://influx:8086. Data to the InfluxDb can only be imported in line protocol format.

Download sample dataset:

curl https://s3.amazonaws.com/noaa.water-database/NOAA_data.txt -o NOAA_data.txt

Import data to test database

docker cp NOAA_data.txt influx-source:/
docker exec -i influx-source influx -import -path=NOAA_data.txt -precision=s -database=test