Support streaming data sources

[rcrichton]

As it stands the pipeline is configured to support batches and achieves the incremental mode using micro batches. True support for streaming data into the pipeline would allow for a more real time ingestion of data and also allow for data unbounded data sources such as Kafka (or any event/messaging/subscription system).