Concurrent reads from same topic

[bosak]

Hi
In my application, i need to read (lets say from beginning) particular topic-partition in multiple concurrently running processes, which are started on demand.

So naturally i want multiple brod_consumers over same topic-partition, to keep current offset per my subscriber.

As i understand, the way ho to achieve this is to use multiple brod_clients. It indeed works. Question is as these clients are started on demand, clearly not as intended by design of brod. Are there any disadvantages?

I am aware of redundant tcp socket, and that client name is atom (does it need to be?)

Would it be possible to have multiple brod_consumers for same topic-partition under single brod_client?

[zmstone]

You are right, the disadvantage of limiting one consumer topic per client is the sockets will be redundant.
The advantage is the per-partition worker process tracks only one state
(the last acked offset from subscriber, last requested corr-id etc.)

To to support more consumers per-client would imply the callers to either identify themselves with an ID, or assign brod_consumer an ID, and use the ID in consumer/subscriber related API calls.
e.g. brod:start_consumer brod:get_consumer

[bosak]

Thanks for reply,
I guess with socket redundancy it is ok, as I shall not have big number of concucurent clients running at a same time. So multi client approach is probably fine.

Still I would appreciate if you remove is_atom(ClientId) guard, to support my use_case in a bit cleaner way (i woudl like to generate ClientId rather than cycle and reuse).

[zmstone]

It's a bit tricky but doable.
tricky because the client id atom is used to register the process also to create a named ets table.
I'll see if we can use pg2 for it if there is no atom ID provided.

[k32]

A newer discussion was here: #369