Add task batching doc #150

ryamagishi · 2022-04-10T12:32:12Z

Motivation

To add a documentation for BathingProcessor like task-compaction.adoc.

Related Issue: #128
Related PR: #139

ocadaruma

Thanks for the follow-up!

Left some feedbacks.
Also would you mind updating example project? (https://github.com/line/decaton/tree/master/docs/example/src/main/java/example)

ocadaruma · 2022-04-12T02:32:54Z

docs/task-batching.adoc

+When downstream-DB supports batching I/O (which often very efficient)
+
+== Usage
+To use `Task Batching`, you only need to instantiate `BatchingProcessor`.


Practically, developers will create their own class that inherits BatchingProcessor rather direct instantiation I guess.
So you only need to implement BatchingProcessor sounds more appropriate?

The description and examples have been updated to use InsertHelloTaskBatchingProcessor, which inherits BatchingProcessor<HelloTask>.

ocadaruma · 2022-04-12T02:35:19Z

docs/task-batching.adoc

+                                   )
+    }
+
+    private static BatchingProcessor<HelloTask> createBatchingProcessor(long lingerMillis, int capacity) {


As above comment, I guess the practical usage would be implementing a class that inherits BatchingProcessor.

Then how about fixing this example as like so?

I did the same. #150 (comment)

ocadaruma · 2022-04-12T02:38:27Z

docs/task-batching.adoc

+[source,java]
+.HelloTask.java
+----
+public class HelloTask {


Let's make this example more looks like "real", by naming this class as "InsertHelloTask" or something? :)

Sorry, maybe I don't understand this comment properly.

I created InsertHelloTaskBatchingProcessor and wrote the insertion points in the commet, but please let me know if there are any problems.
https://github.com/line/decaton/pull/150/files#diff-6cfdc5f934448411f9f9ff81c4cf114233a240ab01765dec212c7eac322f9961R70

ocadaruma · 2022-04-12T02:40:06Z

docs/task-batching.adoc

+In this section, we will briefly explain how is Task Batching implemented.
+All the magic happened in `BatchingProcessor` when a task comes to this processor the following things happen:
+
+1. The task will be put into an in-memory window.
+2. When the size or time reach each limit, `processBatchingTasks(List)` is called with stored `batchingTasks`.


I think what we should include in "Implementation" section is something that developers should notice which comes from its implementation detail (rather than its "API"), rather than just a "how this feature is implemented".

e.g. In rate-limiting doc, it's mentioned that it adopts token-bucket algorithm which allows some "bursting" against configured rate-limit, so developers can notice that it's not suitable when they have to limit the rate strictly. (https://github.com/line/decaton/blob/master/docs/rate-limiting.adoc#implementation)

Then, we don't need to have Implementation section for BatchingProcessor I think? Or do you have some ideas you want to mention in this section?

Thanks for the detailed explanation.
Removed the "Implementation" section.

ocadaruma · 2024-07-16T02:55:16Z

@ryamagishi Hi, long time no see. Do you still have a plan to complete this PR?

ryamagishi · 2024-07-19T06:19:19Z

Hi @ocadaruma, I apologize for the delay and for forgetting about this PR. If you would permit, I would like to complete it by the end of this month. I need some time to refresh my memory on the work, so I appreciate your understanding.

ocadaruma · 2024-07-19T06:47:25Z

If you would permit, I would like to complete it by the end of this month

That's totally fine. Thank you!

…BatchingProcessor instead of direct instantiation.

ryamagishi · 2024-07-27T07:22:59Z

@ocadaruma
I apologize for the late response.
I have corrected the parts you commented on.
If there is anything that needs to be corrected, I will deal with it right away so that it can be merged.
Please check it when you have time.

ryamagishi · 2024-07-27T07:24:14Z

docs/example/build.gradle

@@ -7,7 +7,7 @@ plugins {

 ext {
    DECATON_VERSION = getProperty("version") + (getProperty("snapshot").toBoolean() ? "-SNAPSHOT" : "")
-    PROTOBUF_VERSION = "3.3.0"
+    PROTOBUF_VERSION = "3.22.3"


The example build failed, so I upgraded it.

ocadaruma

Thanks for update, current content LGTM!

I left one more request. Please take a look.

ocadaruma · 2024-07-29T01:09:06Z

docs/task-batching.adoc

Could you add Caution: somewhere to mention below?:

batch-flush is done in BatchingProcessor's scheduled executor thread

Which means, the parallelism of flushing has to be controlled by ProcessorScope, not only by decaton.partition.concurrency config. i.e.:

parallelize flushing per partition: ProcessorScope.PARTITION

parallelize flushing per processor thread: ProcessorScope.THREAD

Some users pointed out that this behavior might be confusing, so it's worth to mention.

Thanks! I've added it.
7b0112a

ryamagishi · 2024-07-30T09:28:07Z

@ocadaruma
I have responded to your comments.
Please take a look.

ocadaruma

Thank you! LGTM

Add task batching doc

0b5959e

ryamagishi force-pushed the add-task-batching-doc branch from e2813da to 0b5959e Compare April 10, 2022 12:33

ocadaruma reviewed Apr 12, 2022

View reviewed changes

ryamagishi added 6 commits July 27, 2024 13:24

Fix task batching doc to explain how to create a class that inherits …

fbedebd

…BatchingProcessor instead of direct instantiation.

Remove Implementation section from task batching doc

98cd998

Upgrade task batching doc base version

777ee8f

Add TaskBatchingMain

fb2f1c2

Upgrade example's PROTOBUF_VERSION

c4fdfcf

Fix task batching doc to have the same code example.TaskBatchingMain

332262a

ryamagishi force-pushed the add-task-batching-doc branch from e0c9d97 to 332262a Compare July 27, 2024 06:41

Rename task batching doc classes

675d58b

ryamagishi commented Jul 27, 2024

View reviewed changes

ocadaruma reviewed Jul 29, 2024

View reviewed changes

ryamagishi force-pushed the add-task-batching-doc branch 2 times, most recently from fc79b8c to 2bd4b5e Compare July 30, 2024 09:16

ryamagishi added 2 commits July 30, 2024 18:20

Fix task batching doc's details

4753029

Add CAUTION to task batching doc

7b0112a

ryamagishi force-pushed the add-task-batching-doc branch from 387fa4f to 7b0112a Compare July 30, 2024 09:22

ocadaruma approved these changes Jul 30, 2024

View reviewed changes

ocadaruma merged commit 08614dd into line:master Jul 30, 2024
5 checks passed

ocadaruma mentioned this pull request Aug 20, 2024

Provide BatchingProcessor as a built-in processor #128

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add task batching doc #150

Add task batching doc #150

ryamagishi commented Apr 10, 2022

ocadaruma left a comment

ocadaruma Apr 12, 2022

ryamagishi Jul 27, 2024

ocadaruma Apr 12, 2022

ryamagishi Jul 27, 2024

ocadaruma Apr 12, 2022

ryamagishi Jul 27, 2024

ocadaruma Apr 12, 2022

ryamagishi Jul 27, 2024

ocadaruma commented Jul 16, 2024

ryamagishi commented Jul 19, 2024

ocadaruma commented Jul 19, 2024

ryamagishi commented Jul 27, 2024

ryamagishi Jul 27, 2024

ocadaruma left a comment

ocadaruma Jul 29, 2024

ryamagishi Jul 30, 2024

ryamagishi commented Jul 30, 2024

ocadaruma left a comment

Add task batching doc #150

Add task batching doc #150

Conversation

ryamagishi commented Apr 10, 2022

Motivation

ocadaruma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocadaruma commented Jul 16, 2024

ryamagishi commented Jul 19, 2024

ocadaruma commented Jul 19, 2024

ryamagishi commented Jul 27, 2024

Choose a reason for hiding this comment

ocadaruma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryamagishi commented Jul 30, 2024

ocadaruma left a comment

Choose a reason for hiding this comment