diff --git a/docs/api-reference/dataframe.md b/docs/api-reference/dataframe.md
index ce7a51de5..1c80813c1 100644
--- a/docs/api-reference/dataframe.md
+++ b/docs/api-reference/dataframe.md
@@ -1008,7 +1008,7 @@ operations, but branches can still be generated from its originating SDF.
class StreamingSeries(BaseStreaming)
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L70)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L60)
`StreamingSeries` are typically generated by `StreamingDataframes` when getting
elements from, or performing certain operations on, a `StreamingDataframe`,
@@ -1075,7 +1075,7 @@ def from_apply_callback(cls, func: ApplyWithMetadataCallback,
sdf_id: int) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L132)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L126)
Create a StreamingSeries from a function.
@@ -1104,7 +1104,7 @@ instance of `StreamingSeries`
def apply(func: ApplyCallback) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L155)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L149)
Add a callable to the execution list for this series.
@@ -1156,7 +1156,7 @@ a new `StreamingSeries` with the new callable added
def compose_returning() -> ReturningExecutor
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L189)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L183)
Compose a list of functions from this StreamingSeries and its parents into one
@@ -1187,7 +1187,7 @@ def compose(
None]] = None) -> VoidExecutor
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L204)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L198)
Compose all functions of this StreamingSeries into one big closure.
@@ -1245,7 +1245,7 @@ def test(value: Any,
ctx: Optional[MessageContext] = None) -> Any
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L248)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L242)
A shorthand to test `StreamingSeries` with provided value
@@ -1277,7 +1277,7 @@ result of `StreamingSeries`
def isin(other: Container) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L304)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L313)
Check if series value is in "other".
@@ -1322,7 +1322,7 @@ new StreamingSeries
def contains(other: Union[Self, object]) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L331)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L344)
Check if series value contains "other"
@@ -1367,7 +1367,7 @@ new StreamingSeries
def is_(other: Union[Self, object]) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L356)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L369)
Check if series value refers to the same object as `other`
@@ -1409,7 +1409,7 @@ new StreamingSeries
def isnot(other: Union[Self, object]) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L379)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L392)
Check if series value does not refer to the same object as `other`
@@ -1452,7 +1452,7 @@ new StreamingSeries
def isnull() -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L403)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L416)
Check if series value is None.
@@ -1489,7 +1489,7 @@ new StreamingSeries
def notnull() -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L426)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L439)
Check if series value is not None.
@@ -1526,7 +1526,7 @@ new StreamingSeries
def abs() -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L449)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L462)
Get absolute value of the series value.
diff --git a/docs/api-reference/quixstreams.md b/docs/api-reference/quixstreams.md
index 9b21e0309..d25cf868d 100644
--- a/docs/api-reference/quixstreams.md
+++ b/docs/api-reference/quixstreams.md
@@ -1641,7 +1641,7 @@ operations, but branches can still be generated from its originating SDF.
class StreamingSeries(BaseStreaming)
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L70)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L60)
`StreamingSeries` are typically generated by `StreamingDataframes` when getting
elements from, or performing certain operations on, a `StreamingDataframe`,
@@ -1700,7 +1700,7 @@ def from_apply_callback(cls, func: ApplyWithMetadataCallback,
sdf_id: int) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L132)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L126)
Create a StreamingSeries from a function.
@@ -1723,7 +1723,7 @@ instance of `StreamingSeries`
def apply(func: ApplyCallback) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L155)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L149)
Add a callable to the execution list for this series.
@@ -1767,7 +1767,7 @@ a new `StreamingSeries` with the new callable added
def compose_returning() -> ReturningExecutor
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L189)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L183)
Compose a list of functions from this StreamingSeries and its parents into one
@@ -1794,7 +1794,7 @@ def compose(
None]] = None) -> VoidExecutor
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L204)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L198)
Compose all functions of this StreamingSeries into one big closure.
@@ -1844,7 +1844,7 @@ def test(value: Any,
ctx: Optional[MessageContext] = None) -> Any
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L248)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L242)
A shorthand to test `StreamingSeries` with provided value
@@ -1870,7 +1870,7 @@ result of `StreamingSeries`
def isin(other: Container) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L304)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L313)
Check if series value is in "other".
@@ -1907,7 +1907,7 @@ new StreamingSeries
def contains(other: Union[Self, object]) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L331)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L344)
Check if series value contains "other"
@@ -1944,7 +1944,7 @@ new StreamingSeries
def is_(other: Union[Self, object]) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L356)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L369)
Check if series value refers to the same object as `other`
@@ -1978,7 +1978,7 @@ new StreamingSeries
def isnot(other: Union[Self, object]) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L379)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L392)
Check if series value does not refer to the same object as `other`
@@ -2013,7 +2013,7 @@ new StreamingSeries
def isnull() -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L403)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L416)
Check if series value is None.
@@ -2044,7 +2044,7 @@ new StreamingSeries
def notnull() -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L426)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L439)
Check if series value is not None.
@@ -2075,7 +2075,7 @@ new StreamingSeries
def abs() -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L449)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L462)
Get absolute value of the series value.
@@ -2370,7 +2370,7 @@ class FixedTimeWindow()
def final() -> "StreamingDataFrame"
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L129)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L127)
Apply the window aggregation and return results only when the windows are
closed.
@@ -2401,7 +2401,7 @@ can remain unprocessed until the message the same key is received.
def current() -> "StreamingDataFrame"
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L167)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L165)
Apply the window transformation to the StreamingDataFrame to return results
for each updated window.
@@ -2429,7 +2429,7 @@ regardless of whether the window is closed or not.
```python
def get_window_ranges(timestamp_ms: int,
duration_ms: int,
- step_ms: Optional[int] = None) -> Deque[Tuple[int, int]]
+ step_ms: Optional[int] = None) -> Deque[tuple[int, int]]
```
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/base.py#L17)
@@ -2666,7 +2666,7 @@ def add_apply(func: Union[
metadata: bool = False) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L186)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L190)
Add an "apply" function to the Stream.
@@ -2697,7 +2697,7 @@ def add_update(func: Union[UpdateCallback, UpdateWithMetadataCallback],
metadata: bool = False) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L227)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L231)
Add an "update" function to the Stream, that will mutate the input value.
@@ -2725,7 +2725,7 @@ def add_transform(func: Union[TransformCallback, TransformExpandedCallback],
expand: bool = False) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L261)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L265)
Add a "transform" function to the Stream, that will mutate the input value.
@@ -2754,7 +2754,7 @@ a new Stream derived from the current one
def diff(other: Self) -> Self
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L285)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L289)
Takes the difference between Streams `self` and `other` based on their last
@@ -2787,7 +2787,7 @@ a new independent `Stream` instance whose root begins at the diff
def root_path(allow_splits=True) -> List[Self]
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L350)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L354)
Return a list of all parent Streams including the node itself.
@@ -2807,7 +2807,7 @@ a list of `Stream` objects
def full_tree() -> List[Self]
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L371)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L375)
Starts at tree root and finds every Stream in the tree (including splits).
@@ -2829,7 +2829,7 @@ def compose(
None]] = None) -> VoidExecutor
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L378)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L382)
Generate an "executor" closure by mapping all relatives of this `Stream` and
@@ -2861,7 +2861,7 @@ the stream has transform functions in the tree. Default - True.
def compose_returning() -> ReturningExecutor
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L435)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L439)
Compose a list of functions from this `Stream` and its parents into one
big closure that always returns the transformed record.
@@ -11184,6 +11184,18 @@ The iterable should output dicts with the following data/naming structure:
## quixstreams.sources.community.file.file
+
+
+### FileFetcher
+
+```python
+class FileFetcher()
+```
+
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L22)
+
+Serves individual files while downloading another in the background.
+
### FileSource
@@ -11192,7 +11204,7 @@ The iterable should output dicts with the following data/naming structure:
class FileSource(Source)
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L19)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L70)
Ingest a set of files from a desired origin into Kafka by iterating through the
provided folder and processing all nested files within it.
@@ -11262,10 +11274,10 @@ def __init__(directory: Union[str, Path],
compression: Optional[CompressionName] = None,
replay_speed: float = 1.0,
name: Optional[str] = None,
- shutdown_timeout: float = 10)
+ shutdown_timeout: float = 30)
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L79)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L130)
**Arguments**:
@@ -11292,7 +11304,7 @@ to gracefully shutdown
def default_topic() -> Topic
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L152)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L204)
Uses the file structure to generate the desired partition count for the
diff --git a/docs/api-reference/sources.md b/docs/api-reference/sources.md
index 7f76c5d9f..148cd30a1 100644
--- a/docs/api-reference/sources.md
+++ b/docs/api-reference/sources.md
@@ -822,6 +822,18 @@ For other parameters See `quixstreams.sources.kafka.KafkaReplicatorSource`
## quixstreams.sources.community.file.file
+
+
+### FileFetcher
+
+```python
+class FileFetcher()
+```
+
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L22)
+
+Serves individual files while downloading another in the background.
+
### FileSource
@@ -830,7 +842,7 @@ For other parameters See `quixstreams.sources.kafka.KafkaReplicatorSource`
class FileSource(Source)
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L19)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L70)
Ingest a set of files from a desired origin into Kafka by iterating through the
provided folder and processing all nested files within it.
@@ -902,10 +914,10 @@ def __init__(directory: Union[str, Path],
compression: Optional[CompressionName] = None,
replay_speed: float = 1.0,
name: Optional[str] = None,
- shutdown_timeout: float = 10)
+ shutdown_timeout: float = 30)
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L79)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L130)
@@ -936,7 +948,7 @@ to gracefully shutdown
def default_topic() -> Topic
```
-[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L152)
+[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L204)
Uses the file structure to generate the desired partition count for the