diff --git a/docs/api-reference/dataframe.md b/docs/api-reference/dataframe.md index ce7a51de5..1c80813c1 100644 --- a/docs/api-reference/dataframe.md +++ b/docs/api-reference/dataframe.md @@ -1008,7 +1008,7 @@ operations, but branches can still be generated from its originating SDF. class StreamingSeries(BaseStreaming) ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L70) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L60) `StreamingSeries` are typically generated by `StreamingDataframes` when getting elements from, or performing certain operations on, a `StreamingDataframe`, @@ -1075,7 +1075,7 @@ def from_apply_callback(cls, func: ApplyWithMetadataCallback, sdf_id: int) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L132) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L126) Create a StreamingSeries from a function. @@ -1104,7 +1104,7 @@ instance of `StreamingSeries` def apply(func: ApplyCallback) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L155) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L149) Add a callable to the execution list for this series. @@ -1156,7 +1156,7 @@ a new `StreamingSeries` with the new callable added def compose_returning() -> ReturningExecutor ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L189) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L183) Compose a list of functions from this StreamingSeries and its parents into one @@ -1187,7 +1187,7 @@ def compose( None]] = None) -> VoidExecutor ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L204) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L198) Compose all functions of this StreamingSeries into one big closure. @@ -1245,7 +1245,7 @@ def test(value: Any, ctx: Optional[MessageContext] = None) -> Any ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L248) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L242) A shorthand to test `StreamingSeries` with provided value @@ -1277,7 +1277,7 @@ result of `StreamingSeries` def isin(other: Container) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L304) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L313) Check if series value is in "other". @@ -1322,7 +1322,7 @@ new StreamingSeries def contains(other: Union[Self, object]) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L331) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L344) Check if series value contains "other" @@ -1367,7 +1367,7 @@ new StreamingSeries def is_(other: Union[Self, object]) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L356) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L369) Check if series value refers to the same object as `other` @@ -1409,7 +1409,7 @@ new StreamingSeries def isnot(other: Union[Self, object]) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L379) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L392) Check if series value does not refer to the same object as `other` @@ -1452,7 +1452,7 @@ new StreamingSeries def isnull() -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L403) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L416) Check if series value is None. @@ -1489,7 +1489,7 @@ new StreamingSeries def notnull() -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L426) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L439) Check if series value is not None. @@ -1526,7 +1526,7 @@ new StreamingSeries def abs() -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L449) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L462) Get absolute value of the series value. diff --git a/docs/api-reference/quixstreams.md b/docs/api-reference/quixstreams.md index 9b21e0309..d25cf868d 100644 --- a/docs/api-reference/quixstreams.md +++ b/docs/api-reference/quixstreams.md @@ -1641,7 +1641,7 @@ operations, but branches can still be generated from its originating SDF. class StreamingSeries(BaseStreaming) ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L70) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L60) `StreamingSeries` are typically generated by `StreamingDataframes` when getting elements from, or performing certain operations on, a `StreamingDataframe`, @@ -1700,7 +1700,7 @@ def from_apply_callback(cls, func: ApplyWithMetadataCallback, sdf_id: int) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L132) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L126) Create a StreamingSeries from a function. @@ -1723,7 +1723,7 @@ instance of `StreamingSeries` def apply(func: ApplyCallback) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L155) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L149) Add a callable to the execution list for this series. @@ -1767,7 +1767,7 @@ a new `StreamingSeries` with the new callable added def compose_returning() -> ReturningExecutor ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L189) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L183) Compose a list of functions from this StreamingSeries and its parents into one @@ -1794,7 +1794,7 @@ def compose( None]] = None) -> VoidExecutor ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L204) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L198) Compose all functions of this StreamingSeries into one big closure. @@ -1844,7 +1844,7 @@ def test(value: Any, ctx: Optional[MessageContext] = None) -> Any ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L248) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L242) A shorthand to test `StreamingSeries` with provided value @@ -1870,7 +1870,7 @@ result of `StreamingSeries` def isin(other: Container) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L304) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L313) Check if series value is in "other". @@ -1907,7 +1907,7 @@ new StreamingSeries def contains(other: Union[Self, object]) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L331) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L344) Check if series value contains "other" @@ -1944,7 +1944,7 @@ new StreamingSeries def is_(other: Union[Self, object]) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L356) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L369) Check if series value refers to the same object as `other` @@ -1978,7 +1978,7 @@ new StreamingSeries def isnot(other: Union[Self, object]) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L379) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L392) Check if series value does not refer to the same object as `other` @@ -2013,7 +2013,7 @@ new StreamingSeries def isnull() -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L403) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L416) Check if series value is None. @@ -2044,7 +2044,7 @@ new StreamingSeries def notnull() -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L426) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L439) Check if series value is not None. @@ -2075,7 +2075,7 @@ new StreamingSeries def abs() -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L449) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L462) Get absolute value of the series value. @@ -2370,7 +2370,7 @@ class FixedTimeWindow() def final() -> "StreamingDataFrame" ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L129) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L127) Apply the window aggregation and return results only when the windows are closed. @@ -2401,7 +2401,7 @@ can remain unprocessed until the message the same key is received. def current() -> "StreamingDataFrame" ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L167) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L165) Apply the window transformation to the StreamingDataFrame to return results for each updated window. @@ -2429,7 +2429,7 @@ regardless of whether the window is closed or not. ```python def get_window_ranges(timestamp_ms: int, duration_ms: int, - step_ms: Optional[int] = None) -> Deque[Tuple[int, int]] + step_ms: Optional[int] = None) -> Deque[tuple[int, int]] ``` [[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/base.py#L17) @@ -2666,7 +2666,7 @@ def add_apply(func: Union[ metadata: bool = False) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L186) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L190) Add an "apply" function to the Stream. @@ -2697,7 +2697,7 @@ def add_update(func: Union[UpdateCallback, UpdateWithMetadataCallback], metadata: bool = False) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L227) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L231) Add an "update" function to the Stream, that will mutate the input value. @@ -2725,7 +2725,7 @@ def add_transform(func: Union[TransformCallback, TransformExpandedCallback], expand: bool = False) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L261) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L265) Add a "transform" function to the Stream, that will mutate the input value. @@ -2754,7 +2754,7 @@ a new Stream derived from the current one def diff(other: Self) -> Self ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L285) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L289) Takes the difference between Streams `self` and `other` based on their last @@ -2787,7 +2787,7 @@ a new independent `Stream` instance whose root begins at the diff def root_path(allow_splits=True) -> List[Self] ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L350) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L354) Return a list of all parent Streams including the node itself. @@ -2807,7 +2807,7 @@ a list of `Stream` objects def full_tree() -> List[Self] ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L371) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L375) Starts at tree root and finds every Stream in the tree (including splits). @@ -2829,7 +2829,7 @@ def compose( None]] = None) -> VoidExecutor ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L378) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L382) Generate an "executor" closure by mapping all relatives of this `Stream` and @@ -2861,7 +2861,7 @@ the stream has transform functions in the tree. Default - True. def compose_returning() -> ReturningExecutor ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L435) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L439) Compose a list of functions from this `Stream` and its parents into one big closure that always returns the transformed record. @@ -11184,6 +11184,18 @@ The iterable should output dicts with the following data/naming structure: ## quixstreams.sources.community.file.file + + +### FileFetcher + +```python +class FileFetcher() +``` + +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L22) + +Serves individual files while downloading another in the background. + ### FileSource @@ -11192,7 +11204,7 @@ The iterable should output dicts with the following data/naming structure: class FileSource(Source) ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L19) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L70) Ingest a set of files from a desired origin into Kafka by iterating through the provided folder and processing all nested files within it. @@ -11262,10 +11274,10 @@ def __init__(directory: Union[str, Path], compression: Optional[CompressionName] = None, replay_speed: float = 1.0, name: Optional[str] = None, - shutdown_timeout: float = 10) + shutdown_timeout: float = 30) ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L79) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L130) **Arguments**: @@ -11292,7 +11304,7 @@ to gracefully shutdown def default_topic() -> Topic ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L152) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L204) Uses the file structure to generate the desired partition count for the diff --git a/docs/api-reference/sources.md b/docs/api-reference/sources.md index 7f76c5d9f..148cd30a1 100644 --- a/docs/api-reference/sources.md +++ b/docs/api-reference/sources.md @@ -822,6 +822,18 @@ For other parameters See `quixstreams.sources.kafka.KafkaReplicatorSource` ## quixstreams.sources.community.file.file + + +### FileFetcher + +```python +class FileFetcher() +``` + +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L22) + +Serves individual files while downloading another in the background. + ### FileSource @@ -830,7 +842,7 @@ For other parameters See `quixstreams.sources.kafka.KafkaReplicatorSource` class FileSource(Source) ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L19) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L70) Ingest a set of files from a desired origin into Kafka by iterating through the provided folder and processing all nested files within it. @@ -902,10 +914,10 @@ def __init__(directory: Union[str, Path], compression: Optional[CompressionName] = None, replay_speed: float = 1.0, name: Optional[str] = None, - shutdown_timeout: float = 10) + shutdown_timeout: float = 30) ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L79) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L130)
@@ -936,7 +948,7 @@ to gracefully shutdown def default_topic() -> Topic ``` -[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L152) +[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L204) Uses the file structure to generate the desired partition count for the