Skip to content

Commit

Permalink
Update documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@gwaramadze @github-actions
gwaramadze authored and github-actions[bot] committed Jan 13, 2025
1 parent 53515e5 commit 1cc10d6
Show file tree
Hide file tree
Showing 3 changed files with 69 additions and 45 deletions.
26 changes: 13 additions & 13 deletions docs/api-reference/dataframe.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1008,7 +1008,7 @@ operations, but branches can still be generated from its originating SDF.
class StreamingSeries(BaseStreaming)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L70)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L60)

`StreamingSeries` are typically generated by `StreamingDataframes` when getting
elements from, or performing certain operations on, a `StreamingDataframe`,
Expand Down Expand Up @@ -1075,7 +1075,7 @@ def from_apply_callback(cls, func: ApplyWithMetadataCallback,
sdf_id: int) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L132)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L126)

Create a StreamingSeries from a function.

Expand Down Expand Up @@ -1104,7 +1104,7 @@ instance of `StreamingSeries`
def apply(func: ApplyCallback) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L155)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L149)

Add a callable to the execution list for this series.

Expand Down Expand Up @@ -1156,7 +1156,7 @@ a new `StreamingSeries` with the new callable added
def compose_returning() -> ReturningExecutor
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L189)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L183)

Compose a list of functions from this StreamingSeries and its parents into one

Expand Down Expand Up @@ -1187,7 +1187,7 @@ def compose(
None]] = None) -> VoidExecutor
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L204)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L198)

Compose all functions of this StreamingSeries into one big closure.

Expand Down Expand Up @@ -1245,7 +1245,7 @@ def test(value: Any,
ctx: Optional[MessageContext] = None) -> Any
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L248)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L242)

A shorthand to test `StreamingSeries` with provided value

Expand Down Expand Up @@ -1277,7 +1277,7 @@ result of `StreamingSeries`
def isin(other: Container) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L304)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L313)

Check if series value is in "other".

Expand Down Expand Up @@ -1322,7 +1322,7 @@ new StreamingSeries
def contains(other: Union[Self, object]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L331)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L344)

Check if series value contains "other"

Expand Down Expand Up @@ -1367,7 +1367,7 @@ new StreamingSeries
def is_(other: Union[Self, object]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L356)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L369)

Check if series value refers to the same object as `other`

Expand Down Expand Up @@ -1409,7 +1409,7 @@ new StreamingSeries
def isnot(other: Union[Self, object]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L379)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L392)

Check if series value does not refer to the same object as `other`

Expand Down Expand Up @@ -1452,7 +1452,7 @@ new StreamingSeries
def isnull() -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L403)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L416)

Check if series value is None.

Expand Down Expand Up @@ -1489,7 +1489,7 @@ new StreamingSeries
def notnull() -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L426)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L439)

Check if series value is not None.

Expand Down Expand Up @@ -1526,7 +1526,7 @@ new StreamingSeries
def abs() -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L449)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L462)

Get absolute value of the series value.

Expand Down
68 changes: 40 additions & 28 deletions docs/api-reference/quixstreams.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1641,7 +1641,7 @@ operations, but branches can still be generated from its originating SDF.
class StreamingSeries(BaseStreaming)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L70)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L60)

`StreamingSeries` are typically generated by `StreamingDataframes` when getting
elements from, or performing certain operations on, a `StreamingDataframe`,
Expand Down Expand Up @@ -1700,7 +1700,7 @@ def from_apply_callback(cls, func: ApplyWithMetadataCallback,
sdf_id: int) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L132)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L126)

Create a StreamingSeries from a function.

Expand All @@ -1723,7 +1723,7 @@ instance of `StreamingSeries`
def apply(func: ApplyCallback) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L155)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L149)

Add a callable to the execution list for this series.

Expand Down Expand Up @@ -1767,7 +1767,7 @@ a new `StreamingSeries` with the new callable added
def compose_returning() -> ReturningExecutor
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L189)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L183)

Compose a list of functions from this StreamingSeries and its parents into one

Expand All @@ -1794,7 +1794,7 @@ def compose(
None]] = None) -> VoidExecutor
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L204)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L198)

Compose all functions of this StreamingSeries into one big closure.

Expand Down Expand Up @@ -1844,7 +1844,7 @@ def test(value: Any,
ctx: Optional[MessageContext] = None) -> Any
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L248)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L242)

A shorthand to test `StreamingSeries` with provided value

Expand All @@ -1870,7 +1870,7 @@ result of `StreamingSeries`
def isin(other: Container) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L304)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L313)

Check if series value is in "other".

Expand Down Expand Up @@ -1907,7 +1907,7 @@ new StreamingSeries
def contains(other: Union[Self, object]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L331)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L344)

Check if series value contains "other"

Expand Down Expand Up @@ -1944,7 +1944,7 @@ new StreamingSeries
def is_(other: Union[Self, object]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L356)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L369)

Check if series value refers to the same object as `other`

Expand Down Expand Up @@ -1978,7 +1978,7 @@ new StreamingSeries
def isnot(other: Union[Self, object]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L379)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L392)

Check if series value does not refer to the same object as `other`

Expand Down Expand Up @@ -2013,7 +2013,7 @@ new StreamingSeries
def isnull() -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L403)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L416)

Check if series value is None.

Expand Down Expand Up @@ -2044,7 +2044,7 @@ new StreamingSeries
def notnull() -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L426)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L439)

Check if series value is not None.

Expand Down Expand Up @@ -2075,7 +2075,7 @@ new StreamingSeries
def abs() -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L449)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/series.py#L462)

Get absolute value of the series value.

Expand Down Expand Up @@ -2370,7 +2370,7 @@ class FixedTimeWindow()
def final() -> "StreamingDataFrame"
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L129)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L127)

Apply the window aggregation and return results only when the windows are
closed.
Expand Down Expand Up @@ -2401,7 +2401,7 @@ can remain unprocessed until the message the same key is received.
def current() -> "StreamingDataFrame"
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L167)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/time_based.py#L165)

Apply the window transformation to the StreamingDataFrame to return results
for each updated window.
Expand Down Expand Up @@ -2429,7 +2429,7 @@ regardless of whether the window is closed or not.
```python
def get_window_ranges(timestamp_ms: int,
duration_ms: int,
step_ms: Optional[int] = None) -> Deque[Tuple[int, int]]
step_ms: Optional[int] = None) -> Deque[tuple[int, int]]
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/windows/base.py#L17)
Expand Down Expand Up @@ -2666,7 +2666,7 @@ def add_apply(func: Union[
metadata: bool = False) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L186)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L190)

Add an "apply" function to the Stream.

Expand Down Expand Up @@ -2697,7 +2697,7 @@ def add_update(func: Union[UpdateCallback, UpdateWithMetadataCallback],
metadata: bool = False) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L227)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L231)

Add an "update" function to the Stream, that will mutate the input value.

Expand Down Expand Up @@ -2725,7 +2725,7 @@ def add_transform(func: Union[TransformCallback, TransformExpandedCallback],
expand: bool = False) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L261)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L265)

Add a "transform" function to the Stream, that will mutate the input value.

Expand Down Expand Up @@ -2754,7 +2754,7 @@ a new Stream derived from the current one
def diff(other: Self) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L285)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L289)

Takes the difference between Streams `self` and `other` based on their last

Expand Down Expand Up @@ -2787,7 +2787,7 @@ a new independent `Stream` instance whose root begins at the diff
def root_path(allow_splits=True) -> List[Self]
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L350)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L354)

Return a list of all parent Streams including the node itself.

Expand All @@ -2807,7 +2807,7 @@ a list of `Stream` objects
def full_tree() -> List[Self]
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L371)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L375)

Starts at tree root and finds every Stream in the tree (including splits).

Expand All @@ -2829,7 +2829,7 @@ def compose(
None]] = None) -> VoidExecutor
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L378)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L382)

Generate an "executor" closure by mapping all relatives of this `Stream` and

Expand Down Expand Up @@ -2861,7 +2861,7 @@ the stream has transform functions in the tree. Default - True.
def compose_returning() -> ReturningExecutor
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L435)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/core/stream/stream.py#L439)

Compose a list of functions from this `Stream` and its parents into one
big closure that always returns the transformed record.
Expand Down Expand Up @@ -11184,6 +11184,18 @@ The iterable should output dicts with the following data/naming structure:

## quixstreams.sources.community.file.file

<a id="quixstreams.sources.community.file.file.FileFetcher"></a>

### FileFetcher

```python
class FileFetcher()
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L22)

Serves individual files while downloading another in the background.

<a id="quixstreams.sources.community.file.file.FileSource"></a>

### FileSource
Expand All @@ -11192,7 +11204,7 @@ The iterable should output dicts with the following data/naming structure:
class FileSource(Source)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L19)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L70)

Ingest a set of files from a desired origin into Kafka by iterating through the
provided folder and processing all nested files within it.
Expand Down Expand Up @@ -11262,10 +11274,10 @@ def __init__(directory: Union[str, Path],
compression: Optional[CompressionName] = None,
replay_speed: float = 1.0,
name: Optional[str] = None,
shutdown_timeout: float = 10)
shutdown_timeout: float = 30)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L79)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L130)

**Arguments**:

Expand All @@ -11292,7 +11304,7 @@ to gracefully shutdown
def default_topic() -> Topic
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L152)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/community/file/file.py#L204)

Uses the file structure to generate the desired partition count for the

Expand Down
Loading

0 comments on commit 1cc10d6

Please sign in to comment.