Skip to content

Commit

Permalink
add code block to documentation
Browse files Browse the repository at this point in the history
PiperOrigin-RevId: 331557289
  • Loading branch information
andylou2 authored and tfx-copybara committed Sep 14, 2020
1 parent 74ee9bc commit b1b9cff
Show file tree
Hide file tree
Showing 15 changed files with 105 additions and 8 deletions.
10 changes: 10 additions & 0 deletions struct2tensor/expression_impl/broadcast.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
Suppose you have an expr representing:
```
+
|
+-session* (stars indicate repeated)
Expand All @@ -38,9 +39,17 @@
event: {}
val: 20
}
```
Then:
```
broadcast.broadcast(expr, path.Path(["session","val"]), "event", "nv")
```
becomes:
```
+
|
+---session* (stars indicate repeated)
Expand Down Expand Up @@ -68,6 +77,7 @@
event: {nv: 20}
val: 20
}
```
"""

Expand Down
4 changes: 4 additions & 0 deletions struct2tensor/expression_impl/depth_limit.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,26 @@
Suppose you have an expression expr modeled as:
```
*
\
A
/ \
D B
\
C
```
if expr_2 = depth_limit.limit_depth(expr, 2)
You get:
```
*
\
A
/ \
D B
```
"""

Expand Down
5 changes: 4 additions & 1 deletion struct2tensor/expression_impl/filter_expression.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
Beginning with the struct:
```
root =
-----*----------------------------------------------------
/ \ \
Expand All @@ -45,15 +46,17 @@
new_doc0-----------
\ \ \
bar:"b" bar:"c" keep_me:True
```
filter_by_sibling allows you to filter an expression by a optional boolean
child field.
The following call will have the same effect as above:
```
root_2 = filter_expression.filter_by_child(
root, path.create_path("doc"), "keep_me", "new_doc")
```
"""

Expand Down
18 changes: 16 additions & 2 deletions struct2tensor/expression_impl/index.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
other elements.
Given:
```
session: {
event: {
val: 111
Expand All @@ -37,9 +39,15 @@
val: 1
}
}
```
```
get_positional_index(expr, path.Path(["event","val"]), "val_index")
```
get_positional_index(expr, path.Path(["event","val"]), "val_index") yields:
yields:
```
session: {
event: {
val: 111
Expand All @@ -65,9 +73,14 @@
val_index: 0
}
}
```
get_index_from_end(expr, path.Path(["event","val"]), "neg_val_index") yields:
```
get_index_from_end(expr, path.Path(["event","val"]), "neg_val_index")
```
yields:
```
session: {
event: {
val: 111
Expand All @@ -93,6 +106,7 @@
neg_val_index: -1
}
}
```
These methods are useful when you want to depend upon the index of a field.
For example, if you want to filter examples based upon their index, or
Expand Down
12 changes: 11 additions & 1 deletion struct2tensor/expression_impl/map_prensor.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
and map_ragged_tensor.
Assume expr is:
```
session: {
event: {
val_a: 10
Expand All @@ -35,12 +37,15 @@
val_b: 5
}
}
```
Either of the following alternatives will add val_a and val_b
to create val_sum.
map_sparse_tensor converts val_a and val_b to sparse tensors,
and then add them to produce val_sum.
```
new_root = map_prensor.map_sparse_tensor(
expr,
path.Path(["event"]),
Expand All @@ -49,9 +54,12 @@
False,
tf.int32,
"val_sum")
```
map_ragged_tensor converts val_a and val_b to ragged tensors,
and then add them to produce val_sum.
```
new_root = map_prensor.map_ragged_tensor(
expr,
path.Path(["event"]),
Expand All @@ -60,9 +68,11 @@
False,
tf.int32,
"val_sum")
```
The result of either is:
```
session: {
event: {
val_a: 10
Expand All @@ -85,7 +95,7 @@
val_sum: 5
}
}
```
"""

Expand Down
11 changes: 10 additions & 1 deletion struct2tensor/expression_impl/map_prensor_to_prensor.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,23 +20,30 @@
For example, suppose you have an op my_op, that takes a prensor of the form:
```
event
/ \
foo bar
```
and produces a prensor of the form my_result_schema:
```
event
/ \
foo2 bar2
```
```
my_result_schema = create_schema(
is_repeated=True,
children={"foo2":{is_repeated:True, dtype:tf.int64},
"bar2":{is_repeated:False, dtype:tf.int64}})
```
If you give it an expression original with the schema:
```
session
|
event
Expand All @@ -48,15 +55,17 @@
path.Path(["session","event"]),
my_op,
my_result_schema)
```
Result will have the schema:
```
session
|
event--------
/ \ \ \
foo bar foo2 bar2
```
"""

Expand Down
3 changes: 3 additions & 0 deletions struct2tensor/expression_impl/parquet.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,16 @@
"""Apache Parquet Dataset.
Example usage:
```
exp = create_expression_from_parquet_file(filenames)
docid_project_exp = project.project(exp, [path.Path(["DocId"])])
pqds = parquet_dataset.calculate_parquet_values([docid_project_exp], exp,
filenames, batch_size)
for prensors in pqds:
doc_id_prensor = prensors[0]
```
"""

Expand Down
7 changes: 7 additions & 0 deletions struct2tensor/expression_impl/parse_message_level_ex.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
Specifically, consider google.protobuf.Any and proto maps:
```
package foo.bar;
message MyMessage {
Expand All @@ -35,19 +36,24 @@
int32 my_int = 1;
...
}
```
Then for MyMessage, the path my_any.(foo.bar.Baz).my_int is an optional path.
Also, my_map[x].my_int is an optional path.
```
MyMessage--------------
\ my_any? \ my_map[x]
* *
\ (foo.bar.Baz)? \ my_int?
* *
\ my_int?
*
```
Thus, we can run:
```
my_message_serialized_tensor = ...
my_message_parsed = parse_message_level_ex(
Expand All @@ -61,6 +67,7 @@
my_any_serialized,
Any.DESCRIPTOR,
{"(foo.bar.Baz)"})
```
At this point, my_message_parsed["my_map[x]"].value AND
my_any_parsed["(foo.bar.Baz)"].value are serialized Baz tensors.
Expand Down
3 changes: 3 additions & 0 deletions struct2tensor/expression_impl/placeholder.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,14 @@
will bind the prensor with the appropriate placeholder expression.
Sample usage:
```
placeholder_exp = placeholder.create_expression_from_schema(schema)
new_exp = expression_queries(placeholder_exp, ..)
result = calculate.calculate_values([new_exp],
feed_dict={placeholder_exp: pren})
# placeholder_exp requires a feed_dict to be passed in when calculating
```
"""

Expand Down
2 changes: 2 additions & 0 deletions struct2tensor/expression_impl/project.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,12 @@
Example:
```
expr = ...
new_expr = project.project(expr, [path.Path(["foo","bar"]),
path.Path(["x", "y"])])
[prensor_result] = calculate.calculate_prensors([new_expr])
```
prensor_result now has two paths, "foo.bar" and "x.y".
Expand Down
12 changes: 10 additions & 2 deletions struct2tensor/expression_impl/promote.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
For example, suppose an expr represents:
```
+
|
+-session* (stars indicate repeated)
Expand Down Expand Up @@ -48,8 +48,15 @@
}
}
promote.promote(expr, path.Path(["session", "event", "val"]), nval) produces:
```
```
promote.promote(expr, path.Path(["session", "event", "val"]), nval)
```
produces:
```
+
|
+-session* (stars indicate repeated)
Expand Down Expand Up @@ -85,6 +92,7 @@
nval: 7
nval: 1
}
```
"""

Expand Down
Loading

0 comments on commit b1b9cff

Please sign in to comment.