Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Redis Storage and add JSON support #78

Merged
merged 37 commits into from
Nov 21, 2023
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
32f8a4c
initial JSON pass
tylerhutcherson Oct 19, 2023
5d278bc
fix tests
tylerhutcherson Nov 1, 2023
8c9240a
reformat and lint
tylerhutcherson Nov 2, 2023
de709c8
add checks for bytes when loading
tylerhutcherson Nov 2, 2023
4f14b58
factor out storage into separate module
tylerhutcherson Nov 8, 2023
abc9940
updates to schema to enable storage
tylerhutcherson Nov 8, 2023
c10f92d
harden storage and index classes
tylerhutcherson Nov 8, 2023
47088fd
remove unused import
tylerhutcherson Nov 9, 2023
8e93b0e
storage docstrings and updates
tylerhutcherson Nov 9, 2023
83ee19a
update tests
tylerhutcherson Nov 9, 2023
057127e
update code to fix tests
tylerhutcherson Nov 9, 2023
579c37e
formatting
tylerhutcherson Nov 9, 2023
f410d40
address json issues and update tests
tylerhutcherson Nov 10, 2023
ddc4a09
add query unit tests
tylerhutcherson Nov 10, 2023
e98a799
update search index api ordering
tylerhutcherson Nov 13, 2023
414c495
pin openai version
tylerhutcherson Nov 13, 2023
f3a8cb9
user guide updates
tylerhutcherson Nov 13, 2023
e8fa418
update index property docstrings
tylerhutcherson Nov 13, 2023
4493bd8
simple json.hash guide
tylerhutcherson Nov 13, 2023
80679b6
more updates
tylerhutcherson Nov 14, 2023
deae314
fix formatting, still need to do mypy
tylerhutcherson Nov 14, 2023
91543a2
fix tests
tylerhutcherson Nov 16, 2023
e4919cd
add new test file
tylerhutcherson Nov 16, 2023
28b65a4
Format
Nov 17, 2023
7b86de7
all test pass
Nov 17, 2023
194bd50
cleanup, formatting, and tests
tylerhutcherson Nov 17, 2023
5cd5f87
unify redis_url usage
tylerhutcherson Nov 17, 2023
cd4b625
address comments
tylerhutcherson Nov 17, 2023
74eed62
initial pass at holding schema directly
tylerhutcherson Nov 20, 2023
77777ad
use index method wrappers and add async support
tylerhutcherson Nov 20, 2023
54ff792
WIP updates reverting schema refactor and wrapping doc strings
tylerhutcherson Nov 21, 2023
d21943e
move utils decorators and rename schema gen
tylerhutcherson Nov 21, 2023
4b6e5b9
fix comment wrapping and spacing
tylerhutcherson Nov 21, 2023
5d622b6
get tests to pass
tylerhutcherson Nov 21, 2023
0684735
final pass at tests and formatting
tylerhutcherson Nov 21, 2023
ab2f3ff
notebook re runs
tylerhutcherson Nov 21, 2023
5206960
Remove storage reference
tylerhutcherson Nov 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/_extension/gallery_directive.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
"""A directive to generate a gallery of images from structured data.

Generating a gallery of images that are all the same size is a common
pattern in documentation, and this can be cumbersome if the gallery is
generated programmatically. This directive wraps this particular use-case
in a helper-directive to generate it with a single YAML configuration file.
Generating a gallery of images that are all the same size is a common pattern in
documentation, and this can be cumbersome if the gallery is generated
programmatically. This directive wraps this particular use-case in a helper-
directive to generate it with a single YAML configuration file.

It currently exists for maintainers of the pydata-sphinx-theme,
but might be abstracted into a standalone package if it proves useful.
It currently exists for maintainers of the pydata-sphinx-theme, but might be
abstracted into a standalone package if it proves useful.
"""
from pathlib import Path
from typing import Any, Dict, List
Expand Down
29 changes: 20 additions & 9 deletions docs/api/searchindex.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,23 @@ SearchIndex
.. autosummary::

SearchIndex.__init__
SearchIndex.client
SearchIndex.name
SearchIndex.prefix
SearchIndex.key_separator
SearchIndex.storage
tylerhutcherson marked this conversation as resolved.
Show resolved Hide resolved
SearchIndex.storage_type
SearchIndex.from_yaml
SearchIndex.from_dict
SearchIndex.from_existing
SearchIndex.connect
SearchIndex.create
SearchIndex.load
SearchIndex.search
SearchIndex.query
SearchIndex.create
SearchIndex.delete
SearchIndex.load
SearchIndex.client
SearchIndex.connect
SearchIndex.disconnect
SearchIndex.info
SearchIndex.disconnect



Expand All @@ -44,17 +49,23 @@ AsyncSearchIndex
.. autosummary::

AsyncSearchIndex.__init__
AsyncSearchIndex.client
AsyncSearchIndex.name
AsyncSearchIndex.prefix
AsyncSearchIndex.key_separator
AsyncSearchIndex.storage
AsyncSearchIndex.storage_type
AsyncSearchIndex.from_yaml
AsyncSearchIndex.from_dict
AsyncSearchIndex.from_existing
AsyncSearchIndex.connect
AsyncSearchIndex.create
AsyncSearchIndex.load
AsyncSearchIndex.search
AsyncSearchIndex.query
AsyncSearchIndex.create
AsyncSearchIndex.delete
AsyncSearchIndex.load
AsyncSearchIndex.connect
AsyncSearchIndex.disconnect
AsyncSearchIndex.info
AsyncSearchIndex.disconnect



Expand Down
2 changes: 1 addition & 1 deletion docs/examples/openai_qna.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
"source": [
"# first we need to install a few things\n",
"\n",
"!pip install pandas wget tenacity tiktoken openai"
"!pip install pandas wget tenacity tiktoken openai==0.28.1"
]
},
{
Expand Down
6 changes: 5 additions & 1 deletion docs/user_guide/getting_started_01.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@
"index:\n",
" name: user_index\n",
" prefix: user\n",
" storage_type: hash\n",
" key_separator: ':'\n",
"\n",
"fields:\n",
" # define tag fields\n",
Expand Down Expand Up @@ -162,6 +164,8 @@
" \"index\": {\n",
" \"name\": \"user_index\",\n",
" \"prefix\": \"user\",\n",
" \"storage_type\": \"hash\",\n",
" \"key_separator\": \":\"\n",
" },\n",
" \"fields\": {\n",
" \"tag\": [{\"name\": \"credit_score\"}],\n",
Expand Down Expand Up @@ -465,7 +469,7 @@
],
"source": [
"# create a new SearchIndex instance from an existing index\n",
"existing_index = SearchIndex.from_existing(\"user_index\", \"redis://localhost:6379\")\n",
"existing_index = SearchIndex.from_existing(name=\"user_index\", redis_url=\"redis://localhost:6379\")\n",
"\n",
"# run the same query\n",
"results = existing_index.query(query)\n",
Expand Down
Loading
Loading