Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Only index that are needed in indexing store #1193

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

refactor: Only index that are needed in indexing store #1193

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion query/filter/filter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@ var (
type Filter interface {
// Matches returns true if the input doc passes the filter, otherwise false
Matches(doc []byte, metadata []byte) bool
// MatchesDoc similar to Matches but used when document is already parsed
// MatchesDoc similar to Matches but used when document is already parsed. This is mainly used by
// standalone search index.
MatchesDoc(doc map[string]any) bool
ToSearchFilter() string
// IsSearchIndexed to let caller knows if there is any fields in the query not indexed in search. This
Expand Down
4 changes: 4 additions & 0 deletions query/filter/selector.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,10 @@ func (s *Selector) ToSearchFilter() string {
}

func (s *Selector) IsSearchIndexed() bool {
if !s.Field.SearchIndexed {
return false
}

switch {
case s.Field.DataType == schema.DoubleType:
v, ok := s.Matcher.GetValue().(*value.DoubleValue)
Expand Down
47 changes: 12 additions & 35 deletions schema/search.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,25 +458,12 @@ func (s *ImplicitSearchIndex) buildSearchSchema(searchStoreName string) {
tsFields := make([]tsApi.Field, 0, len(s.QueryableFields))

for _, f := range s.QueryableFields {
// the implicit search index by default index all the fields that are indexable and same applies to facet/sort.
shouldIndex := SupportedSearchIndexableType(f.DataType, f.SubType)
shouldFacet := DefaultFacetableType(f.DataType)
shouldSort := DefaultSortableType(f.DataType)

if !shouldSort && f.Sortable {
// honor schema i.e. in case of strings user can explicitly enable sorting.
shouldSort = true
}
if !shouldFacet && f.Faceted {
shouldFacet = true
}

tsFields = append(tsFields, tsApi.Field{
Name: f.Name(),
Type: f.SearchType,
Facet: &shouldFacet,
Index: &shouldIndex,
Sort: &shouldSort,
Facet: &f.Faceted,
Index: &f.SearchIndexed,
Sort: &f.Sortable,
Optional: &ptrTrue,
NumDim: f.Dimensions,
})
Expand All @@ -486,9 +473,9 @@ func (s *ImplicitSearchIndex) buildSearchSchema(searchStoreName string) {
tsFields = append(tsFields, tsApi.Field{
Name: f.InMemoryName(),
Type: f.SearchType,
Facet: &shouldFacet,
Index: &shouldIndex,
Sort: &shouldSort,
Facet: &f.Faceted,
Index: &f.SearchIndexed,
Sort: &f.Sortable,
Optional: &ptrTrue,
})
}
Expand Down Expand Up @@ -531,26 +518,16 @@ func (s *ImplicitSearchIndex) GetSearchDeltaFields(existingFields []*QueryableFi
e := existingFieldMap[f.FieldName]
delete(existingFieldMap, f.FieldName)

shouldIndex := SupportedSearchIndexableType(f.DataType, f.SubType)
shouldFacet := DefaultFacetableType(f.DataType)
shouldSort := DefaultSortableType(f.DataType)
if !shouldSort && f.Sortable {
shouldSort = true
}
if !shouldFacet && f.Faceted {
shouldFacet = true
}

stateChanged := false
if e != nil {
inSearchState, found := fieldsInSearchMap[f.FieldName]
if found && inSearchState.Index != nil && *inSearchState.Index != shouldIndex {
if found && inSearchState.Index != nil && *inSearchState.Index != f.SearchIndexed {
stateChanged = true
}
if found && inSearchState.Facet != nil && *inSearchState.Facet != shouldFacet {
if found && inSearchState.Facet != nil && *inSearchState.Facet != f.Faceted {
stateChanged = true
}
if found && inSearchState.Sort != nil && *inSearchState.Sort != shouldSort {
if found && inSearchState.Sort != nil && *inSearchState.Sort != f.Sortable {
stateChanged = true
}

Expand Down Expand Up @@ -579,9 +556,9 @@ func (s *ImplicitSearchIndex) GetSearchDeltaFields(existingFields []*QueryableFi
tsFields = append(tsFields, tsApi.Field{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We adding to the tsFields here but all the values would be false if we have no search fields. What would then happen for typesense?
I was testing this on the background search branch and what I did was not add the field to the tsFields if there was no search field But this caused the crashes in collection creation

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the fields false and optional true mean storing in search but not indexing. This is needed so we don't need to read from the database when we get search results. You should not change that because that will break the search as then search results won't have full data.

Name: f.FieldName,
Type: f.SearchType,
Facet: &shouldFacet,
Index: &shouldIndex,
Sort: &shouldSort,
Facet: &f.Faceted,
Index: &f.SearchIndexed,
Sort: &f.Sortable,
Optional: &ptrTrue,
NumDim: f.Dimensions,
})
Expand Down