Add search diagnostics to deadline exceeded exceptions #621
+67
−3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sarthakn7
reviewed
Feb 3, 2024
sarthakn7
reviewed
Feb 5, 2024
| @@ -191,7 +191,8 @@ public SearchResponse handle(IndexState indexState, SearchRequest searchRequest) | |||
|
|
|||
| diagnostics.setFirstPassSearchTimeMs(((System.nanoTime() - searchStartTime) / 1000000.0)); | |||
|
|
|||
| DeadlineUtils.checkDeadline("SearchHandler: post recall", "SEARCH"); | |||
| DeadlineUtils.checkDeadline( | |||
There was a problem hiding this comment.
Maybe add a diagnostics parameter to checkDeadline function, and append it to the message only when the error is being thrown? You can also consider keeping the existing function and adding a new checkDeadline with additional parameter in the signature.
There was a problem hiding this comment.
The method is getting used in many places in the code, so I will create a new method with the additional param.
swekannan
commented
Feb 5, 2024
| @@ -191,7 +191,7 @@ public SearchResponse handle(IndexState indexState, SearchRequest searchRequest) | |||
|
|
|||
| diagnostics.setFirstPassSearchTimeMs(((System.nanoTime() - searchStartTime) / 1000000.0)); | |||
|
|
|||
| DeadlineUtils.checkDeadline("SearchHandler: post recall", "SEARCH"); | |||
| DeadlineUtils.checkDeadline("SearchHandler: post recall", diagnostics.build(), "SEARCH"); | |||
There was a problem hiding this comment.
@sarthakn7 I am a bit unsure about this part. Is it okay to log the build diagnostics or should I do some string conversion before logging? What would be the preferred way?
There was a problem hiding this comment.
You can convert it to json before printing if you want to, but it's toString is also fine. You can also just log the Builder object rather than calling build.
There was a problem hiding this comment.
Ok. I used toString() in this case
sarthakn7
reviewed
Feb 5, 2024
| Deadline deadline = Context.current().getDeadline(); | ||
| if (deadline != null && deadline.isExpired()) { | ||
| DeadlineMetrics.nrtDeadlineCancelCount.labels(operation).inc(); | ||
| System.out.println(diagnostics); |
| @@ -191,7 +191,7 @@ public SearchResponse handle(IndexState indexState, SearchRequest searchRequest) | |||
|
|
|||
| diagnostics.setFirstPassSearchTimeMs(((System.nanoTime() - searchStartTime) / 1000000.0)); | |||
|
|
|||
| DeadlineUtils.checkDeadline("SearchHandler: post recall", "SEARCH"); | |||
| DeadlineUtils.checkDeadline("SearchHandler: post recall", diagnostics.build(), "SEARCH"); | |||
There was a problem hiding this comment.
You can convert it to json before printing if you want to, but it's toString is also fine. You can also just log the Builder object rather than calling build.
sarthakn7
approved these changes
Feb 7, 2024
sarthakn7
added a commit
that referenced
this pull request
Mar 1, 2024
* Fix node resolver for empty/missing files (#587) * Autogenerated JaCoCo coverage badge * Bump to v0.26.1 (#588) * Autogenerated JaCoCo coverage badge * parallel explain for nrtsearch (#589) * parallel explain for nrtsearch * fix nit * add explain for multi segment parallel * refactor isExplain in different contexts * make NRTSearch empty boolean query result constant score 1 (#592) * make NRTSearch empty boolean query result constant score 1 * fix tests * soft exception for fvh failures (#593) * Support unit in SortType (#590) create sort context and support unit in lat_lon distance sort * Autogenerated JaCoCo coverage badge * Upgrade dependencies for snyk check (#596) * Fix example-plugin test (#598) * Add page fault metrics (#599) * Add additional merge metrics (#607) * Add live setting for verbose index metrics (#608) * Add live setting for verbose index metrics * Address review comment * Autogenerated JaCoCo coverage badge * Bump to v0.29.0 (#609) * Add live index settings override from config file (#610) * Autogenerated JaCoCo coverage badge * Add ability to update local index live settings (#611) * Add deadline cancellation for indexing (#612) * Add geo polygon query type (#613) * Add geo polygon query type * Detect closed polygons, update docs * Autogenerated JaCoCo coverage badge * Bump to v0.30.0 (#615) * Testing readthedocs web hooks (#616) * Add a bare minimum readthedocs config file (#617) * log more info when fvh failed (#618) log more info when fvh failed * Avoid calling query.toString() (#619) * add sts for web identity auth (#620) add sts for web identity auth * Add search diagnostics to deadline exceeded exceptions (#621) * Add search diagnostics to deadline exceeded exceptions --------- Co-authored-by: swekannan <shwethu.kannan93@gmail.com> * Fixes and spotless apply * Updated grpc-gateway --------- Co-authored-by: Andrew Prudhomme <asp@yelp.com> Co-authored-by: github_actions <runner@fv-az248-700.3twvhzoricxu3figbgb44chhog.cx.internal.cloudapp.net> Co-authored-by: github_actions <runner@fv-az887-622.kyhrjabtieueri5x0cgfkhfc1a.bx.internal.cloudapp.net> Co-authored-by: Tao Yu <taoyu@yelp.com> Co-authored-by: waziqi89 <89210409+waziqi89@users.noreply.github.com> Co-authored-by: github_actions <runner@fv-az248-372.3twvhzoricxu3figbgb44chhog.cx.internal.cloudapp.net> Co-authored-by: github_actions <runner@fv-az574-753.esnmgxn14wlejbqjnvhsbsbtxa.dx.internal.cloudapp.net> Co-authored-by: github_actions <runner@fv-az1272-720.grsihaubamwerhiryzjrxtypna.phxx.internal.cloudapp.net> Co-authored-by: github_actions <runner@fv-az736-601.alpbqrzxv30uzkvtn2qktnuusd.cx.internal.cloudapp.net> Co-authored-by: Mohammad Mohtasham <mohm@yelp.com> Co-authored-by: swekannan <62616258+swekannan@users.noreply.github.com> Co-authored-by: swekannan <shwethu.kannan93@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add search diagnostics for deadline exceeded exceptions for easier debugging of latencies.