Use 50, 200, 500 max token counts (PRIME-654) #1137
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to investigate https://linear.app/sourcegraph/issue/PRIME-654/investigate-why-the-fastpathnon-fastpath-diff-decreases-with-longer, and answer the question of whether the back end seems to contribute to our observations there (increased latency diff between SG→CG and direct CG with shorter completions).
It uses the somewhat arbitrary token counts 50, 200, and 500 to simulate single-line, multiline, and long code completions. In all cases, it uses the messages
talking to Claude 3 Haiku (for cost efficiency) at temperature 0.0 and streaming ON.
To fully understand the changes in this PR, see also #1138 which this PR builds on.
Test plan
(I don't intend to merge this.)
Caveat: I was not extremely rigorous in looking into the exact differences between the requests assembled by SG and by the
src-cli
script. I trusted that we successfully converted the same arguments passed to SG to the format CG needs them. Whitespace in the body and some HTTP headers are probably different, though. This can be tested and improved if needed.