Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix quantized cache #30

Merged
merged 32 commits into from
Dec 11, 2024
Merged

Fix quantized cache #30

merged 32 commits into from
Dec 11, 2024

Conversation

maxjeblick
Copy link
Collaborator

This PR fixes the failing quantized cache for answers > 1.

  • Quantized cache also uses a key/value cache (of dummy 0s) that needs to be pruned after answer generation. This caused the original issue.
  • Quantized cache uses _seen_tokens to determine the cache sequence length, adjust this value to be in sync with the actual kv cache.
  • Added several tests (including integration tests for ruler)

Fixes #27

@maxjeblick
fix generate_answer for quantized cache
cfa4e58
@maxjeblick
fix value chache pruning
190b3c4
@maxjeblick
improve test
4255ba1
@maxjeblick
improve test
d766ce9
@maxjeblick
add integration tests
9c30df1
@maxjeblick
Merge branch 'main' into max/fix_quanto_cache
4df78ef 
# Conflicts:
#	kvpress/pipeline.py
#	kvpress/presses/base_press.py
#	tests/presses/test_presses.py
#	tests/test_per_layer_compression_wrapper.py
#	tests/test_pipeline.py
@maxjeblick
get correct context length
f753eae
@maxjeblick
fix qunatized key cache
7649ab9
@maxjeblick
fix qunatized key cache
11a4c45
@maxjeblick
fix test
cfec9b8
@maxjeblick
fix test
1bfe667
@maxjeblick
fix test
d46cc55
@maxjeblick
add more asserts
524fe47
@maxjeblick
fix test
c07cc4d
@maxjeblick
fix test
6ff4bc9
@maxjeblick
fix test
960e05e
@maxjeblick
Merge branch 'main' into max/fix_quanto_cache
5c5e1b9 
# Conflicts:
#	kvpress/__init__.py
#	kvpress/presses/scorer_press.py
@maxjeblick
fix merge conflicts
58bc8ae
@maxjeblick
fix failing tests
16e8671
@maxjeblick
import flash attn skip
d9887ea
@maxjeblick
fix test
f5d9d59
@maxjeblick
add integration tests
c620cb0
@maxjeblick
add integration tests
ca4f0ba
@maxjeblick
add integration tests
702888a
@maxjeblick
add fixture
4369c02
@maxjeblick
easen up test
22c9549
@maxjeblick
undo vvariable extraction
4b66477
@maxjeblick
undo newlines
8006a75
@maxjeblick maxjeblick requested a review from SimJeg December 10, 2024 17:10
@maxjeblick maxjeblick mentioned this pull request Dec 11, 2024
Merged
SimJeg
SimJeg reviewed Dec 11, 2024
View reviewed changes
tests/integration/test_ruler.py Outdated Show resolved Hide resolved
tests/integration/test_ruler.py Show resolved Hide resolved
@maxjeblick maxjeblick merged commit 7503f0d into main Dec 11, 2024
2 checks passed
@maxjeblick maxjeblick deleted the max/fix_quanto_cache branch December 11, 2024 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SimJeg SimJeg SimJeg approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

QuantoQuantizedCache fails with number of questions > 1
2 participants
@maxjeblick @SimJeg