Skip to content

Commit

Permalink
Reintroduce some of blaze2004's changes to tokenizer file:
Browse files Browse the repository at this point in the history 
* Refactor tokenizer into a class
* Allow passing custom vocab and merge data
* Allow passing custom tests
  • Loading branch information
@belladoreai
belladoreai committed Mar 23, 2024
1 parent 79acbc4 commit 27bd8ed
Showing 1 changed file with 355 additions and 348 deletions.
Loading

0 comments on commit 27bd8ed

Please sign in to comment.