Fix bug in BPE cache

[andrew-healey]

bramses reported this bug first in #8.

Recreation

const {encode} = require('gpt-3-encoder')

const str = 'toString'
const encoded = encode(str)
// TypeError: bpe(...).split is not a function

Explanation

The bpe() method of the encoder uses a cache like so:

const cache = {};

// ...

function bpe(token){
  if(token in cache) {
    return cache[token];
  }

  // otherwise, find the right word for the token...

  cache[token] = word
  return word
}

But in has some unexpected behavior:

"toString" in {}; // true
({}).toString // f toString() { [native code] }

So calling this.bpe("toString") returns the function Object.prototype.toString. This causes an error.

Fix

I changed the implementation to use a JS Map, which is a better cache data structure that avoids this bug.

const cache = new Map;
cache.has("toString"); // false

[NickHeiner]

I applied this PR locally, and confirmed that it fixes #8.

[NickHeiner]

I applied this in my fork: https://www.npmjs.com/package/@nick.heiner/gpt-3-encoder.

[asontha]

Can we get this merged?