Unaligned bit arrays on the JavaScript target

[richard-viney]

Summary 📘

This PR adds support for unaligned bit arrays on the JavaScript target.

In expressions:

• Arbitrary sized integer segments:

  <<1:4>>
  <<12:9-little>>
  <<12:29-big>>
  <<1234:100-little>>

• Arbitrary sized bits segments:

  <<<<0xABCD:15>>:bits-10>>

In patterns:

• Arbitrary sized int segments:

  let assert <<_:7, i:19-little-signed>> = <<0xABCDEF12:26>>

• Sized and unsized bits segments:

  let assert <<_:7, a:bits-3, b:size(14)-bits, c:bits>> = <<0xABCDEF:24, 0x1234:16>>

There is a compiler warning if the above features are used when gleam.toml specifies a version < v1.8.0.

---

Implementation Details 🛠️

• The BitArray class in the prelude now has bitSize, byteSize, bitOffset, and rawBuffer properties.
  • The value of any unused high bits in the buffer's first byte, and any unused low bits in the buffer's final byte, are undefined.
  • Public API for use by FFI code is now: bitSize, byteSize, bitOffset, rawBuffer, byteAt(index).
  • Deprecated APIs still used by existing FFI code: get buffer(), get length(). Using these emits a deprecation warning at runtime, and throws an exception if the bit array is unaligned (i.e. bitOffset isn't zero or bitSize isn't a multiple of 8).
• JSDoc annotations have been added to allow type-checking by adding // @ts-check to the top of the file.

---

Implications for @external JavaScript code 🌍

• All existing JavaScript FFI code that operates on bit arrays needs to be updated to support unaligned bit arrays.
• Code that isn't updated will:
  • Continue to work, provided it is only used with aligned bit arrays.
  • Emit deprecation warnings at runtime on use of BitArray.length and BitArray.buffer.
  • Throw a runtime exception if these deprecated APIs are called on a unaligned bit array.
• No existing code breaks because unaligned bit arrays on JavaScript weren't previously possible.

---

Implications for gleam/stdlib 🤝

• I have the updates for gleam/stdlib ready, mostly affecting gleam/bit_array. It can only be merged once this PR goes in as its tests won't run on stable Gleam. It may be necessary to run the new stdlib tests on nightly for a short period, with them segregated into their own file so they can be included/excluded depending on the active Gleam version. I'll sort that out once this PR makes it through review.
• Future stdlib versions that support unaligned bit arrays on JavaScript work fine on Gleam versions < 1.8.0, there are no compatibility concerns there.

---

Testing 🧪

There's certainly some complexity and tricky bitwise operations here, mostly in the JavaScript prelude. The following has been done to ensure correctness:

• Many new tests added to language_tests.gleam, and test/javascript_prelude.
• Every path and branch through the code that performs slicing, concatenation, and conversion to ints/floats is covered by at least one test.
• Extensive fuzzing has been performed on these operations which validated millions of combinations of bit array contents, segment sizes, offsets, endianness, signedness, etc. on JavaScript against the result on the Erlang target.
• Issues found during fuzz testing were fixed and added to the language tests and prelude tests.

---

✨✨✨

[lpil]

Wow, what a fantastic bit of work! Thank you!

I've not digested these changes properly yet, but I have some initial questions from my first review.

There's a lot of public functions in the API of the bit array class now, but generally the aim is to have none. What is the motivation for having these?

Similar for the deprecated functions, we can remove them.

There's quite a lot of code in the class. Could we move them to free-standing functions and have the generated code only use them if absolutely necessary? That would help with JavaScript bundlers performing dead code elimination.

Some existing tests have been removed or changed, why is this? New features shouldn't alter existing tests, it makes it harder to review and to prevent regressions.

Thanks again!

[richard-viney]

Thanks for taking a look!

There's a lot of public functions in the API of the bit array class now, but generally the aim is to have none. What is the motivation for having these?

These have been reduced down to the following: bitSize, byteSize, and byteAt().

Similar for the deprecated functions, we can remove them.

I've removed all except the BitArray.length and BitArray.buffer` accessors, because removing these would break most/all JS FFI code that operates on bit arrays.

There's quite a lot of code in the class. Could we move them to free-standing functions and have the generated code only use them if absolutely necessary? That would help with JavaScript bundlers performing dead code elimination.

Done.

Some existing tests have been removed or changed, why is this? New features shouldn't alter existing tests, it makes it harder to review and to prevent regressions.

The diff on the tests looked a bit more complex in places than it actually was so I've moved some things around to make it easier to digest. Some existing tests did need to be tweaked or removed, e.g. those that were testing for compilation errors if unaligned bit arrays were used on the JS target.

[richard-viney]

This PR has been updated to make slicing bit arrays an O(1) operation under all circumstances, matching Erlang's performance characteristics. The BitArray.bitOffset member was added to enable this.

The main write-up has been updated for this change.