Should the UtfCodepoint type be renamed?

[mooreryan]

I wonder if the UtfCodepoint type and its associated functions in the stdlib string module should be renamed.

I haven't seen the term UTF codepoint used outside of Gleam's stdlib and prelude really. That doesn't necessarily mean that the term is technically incorrect, but it does seem to indicate that it may not be the best term to use.

The UtfCodepoint type and its functions from the stdlib are dealing with what is generally called "unicode code point" or just "code point". For example, in these docs the thing that Gleam calls UtfCodepoint is called a unicode code point or simply code point, rather than UTF code point.

• Chapter 2, Section 4 -- Code Points and Characters of the Unicode core spec
• [Chapter 3 -- Conformance] of the Unicode core spec
• The Unicode glossary
• Using Unicode in Erlang
• The Erlang unicode module docs
• Elixir's String module

I've pulled a couple of definitions from the Unicode glossary, for reference.

Code Point. (1) Any value in the Unicode codespace; that is, the range of integers from 0 to 10FFFF16. (See definition D10 in Section 3.4, Characters and Encoding.) Not all code points are assigned to encoded characters. See code point type. (2) A value, or position, for a character, in any coded character set.

UTF. Acronym for Unicode (or UCS) Transformation Format.

Character Encoding Scheme. A character encoding form plus byte serialization. There are seven character encoding schemes in Unicode: UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, and UTF-32LE.

Potential Alternative Names

So it seems that a better name may be something like

• UnicodeCodepoint (or UnicodeCodePoint): explicit, and seems to follow with accepted definitions, but potentially awkward sounding
• Codepoint (or CodePoint): shorter and nicer to type, but it does assume that we're working in the Unicode codespace (which is probably a fine assumption for Gleam)
• Ucp: short and looks nice in the function names, but maybe not very gleamy to introduce an acronym like this

Current functions in the stdlib:

• from_utf_codepoints
• to_utf_codepoints
• utf_codepoint
• utf_codepoint_to_int

Could look like this with one of those new names:

• from_unicode_codepoints
• to_unicode_codepoints
• unicode_codepoint
• unicode_codepoint_to_int

or

• from_codepoints
• to_codepoints
• codepoint
• codepoint_to_int

or

• from_ucp
• to_ucp
• ucp
• ucp_to_int

Summary

• UtfCodepoint does not seem to follow with the general usage of the terms involved (unicode, UTF, code point).
• It is likely that a different name for the type and its associated functions would better reflect its semantics.

[mooreryan]

Another data point: the Gleam compiler error for a bad unicode code point refers to "Invalid Unicode codepoint" rather than "Invalid UTF codepoint".

error: Syntax error
   ┌─ /src/main.gleam:18:18
   │
18 │   io.debug("yo - \u{D800}")  
   │                  ^^^^^^^^ Invalid Unicode codepoint

(That error was generated on the Gleam tour page.)

[lpil]

Hello! Thank you for this thorough write up.

I agree with you that the current name is a mistake, and it would be good to correct that.

I spoke to the team and we're not in agreement that it's worth the deprecation. The old name would continue to exist until Gleam v2 (which may never happen), emitting a warning if used, and that's some baggage the language ideally would not have.

Do you have thoughts on this?

[mooreryan]

I do think the name is not the correct one, but it is a good point about the deprecation potentially being more trouble than it is worth.

I wonder how many others find the name confusing...maybe it has not really been a problem in practice?

[inoas-nbw]

The old name would continue to exist until Gleam v2 (which may never happen), emitting a warning if used, and that's some baggage the language ideally would not have.

That sounds great! stdlib would be updated to use the new Type I suppose?