iTranslated by AI
How to Design Numeric Types When Creating a New Programming Language
This article summarizes my personal views on how to design numeric types when creating a new programming language.
Number Systems
Languages like JavaScript (pre-BigInt) and Lua (up to 5.2) use floating-point numbers as their sole numeric type, representing both integers and real numbers as the same "number" type. While such a design might be acceptable for a minimal language, if you are building a practical language that also considers native compilation, I believe you should avoid conflating integers and real numbers.
In particular, just because you are building a language that compiles to JavaScript, you don't necessarily have to mimic its numeric type design.
Representing integers as floating-point numbers can lead to unexpected performance degradation. Recently (February 2023), the following tweet became a topic of discussion:
This appears to be due to the distinction between positive and negative zero; positive zero is treated as an integer internally, while negative zero is not. Such issues might be unique to runtimes with advanced optimizations, but they are noteworthy nonetheless.
Beyond integer and floating-point types, other numeric types to consider include rational numbers and complex numbers. Scheme, for instance, maintains a hierarchy of integers, rationals, reals (floating-point), and complex numbers, which it calls a "numerical tower." This might serve as a good reference if you want to provide a rich set of numeric types.
Avoid Simply Naming a Type "number"
You should avoid simply naming a numeric type "number." You should choose names that clarify what kind of "number" it represents, such as integer for integers, float or real for floating-point real numbers, and complex for complex numbers.
As a bad example, PureScript calls its floating-point type Number. Since there is an Int type separate from the Number type, Number is strictly intended to be a floating-point type, and it should have been named something like Float, Float64, or Double.
Behavior of Integer Overflow
If you provide fixed-length integers, you must consider what happens during overflow. Possible options include:
- Raise an exception
- Example: Signed integers in Standard ML
- Wrap around
- Example: Java, Lua 5.3
- Convert to a floating-point number
- Example: PHP
- JavaScript's "treating regular integers as floating-point numbers" could be seen as a variation of this.
- Treat as undefined behavior
- Example: Signed integers in C
- Do not provide fixed-length integers at all; rely on arbitrary-precision integers
- Example: Ruby, Python
I believe the educationally sound approaches are either "raise an exception" or "rely on arbitrary-precision integers."
It is easy to imagine that addition, subtraction, and multiplication can overflow, but division, unary negation, and absolute value can also cause overflows, particularly when dealing with the minimum value in two's complement representation.
When implementing wrap-around arithmetic in C, you should avoid undefined behavior. Specifically, you should cast to an unsigned integer or check individually with conditional branches. You might want to refer to the implementation in Lua 5.3.
Floating-Point Numbers
Creating a language runtime requires a certain level of knowledge about floating-point numbers.
Optimization
Implementing optimizations based on the assumption of mathematical real numbers can change the observable behavior of a program. For example, converting x + 0.0 to x changes the result if x is negative zero or a signaling NaN.
Some of the rules regarding floating-point optimizations in GCC are summarized here:
Well, if you consider every single edge case, you will hardly be able to implement any optimizations, so adopting a policy like "ignore signaling NaNs" might be a good approach. If you ignore signaling NaNs, optimizations such as changing x * 1.0 to x or x - 0.0 to x become possible.
For a domain-specific language, ignoring the fine details of floating-point numbers to prioritize optimization is also acceptable. There is little reason to worry about the sign of zero in deep learning applications (unless you are concerned with reproducibility).
The Sign of Zero
Forgetting about negative zero can lead to problems. When outputting a constant table in a program's object code, using == to determine equivalence can result in positive and negative zeros being unified. MinCaml has this issue:
Here are a few properties of the sign of zero that you should at least be aware of:
- They are treated as equivalent by standard equality operators like
==:0.0 == -0.0 - Differences can be observed when passed to functions like reciprocal or
atan2:1.0 / 0.0 != 1.0 / (-0.0)- Therefore, performing optimizations or code generation that conflates positive and negative zeros may change program behavior.
- Negative zero can be generated by operations like
(-1) * 0, underflow(-1e-300) * 1e-300, or sign inversion. In standard rounding modes, negative zero appears as the result of addition or subtraction only if the input already contains a negative zero.
Complex Numbers
When implementing complex numbers composed of floating-point numbers, calculating them strictly according to mathematical formulas can yield undesirable results.
For example, when calculating the absolute value of 1e300 + 1e300i, if you use sqrt(x * x + y * y), the intermediate x * x will overflow, returning infinity even though the true result 1.4142135623730952e+300 is well within the range of double-precision floating-point numbers. This is unnecessary overflow/underflow. You can avoid this by using the hypot function to calculate the absolute value.
Similarly, when calculating 1 / (1e300 + 1e300i) using the definition (x - yi) / (x * x + y * y), the same issue occurs.
Another problem is that when calculating (inf + inf i) * (1 + 0 i), the result should ideally be some form of infinity, but a naive implementation of multiplication results in both the real and imaginary parts becoming NaN.
I have previously addressed these topics on my blog:
- Floating-point functions and overflow
- Caution regarding complex arithmetic with floating-point numbers
- Rating implementations of complex numbers in programming languages
Addressing these issues carries a cost. If calculation speed is critical for your language, it might be beneficial to provide a "mathematical formula calculation mode" that can be used at the programmer's discretion. C has a feature called #pragma STDC CX_LIMITED_RANGE for this purpose.
min / max
When deciding the specification for min and max functions that return the smaller or larger of two floating-point numbers, there are two points of caution.
First, consider whether to handle the sign of zero or ignore it. Defining -0 < +0 (i.e., min(-0, +0) = min(+0, -0) = -0 and max(-0, +0) = max(+0, -0) = +0) is elegant, but naive implementations will not satisfy this property.
Second, what to do about NaNs. There are generally two schools of thought: "propagate the NaN" or "treat NaN as missing input and return the non-NaN input if one exists." A naive implementation will satisfy neither property.
I have written an article on this previously:
Hexadecimal Notation
Modern IEEE 754 specifies a hexadecimal notation for floating-point numbers, such as 0xcafe.cafep0. Therefore, if you allow dot-based member access immediately following an integer literal, you will run into trouble if you decide to introduce hexadecimal floating-point notation later.
I'll introduce how several existing languages handle member access immediately after integer literals.
Lua does not allow direct member access to literals at all; you must wrap them in parentheses (you cannot write "%.5f":format(math.pi), you must write ("%.5f"):format(math.pi)). Therefore, introducing hexadecimal floating-point literals in Lua 5.2 did not break compatibility.
JavaScript basically allows member access to literals, but due to lexical analysis constraints, you cannot write dot-based member access immediately after a decimal integer literal. Because this is a lexical analysis constraint, inserting a space works (1.toString() is invalid, but 1 .toString() is OK; 1.0.toString() or 1..toString() are also valid). On the other hand, you can write dot-based member access immediately after a hexadecimal integer literal (0xcafe.cafep0 is equivalent to 0xcafe["cafep0"]), so introducing hexadecimal floating-point literals would break compatibility.
Perl supports hexadecimal floating-point literals, and if the syntax does not match that, it is treated as a dot attached to an integer literal. In other words, print 0x42.cafep0 prints 66.7929382324219, but print 0x42.cafe prints 66cafe (Perl's . is a string concatenation operator). This suggests that it performs look-ahead during lexical analysis.
Support for Decimal Floating-Point Numbers
When creating a language that considers educational use, you should consider if it can support decimal floating-point numbers. Standard binary floating-point numbers are notorious for not handling decimal fractions well, leading to the well-known "0.1 + 0.1 + 0.1 does not equal 0.3" problem, and providing educators and learners with a way to avoid this is valuable.
If your language is primarily for educational purposes, it might be worth making decimal the default floating-point type.
If implementing it yourself is difficult, using something like mpdecimal seems appropriate. CPython appears to use mpdecimal as well.
Appendix: Array Size and the int Type
What type should be used for array sizes and indices? C and C++ have a dedicated type called size_t, but many other languages use the int type for array sizes and indices.
This is fine if the size of the int type is not fixed by the language specification and is decided by the execution environment as 32-bit or 64-bit. Even if the size of the int type is fixed at 64 bits, it should be fine (for now).
However, a problem arises in languages that fix int at 32 bits by language specification while using it for array sizes and indices. Namely, the size of arrays that can be handled on 64-bit environments is unnecessarily restricted.
Specifically, Java and .NET are affected by this problem. .NET seems to have introduced properties like LongLength, but older code still cannot handle large arrays regardless.
Postscript: TypeScript and the number Type
I wrote, "Just because you are building a language that compiles to JavaScript, you don't necessarily have to mimic its numeric type design," but one prominent language that compiles to JavaScript—TypeScript—has only number (and bigint) as its numeric types.
Given the premise of compiling to JavaScript, the choice to only have the number type seems natural, but is this premise valid in the first place? Successful languages eventually want to expand their compilation targets. Scala and Kotlin started as JVM languages but now have backends that generate JavaScript or native code. PureScript started as an altJS but unofficially has a variant that compiles natively (via languages like C++). Similarly, couldn't a backend that natively compiles TypeScript be added?
When dreaming of native compilation for TypeScript (or a subset thereof), the first hurdle is the problem of "what to do with the number type." If you represent it simply using floating-point numbers, you cannot use the CPU's integer arithmetic units. Also, processes to convert to integers (ToInt32, ToUint32) will likely be costly (except on certain Arm CPUs).
In current high-performance JavaScript runtimes, optimizations are implemented to represent values that can fit into a 32-bit integer as a 32-bit (or 31-bit?) integer internally. If you try to use that representation method in AOT compilation, you have to insert conditional branches for every number type operation, which seems difficult.
I haven't experimented with this myself, so this is just speculation. If anyone has tried it, please let me know.
Well, even if there were no issues with the number type in TypeScript, there are still the issues of an unsound type system and structural subtyping waiting to hinder AOT compilation, so native compilation might not work out well either way (hence, it's just a "dream"). Nevertheless, I believe there are drawbacks to numeric types being just number.
By the way, Dart, which appeared a bit before TypeScript, separates int and double and supports native compilation. The int type becomes a 64-bit integer during native compilation and uses floating-point numbers when compiling to JavaScript. While I think the class-based nature and type soundness also contribute to its ability to be natively compiled,
Also, there is AssemblyScript, a derivative language of TypeScript, which also separates integer types from floating-point types.
Discussion