From 07a35f18294546b77ca1f87c9c79555d9cd692df Mon Sep 17 00:00:00 2001 From: Brandon Lucas Date: Sat, 14 Feb 2026 02:20:41 -0500 Subject: [PATCH] docs: add stress test findings with identified bugs and gaps Documents findings from systematic testing of parser, typechecker, and REPL: - 5 critical bugs (record equality, invalid escapes, unknown effects, etc.) - Parser missing features (hex literals, list patterns, guard clauses) - Poor error messages for common mistakes - REPL issues (:type doesn't show type, interpolation syntax) Co-Authored-By: Claude Opus 4.5 --- docs/STRESS_TEST_FINDINGS.md | 173 +++++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 docs/STRESS_TEST_FINDINGS.md diff --git a/docs/STRESS_TEST_FINDINGS.md b/docs/STRESS_TEST_FINDINGS.md new file mode 100644 index 0000000..7117bdc --- /dev/null +++ b/docs/STRESS_TEST_FINDINGS.md @@ -0,0 +1,173 @@ +# Lux Stress Test Findings + +This document summarizes issues found during systematic stress testing of the Lux parser, typechecker, and REPL. + +## Critical Bugs + +### 1. Record Equality Returns False for Equal Records +```lux +let r1 = { x: 1, y: 2 } +let r2 = { x: 1, y: 2 } +r1 == r2 // Returns false! Should be true +``` +**Impact**: High - breaks expected semantics for record comparison +**Location**: Likely in interpreter's equality logic for `Value::Record` + +### 2. Invalid Escape Sequences Silently Accepted +```lux +"\z\q\w" // Returns "zqw" - backslashes silently dropped +"\x41" // Returns "x41" - hex escapes not supported but no error +"\u0041" // Returns "u0041" - unicode escapes not supported but no error +``` +**Impact**: High - users expect invalid escapes to error +**Fix**: Parser should reject unknown escape sequences + +### 3. Unknown Effects Silently Accepted in Declarations +```lux +fn test(): Unit with {CompletelyFakeEffect} = () // No error! +``` +**Impact**: Medium - typos in effect names not caught until call site +**Fix**: Validate effect names during function declaration checking + +### 4. No Forward References for Mutual Recursion +```lux +fn isEven(n: Int): Bool = if n == 0 then true else isOdd(n - 1) // Error: isOdd undefined +fn isOdd(n: Int): Bool = if n == 0 then false else isEven(n - 1) +``` +**Impact**: Medium - common pattern not supported +**Fix**: Two-pass type checking or explicit forward declarations + +### 5. Circular Type Definitions Silently Fail +```lux +type A = B +type B = A +// No output, no error +``` +**Impact**: Medium - should produce clear error about circular definition + +## Parser Issues + +### Missing Features + +| Feature | Status | Notes | +|---------|--------|-------| +| Hex literals (`0x10`) | Not supported | Generic parse error | +| Binary literals (`0b101`) | Not supported | Generic parse error | +| Octal literals (`0o77`) | Not supported | Generic parse error | +| Scientific notation (`1e10`) | Not supported | Generic parse error | +| Leading dot floats (`.5`) | Not supported | Generic parse error | +| Trailing dot floats (`1.`) | Not supported | Generic parse error | +| Underscore separators (`1_000_000`) | **Supported** | Works correctly | +| List patterns in match (`[a, b, c]`) | Not supported | Parse error | +| Guard clauses (`x if x > 0 =>`) | Not working | Shows redundant pattern error | + +### Poor Error Messages + +| Input | Current Error | Better Error | +|-------|---------------|--------------| +| `[1 2 3]` | "Expected declaration" | "Missing comma between list elements" | +| `{ x 1 }` | "Undefined variable: x" | "Missing colon in record field" | +| `funtion foo()` | "Expected declaration" | "Unknown keyword 'funtion'. Did you mean 'fn'?" | +| `(1 + 2` | No error | "Unclosed parenthesis" | +| `{ x: 1` | No error | "Unclosed brace" | +| `[1,, 2]` | No output | "Unexpected comma" | + +### Identifier Issues +- Double underscore identifiers (`__name`) not allowed +- Unicode identifiers (`café`) partially work but may have issues + +## Type Checker Issues + +### Good Error Messages +- Type mismatches: Clear "Cannot unify X with Y" +- Undefined variables: Includes "Did you mean..." suggestions +- Wrong argument count: Clear arity mismatch message +- Non-exhaustive match: Lists uncovered patterns +- Effect mismatches: Shows expected vs actual effects +- Undefined record field: Clear "Record has no field" +- Calling non-function: Clear type mismatch + +### Issues +- `:type 42` shows "(type checking passed)" but not the actual type +- No type inference for function parameters - must be annotated + +## Runtime Issues + +### Division Handling +- Integer `/0`: Proper "Division by zero" error +- Integer `%0`: Proper "Modulo by zero" error +- Float `/0.0`: Returns `inf` (IEEE 754 compliant) + +### Recursion +- Deep recursion (10,000 calls): Works! Good tail call optimization +- Infinite recursion: Eventually times out (expected) + +### Missing Stdlib Functions +- `String.get(s, i)` - not implemented +- `String.charAt(s, i)` - not implemented + +## REPL Issues + +### Working Well +- State persistence between lines +- Function redefinition +- Multi-line input +- `:help`, `:quit`, `:load` commands +- Tab completion (noted in help) + +### Issues +- `:type expr` doesn't show the type, just "(type checking passed)" +- No `:ast` or `:tokens` debugging commands +- String interpolation syntax is `{x}` not `${x}` (unexpected) + - `"Hello {name}"` works + - `"Hello ${name}"` outputs `"Hello $"` (keeps literal `$`) + +## Comparison & Equality + +| Type | `==` Works? | +|------|-------------| +| Int | Yes | +| Float | Yes | +| Bool | Yes | +| String | Yes | +| List | Yes | +| Tuple | Yes | +| Option/Result | Yes | +| **Record** | **NO (BUG)** | +| Function | No (expected) | + +## Short-Circuit Evaluation + +Works correctly: +- `true || crash()` - doesn't call `crash()` +- `false && crash()` - doesn't call `crash()` +- `if true then 1 else crash()` - doesn't call `crash()` + +## Recommendations + +### High Priority +1. Fix record equality +2. Error on invalid escape sequences +3. Validate effect names at declaration time + +### Medium Priority +4. Add forward reference support for mutual recursion +5. Improve parser error messages with context +6. Add list pattern matching support +7. Error on circular type definitions + +### Low Priority +8. Add hex/binary/octal literal support +9. Add scientific notation support +10. Fix `:type` command to show actual type +11. Document actual string interpolation syntax + +## Test Commands Used + +```bash +# Run stress tests +./target/release/lux test examples/test_math.lux + +# Test specific edge case +echo 'let r1 = {x: 1}; let r2 = {x: 1}; r1 == r2' | ./target/release/lux +```