docs: add stress test findings with identified bugs and gaps
Documents findings from systematic testing of parser, typechecker, and REPL: - 5 critical bugs (record equality, invalid escapes, unknown effects, etc.) - Parser missing features (hex literals, list patterns, guard clauses) - Poor error messages for common mistakes - REPL issues (:type doesn't show type, interpolation syntax) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
173
docs/STRESS_TEST_FINDINGS.md
Normal file
173
docs/STRESS_TEST_FINDINGS.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# Lux Stress Test Findings
|
||||
|
||||
This document summarizes issues found during systematic stress testing of the Lux parser, typechecker, and REPL.
|
||||
|
||||
## Critical Bugs
|
||||
|
||||
### 1. Record Equality Returns False for Equal Records
|
||||
```lux
|
||||
let r1 = { x: 1, y: 2 }
|
||||
let r2 = { x: 1, y: 2 }
|
||||
r1 == r2 // Returns false! Should be true
|
||||
```
|
||||
**Impact**: High - breaks expected semantics for record comparison
|
||||
**Location**: Likely in interpreter's equality logic for `Value::Record`
|
||||
|
||||
### 2. Invalid Escape Sequences Silently Accepted
|
||||
```lux
|
||||
"\z\q\w" // Returns "zqw" - backslashes silently dropped
|
||||
"\x41" // Returns "x41" - hex escapes not supported but no error
|
||||
"\u0041" // Returns "u0041" - unicode escapes not supported but no error
|
||||
```
|
||||
**Impact**: High - users expect invalid escapes to error
|
||||
**Fix**: Parser should reject unknown escape sequences
|
||||
|
||||
### 3. Unknown Effects Silently Accepted in Declarations
|
||||
```lux
|
||||
fn test(): Unit with {CompletelyFakeEffect} = () // No error!
|
||||
```
|
||||
**Impact**: Medium - typos in effect names not caught until call site
|
||||
**Fix**: Validate effect names during function declaration checking
|
||||
|
||||
### 4. No Forward References for Mutual Recursion
|
||||
```lux
|
||||
fn isEven(n: Int): Bool = if n == 0 then true else isOdd(n - 1) // Error: isOdd undefined
|
||||
fn isOdd(n: Int): Bool = if n == 0 then false else isEven(n - 1)
|
||||
```
|
||||
**Impact**: Medium - common pattern not supported
|
||||
**Fix**: Two-pass type checking or explicit forward declarations
|
||||
|
||||
### 5. Circular Type Definitions Silently Fail
|
||||
```lux
|
||||
type A = B
|
||||
type B = A
|
||||
// No output, no error
|
||||
```
|
||||
**Impact**: Medium - should produce clear error about circular definition
|
||||
|
||||
## Parser Issues
|
||||
|
||||
### Missing Features
|
||||
|
||||
| Feature | Status | Notes |
|
||||
|---------|--------|-------|
|
||||
| Hex literals (`0x10`) | Not supported | Generic parse error |
|
||||
| Binary literals (`0b101`) | Not supported | Generic parse error |
|
||||
| Octal literals (`0o77`) | Not supported | Generic parse error |
|
||||
| Scientific notation (`1e10`) | Not supported | Generic parse error |
|
||||
| Leading dot floats (`.5`) | Not supported | Generic parse error |
|
||||
| Trailing dot floats (`1.`) | Not supported | Generic parse error |
|
||||
| Underscore separators (`1_000_000`) | **Supported** | Works correctly |
|
||||
| List patterns in match (`[a, b, c]`) | Not supported | Parse error |
|
||||
| Guard clauses (`x if x > 0 =>`) | Not working | Shows redundant pattern error |
|
||||
|
||||
### Poor Error Messages
|
||||
|
||||
| Input | Current Error | Better Error |
|
||||
|-------|---------------|--------------|
|
||||
| `[1 2 3]` | "Expected declaration" | "Missing comma between list elements" |
|
||||
| `{ x 1 }` | "Undefined variable: x" | "Missing colon in record field" |
|
||||
| `funtion foo()` | "Expected declaration" | "Unknown keyword 'funtion'. Did you mean 'fn'?" |
|
||||
| `(1 + 2` | No error | "Unclosed parenthesis" |
|
||||
| `{ x: 1` | No error | "Unclosed brace" |
|
||||
| `[1,, 2]` | No output | "Unexpected comma" |
|
||||
|
||||
### Identifier Issues
|
||||
- Double underscore identifiers (`__name`) not allowed
|
||||
- Unicode identifiers (`café`) partially work but may have issues
|
||||
|
||||
## Type Checker Issues
|
||||
|
||||
### Good Error Messages
|
||||
- Type mismatches: Clear "Cannot unify X with Y"
|
||||
- Undefined variables: Includes "Did you mean..." suggestions
|
||||
- Wrong argument count: Clear arity mismatch message
|
||||
- Non-exhaustive match: Lists uncovered patterns
|
||||
- Effect mismatches: Shows expected vs actual effects
|
||||
- Undefined record field: Clear "Record has no field"
|
||||
- Calling non-function: Clear type mismatch
|
||||
|
||||
### Issues
|
||||
- `:type 42` shows "(type checking passed)" but not the actual type
|
||||
- No type inference for function parameters - must be annotated
|
||||
|
||||
## Runtime Issues
|
||||
|
||||
### Division Handling
|
||||
- Integer `/0`: Proper "Division by zero" error
|
||||
- Integer `%0`: Proper "Modulo by zero" error
|
||||
- Float `/0.0`: Returns `inf` (IEEE 754 compliant)
|
||||
|
||||
### Recursion
|
||||
- Deep recursion (10,000 calls): Works! Good tail call optimization
|
||||
- Infinite recursion: Eventually times out (expected)
|
||||
|
||||
### Missing Stdlib Functions
|
||||
- `String.get(s, i)` - not implemented
|
||||
- `String.charAt(s, i)` - not implemented
|
||||
|
||||
## REPL Issues
|
||||
|
||||
### Working Well
|
||||
- State persistence between lines
|
||||
- Function redefinition
|
||||
- Multi-line input
|
||||
- `:help`, `:quit`, `:load` commands
|
||||
- Tab completion (noted in help)
|
||||
|
||||
### Issues
|
||||
- `:type expr` doesn't show the type, just "(type checking passed)"
|
||||
- No `:ast` or `:tokens` debugging commands
|
||||
- String interpolation syntax is `{x}` not `${x}` (unexpected)
|
||||
- `"Hello {name}"` works
|
||||
- `"Hello ${name}"` outputs `"Hello $<value>"` (keeps literal `$`)
|
||||
|
||||
## Comparison & Equality
|
||||
|
||||
| Type | `==` Works? |
|
||||
|------|-------------|
|
||||
| Int | Yes |
|
||||
| Float | Yes |
|
||||
| Bool | Yes |
|
||||
| String | Yes |
|
||||
| List | Yes |
|
||||
| Tuple | Yes |
|
||||
| Option/Result | Yes |
|
||||
| **Record** | **NO (BUG)** |
|
||||
| Function | No (expected) |
|
||||
|
||||
## Short-Circuit Evaluation
|
||||
|
||||
Works correctly:
|
||||
- `true || crash()` - doesn't call `crash()`
|
||||
- `false && crash()` - doesn't call `crash()`
|
||||
- `if true then 1 else crash()` - doesn't call `crash()`
|
||||
|
||||
## Recommendations
|
||||
|
||||
### High Priority
|
||||
1. Fix record equality
|
||||
2. Error on invalid escape sequences
|
||||
3. Validate effect names at declaration time
|
||||
|
||||
### Medium Priority
|
||||
4. Add forward reference support for mutual recursion
|
||||
5. Improve parser error messages with context
|
||||
6. Add list pattern matching support
|
||||
7. Error on circular type definitions
|
||||
|
||||
### Low Priority
|
||||
8. Add hex/binary/octal literal support
|
||||
9. Add scientific notation support
|
||||
10. Fix `:type` command to show actual type
|
||||
11. Document actual string interpolation syntax
|
||||
|
||||
## Test Commands Used
|
||||
|
||||
```bash
|
||||
# Run stress tests
|
||||
./target/release/lux test examples/test_math.lux
|
||||
|
||||
# Test specific edge case
|
||||
echo 'let r1 = {x: 1}; let r2 = {x: 1}; r1 == r2' | ./target/release/lux
|
||||
```
|
||||
Reference in New Issue
Block a user