feat: add Test effect and native testing framework
- Add Test effect with operations: assert, assertEqual, assertNotEqual, assertTrue, assertFalse, fail - Implement Test effect handlers in interpreter with TestResults tracking - Add values_equal method for comparing Value types in tests - Update lux test command to discover and run test_* functions - Create example test files: test_math.lux, test_lists.lux - Add TESTING_DESIGN.md documentation - Fix AST mismatches in C backend and compiler.rs for compatibility Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
307
docs/TESTING_DESIGN.md
Normal file
307
docs/TESTING_DESIGN.md
Normal file
@@ -0,0 +1,307 @@
|
||||
# Lux Testing Framework Design
|
||||
|
||||
## Overview
|
||||
|
||||
Lux needs a native testing framework that leverages its effect system. The key insight: **testing is an effect** - assertions, test discovery, and reporting are all side effects.
|
||||
|
||||
## Survey of Testing Approaches
|
||||
|
||||
### 1. Rust Style (`#[test]` attributes)
|
||||
```rust
|
||||
#[test]
|
||||
fn test_addition() {
|
||||
assert_eq!(2 + 2, 4);
|
||||
}
|
||||
```
|
||||
**Pros:** Familiar, built-in, zero runtime overhead
|
||||
**Cons:** Requires attribute/macro system
|
||||
|
||||
### 2. Elm Style (describe/test/expect)
|
||||
```elm
|
||||
suite = describe "Math"
|
||||
[ test "addition" <| \_ -> Expect.equal 4 (2 + 2)
|
||||
, test "subtraction" <| \_ -> Expect.equal 0 (2 - 2)
|
||||
]
|
||||
```
|
||||
**Pros:** Readable, hierarchical
|
||||
**Cons:** Verbose, needs custom runner
|
||||
|
||||
### 3. Zig Style (inline test blocks)
|
||||
```zig
|
||||
test "addition" {
|
||||
try std.testing.expect(2 + 2 == 4);
|
||||
}
|
||||
```
|
||||
**Pros:** Inline with code, comptime evaluated
|
||||
**Cons:** Needs language support
|
||||
|
||||
### 4. Property-Based (QuickCheck/Hypothesis)
|
||||
```haskell
|
||||
prop_reverse :: [Int] -> Bool
|
||||
prop_reverse xs = reverse (reverse xs) == xs
|
||||
```
|
||||
**Pros:** Finds edge cases automatically
|
||||
**Cons:** Complex to implement
|
||||
|
||||
### 5. Expect Tests (OCaml/Jane Street)
|
||||
```ocaml
|
||||
let%expect_test "hello" =
|
||||
print_endline "Hello";
|
||||
[%expect {| Hello |}]
|
||||
```
|
||||
**Pros:** Self-updating, snapshot testing
|
||||
**Cons:** Requires tooling
|
||||
|
||||
## Recommended: Effect-Based Testing
|
||||
|
||||
Since Lux has effects, testing should be an effect. This is idiomatic and enables powerful features.
|
||||
|
||||
### Design
|
||||
|
||||
```lux
|
||||
// Built-in Test effect
|
||||
effect Test {
|
||||
fn assert(condition: Bool, message: String): Unit
|
||||
fn assertEqual<T>(expected: T, actual: T): Unit
|
||||
fn assertNotEqual<T>(a: T, b: T): Unit
|
||||
fn fail(message: String): Unit
|
||||
fn skip(reason: String): Unit
|
||||
fn todo(description: String): Unit
|
||||
}
|
||||
|
||||
// Test declaration syntax (new keyword)
|
||||
test "addition works" {
|
||||
Test.assertEqual(4, 2 + 2)
|
||||
}
|
||||
|
||||
test "string concatenation" {
|
||||
let result = "hello" + " " + "world"
|
||||
Test.assertEqual("hello world", result)
|
||||
}
|
||||
|
||||
// Grouped tests
|
||||
describe "List operations" {
|
||||
test "map doubles values" {
|
||||
let input = [1, 2, 3]
|
||||
let output = List.map(input, fn(x) => x * 2)
|
||||
Test.assertEqual([2, 4, 6], output)
|
||||
}
|
||||
|
||||
test "filter keeps matches" {
|
||||
let input = [1, 2, 3, 4, 5]
|
||||
let output = List.filter(input, fn(x) => x > 2)
|
||||
Test.assertEqual([3, 4, 5], output)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Why This Design?
|
||||
|
||||
1. **Idiomatic**: Testing is just another effect
|
||||
2. **Mockable**: Can provide different Test handlers for different scenarios
|
||||
3. **Parallel-safe**: Effect isolation prevents test interference
|
||||
4. **Extensible**: Users can define custom assertion effects
|
||||
5. **Property testing**: Can add `Test.forAll` for property-based testing
|
||||
|
||||
### Test Runner
|
||||
|
||||
```bash
|
||||
# Run all tests in a file
|
||||
lux test examples/math_test.lux
|
||||
|
||||
# Run all tests in directory
|
||||
lux test tests/
|
||||
|
||||
# Run specific test
|
||||
lux test tests/math.lux --filter "addition"
|
||||
|
||||
# Watch mode
|
||||
lux test --watch
|
||||
```
|
||||
|
||||
### Alternative: Annotation-Based (Simpler)
|
||||
|
||||
If we want simpler implementation:
|
||||
|
||||
```lux
|
||||
// Functions prefixed with test_ are tests
|
||||
fn test_addition(): Unit with {Test} = {
|
||||
Test.assertEqual(4, 2 + 2)
|
||||
}
|
||||
|
||||
fn test_strings(): Unit with {Test} = {
|
||||
Test.assertEqual("hello", "hel" + "lo")
|
||||
}
|
||||
```
|
||||
|
||||
Run with:
|
||||
```bash
|
||||
lux test file.lux # Finds all test_* functions
|
||||
```
|
||||
|
||||
## Chosen Approach: Hybrid
|
||||
|
||||
Combine both for flexibility:
|
||||
|
||||
### Phase 1: Function-Based (Simple, Immediate)
|
||||
|
||||
```lux
|
||||
// Any function starting with test_ is a test
|
||||
fn test_addition(): Unit with {Test} =
|
||||
Test.assertEqual(4, 2 + 2)
|
||||
|
||||
fn test_factorial(): Unit with {Test} = {
|
||||
Test.assertEqual(1, factorial(0))
|
||||
Test.assertEqual(1, factorial(1))
|
||||
Test.assertEqual(120, factorial(5))
|
||||
}
|
||||
|
||||
// Helper for multiple assertions
|
||||
fn test_list_operations(): Unit with {Test} = {
|
||||
// Each assertion is independent
|
||||
Test.assertEqual([2, 4], List.map([1, 2], fn(x) => x * 2))
|
||||
Test.assertEqual([2, 3], List.filter([1, 2, 3], fn(x) => x > 1))
|
||||
Test.assertEqual(6, List.fold([1, 2, 3], 0, fn(a, b) => a + b))
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 2: Block Syntax (Later)
|
||||
|
||||
```lux
|
||||
test "descriptive name" {
|
||||
Test.assertEqual(4, 2 + 2)
|
||||
}
|
||||
|
||||
describe "module name" {
|
||||
test "feature 1" { ... }
|
||||
test "feature 2" { ... }
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Step 1: Test Effect
|
||||
|
||||
Add built-in `Test` effect to the type system:
|
||||
|
||||
```rust
|
||||
// In types.rs, add Test effect
|
||||
EffectDef {
|
||||
name: "Test".to_string(),
|
||||
operations: vec![
|
||||
EffectOpDef { name: "assert", params: vec![("cond", Bool), ("msg", String)], return_type: Unit },
|
||||
EffectOpDef { name: "assertEqual", params: vec![("expected", T), ("actual", T)], return_type: Unit },
|
||||
EffectOpDef { name: "fail", params: vec![("msg", String)], return_type: Unit },
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Test Handler (Interpreter)
|
||||
|
||||
```rust
|
||||
// In interpreter.rs
|
||||
fn handle_test_effect(&mut self, op: &str, args: &[Value]) -> Result<Value, RuntimeError> {
|
||||
match op {
|
||||
"assert" => {
|
||||
let cond = args[0].as_bool()?;
|
||||
let msg = args[1].as_string()?;
|
||||
if !cond {
|
||||
self.test_failures.push(TestFailure { message: msg, .. });
|
||||
}
|
||||
Ok(Value::Unit)
|
||||
}
|
||||
"assertEqual" => {
|
||||
let expected = &args[0];
|
||||
let actual = &args[1];
|
||||
if expected != actual {
|
||||
self.test_failures.push(TestFailure {
|
||||
message: format!("Expected {:?}, got {:?}", expected, actual),
|
||||
..
|
||||
});
|
||||
}
|
||||
Ok(Value::Unit)
|
||||
}
|
||||
// ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Test Discovery
|
||||
|
||||
```rust
|
||||
// Find all test_* functions in a file
|
||||
fn discover_tests(program: &Program) -> Vec<&FunctionDecl> {
|
||||
program.declarations.iter().filter_map(|d| {
|
||||
if let Declaration::Function(f) = d {
|
||||
if f.name.name.starts_with("test_") {
|
||||
return Some(f);
|
||||
}
|
||||
}
|
||||
None
|
||||
}).collect()
|
||||
}
|
||||
```
|
||||
|
||||
### Step 4: Test Runner CLI
|
||||
|
||||
```bash
|
||||
lux test file.lux
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Running tests in file.lux...
|
||||
|
||||
test_addition ... OK
|
||||
test_factorial ... OK
|
||||
test_strings ... FAIL
|
||||
Expected: "hello world"
|
||||
Actual: "helloworld"
|
||||
at file.lux:15
|
||||
|
||||
Results: 2 passed, 1 failed
|
||||
```
|
||||
|
||||
## Property-Based Testing (Future)
|
||||
|
||||
```lux
|
||||
// Properties are functions that return Bool
|
||||
fn prop_reverse_twice<T>(xs: List<T>): Bool =
|
||||
List.reverse(List.reverse(xs)) == xs
|
||||
|
||||
fn prop_sort_idempotent(xs: List<Int>): Bool =
|
||||
List.sort(List.sort(xs)) == List.sort(xs)
|
||||
|
||||
// Test runner generates random inputs
|
||||
test "reverse is involution" {
|
||||
Test.forAll(prop_reverse_twice)
|
||||
}
|
||||
```
|
||||
|
||||
## Expect Tests (Future)
|
||||
|
||||
```lux
|
||||
test "json output" {
|
||||
let data = { name: "Alice", age: 30 }
|
||||
let json = Json.stringify(data)
|
||||
|
||||
// This gets auto-updated by the test runner
|
||||
Test.expect(json, |||
|
||||
{"name":"Alice","age":30}
|
||||
|||)
|
||||
}
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
| Phase | Feature | Effort |
|
||||
|-------|---------|--------|
|
||||
| 1 | Test effect + test_* discovery | 1-2 days |
|
||||
| 2 | Test runner CLI (`lux test`) | 1 day |
|
||||
| 3 | Nice output formatting | 0.5 day |
|
||||
| 4 | `test "name" {}` syntax | 1 day |
|
||||
| 5 | `describe` blocks | 0.5 day |
|
||||
| 6 | Property-based testing | 2-3 days |
|
||||
| 7 | Expect/snapshot tests | 2 days |
|
||||
|
||||
**Recommendation:** Start with Phase 1-3 (function-based tests with CLI) for immediate value.
|
||||
Reference in New Issue
Block a user