lux/docs/TESTING_DESIGN.md

Lux Testing Framework Design

Overview

Lux needs a native testing framework that leverages its effect system. The key insight: testing is an effect - assertions, test discovery, and reporting are all side effects.

Survey of Testing Approaches

1. Rust Style (#[test] attributes)

#[test]
fn test_addition() {
    assert_eq!(2 + 2, 4);
}

Pros: Familiar, built-in, zero runtime overhead Cons: Requires attribute/macro system

2. Elm Style (describe/test/expect)

suite = describe "Math"
    [ test "addition" <| \_ -> Expect.equal 4 (2 + 2)
    , test "subtraction" <| \_ -> Expect.equal 0 (2 - 2)
    ]

Pros: Readable, hierarchical Cons: Verbose, needs custom runner

3. Zig Style (inline test blocks)

test "addition" {
    try std.testing.expect(2 + 2 == 4);
}

Pros: Inline with code, comptime evaluated Cons: Needs language support

4. Property-Based (QuickCheck/Hypothesis)

prop_reverse :: [Int] -> Bool
prop_reverse xs = reverse (reverse xs) == xs

Pros: Finds edge cases automatically Cons: Complex to implement

5. Expect Tests (OCaml/Jane Street)

let%expect_test "hello" =
  print_endline "Hello";
  [%expect {| Hello |}]

Pros: Self-updating, snapshot testing Cons: Requires tooling

Recommended: Effect-Based Testing

Since Lux has effects, testing should be an effect. This is idiomatic and enables powerful features.

Design

// Built-in Test effect
effect Test {
    fn assert(condition: Bool, message: String): Unit
    fn assertEqual<T>(expected: T, actual: T): Unit
    fn assertNotEqual<T>(a: T, b: T): Unit
    fn fail(message: String): Unit
    fn skip(reason: String): Unit
    fn todo(description: String): Unit
}

// Test declaration syntax (new keyword)
test "addition works" {
    Test.assertEqual(4, 2 + 2)
}

test "string concatenation" {
    let result = "hello" + " " + "world"
    Test.assertEqual("hello world", result)
}

// Grouped tests
describe "List operations" {
    test "map doubles values" {
        let input = [1, 2, 3]
        let output = List.map(input, fn(x) => x * 2)
        Test.assertEqual([2, 4, 6], output)
    }

    test "filter keeps matches" {
        let input = [1, 2, 3, 4, 5]
        let output = List.filter(input, fn(x) => x > 2)
        Test.assertEqual([3, 4, 5], output)
    }
}

Why This Design?

1. Idiomatic: Testing is just another effect
2. Mockable: Can provide different Test handlers for different scenarios
3. Parallel-safe: Effect isolation prevents test interference
4. Extensible: Users can define custom assertion effects
5. Property testing: Can add Test.forAll for property-based testing

Test Runner

# Run all tests in a file
lux test examples/math_test.lux

# Run all tests in directory
lux test tests/

# Run specific test
lux test tests/math.lux --filter "addition"

# Watch mode
lux test --watch

Alternative: Annotation-Based (Simpler)

If we want simpler implementation:

// Functions prefixed with test_ are tests
fn test_addition(): Unit with {Test} = {
    Test.assertEqual(4, 2 + 2)
}

fn test_strings(): Unit with {Test} = {
    Test.assertEqual("hello", "hel" + "lo")
}

Run with:

lux test file.lux  # Finds all test_* functions

Chosen Approach: Hybrid

Combine both for flexibility:

Phase 1: Function-Based (Simple, Immediate)

// Any function starting with test_ is a test
fn test_addition(): Unit with {Test} =
    Test.assertEqual(4, 2 + 2)

fn test_factorial(): Unit with {Test} = {
    Test.assertEqual(1, factorial(0))
    Test.assertEqual(1, factorial(1))
    Test.assertEqual(120, factorial(5))
}

// Helper for multiple assertions
fn test_list_operations(): Unit with {Test} = {
    // Each assertion is independent
    Test.assertEqual([2, 4], List.map([1, 2], fn(x) => x * 2))
    Test.assertEqual([2, 3], List.filter([1, 2, 3], fn(x) => x > 1))
    Test.assertEqual(6, List.fold([1, 2, 3], 0, fn(a, b) => a + b))
}

Phase 2: Block Syntax (Later)

test "descriptive name" {
    Test.assertEqual(4, 2 + 2)
}

describe "module name" {
    test "feature 1" { ... }
    test "feature 2" { ... }
}

Implementation Plan

Step 1: Test Effect

Add built-in Test effect to the type system:

// In types.rs, add Test effect
EffectDef {
    name: "Test".to_string(),
    operations: vec![
        EffectOpDef { name: "assert", params: vec![("cond", Bool), ("msg", String)], return_type: Unit },
        EffectOpDef { name: "assertEqual", params: vec![("expected", T), ("actual", T)], return_type: Unit },
        EffectOpDef { name: "fail", params: vec![("msg", String)], return_type: Unit },
    ],
}

Step 2: Test Handler (Interpreter)

// In interpreter.rs
fn handle_test_effect(&mut self, op: &str, args: &[Value]) -> Result<Value, RuntimeError> {
    match op {
        "assert" => {
            let cond = args[0].as_bool()?;
            let msg = args[1].as_string()?;
            if !cond {
                self.test_failures.push(TestFailure { message: msg, .. });
            }
            Ok(Value::Unit)
        }
        "assertEqual" => {
            let expected = &args[0];
            let actual = &args[1];
            if expected != actual {
                self.test_failures.push(TestFailure {
                    message: format!("Expected {:?}, got {:?}", expected, actual),
                    ..
                });
            }
            Ok(Value::Unit)
        }
        // ...
    }
}

Step 3: Test Discovery

// Find all test_* functions in a file
fn discover_tests(program: &Program) -> Vec<&FunctionDecl> {
    program.declarations.iter().filter_map(|d| {
        if let Declaration::Function(f) = d {
            if f.name.name.starts_with("test_") {
                return Some(f);
            }
        }
        None
    }).collect()
}

Step 4: Test Runner CLI

lux test file.lux

Output:

Running tests in file.lux...

  test_addition ... OK
  test_factorial ... OK
  test_strings ... FAIL
    Expected: "hello world"
    Actual:   "helloworld"
    at file.lux:15

Results: 2 passed, 1 failed

Property-Based Testing (Future)

// Properties are functions that return Bool
fn prop_reverse_twice<T>(xs: List<T>): Bool =
    List.reverse(List.reverse(xs)) == xs

fn prop_sort_idempotent(xs: List<Int>): Bool =
    List.sort(List.sort(xs)) == List.sort(xs)

// Test runner generates random inputs
test "reverse is involution" {
    Test.forAll(prop_reverse_twice)
}

Expect Tests (Future)

test "json output" {
    let data = { name: "Alice", age: 30 }
    let json = Json.stringify(data)

    // This gets auto-updated by the test runner
    Test.expect(json, |||
        {"name":"Alice","age":30}
    |||)
}

Summary

Phase	Feature	Effort
1	Test effect + test_* discovery	1-2 days
2	Test runner CLI (lux test)	1 day
3	Nice output formatting	0.5 day
4	test "name" {} syntax	1 day
5	describe blocks	0.5 day
6	Property-based testing	2-3 days
7	Expect/snapshot tests	2 days

Recommendation: Start with Phase 1-3 (function-based tests with CLI) for immediate value.