feat: add property-based testing framework
Implements property-based testing infrastructure: stdlib/testing.lux: - Generators: genInt, genIntList, genString, genBool, etc. - Shrinking helpers: shrinkInt, shrinkList, shrinkString - Property helpers: isSorted, sameElements examples/property_testing.lux: - 10 property tests demonstrating the framework - Tests for: involution, commutativity, associativity, identity - 100 iterations per property with random inputs docs/guide/14-property-testing.md: - Complete guide to property-based testing - Generator patterns and common properties - Best practices and examples Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
300
docs/guide/14-property-testing.md
Normal file
300
docs/guide/14-property-testing.md
Normal file
@@ -0,0 +1,300 @@
|
||||
# Property-Based Testing
|
||||
|
||||
Property-based testing is a powerful testing technique where you define properties that should hold for all inputs, and the testing framework generates random inputs to verify those properties. This guide shows how to use property-based testing in Lux.
|
||||
|
||||
## Overview
|
||||
|
||||
Instead of writing tests with specific inputs:
|
||||
|
||||
```lux
|
||||
fn test_reverse(): Unit with {Test} = {
|
||||
Test.assertEqual([3, 2, 1], List.reverse([1, 2, 3]))
|
||||
Test.assertEqual([], List.reverse([]))
|
||||
}
|
||||
```
|
||||
|
||||
Property-based testing verifies general properties:
|
||||
|
||||
```lux
|
||||
// Property: Reversing a list twice gives back the original list
|
||||
fn testReverseInvolutive(n: Int): Bool with {Console, Random} = {
|
||||
if n <= 0 then true
|
||||
else {
|
||||
let xs = genIntList(0, 100, 20)
|
||||
if List.reverse(List.reverse(xs)) == xs then
|
||||
testReverseInvolutive(n - 1)
|
||||
else
|
||||
false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Generators
|
||||
|
||||
Generators create random test data. The `Random` effect provides the building blocks.
|
||||
|
||||
### Basic Generators
|
||||
|
||||
```lux
|
||||
// Generate random integer in range
|
||||
fn genInt(min: Int, max: Int): Int with {Random} =
|
||||
Random.int(min, max)
|
||||
|
||||
// Generate random boolean
|
||||
fn genBool(): Bool with {Random} =
|
||||
Random.bool()
|
||||
|
||||
// Generate random float
|
||||
fn genFloat(): Float with {Random} =
|
||||
Random.float()
|
||||
```
|
||||
|
||||
### String Generators
|
||||
|
||||
```lux
|
||||
let CHARS = "abcdefghijklmnopqrstuvwxyz"
|
||||
|
||||
// Generate random character
|
||||
fn genChar(): String with {Random} = {
|
||||
let idx = Random.int(0, 25)
|
||||
String.substring(CHARS, idx, idx + 1)
|
||||
}
|
||||
|
||||
// Generate random string up to maxLen characters
|
||||
fn genString(maxLen: Int): String with {Random} = {
|
||||
let len = Random.int(0, maxLen)
|
||||
genStringHelper(len)
|
||||
}
|
||||
|
||||
fn genStringHelper(len: Int): String with {Random} = {
|
||||
if len <= 0 then ""
|
||||
else genChar() + genStringHelper(len - 1)
|
||||
}
|
||||
```
|
||||
|
||||
### List Generators
|
||||
|
||||
```lux
|
||||
// Generate random list of integers
|
||||
fn genIntList(min: Int, max: Int, maxLen: Int): List<Int> with {Random} = {
|
||||
let len = Random.int(0, maxLen)
|
||||
genIntListHelper(min, max, len)
|
||||
}
|
||||
|
||||
fn genIntListHelper(min: Int, max: Int, len: Int): List<Int> with {Random} = {
|
||||
if len <= 0 then []
|
||||
else List.concat([Random.int(min, max)], genIntListHelper(min, max, len - 1))
|
||||
}
|
||||
```
|
||||
|
||||
## Writing Property Tests
|
||||
|
||||
### Pattern: Recursive Test Function
|
||||
|
||||
The most common pattern is a recursive function that runs N iterations:
|
||||
|
||||
```lux
|
||||
fn testProperty(n: Int, count: Int): Bool with {Console, Random} = {
|
||||
if n <= 0 then {
|
||||
Console.print(" PASS property_name (" + toString(count) + " tests)")
|
||||
true
|
||||
} else {
|
||||
// Generate random inputs
|
||||
let x = genInt(0, 100)
|
||||
let y = genInt(0, 100)
|
||||
|
||||
// Check property
|
||||
if x + y == y + x then
|
||||
testProperty(n - 1, count)
|
||||
else {
|
||||
Console.print(" FAIL property_name")
|
||||
Console.print(" Counterexample: x=" + toString(x) + ", y=" + toString(y))
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Common Properties
|
||||
|
||||
Here are properties commonly verified in property-based testing:
|
||||
|
||||
**Involution** - Applying a function twice returns the original value:
|
||||
```lux
|
||||
// f(f(x)) == x
|
||||
List.reverse(List.reverse(xs)) == xs
|
||||
```
|
||||
|
||||
**Idempotence** - Applying a function multiple times has the same effect as once:
|
||||
```lux
|
||||
// f(f(x)) == f(x)
|
||||
List.sort(List.sort(xs)) == List.sort(xs)
|
||||
```
|
||||
|
||||
**Commutativity** - Order of arguments doesn't matter:
|
||||
```lux
|
||||
// f(a, b) == f(b, a)
|
||||
a + b == b + a
|
||||
a * b == b * a
|
||||
```
|
||||
|
||||
**Associativity** - Grouping doesn't matter:
|
||||
```lux
|
||||
// f(f(a, b), c) == f(a, f(b, c))
|
||||
(a + b) + c == a + (b + c)
|
||||
(a * b) * c == a * (b * c)
|
||||
```
|
||||
|
||||
**Identity** - An element that doesn't change the result:
|
||||
```lux
|
||||
x + 0 == x
|
||||
x * 1 == x
|
||||
List.concat(xs, []) == xs
|
||||
```
|
||||
|
||||
**Length Preservation**:
|
||||
```lux
|
||||
List.length(List.reverse(xs)) == List.length(xs)
|
||||
List.length(List.map(xs, f)) == List.length(xs)
|
||||
```
|
||||
|
||||
**Length Addition**:
|
||||
```lux
|
||||
List.length(List.concat(xs, ys)) == List.length(xs) + List.length(ys)
|
||||
String.length(s1 + s2) == String.length(s1) + String.length(s2)
|
||||
```
|
||||
|
||||
## Complete Example
|
||||
|
||||
```lux
|
||||
// Property-Based Testing Example
|
||||
// Run with: lux examples/property_testing.lux
|
||||
|
||||
let CHARS = "abcdefghijklmnopqrstuvwxyz"
|
||||
|
||||
// Generators
|
||||
fn genInt(min: Int, max: Int): Int with {Random} =
|
||||
Random.int(min, max)
|
||||
|
||||
fn genIntList(min: Int, max: Int, maxLen: Int): List<Int> with {Random} = {
|
||||
let len = Random.int(0, maxLen)
|
||||
genIntListHelper(min, max, len)
|
||||
}
|
||||
|
||||
fn genIntListHelper(min: Int, max: Int, len: Int): List<Int> with {Random} = {
|
||||
if len <= 0 then []
|
||||
else List.concat([Random.int(min, max)], genIntListHelper(min, max, len - 1))
|
||||
}
|
||||
|
||||
// Test helper
|
||||
fn printResult(name: String, passed: Bool, count: Int): Unit with {Console} = {
|
||||
if passed then
|
||||
Console.print(" PASS " + name + " (" + toString(count) + " tests)")
|
||||
else
|
||||
Console.print(" FAIL " + name)
|
||||
}
|
||||
|
||||
// Property: reverse(reverse(xs)) == xs
|
||||
fn testReverseInvolutive(n: Int, count: Int): Bool with {Console, Random} = {
|
||||
if n <= 0 then {
|
||||
printResult("reverse(reverse(xs)) == xs", true, count)
|
||||
true
|
||||
} else {
|
||||
let xs = genIntList(0, 100, 20)
|
||||
if List.reverse(List.reverse(xs)) == xs then
|
||||
testReverseInvolutive(n - 1, count)
|
||||
else {
|
||||
printResult("reverse(reverse(xs)) == xs", false, count - n + 1)
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Property: addition is commutative
|
||||
fn testAddCommutative(n: Int, count: Int): Bool with {Console, Random} = {
|
||||
if n <= 0 then {
|
||||
printResult("a + b == b + a", true, count)
|
||||
true
|
||||
} else {
|
||||
let a = genInt(-1000, 1000)
|
||||
let b = genInt(-1000, 1000)
|
||||
if a + b == b + a then
|
||||
testAddCommutative(n - 1, count)
|
||||
else {
|
||||
printResult("a + b == b + a", false, count - n + 1)
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn main(): Unit with {Console, Random} = {
|
||||
Console.print("Property-Based Testing Demo")
|
||||
Console.print("")
|
||||
Console.print("Running 100 iterations per property...")
|
||||
Console.print("")
|
||||
|
||||
testReverseInvolutive(100, 100)
|
||||
testAddCommutative(100, 100)
|
||||
|
||||
Console.print("")
|
||||
Console.print("All tests completed!")
|
||||
}
|
||||
|
||||
let result = run main() with {}
|
||||
```
|
||||
|
||||
## Stdlib Testing Module
|
||||
|
||||
The `stdlib/testing.lux` module provides pre-built generators:
|
||||
|
||||
```lux
|
||||
import stdlib.testing
|
||||
|
||||
// Available generators:
|
||||
genInt(min, max) // Random integer in range
|
||||
genIntUpTo(max) // Random integer 0 to max
|
||||
genPositiveInt(max) // Random integer 1 to max
|
||||
genBool() // Random boolean
|
||||
genChar() // Random lowercase letter
|
||||
genAlphaNum() // Random alphanumeric character
|
||||
genString(maxLen) // Random string
|
||||
genStringOfLength(len) // Random string of exact length
|
||||
genIntList(min, max, maxLen) // Random list of integers
|
||||
genBoolList(maxLen) // Random list of booleans
|
||||
genStringList(maxStrLen, maxLen) // Random list of strings
|
||||
|
||||
// Helper functions:
|
||||
shrinkInt(n) // Shrink integer toward zero
|
||||
shrinkList(xs) // Shrink list by removing elements
|
||||
shrinkString(s) // Shrink string by removing characters
|
||||
isSorted(xs) // Check if list is sorted
|
||||
sameElements(xs, ys) // Check if lists have same elements
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Start with many iterations**: Use 100+ iterations per property to catch edge cases.
|
||||
|
||||
2. **Test edge cases explicitly**: Property tests are great for general cases, but also write unit tests for known edge cases.
|
||||
|
||||
3. **Keep properties simple**: Each property should test one thing. Complex properties are harder to debug.
|
||||
|
||||
4. **Use good generators**: Match the distribution of your generators to realistic inputs.
|
||||
|
||||
5. **Print counterexamples**: When a test fails, print the failing inputs to help debugging.
|
||||
|
||||
6. **Combine with shrinking**: Shrinking finds minimal failing inputs, making debugging easier.
|
||||
|
||||
## Limitations
|
||||
|
||||
Current limitations of property-based testing in Lux:
|
||||
|
||||
- No automatic shrinking (must be done manually)
|
||||
- No seed control for reproducible tests
|
||||
- No integration with `lux test` command (uses `Random` effect)
|
||||
|
||||
## See Also
|
||||
|
||||
- [Testing Guide](../testing.md) - Unit testing with the Test effect
|
||||
- [Effects Guide](./05-effects.md) - Understanding the Random effect
|
||||
- [Example](../../examples/property_testing.lux) - Complete working example
|
||||
Reference in New Issue
Block a user