blu/lux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 13 Wiki Activity

fix: make all example programs work correctly

Browse Source 
- Add string concatenation support to + operator in typechecker
- Register ADT constructors in both type environment and interpreter
- Bind handlers as values so they can be referenced in run...with
- Fix effect checking to use subset instead of exact match
- Add built-in effects (Console, Fail, State) to run block contexts
- Suppress dead code warnings in diagnostics, modules, parser

Update all example programs with:
- Expected output documented in comments
- Proper run...with statements to execute code

Add new example programs:
- behavioral.lux: pure, idempotent, deterministic, commutative functions
- pipelines.lux: pipe operator demonstrations
- statemachine.lux: ADT-based state machines
- tailcall.lux: tail call optimization examples
- traits.lux: type classes and pattern matching

Add documentation:
- docs/IMPLEMENTATION_PLAN.md: feature roadmap and status
- docs/PERFORMANCE_AND_TRADEOFFS.md: performance analysis

Add benchmarks for performance testing.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Lucas
2026-02-13 09:05:06 -05:00
parent 20bf75a5f8
commit 15a820a467
25 changed files with 1210 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

303
docs/IMPLEMENTATION_PLAN.md Normal file
View File

@@ -0,0 +1,303 @@
# Lux Language Implementation Plan
## Current Status Summary
### What's Working (147 tests passing)
**Core Language:**
- Lexer and parser for core syntax
- AST representation
- Type checker with Hindley-Milner inference
- Interpreter with REPL
- Tail call optimization (TCO)
- Pattern matching with exhaustiveness checking
- Algebraic data types (ADTs) with constructors
- Records and tuples
- Higher-order functions and closures
- Pipe operator (`|>`)
- Documentation comments
**Effect System:**
- Effect declarations
- Effect signatures on functions (`with {Effect}`)
- Handler definitions
- `run ... with` syntax
- Effect inference within function bodies
- Effect subset checking (pure functions callable in effectful contexts)
**Type Classes:**
- Trait declarations
- Implementation blocks
- Basic trait method dispatch
**Behavioral Properties:**
- Property declarations (`is pure`, `is idempotent`, etc.)
- Property parsing and storage in AST
**Tooling:**
- REPL with history and autocomplete
- LSP server (diagnostics, hover, completions)
- Elm-style error messages
### Example Programs Working
1. `hello.lux` - Basic effect usage
2. `factorial.lux` - Recursion
3. `effects.lux` - Custom effects and handlers
4. `datatypes.lux` - ADTs and pattern matching
5. `functional.lux` - Higher-order functions
6. `traits.lux` - Type classes
7. `behavioral.lux` - Behavioral properties
8. `tailcall.lux` - Tail call optimization
9. `statemachine.lux` - State machines with ADTs
10. `pipelines.lux` - Pipe operator
---
## Missing Features Analysis
### Priority 1: Critical Missing Features
#### 1.1 Generic Type Parameters
**Status:** Parser supports `Option<Int>` syntax but type system doesn't fully support generics.
**What's Missing:**
- Type parameter declarations in type definitions (`type List<T> = ...`)
- Type application in type checking
- Generic function parameters (`fn map<T, U>(f: fn(T): U, list: List<T>): List<U>`)
- Type parameter constraints (`where T: Eq`)
**Implementation Steps:**
1. Add type parameter support to type definitions in AST
2. Implement kind checking (types vs type constructors)
3. Add type application rules to unification
4. Support polymorphic function instantiation
#### 1.2 String Interpolation
**Status:** Not implemented. Currently requires manual `toString()` calls.
**What's Missing:**
- Parser support for `"Hello, {name}!"` syntax
- Type checking for interpolated expressions
- Runtime string building
**Implementation Steps:**
1. Add string interpolation tokens to lexer
2. Parse interpolated strings into AST (list of parts)
3. Desugar to string concatenation
#### 1.3 Better Error Messages
**Status:** Basic error messages exist but can be improved.
**What's Missing:**
- Source code context in all errors
- Type diff display for mismatches
- Suggestions for common mistakes
- Error recovery in parser
**Implementation Steps:**
1. Ensure all errors include span information
2. Implement error recovery in parser
3. Add "did you mean?" suggestions
### Priority 2: Effect System Completion
#### 2.1 Effect Polymorphism
**Status:** Not implemented.
**What's Missing:**
- Functions generic over their effects
- Effect variables in type signatures
- Effect constraints
**Example syntax:**
```lux
fn withRetry<E>(action: fn(): T with E, attempts: Int): T with E = ...
```
**Implementation Steps:**
1. Add effect variables to effect representation
2. Implement effect unification with variables
3. Support effect quantification in type schemes
#### 2.2 Built-in Effects
**Status:** Only Console effect is built-in.
**Missing Effects:**
- `State<S>` - get/put state
- `Reader<R>` - read-only environment
- `Fail` - early returns/exceptions
- `Random` - random number generation
- `Time` - current time, delays
- `Async` - async/await
**Implementation Steps:**
1. Define effect interfaces in prelude
2. Implement handlers in runtime
3. Add effect-specific type checking rules
#### 2.3 Resumable Handlers
**Status:** Handlers exist but may not support continuation resumption.
**What's Missing:**
- `resume` keyword to continue computation
- Multi-shot continuations
- Proper effect handler semantics
### Priority 3: Schema Evolution
#### 3.1 Versioned Types
**Status:** Parser supports `@v1` syntax but runtime doesn't use it.
**What's Missing:**
- Version tracking in type system
- Migration function generation
- Compatibility checking
- Codec generation
**Implementation Steps:**
1. Track version in type representation
2. Implement migration chain resolution
3. Add compatibility rules to type checker
4. Generate serialization code
### Priority 4: Module System
#### 4.1 Complete Import/Export
**Status:** Basic imports work but incomplete.
**What's Missing:**
- Re-exports
- Module aliases working properly
- Circular dependency detection (exists but needs testing)
- Package/namespace management
**Implementation Steps:**
1. Implement proper module resolution
2. Add re-export syntax
3. Support qualified names everywhere
4. Add package.lux for project configuration
### Priority 5: Code Generation
#### 5.1 Compile to Target
**Status:** Interpreter only.
**Target Options:**
1. **WASM** - Best for web and portable deployment
2. **JavaScript** - Easiest web integration
3. **LLVM IR** - Native performance
4. **Custom bytecode** - VM-based execution
**Implementation Steps:**
1. Design intermediate representation (IR)
2. Implement IR generation from AST
3. Implement backend for chosen target
4. Add optimization passes
### Priority 6: Tooling
#### 6.1 Package Manager
**What's Needed:**
- Package registry
- Dependency resolution
- Version management
- Build system integration
#### 6.2 Standard Library
**What's Needed:**
- Collections (Map, Set, Array)
- String utilities
- Math functions
- File I/O
- Network I/O
- JSON/YAML parsing
#### 6.3 Debugger
**What's Needed:**
- Breakpoints
- Step execution
- Variable inspection
- Stack traces
---
## Recommended Implementation Order
### Phase 1: Language Completeness (Essential)
1. **Generic type parameters** - Required for proper List, Option, etc.
2. **String interpolation** - Major usability improvement
3. **Better error messages** - Critical for adoption
### Phase 2: Effect System Maturity
4. **Built-in effects** (State, Fail, Reader)
5. **Effect polymorphism**
6. **Resumable handlers**
### Phase 3: Production Readiness
7. **Complete module system**
8. **Standard library**
9. **Package manager**
### Phase 4: Performance
10. **Code generation** (WASM or JS first)
11. **Optimization passes**
12. **Incremental compilation**
### Phase 5: Advanced Features
13. **Schema evolution**
14. **Refinement types**
15. **SMT solver integration**
---
## Technical Debt
### Known Issues Fixed During Testing
1. ✅ String concatenation with `+` operator wasn't type-checked correctly
2. ✅ ADT constructors weren't registered in type environment
3. ✅ ADT constructors weren't registered in interpreter environment
4. ✅ Handlers weren't accessible as values
5. ✅ Effect checking was too strict (required exact match instead of subset)
6. ✅ `total` keyword conflicts with variable names in examples
### Remaining Technical Debt
1. Dead code in diagnostics, modules, parser (suppressed with `#![allow(dead_code)]`)
2. Some test utilities not fully utilized
3. LSP server basic but could be expanded
4. Error recovery in parser incomplete
---
## Feature Comparison with Other Languages
| Feature | Lux | Koka | Haskell | Rust | TypeScript |
|---------|-----|------|---------|------|------------|
| Algebraic Effects | ✅ | ✅ | Via libs | ❌ | ❌ |
| Type Inference | ✅ | ✅ | ✅ | Partial | Partial |
| ADTs | ✅ | ✅ | ✅ | ✅ | Via unions |
| Pattern Matching | ✅ | ✅ | ✅ | ✅ | Limited |
| Generics | Partial | ✅ | ✅ | ✅ | ✅ |
| Type Classes | Basic | ✅ | ✅ | ✅ | ❌ |
| Effect Polymorphism | ❌ | ✅ | Via mtl | N/A | N/A |
| Schema Evolution | Planned | ❌ | ❌ | ❌ | ❌ |
| Refinement Types | Planned | ❌ | Via LH | ❌ | ❌ |
| Tail Call Opt | ✅ | ✅ | ✅ | Limited | ❌ |
| REPL | ✅ | ✅ | ✅ | Limited | ✅ |
| LSP | Basic | ✅ | ✅ | ✅ | ✅ |
---
## Unique Value Proposition
Lux differentiates itself through:
1. **First-class algebraic effects** - Making side effects explicit, testable, and composable
2. **Schema evolution** (planned) - Type-safe data migrations built into the language
3. **Behavioral types** (planned) - Compile-time verification of properties like purity and totality
4. **Developer experience** - Elm-style errors, REPL, LSP support
The combination of these features makes Lux particularly suited for:
- Building reliable backend services
- Applications with complex state management
- Systems requiring careful versioning and migration
- Projects where testing and verification are critical

351
docs/PERFORMANCE_AND_TRADEOFFS.md Normal file
View File

@@ -0,0 +1,351 @@
# Lux Performance Characteristics and Language Tradeoffs
## Executive Summary
Lux is a tree-walking interpreted language with algebraic effects. This document analyzes its performance characteristics, compares it to other languages, and explains the design tradeoffs made.
**Key Performance Characteristics:**
- **Interpretation overhead:** ~100-1000x slower than native compiled languages
- **Tail call optimization:** Effective, prevents stack overflow
- **Effect handling:** ~10-20% overhead per effect operation
- **Memory:** Reference counting for closures, aggressive cloning for collections
---
## Benchmark Results
### Test System
Benchmarks run via tree-walking interpreter in release mode.
### Results Summary
| Benchmark | Time | Operations | Ops/sec | Notes |
|-----------|------|------------|---------|-------|
| Fibonacci (naive, n=30) | 34,980ms | ~1.3M calls | 37K | Exponential recursion |
| Fibonacci (TCO, n=100K) | 498ms | 100K iterations | 200K | Tail-call optimized |
| List operations (10K) | 461ms | 30K ops | 65K | map+filter+fold |
| Pattern matching (32K nodes) | 964ms | 65K matches | 67K | Tree traversal |
| Closures (100K calls) | 538ms | 100K closures | 186K | Closure creation + calls |
| String ops (1K concat) | 457ms | 1K concats | 2.2K | String building |
### Analysis
**Naive Recursion is Expensive:**
- fib(30) takes 35 seconds due to exponential call overhead
- Each function call involves: environment extension, parameter binding, AST traversal
- Compare: Python ~2s, JavaScript ~0.05s, Rust ~0.001s
**TCO is Effective:**
- fib(100,000) completes in 500ms without stack overflow
- Linear time, constant stack space
- The trampoline approach works well
**Collection Operations Have Cloning Overhead:**
- List.map/filter/fold clone the entire list to extract from Value enum
- Pre-allocation in List.map helps but cloning dominates
- Larger lists will show worse performance
---
## Implementation Details
### Evaluation Strategy: Tree-Walking Interpreter
```
Source Code → Lexer → Tokens → Parser → AST → Interpreter → Value
```
**Pros:**
- Simple to implement and debug
- Direct correspondence between AST and execution
- Easy to add new features
**Cons:**
- No optimization passes
- Repeated AST traversal
- No instruction caching
- ~100-1000x slower than bytecode/native
**Comparison:**
| Language | Strategy | Relative Speed |
|----------|----------|----------------|
| Lux | Tree-walking | 1x (baseline) |
| Python | Bytecode VM | 10-50x faster |
| JavaScript (V8) | JIT compiled | 100-500x faster |
| Haskell (GHC) | Native compiled | 500-2000x faster |
| Rust | Native compiled | 1000-5000x faster |
### Value Representation
```rust
pub enum Value {
Int(i64), // Unboxed, 8 bytes
Float(f64), // Unboxed, 8 bytes
Bool(bool), // Unboxed, 1 byte
String(String), // Heap-allocated, ~24 bytes + data
List(Vec<Value>), // Heap-allocated, ~24 bytes + n*size(Value)
Function(Rc<Closure>), // Reference-counted, 8 bytes pointer
Constructor { ... }, // Tagged union
...
}
```
**Memory Overhead:**
- Each `Value` is ~40-80 bytes due to enum discriminant + largest variant
- Lists are `Vec<Value>`, so each element is a full `Value` enum
- No small-value optimization
**Tradeoffs:**
| Aspect | Lux Approach | Alternative | Tradeoff |
|--------|--------------|-------------|----------|
| Primitives | Unboxed in enum | NaN-boxing | Simpler code, more memory |
| Strings | Owned String | Interned/Rc | Simpler, more copying |
| Lists | Vec<Value> | Rc<Vec<Rc<Value>>> | Simpler, expensive clone |
| Closures | Rc<Closure> | Owned | Cheap sharing, GC needed |
### Closure Capture
```rust
pub struct Closure {
params: Vec<String>,
body: Expr,
env: Env, // Entire lexical environment
}
pub struct Env {
bindings: Rc<RefCell<HashMap<String, Value>>>,
parent: Option<Box<Env>>,
}
```
**Characteristics:**
- Closures capture the entire environment chain (lexical scoping)
- Environment lookup is O(depth) - traverses parent chain
- Variable access clones the value (expensive for large values)
**Comparison:**
| Language | Capture Strategy | Lookup Cost |
|----------|------------------|-------------|
| Lux | Scope chain | O(depth) |
| JavaScript | Scope chain | O(depth), optimized |
| Python | Cell references | O(1) after first access |
| Rust | Move/borrow | O(1), compile-time resolved |
### Effect Handling
```rust
fn handle_effect(&mut self, request: EffectRequest) -> Result<Value, RuntimeError> {
// Linear search through handler stack (LIFO)
for handler in self.handler_stack.iter().rev() {
if handler.effect == request.effect {
// Clone handler environment and execute
...
}
}
}
```
**Overhead per Effect Operation:**
1. Create `EffectRequest` struct
2. Linear search through handler stack (typically O(1-5))
3. Clone handler environment
4. Execute handler body
5. Return value
**Comparison with Other Approaches:**
| Approach | Overhead | Flexibility |
|----------|----------|-------------|
| Lux (runtime handlers) | ~10-20% | High - dynamic dispatch |
| Koka (evidence passing) | ~1-5% | High - optimized |
| Haskell mtl (transformers) | ~5-10% | Medium - static |
| Rust (traits) | 0% | Low - compile-time only |
### Tail Call Optimization
```rust
pub enum EvalResult {
Value(Value),
Effect(EffectRequest),
TailCall { func, args, span }, // Trampoline marker
}
// Trampoline loop
loop {
match result {
EvalResult::Value(v) => return Ok(v),
EvalResult::TailCall { func, args, span } => {
result = self.eval_call(func, args, span)?;
}
}
}
```
**Characteristics:**
- Explicit tail position tracking via `tail: bool` parameter
- TailCall variant prevents stack growth
- Only function calls in tail position are optimized
- Arguments are always evaluated eagerly before tail call
**Comparison:**
| Language | TCO Support | Mechanism |
|----------|-------------|-----------|
| Lux | Full | Trampoline |
| Scheme | Full | Required by spec |
| Haskell | Full | Lazy evaluation + STG |
| JavaScript | Safari only | Implementation-dependent |
| Python | None | Explicit recursion limit |
| Rust | Limited | LLVM optimization |
---
## Language Tradeoffs
### 1. Safety vs Performance
**Choice: Safety First**
| Decision | Safety Benefit | Performance Cost |
|----------|----------------|------------------|
| Immutable values | No data races | Clone on every modification |
| Explicit effects | No hidden side effects | Handler lookup overhead |
| Type checking | Catch errors early | Compile-time overhead |
| Exhaustive matching | No missed cases | Runtime pattern matching |
### 2. Simplicity vs Optimization
**Choice: Simplicity First**
| Decision | Simplicity Benefit | Lost Optimization |
|----------|-------------------|-------------------|
| Tree-walking | Easy to implement | No bytecode caching |
| Value enum | Uniform handling | No NaN-boxing |
| Clone semantics | Predictable memory | No move optimization |
| No mutation | No aliasing issues | Can't update in place |
### 3. Expressiveness vs Compilation
**Choice: Expressiveness First**
| Feature | Expressiveness Benefit | Compilation Challenge |
|---------|------------------------|----------------------|
| Algebraic effects | Composable side effects | Hard to optimize |
| First-class handlers | Runtime flexibility | Dynamic dispatch |
| Effect polymorphism (planned) | Generic effect code | Complex inference |
| Refinement types (planned) | Precise specifications | SMT solver needed |
### 4. Comparison Matrix
| Aspect | Lux | Koka | Haskell | Rust | TypeScript |
|--------|-----|------|---------|------|------------|
| **Execution** | Interpreted | Compiled | Compiled | Compiled | JIT |
| **Effects** | Algebraic | Algebraic | Monads | Traits | Promises |
| **Memory** | RC + Clone | RC + Reuse | GC | Ownership | GC |
| **Mutability** | Immutable | Immutable | Immutable | Controlled | Mutable |
| **TCO** | Trampoline | Native | Native | LLVM | No |
| **Typing** | HM Inference | HM + Effects | HM + Extensions | Explicit | Structural |
---
## How to Measure Performance
### Running Benchmarks
```bash
# Run a specific benchmark
nix develop --command cargo run --release -- benchmarks/fibonacci.lux
# Time a benchmark
time nix develop --command cargo run --release -- benchmarks/fibonacci_tco.lux
# Run with effect tracing (slower but shows effect operations)
# In REPL: :trace on
```
### Benchmark Suite
| File | Tests | Expected Time |
|------|-------|---------------|
| `fibonacci.lux` | Function call overhead | ~35s (fib 30) |
| `fibonacci_tco.lux` | Tail call optimization | ~0.5s (fib 100K) |
| `list_operations.lux` | Collection performance | ~0.5s (10K elements) |
| `pattern_matching.lux` | ADT matching | ~1s (32K nodes) |
| `effects.lux` | Effect dispatch | ~0.4s (10K effects) |
| `closures.lux` | Closure performance | ~0.5s (100K closures) |
| `strings.lux` | String operations | ~0.5s (1K concats) |
### Key Metrics to Measure
1. **Function calls per second**: Use recursive fibonacci
2. **Effect operations per second**: Use counter effect benchmark
3. **Pattern matches per second**: Use tree traversal
4. **Closure creations per second**: Use makeAdder benchmark
5. **List operations per second**: Use map/filter/fold chain
6. **Memory usage**: Monitor with system tools (not built-in yet)
### Comparison Benchmarks
To compare with other languages, implement the same algorithms:
**Fibonacci (n=30) comparison:**
```
Lux (interpreted): ~35,000 ms
Python 3: ~2,000 ms
Node.js: ~50 ms
Haskell (ghci): ~200 ms
Haskell (compiled): ~5 ms
Rust: ~1 ms
```
---
## Performance Improvement Opportunities
### Short-term (Interpreter Improvements)
1. **Bytecode compilation**: Convert AST to bytecode for faster dispatch
2. **Value representation**: Use NaN-boxing for primitives
3. **Environment optimization**: Use flat closure representation
4. **List operations**: Avoid cloning by using Rc<Vec<Rc<Value>>>
5. **String interning**: Deduplicate string values
### Medium-term (New Backend)
1. **WASM compilation**: Target WebAssembly for portable native speed
2. **JavaScript emission**: Leverage V8/SpiderMonkey JIT
3. **LLVM backend**: Generate native code via LLVM IR
### Long-term (Advanced Optimizations)
1. **Effect fusion**: Combine adjacent effect operations
2. **Inlining**: Inline small functions
3. **Specialization**: Generate specialized code for monomorphic calls
4. **Escape analysis**: Stack-allocate non-escaping values
### Estimated Speedup Potential
| Optimization | Expected Speedup | Effort |
|--------------|------------------|--------|
| Bytecode VM | 5-10x | Medium |
| NaN-boxing | 1.5-2x | Low |
| Flat closures | 2-3x | Medium |
| WASM backend | 50-100x | High |
| LLVM backend | 100-500x | Very High |
---
## Conclusion
Lux prioritizes **expressiveness, safety, and simplicity** over raw performance. The current interpreter is suitable for:
- Prototyping and development
- Educational purposes
- Small scripts and tools
- Testing effect-based designs
For production workloads requiring high performance, a compilation backend would be necessary. The language design is amenable to efficient compilation - algebraic effects can be compiled using CPS transformation or evidence passing, and the pure functional core can benefit from standard optimizations.
The key insight is that Lux's performance ceiling is set by implementation choices (interpreter vs compiler), not fundamental language limitations. Languages like Koka demonstrate that algebraic effects can be compiled efficiently.