lux/docs/IMPLEMENTATION_PLAN.md

# Lux Language Implementation Plan

## Current Status Summary

### What's Working (222 tests passing)

**Core Language:**
- Lexer and parser for core syntax
- AST representation
- Type checker with Hindley-Milner inference
- Interpreter with REPL
- Tail call optimization (TCO)
- Pattern matching with exhaustiveness checking
- Algebraic data types (ADTs) with constructors
- Records and tuples
- Higher-order functions and closures
- Pipe operator (`|>`)
- Documentation comments
- **Generic type parameters** (`fn map<T, U>`, `type List<T>`)
- **String interpolation** (`"Hello {name}!"`)
- **Escape sequences** (`\n`, `\t`, `\"`, `\{`, `\}`)

**Effect System:**
- Effect declarations
- Effect signatures on functions (`with {Effect}`)
- Handler definitions
- `run ... with` syntax
- Effect inference within function bodies
- Effect subset checking (pure functions callable in effectful contexts)

**Built-in Effects:**
- `Console` - print, readLine, readInt
- `File` - read, write, exists, delete, listDir, createDir
- `Http` - get, post, put, delete (HTTP client)
- `HttpServer` - listen, accept, respond, respondWithHeaders, stop (HTTP server)
- `Random` - int, float, range, bool, element
- `Time` - now, sleep

**Type Classes:**
- Trait declarations
- Implementation blocks
- Basic trait method dispatch

**Behavioral Properties:**
- Property declarations (`is pure`, `is idempotent`, `is total`, `is deterministic`, `is commutative`)
- Property parsing and storage in AST
- Pure function verification (no effects allowed)
- Deterministic verification (no Random/Time effects)
- Commutative verification (operator-based analysis)
- Idempotent verification (pattern-based analysis)
- Total verification (structural recursion, termination checking)

**Module System:**
- Import/export with `pub` visibility
- Aliased imports (`import foo as bar`)
- Selective imports (`import foo.{a, b}`)
- Wildcard imports (`import foo.*`)
- Circular dependency detection
- Transitive imports

**JIT Compiler (Cranelift):**
- Integer arithmetic and comparisons
- Conditionals, let bindings, blocks
- Function calls including recursion
- ~160x speedup over interpreter for numeric code
- CLI integration via `lux compile <file>`

**Standard Library:**
- String operations (substring, length, split, trim, etc.)
- List operations (map, filter, fold, etc.)
- JSON parsing and serialization
- Math functions

**Tooling:**
- REPL with history and autocomplete
- LSP server (diagnostics, hover, completions)
- Watch mode for auto-recompilation
- Source-context error messages

### Example Programs Working
1. `hello.lux` - Basic effect usage
2. `factorial.lux` - Recursion
3. `effects.lux` - Custom effects and handlers
4. `datatypes.lux` - ADTs and pattern matching
5. `functional.lux` - Higher-order functions
6. `traits.lux` - Type classes
7. `behavioral.lux` - Behavioral properties
8. `tailcall.lux` - Tail call optimization
9. `statemachine.lux` - State machines with ADTs
10. `pipelines.lux` - Pipe operator
11. `generics.lux` - Generic type parameters
12. `json.lux` - JSON parsing and serialization
13. `jit_test.lux` - JIT compilation demo
14. `guessing_game.lux` - Console input/output
15. `http_server.lux` - HTTP server with routing
16. `examples/modules/main.lux` - Module imports
17. `examples/modules/main_selective.lux` - Selective imports
18. `examples/modules/main_wildcard.lux` - Wildcard imports

---

## Missing Features Analysis

### Priority 1: Critical Missing Features

#### 1.1 Generic Type Parameters
**Status:** ✅ Complete

Generic type parameters are fully working:
- Type parameter declarations in type definitions (`type List<T> = ...`)
- Generic function parameters (`fn map<T, U>(f: fn(T): U, list: List<T>): List<U>`)
- Type application and instantiation
- See `examples/generics.lux` for working examples

**Still Missing (nice-to-have):**
- Type parameter constraints (`where T: Eq`)

#### 1.2 String Interpolation
**Status:** ✅ Complete

String interpolation is fully working:
- `"Hello, {name}!"` syntax
- Nested expressions: `"Result: {1 + 2}"`
- Escape sequences: `\{`, `\}`, `\n`, `\t`, `\"`, `\\`

#### 1.3 Better Error Messages
**Status:** ⚠️ Partial

**What's Working:**
- Source code context with line/column numbers
- Caret pointing to error location
- Color-coded error output

**What's Missing:**
- Type diff display for mismatches
- "Did you mean?" suggestions
- Error recovery in parser

**Implementation Steps:**
1. Add Levenshtein distance for suggestions
2. Implement error recovery in parser
3. Add type diff visualization

### Priority 2: Effect System Completion

#### 2.1 Effect Polymorphism
**Status:** Not implemented.

**What's Missing:**
- Functions generic over their effects
- Effect variables in type signatures
- Effect constraints

**Example syntax:**
```lux
fn withRetry<E>(action: fn(): T with E, attempts: Int): T with E = ...
```

**Implementation Steps:**
1. Add effect variables to effect representation
2. Implement effect unification with variables
3. Support effect quantification in type schemes

#### 2.2 Built-in Effects
**Status:** ✅ Core effects complete

**Working Effects:**
- `Console` - print, readLine, readInt
- `File` - read, write, exists, delete, listDir, createDir
- `Http` - get, post, put, delete (HTTP client)
- `HttpServer` - listen, accept, respond, respondWithHeaders, stop
- `Random` - int, float, range, bool, element
- `Time` - now, sleep
- `State` - get, put (built-in)
- `Reader` - ask (built-in)

**Missing Effects (nice-to-have):**
- `Async` - async/await pattern
- Generic effect parameters (`State<S>`)

#### 2.3 Resumable Handlers
**Status:** Handlers exist but may not support continuation resumption.

**What's Missing:**
- `resume` keyword to continue computation
- Multi-shot continuations
- Proper effect handler semantics

### Priority 3: Schema Evolution

#### 3.1 Versioned Types
**Status:** ✅ Type system integration complete

**What's Working:**
- Version annotations preserved in type system (`Int @v1`, `User @v2`)
- Version mismatch detection at compile time
- Version constraints: `@v1` (exact), `@v2+` (at least), `@latest` (any)
- Versioned type declarations tracked
- Migration bodies stored for future execution

**Still Missing (nice-to-have):**
- Auto-migration generation
- Version-aware serialization/codecs

### Priority 4: Module System

#### 4.1 Complete Import/Export
**Status:** ✅ Complete.

The module system is fully functional with:
- `import foo/bar` - basic imports
- `import foo/bar as alias` - aliased imports
- `import foo.{a, b}` - selective imports
- `import foo.*` - wildcard imports
- `pub fn` visibility for exports
- Circular dependency detection (tested)
- Module caching
- Transitive imports
- Type checking across modules

**Still Missing (nice-to-have):**
- Re-exports (`pub import`)
- Package manifest (`package.lux`)

### Priority 5: Code Generation

#### 5.1 Compile to Target
**Status:** ⚠️ Partial - JIT works for numeric code

**What's Working:**
- `src/compiler.rs` - Cranelift JIT compiler (539 lines)
- Integer arithmetic, comparisons, boolean ops
- Conditionals, let bindings, function calls, blocks
- Recursive functions (fib(30) compiles and runs)
- CLI integration: `lux compile <file> [--benchmark]`
- ~160x speedup over interpreter for numeric code

**What's Missing in JIT:**
- Strings, floats, lists, records, tuples
- Pattern matching, ADTs
- Effects and handlers

**Target Options for Full Compilation:**
1. **WASM** - Best for web and portable deployment
2. **JavaScript** - Easiest web integration
3. **Extend Cranelift** - Build on existing JIT

**Implementation Steps:**
1. ~~Expose JIT via CLI command~~ ✅ Done
2. Add support for more types (strings, lists, etc.)
3. Compile pattern matching
4. Handle effects (CPS transform or evidence passing)

### Priority 6: Tooling

#### 6.1 Package Manager
**What's Needed:**
- Package registry
- Dependency resolution
- Version management
- Build system integration

#### 6.2 Standard Library
**What's Needed:**
- Collections (Map, Set, Array)
- String utilities
- Math functions
- File I/O
- Network I/O
- JSON/YAML parsing

#### 6.3 Debugger
**What's Needed:**
- Breakpoints
- Step execution
- Variable inspection
- Stack traces

---

## Recommended Implementation Order

### Phase 1: Language Completeness (Essential)
1. ~~**Generic type parameters**~~ ✅ Done
2. ~~**String interpolation**~~ ✅ Done
3. **Better error messages** - Elm-quality suggestions

### Phase 2: Effect System Maturity
4. ~~**Built-in effects** (Console, File, Http, Random, Time)~~ ✅ Done
5. **Effect polymorphism** - Generic effect parameters
6. **Fail effect** - Proper error handling pattern
7. **Resumable handlers** - Multi-shot continuations

### Phase 3: Production Readiness
8. ~~**Complete module system**~~ ✅ Done
9. ~~**Standard library basics**~~ ✅ Done (String, List, JSON)
10. **Package manager** - Dependency resolution
11. **Full compilation** - Extend JIT or add WASM/JS backend

### Phase 4: Behavioral Types (Verification)
12. ~~**Total function verification**~~ ✅ Done - Termination checking
13. ~~**Idempotent verification**~~ ✅ Done - Pattern-based analysis
14. ~~**Deterministic verification**~~ ✅ Done - Effect-based analysis
15. ~~**Commutative verification**~~ ✅ Done - Operator analysis
16. **Where clause enforcement** - Constraint checking (basic parsing done)

### Phase 5: Schema Evolution (Data)
17. ~~**Type system version tracking**~~ ✅ Done
18. ~~**Version mismatch detection**~~ ✅ Done
19. **Auto-migration generation** - Generate migration code
20. **Version-aware serialization** - Codecs

### Phase 6: Advanced Features
18. **Refinement types** - SMT solver integration
19. **Async/await effect**
20. **Stream processing**

---

## Technical Debt

### Known Issues Fixed During Testing
1. ✅ String concatenation with `+` operator wasn't type-checked correctly
2. ✅ ADT constructors weren't registered in type environment
3. ✅ ADT constructors weren't registered in interpreter environment
4. ✅ Handlers weren't accessible as values
5. ✅ Effect checking was too strict (required exact match instead of subset)
6. ✅ `total` keyword conflicts with variable names in examples

### Remaining Technical Debt
1. Dead code in diagnostics, modules, parser (suppressed with `#![allow(dead_code)]`)
2. Some test utilities not fully utilized
3. LSP server basic but could be expanded
4. Error recovery in parser incomplete

---

## Feature Comparison with Other Languages

| Feature | Lux | Koka | Haskell | Rust | TypeScript |
|---------|-----|------|---------|------|------------|
| Algebraic Effects | ✅ | ✅ | Via libs | ❌ | ❌ |
| Type Inference | ✅ | ✅ | ✅ | Partial | Partial |
| ADTs | ✅ | ✅ | ✅ | ✅ | Via unions |
| Pattern Matching | ✅ | ✅ | ✅ | ✅ | Limited |
| Generics | ✅ | ✅ | ✅ | ✅ | ✅ |
| Type Classes | Basic | ✅ | ✅ | ✅ | ❌ |
| String Interpolation | ✅ | ✅ | ❌ | ❌ | ✅ |
| Effect Polymorphism | ❌ | ✅ | Via mtl | N/A | N/A |
| Schema Evolution | ✅ | ❌ | ❌ | ❌ | ❌ |
| Behavioral Types | ✅ | ❌ | ❌ | ❌ | ❌ |
| HTTP Server | ✅ | ❌ | Via libs | Via libs | Via libs |
| Refinement Types | Planned | ❌ | Via LH | ❌ | ❌ |
| Tail Call Opt | ✅ | ✅ | ✅ | Limited | ❌ |
| JIT Compilation | ⚠️ Numeric | ✅ | ✅ | N/A | N/A |
| REPL | ✅ | ✅ | ✅ | Limited | ✅ |
| LSP | Basic | ✅ | ✅ | ✅ | ✅ |

---

## Unique Value Proposition

Lux differentiates itself through:

1. **First-class algebraic effects** - Making side effects explicit, testable, and composable
2. **Schema evolution** - Type-safe version tracking with compile-time mismatch detection
3. **Behavioral types** - Compile-time verification of purity, totality, determinism, idempotency
4. **Built-in HTTP server** - Effect-tracked web servers without external frameworks
5. **Developer experience** - Elm-style errors, REPL, LSP support

The combination of these features makes Lux particularly suited for:
- Building reliable backend services with explicit effect tracking
- Applications with complex state management
- Systems requiring careful versioning and migration
- Projects where testing and verification are critical
- Educational use for learning algebraic effects