Brandon Lucas
33b4f57faf
- Add String.fromChar, chars, substring, toUpper, toLower, replace, startsWith, endsWith, join to C backend - Fix record type alias unification by adding expand_type_alias and unify_with_env functions - Update docs to reflect current implementation status - Clean up outdated roadmap items and fix inconsistencies - Add comprehensive language comparison document Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
392 lines
12 KiB
Markdown
392 lines
12 KiB
Markdown
# Lux Language Implementation Plan
|
|
|
|
## Current Status Summary
|
|
|
|
### What's Working (222 tests passing)
|
|
|
|
**Core Language:**
|
|
- Lexer and parser for core syntax
|
|
- AST representation
|
|
- Type checker with Hindley-Milner inference
|
|
- Interpreter with REPL
|
|
- Tail call optimization (TCO)
|
|
- Pattern matching with exhaustiveness checking
|
|
- Algebraic data types (ADTs) with constructors
|
|
- Records and tuples
|
|
- Higher-order functions and closures
|
|
- Pipe operator (`|>`)
|
|
- Documentation comments
|
|
- **Generic type parameters** (`fn map<T, U>`, `type List<T>`)
|
|
- **String interpolation** (`"Hello {name}!"`)
|
|
- **Escape sequences** (`\n`, `\t`, `\"`, `\{`, `\}`)
|
|
|
|
**Effect System:**
|
|
- Effect declarations
|
|
- Effect signatures on functions (`with {Effect}`)
|
|
- Handler definitions
|
|
- `run ... with` syntax
|
|
- Effect inference within function bodies
|
|
- Effect subset checking (pure functions callable in effectful contexts)
|
|
|
|
**Built-in Effects:**
|
|
- `Console` - print, readLine, readInt
|
|
- `File` - read, write, exists, delete, listDir, createDir
|
|
- `Http` - get, post, put, delete (HTTP client)
|
|
- `HttpServer` - listen, accept, respond, respondWithHeaders, stop (HTTP server)
|
|
- `Random` - int, float, range, bool, element
|
|
- `Time` - now, sleep
|
|
|
|
**Type Classes:**
|
|
- Trait declarations
|
|
- Implementation blocks
|
|
- Basic trait method dispatch
|
|
|
|
**Behavioral Properties:**
|
|
- Property declarations (`is pure`, `is idempotent`, `is total`, `is deterministic`, `is commutative`)
|
|
- Property parsing and storage in AST
|
|
- Pure function verification (no effects allowed)
|
|
- Deterministic verification (no Random/Time effects)
|
|
- Commutative verification (operator-based analysis)
|
|
- Idempotent verification (pattern-based analysis)
|
|
- Total verification (structural recursion, termination checking)
|
|
|
|
**Module System:**
|
|
- Import/export with `pub` visibility
|
|
- Aliased imports (`import foo as bar`)
|
|
- Selective imports (`import foo.{a, b}`)
|
|
- Wildcard imports (`import foo.*`)
|
|
- Circular dependency detection
|
|
- Transitive imports
|
|
|
|
**JIT Compiler (Cranelift):**
|
|
- Integer arithmetic and comparisons
|
|
- Conditionals, let bindings, blocks
|
|
- Function calls including recursion
|
|
- ~160x speedup over interpreter for numeric code
|
|
- CLI integration via `lux compile <file>`
|
|
|
|
**Standard Library:**
|
|
- String operations (substring, length, split, trim, etc.)
|
|
- List operations (map, filter, fold, etc.)
|
|
- JSON parsing and serialization
|
|
- Math functions
|
|
|
|
**Tooling:**
|
|
- REPL with history and autocomplete
|
|
- LSP server (diagnostics, hover, completions)
|
|
- Watch mode for auto-recompilation
|
|
- Source-context error messages
|
|
|
|
### Example Programs Working
|
|
1. `hello.lux` - Basic effect usage
|
|
2. `factorial.lux` - Recursion
|
|
3. `effects.lux` - Custom effects and handlers
|
|
4. `datatypes.lux` - ADTs and pattern matching
|
|
5. `functional.lux` - Higher-order functions
|
|
6. `traits.lux` - Type classes
|
|
7. `behavioral.lux` - Behavioral properties
|
|
8. `tailcall.lux` - Tail call optimization
|
|
9. `statemachine.lux` - State machines with ADTs
|
|
10. `pipelines.lux` - Pipe operator
|
|
11. `generics.lux` - Generic type parameters
|
|
12. `json.lux` - JSON parsing and serialization
|
|
13. `jit_test.lux` - JIT compilation demo
|
|
14. `guessing_game.lux` - Console input/output
|
|
15. `http_server.lux` - HTTP server with routing
|
|
16. `examples/modules/main.lux` - Module imports
|
|
17. `examples/modules/main_selective.lux` - Selective imports
|
|
18. `examples/modules/main_wildcard.lux` - Wildcard imports
|
|
|
|
---
|
|
|
|
## Missing Features Analysis
|
|
|
|
### Priority 1: Critical Missing Features
|
|
|
|
#### 1.1 Generic Type Parameters
|
|
**Status:** ✅ Complete
|
|
|
|
Generic type parameters are fully working:
|
|
- Type parameter declarations in type definitions (`type List<T> = ...`)
|
|
- Generic function parameters (`fn map<T, U>(f: fn(T): U, list: List<T>): List<U>`)
|
|
- Type application and instantiation
|
|
- See `examples/generics.lux` for working examples
|
|
|
|
**Still Missing (nice-to-have):**
|
|
- Type parameter constraints (`where T: Eq`)
|
|
|
|
#### 1.2 String Interpolation
|
|
**Status:** ✅ Complete
|
|
|
|
String interpolation is fully working:
|
|
- `"Hello, {name}!"` syntax
|
|
- Nested expressions: `"Result: {1 + 2}"`
|
|
- Escape sequences: `\{`, `\}`, `\n`, `\t`, `\"`, `\\`
|
|
|
|
#### 1.3 Better Error Messages
|
|
**Status:** ✅ Complete (Elm-quality)
|
|
|
|
**What's Working:**
|
|
- Source code context with line/column numbers
|
|
- Caret pointing to error location
|
|
- Color-coded error output
|
|
- Field suggestions for unknown fields
|
|
- Error categorization
|
|
- Improved hints
|
|
|
|
**Nice-to-have (not critical):**
|
|
- Error recovery in parser
|
|
- Type diff visualization
|
|
|
|
### Priority 2: Effect System Completion
|
|
|
|
#### 2.1 Effect Polymorphism
|
|
**Status:** Not implemented.
|
|
|
|
**What's Missing:**
|
|
- Functions generic over their effects
|
|
- Effect variables in type signatures
|
|
- Effect constraints
|
|
|
|
**Example syntax:**
|
|
```lux
|
|
fn withRetry<E>(action: fn(): T with E, attempts: Int): T with E = ...
|
|
```
|
|
|
|
**Implementation Steps:**
|
|
1. Add effect variables to effect representation
|
|
2. Implement effect unification with variables
|
|
3. Support effect quantification in type schemes
|
|
|
|
#### 2.2 Built-in Effects
|
|
**Status:** ✅ Core effects complete
|
|
|
|
**Working Effects:**
|
|
- `Console` - print, readLine, readInt
|
|
- `File` - read, write, exists, delete, listDir, createDir
|
|
- `Http` - get, post, put, delete (HTTP client)
|
|
- `HttpServer` - listen, accept, respond, respondWithHeaders, stop
|
|
- `Random` - int, float, range, bool, element
|
|
- `Time` - now, sleep
|
|
- `State` - get, put (built-in)
|
|
- `Reader` - ask (built-in)
|
|
|
|
**Missing Effects (nice-to-have):**
|
|
- `Async` - async/await pattern
|
|
- Generic effect parameters (`State<S>`)
|
|
|
|
#### 2.3 Resumable Handlers
|
|
**Status:** Handlers exist but may not support continuation resumption.
|
|
|
|
**What's Missing:**
|
|
- `resume` keyword to continue computation
|
|
- Multi-shot continuations
|
|
- Proper effect handler semantics
|
|
|
|
### Priority 3: Schema Evolution
|
|
|
|
#### 3.1 Versioned Types
|
|
**Status:** ✅ Type system integration complete
|
|
|
|
**What's Working:**
|
|
- Version annotations preserved in type system (`Int @v1`, `User @v2`)
|
|
- Version mismatch detection at compile time
|
|
- Version constraints: `@v1` (exact), `@v2+` (at least), `@latest` (any)
|
|
- Versioned type declarations tracked
|
|
- Migration bodies stored for future execution
|
|
|
|
**What's Working:**
|
|
- Auto-migration generation
|
|
|
|
**Still Missing (nice-to-have):**
|
|
- Version-aware serialization/codecs
|
|
|
|
### Priority 4: Module System
|
|
|
|
#### 4.1 Complete Import/Export
|
|
**Status:** ✅ Complete.
|
|
|
|
The module system is fully functional with:
|
|
- `import foo/bar` - basic imports
|
|
- `import foo/bar as alias` - aliased imports
|
|
- `import foo.{a, b}` - selective imports
|
|
- `import foo.*` - wildcard imports
|
|
- `pub fn` visibility for exports
|
|
- Circular dependency detection (tested)
|
|
- Module caching
|
|
- Transitive imports
|
|
- Type checking across modules
|
|
|
|
**Still Missing (nice-to-have):**
|
|
- Re-exports (`pub import`)
|
|
- Package manifest (`package.lux`)
|
|
|
|
### Priority 5: Code Generation
|
|
|
|
#### 5.1 Compile to Target
|
|
**Status:** ⚠️ Partial - JIT works for numeric code
|
|
|
|
**What's Working:**
|
|
- `src/compiler.rs` - Cranelift JIT compiler (539 lines)
|
|
- Integer arithmetic, comparisons, boolean ops
|
|
- Conditionals, let bindings, function calls, blocks
|
|
- Recursive functions (fib(30) compiles and runs)
|
|
- CLI integration: `lux compile <file> [--benchmark]`
|
|
- ~160x speedup over interpreter for numeric code
|
|
|
|
**What's Missing in JIT:**
|
|
- Strings, floats, lists, records, tuples
|
|
- Pattern matching, ADTs
|
|
- Effects and handlers
|
|
|
|
**Target Options for Full Compilation:**
|
|
1. **WASM** - Best for web and portable deployment
|
|
2. **JavaScript** - Easiest web integration
|
|
3. **Extend Cranelift** - Build on existing JIT
|
|
|
|
**Implementation Steps:**
|
|
1. ~~Expose JIT via CLI command~~ ✅ Done
|
|
2. Add support for more types (strings, lists, etc.)
|
|
3. Compile pattern matching
|
|
4. Handle effects (CPS transform or evidence passing)
|
|
|
|
### Priority 6: Tooling
|
|
|
|
#### 6.1 Package Manager
|
|
**Status:** ✅ Complete
|
|
|
|
**What's Working:**
|
|
- `lux pkg init` - Initialize project with lux.toml
|
|
- `lux pkg add/remove` - Manage dependencies
|
|
- `lux pkg install` - Install from lux.toml
|
|
- Git and local path dependencies
|
|
- Package registry (`lux registry`)
|
|
- CLI commands (search, publish)
|
|
|
|
**Still Missing (nice-to-have):**
|
|
- Version conflict resolution
|
|
|
|
#### 6.2 Standard Library
|
|
**Status:** ✅ Complete
|
|
|
|
**What's Working:**
|
|
- String operations (substring, length, split, trim, etc.)
|
|
- List operations (map, filter, fold, etc.)
|
|
- Option and Result operations
|
|
- Math functions
|
|
- JSON parsing and serialization
|
|
|
|
**Still Missing (nice-to-have):**
|
|
- Collections (Map, Set)
|
|
- YAML parsing
|
|
|
|
#### 6.3 Debugger
|
|
**Status:** ✅ Basic
|
|
|
|
**What's Working:**
|
|
- Basic debugger
|
|
|
|
**Nice-to-have:**
|
|
- Breakpoints
|
|
- Step execution
|
|
- Variable inspection
|
|
|
|
---
|
|
|
|
## Recommended Implementation Order
|
|
|
|
### Phase 1: Language Completeness (Essential)
|
|
1. ~~**Generic type parameters**~~ ✅ Done
|
|
2. ~~**String interpolation**~~ ✅ Done
|
|
3. **Better error messages** - Elm-quality suggestions
|
|
|
|
### Phase 2: Effect System Maturity
|
|
4. ~~**Built-in effects** (Console, File, Http, Random, Time)~~ ✅ Done
|
|
5. **Effect polymorphism** - Generic effect parameters
|
|
6. **Fail effect** - Proper error handling pattern
|
|
7. **Resumable handlers** - Multi-shot continuations
|
|
|
|
### Phase 3: Production Readiness
|
|
8. ~~**Complete module system**~~ ✅ Done
|
|
9. ~~**Standard library basics**~~ ✅ Done (String, List, JSON)
|
|
10. **Package manager** - Dependency resolution
|
|
11. **Full compilation** - Extend JIT or add WASM/JS backend
|
|
|
|
### Phase 4: Behavioral Types (Verification)
|
|
12. ~~**Total function verification**~~ ✅ Done - Termination checking
|
|
13. ~~**Idempotent verification**~~ ✅ Done - Pattern-based analysis
|
|
14. ~~**Deterministic verification**~~ ✅ Done - Effect-based analysis
|
|
15. ~~**Commutative verification**~~ ✅ Done - Operator analysis
|
|
16. ~~**Where clause enforcement**~~ ✅ Done - Property constraints
|
|
|
|
### Phase 5: Schema Evolution (Data)
|
|
17. ~~**Type system version tracking**~~ ✅ Done
|
|
18. ~~**Version mismatch detection**~~ ✅ Done
|
|
19. **Auto-migration generation** - Generate migration code
|
|
20. **Version-aware serialization** - Codecs
|
|
|
|
### Phase 6: Advanced Features
|
|
18. **Refinement types** - SMT solver integration
|
|
19. **Async/await effect**
|
|
20. **Stream processing**
|
|
|
|
---
|
|
|
|
## Technical Debt
|
|
|
|
### Known Issues Fixed During Testing
|
|
1. ✅ String concatenation with `+` operator wasn't type-checked correctly
|
|
2. ✅ ADT constructors weren't registered in type environment
|
|
3. ✅ ADT constructors weren't registered in interpreter environment
|
|
4. ✅ Handlers weren't accessible as values
|
|
5. ✅ Effect checking was too strict (required exact match instead of subset)
|
|
6. ✅ `total` keyword conflicts with variable names in examples
|
|
|
|
### Remaining Technical Debt
|
|
1. Dead code in diagnostics, modules, parser (suppressed with `#![allow(dead_code)]`)
|
|
2. Some test utilities not fully utilized
|
|
3. LSP server basic but could be expanded
|
|
4. Error recovery in parser incomplete
|
|
|
|
---
|
|
|
|
## Feature Comparison with Other Languages
|
|
|
|
| Feature | Lux | Koka | Haskell | Rust | TypeScript |
|
|
|---------|-----|------|---------|------|------------|
|
|
| Algebraic Effects | ✅ | ✅ | Via libs | ❌ | ❌ |
|
|
| Type Inference | ✅ | ✅ | ✅ | Partial | Partial |
|
|
| ADTs | ✅ | ✅ | ✅ | ✅ | Via unions |
|
|
| Pattern Matching | ✅ | ✅ | ✅ | ✅ | Limited |
|
|
| Generics | ✅ | ✅ | ✅ | ✅ | ✅ |
|
|
| Type Classes | Basic | ✅ | ✅ | ✅ | ❌ |
|
|
| String Interpolation | ✅ | ✅ | ❌ | ❌ | ✅ |
|
|
| Effect Polymorphism | ❌ | ✅ | Via mtl | N/A | N/A |
|
|
| Schema Evolution | ✅ | ❌ | ❌ | ❌ | ❌ |
|
|
| Behavioral Types | ✅ | ❌ | ❌ | ❌ | ❌ |
|
|
| HTTP Server | ✅ | ❌ | Via libs | Via libs | Via libs |
|
|
| Refinement Types | Planned | ❌ | Via LH | ❌ | ❌ |
|
|
| Tail Call Opt | ✅ | ✅ | ✅ | Limited | ❌ |
|
|
| JIT Compilation | ⚠️ Numeric | ✅ | ✅ | N/A | N/A |
|
|
| REPL | ✅ | ✅ | ✅ | Limited | ✅ |
|
|
| LSP | Basic | ✅ | ✅ | ✅ | ✅ |
|
|
|
|
---
|
|
|
|
## Unique Value Proposition
|
|
|
|
Lux differentiates itself through:
|
|
|
|
1. **First-class algebraic effects** - Making side effects explicit, testable, and composable
|
|
2. **Schema evolution** - Type-safe version tracking with compile-time mismatch detection
|
|
3. **Behavioral types** - Compile-time verification of purity, totality, determinism, idempotency
|
|
4. **Built-in HTTP server** - Effect-tracked web servers without external frameworks
|
|
5. **Developer experience** - Elm-style errors, REPL, LSP support
|
|
|
|
The combination of these features makes Lux particularly suited for:
|
|
- Building reliable backend services with explicit effect tracking
|
|
- Applications with complex state management
|
|
- Systems requiring careful versioning and migration
|
|
- Projects where testing and verification are critical
|
|
- Educational use for learning algebraic effects
|