lux/docs/ROADMAP.md

# Lux Development Roadmap

*Targeting practical use cases in priority order.*

---

## Feature Status Summary

### Schema Evolution
| Component | Status |
|-----------|--------|
| Parser (`@v1`, `@v2`, `from @v1 = ...`) | ✅ Complete |
| AST representation | ✅ Complete |
| Runtime versioned values | ✅ Complete |
| Schema registry & compatibility checking | ✅ Complete |
| Basic migration execution | ✅ Complete |
| Type system integration | ✅ Complete |
| Auto-migration generation | ✅ Complete |
| Serialization/codec support | ❌ Missing |

### Behavioral Types
| Component | Status |
|-----------|--------|
| Parser (`is pure`, `is total`, etc.) | ✅ Complete |
| AST & PropertySet | ✅ Complete |
| Pure function checking (no effects) | ✅ Complete |
| Total verification (no Fail, structural recursion) | ✅ Complete |
| Idempotent verification (pattern-based) | ✅ Complete |
| Deterministic verification (no Random/Time) | ✅ Complete |
| Commutative verification (2 params, commutative op) | ✅ Complete |
| Where clause property constraints | ✅ Complete |

---

## Use Case 1: Backend Services with Complex Business Logic

**Value Proposition:** Effect tracking shows exactly what each function does. Testing is trivial.

### Phase 1.1: Make It Buildable (Current Blockers)

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Generic type parameters | P0 | — | ✅ Complete |
| String interpolation | P1 | — | ✅ Complete |
| File effect (read/write) | P1 | — | ✅ Complete |
| HTTP effect (client) | P1 | — | ✅ Complete |
| JSON parsing/serialization | P1 | — | ✅ Complete |

### Phase 1.2: Database & Persistence

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| SQL effect (query, execute) | P1 | 2 weeks | ✅ Complete |
| Transaction effect | P2 | 1 week | ✅ Complete |
| Connection pooling | P2 | 1 week | ❌ Missing |
| PostgreSQL support | P1 | 2 weeks | ✅ Complete |

### Phase 1.3: Web Server Framework

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| HTTP server effect | P1 | 2 weeks | ✅ Complete |
| Routing DSL | P2 | 1 week | ❌ Missing |
| Middleware pattern | P2 | 1 week | ❌ Missing |
| Request/Response types | P1 | 3 days | ✅ Complete (via HttpServer effect) |

### Phase 1.4: Developer Experience

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Elm-quality error messages | P1 | 2 weeks | ✅ Complete (field suggestions, error categories) |
| Full JS compilation | P2 | 4 weeks | ✅ Complete |
| Hot reload / watch mode | P2 | — | ✅ Complete |
| Debugger improvements | P3 | 2 weeks | ✅ Basic |
| C backend CLI integration | P1 | — | ✅ Complete (`lux compile`) |

### Success Criteria for Use Case 1
- [ ] Can build a REST API with database access
- [ ] Can swap database handler for testing without mocks
- [ ] Effect signatures document all side effects
- [ ] Compile-time guarantees prevent common bugs

---

## Use Case 2: High-Reliability Systems

**Value Proposition:** Behavioral types provide compile-time guarantees (`is total`, `is idempotent`).

### Phase 2.1: Complete Behavioral Type Verification

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Total function verification | P1 | 2 weeks | ✅ Complete |
| Idempotent verification | P1 | 2 weeks | ✅ Complete |
| Deterministic verification | P1 | 1 week | ✅ Complete |
| Where clause enforcement | P1 | 1 week | ✅ Complete |

**Implementation approach (completed):**
- **Total:** Checks for no Fail effect + structural recursion for termination
- **Idempotent:** Pattern-based recognition (constants, identity, clamping, abs, projections)
- **Deterministic:** Effect analysis (no Random or Time effects)
- **Commutative:** Requires 2 params with commutative operation (+, *, min, max, etc.)

### Phase 2.2: Refinement Types (Stretch Goal)

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Basic refinements (`Int where self > 0`) | P2 | 3 weeks | ❌ Missing |
| SMT solver integration (Z3) | P3 | 4 weeks | ❌ Missing |
| Contract checking | P3 | 2 weeks | ❌ Missing |

### Phase 2.3: Fault Tolerance Patterns

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Retry effect with backoff | P2 | 1 week | ❌ Missing |
| Circuit breaker pattern | P2 | 1 week | ❌ Missing |
| Supervision trees (Elixir-style) | P3 | 3 weeks | ❌ Missing |

### Success Criteria for Use Case 2
- [ ] Compiler verifies `is total` functions always terminate
- [ ] Compiler verifies `is idempotent` functions are safe to retry
- [ ] Can build payment processing with idempotency guarantees
- [ ] Effect system enables fault isolation

---

## Use Case 3: Data Pipelines / ETL Systems

**Value Proposition:** Schema evolution handles changing data formats gracefully.

### Phase 3.1: Complete Schema Evolution

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Type system version tracking | P1 | 1 week | ✅ Complete |
| Auto-migration generation | P1 | 2 weeks | ✅ Complete |
| Version compatibility errors | P1 | 1 week | ✅ Complete |
| Migration chain optimization | P2 | 1 week | ✅ Complete |

### Phase 3.2: Serialization Support

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| JSON codec generation | P1 | 2 weeks | ❌ Missing |
| Version-aware deserialization | P1 | 1 week | ❌ Missing |
| Binary format support | P3 | 2 weeks | ❌ Missing |
| Avro/Protobuf interop | P3 | 3 weeks | ❌ Missing |

### Phase 3.3: Stream Processing

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Stream effect | P2 | 2 weeks | ❌ Missing |
| Batch processing patterns | P2 | 1 week | ❌ Missing |
| Backpressure handling | P3 | 2 weeks | ❌ Missing |

### Success Criteria for Use Case 3
- [ ] Can define versioned types with automatic migrations
- [ ] Compiler catches version mismatches
- [ ] Can deserialize old data formats automatically
- [ ] Can process data streams with effect tracking

---

## Use Case 4: Teaching Functional Programming

**Value Proposition:** Effects are more intuitive than monads for learning.

### Phase 4.1: Educational Materials

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Beginner tutorial (effects intro) | P1 | 1 week | ⚠️ Basic |
| Interactive examples | P2 | 1 week | ❌ Missing |
| Comparison guides (vs Haskell monads) | P2 | 3 days | ❌ Missing |
| Video tutorials | P3 | 2 weeks | ❌ Missing |

### Phase 4.2: REPL Improvements

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Step-by-step evaluation | P2 | 1 week | ❌ Missing |
| Effect visualization | P2 | 2 weeks | ❌ Missing |
| Type hole support (`_` placeholders) | P2 | 1 week | ❌ Missing |

### Phase 4.3: Error Message Excellence

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Elm-style error catalog | P1 | 2 weeks | ❌ Missing |
| "Did you mean?" suggestions | P1 | 1 week | ❌ Missing |
| Example-based hints | P2 | 1 week | ❌ Missing |
| Beginner-friendly mode | P3 | 1 week | ❌ Missing |

### Success Criteria for Use Case 4
- [ ] New FP learner can understand effects in < 1 hour
- [ ] Error messages guide toward solutions
- [ ] Clear progression from pure functions to effects to handlers

---

## Cross-Cutting Concerns (All Use Cases)

### Ecosystem

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| Package manager (lux pkg) | P1 | 3 weeks | ✅ Complete |
| Module loader integration | P1 | 1 week | ✅ Complete |
| Package registry server | P2 | 2 weeks | ✅ Complete |
| Registry CLI (search, publish) | P2 | 1 week | ✅ Complete |
| Lock file generation | P1 | 1 week | ✅ Complete |
| Version constraint parsing | P1 | 1 week | ✅ Complete |
| Transitive dependency resolution | P2 | 2 weeks | ⚠️ Basic (direct deps only) |

**Package Manager Features:**
- `lux pkg init` - Initialize project with lux.toml
- `lux pkg add/remove` - Manage dependencies
- `lux pkg install` - Install from lux.toml
- Git and local path dependencies
- Automatic module resolution from .lux_packages/

### Tooling

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| LSP completions | P1 | 1 week | ✅ Complete (module-specific completions) |
| LSP go-to-definition | P1 | 1 week | ✅ Complete (functions, lets, types) |
| Formatter | P2 | — | ✅ Complete |
| Documentation generator | P2 | 1 week | ❌ Missing |

### Compilation

| Task | Priority | Effort | Status |
|------|----------|--------|--------|
| C backend (basic) | P1 | — | ✅ Complete (functions, Console.print) |
| Extend C backend (closures) | P1 | — | ✅ Complete |
| Extend C backend (pattern matching) | P1 | — | ✅ Complete |
| Extend C backend (lists) | P1 | — | ✅ Complete |
| JS backend | P2 | 4 weeks | ✅ Complete |
| WASM backend | P3 | 4 weeks | ❌ Missing |

**JS Backend Features:**
- Core language: functions, closures, ADTs, pattern matching
- Standard library: String, List, Option, Result, Math, JSON
- Effects: Console, Random, Time, Http, Dom
- Browser support: Html module, TEA runtime, DOM manipulation
- CLI: `lux compile --target js`

---

## Recommended Implementation Order

### Quarter 1: Foundation ✅ COMPLETE

1. ~~**Generic type parameters**~~ ✅ Done
2. ~~**String interpolation**~~ ✅ Done
3. ~~**File effect**~~ ✅ Done
4. ~~**HTTP client effect**~~ ✅ Done
5. ~~**JSON support**~~ ✅ Done
6. ~~**Elm-quality errors**~~ ✅ Done

### Quarter 2: Backend Services (Use Case 1) ✅ COMPLETE

7. ~~**HTTP server effect**~~ ✅ Done
8. ~~**SQL effect**~~ ✅ Done
9. ~~**Full JS compilation**~~ ✅ Done
10. ~~**Package manager**~~ ✅ Done

### Quarter 3: Reliability (Use Case 2) ✅ COMPLETE

11. ~~**Behavioral type verification**~~ ✅ Done
12. ~~**Where clause enforcement**~~ ✅ Done
13. ~~**Schema evolution completion**~~ ✅ Done
14. ~~**Auto-migration generation**~~ ✅ Done

### Quarter 4: Polish (Use Cases 3 & 4)

15. **Serialization codecs** — Data pipelines
16. **Educational materials** — Teaching
17. **Refinement types** — Advanced guarantees
18. **Stream processing** — ETL use case

---

## What's Already Done (Often Forgotten)

**Core Language:**
- ✅ Generic type parameters (`fn map<T, U>`, `type List<T>`)
- ✅ String interpolation (`"Hello {name}!"`)
- ✅ Pattern matching with exhaustiveness checking
- ✅ Algebraic data types
- ✅ Type inference (Hindley-Milner)
- ✅ Tail call optimization

**Effect System:**
- ✅ Effect declarations and handlers
- ✅ Console effect (print, readLine, readInt)
- ✅ File effect (read, write, exists, delete, listDir, mkdir)
- ✅ HTTP client effect (get, post, put, delete)
- ✅ HTTP server effect (listen, accept, respond, stop)
- ✅ Process effect (exec, env, args, cwd, exit)
- ✅ Random effect (int, float, range, bool)
- ✅ Time effect (now, sleep)
- ✅ Test effect (assert, assertEqual, assertTrue, assertFalse)
- ✅ SQL effect (SQLite with transactions)
- ✅ Postgres effect (PostgreSQL connections)

**Module System:**
- ✅ Imports, exports, aliases
- ✅ Selective imports (`import foo.{a, b}`)
- ✅ Wildcard imports (`import foo.*`)
- ✅ Circular dependency detection

**Standard Library:**
- ✅ String operations (split, join, trim, contains, replace, toUpper, toLower, etc.)
- ✅ List operations (map, filter, fold, head, tail, concat, reverse, find, etc.)
- ✅ Option operations (map, flatMap, getOrElse, isSome, isNone)
- ✅ Result operations (map, flatMap, getOrElse, isOk, isErr)
- ✅ Math operations (abs, min, max, sqrt, pow, floor, ceil, round)
- ✅ JSON parsing/serialization (parse, stringify, get, object, array)

**Tooling:**
- ✅ C backend (functions, closures, pattern matching, lists)
- ✅ JS backend (full language support, browser & Node.js)
- ✅ REPL with history
- ✅ LSP server (diagnostics, hover, completions, go-to-definition, references, symbols)
- ✅ Formatter
- ✅ Watch mode
- ✅ Debugger (basic)

**Advanced:**
- ✅ Schema evolution (parser, runtime, migrations, compatibility checking)
- ✅ Behavioral types (pure, total, idempotent, deterministic, commutative verification)

---

## Success Metrics

### Use Case 1: Backend Services
- Can build and deploy a production API
- 10+ companies using Lux for backends

### Use Case 2: High-Reliability
- Compiler catches idempotency violations
- Used in fintech/healthcare context

### Use Case 3: Data Pipelines
- Schema migrations happen automatically
- Zero data loss from version mismatches

### Use Case 4: Teaching
- Featured in FP courses
- "Effects finally make sense" testimonials