lux/docs/SQL_DESIGN_ANALYSIS.md

# SQL in Lux: Built-in Effect vs Package

## Executive Summary

This document analyzes whether SQL database access should be a built-in language feature (as it currently is) or a separate package. After comparing approaches across 12+ languages, the recommendation is:

**Keep SQL as a built-in effect, but refactor the implementation to be more modular.**

## Current Implementation

Lux currently implements SQL as a built-in effect:

```lux
fn main(): Unit with {Console, Sql} = {
    let db = Sql.openMemory()
    Sql.execute(db, "CREATE TABLE users (...)")
    let users = Sql.query(db, "SELECT * FROM users")
    Sql.close(db)
}
```

The implementation uses rusqlite (SQLite) compiled directly into the Lux binary.

## How Other Languages Handle Database Access

### Languages with Built-in Database Support

| Language | Approach | Notes |
|----------|----------|-------|
| **Python** | `sqlite3` in stdlib | Most languages have SQLite in stdlib |
| **Ruby** | `sqlite3` gem + AR are common | ActiveRecord is de facto standard |
| **Go** | `database/sql` interface in stdlib | Drivers are packages |
| **Elixir** | Ecto as separate package | But universally used |
| **PHP** | PDO in core | Multiple backends |

### Languages with Package-Only Database Support

| Language | Approach | Notes |
|----------|----------|-------|
| **Rust** | rusqlite, diesel, sqlx packages | No stdlib database |
| **Node.js** | pg, mysql2, better-sqlite3 | Packages only |
| **Haskell** | postgresql-simple, persistent | Packages only |
| **OCaml** | caqti, postgresql-ocaml | Packages only |

### Analysis of Each Approach

#### Go's Model: Interface in Stdlib + Driver Packages

```go
import (
    "database/sql"
    _ "github.com/lib/pq"  // PostgreSQL driver
)

db, _ := sql.Open("postgres", "...")
rows, _ := db.Query("SELECT * FROM users")
```

**Pros:**
- Standard interface for all databases
- Type-safe at compile time
- Drivers are swappable

**Cons:**
- Requires understanding interfaces
- Need external packages for actual database

#### Python's Model: SQLite in Stdlib

```python
import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()
c.execute('SELECT * FROM users')
```

**Pros:**
- Zero dependencies for getting started
- Great for learning/prototyping
- Always available

**Cons:**
- Other databases need packages
- stdlib vs package API differences

#### Rust's Model: Everything is Packages

```rust
use rusqlite::{Connection, Result};

fn main() -> Result<()> {
    let conn = Connection::open("test.db")?;
    conn.execute("CREATE TABLE users (...)", [])?;
    Ok(())
}
```

**Pros:**
- Minimal core language
- Best-in-class implementations
- Clear ownership

**Cons:**
- Cargo.toml management
- Version conflicts possible
- Learning curve for package ecosystem

#### Elixir's Model: Strong Package Ecosystem

```elixir
# Ecto is technically a package but universally used
Repo.all(from u in User, where: u.age > 18)
```

**Pros:**
- Best API emerges naturally
- Core team can focus on language
- Community ownership

**Cons:**
- Package can become outdated
- Multiple competing solutions

## Arguments For Built-in SQL

### 1. Effect System Integration

The most compelling argument: **SQL fits naturally into Lux's effect system.**

```lux
// The effect signature documents database access
fn fetchUser(id: Int): User with {Sql} = { ... }

// Handlers enable testing without mocks
handler testDatabase(): Sql { ... }
```

This is harder to achieve with packages - they'd need to integrate deeply with the effect system.

### 2. Zero-Dependency Getting Started

New users can immediately:
- Follow tutorials that use databases
- Build real applications
- Learn effects with practical examples

```bash
lux run database_example.lux
# Just works - no package installation
```

### 3. Guaranteed API Stability

Built-in effects have stable, documented APIs. Package APIs can change between versions.

### 4. Teaching Functional Effects

SQL is an excellent teaching example for effects:
- Clear side effects (I/O to database)
- Handler swapping for testing
- Transaction scoping

### 5. Practical Utility

90%+ of real applications need database access. Making it trivial benefits most users.

## Arguments For SQL as Package

### 1. Smaller Binary Size

rusqlite adds significant binary size (~2-3MB). Package-based approach lets users opt-in.

### 2. Database Backend Choice

Currently locked to SQLite. A package ecosystem could offer:
- `lux-sqlite`
- `lux-postgres`
- `lux-mysql`
- `lux-mongodb`

### 3. Faster Core Language Evolution

Core team focuses on language; community builds integrations.

### 4. Better Specialization

Dedicated package maintainers might build better database tooling than core team.

### 5. Multiple Competing Implementations

Competition drives quality. The best SQL package wins adoption.

## Comparison Matrix

| Factor | Built-in | Package |
|--------|----------|---------|
| Effect integration | Excellent | Needs design work |
| Learning curve | Low | Medium |
| Binary size | Larger | User controls |
| Database options | Limited | Unlimited |
| API stability | Guaranteed | Version-dependent |
| Getting started | Instant | Requires install |
| Testing story | Built-in handlers | Package-specific |
| Maintenance burden | Core team | Community |

## Recommendation

### Keep SQL as Built-in Effect, With Changes

**Rationale:**

1. **Effect system is Lux's differentiator** - SQL showcases it perfectly
2. **Practicality matters** - 90% of apps need databases
3. **Teaching value** - SQL is ideal for learning effects
4. **Handler testing** - Built-in integration enables powerful testing

### Proposed Architecture

```
Core Lux
├── Sql effect (interface only)
│   ├── open/close
│   ├── execute/query
│   └── transaction operations
│
└── Default SQLite handler (built-in)
    └── Uses rusqlite

Future packages (optional)
├── lux-postgres  -- PostgreSQL handler
├── lux-mysql     -- MySQL handler
└── lux-redis     -- Redis (key-value, not Sql)
```

### Specific Changes to Consider

1. **Make SQLite compilation optional**
   ```toml
   # Cargo.toml
   [features]
   default = ["sqlite"]
   sqlite = ["rusqlite"]
   ```

2. **Define stable Sql effect interface**
   ```lux
   effect Sql {
       fn open(path: String): SqlConn
       fn close(conn: SqlConn): Unit
       fn execute(conn: SqlConn, sql: String): Int
       fn query(conn: SqlConn, sql: String): List<SqlRow>
       // ...
   }
   ```

3. **Allow package handlers to implement Sql**
   ```lux
   // In lux-postgres package
   handler postgresHandler(connStr: String): Sql { ... }

   // Usage
   run myApp() with {
       Sql -> postgresHandler("postgres://...")
   }
   ```

4. **Add connection pooling to core**
   Important for production, should be standard.

## Comparison to Similar Decisions

### Console Effect

Console is built-in. Nobody questions this because:
- Universally needed
- Simple interface
- Hard to get wrong

SQL is similar but more complex.

### HTTP Effect

HTTP client is built-in in Lux. This was the right call because:
- Most apps need HTTP
- Complex to implement well
- Effect system integration important

SQL follows same reasoning.

### File Effect

File I/O is built-in. Same rationale applies.

## What Other Effect-System Languages Do

| Language | Database | Built-in? |
|----------|----------|-----------|
| **Koka** | No database support | N/A |
| **Eff** | No database support | N/A |
| **Frank** | No database support | N/A |
| **Unison** | Abilities + packages | Both |

Lux is pioneering practical effects. Built-in SQL makes sense.

## Conclusion

SQL should remain a built-in effect in Lux because:

1. It demonstrates the power of effects for real-world use
2. It enables the handler-based testing story
3. It removes friction for most applications
4. It serves as a teaching example for effects

However, the implementation should evolve to:
- Support multiple database backends via handlers
- Make SQLite optional for minimal binaries
- Provide connection pooling
- Add parameterized query support

This hybrid approach gives users the best of both worlds: immediate productivity with built-in SQLite, and flexibility through package-provided handlers for other databases.

---

## Future Work

1. **Parameterized queries** - Critical for SQL injection prevention
2. **Connection pooling** - Required for production servers
3. **PostgreSQL handler** - Most requested database
4. **Migration support** - Schema evolution tooling
5. **Type-safe queries** - Compile-time SQL checking (ambitious)