lux/docs/guide/13-schema-evolution.md

Chapter 13: Schema Evolution

Data structures change over time. Fields get added, removed, or renamed. Types get split or merged. Without careful handling, these changes break systems—old data can't be read, services fail, migrations corrupt data.

Lux's schema evolution system makes these changes safe and automatic.

The Problem

Consider a real scenario:

// Version 1: Simple user
type User {
    name: String
}

// Later, you need email addresses
type User {
    name: String,
    email: String   // Breaking change! Old data doesn't have this.
}

In most languages, this breaks everything. Existing users in your database don't have email addresses. Deserializing old data fails. Services crash.

Lux solves this with versioned types and automatic migrations.

Versioned Types

Add a version annotation to any type:

// Version 1: Original definition
type User @v1 {
    name: String
}

// Version 2: Added email field
type User @v2 {
    name: String,
    email: String,

    // How to migrate from v1
    from @v1 = { name: old.name, email: "unknown@example.com" }
}

// Version 3: Split name into first/last
type User @v3 {
    firstName: String,
    lastName: String,
    email: String,

    // How to migrate from v2
    from @v2 = {
        firstName: String.split(old.name, " ") |> List.head |> Option.getOrElse(""),
        lastName: String.split(old.name, " ") |> List.tail |> List.head |> Option.getOrElse(""),
        email: old.email
    }
}

The @latest alias always refers to the most recent version:

type User @latest {
    firstName: String,
    lastName: String,
    email: String,

    from @v2 = { ... }
}

// These are equivalent:
fn createUser(first: String, last: String, email: String): User@latest = ...
fn createUser(first: String, last: String, email: String): User@v3 = ...

Migration Syntax

Basic Migration

type Config @v2 {
    theme: String,
    fontSize: Int,

    // 'old' refers to the v1 value
    from @v1 = {
        theme: old.theme,
        fontSize: 14  // New field with default
    }
}

Computed Fields

type Order @v2 {
    items: List<Item>,
    total: Int,
    itemCount: Int,  // New computed field

    from @v1 = {
        items: old.items,
        total: old.total,
        itemCount: List.length(old.items)
    }
}

Removing Fields

When removing fields, simply don't include them in the new version:

type Settings @v1 {
    theme: String,
    legacyMode: Bool,  // To be removed
    volume: Int
}

type Settings @v2 {
    theme: String,
    volume: Int,

    // legacyMode is dropped - just don't migrate it
    from @v1 = {
        theme: old.theme,
        volume: old.volume
    }
}

Renaming Fields

type Product @v1 {
    name: String,
    cost: Int  // Old field name
}

type Product @v2 {
    name: String,
    price: Int,  // Renamed from 'cost'

    from @v1 = {
        name: old.name,
        price: old.cost  // Map old field to new name
    }
}

Complex Transformations

type Address @v1 {
    fullAddress: String  // "123 Main St, New York, NY 10001"
}

type Address @v2 {
    street: String,
    city: String,
    state: String,
    zip: String,

    from @v1 = {
        let parts = String.split(old.fullAddress, ", ")
        {
            street: List.get(parts, 0) |> Option.getOrElse(""),
            city: List.get(parts, 1) |> Option.getOrElse(""),
            state: List.get(parts, 2)
                |> Option.map(fn(s: String): String => String.split(s, " ") |> List.head |> Option.getOrElse(""))
                |> Option.getOrElse(""),
            zip: List.get(parts, 2)
                |> Option.map(fn(s: String): String => String.split(s, " ") |> List.last |> Option.getOrElse(""))
                |> Option.getOrElse("")
        }
    }
}

Working with Versioned Values

The Schema module provides runtime operations for versioned values:

Creating Versioned Values

// Create a value tagged with a specific version
let userV1 = Schema.versioned("User", 1, { name: "Alice" })
let userV2 = Schema.versioned("User", 2, { name: "Alice", email: "alice@example.com" })

Checking Versions

let user = Schema.versioned("User", 1, { name: "Alice" })
let version = Schema.getVersion(user)  // Returns 1

// Version-aware logic
if version < 2 then
    Console.print("Legacy user format")
else
    Console.print("Modern user format")

Migrating Values

// Migrate to a specific version
let userV1 = Schema.versioned("User", 1, { name: "Alice" })
let userV2 = Schema.migrate(userV1, 2)  // Uses declared migration

let version = Schema.getVersion(userV2)  // Now 2

// Chain migrations (v1 -> v2 -> v3)
let userV3 = Schema.migrate(userV1, 3)  // Applies v1->v2, then v2->v3

Auto-Generated Migrations

For simple changes, Lux can automatically generate migrations:

type Profile @v1 {
    name: String
}

// Adding a field with a default? Migration is auto-generated
type Profile @v2 {
    name: String,
    bio: String = ""  // Default value provided
}

// The compiler generates this for you:
// from @v1 = { name: old.name, bio: "" }

Auto-migration works for:

• Adding fields with default values
• Keeping existing fields unchanged

You must write explicit migrations for:

• Field renaming
• Field removal (to confirm intent)
• Type changes
• Computed/derived fields

Practical Examples

Example 1: API Response Versioning

type ApiResponse @v1 {
    status: String,
    data: String
}

type ApiResponse @v2 {
    status: String,
    data: String,
    meta: { timestamp: Int, version: String },

    from @v1 = {
        status: old.status,
        data: old.data,
        meta: { timestamp: 0, version: "legacy" }
    }
}

// Version-aware API client
fn handleResponse(raw: ApiResponse@v1): ApiResponse@v2 = {
    Schema.migrate(Schema.versioned("ApiResponse", 1, raw), 2)
}

Example 2: Database Record Evolution

// Original schema
type Customer @v1 {
    name: String,
    address: String
}

// Split address into components
type Customer @v2 {
    name: String,
    street: String,
    city: String,
    country: String,

    from @v1 = {
        let parts = String.split(old.address, ", ")
        {
            name: old.name,
            street: List.get(parts, 0) |> Option.getOrElse(old.address),
            city: List.get(parts, 1) |> Option.getOrElse("Unknown"),
            country: List.get(parts, 2) |> Option.getOrElse("Unknown")
        }
    }
}

// Load and migrate on read
fn loadCustomer(id: String): Customer@v2 with {Database} = {
    let record = Database.query("SELECT * FROM customers WHERE id = ?", [id])
    let version = record.schema_version  // Stored version

    if version == 1 then
        let v1 = Schema.versioned("Customer", 1, {
            name: record.name,
            address: record.address
        })
        Schema.migrate(v1, 2)
    else
        { name: record.name, street: record.street, city: record.city, country: record.country }
}

Example 3: Configuration Files

type AppConfig @v1 {
    debug: Bool,
    port: Int
}

type AppConfig @v2 {
    debug: Bool,
    port: Int,
    logLevel: String,  // New in v2

    from @v1 = {
        debug: old.debug,
        port: old.port,
        logLevel: if old.debug then "debug" else "info"
    }
}

type AppConfig @v3 {
    environment: String,  // Replaces debug flag
    port: Int,
    logLevel: String,

    from @v2 = {
        environment: if old.debug then "development" else "production",
        port: old.port,
        logLevel: old.logLevel
    }
}

// Load config with automatic migration
fn loadConfig(path: String): AppConfig@v3 with {File} = {
    let json = File.read(path)
    let parsed = Json.parse(json)
    let version = Json.getInt(parsed, "version") |> Option.getOrElse(1)

    match version {
        1 => {
            let v1 = Schema.versioned("AppConfig", 1, {
                debug: Json.getBool(parsed, "debug") |> Option.getOrElse(false),
                port: Json.getInt(parsed, "port") |> Option.getOrElse(8080)
            })
            Schema.migrate(v1, 3)
        },
        2 => {
            let v2 = Schema.versioned("AppConfig", 2, {
                debug: Json.getBool(parsed, "debug") |> Option.getOrElse(false),
                port: Json.getInt(parsed, "port") |> Option.getOrElse(8080),
                logLevel: Json.getString(parsed, "logLevel") |> Option.getOrElse("info")
            })
            Schema.migrate(v2, 3)
        },
        _ => {
            // Already v3
            {
                environment: Json.getString(parsed, "environment") |> Option.getOrElse("production"),
                port: Json.getInt(parsed, "port") |> Option.getOrElse(8080),
                logLevel: Json.getString(parsed, "logLevel") |> Option.getOrElse("info")
            }
        }
    }
}

Example 4: Event Sourcing

// Event types evolve over time
type UserCreated @v1 {
    userId: String,
    name: String,
    timestamp: Int
}

type UserCreated @v2 {
    userId: String,
    name: String,
    email: String,
    createdAt: Int,  // Renamed from timestamp

    from @v1 = {
        userId: old.userId,
        name: old.name,
        email: "",  // Not captured in v1
        createdAt: old.timestamp
    }
}

// Process events regardless of version
fn processEvent(event: UserCreated@v1 | UserCreated@v2): Unit with {Console} = {
    let normalized = Schema.migrate(event, 2)  // Always work with v2
    Console.print("User created: " + normalized.name + " at " + toString(normalized.createdAt))
}

Compile-Time Safety

The compiler catches schema evolution errors:

type User @v2 {
    name: String,
    email: String

    // ERROR: Migration references non-existent field
    from @v1 = { name: old.username, email: old.email }
    //                    ^^^^^^^^ 'username' does not exist in User@v1
}

type User @v2 {
    name: String,
    email: String

    // ERROR: Migration missing required field
    from @v1 = { name: old.name }
    //                           ^ Missing 'email' field
}

type User @v2 {
    name: String,
    age: Int

    // ERROR: Type mismatch in migration
    from @v1 = { name: old.name, age: old.birthYear }
    //                                ^^^^^^^^^^^^^ Expected Int, found String
}

Compatibility Checking

Lux tracks compatibility between versions:

Change Type	Backward Compatible	Forward Compatible
Add optional field (with default)	Yes	Yes
Add required field	No	Yes (with migration)
Remove field	Yes (with migration)	No
Rename field	No	No (need migration)
Change field type	No	No (need migration)

The compiler warns about breaking changes:

type User @v1 {
    name: String,
    email: String
}

type User @v2 {
    name: String
    // Warning: Removing 'email' is a breaking change
    // Existing v2 consumers expect this field
}

Best Practices

1. Always Version Production Types

// Good: Versioned from the start
type Order @v1 {
    id: String,
    items: List<Item>,
    total: Int
}

// Bad: Unversioned type is hard to evolve
type Order {
    id: String,
    items: List<Item>,
    total: Int
}

2. Keep Migrations Simple

// Good: Simple, direct mapping
from @v1 = {
    name: old.name,
    email: old.email |> Option.getOrElse("")
}

// Avoid: Complex logic in migrations
from @v1 = {
    name: old.name,
    email: {
        // Don't put complex business logic here
        let domain = inferDomainFromName(old.name)
        let local = String.toLower(String.replace(old.name, " ", "."))
        local + "@" + domain
    }
}

3. Test Migrations

fn testUserMigration(): Unit with {Test} = {
    let v1User = Schema.versioned("User", 1, { name: "Alice" })
    let v2User = Schema.migrate(v1User, 2)

    Test.assertEqual(v2User.name, "Alice")
    Test.assertEqual(v2User.email, "unknown@example.com")
}

4. Document Breaking Changes

type User @v3 {
    // BREAKING: 'name' split into firstName/lastName
    // Migration: name.split(" ")[0] -> firstName, name.split(" ")[1] -> lastName
    firstName: String,
    lastName: String,
    email: String,

    from @v2 = { ... }
}

Schema Module Reference

Function	Description
Schema.versioned(typeName, version, value)	Create a versioned value
Schema.getVersion(value)	Get the version of a value
Schema.migrate(value, targetVersion)	Migrate to a target version
Schema.isCompatible(v1, v2)	Check if versions are compatible

Summary

Schema evolution in Lux provides:

• Versioned types with @v1, @v2, @latest annotations
• Explicit migrations with from @vN = { ... } syntax
• Automatic migrations for simple field additions with defaults
• Runtime operations via the Schema module
• Compile-time safety catching migration errors early
• Migration chaining for multi-step upgrades

This system ensures your data can evolve safely over time, without breaking existing code or losing information.

What's Next?

• Tutorials - Build real projects
• Standard Library Reference - Complete API docs