lux/PACKAGES.md

# Lux Package Ecosystem Plan

## Current State

### Stdlib (built-in)
| Module | Coverage |
|--------|----------|
| String | Comprehensive (split, join, trim, indexOf, replace, etc.) |
| List | Good (map, filter, fold, head, tail, concat, range, find, any, all, take, drop) |
| Option | Basic (map, flatMap, getOrElse, isSome, isNone) |
| Result | Basic (map, flatMap, getOrElse, isOk, isErr) |
| Math | Basic (abs, min, max, sqrt, pow, floor, ceil, round) |
| Json | Comprehensive (parse, stringify, get, typed extractors, constructors) |
| File | Good (read, write, append, exists, delete, readDir, isDir, mkdir) |
| Console | Good (print, read, readLine, readInt) |
| Process | Good (exec, execStatus, env, args, exit, cwd) |
| Http | Basic (get, post, put, delete, setHeader) |
| HttpServer | Basic (listen, accept, respond) |
| Time | Minimal (now, sleep) |
| Random | Basic (int, float, bool) |
| Sql | Good (SQLite: open, query, execute, transactions) |
| Postgres | Good (connect, query, execute, transactions) |
| Schema | Niche (versioned data migration) |
| Test | Good (assert, assertEqual, assertTrue) |
| Concurrent | Experimental (spawn, await, yield, cancel) |
| Channel | Experimental (create, send, receive) |

### Registry (pkgs.lux) - 3 packages
| Package | Version | Notes |
|---------|---------|-------|
| json | 1.0.0 | Wraps stdlib Json with convenience functions (getPath, getString, etc.) |
| http-client | 0.1.0 | Wraps stdlib Http with JSON helpers, URL encoding |
| testing | 0.1.0 | Wraps stdlib Test with describe/it structure |

---

## Gap Analysis

### What's Missing vs Other Languages

Compared to ecosystems like Rust/cargo, Go, Python, Elm, Gleam:

| Category | Gap | Impact | Notes |
|----------|-----|--------|-------|
| **Collections** | No HashMap, Set, Queue, Stack | Critical | List-of-pairs with O(n) lookup is the only option |
| **Sorting** | No List.sort or List.sortBy | High | Must implement insertion sort manually |
| **Date/Time** | Only `Time.now()` (epoch ms), no parsing/formatting | High | blu-site does string-based date formatting manually |
| **Markdown** | No markdown parser | High | blu-site has 300+ lines of hand-rolled markdown |
| **XML/RSS** | No XML generation | High | Can't generate RSS feeds or sitemaps |
| **Regex** | No pattern matching on strings | High | Character-by-character scanning required |
| **Path** | No file path utilities | Medium | basename/dirname manually reimplemented |
| **YAML/TOML** | No config file parsing (beyond JSON) | Medium | Frontmatter parsing is manual |
| **Template** | No string templating | Medium | HTML built via raw string concatenation |
| **URL** | No URL parsing/encoding | Medium | http-client has basic urlEncode but no parser |
| **Crypto** | No hashing (SHA256, etc.) | Medium | Can't do checksums, content hashing |
| **Base64** | No encoding/decoding | Low | Needed for data URIs, some auth |
| **CSV** | No CSV parsing | Low | Common data format |
| **UUID** | No UUID generation | Low | Useful for IDs |
| **Logging** | No structured logging | Low | Just Console.print |
| **CLI** | No argument parsing library | Low | Manual arg handling |

### What Should Be Stdlib vs Package

**Should be stdlib additions** (too fundamental to be packages):
- HashMap / Map type (requires runtime support)
- List.sort / List.sortBy (fundamental operation)
- Better Time module (date parsing, formatting)
- Regex (needs runtime/C support for performance)
- Path module (cross-platform file path handling)

**Should be packages** (application-level, opinionated, composable):
- markdown
- xml
- rss/atom
- frontmatter
- template
- csv
- crypto
- ssg (static site generator framework)

---

## Priority Package Plans

Ordered by what unblocks blu-site fixes first, then general ecosystem value.

---

### Package 1: `markdown` (Priority: HIGHEST)

**Why:** The 300-line markdown parser in blu-site's main.lux is general-purpose code that belongs in a reusable package. It's also the most complex part of blu-site and has known bugs (e.g., `### ` inside list items renders literally).

**Scope:**
```
markdown/
  lux.toml
  lib.lux          # Public API: parse, parseInline
  src/
    inline.lux     # Inline parsing (bold, italic, links, images, code)
    block.lux      # Block parsing (headings, lists, code blocks, blockquotes, hr)
    types.lux      # AST types (optional - could emit HTML directly)
```

**Public API:**
```lux
// Convert markdown string to HTML string
pub fn toHtml(markdown: String): String

// Convert inline markdown only (no blocks)
pub fn inlineToHtml(text: String): String

// Escape HTML entities
pub fn escapeHtml(s: String): String
```

**Improvements over current blu-site code:**
- Fix heading-inside-list-item rendering (`- ### Title` should work)
- Support nested lists (currently flat only)
- Support reference-style links `[text][ref]`
- Handle edge cases (empty lines in code blocks, nested blockquotes)
- Proper HTML entity escaping in more contexts

**Depends on:** Nothing (pure string processing)

**Estimated size:** ~400-500 lines of Lux

---

### Package 2: `xml` (Priority: HIGH)

**Why:** Needed for RSS/Atom feed generation, sitemap.xml, and robots.txt generation. General-purpose XML builder that doesn't try to parse XML (which would need regex), just emits it.

**Scope:**
```
xml/
  lux.toml
  lib.lux          # Public API: element, document, serialize
```

**Public API:**
```lux
type XmlNode =
    | Element(String, List<XmlAttr>, List<XmlNode>)
    | Text(String)
    | CData(String)
    | Comment(String)
    | Declaration(String, String)  // version, encoding

type XmlAttr =
    | Attr(String, String)

// Build an XML element
pub fn element(tag: String, attrs: List<XmlAttr>, children: List<XmlNode>): XmlNode

// Build a text node (auto-escapes)
pub fn text(content: String): XmlNode

// Build a CDATA section
pub fn cdata(content: String): XmlNode

// Serialize XML tree to string
pub fn serialize(node: XmlNode): String

// Serialize with XML declaration header
pub fn document(version: String, encoding: String, root: XmlNode): String

// Convenience: self-closing element
pub fn selfClosing(tag: String, attrs: List<XmlAttr>): XmlNode
```

**Depends on:** Nothing

**Estimated size:** ~150-200 lines

---

### Package 3: `rss` (Priority: HIGH)

**Why:** Directly needed for blu-site's #6 priority fix (add RSS feed). Builds on `xml` package.

**Scope:**
```
rss/
  lux.toml         # depends on xml
  lib.lux          # Public API: feed, item, toXml, toAtom
```

**Public API:**
```lux
type FeedInfo =
    | FeedInfo(String, String, String, String, String)
    // title, link, description, language, lastBuildDate

type FeedItem =
    | FeedItem(String, String, String, String, String, String)
    // title, link, description, pubDate, guid, categories (comma-separated)

// Generate RSS 2.0 XML string
pub fn toRss(info: FeedInfo, items: List<FeedItem>): String

// Generate Atom 1.0 XML string
pub fn toAtom(info: FeedInfo, items: List<FeedItem>): String
```

**Depends on:** `xml`

**Estimated size:** ~100-150 lines

---

### Package 4: `frontmatter` (Priority: HIGH)

**Why:** blu-site has ~50 lines of fragile frontmatter parsing. This is a common need for any content-driven Lux project. The current parser uses `String.indexOf(line, ": ")` which breaks on values containing `: `.

**Scope:**
```
frontmatter/
  lux.toml
  lib.lux          # Public API: parse
```

**Public API:**
```lux
type FrontmatterResult =
    | FrontmatterResult(List<(String, String)>, String)
    // key-value pairs, remaining body

// Parse frontmatter from a string (--- delimited YAML-like header)
pub fn parse(content: String): FrontmatterResult

// Get a value by key from parsed frontmatter
pub fn get(pairs: List<(String, String)>, key: String): Option<String>

// Get a value or default
pub fn getOrDefault(pairs: List<(String, String)>, key: String, default: String): String

// Parse a space-separated tag string into a list
pub fn parseTags(tagString: String): List<String>
```

**Improvements over current blu-site code:**
- Handle values with `: ` in them (only split on first `: `)
- Handle multi-line values (indented continuation)
- Handle quoted values with embedded newlines
- Strip quotes from values consistently

**Depends on:** Nothing

**Estimated size:** ~100-150 lines

---

### Package 5: `path` (Priority: MEDIUM)

**Why:** blu-site manually implements `basename` and `dirname`. Any file-processing Lux program needs these. Tiny but universally useful.

**Scope:**
```
path/
  lux.toml
  lib.lux
```

**Public API:**
```lux
// Get filename from path: "/foo/bar.txt" -> "bar.txt"
pub fn basename(p: String): String

// Get directory from path: "/foo/bar.txt" -> "/foo"
pub fn dirname(p: String): String

// Get file extension: "file.txt" -> "txt", "file" -> ""
pub fn extension(p: String): String

// Remove file extension: "file.txt" -> "file"
pub fn stem(p: String): String

// Join path segments: join("foo", "bar") -> "foo/bar"
pub fn join(a: String, b: String): String

// Normalize path: "foo//bar/../baz" -> "foo/baz"
pub fn normalize(p: String): String

// Check if path is absolute
pub fn isAbsolute(p: String): Bool
```

**Depends on:** Nothing

**Estimated size:** ~80-120 lines

---

### Package 6: `sitemap` (Priority: MEDIUM)

**Why:** Directly needed for blu-site's #9 priority fix. Simple package that generates sitemap.xml.

**Scope:**
```
sitemap/
  lux.toml         # depends on xml
  lib.lux
```

**Public API:**
```lux
type SitemapEntry =
    | SitemapEntry(String, String, String, String)
    // url, lastmod (ISO date), changefreq, priority

// Generate sitemap.xml string
pub fn generate(entries: List<SitemapEntry>): String

// Generate a simple robots.txt pointing to the sitemap
pub fn robotsTxt(sitemapUrl: String): String
```

**Depends on:** `xml`

**Estimated size:** ~50-70 lines

---

### Package 7: `ssg` (Priority: LOW - future)

**Why:** Once markdown, frontmatter, rss, sitemap, and path packages exist, the remaining logic in blu-site's main.lux is generic SSG framework code: read content dirs, parse posts, sort by date, generate section indexes, generate tag pages, copy static assets. This could be extracted into a framework package that other Lux users could use to build their own static sites.

**This should wait** until the foundation packages above are stable and battle-tested through blu-site usage.

---

## Non-Package Stdlib Improvements Needed

These gaps are too fundamental to be packages and should be added to the Lux language itself:

### HashMap (Critical)
Every package above that needs key-value lookups (frontmatter, xml attributes, etc.) is working around the lack of HashMap with `List<(String, String)>`. This is O(n) per lookup and makes code verbose. A stdlib `Map` module would transform the ecosystem.

### List.sort / List.sortBy (High)
blu-site implements insertion sort manually. Every content-driven app needs sorting. This should be a stdlib function.

### Time.format / Time.parse (High)
blu-site manually parses "2025-01-15" by substring extraction and maps month numbers to names. A proper date/time library (even just ISO 8601 parsing and basic formatting) would help every package above.

---

## Implementation Order

```
Phase 1 (unblock blu-site fixes):
  1. markdown     - extract from blu-site, fix bugs, publish
  2. frontmatter  - extract from blu-site, improve robustness
  3. path         - tiny, universally useful
  4. xml          - needed by rss and sitemap

Phase 2 (complete blu-site features):
  5. rss          - depends on xml
  6. sitemap      - depends on xml

Phase 3 (ecosystem growth):
  7. template     - string templating (mustache-like)
  8. csv          - data processing
  9. cli          - argument parsing
  10. ssg         - framework extraction from blu-site
```

Each package should be developed in its own directory under `~/src/`, published to the git.qrty.ink registry, and tested by integrating it into blu-site.