Initial commit: Ultimate Notetaking, Sync & Backup System

A NixOS-based system for managing personal data across three tiers:
- Tier 1: Configuration (shareable via git)
- Tier 2: Syncable data (nb + Syncthing)
- Tier 3: Large data (self-hosted services + backup)

Includes:
- NixOS modules for nb, Syncthing, backup (restic)
- Server modules for Forgejo, Immich, Jellyfin
- Helper scripts (usync, ustatus)
- Comprehensive documentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
2026-02-13 01:44:00 -05:00
commit b40ac99524
17 changed files with 3151 additions and 0 deletions

267
docs/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,267 @@
# System Architecture
## Design Principles
### 1. Declarative Everything
Every aspect of this system is declared in Nix. No imperative setup steps, no hidden state.
```
Desired State (Nix) → nixos-rebuild → Actual State
```
### 2. Separation of Concerns
```
┌──────────────────────────────────────────────────────────────┐
│ CONFIGURATION LAYER │
│ What: How machines should be configured │
│ Where: This repo (flake.nix, modules/) │
│ Managed by: Git + GitHub │
│ Shareable: Yes (public) │
└──────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────┐
│ DATA LAYER │
│ │
│ ┌────────────────────┐ ┌────────────────────┐ │
│ │ TIER 2: Sync │ │ TIER 3: Backup │ │
│ │ │ │ │ │
│ │ ~/notes/ (nb) │ │ Photos (Immich) │ │
│ │ ~/Documents/ │ │ Media (Jellyfin) │ │
│ │ │ │ Repos (Forgejo) │ │
│ │ Managed by: │ │ │ │
│ │ - nb (git sync) │ │ Managed by: │ │
│ │ - Syncthing │ │ - Server apps │ │
│ │ │ │ - restic backup │ │
│ │ Multi-device: Yes │ │ Multi-device: No │ │
│ │ Real-time: Yes │ │ (on-demand access)│ │
│ └────────────────────┘ └────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
```
### 3. No Vendor Lock-in
Every component can be replaced:
| Component | Could be replaced with |
|-----------|----------------------|
| nb | Obsidian + git, org-mode, plain vim + git |
| Syncthing | Unison, rclone bisync |
| Forgejo | Gitea, GitLab, bare git |
| Immich | PhotoPrism, any DAM |
| Jellyfin | Plex (non-FOSS), Kodi |
| restic | borg, duplicity |
### 4. Offline-First
All Tier 2 data is fully available offline:
- nb notebooks are local git repos
- Syncthing folders are local directories
- Sync happens when connectivity allows
Tier 3 data is available via caching or on-demand download.
## Data Flow
### Tier 1: Configuration
```
┌─────────┐ git push ┌─────────┐
│ Local │ ───────────────► │ GitHub │
│ Repo │ ◄─────────────── │ (pub) │
└────┬────┘ git pull └─────────┘
│ nixos-rebuild switch
┌─────────┐
│ Machine │
│ State │
└─────────┘
```
### Tier 2: Notes (nb)
```
┌─────────┐ ┌─────────┐
│ Device │ nb sync │ Forgejo │
│ A │ ◄────────────────► │ (priv) │
└─────────┘ └────┬────┘
┌─────────┐ nb sync ┌────▼────┐
│ Device │ ◄────────────────► │ Forgejo │
│ B │ │ (priv) │
└─────────┘ └─────────┘
Conflict Resolution: Git 3-way merge
History: Full git history preserved
```
### Tier 2: Documents (Syncthing)
```
┌─────────┐ ┌─────────┐
│ Device │ ◄──── P2P ───────► │ Device │
│ A │ │ B │
└────┬────┘ └────┬────┘
│ │
│ ┌─────────┐ │
└────────►│ Device │◄─────────┘
│ C │
└─────────┘
Conflict Resolution: .sync-conflict files (manual)
History: Syncthing versioning (configurable)
```
### Tier 3: Large Data
```
┌─────────┐ upload ┌─────────┐
│ Device │ ──────────────► │ Server │
│ │ ◄────────────── │ Immich/ │
└─────────┘ on-demand │ Jellyfin│
└────┬────┘
│ restic
┌────▼────┐
│ Backup │
│ B2/NAS │
└─────────┘
```
## Module Dependency Graph
```
flake.nix
┌───────────────┼───────────────┐
▼ ▼ ▼
hosts/ modules/ home/
desktop (shared) default.nix
laptop │
server │
│ │
▼ ▼
┌──────────────────────────────────┐
│ modules/ │
│ ┌──────┐ ┌───────────┐ ┌─────┐ │
│ │ nb │ │ syncthing │ │nvim │ │
│ └──────┘ └───────────┘ └─────┘ │
│ ┌────────┐ │
│ │ backup │ │
│ └────────┘ │
│ │
│ ┌─────────── server/ ────────┐ │
│ │ forgejo immich jellyfin │ │
│ └───────────────────────────┘ │
└──────────────────────────────────┘
```
## Security Model
### Secrets
```
┌─────────────────────────────────────────┐
│ sops-nix │
│ │
│ secrets.yaml (encrypted in repo) │
│ │ │
│ │ age/gpg key (not in repo) │
│ ▼ │
│ Decrypted at activation time │
│ Placed in /run/secrets/ │
│ Permissions set per-secret │
└─────────────────────────────────────────┘
```
### Network Security
- All sync over TLS (Syncthing) or SSH (nb, Forgejo)
- Server services behind reverse proxy (Caddy/nginx)
- No ports exposed except 443 (HTTPS)
- Tailscale/WireGuard for private network (optional)
## Backup Strategy
```
┌────────────────────────────────────────────────────────────┐
│ 3-2-1 Backup Rule │
│ │
│ 3 copies: │
│ ├── Primary: Local device │
│ ├── Secondary: Home server │
│ └── Tertiary: Offsite (B2/cloud) │
│ │
│ 2 media types: │
│ ├── SSD (local) │
│ └── Cloud object storage (offsite) │
│ │
│ 1 offsite: │
│ └── Backblaze B2 / AWS S3 / etc. │
└────────────────────────────────────────────────────────────┘
Backup Schedule (systemd timers):
├── Tier 2 (notes/docs): Daily, keep 30 days
├── Tier 3 (media): Weekly, keep 90 days
└── Server state: Daily, keep 14 days
```
## Scaling Considerations
### Adding a New Machine
1. Create `hosts/<hostname>/configuration.nix`
2. Add hardware-configuration.nix
3. Add to flake.nix outputs
4. Run `nixos-rebuild switch --flake .#<hostname>`
5. nb automatically syncs when `nb sync` is run
6. Syncthing auto-discovers via device IDs in config
### Adding a New User
1. Add user to `hosts/<hostname>/configuration.nix`
2. Create home-manager config
3. Set up nb notebooks for user
4. Add Syncthing device ID
### Multiple Servers
The architecture supports multiple servers:
```
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Server │ │ Server │ │ Server │
│ Home │ │ VPS │ │ NAS │
│ (media) │ │ (Forgejo│ │(backup) │
└─────────┘ └─────────┘ └─────────┘
```
Each server has its own host config in `hosts/`.
## Failure Modes
| Failure | Impact | Recovery |
|---------|--------|----------|
| Device lost | Tier 2 data unavailable locally | Sync from other devices or Forgejo |
| Server down | Tier 3 unavailable, nb sync fails | Use local cache, fix server |
| Backup failure | No new backups | Alert (systemd OnFailure), investigate |
| Sync conflict | Manual resolution needed | Resolve .sync-conflict or git merge |
| Internet down | Can't sync | Work offline, sync when back |
## Future Considerations
### Potential Additions
1. **Encrypted Tier 2**: Full-disk encryption + nb's GPG encryption
2. **Mobile support**: Termux + Unison, or Syncthing-Fork
3. **Real-time collaboration**: Automerge/CRDT-based notes
4. **Monitoring**: Prometheus + Grafana for sync/backup status
### Not In Scope
1. Windows support (NixOS only)
2. Proprietary services integration
3. Enterprise/team features
4. Real-time streaming replication

240
docs/LLM-CONTEXT.md Normal file
View File

@@ -0,0 +1,240 @@
# LLM Context Document
This document provides context for AI assistants (Claude, GPT, Copilot, etc.) working with this codebase.
## Project Overview
**Name**: Ultimate Notetaking, Sync & Backup System
**Purpose**: NixOS-based declarative system for managing personal data across multiple machines
**Primary User**: Linux users, especially NixOS, who want full control over their data
## Core Concepts
### The Three-Tier Data Model
```
TIER 1: Configuration (Shareable)
├── What: System config, dotfiles, flake.nix
├── Where: This git repo (public on GitHub)
├── How: NixOS + home-manager + sops-nix
└── Philosophy: Anyone can clone and use this config
TIER 2: Syncable Data (Private, Multi-device)
├── What: Notes, documents, writings
├── Where: ~/notes/ (nb), ~/Documents/ (Syncthing)
├── How: nb (git-backed) for text, Syncthing for binary
└── Philosophy: Edit anywhere, sync automatically
TIER 3: Large Data (Backed up, On-demand)
├── What: Photos, videos, large repos
├── Where: Self-hosted servers (Immich, Jellyfin, Forgejo)
├── How: Upload to server, access on-demand, restic backup
└── Philosophy: Don't sync everything; backup everything
```
### Key Tools
| Tool | Purpose | Tier |
|------|---------|------|
| NixOS | Declarative system configuration | 1 |
| home-manager | Declarative user configuration | 1 |
| sops-nix | Encrypted secrets in git | 1 |
| nb | CLI notebook with git sync | 2 |
| Syncthing | P2P file synchronization | 2 |
| Neovim | Primary editor | 1, 2 |
| Forgejo | Self-hosted git (GitHub alternative) | 3 |
| Immich | Self-hosted photos (Google Photos alternative) | 3 |
| Jellyfin | Self-hosted media (Plex alternative) | 3 |
| restic | Encrypted, deduplicated backups | 3 |
| rclone | Cloud storage Swiss Army knife | 3 |
## File Structure
```
flake.nix # Nix flake entry point
├── inputs: nixpkgs, home-manager, sops-nix
├── outputs: nixosConfigurations, homeConfigurations, devShells
└── imports modules for each host
modules/
├── nb.nix # nb installation and config
├── syncthing.nix # Declarative Syncthing
├── neovim.nix # Neovim with nb integration
├── backup.nix # restic systemd timers
└── server/ # Server-only modules
├── forgejo.nix
├── immich.nix
└── jellyfin.nix
hosts/
├── desktop/ # Desktop-specific config
├── laptop/ # Laptop-specific config
└── server/ # Server-specific config
home/
└── default.nix # home-manager user config
scripts/
├── usync # Unified sync (nb + Syncthing)
├── ubackup # Manual backup trigger
└── ustatus # Status dashboard
```
## Common Tasks
### User wants to add a new notebook
1. Edit `modules/nb.nix`
2. Add notebook to `programs.nb.notebooks`
3. Run `nixos-rebuild switch`
4. Run `nb notebooks` to verify
### User wants to add a Syncthing folder
1. Edit `modules/syncthing.nix`
2. Add folder to `services.syncthing.settings.folders`
3. Add device IDs if new devices
4. Run `nixos-rebuild switch`
### User wants to change backup paths
1. Edit `modules/backup.nix`
2. Modify `services.restic.backups.default.paths`
3. Run `nixos-rebuild switch`
### User wants to add a new host
1. Create `hosts/<hostname>/configuration.nix`
2. Add to `flake.nix` outputs
3. Configure host-specific settings
4. Run `nixos-rebuild switch --flake .#<hostname>`
## Code Patterns
### Module Structure
All modules follow this pattern:
```nix
{ config, lib, pkgs, ... }:
with lib;
let
cfg = config.services.ourService;
in {
options.services.ourService = {
enable = mkEnableOption "our service";
someOption = mkOption {
type = types.str;
default = "default-value";
description = "Description of the option";
};
};
config = mkIf cfg.enable {
# Implementation
};
}
```
### Home-manager Integration
User-level config uses home-manager:
```nix
{ config, pkgs, ... }:
{
programs.git = {
enable = true;
userName = "User Name";
userEmail = "user@example.com";
};
}
```
### Secrets Management
Secrets use sops-nix:
```nix
{
sops.secrets.syncthing-key = {
sopsFile = ./secrets/secrets.yaml;
owner = "user";
};
}
```
## Constraints
1. **Open source only**: No proprietary tools
2. **NixOS-centric**: Must work with `nixos-rebuild`
3. **Flakes required**: All Nix code uses flakes
4. **No cloud dependencies**: Must work fully self-hosted (cloud optional)
5. **CLI-first**: GUI is optional, never required
6. **Markdown for notes**: Plain text, portable, version-control friendly
## When Helping Users
### DO:
- Suggest declarative (Nix) solutions
- Recommend nb for text/notes
- Recommend Syncthing for binary files
- Use systemd for services
- Use home-manager for user config
- Keep modules composable
### DON'T:
- Suggest proprietary alternatives
- Hardcode paths (use options)
- Mix system and user config inappropriately
- Ignore the three-tier model
- Suggest Syncthing for notes (use nb instead)
## Debugging
### Common Issues
| Symptom | Likely Cause | Solution |
|---------|--------------|----------|
| nb sync fails | SSH key not available | Check `ssh-add -l` |
| Syncthing not starting | Declarative config conflict | Check for GUI-made changes |
| Backup fails | Repository not initialized | Run `restic init` |
| Module not found | Not imported in flake.nix | Add to imports |
### Useful Commands
```bash
# Check Nix config
nix flake check
# Show Syncthing status
syncthing cli show system
# nb sync status
nb sync status
# restic backup status
restic -r <repo> snapshots
```
## Extension Points
Users commonly want to extend:
1. **Additional notebooks**: Add to `modules/nb.nix`
2. **New Syncthing folders**: Add to `modules/syncthing.nix`
3. **Backup exclusions**: Modify `modules/backup.nix`
4. **Server services**: Add new modules under `modules/server/`
5. **Editor config**: Modify `modules/neovim.nix`
## Related Documentation
- [Architecture](./ARCHITECTURE.md) - Detailed system design
- [Sync Tools Comparison](./research/sync-tools-comparison.md) - Why we chose these tools
- [nb documentation](https://xwmx.github.io/nb/)
- [NixOS manual](https://nixos.org/manual/nixos/stable/)
- [home-manager manual](https://nix-community.github.io/home-manager/)

View File

@@ -0,0 +1,271 @@
# File Synchronization Tools: Comprehensive Comparison
*Research conducted: February 2026*
## Executive Summary
After extensive research, **Syncthing is not buggy—but it is complex and unforgiving**. Most reported "bugs" are actually:
1. Misunderstanding of how P2P sync works
2. Conflict handling that differs from user expectations
3. Android-specific issues (now largely resolved with forks)
4. Users treating it as a backup tool (it's not)
For a NixOS-based notes/data sync system, the recommended approach is:
- **Tier 2 (Notes)**: `nb` with git sync to private Forgejo (explicit, versioned)
- **Tier 2 (Documents)**: Syncthing OR Unison (based on preference)
- **Tier 3 (Large files)**: rclone + restic to self-hosted or cloud storage
---
## Tool-by-Tool Analysis
### Syncthing
**What it is**: Continuous, real-time, P2P file synchronization
**Strengths**:
- Truly decentralized (no server required)
- Real-time sync via filesystem watching
- Handles file renames/moves efficiently
- Cross-platform with good GUI
- Battle-tested with large user base
- Declarative NixOS module available
**Weaknesses**:
- **Conflict handling is file-level**: Creates `.sync-conflict-*` files that must be manually resolved
- **Not a backup**: Deletions propagate immediately; if you delete on one device, it's gone everywhere
- **Android app discontinued** (Dec 2024): Use [Syncthing-Fork](https://github.com/catfriend1/syncthing-android) instead
- **Complexity ceiling**: Multi-device topologies can create unexpected sync loops
- **No ownership/permission sync**: By design, to avoid requiring root
- **OOM issues in v2.0**: On low-memory devices (1GB RAM) with large file sets
**Common Misconceptions**:
| Issue Reported | Actual Cause |
|----------------|--------------|
| "Files disappearing" | Deletions syncing correctly; user expected backup behavior |
| "Sync stuck at 99%" | Large files still transferring, or ignored files being counted |
| "Random conflicts" | Multiple devices modifying same file; Syncthing is doing its job |
| "Ignoring read-only" | Misconfigured folder type; "Receive Only" ≠ read-only filesystem |
**Verdict**: Syncthing works well if you understand that it's **sync, not backup**. For notes that change frequently on multiple devices, conflicts will happen.
**Sources**:
- [Syncthing GitHub Issues](https://github.com/syncthing/syncthing/issues)
- [Syncthing 2.0 OOM Discussion](https://forum.syncthing.net/t/syncthing-2-0-1-oom/24824)
- [Lobsters: "Do not use Syncthing"](https://lobste.rs/s/pacmpc/do_not_use_syncthing)
- [Syncthing Android Discontinuation](https://www.ghacks.net/2024/10/21/syncthing-for-android-is-being-discontinued-but-theres-an-alternative-app-you-can-switch-to/)
---
### Unison
**What it is**: Bidirectional file synchronizer, runs on-demand (not continuous)
**Strengths**:
- **Explicit sync**: You run it when you want, see exactly what will happen
- **Proper conflict detection**: Shows both versions, lets you choose
- **SSH-native**: Works over SSH, no extra protocols
- **Version 2.52+ is OCaml-independent**: No more version matching hell
- **ACL/xattr support** (v2.53+): Can sync permissions and extended attributes
- **Hub-and-spoke works great**: Sync all devices to a NAS/server
- **Lower battery usage** than continuous sync
**Weaknesses**:
- **Manual invocation**: No background daemon (by design, but requires discipline)
- **File rename detection is poor**: Rename = delete + create, potentially losing history
- **GUI is deprecated**: lablgtk maintenance uncertain; CLI is the future
- **Small maintainer base**: ~2.5 people, 0.1 FTE
- **Learning curve**: Profile configuration can be confusing
**Best for**: Users who want explicit control over sync timing and conflict resolution
**Sources**:
- [Unison GitHub](https://github.com/bcpierce00/unison)
- [Migration from Syncthing to Unison](https://codelearn.me/2025/12/29/from-syncthing-to-unison.html)
- [Syncthing Forum Comparison](https://forum.syncthing.net/t/syncthing-vs-rsync-vs-unison/6298)
---
### rsync
**What it is**: One-way file copying/mirroring tool
**Strengths**:
- Rock solid, ubiquitous
- Efficient delta transfers
- Perfect for backups
**Weaknesses**:
- **Unidirectional only**: Not a sync tool
- No conflict handling (overwrites target)
**Best for**: Backups, deployments, one-way mirroring. Not for bidirectional sync.
---
### rclone (with bisync)
**What it is**: Cloud storage Swiss Army knife, with experimental bidirectional sync
**Strengths**:
- Supports 70+ cloud backends
- `bisync` command provides bidirectional sync (as of v1.66)
- Checksum-based comparison available
- Lock files prevent concurrent corruption
- Great for cloud ↔ local sync
**Weaknesses**:
- **bisync is "advanced"**: Documentation warns of potential data loss
- Designed for cloud storage, not LAN sync
- More complex ignore pattern syntax than Syncthing
- No real-time watching (cron-based)
**Best for**: Syncing with cloud storage (S3, B2, Drive, etc.), Tier 3 backup
**Sources**:
- [rclone bisync documentation](https://rclone.org/bisync/)
---
### Mutagen
**What it is**: Fast bidirectional sync for remote development
**Strengths**:
- Designed for code sync to remote containers/servers
- Very fast (near-native Docker volume performance)
- Real-time filesystem watching
- Multiple sync modes (two-way-safe, one-way, etc.)
- SSH and Docker native
**Weaknesses**:
- **Ephemeral by design**: Not meant for long-term sync
- Less configuration flexibility than Syncthing
- Primarily for developer workflows
**Best for**: Remote development, Docker volumes, SSH workspaces. Overkill for notes.
**Sources**:
- [Mutagen GitHub](https://github.com/mutagen-io/mutagen)
- [Mutagen Documentation](https://mutagen.io/documentation/synchronization/)
---
### nb (Notebook CLI)
**What it is**: CLI note-taking with built-in git sync
**Strengths**:
- **Git-native**: Every notebook is a git repo
- Explicit sync via `nb sync`
- Full history, branching, proper merges
- Works with any git remote (GitHub, Gitea, Forgejo)
- Markdown by default, encryption optional
- Single portable shell script
- `$EDITOR` integration (neovim-friendly)
**Weaknesses**:
- **Explicit sync only**: Must run `nb sync` (or set up cron/systemd)
- Git conflicts require git knowledge to resolve
- No GUI (by design)
**Best for**: Notes and writing. This is the ideal Tier 2 solution for text content.
**Sources**:
- [nb GitHub](https://github.com/xwmx/nb)
- [nb Documentation](https://xwmx.github.io/nb/)
---
### CRDTs (Automerge, Yjs)
**What it is**: Conflict-free Replicated Data Types for automatic merging
**Strengths**:
- **No conflicts by design**: Concurrent edits automatically merge
- Real-time collaboration possible
- Local-first architecture
- Automerge 3.0 has 10x lower memory usage
**Weaknesses**:
- **Application-level, not filesystem-level**: Requires app integration
- Not a drop-in replacement for file sync
- Larger file sizes than plain text
- Requires specialized tooling
**Best for**: Future note-taking apps, real-time collaboration. Not ready for filesystem sync.
**Sources**:
- [Automerge GitHub](https://github.com/automerge/automerge)
- [Local, First, Forever](https://tonsky.me/blog/crdt-filesync/)
---
## Recommendation Matrix
| Use Case | Recommended Tool | Reasoning |
|----------|------------------|-----------|
| **Notes (Markdown)** | `nb` with git | Explicit sync, full history, proper conflict resolution |
| **Documents (Office, misc)** | Syncthing OR Unison | Syncthing for "set and forget", Unison for control |
| **Code/Dotfiles** | Git + home-manager | Version control is the right tool |
| **Photos** | Immich + rclone | Specialized tool for media |
| **Large files (video, etc.)** | rclone to B2/S3/NAS | Cloud storage with proper backup |
| **Backup** | restic or borg | Deduplication, encryption, versioning |
---
## Why "Syncthing is Buggy" is Usually Wrong
The perception of bugginess typically comes from:
1. **Expectation mismatch**: Users expect Dropbox behavior (cloud-centric) but get P2P behavior
2. **Conflict = failure to them**: Syncthing correctly identifies conflicts; users see this as a bug
3. **Android battery drain**: Real issue, but fixed in forks with battery optimization
4. **Propagated deletions**: Syncthing does what it's supposed to; users wanted a backup
5. **Complex topologies**: 5+ devices with selective sync creates edge cases
**If you experienced "bugginess" with Syncthing, ask yourself**:
- Did you have proper backups outside of Syncthing?
- Were multiple devices editing the same files?
- Did you understand the difference between "Send Only" / "Receive Only" / "Send & Receive"?
- Was your Android device killing the app in the background?
---
## Final Recommendation for This Project
```
┌─────────────────────────────────────────────────────────────┐
│ TIER 1: Configuration │
│ Tool: Git (public GitHub repo) │
│ NixOS flake + home-manager, sops-nix for secrets │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ TIER 2: Syncable Data │
│ │
│ Notes & Writing: │
│ ├── Tool: nb (with git sync to private Forgejo) │
│ ├── Why: Explicit sync, full history, proper merges │
│ └── Conflict handling: Git-style (user resolves) │
│ │
│ Other Documents: │
│ ├── Tool: Syncthing (if you want auto-sync) │
│ │ OR Unison (if you want manual control) │
│ ├── Why: Real-time for docs that change less frequently │
│ └── Conflict handling: .sync-conflict files (Syncthing) │
│ or interactive (Unison) │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ TIER 3: Large Data │
│ │
│ Photos: Immich (self-hosted) │
│ Media: Jellyfin (self-hosted) │
│ Repos: Forgejo (self-hosted) │
│ Backup: restic → B2/Backblaze/NAS │
│ Access: On-demand download via rclone or web UI │
└─────────────────────────────────────────────────────────────┘
```
The key insight: **Use git for text, file-sync for binary, and specialized tools for media**.