Initial commit: Ultimate Notetaking, Sync & Backup System
A NixOS-based system for managing personal data across three tiers: - Tier 1: Configuration (shareable via git) - Tier 2: Syncable data (nb + Syncthing) - Tier 3: Large data (self-hosted services + backup) Includes: - NixOS modules for nb, Syncthing, backup (restic) - Server modules for Forgejo, Immich, Jellyfin - Helper scripts (usync, ustatus) - Comprehensive documentation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
267
docs/ARCHITECTURE.md
Normal file
267
docs/ARCHITECTURE.md
Normal file
@@ -0,0 +1,267 @@
|
||||
# System Architecture
|
||||
|
||||
## Design Principles
|
||||
|
||||
### 1. Declarative Everything
|
||||
|
||||
Every aspect of this system is declared in Nix. No imperative setup steps, no hidden state.
|
||||
|
||||
```
|
||||
Desired State (Nix) → nixos-rebuild → Actual State
|
||||
```
|
||||
|
||||
### 2. Separation of Concerns
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ CONFIGURATION LAYER │
|
||||
│ What: How machines should be configured │
|
||||
│ Where: This repo (flake.nix, modules/) │
|
||||
│ Managed by: Git + GitHub │
|
||||
│ Shareable: Yes (public) │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ DATA LAYER │
|
||||
│ │
|
||||
│ ┌────────────────────┐ ┌────────────────────┐ │
|
||||
│ │ TIER 2: Sync │ │ TIER 3: Backup │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ ~/notes/ (nb) │ │ Photos (Immich) │ │
|
||||
│ │ ~/Documents/ │ │ Media (Jellyfin) │ │
|
||||
│ │ │ │ Repos (Forgejo) │ │
|
||||
│ │ Managed by: │ │ │ │
|
||||
│ │ - nb (git sync) │ │ Managed by: │ │
|
||||
│ │ - Syncthing │ │ - Server apps │ │
|
||||
│ │ │ │ - restic backup │ │
|
||||
│ │ Multi-device: Yes │ │ Multi-device: No │ │
|
||||
│ │ Real-time: Yes │ │ (on-demand access)│ │
|
||||
│ └────────────────────┘ └────────────────────┘ │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 3. No Vendor Lock-in
|
||||
|
||||
Every component can be replaced:
|
||||
|
||||
| Component | Could be replaced with |
|
||||
|-----------|----------------------|
|
||||
| nb | Obsidian + git, org-mode, plain vim + git |
|
||||
| Syncthing | Unison, rclone bisync |
|
||||
| Forgejo | Gitea, GitLab, bare git |
|
||||
| Immich | PhotoPrism, any DAM |
|
||||
| Jellyfin | Plex (non-FOSS), Kodi |
|
||||
| restic | borg, duplicity |
|
||||
|
||||
### 4. Offline-First
|
||||
|
||||
All Tier 2 data is fully available offline:
|
||||
- nb notebooks are local git repos
|
||||
- Syncthing folders are local directories
|
||||
- Sync happens when connectivity allows
|
||||
|
||||
Tier 3 data is available via caching or on-demand download.
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Tier 1: Configuration
|
||||
|
||||
```
|
||||
┌─────────┐ git push ┌─────────┐
|
||||
│ Local │ ───────────────► │ GitHub │
|
||||
│ Repo │ ◄─────────────── │ (pub) │
|
||||
└────┬────┘ git pull └─────────┘
|
||||
│
|
||||
│ nixos-rebuild switch
|
||||
▼
|
||||
┌─────────┐
|
||||
│ Machine │
|
||||
│ State │
|
||||
└─────────┘
|
||||
```
|
||||
|
||||
### Tier 2: Notes (nb)
|
||||
|
||||
```
|
||||
┌─────────┐ ┌─────────┐
|
||||
│ Device │ nb sync │ Forgejo │
|
||||
│ A │ ◄────────────────► │ (priv) │
|
||||
└─────────┘ └────┬────┘
|
||||
│
|
||||
┌─────────┐ nb sync ┌────▼────┐
|
||||
│ Device │ ◄────────────────► │ Forgejo │
|
||||
│ B │ │ (priv) │
|
||||
└─────────┘ └─────────┘
|
||||
|
||||
Conflict Resolution: Git 3-way merge
|
||||
History: Full git history preserved
|
||||
```
|
||||
|
||||
### Tier 2: Documents (Syncthing)
|
||||
|
||||
```
|
||||
┌─────────┐ ┌─────────┐
|
||||
│ Device │ ◄──── P2P ───────► │ Device │
|
||||
│ A │ │ B │
|
||||
└────┬────┘ └────┬────┘
|
||||
│ │
|
||||
│ ┌─────────┐ │
|
||||
└────────►│ Device │◄─────────┘
|
||||
│ C │
|
||||
└─────────┘
|
||||
|
||||
Conflict Resolution: .sync-conflict files (manual)
|
||||
History: Syncthing versioning (configurable)
|
||||
```
|
||||
|
||||
### Tier 3: Large Data
|
||||
|
||||
```
|
||||
┌─────────┐ upload ┌─────────┐
|
||||
│ Device │ ──────────────► │ Server │
|
||||
│ │ ◄────────────── │ Immich/ │
|
||||
└─────────┘ on-demand │ Jellyfin│
|
||||
└────┬────┘
|
||||
│ restic
|
||||
┌────▼────┐
|
||||
│ Backup │
|
||||
│ B2/NAS │
|
||||
└─────────┘
|
||||
```
|
||||
|
||||
## Module Dependency Graph
|
||||
|
||||
```
|
||||
flake.nix
|
||||
│
|
||||
┌───────────────┼───────────────┐
|
||||
▼ ▼ ▼
|
||||
hosts/ modules/ home/
|
||||
desktop (shared) default.nix
|
||||
laptop │
|
||||
server │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────┐
|
||||
│ modules/ │
|
||||
│ ┌──────┐ ┌───────────┐ ┌─────┐ │
|
||||
│ │ nb │ │ syncthing │ │nvim │ │
|
||||
│ └──────┘ └───────────┘ └─────┘ │
|
||||
│ ┌────────┐ │
|
||||
│ │ backup │ │
|
||||
│ └────────┘ │
|
||||
│ │
|
||||
│ ┌─────────── server/ ────────┐ │
|
||||
│ │ forgejo immich jellyfin │ │
|
||||
│ └───────────────────────────┘ │
|
||||
└──────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Security Model
|
||||
|
||||
### Secrets
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ sops-nix │
|
||||
│ │
|
||||
│ secrets.yaml (encrypted in repo) │
|
||||
│ │ │
|
||||
│ │ age/gpg key (not in repo) │
|
||||
│ ▼ │
|
||||
│ Decrypted at activation time │
|
||||
│ Placed in /run/secrets/ │
|
||||
│ Permissions set per-secret │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Network Security
|
||||
|
||||
- All sync over TLS (Syncthing) or SSH (nb, Forgejo)
|
||||
- Server services behind reverse proxy (Caddy/nginx)
|
||||
- No ports exposed except 443 (HTTPS)
|
||||
- Tailscale/WireGuard for private network (optional)
|
||||
|
||||
## Backup Strategy
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────┐
|
||||
│ 3-2-1 Backup Rule │
|
||||
│ │
|
||||
│ 3 copies: │
|
||||
│ ├── Primary: Local device │
|
||||
│ ├── Secondary: Home server │
|
||||
│ └── Tertiary: Offsite (B2/cloud) │
|
||||
│ │
|
||||
│ 2 media types: │
|
||||
│ ├── SSD (local) │
|
||||
│ └── Cloud object storage (offsite) │
|
||||
│ │
|
||||
│ 1 offsite: │
|
||||
│ └── Backblaze B2 / AWS S3 / etc. │
|
||||
└────────────────────────────────────────────────────────────┘
|
||||
|
||||
Backup Schedule (systemd timers):
|
||||
├── Tier 2 (notes/docs): Daily, keep 30 days
|
||||
├── Tier 3 (media): Weekly, keep 90 days
|
||||
└── Server state: Daily, keep 14 days
|
||||
```
|
||||
|
||||
## Scaling Considerations
|
||||
|
||||
### Adding a New Machine
|
||||
|
||||
1. Create `hosts/<hostname>/configuration.nix`
|
||||
2. Add hardware-configuration.nix
|
||||
3. Add to flake.nix outputs
|
||||
4. Run `nixos-rebuild switch --flake .#<hostname>`
|
||||
5. nb automatically syncs when `nb sync` is run
|
||||
6. Syncthing auto-discovers via device IDs in config
|
||||
|
||||
### Adding a New User
|
||||
|
||||
1. Add user to `hosts/<hostname>/configuration.nix`
|
||||
2. Create home-manager config
|
||||
3. Set up nb notebooks for user
|
||||
4. Add Syncthing device ID
|
||||
|
||||
### Multiple Servers
|
||||
|
||||
The architecture supports multiple servers:
|
||||
|
||||
```
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ Server │ │ Server │ │ Server │
|
||||
│ Home │ │ VPS │ │ NAS │
|
||||
│ (media) │ │ (Forgejo│ │(backup) │
|
||||
└─────────┘ └─────────┘ └─────────┘
|
||||
```
|
||||
|
||||
Each server has its own host config in `hosts/`.
|
||||
|
||||
## Failure Modes
|
||||
|
||||
| Failure | Impact | Recovery |
|
||||
|---------|--------|----------|
|
||||
| Device lost | Tier 2 data unavailable locally | Sync from other devices or Forgejo |
|
||||
| Server down | Tier 3 unavailable, nb sync fails | Use local cache, fix server |
|
||||
| Backup failure | No new backups | Alert (systemd OnFailure), investigate |
|
||||
| Sync conflict | Manual resolution needed | Resolve .sync-conflict or git merge |
|
||||
| Internet down | Can't sync | Work offline, sync when back |
|
||||
|
||||
## Future Considerations
|
||||
|
||||
### Potential Additions
|
||||
|
||||
1. **Encrypted Tier 2**: Full-disk encryption + nb's GPG encryption
|
||||
2. **Mobile support**: Termux + Unison, or Syncthing-Fork
|
||||
3. **Real-time collaboration**: Automerge/CRDT-based notes
|
||||
4. **Monitoring**: Prometheus + Grafana for sync/backup status
|
||||
|
||||
### Not In Scope
|
||||
|
||||
1. Windows support (NixOS only)
|
||||
2. Proprietary services integration
|
||||
3. Enterprise/team features
|
||||
4. Real-time streaming replication
|
||||
Reference in New Issue
Block a user