Initial commit: Ultimate Notetaking, Sync & Backup System

A NixOS-based system for managing personal data across three tiers: - Tier 1: Configuration (shareable via git) - Tier 2: Syncable data (nb + Syncthing) - Tier 3: Large data (self-hosted services + backup) Includes: - NixOS modules for nb, Syncthing, backup (restic) - Server modules for Forgejo, Immich, Jellyfin - Helper scripts (usync, ustatus) - Comprehensive documentation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-13 01:44:00 -05:00
commit b40ac99524
17 changed files with 3151 additions and 0 deletions
--- a/docs/research/sync-tools-comparison.md
+++ b/docs/research/sync-tools-comparison.md
@@ -0,0 +1,271 @@
+# File Synchronization Tools: Comprehensive Comparison
+
+*Research conducted: February 2026*
+
+## Executive Summary
+
+After extensive research, **Syncthing is not buggy—but it is complex and unforgiving**. Most reported "bugs" are actually:
+1. Misunderstanding of how P2P sync works
+2. Conflict handling that differs from user expectations
+3. Android-specific issues (now largely resolved with forks)
+4. Users treating it as a backup tool (it's not)
+
+For a NixOS-based notes/data sync system, the recommended approach is:
+- **Tier 2 (Notes)**: `nb` with git sync to private Forgejo (explicit, versioned)
+- **Tier 2 (Documents)**: Syncthing OR Unison (based on preference)
+- **Tier 3 (Large files)**: rclone + restic to self-hosted or cloud storage
+
+---
+
+## Tool-by-Tool Analysis
+
+### Syncthing
+
+**What it is**: Continuous, real-time, P2P file synchronization
+
+**Strengths**:
+- Truly decentralized (no server required)
+- Real-time sync via filesystem watching
+- Handles file renames/moves efficiently
+- Cross-platform with good GUI
+- Battle-tested with large user base
+- Declarative NixOS module available
+
+**Weaknesses**:
+- **Conflict handling is file-level**: Creates `.sync-conflict-*` files that must be manually resolved
+- **Not a backup**: Deletions propagate immediately; if you delete on one device, it's gone everywhere
+- **Android app discontinued** (Dec 2024): Use [Syncthing-Fork](https://github.com/catfriend1/syncthing-android) instead
+- **Complexity ceiling**: Multi-device topologies can create unexpected sync loops
+- **No ownership/permission sync**: By design, to avoid requiring root
+- **OOM issues in v2.0**: On low-memory devices (1GB RAM) with large file sets
+
+**Common Misconceptions**:
+| Issue Reported | Actual Cause |
+|----------------|--------------|
+| "Files disappearing" | Deletions syncing correctly; user expected backup behavior |
+| "Sync stuck at 99%" | Large files still transferring, or ignored files being counted |
+| "Random conflicts" | Multiple devices modifying same file; Syncthing is doing its job |
+| "Ignoring read-only" | Misconfigured folder type; "Receive Only" ≠ read-only filesystem |
+
+**Verdict**: Syncthing works well if you understand that it's **sync, not backup**. For notes that change frequently on multiple devices, conflicts will happen.
+
+**Sources**:
+- [Syncthing GitHub Issues](https://github.com/syncthing/syncthing/issues)
+- [Syncthing 2.0 OOM Discussion](https://forum.syncthing.net/t/syncthing-2-0-1-oom/24824)
+- [Lobsters: "Do not use Syncthing"](https://lobste.rs/s/pacmpc/do_not_use_syncthing)
+- [Syncthing Android Discontinuation](https://www.ghacks.net/2024/10/21/syncthing-for-android-is-being-discontinued-but-theres-an-alternative-app-you-can-switch-to/)
+
+---
+
+### Unison
+
+**What it is**: Bidirectional file synchronizer, runs on-demand (not continuous)
+
+**Strengths**:
+- **Explicit sync**: You run it when you want, see exactly what will happen
+- **Proper conflict detection**: Shows both versions, lets you choose
+- **SSH-native**: Works over SSH, no extra protocols
+- **Version 2.52+ is OCaml-independent**: No more version matching hell
+- **ACL/xattr support** (v2.53+): Can sync permissions and extended attributes
+- **Hub-and-spoke works great**: Sync all devices to a NAS/server
+- **Lower battery usage** than continuous sync
+
+**Weaknesses**:
+- **Manual invocation**: No background daemon (by design, but requires discipline)
+- **File rename detection is poor**: Rename = delete + create, potentially losing history
+- **GUI is deprecated**: lablgtk maintenance uncertain; CLI is the future
+- **Small maintainer base**: ~2.5 people, 0.1 FTE
+- **Learning curve**: Profile configuration can be confusing
+
+**Best for**: Users who want explicit control over sync timing and conflict resolution
+
+**Sources**:
+- [Unison GitHub](https://github.com/bcpierce00/unison)
+- [Migration from Syncthing to Unison](https://codelearn.me/2025/12/29/from-syncthing-to-unison.html)
+- [Syncthing Forum Comparison](https://forum.syncthing.net/t/syncthing-vs-rsync-vs-unison/6298)
+
+---
+
+### rsync
+
+**What it is**: One-way file copying/mirroring tool
+
+**Strengths**:
+- Rock solid, ubiquitous
+- Efficient delta transfers
+- Perfect for backups
+
+**Weaknesses**:
+- **Unidirectional only**: Not a sync tool
+- No conflict handling (overwrites target)
+
+**Best for**: Backups, deployments, one-way mirroring. Not for bidirectional sync.
+
+---
+
+### rclone (with bisync)
+
+**What it is**: Cloud storage Swiss Army knife, with experimental bidirectional sync
+
+**Strengths**:
+- Supports 70+ cloud backends
+- `bisync` command provides bidirectional sync (as of v1.66)
+- Checksum-based comparison available
+- Lock files prevent concurrent corruption
+- Great for cloud ↔ local sync
+
+**Weaknesses**:
+- **bisync is "advanced"**: Documentation warns of potential data loss
+- Designed for cloud storage, not LAN sync
+- More complex ignore pattern syntax than Syncthing
+- No real-time watching (cron-based)
+
+**Best for**: Syncing with cloud storage (S3, B2, Drive, etc.), Tier 3 backup
+
+**Sources**:
+- [rclone bisync documentation](https://rclone.org/bisync/)
+
+---
+
+### Mutagen
+
+**What it is**: Fast bidirectional sync for remote development
+
+**Strengths**:
+- Designed for code sync to remote containers/servers
+- Very fast (near-native Docker volume performance)
+- Real-time filesystem watching
+- Multiple sync modes (two-way-safe, one-way, etc.)
+- SSH and Docker native
+
+**Weaknesses**:
+- **Ephemeral by design**: Not meant for long-term sync
+- Less configuration flexibility than Syncthing
+- Primarily for developer workflows
+
+**Best for**: Remote development, Docker volumes, SSH workspaces. Overkill for notes.
+
+**Sources**:
+- [Mutagen GitHub](https://github.com/mutagen-io/mutagen)
+- [Mutagen Documentation](https://mutagen.io/documentation/synchronization/)
+
+---
+
+### nb (Notebook CLI)
+
+**What it is**: CLI note-taking with built-in git sync
+
+**Strengths**:
+- **Git-native**: Every notebook is a git repo
+- Explicit sync via `nb sync`
+- Full history, branching, proper merges
+- Works with any git remote (GitHub, Gitea, Forgejo)
+- Markdown by default, encryption optional
+- Single portable shell script
+- `$EDITOR` integration (neovim-friendly)
+
+**Weaknesses**:
+- **Explicit sync only**: Must run `nb sync` (or set up cron/systemd)
+- Git conflicts require git knowledge to resolve
+- No GUI (by design)
+
+**Best for**: Notes and writing. This is the ideal Tier 2 solution for text content.
+
+**Sources**:
+- [nb GitHub](https://github.com/xwmx/nb)
+- [nb Documentation](https://xwmx.github.io/nb/)
+
+---
+
+### CRDTs (Automerge, Yjs)
+
+**What it is**: Conflict-free Replicated Data Types for automatic merging
+
+**Strengths**:
+- **No conflicts by design**: Concurrent edits automatically merge
+- Real-time collaboration possible
+- Local-first architecture
+- Automerge 3.0 has 10x lower memory usage
+
+**Weaknesses**:
+- **Application-level, not filesystem-level**: Requires app integration
+- Not a drop-in replacement for file sync
+- Larger file sizes than plain text
+- Requires specialized tooling
+
+**Best for**: Future note-taking apps, real-time collaboration. Not ready for filesystem sync.
+
+**Sources**:
+- [Automerge GitHub](https://github.com/automerge/automerge)
+- [Local, First, Forever](https://tonsky.me/blog/crdt-filesync/)
+
+---
+
+## Recommendation Matrix
+
+| Use Case | Recommended Tool | Reasoning |
+|----------|------------------|-----------|
+| **Notes (Markdown)** | `nb` with git | Explicit sync, full history, proper conflict resolution |
+| **Documents (Office, misc)** | Syncthing OR Unison | Syncthing for "set and forget", Unison for control |
+| **Code/Dotfiles** | Git + home-manager | Version control is the right tool |
+| **Photos** | Immich + rclone | Specialized tool for media |
+| **Large files (video, etc.)** | rclone to B2/S3/NAS | Cloud storage with proper backup |
+| **Backup** | restic or borg | Deduplication, encryption, versioning |
+
+---
+
+## Why "Syncthing is Buggy" is Usually Wrong
+
+The perception of bugginess typically comes from:
+
+1. **Expectation mismatch**: Users expect Dropbox behavior (cloud-centric) but get P2P behavior
+2. **Conflict = failure to them**: Syncthing correctly identifies conflicts; users see this as a bug
+3. **Android battery drain**: Real issue, but fixed in forks with battery optimization
+4. **Propagated deletions**: Syncthing does what it's supposed to; users wanted a backup
+5. **Complex topologies**: 5+ devices with selective sync creates edge cases
+
+**If you experienced "bugginess" with Syncthing, ask yourself**:
+- Did you have proper backups outside of Syncthing?
+- Were multiple devices editing the same files?
+- Did you understand the difference between "Send Only" / "Receive Only" / "Send & Receive"?
+- Was your Android device killing the app in the background?
+
+---
+
+## Final Recommendation for This Project
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  TIER 1: Configuration                                       │
+│  Tool: Git (public GitHub repo)                              │
+│  NixOS flake + home-manager, sops-nix for secrets           │
+└─────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────┐
+│  TIER 2: Syncable Data                                       │
+│                                                              │
+│  Notes & Writing:                                            │
+│  ├── Tool: nb (with git sync to private Forgejo)            │
+│  ├── Why: Explicit sync, full history, proper merges        │
+│  └── Conflict handling: Git-style (user resolves)           │
+│                                                              │
+│  Other Documents:                                            │
+│  ├── Tool: Syncthing (if you want auto-sync)                │
+│  │   OR Unison (if you want manual control)                 │
+│  ├── Why: Real-time for docs that change less frequently    │
+│  └── Conflict handling: .sync-conflict files (Syncthing)    │
+│                        or interactive (Unison)              │
+└─────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────┐
+│  TIER 3: Large Data                                          │
+│                                                              │
+│  Photos: Immich (self-hosted)                                │
+│  Media: Jellyfin (self-hosted)                               │
+│  Repos: Forgejo (self-hosted)                                │
+│  Backup: restic → B2/Backblaze/NAS                          │
+│  Access: On-demand download via rclone or web UI            │
+└─────────────────────────────────────────────────────────────┘
+```
+
+The key insight: **Use git for text, file-sync for binary, and specialized tools for media**.