Initial commit: Ultimate Notetaking, Sync & Backup System
A NixOS-based system for managing personal data across three tiers: - Tier 1: Configuration (shareable via git) - Tier 2: Syncable data (nb + Syncthing) - Tier 3: Large data (self-hosted services + backup) Includes: - NixOS modules for nb, Syncthing, backup (restic) - Server modules for Forgejo, Immich, Jellyfin - Helper scripts (usync, ustatus) - Comprehensive documentation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
267
docs/ARCHITECTURE.md
Normal file
267
docs/ARCHITECTURE.md
Normal file
@@ -0,0 +1,267 @@
|
||||
# System Architecture
|
||||
|
||||
## Design Principles
|
||||
|
||||
### 1. Declarative Everything
|
||||
|
||||
Every aspect of this system is declared in Nix. No imperative setup steps, no hidden state.
|
||||
|
||||
```
|
||||
Desired State (Nix) → nixos-rebuild → Actual State
|
||||
```
|
||||
|
||||
### 2. Separation of Concerns
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ CONFIGURATION LAYER │
|
||||
│ What: How machines should be configured │
|
||||
│ Where: This repo (flake.nix, modules/) │
|
||||
│ Managed by: Git + GitHub │
|
||||
│ Shareable: Yes (public) │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ DATA LAYER │
|
||||
│ │
|
||||
│ ┌────────────────────┐ ┌────────────────────┐ │
|
||||
│ │ TIER 2: Sync │ │ TIER 3: Backup │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ ~/notes/ (nb) │ │ Photos (Immich) │ │
|
||||
│ │ ~/Documents/ │ │ Media (Jellyfin) │ │
|
||||
│ │ │ │ Repos (Forgejo) │ │
|
||||
│ │ Managed by: │ │ │ │
|
||||
│ │ - nb (git sync) │ │ Managed by: │ │
|
||||
│ │ - Syncthing │ │ - Server apps │ │
|
||||
│ │ │ │ - restic backup │ │
|
||||
│ │ Multi-device: Yes │ │ Multi-device: No │ │
|
||||
│ │ Real-time: Yes │ │ (on-demand access)│ │
|
||||
│ └────────────────────┘ └────────────────────┘ │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 3. No Vendor Lock-in
|
||||
|
||||
Every component can be replaced:
|
||||
|
||||
| Component | Could be replaced with |
|
||||
|-----------|----------------------|
|
||||
| nb | Obsidian + git, org-mode, plain vim + git |
|
||||
| Syncthing | Unison, rclone bisync |
|
||||
| Forgejo | Gitea, GitLab, bare git |
|
||||
| Immich | PhotoPrism, any DAM |
|
||||
| Jellyfin | Plex (non-FOSS), Kodi |
|
||||
| restic | borg, duplicity |
|
||||
|
||||
### 4. Offline-First
|
||||
|
||||
All Tier 2 data is fully available offline:
|
||||
- nb notebooks are local git repos
|
||||
- Syncthing folders are local directories
|
||||
- Sync happens when connectivity allows
|
||||
|
||||
Tier 3 data is available via caching or on-demand download.
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Tier 1: Configuration
|
||||
|
||||
```
|
||||
┌─────────┐ git push ┌─────────┐
|
||||
│ Local │ ───────────────► │ GitHub │
|
||||
│ Repo │ ◄─────────────── │ (pub) │
|
||||
└────┬────┘ git pull └─────────┘
|
||||
│
|
||||
│ nixos-rebuild switch
|
||||
▼
|
||||
┌─────────┐
|
||||
│ Machine │
|
||||
│ State │
|
||||
└─────────┘
|
||||
```
|
||||
|
||||
### Tier 2: Notes (nb)
|
||||
|
||||
```
|
||||
┌─────────┐ ┌─────────┐
|
||||
│ Device │ nb sync │ Forgejo │
|
||||
│ A │ ◄────────────────► │ (priv) │
|
||||
└─────────┘ └────┬────┘
|
||||
│
|
||||
┌─────────┐ nb sync ┌────▼────┐
|
||||
│ Device │ ◄────────────────► │ Forgejo │
|
||||
│ B │ │ (priv) │
|
||||
└─────────┘ └─────────┘
|
||||
|
||||
Conflict Resolution: Git 3-way merge
|
||||
History: Full git history preserved
|
||||
```
|
||||
|
||||
### Tier 2: Documents (Syncthing)
|
||||
|
||||
```
|
||||
┌─────────┐ ┌─────────┐
|
||||
│ Device │ ◄──── P2P ───────► │ Device │
|
||||
│ A │ │ B │
|
||||
└────┬────┘ └────┬────┘
|
||||
│ │
|
||||
│ ┌─────────┐ │
|
||||
└────────►│ Device │◄─────────┘
|
||||
│ C │
|
||||
└─────────┘
|
||||
|
||||
Conflict Resolution: .sync-conflict files (manual)
|
||||
History: Syncthing versioning (configurable)
|
||||
```
|
||||
|
||||
### Tier 3: Large Data
|
||||
|
||||
```
|
||||
┌─────────┐ upload ┌─────────┐
|
||||
│ Device │ ──────────────► │ Server │
|
||||
│ │ ◄────────────── │ Immich/ │
|
||||
└─────────┘ on-demand │ Jellyfin│
|
||||
└────┬────┘
|
||||
│ restic
|
||||
┌────▼────┐
|
||||
│ Backup │
|
||||
│ B2/NAS │
|
||||
└─────────┘
|
||||
```
|
||||
|
||||
## Module Dependency Graph
|
||||
|
||||
```
|
||||
flake.nix
|
||||
│
|
||||
┌───────────────┼───────────────┐
|
||||
▼ ▼ ▼
|
||||
hosts/ modules/ home/
|
||||
desktop (shared) default.nix
|
||||
laptop │
|
||||
server │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────┐
|
||||
│ modules/ │
|
||||
│ ┌──────┐ ┌───────────┐ ┌─────┐ │
|
||||
│ │ nb │ │ syncthing │ │nvim │ │
|
||||
│ └──────┘ └───────────┘ └─────┘ │
|
||||
│ ┌────────┐ │
|
||||
│ │ backup │ │
|
||||
│ └────────┘ │
|
||||
│ │
|
||||
│ ┌─────────── server/ ────────┐ │
|
||||
│ │ forgejo immich jellyfin │ │
|
||||
│ └───────────────────────────┘ │
|
||||
└──────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Security Model
|
||||
|
||||
### Secrets
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ sops-nix │
|
||||
│ │
|
||||
│ secrets.yaml (encrypted in repo) │
|
||||
│ │ │
|
||||
│ │ age/gpg key (not in repo) │
|
||||
│ ▼ │
|
||||
│ Decrypted at activation time │
|
||||
│ Placed in /run/secrets/ │
|
||||
│ Permissions set per-secret │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Network Security
|
||||
|
||||
- All sync over TLS (Syncthing) or SSH (nb, Forgejo)
|
||||
- Server services behind reverse proxy (Caddy/nginx)
|
||||
- No ports exposed except 443 (HTTPS)
|
||||
- Tailscale/WireGuard for private network (optional)
|
||||
|
||||
## Backup Strategy
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────┐
|
||||
│ 3-2-1 Backup Rule │
|
||||
│ │
|
||||
│ 3 copies: │
|
||||
│ ├── Primary: Local device │
|
||||
│ ├── Secondary: Home server │
|
||||
│ └── Tertiary: Offsite (B2/cloud) │
|
||||
│ │
|
||||
│ 2 media types: │
|
||||
│ ├── SSD (local) │
|
||||
│ └── Cloud object storage (offsite) │
|
||||
│ │
|
||||
│ 1 offsite: │
|
||||
│ └── Backblaze B2 / AWS S3 / etc. │
|
||||
└────────────────────────────────────────────────────────────┘
|
||||
|
||||
Backup Schedule (systemd timers):
|
||||
├── Tier 2 (notes/docs): Daily, keep 30 days
|
||||
├── Tier 3 (media): Weekly, keep 90 days
|
||||
└── Server state: Daily, keep 14 days
|
||||
```
|
||||
|
||||
## Scaling Considerations
|
||||
|
||||
### Adding a New Machine
|
||||
|
||||
1. Create `hosts/<hostname>/configuration.nix`
|
||||
2. Add hardware-configuration.nix
|
||||
3. Add to flake.nix outputs
|
||||
4. Run `nixos-rebuild switch --flake .#<hostname>`
|
||||
5. nb automatically syncs when `nb sync` is run
|
||||
6. Syncthing auto-discovers via device IDs in config
|
||||
|
||||
### Adding a New User
|
||||
|
||||
1. Add user to `hosts/<hostname>/configuration.nix`
|
||||
2. Create home-manager config
|
||||
3. Set up nb notebooks for user
|
||||
4. Add Syncthing device ID
|
||||
|
||||
### Multiple Servers
|
||||
|
||||
The architecture supports multiple servers:
|
||||
|
||||
```
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ Server │ │ Server │ │ Server │
|
||||
│ Home │ │ VPS │ │ NAS │
|
||||
│ (media) │ │ (Forgejo│ │(backup) │
|
||||
└─────────┘ └─────────┘ └─────────┘
|
||||
```
|
||||
|
||||
Each server has its own host config in `hosts/`.
|
||||
|
||||
## Failure Modes
|
||||
|
||||
| Failure | Impact | Recovery |
|
||||
|---------|--------|----------|
|
||||
| Device lost | Tier 2 data unavailable locally | Sync from other devices or Forgejo |
|
||||
| Server down | Tier 3 unavailable, nb sync fails | Use local cache, fix server |
|
||||
| Backup failure | No new backups | Alert (systemd OnFailure), investigate |
|
||||
| Sync conflict | Manual resolution needed | Resolve .sync-conflict or git merge |
|
||||
| Internet down | Can't sync | Work offline, sync when back |
|
||||
|
||||
## Future Considerations
|
||||
|
||||
### Potential Additions
|
||||
|
||||
1. **Encrypted Tier 2**: Full-disk encryption + nb's GPG encryption
|
||||
2. **Mobile support**: Termux + Unison, or Syncthing-Fork
|
||||
3. **Real-time collaboration**: Automerge/CRDT-based notes
|
||||
4. **Monitoring**: Prometheus + Grafana for sync/backup status
|
||||
|
||||
### Not In Scope
|
||||
|
||||
1. Windows support (NixOS only)
|
||||
2. Proprietary services integration
|
||||
3. Enterprise/team features
|
||||
4. Real-time streaming replication
|
||||
240
docs/LLM-CONTEXT.md
Normal file
240
docs/LLM-CONTEXT.md
Normal file
@@ -0,0 +1,240 @@
|
||||
# LLM Context Document
|
||||
|
||||
This document provides context for AI assistants (Claude, GPT, Copilot, etc.) working with this codebase.
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Name**: Ultimate Notetaking, Sync & Backup System
|
||||
**Purpose**: NixOS-based declarative system for managing personal data across multiple machines
|
||||
**Primary User**: Linux users, especially NixOS, who want full control over their data
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### The Three-Tier Data Model
|
||||
|
||||
```
|
||||
TIER 1: Configuration (Shareable)
|
||||
├── What: System config, dotfiles, flake.nix
|
||||
├── Where: This git repo (public on GitHub)
|
||||
├── How: NixOS + home-manager + sops-nix
|
||||
└── Philosophy: Anyone can clone and use this config
|
||||
|
||||
TIER 2: Syncable Data (Private, Multi-device)
|
||||
├── What: Notes, documents, writings
|
||||
├── Where: ~/notes/ (nb), ~/Documents/ (Syncthing)
|
||||
├── How: nb (git-backed) for text, Syncthing for binary
|
||||
└── Philosophy: Edit anywhere, sync automatically
|
||||
|
||||
TIER 3: Large Data (Backed up, On-demand)
|
||||
├── What: Photos, videos, large repos
|
||||
├── Where: Self-hosted servers (Immich, Jellyfin, Forgejo)
|
||||
├── How: Upload to server, access on-demand, restic backup
|
||||
└── Philosophy: Don't sync everything; backup everything
|
||||
```
|
||||
|
||||
### Key Tools
|
||||
|
||||
| Tool | Purpose | Tier |
|
||||
|------|---------|------|
|
||||
| NixOS | Declarative system configuration | 1 |
|
||||
| home-manager | Declarative user configuration | 1 |
|
||||
| sops-nix | Encrypted secrets in git | 1 |
|
||||
| nb | CLI notebook with git sync | 2 |
|
||||
| Syncthing | P2P file synchronization | 2 |
|
||||
| Neovim | Primary editor | 1, 2 |
|
||||
| Forgejo | Self-hosted git (GitHub alternative) | 3 |
|
||||
| Immich | Self-hosted photos (Google Photos alternative) | 3 |
|
||||
| Jellyfin | Self-hosted media (Plex alternative) | 3 |
|
||||
| restic | Encrypted, deduplicated backups | 3 |
|
||||
| rclone | Cloud storage Swiss Army knife | 3 |
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
flake.nix # Nix flake entry point
|
||||
├── inputs: nixpkgs, home-manager, sops-nix
|
||||
├── outputs: nixosConfigurations, homeConfigurations, devShells
|
||||
└── imports modules for each host
|
||||
|
||||
modules/
|
||||
├── nb.nix # nb installation and config
|
||||
├── syncthing.nix # Declarative Syncthing
|
||||
├── neovim.nix # Neovim with nb integration
|
||||
├── backup.nix # restic systemd timers
|
||||
└── server/ # Server-only modules
|
||||
├── forgejo.nix
|
||||
├── immich.nix
|
||||
└── jellyfin.nix
|
||||
|
||||
hosts/
|
||||
├── desktop/ # Desktop-specific config
|
||||
├── laptop/ # Laptop-specific config
|
||||
└── server/ # Server-specific config
|
||||
|
||||
home/
|
||||
└── default.nix # home-manager user config
|
||||
|
||||
scripts/
|
||||
├── usync # Unified sync (nb + Syncthing)
|
||||
├── ubackup # Manual backup trigger
|
||||
└── ustatus # Status dashboard
|
||||
```
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### User wants to add a new notebook
|
||||
|
||||
1. Edit `modules/nb.nix`
|
||||
2. Add notebook to `programs.nb.notebooks`
|
||||
3. Run `nixos-rebuild switch`
|
||||
4. Run `nb notebooks` to verify
|
||||
|
||||
### User wants to add a Syncthing folder
|
||||
|
||||
1. Edit `modules/syncthing.nix`
|
||||
2. Add folder to `services.syncthing.settings.folders`
|
||||
3. Add device IDs if new devices
|
||||
4. Run `nixos-rebuild switch`
|
||||
|
||||
### User wants to change backup paths
|
||||
|
||||
1. Edit `modules/backup.nix`
|
||||
2. Modify `services.restic.backups.default.paths`
|
||||
3. Run `nixos-rebuild switch`
|
||||
|
||||
### User wants to add a new host
|
||||
|
||||
1. Create `hosts/<hostname>/configuration.nix`
|
||||
2. Add to `flake.nix` outputs
|
||||
3. Configure host-specific settings
|
||||
4. Run `nixos-rebuild switch --flake .#<hostname>`
|
||||
|
||||
## Code Patterns
|
||||
|
||||
### Module Structure
|
||||
|
||||
All modules follow this pattern:
|
||||
|
||||
```nix
|
||||
{ config, lib, pkgs, ... }:
|
||||
|
||||
with lib;
|
||||
|
||||
let
|
||||
cfg = config.services.ourService;
|
||||
in {
|
||||
options.services.ourService = {
|
||||
enable = mkEnableOption "our service";
|
||||
|
||||
someOption = mkOption {
|
||||
type = types.str;
|
||||
default = "default-value";
|
||||
description = "Description of the option";
|
||||
};
|
||||
};
|
||||
|
||||
config = mkIf cfg.enable {
|
||||
# Implementation
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Home-manager Integration
|
||||
|
||||
User-level config uses home-manager:
|
||||
|
||||
```nix
|
||||
{ config, pkgs, ... }:
|
||||
|
||||
{
|
||||
programs.git = {
|
||||
enable = true;
|
||||
userName = "User Name";
|
||||
userEmail = "user@example.com";
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Secrets Management
|
||||
|
||||
Secrets use sops-nix:
|
||||
|
||||
```nix
|
||||
{
|
||||
sops.secrets.syncthing-key = {
|
||||
sopsFile = ./secrets/secrets.yaml;
|
||||
owner = "user";
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## Constraints
|
||||
|
||||
1. **Open source only**: No proprietary tools
|
||||
2. **NixOS-centric**: Must work with `nixos-rebuild`
|
||||
3. **Flakes required**: All Nix code uses flakes
|
||||
4. **No cloud dependencies**: Must work fully self-hosted (cloud optional)
|
||||
5. **CLI-first**: GUI is optional, never required
|
||||
6. **Markdown for notes**: Plain text, portable, version-control friendly
|
||||
|
||||
## When Helping Users
|
||||
|
||||
### DO:
|
||||
- Suggest declarative (Nix) solutions
|
||||
- Recommend nb for text/notes
|
||||
- Recommend Syncthing for binary files
|
||||
- Use systemd for services
|
||||
- Use home-manager for user config
|
||||
- Keep modules composable
|
||||
|
||||
### DON'T:
|
||||
- Suggest proprietary alternatives
|
||||
- Hardcode paths (use options)
|
||||
- Mix system and user config inappropriately
|
||||
- Ignore the three-tier model
|
||||
- Suggest Syncthing for notes (use nb instead)
|
||||
|
||||
## Debugging
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Symptom | Likely Cause | Solution |
|
||||
|---------|--------------|----------|
|
||||
| nb sync fails | SSH key not available | Check `ssh-add -l` |
|
||||
| Syncthing not starting | Declarative config conflict | Check for GUI-made changes |
|
||||
| Backup fails | Repository not initialized | Run `restic init` |
|
||||
| Module not found | Not imported in flake.nix | Add to imports |
|
||||
|
||||
### Useful Commands
|
||||
|
||||
```bash
|
||||
# Check Nix config
|
||||
nix flake check
|
||||
|
||||
# Show Syncthing status
|
||||
syncthing cli show system
|
||||
|
||||
# nb sync status
|
||||
nb sync status
|
||||
|
||||
# restic backup status
|
||||
restic -r <repo> snapshots
|
||||
```
|
||||
|
||||
## Extension Points
|
||||
|
||||
Users commonly want to extend:
|
||||
|
||||
1. **Additional notebooks**: Add to `modules/nb.nix`
|
||||
2. **New Syncthing folders**: Add to `modules/syncthing.nix`
|
||||
3. **Backup exclusions**: Modify `modules/backup.nix`
|
||||
4. **Server services**: Add new modules under `modules/server/`
|
||||
5. **Editor config**: Modify `modules/neovim.nix`
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Architecture](./ARCHITECTURE.md) - Detailed system design
|
||||
- [Sync Tools Comparison](./research/sync-tools-comparison.md) - Why we chose these tools
|
||||
- [nb documentation](https://xwmx.github.io/nb/)
|
||||
- [NixOS manual](https://nixos.org/manual/nixos/stable/)
|
||||
- [home-manager manual](https://nix-community.github.io/home-manager/)
|
||||
271
docs/research/sync-tools-comparison.md
Normal file
271
docs/research/sync-tools-comparison.md
Normal file
@@ -0,0 +1,271 @@
|
||||
# File Synchronization Tools: Comprehensive Comparison
|
||||
|
||||
*Research conducted: February 2026*
|
||||
|
||||
## Executive Summary
|
||||
|
||||
After extensive research, **Syncthing is not buggy—but it is complex and unforgiving**. Most reported "bugs" are actually:
|
||||
1. Misunderstanding of how P2P sync works
|
||||
2. Conflict handling that differs from user expectations
|
||||
3. Android-specific issues (now largely resolved with forks)
|
||||
4. Users treating it as a backup tool (it's not)
|
||||
|
||||
For a NixOS-based notes/data sync system, the recommended approach is:
|
||||
- **Tier 2 (Notes)**: `nb` with git sync to private Forgejo (explicit, versioned)
|
||||
- **Tier 2 (Documents)**: Syncthing OR Unison (based on preference)
|
||||
- **Tier 3 (Large files)**: rclone + restic to self-hosted or cloud storage
|
||||
|
||||
---
|
||||
|
||||
## Tool-by-Tool Analysis
|
||||
|
||||
### Syncthing
|
||||
|
||||
**What it is**: Continuous, real-time, P2P file synchronization
|
||||
|
||||
**Strengths**:
|
||||
- Truly decentralized (no server required)
|
||||
- Real-time sync via filesystem watching
|
||||
- Handles file renames/moves efficiently
|
||||
- Cross-platform with good GUI
|
||||
- Battle-tested with large user base
|
||||
- Declarative NixOS module available
|
||||
|
||||
**Weaknesses**:
|
||||
- **Conflict handling is file-level**: Creates `.sync-conflict-*` files that must be manually resolved
|
||||
- **Not a backup**: Deletions propagate immediately; if you delete on one device, it's gone everywhere
|
||||
- **Android app discontinued** (Dec 2024): Use [Syncthing-Fork](https://github.com/catfriend1/syncthing-android) instead
|
||||
- **Complexity ceiling**: Multi-device topologies can create unexpected sync loops
|
||||
- **No ownership/permission sync**: By design, to avoid requiring root
|
||||
- **OOM issues in v2.0**: On low-memory devices (1GB RAM) with large file sets
|
||||
|
||||
**Common Misconceptions**:
|
||||
| Issue Reported | Actual Cause |
|
||||
|----------------|--------------|
|
||||
| "Files disappearing" | Deletions syncing correctly; user expected backup behavior |
|
||||
| "Sync stuck at 99%" | Large files still transferring, or ignored files being counted |
|
||||
| "Random conflicts" | Multiple devices modifying same file; Syncthing is doing its job |
|
||||
| "Ignoring read-only" | Misconfigured folder type; "Receive Only" ≠ read-only filesystem |
|
||||
|
||||
**Verdict**: Syncthing works well if you understand that it's **sync, not backup**. For notes that change frequently on multiple devices, conflicts will happen.
|
||||
|
||||
**Sources**:
|
||||
- [Syncthing GitHub Issues](https://github.com/syncthing/syncthing/issues)
|
||||
- [Syncthing 2.0 OOM Discussion](https://forum.syncthing.net/t/syncthing-2-0-1-oom/24824)
|
||||
- [Lobsters: "Do not use Syncthing"](https://lobste.rs/s/pacmpc/do_not_use_syncthing)
|
||||
- [Syncthing Android Discontinuation](https://www.ghacks.net/2024/10/21/syncthing-for-android-is-being-discontinued-but-theres-an-alternative-app-you-can-switch-to/)
|
||||
|
||||
---
|
||||
|
||||
### Unison
|
||||
|
||||
**What it is**: Bidirectional file synchronizer, runs on-demand (not continuous)
|
||||
|
||||
**Strengths**:
|
||||
- **Explicit sync**: You run it when you want, see exactly what will happen
|
||||
- **Proper conflict detection**: Shows both versions, lets you choose
|
||||
- **SSH-native**: Works over SSH, no extra protocols
|
||||
- **Version 2.52+ is OCaml-independent**: No more version matching hell
|
||||
- **ACL/xattr support** (v2.53+): Can sync permissions and extended attributes
|
||||
- **Hub-and-spoke works great**: Sync all devices to a NAS/server
|
||||
- **Lower battery usage** than continuous sync
|
||||
|
||||
**Weaknesses**:
|
||||
- **Manual invocation**: No background daemon (by design, but requires discipline)
|
||||
- **File rename detection is poor**: Rename = delete + create, potentially losing history
|
||||
- **GUI is deprecated**: lablgtk maintenance uncertain; CLI is the future
|
||||
- **Small maintainer base**: ~2.5 people, 0.1 FTE
|
||||
- **Learning curve**: Profile configuration can be confusing
|
||||
|
||||
**Best for**: Users who want explicit control over sync timing and conflict resolution
|
||||
|
||||
**Sources**:
|
||||
- [Unison GitHub](https://github.com/bcpierce00/unison)
|
||||
- [Migration from Syncthing to Unison](https://codelearn.me/2025/12/29/from-syncthing-to-unison.html)
|
||||
- [Syncthing Forum Comparison](https://forum.syncthing.net/t/syncthing-vs-rsync-vs-unison/6298)
|
||||
|
||||
---
|
||||
|
||||
### rsync
|
||||
|
||||
**What it is**: One-way file copying/mirroring tool
|
||||
|
||||
**Strengths**:
|
||||
- Rock solid, ubiquitous
|
||||
- Efficient delta transfers
|
||||
- Perfect for backups
|
||||
|
||||
**Weaknesses**:
|
||||
- **Unidirectional only**: Not a sync tool
|
||||
- No conflict handling (overwrites target)
|
||||
|
||||
**Best for**: Backups, deployments, one-way mirroring. Not for bidirectional sync.
|
||||
|
||||
---
|
||||
|
||||
### rclone (with bisync)
|
||||
|
||||
**What it is**: Cloud storage Swiss Army knife, with experimental bidirectional sync
|
||||
|
||||
**Strengths**:
|
||||
- Supports 70+ cloud backends
|
||||
- `bisync` command provides bidirectional sync (as of v1.66)
|
||||
- Checksum-based comparison available
|
||||
- Lock files prevent concurrent corruption
|
||||
- Great for cloud ↔ local sync
|
||||
|
||||
**Weaknesses**:
|
||||
- **bisync is "advanced"**: Documentation warns of potential data loss
|
||||
- Designed for cloud storage, not LAN sync
|
||||
- More complex ignore pattern syntax than Syncthing
|
||||
- No real-time watching (cron-based)
|
||||
|
||||
**Best for**: Syncing with cloud storage (S3, B2, Drive, etc.), Tier 3 backup
|
||||
|
||||
**Sources**:
|
||||
- [rclone bisync documentation](https://rclone.org/bisync/)
|
||||
|
||||
---
|
||||
|
||||
### Mutagen
|
||||
|
||||
**What it is**: Fast bidirectional sync for remote development
|
||||
|
||||
**Strengths**:
|
||||
- Designed for code sync to remote containers/servers
|
||||
- Very fast (near-native Docker volume performance)
|
||||
- Real-time filesystem watching
|
||||
- Multiple sync modes (two-way-safe, one-way, etc.)
|
||||
- SSH and Docker native
|
||||
|
||||
**Weaknesses**:
|
||||
- **Ephemeral by design**: Not meant for long-term sync
|
||||
- Less configuration flexibility than Syncthing
|
||||
- Primarily for developer workflows
|
||||
|
||||
**Best for**: Remote development, Docker volumes, SSH workspaces. Overkill for notes.
|
||||
|
||||
**Sources**:
|
||||
- [Mutagen GitHub](https://github.com/mutagen-io/mutagen)
|
||||
- [Mutagen Documentation](https://mutagen.io/documentation/synchronization/)
|
||||
|
||||
---
|
||||
|
||||
### nb (Notebook CLI)
|
||||
|
||||
**What it is**: CLI note-taking with built-in git sync
|
||||
|
||||
**Strengths**:
|
||||
- **Git-native**: Every notebook is a git repo
|
||||
- Explicit sync via `nb sync`
|
||||
- Full history, branching, proper merges
|
||||
- Works with any git remote (GitHub, Gitea, Forgejo)
|
||||
- Markdown by default, encryption optional
|
||||
- Single portable shell script
|
||||
- `$EDITOR` integration (neovim-friendly)
|
||||
|
||||
**Weaknesses**:
|
||||
- **Explicit sync only**: Must run `nb sync` (or set up cron/systemd)
|
||||
- Git conflicts require git knowledge to resolve
|
||||
- No GUI (by design)
|
||||
|
||||
**Best for**: Notes and writing. This is the ideal Tier 2 solution for text content.
|
||||
|
||||
**Sources**:
|
||||
- [nb GitHub](https://github.com/xwmx/nb)
|
||||
- [nb Documentation](https://xwmx.github.io/nb/)
|
||||
|
||||
---
|
||||
|
||||
### CRDTs (Automerge, Yjs)
|
||||
|
||||
**What it is**: Conflict-free Replicated Data Types for automatic merging
|
||||
|
||||
**Strengths**:
|
||||
- **No conflicts by design**: Concurrent edits automatically merge
|
||||
- Real-time collaboration possible
|
||||
- Local-first architecture
|
||||
- Automerge 3.0 has 10x lower memory usage
|
||||
|
||||
**Weaknesses**:
|
||||
- **Application-level, not filesystem-level**: Requires app integration
|
||||
- Not a drop-in replacement for file sync
|
||||
- Larger file sizes than plain text
|
||||
- Requires specialized tooling
|
||||
|
||||
**Best for**: Future note-taking apps, real-time collaboration. Not ready for filesystem sync.
|
||||
|
||||
**Sources**:
|
||||
- [Automerge GitHub](https://github.com/automerge/automerge)
|
||||
- [Local, First, Forever](https://tonsky.me/blog/crdt-filesync/)
|
||||
|
||||
---
|
||||
|
||||
## Recommendation Matrix
|
||||
|
||||
| Use Case | Recommended Tool | Reasoning |
|
||||
|----------|------------------|-----------|
|
||||
| **Notes (Markdown)** | `nb` with git | Explicit sync, full history, proper conflict resolution |
|
||||
| **Documents (Office, misc)** | Syncthing OR Unison | Syncthing for "set and forget", Unison for control |
|
||||
| **Code/Dotfiles** | Git + home-manager | Version control is the right tool |
|
||||
| **Photos** | Immich + rclone | Specialized tool for media |
|
||||
| **Large files (video, etc.)** | rclone to B2/S3/NAS | Cloud storage with proper backup |
|
||||
| **Backup** | restic or borg | Deduplication, encryption, versioning |
|
||||
|
||||
---
|
||||
|
||||
## Why "Syncthing is Buggy" is Usually Wrong
|
||||
|
||||
The perception of bugginess typically comes from:
|
||||
|
||||
1. **Expectation mismatch**: Users expect Dropbox behavior (cloud-centric) but get P2P behavior
|
||||
2. **Conflict = failure to them**: Syncthing correctly identifies conflicts; users see this as a bug
|
||||
3. **Android battery drain**: Real issue, but fixed in forks with battery optimization
|
||||
4. **Propagated deletions**: Syncthing does what it's supposed to; users wanted a backup
|
||||
5. **Complex topologies**: 5+ devices with selective sync creates edge cases
|
||||
|
||||
**If you experienced "bugginess" with Syncthing, ask yourself**:
|
||||
- Did you have proper backups outside of Syncthing?
|
||||
- Were multiple devices editing the same files?
|
||||
- Did you understand the difference between "Send Only" / "Receive Only" / "Send & Receive"?
|
||||
- Was your Android device killing the app in the background?
|
||||
|
||||
---
|
||||
|
||||
## Final Recommendation for This Project
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ TIER 1: Configuration │
|
||||
│ Tool: Git (public GitHub repo) │
|
||||
│ NixOS flake + home-manager, sops-nix for secrets │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ TIER 2: Syncable Data │
|
||||
│ │
|
||||
│ Notes & Writing: │
|
||||
│ ├── Tool: nb (with git sync to private Forgejo) │
|
||||
│ ├── Why: Explicit sync, full history, proper merges │
|
||||
│ └── Conflict handling: Git-style (user resolves) │
|
||||
│ │
|
||||
│ Other Documents: │
|
||||
│ ├── Tool: Syncthing (if you want auto-sync) │
|
||||
│ │ OR Unison (if you want manual control) │
|
||||
│ ├── Why: Real-time for docs that change less frequently │
|
||||
│ └── Conflict handling: .sync-conflict files (Syncthing) │
|
||||
│ or interactive (Unison) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ TIER 3: Large Data │
|
||||
│ │
|
||||
│ Photos: Immich (self-hosted) │
|
||||
│ Media: Jellyfin (self-hosted) │
|
||||
│ Repos: Forgejo (self-hosted) │
|
||||
│ Backup: restic → B2/Backblaze/NAS │
|
||||
│ Access: On-demand download via rclone or web UI │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The key insight: **Use git for text, file-sync for binary, and specialized tools for media**.
|
||||
Reference in New Issue
Block a user