filedb

filedb is a Go CLI utility for indexing file metadata across multiple libraries and collections.

Library: logical grouping of collections, persisted as its own DuckDB database file.
Collection: a backup-like dataset backed by either:
- fs: local filesystem path
- rclone: rclone lsjson listing for a remote/path

Checks are informational only. The tool does not modify or delete files.

Storage

Default application directory: ~/.local/filedb
Library databases: ~/.local/filedb/libraries/*.duckdb (one database per library)
Override app directory with: --app-dir <path>

Features

Create/list libraries
Create/list collections inside libraries
Update one collection or a full library with one operation (file set + metadata only; no MD5 processing)
Update MD5 hashes with dedicated update-hashes command
Store metadata: size, mod time, create time (if available), MD5 (if possible)
Compare indexed metadata with current live files (compare-live)
Offline duplicate checking inside indexed collection (compare-offline)
Store reports inside each library DB by default
Optional report export to JSON or HTML and optional no-store mode

Build

go build -o filedb ./cmd/filedb

Usage

filedb library-create --name archive
filedb collection-create --library archive --name tape-001 --type fs --source /data/tape-001
filedb collection-create --library archive --name remote-photos --type rclone --source myremote:photos
filedb collection-create --library archive --name remote-root --type rclone --remote myremote
filedb collection-create --library archive --name remote-subdir --type rclone --remote myremote --remote-path photos/2024

filedb update --library archive --collection tape-001
filedb update --library archive

filedb update-hashes --library archive --collection tape-001
filedb update-hashes --library archive --force-all
filedb update-hashes --library archive --modified-only
filedb update-hashes --library archive --collection tape-001 --file path/in/collection/file.iso
filedb update-hashes --library archive --collection tape-001 --file-list /tmp/files.txt

filedb compare-live --library archive --metadata size,md5 --collection tape-001
filedb compare-live --library archive --metadata size,md5
filedb compare-offline --library archive --metadata filename,size,md5,mtime --collection tape-001 --collection tape-002

filedb compare-offline --library archive --metadata filename,size,md5 --no-store-report --export html --output report.html

filedb report-list
filedb report-list --library archive
filedb report-show --id <report-id> --library archive
filedb report-export --id <report-id> --library archive --format json --output report.json

With custom app directory:

filedb --app-dir /tmp/filedb-data library-list

Metadata Fields

For comparison keys (--metadata):

filename (offline only; not allowed in compare-live)
size
mtime
ctime
md5

Example:

filedb compare-offline --library archive --metadata filename,size,mtime

Notes

update does not compute hashes. Use update-hashes for MD5 processing.
ctime availability is OS/filesystem dependent.
Rclone collections require rclone in PATH (validated at collection creation and indexing).
For rclone collections use either --source <remote-or-remote:path> or --remote <remote> [--remote-path <path>].
update skips disconnected/unreachable filesystem sources and rclone indexing failures, and continues with other collections.