99 lines
3.7 KiB
Markdown
99 lines
3.7 KiB
Markdown
# filedb
|
|
|
|
`filedb` is a Go CLI utility for indexing file metadata across multiple libraries and collections.
|
|
|
|
- Library: logical grouping of collections, persisted as its own DuckDB database file.
|
|
- Collection: a backup-like dataset backed by either:
|
|
- `fs`: local filesystem path
|
|
- `rclone`: `rclone lsjson` listing for a remote/path
|
|
|
|
Checks are informational only. The tool does not modify or delete files.
|
|
|
|
## Storage
|
|
|
|
- Default application directory: `~/.local/filedb`
|
|
- Library databases: `~/.local/filedb/libraries/*.duckdb` (one database per library)
|
|
- Override app directory with: `--app-dir <path>`
|
|
|
|
## Features
|
|
|
|
- Create/list libraries
|
|
- Create/list collections inside libraries
|
|
- Delete libraries or collections
|
|
- Move a collection between libraries
|
|
- Update one collection or a full library with one operation (file set + metadata only; no MD5 processing)
|
|
- Update MD5 hashes with dedicated `update-hashes` command
|
|
- Store metadata: size, mod time, create time (if available), MD5 (if possible)
|
|
- Compare indexed metadata with current live files (`compare-live`)
|
|
- Offline duplicate checking inside indexed collection (`compare-offline`)
|
|
- Store reports inside each library DB by default
|
|
- Optional report export to JSON or HTML and optional no-store mode
|
|
|
|
## Build
|
|
|
|
```bash
|
|
go build -o filedb ./cmd/filedb
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
filedb library-create --name archive
|
|
filedb library-delete --name old-archive
|
|
filedb collection-create --library archive --name tape-001 --type fs --source /data/tape-001
|
|
filedb collection-create --library archive --name remote-photos --type rclone --source myremote:photos
|
|
filedb collection-create --library archive --name remote-root --type rclone --source myremote:
|
|
filedb collection-delete --library archive --name tape-001
|
|
filedb collection-move --library archive --name remote-photos --target-library cold-storage
|
|
|
|
filedb update --library archive --collection tape-001
|
|
filedb update --library archive
|
|
|
|
filedb update-hashes --library archive --collection tape-001
|
|
filedb update-hashes --library archive --force-all
|
|
filedb update-hashes --library archive --modified-only
|
|
filedb update-hashes --library archive --collection tape-001 --file path/in/collection/file.iso
|
|
filedb update-hashes --library archive --collection tape-001 --file-list /tmp/files.txt
|
|
|
|
filedb compare-live --library archive --metadata size,md5 --collection tape-001
|
|
filedb compare-live --library archive --metadata size,md5
|
|
filedb compare-offline --library archive --metadata filename,size,md5,mtime --collection tape-001 --collection tape-002
|
|
|
|
filedb compare-offline --library archive --metadata filename,size,md5 --no-store-report --export html --output report.html
|
|
|
|
filedb report-list
|
|
filedb report-list --library archive
|
|
filedb report-show --id <report-id> --library archive
|
|
filedb report-export --id <report-id> --library archive --format json --output report.json
|
|
```
|
|
|
|
With custom app directory:
|
|
|
|
```bash
|
|
filedb --app-dir /tmp/filedb-data library-list
|
|
```
|
|
|
|
## Metadata Fields
|
|
|
|
For comparison keys (`--metadata`):
|
|
|
|
- `filename` (offline only; not allowed in `compare-live`)
|
|
- `size`
|
|
- `mtime`
|
|
- `ctime`
|
|
- `md5`
|
|
|
|
Example:
|
|
|
|
```bash
|
|
filedb compare-offline --library archive --metadata filename,size,mtime
|
|
```
|
|
|
|
## Notes
|
|
|
|
- `update` does not compute hashes. Use `update-hashes` for MD5 processing.
|
|
- `ctime` availability is OS/filesystem dependent.
|
|
- Rclone collections require `rclone` in `PATH` (validated at collection creation and indexing).
|
|
- `collection-create --source` is validated at create time: filesystem sources must be existing directories, and rclone sources must use `remote:` or `remote:path` syntax.
|
|
- `update` skips disconnected/unreachable filesystem sources and rclone indexing failures, and continues with other collections.
|