# filedb

`filedb` is a Go CLI utility for indexing file metadata across multiple libraries and collections.

- Library: logical grouping of collections, persisted as its own DuckDB database file.
- Collection: a backup-like dataset backed by either:
  - `fs`: local filesystem path
  - `rclone`: `rclone lsjson` listing for a remote/path

Checks are informational only. The tool does not modify or delete files.

## Storage

- Default application directory: `~/.local/filedb`
- Library databases: `~/.local/filedb/libraries/*.duckdb` (one database per library)
- Override app directory with: `--app-dir <path>`

## Features

- Create/list libraries
- Create/list collections inside libraries
- Update one collection or a full library with one operation (file set + metadata only; no MD5 processing)
- Update MD5 hashes with dedicated `update-hashes` command
- Store metadata: size, mod time, create time (if available), MD5 (if possible)
- Compare indexed metadata with current live files (`compare-live`)
- Offline duplicate checking inside indexed collection (`compare-offline`)
- Store reports inside each library DB by default
- Optional report export to JSON or HTML and optional no-store mode

## Build

```bash
go build -o filedb ./cmd/filedb
```

## Usage

```bash
filedb library-create --name archive
filedb collection-create --library archive --name tape-001 --type fs --source /data/tape-001
filedb collection-create --library archive --name remote-photos --type rclone --source myremote:photos
filedb collection-create --library archive --name remote-root --type rclone --remote myremote
filedb collection-create --library archive --name remote-subdir --type rclone --remote myremote --remote-path photos/2024

filedb update --library archive --collection tape-001
filedb update --library archive

filedb update-hashes --library archive --collection tape-001
filedb update-hashes --library archive --force-all
filedb update-hashes --library archive --modified-only
filedb update-hashes --library archive --collection tape-001 --file path/in/collection/file.iso
filedb update-hashes --library archive --collection tape-001 --file-list /tmp/files.txt

filedb compare-live --library archive --metadata size,md5 --collection tape-001
filedb compare-live --library archive --metadata size,md5
filedb compare-offline --library archive --metadata filename,size,md5,mtime --collection tape-001 --collection tape-002

filedb compare-offline --library archive --metadata filename,size,md5 --no-store-report --export html --output report.html

filedb report-list
filedb report-list --library archive
filedb report-show --id <report-id> --library archive
filedb report-export --id <report-id> --library archive --format json --output report.json
```

With custom app directory:

```bash
filedb --app-dir /tmp/filedb-data library-list
```

## Metadata Fields

For comparison keys (`--metadata`):

- `filename` (offline only; not allowed in `compare-live`)
- `size`
- `mtime`
- `ctime`
- `md5`

Example:

```bash
filedb compare-offline --library archive --metadata filename,size,mtime
```

## Notes

- `update` does not compute hashes. Use `update-hashes` for MD5 processing.
- `ctime` availability is OS/filesystem dependent.
- Rclone collections require `rclone` in `PATH` (validated at collection creation and indexing).
- For rclone collections use either `--source <remote-or-remote:path>` or `--remote <remote> [--remote-path <path>]`.
- `update` skips disconnected/unreachable filesystem sources and rclone indexing failures, and continues with other collections.