commit fdcddc81492c13e3daf52cfc3e1fbc11218ac81f Author: Jess Hallsworth Date: Mon Sep 7 03:41:38 2026 +0000 Initial design for MediaShelf Software design for a Plex library analytics and reclaim-reporting tool. v1 is report-only: no deletion, no filesystem access, Plex API as the sole data source. Movies at item level, TV rolled up to season level. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GVbG48GAXfCZatcmX123Ra diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..1b8cdca --- /dev/null +++ b/.env.example @@ -0,0 +1,30 @@ +# MediaShelf configuration — copy to .env and fill in. +# NEVER commit .env. PLEX_TOKEN is a secret. + +MEDIASHELF_SECRET_KEY=change-me +LOG_LEVEL=INFO +TZ=America/Regina + +# --- Plex connection --- +PLEX_BASE_URL=http://192.168.1.10:32400 +PLEX_TOKEN=your-plex-token-here +PLEX_VERIFY_SSL=true +PLEX_TIMEOUT_S=30 +PLEX_PAGE_SIZE=500 +PLEX_REQUEST_DELAY_MS=0 + +# --- Storage --- +DATABASE_PATH=/data/mediashelf.db + +# --- Scanning --- +SCAN_SCHEDULE_CRON=0 4 * * * +SCAN_FULL_SWEEP_CRON=0 3 * * 0 +SCAN_ON_STARTUP=false +SCAN_LOCK_TIMEOUT_S=7200 + +# --- Reclaim score tuning --- +SCORE_STALE_HORIZON_DAYS=730 +SCORE_AGE_HORIZON_DAYS=1095 +SCORE_POPULAR_AT=3 +SCORE_GRACE_DAYS=30 +SCORE_RECENT_DAYS=90 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..802eaf7 --- /dev/null +++ b/.gitignore @@ -0,0 +1,16 @@ +.env +*.db +*.db-wal +*.db-shm +__pycache__/ +*.py[cod] +.venv/ +venv/ +.pytest_cache/ +.coverage +htmlcov/ +dist/ +build/ +*.egg-info/ +data/ +.DS_Store diff --git a/README.md b/README.md new file mode 100644 index 0000000..940c908 --- /dev/null +++ b/README.md @@ -0,0 +1,47 @@ +# MediaShelf + +A self-hosted web app for figuring out which of the thousands of files in a Plex library +are actually worth keeping. + +MediaShelf scans a Plex Media Server over its HTTP API, builds a local snapshot of every +movie and TV season, and joins together the facts Plex already knows but never shows side +by side — date added, size on disk, file path, owning library, watch count, and last +watched — into a sortable, filterable, chartable grid with a tunable **reclaim score** +that ranks deletion candidates. + +## Status + +**v1 is report-only.** MediaShelf does not delete, move, or modify anything. It produces +a ranked list, saved rule sets, and CSV export. Deletion is designed for in the roadmap +but deliberately not built, so the scanner and the scoring model can be trusted before +anything destructive is wired up. + +Nothing is implemented yet — this repository currently holds the design. + +## What it does + +- Full-library ingest from Plex, no agent on the Plex host, no filesystem mounts +- Movies at item level, TV rolled up to **season** level +- Watch data pulled from the server-wide playback history, so plays by *every* Plex + account are counted — not just the token owner's +- Sort and filter on every metric; charts for size by library, additions over time, + watched vs. unwatched by size, and size vs. last-watched +- A weighted reclaim score with live sliders, and grace rules so it never recommends + something you added last week +- Named, re-runnable saved views — *"unwatched, older than 2 years, over 10 GB"* +- CSV export of any view + +## Planned stack + +Python + Flask, SQLite (WAL), vanilla JS front-end, single container deployed as a +Portainer stack behind Nginx Proxy Manager. + +## Roadmap + +- **v2** — two-stage quarantine-then-purge deletion, with authentication, a path + allowlist, and an audit log +- **v3** — Emby and Jellyfin support behind the existing `MediaProvider` abstraction + +## Documentation + +- [`docs/design.md`](docs/design.md) — the full software design diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..e9d9560 --- /dev/null +++ b/docs/design.md @@ -0,0 +1,1012 @@ +# MediaShelf — Software Design + +**Status:** Draft v1 · **Author:** Claude (spec) for Jess · **Date:** 2026-09-07 + +--- + +## 1. Overview + +MediaShelf is a self-hosted web application that reads a Plex Media Server library, +builds a local analytical snapshot of every item in it, and gives a human a fast way +to answer one question: + +> *Which of these thousands of files are worth keeping, and which are burning disk +> for nothing?* + +It does this by joining together facts Plex already knows but never shows side by +side — when a file was added, how large it is, where it lives on disk, which library +it belongs to, how many times it has been played, and when it was last played — and +exposing them as a sortable, filterable, chartable grid with a tunable "reclaim +score" that ranks deletion candidates. + +**v1 is read-only.** MediaShelf will never delete, move, or modify a file or a Plex +record in this version. Its output is a report: an on-screen ranked list, saved rule +sets, and CSV export. Actual deletion is designed for in §14 but deliberately not +built yet, so the scanner and scoring model can be trusted before anything +destructive is wired up. + +### 1.1 Goals + +- Full-library ingest from Plex over its HTTP API, with no agent installed on the Plex host. +- A durable local snapshot so the UI is instant and Plex isn't hammered on every click. +- Movies at item level; TV rolled up to **season** level. +- Sort/filter/visualize on: date added, last watched, watch count, size on disk, file path, library. +- A weighted **reclaim score** with live sliders, so "what should I delete" is one sort click. +- **Saved views** — named, re-runnable rule sets like *unwatched, older than 2 years, over 10 GB*. +- CSV export of any view. +- A provider abstraction clean enough that Emby and Jellyfin drop in later without + touching the scoring, storage, or UI layers. + +### 1.2 Non-goals (v1) + +- Deleting, moving, or trashing files. Not in v1. See §14.1. +- Writing anything back to Plex (no rating changes, no collection edits, no unmatch/refresh). +- Authentication or multi-user accounts — this is a LAN-only, single-admin tool. See §12. +- Music, photo, and "other video" libraries. Movie and Show sections only. +- Direct filesystem access. MediaShelf never mounts the media shares. All size and path + data comes from the Plex API. See §4.4. +- Transcode/quality analysis, duplicate detection, subtitle management. + +### 1.3 Decisions already made + +| Question | Decision | +|---|---| +| Deletion in v1 | No — report only | +| Stack | Python + Flask, SQLite | +| Deployment | Portainer stack on **Nox** (192.168.1.77), published via Nginx Proxy Manager | +| Auth | None; LAN-only, single admin view, watch data aggregated across all Plex accounts | +| TV granularity | Season | +| Size/path source | Plex API only | +| v1 output | Sortable grid + filters + charts, tunable reclaim score, saved rule sets | +| Implementation | Design doc only for now; code to follow | + +--- + +## 2. Context + +The Plex server is **Loki** (192.168.1.10, Dell R710, Ubuntu 16.04) with roughly 90 TB +of RAID-6 storage across a Dell MD1000 vault. A second vault is powered down awaiting +drive upgrades, so reclaiming space on the live vault has real value — this tool exists +because buying disks is the more expensive alternative. + +MediaShelf runs on **Nox**, not on Loki. Loki's job is serving media; MediaShelf is a +polite HTTP client that talks to it across the LAN and keeps its own database locally. +This also means MediaShelf survives Loki reboots, Plex upgrades, and the eventual +migration of services onto Enki. + +--- + +## 3. Architecture + +``` +┌──────────────────────────── Nox (192.168.1.77) ────────────────────────────┐ +│ │ +│ ┌─────────────────── mediashelf container ───────────────────┐ │ +│ │ │ │ +│ │ Flask app (gunicorn, 2 workers) │ │ +│ │ ├── / → UI (server-rendered shell + JS) │ │ +│ │ ├── /api/* → JSON API │ │ +│ │ └── /healthz │ │ +│ │ │ │ +│ │ APScheduler (in-process, 1 worker only) │ │ +│ │ └── nightly scan job │ │ +│ │ │ │ +│ │ Ingestion pipeline │ │ +│ │ PlexProvider ──▶ normalize ──▶ upsert ──▶ rollup │ │ +│ │ │ │ +│ │ SQLite /data/mediashelf.db (WAL) │ │ +│ └────────────────────────────┬───────────────────────────────┘ │ +│ │ volume: /data │ +└────────────────────────────────┼───────────────────────────────────────────┘ + │ HTTP :32400 + X-Plex-Token + ▼ + Loki (192.168.1.10) — Plex Media Server +``` + +### 3.1 Component responsibilities + +**`mediashelf.providers`** — the pluggable source layer. Defines the `MediaProvider` +protocol and ships one implementation, `PlexProvider`. Everything above this layer +speaks only in MediaShelf's own normalized dataclasses, never in Plex XML. This is the +single seam that Emby/Jellyfin support slots into (§14.2). + +**`mediashelf.ingest`** — orchestrates a scan: pull sections, pull items, pull history, +normalize, upsert into SQLite inside a transaction, recompute season rollups, mark +vanished items, write a `scan` row. + +**`mediashelf.scoring`** — pure functions. Takes a row of metrics plus a weight profile, +returns a 0–100 reclaim score and the component breakdown. No I/O, fully unit-testable. + +**`mediashelf.rules`** — evaluates saved-view rule sets into SQL `WHERE` fragments via a +whitelisted field/operator grammar. Never string-interpolates user input into SQL. + +**`mediashelf.api`** — Flask blueprints returning JSON. + +**`mediashelf.web`** — one server-rendered HTML shell; the grid, charts, and sliders are +client-side JS against `/api`. + +### 3.2 Why Flask + SQLite + +The dataset is small in database terms. A library of ~5,000 movies and ~2,000 seasons +across ~60,000 episodes is a few tens of megabytes. SQLite in WAL mode handles this with +sub-10 ms queries on indexed columns, needs no second container, and backs up by copying +one file. Flask matches the pattern already running for the Mythica portal on Nox, so +deployment, logging, and reverse-proxy conventions carry over unchanged. + +The one real constraint SQLite imposes is a single writer. Since the only writer is the +scan job, and scans run one at a time behind a lock, this is not a limitation. It does +mean **gunicorn must run the scheduler in exactly one worker** — see §11.3. + +--- + +## 4. Plex integration + +### 4.1 Connection + +MediaShelf authenticates with a Plex server token (`X-Plex-Token`) supplied via +environment variable. It talks directly to `http://192.168.1.10:32400` on the LAN — no +plex.tv round trip, no account login, no dependency on Plex's cloud being up. + +All requests carry standard client identification headers so the connection is +identifiable in Plex's own logs: + +``` +X-Plex-Token: +X-Plex-Client-Identifier: mediashelf- +X-Plex-Product: MediaShelf +X-Plex-Version: +Accept: application/json +``` + +Plex returns JSON when `Accept: application/json` is set, which avoids XML parsing +entirely. Every response body is wrapped in a `MediaContainer` object. + +### 4.2 Endpoints used + +| Purpose | Endpoint | +|---|---| +| Server identity / version | `GET /identity` | +| List libraries | `GET /library/sections` | +| Library detail + on-disk locations | `GET /library/sections/{id}` | +| All movies in a section | `GET /library/sections/{id}/all?type=1` | +| All episodes in a show section | `GET /library/sections/{id}/all?type=4` | +| All seasons in a show section | `GET /library/sections/{id}/all?type=3` | +| Global watch history | `GET /status/sessions/history/all` | +| Accounts (for history attribution) | `GET /accounts` | + +Pagination on every list endpoint uses `X-Plex-Container-Start` and +`X-Plex-Container-Size` headers (or the equivalent query params). MediaShelf pages at +**500 items** per request; the `MediaContainer.totalSize` field on the first response +gives the loop its bound. + +### 4.3 The fields that matter + +From each `Video` element on `/library/sections/{id}/all`: + +| Plex attribute | Meaning | MediaShelf column | +|---|---|---| +| `ratingKey` | stable per-server item id | `provider_item_id` | +| `guid` | cross-server content identity | `guid` | +| `type` | `movie` / `episode` / `season` / `show` | `kind` | +| `title`, `year` | display | `title`, `year` | +| `addedAt` | epoch seconds, when Plex first saw the file | `added_at` | +| `updatedAt` | epoch seconds, last metadata change | `updated_at` | +| `duration` | milliseconds | `duration_ms` | +| `viewCount` | plays — **token-account scoped**, see §4.5 | (advisory only) | +| `lastViewedAt` | epoch seconds — token-account scoped | (advisory only) | +| `librarySectionID` | owning library | `library_id` | +| `grandparentRatingKey`, `grandparentTitle` | (episodes) the show | `show_id`, `show_title` | +| `parentRatingKey`, `parentIndex` | (episodes) the season and its number | `season_id`, `season_number` | + +From the nested `Media[].Part[]` elements: + +| Plex attribute | MediaShelf column | +|---|---| +| `Part.file` | `file_path` — absolute path on the Plex host | +| `Part.size` | `size_bytes` | +| `Part.container` | `container` | +| `Media.videoResolution`, `videoCodec`, `audioCodec`, `bitrate` | `resolution`, `video_codec`, `audio_codec`, `bitrate` | + +An item can have several `Media` entries (multiple versions) each with several `Part` +entries (split files). MediaShelf stores **every part as its own row** in `media_part` +and treats an item's size as the sum of its parts. This matters: a movie with a 4K and a +1080p version is one item occupying two files, and a reclaim report that reported only +the first part would understate the win. + +### 4.4 Why Plex-only, and what it costs + +MediaShelf does not mount the media shares. That keeps the container trivially portable, +removes any possibility of it touching a file by accident, and means it works from +anywhere that can reach port 32400. + +The accepted trade-offs: + +- **Size is Plex's number, not the filesystem's.** Plex records `Part.size` at scan + time. If a file is replaced outside Plex and Plex hasn't rescanned, the number is + stale. In practice Plex's own scanner keeps this current; the design accepts drift. +- **Orphans are invisible.** A file on disk that Plex has never indexed will not appear + in MediaShelf at all. Finding those needs a filesystem walk, which is explicitly out + of scope (see §14.3 for the optional-mount future). +- **Paths are Loki-relative.** `file_path` is the path as the Plex server sees it. It is + displayed and exported for a human's benefit, not resolved by MediaShelf. + +### 4.5 Watch data: the important subtlety + +`viewCount` and `lastViewedAt` on a library item are **scoped to the account that owns +the token**. If Jess's token is used, plays by home users, managed users, and shared +users are not counted. For a household library this understates watches badly, and a +reclaim tool that deletes "unwatched" content someone else has been watching is worse +than useless. + +So MediaShelf treats `/status/sessions/history/all` as the source of truth for watch +data. That endpoint returns one row per playback event, server-wide, across all +accounts, each with: + +- `historyKey`, `ratingKey`, `librarySectionID` +- `viewedAt` (epoch seconds) +- `accountID` (which Plex user) +- `type`, `title`, and for episodes the parent/grandparent identifiers + +MediaShelf ingests this into a `watch_event` table and derives, per item: + +- `watch_count` — number of distinct playback events +- `last_watched_at` — max `viewedAt` +- `first_watched_at` — min `viewedAt` +- `distinct_watcher_count` — count of distinct `accountID` + +The single-admin UI shows the aggregate. The per-account data is stored anyway, because +it's free to keep and it's what the "who actually watches this" question will need later. + +**Caveat to verify during the first spike:** Plex prunes session history according to the +server's *Media Playback → Settings → Empty trash / history retention* configuration, and +some servers cap it. If Loki's retention window turns out to be shorter than the useful +analysis window, MediaShelf's own `watch_event` table becomes the long-term record — +which it already is, since it accumulates across scans and never deletes rows. The first +full ingest establishes the baseline; every scan after that only needs events newer than +the last one seen. + +The item-level `viewCount` is still stored as an advisory field, and the UI shows a small +warning badge when `viewCount > 0` but no history events exist for that item — that's the +signature of a watch that happened before history retention began. + +### 4.6 Scan strategy + +**Full scan** — walks every section, every item, every part, and the complete history. +Run on first launch and on demand. Expect a few minutes for a large library. + +**Incremental scan** — the nightly default. Two cheap tricks: + +1. Items: request the section sorted by `addedAt:desc` and stop paging once the page's + oldest `addedAt` predates the last successful scan. This catches additions. To catch + *removals* and metadata edits, a full item sweep still runs weekly (configurable) — + an item silently vanishing from Plex is exactly the event that must not be missed. +2. History: request `/status/sessions/history/all` sorted by `viewedAt:desc` and stop at + the last-seen `viewedAt` watermark. + +**Concurrency** — one scan at a time, enforced by a row in a `scan_lock` table plus an +in-process lock. A scan that dies leaves a stale lock; locks older than +`SCAN_LOCK_TIMEOUT` (default 2 h) are broken automatically and the orphaned scan row is +marked `failed`. + +**Politeness** — requests are issued serially with a small configurable delay +(`PLEX_REQUEST_DELAY_MS`, default 0) and a hard timeout. Loki is a 2010-vintage R710; +the ingest is designed to be slow and steady rather than parallel and aggressive. + +### 4.7 Failure handling + +| Failure | Behaviour | +|---|---| +| Plex unreachable | Scan marked `failed` with the error; UI keeps serving the last good snapshot with a staleness banner | +| Token rejected (401) | Scan fails fast, prominent UI banner, no retry loop | +| Section disappears | Its items are marked `missing` rather than deleted (§5.4) | +| Malformed / partial item | Item skipped, logged, counted in `scan.warning_count`; the scan still succeeds | +| Timeout mid-page | Up to 3 retries with exponential backoff, then the scan fails | + +--- + +## 5. Data model + +SQLite, WAL mode, foreign keys on. Migrations are plain numbered SQL files applied at +startup, tracked in a `schema_version` table — no ORM migration framework. + +### 5.1 Tables + +```sql +-- One row per configured media server. v1 has exactly one, but the column exists +-- so Emby/Jellyfin can be added without a migration. +CREATE TABLE provider ( + id INTEGER PRIMARY KEY, + kind TEXT NOT NULL, -- 'plex' | 'emby' | 'jellyfin' + name TEXT NOT NULL, + base_url TEXT NOT NULL, + server_id TEXT, -- Plex machineIdentifier + version TEXT, + last_scan_id INTEGER, + created_at INTEGER NOT NULL +); + +CREATE TABLE library ( + id INTEGER PRIMARY KEY, + provider_id INTEGER NOT NULL REFERENCES provider(id), + provider_key TEXT NOT NULL, -- Plex section id + title TEXT NOT NULL, + kind TEXT NOT NULL, -- 'movie' | 'show' + locations TEXT, -- JSON array of on-disk roots + scanned_at INTEGER, + UNIQUE (provider_id, provider_key) +); + +-- The unit of analysis. One row per movie, and one row per SEASON. +-- Shows themselves get a row too (kind='show') for grouping and display, +-- but carry no size of their own. +CREATE TABLE media_item ( + id INTEGER PRIMARY KEY, + provider_id INTEGER NOT NULL REFERENCES provider(id), + library_id INTEGER NOT NULL REFERENCES library(id), + provider_item_id TEXT NOT NULL, -- Plex ratingKey + kind TEXT NOT NULL, -- 'movie' | 'show' | 'season' + guid TEXT, + title TEXT NOT NULL, + sort_title TEXT, + year INTEGER, + -- season/show linkage + parent_id INTEGER REFERENCES media_item(id), -- season -> show + season_number INTEGER, + -- provenance + added_at INTEGER, -- epoch s; for a season, MIN over episodes + updated_at INTEGER, + -- rollups (denormalized, recomputed each scan) + episode_count INTEGER NOT NULL DEFAULT 0, + size_bytes INTEGER NOT NULL DEFAULT 0, + duration_ms INTEGER NOT NULL DEFAULT 0, + part_count INTEGER NOT NULL DEFAULT 0, + primary_path TEXT, -- representative file path for display + resolution TEXT, + video_codec TEXT, + -- watch rollups (from watch_event) + watch_count INTEGER NOT NULL DEFAULT 0, + last_watched_at INTEGER, + first_watched_at INTEGER, + distinct_watcher_count INTEGER NOT NULL DEFAULT 0, + provider_view_count INTEGER NOT NULL DEFAULT 0, -- advisory, token-scoped + -- lifecycle + status TEXT NOT NULL DEFAULT 'present', -- 'present' | 'missing' + first_seen_scan_id INTEGER, + last_seen_scan_id INTEGER, + UNIQUE (provider_id, provider_item_id) +); + +-- Episodes are ingested (they carry the size and the history), but they are NOT +-- the unit of analysis. They exist so seasons can be rolled up accurately and so +-- a season can be expanded in the UI. +CREATE TABLE episode ( + id INTEGER PRIMARY KEY, + season_item_id INTEGER NOT NULL REFERENCES media_item(id) ON DELETE CASCADE, + provider_item_id TEXT NOT NULL, + episode_number INTEGER, + title TEXT, + added_at INTEGER, + duration_ms INTEGER, + size_bytes INTEGER NOT NULL DEFAULT 0, + watch_count INTEGER NOT NULL DEFAULT 0, + last_watched_at INTEGER, + status TEXT NOT NULL DEFAULT 'present', + UNIQUE (provider_item_id) +); + +-- Every physical file. A movie with two versions has two rows. +CREATE TABLE media_part ( + id INTEGER PRIMARY KEY, + media_item_id INTEGER REFERENCES media_item(id) ON DELETE CASCADE, + episode_id INTEGER REFERENCES episode(id) ON DELETE CASCADE, + provider_part_id TEXT, + file_path TEXT NOT NULL, + size_bytes INTEGER NOT NULL DEFAULT 0, + container TEXT, + resolution TEXT, + video_codec TEXT, + audio_codec TEXT, + bitrate INTEGER, + CHECK ((media_item_id IS NULL) != (episode_id IS NULL)) +); + +-- One row per playback event, server-wide, all accounts. +CREATE TABLE watch_event ( + id INTEGER PRIMARY KEY, + provider_id INTEGER NOT NULL REFERENCES provider(id), + history_key TEXT NOT NULL, + provider_item_id TEXT NOT NULL, -- the leaf: movie or episode ratingKey + account_id TEXT, + viewed_at INTEGER NOT NULL, + UNIQUE (provider_id, history_key) +); + +CREATE TABLE account ( + id INTEGER PRIMARY KEY, + provider_id INTEGER NOT NULL REFERENCES provider(id), + account_id TEXT NOT NULL, + name TEXT, + UNIQUE (provider_id, account_id) +); + +CREATE TABLE scan ( + id INTEGER PRIMARY KEY, + provider_id INTEGER NOT NULL REFERENCES provider(id), + mode TEXT NOT NULL, -- 'full' | 'incremental' + trigger TEXT NOT NULL, -- 'manual' | 'schedule' | 'startup' + status TEXT NOT NULL, -- 'running' | 'succeeded' | 'failed' + started_at INTEGER NOT NULL, + finished_at INTEGER, + items_seen INTEGER DEFAULT 0, + items_added INTEGER DEFAULT 0, + items_updated INTEGER DEFAULT 0, + items_missing INTEGER DEFAULT 0, + events_added INTEGER DEFAULT 0, + warning_count INTEGER DEFAULT 0, + error TEXT +); + +CREATE TABLE saved_view ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + description TEXT, + rules TEXT NOT NULL, -- JSON rule group, see §7 + sort TEXT, -- JSON: [{field, dir}] + columns TEXT, -- JSON array of column keys + weights TEXT, -- JSON weight profile, see §6 + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL +); + +CREATE TABLE setting ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +); +``` + +### 5.2 Indexes + +```sql +CREATE INDEX ix_item_library ON media_item(library_id, kind, status); +CREATE INDEX ix_item_added ON media_item(added_at); +CREATE INDEX ix_item_lastwatch ON media_item(last_watched_at); +CREATE INDEX ix_item_size ON media_item(size_bytes); +CREATE INDEX ix_item_watchcount ON media_item(watch_count); +CREATE INDEX ix_item_parent ON media_item(parent_id); +CREATE INDEX ix_episode_season ON episode(season_item_id); +CREATE INDEX ix_part_item ON media_part(media_item_id); +CREATE INDEX ix_part_episode ON media_part(episode_id); +CREATE INDEX ix_event_item ON watch_event(provider_item_id); +CREATE INDEX ix_event_viewed ON watch_event(viewed_at); +``` + +A full-text index over `title` uses SQLite FTS5 (`media_item_fts`) kept in sync by +triggers, so the grid's search box is instant even at 100k rows. + +### 5.3 Season rollup rules + +After items and history are ingested, a rollup pass recomputes each season row: + +| Season field | Rule | +|---|---| +| `size_bytes` | `SUM(episode.size_bytes)` | +| `episode_count` | `COUNT(episode)` | +| `duration_ms` | `SUM(episode.duration_ms)` | +| `added_at` | `MIN(episode.added_at)` — when the season first landed | +| `watch_count` | `SUM(episode.watch_count)` | +| `last_watched_at` | `MAX(episode.last_watched_at)` | +| `primary_path` | the common directory prefix of its episodes' paths | + +A show row rolls up the same way from its seasons, for display and for the "delete the +whole show" case. + +The `watch_count` choice deserves a note: summing episode plays means a 24-episode season +watched once has `watch_count = 24`, while a movie watched once has `watch_count = 1`. +Comparing those raw numbers across kinds is meaningless. So the scoring model +(§6) uses a **normalized completion figure** for TV — `watch_count / episode_count` — +rather than the raw sum, and the grid displays both (`24 plays · 1.0× season`). + +### 5.4 Missing items + +Nothing is ever hard-deleted by a scan. When a full sweep completes and an item that was +present is no longer returned by Plex, its `status` flips to `missing` and +`last_seen_scan_id` freezes. Missing items are hidden from the default grid but remain +queryable — they're the record of what left the library and when, which is exactly what +you want the first time something disappears unexpectedly. + +--- + +## 6. The reclaim score + +The score answers: *how much do I gain by deleting this, weighed against how likely I am +to want it?* It is a weighted sum of normalized components, each in `[0, 1]`, scaled to +0–100. Higher means "better candidate for deletion". + +### 6.1 Components + +| Component | Definition | Rationale | +|---|---|---| +| **`size`** | `log10(1 + size_bytes) / log10(1 + max_size_in_library)` | Log scale — the difference between 2 GB and 4 GB matters more than between 60 GB and 62 GB | +| **`staleness`** | `min(days_since_last_watch / STALE_HORIZON, 1)`, where a never-watched item scores `1.0` | How long since anyone cared | +| **`unpopularity`** | `1 - min(normalized_watches / POPULAR_AT, 1)` where `normalized_watches` = `watch_count` for movies, `watch_count / episode_count` for seasons | Rewatched things are keepers | +| **`age`** | `min(days_since_added / AGE_HORIZON, 1)` | Something added last week deserves a chance | +| **`solitude`** | `1 - min(distinct_watcher_count / 3, 1)` | Content only one household member ever touched is safer to cut | + +`STALE_HORIZON` defaults to 730 days, `AGE_HORIZON` to 1095 days, `POPULAR_AT` to 3. + +### 6.2 The formula + +``` +score = 100 × ( w_size · size + + w_staleness · staleness + + w_unpopularity· unpopularity + + w_age · age + + w_solitude · solitude ) + / (w_size + w_staleness + w_unpopularity + w_age + w_solitude) +``` + +Default weight profile: + +```json +{ "size": 0.30, "staleness": 0.30, "unpopularity": 0.25, "age": 0.10, "solitude": 0.05 } +``` + +### 6.3 The grace rule + +Two hard overrides, applied after scoring, because a purely numeric model will +eventually recommend something obviously wrong: + +- **New-arrival grace** — an item with `added_at` inside `GRACE_DAYS` (default 30) is + clamped to score 0 and flagged `grace: new`. You haven't had a chance to watch it yet. +- **Recent-watch grace** — an item watched inside `RECENT_DAYS` (default 90) is clamped + to a maximum of 25 and flagged `grace: recent`. + +Both are toggleable per view, and the UI shows why an item was clamped rather than +silently hiding it. + +### 6.4 Where it's computed + +In SQL, as a generated expression in the query, not as a stored column. Weights change +every time a slider moves; storing the score would mean recomputing 100k rows per drag. +The normalization constants (`max_size_in_library`, horizons) are computed once per +request in a CTE. At this row count the whole thing runs in single-digit milliseconds. + +The same function is implemented in Python in `mediashelf.scoring` for unit tests and CSV +export, with a property test asserting the SQL and Python implementations agree to within +0.01 on a generated corpus. Two implementations of the same formula is a real risk, and +that test is the mitigation. + +--- + +## 7. Saved views and rules + +A saved view bundles four things: a rule set, a sort, a column selection, and a weight +profile. Naming one — *"Big unwatched movies"* — makes it a one-click, re-runnable +report that re-evaluates against fresh data every time it's opened. + +### 7.1 Rule grammar + +Rules are a nested JSON group, evaluated into parameterized SQL: + +```json +{ + "op": "and", + "rules": [ + { "field": "kind", "op": "in", "value": ["movie"] }, + { "field": "size_bytes", "op": "gte", "value": 10737418240 }, + { "field": "watch_count", "op": "eq", "value": 0 }, + { "field": "added_at", "op": "older_than_days", "value": 730 }, + { "op": "or", "rules": [ + { "field": "library_id", "op": "in", "value": [1, 4] }, + { "field": "resolution", "op": "eq", "value": "4k" } + ]} + ] +} +``` + +Fields are drawn from a whitelist mapping field name → column and type. Operators are a +fixed set: `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `in`, `not_in`, `contains`, +`starts_with`, `is_null`, `is_not_null`, `older_than_days`, `newer_than_days`, +`never` (for watch fields). Anything outside the whitelist is a 400. Values are always +bound parameters. There is no SQL string interpolation anywhere in this path. + +### 7.2 Seed views shipped by default + +| Name | Rule | +|---|---| +| Never watched, large | `watch_count = 0 AND size_bytes >= 8 GB` | +| Cold storage | `last_watched_at older than 3 years` | +| One-and-done movies | `kind = movie AND watch_count = 1 AND last_watched_at older than 2 years` | +| Abandoned seasons | `kind = season AND watch_count/episode_count < 0.3 AND added_at older than 1 year` | +| Biggest 100 | sorted by `size_bytes desc`, no filter | +| Recently added | `added_at newer than 30 days` — the sanity check, not a delete list | + +--- + +## 8. HTTP API + +All JSON, all under `/api/v1`. No auth in v1 (§12). + +### 8.1 Library data + +``` +GET /api/v1/items +``` + +Query parameters: + +| Param | Meaning | +|---|---| +| `library_id` | repeatable filter | +| `kind` | `movie` / `season` / `show` | +| `q` | FTS search over title | +| `rules` | URL-encoded JSON rule group (§7.1) | +| `view_id` | apply a saved view instead of ad-hoc rules | +| `sort` | e.g. `reclaim_score:desc`, `size_bytes:desc` | +| `weights` | URL-encoded JSON weight profile | +| `include_missing` | default `false` | +| `page`, `page_size` | page_size max 500 | + +Response: + +```json +{ + "total": 4821, + "page": 1, + "page_size": 100, + "aggregate": { "total_size_bytes": 41234567890123, "item_count": 4821 }, + "items": [ + { + "id": 1183, + "kind": "movie", + "title": "Example Film", + "year": 2011, + "library": { "id": 1, "title": "Movies" }, + "size_bytes": 32212254720, + "added_at": 1490000000, + "last_watched_at": null, + "watch_count": 0, + "distinct_watcher_count": 0, + "episode_count": null, + "primary_path": "/mnt/vault2/movies/Example Film (2011)/Example Film (2011).mkv", + "part_count": 1, + "resolution": "4k", + "reclaim_score": 91.4, + "reclaim_components": { + "size": 0.97, "staleness": 1.0, "unpopularity": 1.0, "age": 1.0, "solitude": 1.0 + }, + "grace": null, + "flags": ["history_gap"] + } + ] +} +``` + +Other endpoints: + +``` +GET /api/v1/items/{id} full detail incl. every part and, for seasons, episodes +GET /api/v1/libraries list with item counts and total size +GET /api/v1/accounts Plex accounts seen in history +GET /api/v1/stats/overview headline numbers for the dashboard +GET /api/v1/stats/size-by-library +GET /api/v1/stats/added-over-time?bucket=month +GET /api/v1/stats/watch-distribution +GET /api/v1/stats/size-vs-lastwatched scatter data, downsampled server-side +``` + +### 8.2 Saved views + +``` +GET /api/v1/views +POST /api/v1/views +GET /api/v1/views/{id} +PUT /api/v1/views/{id} +DELETE /api/v1/views/{id} +``` + +### 8.3 Scans + +``` +GET /api/v1/scans history, newest first +GET /api/v1/scans/current null or the running scan with progress +POST /api/v1/scans { "mode": "full" | "incremental" } → 202 or 409 if one is running +``` + +Scan progress is polled at `/api/v1/scans/current` every 2 s while one is running. No +websockets — polling one row is cheaper than the complexity. + +### 8.4 Export + +``` +GET /api/v1/export.csv? +``` + +Streams the full result set — not just the current page — as CSV with the currently +selected columns plus the reclaim score and its components. This is the v1 deliverable: +the artifact you take away and act on. + +### 8.5 Health + +``` +GET /healthz → { "status": "ok", "db": "ok", "plex": "ok"|"unreachable", + "last_scan_at": 1757000000, "stale": false } +``` + +--- + +## 9. Web UI + +Server-rendered Flask shell, vanilla JS plus two small libraries: a virtualized table +(TanStack Table core or Tabulator) and a charting library. No build step, no npm in the +container — assets are vendored. This keeps the container small and the deploy identical +to the Mythica pattern. + +### 9.1 Screens + +**Dashboard** — the landing page. Headline tiles (total items, total size, size of +never-watched content, size of content untouched for 2+ years, last scan time), plus: + +- *Size by library* — horizontal bar. +- *Added over time* — monthly bars, stacked by library. Shows acquisition habits. +- *Watched vs unwatched by size* — the "how much of this shelf has anyone ever touched" + chart, which is usually the one that motivates the whole exercise. +- *Size vs. last watched* — scatter, log-size on Y, days-since-watch on X. Points in the + upper right are the reclaim targets, and clicking a region filters the grid to it. + +**Library grid** — the main workspace. A virtualized table, every column sortable, with: + +- A filter rail: library checkboxes, kind, size range, added range, last-watched range, + watch-count range, resolution, and a free-text title search. +- A weight panel with five sliders and a live-updating `reclaim_score` column. +- Multi-select via checkboxes, shift-click ranges, and select-all-matching-filter. + In v1 selection drives the aggregate footer ("**312 items selected · 4.1 TB**") and + the CSV export. The buttons that will eventually delete are present but disabled, with + a tooltip explaining v1 is report-only — so the workflow is proven before it's armed. +- Row expansion: a season expands to its episodes; any row expands to its file parts + with full paths. + +**Saved views** — list, create, edit, duplicate, run. Creating one captures the current +filter, sort, columns, and weights, so the flow is *explore, then name what you found*. + +**Item detail** — everything known: all parts with paths and sizes, per-account watch +history timeline, score breakdown showing each component's contribution. + +**Scans** — history table, per-scan counts and duration, a "Scan now" button, live +progress. + +**Settings** — connection status, scan schedule, score defaults, horizons, grace periods. + +### 9.2 Interaction principles + +- Every number that represents bytes is displayed in both human units and, on hover, the + exact byte count. Reclaim decisions turn on real numbers. +- The reclaim score column always shows its components on hover. A score you can't + interrogate is a score you won't trust, and won't act on. +- Filters live in the URL query string, so any view is linkable and back/forward works. +- Nothing in v1 is destructive, and the UI says so plainly in the header rather than + leaving the user to wonder. + +--- + +## 10. Configuration + +All via environment variables, so the Portainer stack is the single source of truth. + +| Variable | Default | Notes | +|---|---|---| +| `MEDIASHELF_SECRET_KEY` | — | required, Flask session signing | +| `PLEX_BASE_URL` | — | required, e.g. `http://192.168.1.10:32400` | +| `PLEX_TOKEN` | — | required, secret | +| `PLEX_VERIFY_SSL` | `true` | | +| `PLEX_TIMEOUT_S` | `30` | | +| `PLEX_PAGE_SIZE` | `500` | | +| `PLEX_REQUEST_DELAY_MS` | `0` | throttle for Loki's sake | +| `DATABASE_PATH` | `/data/mediashelf.db` | on the mounted volume | +| `SCAN_SCHEDULE_CRON` | `0 4 * * *` | nightly incremental at 04:00 | +| `SCAN_FULL_SWEEP_CRON` | `0 3 * * 0` | weekly full sweep, Sunday 03:00 | +| `SCAN_ON_STARTUP` | `false` | | +| `SCAN_LOCK_TIMEOUT_S` | `7200` | | +| `SCORE_STALE_HORIZON_DAYS` | `730` | | +| `SCORE_AGE_HORIZON_DAYS` | `1095` | | +| `SCORE_POPULAR_AT` | `3` | | +| `SCORE_GRACE_DAYS` | `30` | | +| `SCORE_RECENT_DAYS` | `90` | | +| `TZ` | `America/Regina` | so cron times mean what they look like | +| `LOG_LEVEL` | `INFO` | | + +`PLEX_TOKEN` is a secret and must not be committed. The repo ships `.env.example` with +placeholders, and `.env` is gitignored. (Worth noting given the SOAP-password-in-history +problem on the Mythica repo — this one starts clean and stays clean.) + +--- + +## 11. Deployment + +### 11.1 Container + +Single image, `python:3.12-slim` base, multi-stage so build deps don't ship. Runs as a +non-root user. One volume: `/data`. One port: 8080. + +### 11.2 Portainer stack on Nox + +```yaml +services: + mediashelf: + image: mediashelf:latest + container_name: mediashelf + restart: unless-stopped + ports: + - "8085:8080" + volumes: + - mediashelf_data:/data + environment: + - MEDIASHELF_SECRET_KEY=${MEDIASHELF_SECRET_KEY} + - PLEX_BASE_URL=${PLEX_BASE_URL} + - PLEX_TOKEN=${PLEX_TOKEN} + - DATABASE_PATH=/data/mediashelf.db + - SCAN_SCHEDULE_CRON=${SCAN_SCHEDULE_CRON} + - TZ=America/Regina + - LOG_LEVEL=INFO + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request;urllib.request.urlopen('http://localhost:8080/healthz')"] + interval: 60s + timeout: 10s + retries: 3 + +volumes: + mediashelf_data: +``` + +**Every variable used in the stack must be listed under `environment:` here, not only set +in Portainer's UI.** A variable set in Portainer that isn't declared in the compose file +silently does nothing — the same trap that bit `PLAYERMAP_SHOW_GMS` on the Mythica stack. + +Published through Nginx Proxy Manager at `mediashelf.hallsworth.ca`, upstream +`192.168.1.77:8085`. Note the host-port → container-port mapping is 8085 → 8080; NPM's +upstream must point at the **host** port. + +### 11.3 The gunicorn / scheduler constraint + +APScheduler runs in-process. With `--workers 2`, both workers would start a scheduler and +two scans would race for the SQLite write lock. Mitigation: the scheduler starts only in +the worker that successfully acquires an exclusive advisory lock on a lockfile in `/data` +at boot. The `scan_lock` table is the second line of defence. Both are needed; either +alone has a race window. + +### 11.4 Backup + +The database is one file. A weekly `sqlite3 .backup` to the volume, keeping four, is +enough — everything in it is reconstructible from Plex except accumulated `watch_event` +rows beyond Plex's own history retention (§4.5), which is precisely the part worth +backing up. + +--- + +## 12. Security + +MediaShelf is LAN-only with no authentication, by decision. That is a reasonable posture +for a read-only tool behind a firewall, and it is a bad posture the moment v2 adds +deletion. This section records what that means now and what changes later. + +**v1 posture:** + +- No auth, no sessions, no user records. +- Bind to the LAN; if published through NPM, restrict by source IP at the proxy or keep + the hostname internal-only. It should not be reachable from the internet. +- `PLEX_TOKEN` lives in the environment only, never in the database, never in logs, never + in an API response. The `/api/v1/settings` endpoint redacts it. +- No user input reaches SQL except through the whitelisted rule grammar (§7.1) with bound + parameters. +- All rendered strings are escaped; file paths from Plex are treated as untrusted display + data. +- CSRF is not a concern in v1 because nothing mutating exists beyond scan triggering and + saved views. Both use `POST` with a `SameSite=Strict` token anyway, because retrofitting + CSRF protection after deletion exists is exactly the kind of thing that gets skipped. + +**Required before v2 ships deletion:** authentication is not optional. A single admin +password with a signed session, rate-limited login, and an audit log is the minimum. An +unauthenticated endpoint that deletes files on a 90 TB array is an accident waiting for +a misconfigured proxy rule. + +--- + +## 13. Testing + +| Layer | Approach | +|---|---| +| Provider | Recorded Plex JSON fixtures (`responses`/`vcr.py`) covering a movie section, a show section with multi-season shows, multi-version movies, split parts, and a history page. No live server needed in CI. | +| Normalization | Table-driven tests: odd cases like an item with no `Part`, a season with a missing episode, unicode titles, zero-byte parts. | +| Ingest | In-memory SQLite; assert idempotency (running the same scan twice changes nothing), assert `missing` transitions, assert history watermarking doesn't skip or duplicate events. | +| Scoring | Unit tests on known inputs; **property test asserting the SQL and Python implementations agree** (§6.4); tests that grace rules clamp correctly. | +| Rules | Every operator round-trips to correct SQL; injection attempts in field names and values are rejected with 400. | +| API | Flask test client, schema assertions on every response shape. | +| Smoke | A `make smoke` target that runs a real scan against a live Plex and prints the top 20 reclaim candidates — the human sanity check that no unit test replaces. | + +The single most important test is idempotency: a scanner that double-counts sizes or +duplicates history events produces a report that looks plausible and is wrong, which is +worse than one that crashes. + +--- + +## 14. Roadmap + +### 14.1 v2 — Deletion + +The reason v1 is report-only is that the scoring model needs to earn trust before it +gets to remove anything. When it does, the design is **two-stage quarantine, not +delete**: + +1. **Select** items in the UI, review the manifest (every file path, every byte, total). +2. **Quarantine** — MediaShelf records an intent and moves the files to a trash directory + on the same filesystem (a rename, so it's instant and atomic regardless of size), + then triggers a Plex library refresh so the entries disappear from Plex. +3. **Purge** — after a configurable hold (default 14 days) the quarantined files are + permanently removed, either automatically or by an explicit click. Until then, + restore is a rename back. + +This requires the one thing v1 deliberately avoids: filesystem access to the media +shares. The v2 container mounts them read-write and gains a path-mapping layer +translating Plex's paths (`/mnt/vault2/...` as Loki sees them) into container paths. Two +guardrails: a configured allowlist of path prefixes MediaShelf may touch, and a refusal +to act on any path outside them. Plus an immutable audit log of every quarantine and +purge — what, when, how big, and what the score was that recommended it. + +Deletion also makes authentication mandatory (§12) and makes a dry-run mode +non-negotiable: every destructive operation gets a preview that lists exactly what would +happen and requires explicit confirmation. + +### 14.2 v3 — Emby and Jellyfin + +The `MediaProvider` protocol is the whole preparation: + +```python +class MediaProvider(Protocol): + def server_info(self) -> ServerInfo: ... + def libraries(self) -> list[Library]: ... + def items(self, library: Library) -> Iterator[Item]: ... # yields normalized items + def watch_events(self, since: int | None) -> Iterator[WatchEvent]: ... + def accounts(self) -> list[Account]: ... + def refresh_library(self, library: Library) -> None: ... # v2 only +``` + +Jellyfin and Emby share an API lineage, so one `JellyfinProvider` will likely cover both +with a capability flag or two. The mapping is close: `Items` with +`Fields=Path,MediaSources,DateCreated`, `UserData.PlayCount` and `LastPlayedDate` for +watch data, `/Users` for accounts. The notable difference is that Jellyfin's watch data +is genuinely per-user and must be fetched per user and merged, rather than read from one +global history endpoint. + +The `provider` table already carries `kind` and the schema already keys everything on +`(provider_id, provider_item_id)`, so multiple servers can coexist in one database — the +door is open to a combined view across Plex and Jellyfin without a migration. + +### 14.3 Later candidates + +- **Optional filesystem stat** — if the shares happen to be mounted, verify Plex's sizes + against reality and surface orphans Plex never indexed. +- **Duplicate and version analysis** — the same title held at 4K and 1080p is a common + and invisible source of waste. +- **Trend tracking** — MediaShelf already snapshots on a schedule; a chart of library + size over time and a projection of when the vault fills up is nearly free. +- **Tautulli import** — if Tautulli is running, its history database predates Plex's own + retention window and would deepen the watch record considerably. +- **Notifications** — a monthly "here's what's gone cold" summary. + +--- + +## 15. Open questions + +1. **Plex history retention on Loki.** If it's short, the first ingest captures less than + hoped and `watch_event` becomes valuable immediately. Worth checking before the first + scan, and worth knowing whether Tautulli has been running (§14.3). +2. **Multiple Plex libraries of the same kind** — are there several movie sections (e.g. + Movies, Kids, Documentaries)? The design handles it, but the seed views and default + charts get more useful if the real shape is known. +3. **Path structure on the vaults** — knowing the actual roots would let the grid group by + physical vault, which matters when the goal is freeing a *specific* array rather than + space in general. +4. **Where does the image get built?** Nox running Portainer suggests either a build from + a git URL in the stack, or pushing to a registry. Worth settling before the first + deploy. +5. **Is Tautulli running anywhere?** It changes the watch-history story materially.