diff --git a/.env.example b/.env.example index 1b8cdca..668daa6 100644 --- a/.env.example +++ b/.env.example @@ -13,6 +13,19 @@ PLEX_TIMEOUT_S=30 PLEX_PAGE_SIZE=500 PLEX_REQUEST_DELAY_MS=0 +# --- Tautulli (primary watch-history source) --- +# Leave TAUTULLI_BASE_URL empty to disable and fall back to Plex session history. +TAUTULLI_BASE_URL=http://192.168.1.100:8181 +TAUTULLI_API_KEY=your-tautulli-api-key-here +TAUTULLI_TIMEOUT_S=30 +TAUTULLI_PAGE_SIZE=1000 + +# auto | tautulli | plex +HISTORY_SOURCE=auto +SESSION_MERGE_WINDOW_H=6 +COMPLETION_THRESHOLD=85 +ABANDON_CEILING=15 + # --- Storage --- DATABASE_PATH=/data/mediashelf.db @@ -26,5 +39,6 @@ SCAN_LOCK_TIMEOUT_S=7200 SCORE_STALE_HORIZON_DAYS=730 SCORE_AGE_HORIZON_DAYS=1095 SCORE_POPULAR_AT=3 +SCORE_REJECTED_AT=2 SCORE_GRACE_DAYS=30 SCORE_RECENT_DAYS=90 diff --git a/README.md b/README.md index 940c908..2369cf9 100644 --- a/README.md +++ b/README.md @@ -3,11 +3,12 @@ A self-hosted web app for figuring out which of the thousands of files in a Plex library are actually worth keeping. -MediaShelf scans a Plex Media Server over its HTTP API, builds a local snapshot of every -movie and TV season, and joins together the facts Plex already knows but never shows side -by side — date added, size on disk, file path, owning library, watch count, and last -watched — into a sortable, filterable, chartable grid with a tunable **reclaim score** -that ranks deletion candidates. +MediaShelf scans a Plex Media Server over its HTTP API and pulls watch history from +Tautulli, builds a local snapshot of every movie and TV season, and joins together facts +that are never shown side by side — date added, size on disk, file path, owning library, +watch count, last watched, and *how much of it anyone actually finished* — into a +sortable, filterable, chartable grid with a tunable **reclaim score** that ranks deletion +candidates. ## Status @@ -22,13 +23,16 @@ Nothing is implemented yet — this repository currently holds the design. - Full-library ingest from Plex, no agent on the Plex host, no filesystem mounts - Movies at item level, TV rolled up to **season** level -- Watch data pulled from the server-wide playback history, so plays by *every* Plex - account are counted — not just the token owner's +- Watch data from **Tautulli**, so plays by *every* account are counted and a play that + was abandoned after five minutes is distinguished from one that was finished — Plex's + own history reports those identically. Falls back to Plex session history if Tautulli + is unavailable, and says so rather than degrading silently - Sort and filter on every metric; charts for size by library, additions over time, - watched vs. unwatched by size, and size vs. last-watched + finished vs. abandoned vs. never-opened by size, and size vs. last-watched - A weighted reclaim score with live sliders, and grace rules so it never recommends something you added last week -- Named, re-runnable saved views — *"unwatched, older than 2 years, over 10 GB"* +- Named, re-runnable saved views — *"unwatched, older than 2 years, over 10 GB"*, + *"two people started it and nobody finished it"* - CSV export of any view ## Planned stack diff --git a/docs/design.md b/docs/design.md index e9d9560..d7df4e2 100644 --- a/docs/design.md +++ b/docs/design.md @@ -1,6 +1,10 @@ # MediaShelf — Software Design -**Status:** Draft v1 · **Author:** Claude (spec) for Jess · **Date:** 2026-09-07 +**Status:** Draft v1.1 · **Author:** Claude (spec) for Jess · **Date:** 2026-09-07 + +*Revision 1.1 — Tautulli (192.168.1.100:8181) confirmed present and promoted to the +primary watch-history source; Plex's own history demoted to fallback. Deployment path +settled: image built in CI/locally and pushed to Nox's Portainer.* --- @@ -28,6 +32,9 @@ destructive is wired up. ### 1.1 Goals - Full-library ingest from Plex over its HTTP API, with no agent installed on the Plex host. +- Watch history from **Tautulli**, which has been logging every play since it was + installed and knows *how much* of each item was actually watched — not just that a + play started. - A durable local snapshot so the UI is instant and Plex isn't hammered on every click. - Movies at item level; TV rolled up to **season** level. - Sort/filter/visualize on: date added, last watched, watch count, size on disk, file path, library. @@ -57,7 +64,9 @@ destructive is wired up. | Auth | None; LAN-only, single admin view, watch data aggregated across all Plex accounts | | TV granularity | Season | | Size/path source | Plex API only | +| Watch-history source | **Tautulli** primary, Plex session history as fallback | | v1 output | Sortable grid + filters + charts, tunable reclaim score, saved rule sets | +| Image delivery | Built outside Nox, pushed to Nox's Portainer | | Implementation | Design doc only for now; code to follow | --- @@ -97,18 +106,26 @@ migration of services onto Enki. │ │ SQLite /data/mediashelf.db (WAL) │ │ │ └────────────────────────────┬───────────────────────────────┘ │ │ │ volume: /data │ -└────────────────────────────────┼───────────────────────────────────────────┘ - │ HTTP :32400 + X-Plex-Token - ▼ - Loki (192.168.1.10) — Plex Media Server +└──────────┬─────────────────────┴───────────────────────────────────────────┘ + │ │ + │ HTTP :32400 + X-Plex-Token │ HTTP :8181 + apikey + ▼ ▼ + Loki (192.168.1.10) Isis (192.168.1.100:8181) + Plex Media Server Tautulli + · libraries, items, parts · every play since install + · size, path, added_at · per-user, per-item, % complete + · session history (fallback) ``` ### 3.1 Component responsibilities -**`mediashelf.providers`** — the pluggable source layer. Defines the `MediaProvider` -protocol and ships one implementation, `PlexProvider`. Everything above this layer -speaks only in MediaShelf's own normalized dataclasses, never in Plex XML. This is the -single seam that Emby/Jellyfin support slots into (§14.2). +**`mediashelf.providers`** — the pluggable source layer. Defines two protocols: +`MediaProvider` (libraries, items, parts) implemented by `PlexProvider`, and +`HistoryProvider` (watch events) implemented by both `TautulliHistoryProvider` and +`PlexHistoryProvider`. Everything above this layer speaks only in MediaShelf's own +normalized dataclasses. Splitting the two protocols is what lets watch history come +from a different box than the library itself — which, given Tautulli lives on Isis and +Plex on Loki, is not hypothetical. It is also the seam Emby/Jellyfin slot into (§14.2). **`mediashelf.ingest`** — orchestrates a scan: pull sections, pull items, pull history, normalize, upsert into SQLite inside a transaction, recompute season rollups, mark @@ -139,9 +156,13 @@ mean **gunicorn must run the scheduler in exactly one worker** — see §11.3. --- -## 4. Plex integration +## 4. Data sources -### 4.1 Connection +MediaShelf reads from two servers. **Plex on Loki** is the authority on what exists — +libraries, items, files, sizes, paths, added dates. **Tautulli on Isis** is the authority +on what has been watched. Sections 4.1–4.7 cover Plex; 4.8–4.11 cover Tautulli. + +### 4.1 Plex connection MediaShelf authenticates with a Plex server token (`X-Plex-Token`) supplied via environment variable. It talks directly to `http://192.168.1.10:32400` on the LAN — no @@ -230,46 +251,25 @@ The accepted trade-offs: - **Paths are Loki-relative.** `file_path` is the path as the Plex server sees it. It is displayed and exported for a human's benefit, not resolved by MediaShelf. -### 4.5 Watch data: the important subtlety +### 4.5 Why Plex is not the watch-data source -`viewCount` and `lastViewedAt` on a library item are **scoped to the account that owns -the token**. If Jess's token is used, plays by home users, managed users, and shared -users are not counted. For a household library this understates watches badly, and a -reclaim tool that deletes "unwatched" content someone else has been watching is worse -than useless. +Two problems make Plex's own watch data unfit to build a reclaim tool on. -So MediaShelf treats `/status/sessions/history/all` as the source of truth for watch -data. That endpoint returns one row per playback event, server-wide, across all -accounts, each with: +**It is scoped to one account.** `viewCount` and `lastViewedAt` on a library item reflect +only the account that owns the token. Plays by home users, managed users, and shared users +are invisible. For a household library this understates watches badly, and a tool that +recommends deleting "unwatched" content someone else watches weekly is worse than useless. -- `historyKey`, `ratingKey`, `librarySectionID` -- `viewedAt` (epoch seconds) -- `accountID` (which Plex user) -- `type`, `title`, and for episodes the parent/grandparent identifiers +**It only knows that a play started.** `/status/sessions/history/all` records a playback +event. It does not record whether the viewer watched the whole thing or bailed after four +minutes. Those two events mean opposite things for a deletion decision, and Plex reports +them identically. -MediaShelf ingests this into a `watch_event` table and derives, per item: +`/status/sessions/history/all` solves the first problem — it is server-wide across all +accounts — so it remains the **fallback** source, and §4.11 covers when it is used. But +Tautulli solves both, so Tautulli is the primary. -- `watch_count` — number of distinct playback events -- `last_watched_at` — max `viewedAt` -- `first_watched_at` — min `viewedAt` -- `distinct_watcher_count` — count of distinct `accountID` - -The single-admin UI shows the aggregate. The per-account data is stored anyway, because -it's free to keep and it's what the "who actually watches this" question will need later. - -**Caveat to verify during the first spike:** Plex prunes session history according to the -server's *Media Playback → Settings → Empty trash / history retention* configuration, and -some servers cap it. If Loki's retention window turns out to be shorter than the useful -analysis window, MediaShelf's own `watch_event` table becomes the long-term record — -which it already is, since it accumulates across scans and never deletes rows. The first -full ingest establishes the baseline; every scan after that only needs events newer than -the last one seen. - -The item-level `viewCount` is still stored as an advisory field, and the UI shows a small -warning badge when `viewCount > 0` but no history events exist for that item — that's the -signature of a watch that happened before history retention began. - -### 4.6 Scan strategy +### 4.6 Plex scan strategy **Full scan** — walks every section, every item, every part, and the complete history. Run on first launch and on demand. Expect a few minutes for a large library. @@ -302,6 +302,133 @@ the ingest is designed to be slow and steady rather than parallel and aggressive | Malformed / partial item | Item skipped, logged, counted in `scan.warning_count`; the scan still succeeds | | Timeout mid-page | Up to 3 retries with exponential backoff, then the scan fails | +### 4.8 Tautulli — the watch-history source + +Tautulli runs at `http://192.168.1.100:8181` (on Isis) and has been logging every +playback on Loki since the day it was installed. Its database is independent of Plex's: +clearing Plex's history, or Plex pruning its own, does not touch it. That makes it a +deeper and more durable record than anything Plex exposes. + +Its API is a single endpoint with a `cmd` parameter: + +``` +GET http://192.168.1.100:8181/api/v2?apikey=&cmd=& +``` + +Every response is `{"response": {"result": "success"|"error", "message": ..., "data": ...}}`. +A transport-level 200 with `result: "error"` in the body is the normal failure mode, so +**the result field must be checked on every call** — treating HTTP 200 as success is the +mistake this API invites. + +| Purpose | Command | +|---|---| +| Reachability / version | `cmd=get_server_info`, `cmd=arnold` (returns a quote; a cheap liveness ping) | +| Which Plex server it watches | `cmd=get_server_info` → `pms_identifier` | +| Users | `cmd=get_users` | +| Libraries | `cmd=get_libraries` | +| **Watch history** | `cmd=get_history` | +| Per-item size / play count (cross-check) | `cmd=get_library_media_info` | + +### 4.9 Ingesting history + +``` +cmd=get_history + &grouping=0 # do NOT collapse successive plays — we want raw events + &order_column=date + &order_dir=desc + &start= + &length=1000 + &after= # incremental watermark +``` + +`grouping=0` matters. Tautulli's *Group Successive Play History* setting merges a paused- +and-resumed play into one row for display purposes. For counting distinct viewings that +is what we want — but the grouping should be MediaShelf's decision, applied consistently, +not a mirror of however that setting happens to be configured on Isis. So MediaShelf pulls +ungrouped events and does its own session collapsing: **events for the same item by the +same user whose start times fall within `SESSION_MERGE_WINDOW` (default 6 hours) count as +one viewing.** + +The fields MediaShelf keeps from each history row: + +| Tautulli field | Use | +|---|---| +| `row_id` | idempotency key for upsert | +| `reference_id` | Tautulli's own grouping id, kept for cross-checking | +| `date` / `started` / `stopped` | epoch seconds | +| `rating_key` | the leaf item — movie or episode | +| `parent_rating_key`, `grandparent_rating_key` | season and show, for rollup | +| `user_id`, `user`, `friendly_name` | who watched | +| `media_type` | `movie` / `episode` | +| `percent_complete` | **how much was actually watched** | +| `watched_status` | Tautulli's own verdict against its configured threshold | +| `play_duration`, `paused_counter` | real time spent, minus pauses | +| `platform`, `player` | stored, unused in v1 | + +### 4.10 What "watched" means + +This is the part that most changes the quality of the output, so MediaShelf defines it +itself rather than inheriting Tautulli's threshold setting: + +| Derived value | Rule | +|---|---| +| **completed view** | `percent_complete >= COMPLETION_THRESHOLD` (default 85) | +| **partial view** | `percent_complete` between `ABANDON_CEILING` (default 15) and the completion threshold | +| **abandoned view** | `percent_complete < ABANDON_CEILING` — started and bailed | + +MediaShelf uses `percent_complete` rather than `watched_status` as the primary signal +because `watched_status` depends on how the threshold is configured on Isis, and that +configuration can change without warning. `watched_status` is stored anyway and shown in +the UI as a cross-check; a systematic disagreement between the two is worth seeing. + +The rollups per item become: + +- `watch_count` — **completed** views only +- `partial_count`, `abandoned_count` +- `last_watched_at` — most recent completed view +- `last_touched_at` — most recent view of any kind, including abandoned +- `first_watched_at`, `distinct_watcher_count` +- `avg_percent_complete` + +**An abandoned view is evidence for deletion, not against it.** Three people started a +film and all three quit twenty minutes in: that is a stronger signal to delete than a film +nobody ever opened, because the household has now actively rejected it. Plex's history +cannot express this at all, and it is the single biggest reason to prefer Tautulli. The +scoring model uses it (§6.1, `rejection` component). + +### 4.11 Fallback, coverage, and cross-checks + +**When Tautulli is unavailable**, MediaShelf falls back to `PlexHistoryProvider` and +`/status/sessions/history/all`. Events ingested that way are marked `source = 'plex'` and +carry no `percent_complete`, so they count as completed views (the only assumption +available) and are excluded from the `rejection` component. The UI shows a banner +explaining the score is running in degraded mode, because a silently degraded score is a +score that gets trusted when it shouldn't be. + +**Tautulli's coverage horizon.** Tautulli only knows about plays since it was installed. +Anything watched before that is invisible to it — and those items will look never-watched. +On first ingest MediaShelf records `history_coverage_since` = the oldest event Tautulli +holds, and: + +- Items with `added_at` earlier than that date get a `pre_history` flag. +- The UI shows a badge on those rows and a one-line explanation on the dashboard. +- The `staleness` score component is **capped** for `pre_history` items, so a 2009 movie + that was watched in 2015 and never since doesn't get scored as though it had never been + watched at all. + +**Cross-check with `get_library_media_info`.** Tautulli maintains its own per-item table +with `file_size`, `added_at`, `play_count`, and `last_played`. MediaShelf pulls this once +per scan per library and compares it to what Plex reported. Disagreements are recorded on +the scan as warnings, not errors. This is cheap insurance: two independent views of the +same library disagreeing about a file's size is exactly the sort of thing that means a +scan is stale somewhere, and it is much better found by a scan than by a deletion. + +**Joining to Plex.** Both systems use the same Plex `rating_key`, so the join is direct — +no title or path matching, no fuzzy logic. If `get_server_info.pms_identifier` does not +match the `machineIdentifier` from Plex's `/identity`, Tautulli is watching a *different* +Plex server and MediaShelf refuses to use it rather than joining nonsense data together. +That check runs at the start of every scan. + --- ## 5. Data model @@ -364,10 +491,15 @@ CREATE TABLE media_item ( resolution TEXT, video_codec TEXT, -- watch rollups (from watch_event) - watch_count INTEGER NOT NULL DEFAULT 0, - last_watched_at INTEGER, + watch_count INTEGER NOT NULL DEFAULT 0, -- COMPLETED views only + partial_count INTEGER NOT NULL DEFAULT 0, + abandoned_count INTEGER NOT NULL DEFAULT 0, + last_watched_at INTEGER, -- last completed view + last_touched_at INTEGER, -- last view of any kind first_watched_at INTEGER, distinct_watcher_count INTEGER NOT NULL DEFAULT 0, + avg_percent_complete REAL, + pre_history INTEGER NOT NULL DEFAULT 0, -- added before history coverage provider_view_count INTEGER NOT NULL DEFAULT 0, -- advisory, token-scoped -- lifecycle status TEXT NOT NULL DEFAULT 'present', -- 'present' | 'missing' @@ -411,24 +543,50 @@ CREATE TABLE media_part ( ); -- One row per playback event, server-wide, all accounts. +-- Primary source is Tautulli; 'plex' rows are degraded fallback events (§4.11). CREATE TABLE watch_event ( id INTEGER PRIMARY KEY, provider_id INTEGER NOT NULL REFERENCES provider(id), - history_key TEXT NOT NULL, + source TEXT NOT NULL, -- 'tautulli' | 'plex' + source_row_id TEXT NOT NULL, -- Tautulli row_id, or Plex historyKey + reference_id TEXT, -- Tautulli's own grouping id provider_item_id TEXT NOT NULL, -- the leaf: movie or episode ratingKey account_id TEXT, - viewed_at INTEGER NOT NULL, - UNIQUE (provider_id, history_key) + viewed_at INTEGER NOT NULL, -- 'date' / 'started' + stopped_at INTEGER, + play_duration_s INTEGER, + paused_counter_s INTEGER, + percent_complete INTEGER, -- NULL for source='plex' + watched_status REAL, -- Tautulli's verdict, advisory + disposition TEXT NOT NULL, -- 'completed' | 'partial' | 'abandoned' + session_id TEXT, -- MediaShelf's own merge key (§4.9) + media_type TEXT, + platform TEXT, + UNIQUE (provider_id, source, source_row_id) ); CREATE TABLE account ( id INTEGER PRIMARY KEY, provider_id INTEGER NOT NULL REFERENCES provider(id), - account_id TEXT NOT NULL, + account_id TEXT NOT NULL, -- Tautulli user_id / Plex accountID name TEXT, + friendly_name TEXT, UNIQUE (provider_id, account_id) ); +-- What the history record actually covers, so 'never watched' can be +-- distinguished from 'watched before Tautulli existed' (§4.11). +CREATE TABLE history_coverage ( + id INTEGER PRIMARY KEY, + provider_id INTEGER NOT NULL REFERENCES provider(id), + source TEXT NOT NULL, + earliest_event_at INTEGER, + latest_event_at INTEGER, + event_count INTEGER NOT NULL DEFAULT 0, + updated_at INTEGER NOT NULL, + UNIQUE (provider_id, source) +); + CREATE TABLE scan ( id INTEGER PRIMARY KEY, provider_id INTEGER NOT NULL REFERENCES provider(id), @@ -478,6 +636,8 @@ CREATE INDEX ix_part_item ON media_part(media_item_id); CREATE INDEX ix_part_episode ON media_part(episode_id); CREATE INDEX ix_event_item ON watch_event(provider_item_id); CREATE INDEX ix_event_viewed ON watch_event(viewed_at); +CREATE INDEX ix_event_disp ON watch_event(provider_item_id, disposition); +CREATE INDEX ix_event_account ON watch_event(account_id, viewed_at); ``` A full-text index over `title` uses SQLite FTS5 (`media_item_fts`) kept in sync by @@ -493,8 +653,10 @@ After items and history are ingested, a rollup pass recomputes each season row: | `episode_count` | `COUNT(episode)` | | `duration_ms` | `SUM(episode.duration_ms)` | | `added_at` | `MIN(episode.added_at)` — when the season first landed | -| `watch_count` | `SUM(episode.watch_count)` | +| `watch_count` | `SUM(episode.watch_count)` — completed episode views | +| `abandoned_count` | `SUM(episode.abandoned_count)` | | `last_watched_at` | `MAX(episode.last_watched_at)` | +| `last_touched_at` | `MAX(episode.last_touched_at)` | | `primary_path` | the common directory prefix of its episodes' paths | A show row rolls up the same way from its seasons, for display and for the "delete the @@ -531,26 +693,38 @@ to want it?* It is a weighted sum of normalized components, each in `[0, 1]`, sc | **`unpopularity`** | `1 - min(normalized_watches / POPULAR_AT, 1)` where `normalized_watches` = `watch_count` for movies, `watch_count / episode_count` for seasons | Rewatched things are keepers | | **`age`** | `min(days_since_added / AGE_HORIZON, 1)` | Something added last week deserves a chance | | **`solitude`** | `1 - min(distinct_watcher_count / 3, 1)` | Content only one household member ever touched is safer to cut | +| **`rejection`** | `min(abandoned_count / REJECTED_AT, 1)`, zero when `watch_count > 0` | Started and quit is an active verdict — see below | -`STALE_HORIZON` defaults to 730 days, `AGE_HORIZON` to 1095 days, `POPULAR_AT` to 3. +`STALE_HORIZON` defaults to 730 days, `AGE_HORIZON` to 1095 days, `POPULAR_AT` to 3, +`REJECTED_AT` to 2. + +The `rejection` component only exists because Tautulli supplies `percent_complete` +(§4.10). It captures the case a purely count-based model gets backwards: an item three +people started and all abandoned looks "watched" to Plex and looks like a keeper to a +naive score, when in fact the household has tried it and said no. It is deliberately +zeroed the moment anyone completes a view, so a film that was abandoned twice and then +watched through is not penalised. When history is running in Plex-fallback mode this +component is unavailable and drops out of the weighting entirely (§4.11). ### 6.2 The formula ``` -score = 100 × ( w_size · size - + w_staleness · staleness - + w_unpopularity· unpopularity - + w_age · age - + w_solitude · solitude ) - / (w_size + w_staleness + w_unpopularity + w_age + w_solitude) +score = 100 × Σ(wᵢ · componentᵢ) / Σ(wᵢ) ``` Default weight profile: ```json -{ "size": 0.30, "staleness": 0.30, "unpopularity": 0.25, "age": 0.10, "solitude": 0.05 } +{ "size": 0.28, "staleness": 0.26, "unpopularity": 0.22, + "rejection": 0.12, "age": 0.08, "solitude": 0.04 } ``` +The denominator sums only the weights of components that are **available** for that row. +When history is in Plex-fallback mode, `rejection` is dropped and the remaining weights +renormalize, rather than every item silently scoring lower because one component is +pinned at zero. The same applies per-row: a `pre_history` item has its `staleness` +capped (§4.11), and the UI shows which components contributed. + ### 6.3 The grace rule Two hard overrides, applied after scoring, because a purely numeric model will @@ -607,7 +781,9 @@ Rules are a nested JSON group, evaluated into parameterized SQL: Fields are drawn from a whitelist mapping field name → column and type. Operators are a fixed set: `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `in`, `not_in`, `contains`, `starts_with`, `is_null`, `is_not_null`, `older_than_days`, `newer_than_days`, -`never` (for watch fields). Anything outside the whitelist is a 400. Values are always +`never` (for watch fields). The whitelisted fields include the completion metrics — +`watch_count`, `partial_count`, `abandoned_count`, `avg_percent_complete`, +`last_touched_at`, `pre_history`. Anything outside the whitelist is a 400. Values are always bound parameters. There is no SQL string interpolation anywhere in this path. ### 7.2 Seed views shipped by default @@ -618,6 +794,8 @@ bound parameters. There is no SQL string interpolation anywhere in this path. | Cold storage | `last_watched_at older than 3 years` | | One-and-done movies | `kind = movie AND watch_count = 1 AND last_watched_at older than 2 years` | | Abandoned seasons | `kind = season AND watch_count/episode_count < 0.3 AND added_at older than 1 year` | +| Tried and rejected | `abandoned_count >= 2 AND watch_count = 0` — the household said no | +| Bailed in the first 15 min | `avg_percent_complete < 15 AND watch_count = 0 AND size_bytes >= 4 GB` | | Biggest 100 | sorted by `size_bytes desc`, no filter | | Recently added | `added_at newer than 30 days` — the sanity check, not a delete list | @@ -665,18 +843,24 @@ Response: "size_bytes": 32212254720, "added_at": 1490000000, "last_watched_at": null, + "last_touched_at": 1712000000, "watch_count": 0, - "distinct_watcher_count": 0, + "partial_count": 0, + "abandoned_count": 3, + "avg_percent_complete": 8.4, + "distinct_watcher_count": 3, + "pre_history": false, "episode_count": null, "primary_path": "/mnt/vault2/movies/Example Film (2011)/Example Film (2011).mkv", "part_count": 1, "resolution": "4k", "reclaim_score": 91.4, "reclaim_components": { - "size": 0.97, "staleness": 1.0, "unpopularity": 1.0, "age": 1.0, "solitude": 1.0 + "size": 0.97, "staleness": 1.0, "unpopularity": 1.0, + "rejection": 1.0, "age": 1.0, "solitude": 0.0 }, "grace": null, - "flags": ["history_gap"] + "flags": ["rejected"] } ] } @@ -693,6 +877,9 @@ GET /api/v1/stats/size-by-library GET /api/v1/stats/added-over-time?bucket=month GET /api/v1/stats/watch-distribution GET /api/v1/stats/size-vs-lastwatched scatter data, downsampled server-side +GET /api/v1/stats/completion finished / abandoned / never-opened, by size +GET /api/v1/sources per-source status: reachable?, coverage window, + event count, active history source ``` ### 8.2 Saved views @@ -729,7 +916,10 @@ the artifact you take away and act on. ### 8.5 Health ``` -GET /healthz → { "status": "ok", "db": "ok", "plex": "ok"|"unreachable", +GET /healthz → { "status": "ok", "db": "ok", + "plex": "ok"|"unreachable", + "tautulli": "ok"|"unreachable"|"disabled", + "history_source": "tautulli"|"plex", "last_scan_at": 1757000000, "stale": false } ``` @@ -750,7 +940,9 @@ never-watched content, size of content untouched for 2+ years, last scan time), - *Size by library* — horizontal bar. - *Added over time* — monthly bars, stacked by library. Shows acquisition habits. - *Watched vs unwatched by size* — the "how much of this shelf has anyone ever touched" - chart, which is usually the one that motivates the whole exercise. + chart, which is usually the one that motivates the whole exercise. Split three ways + now that completion data exists: **finished · started and abandoned · never opened**. + The middle band is the interesting one and is invisible without Tautulli. - *Size vs. last watched* — scatter, log-size on Y, days-since-watch on X. Points in the upper right are the reclaim targets, and clicking a region filters the grid to it. @@ -769,8 +961,9 @@ never-watched content, size of content untouched for 2+ years, last scan time), **Saved views** — list, create, edit, duplicate, run. Creating one captures the current filter, sort, columns, and weights, so the flow is *explore, then name what you found*. -**Item detail** — everything known: all parts with paths and sizes, per-account watch -history timeline, score breakdown showing each component's contribution. +**Item detail** — everything known: all parts with paths and sizes, a per-account watch +timeline showing each play with how far it got (a row of 8%, 12%, 6% tells the story at a +glance), and the score breakdown showing each component's contribution. **Scans** — history table, per-scan counts and duration, a "Scan now" button, live progress. @@ -802,6 +995,14 @@ All via environment variables, so the Portainer stack is the single source of tr | `PLEX_TIMEOUT_S` | `30` | | | `PLEX_PAGE_SIZE` | `500` | | | `PLEX_REQUEST_DELAY_MS` | `0` | throttle for Loki's sake | +| `TAUTULLI_BASE_URL` | — | e.g. `http://192.168.1.100:8181` — unset disables Tautulli | +| `TAUTULLI_API_KEY` | — | secret | +| `TAUTULLI_TIMEOUT_S` | `30` | | +| `TAUTULLI_PAGE_SIZE` | `1000` | `length` on `get_history` | +| `HISTORY_SOURCE` | `auto` | `auto` \| `tautulli` \| `plex` — `auto` prefers Tautulli, falls back | +| `SESSION_MERGE_WINDOW_H` | `6` | events merged into one viewing (§4.9) | +| `COMPLETION_THRESHOLD` | `85` | percent complete counting as a watch | +| `ABANDON_CEILING` | `15` | percent complete below which a play is "abandoned" | | `DATABASE_PATH` | `/data/mediashelf.db` | on the mounted volume | | `SCAN_SCHEDULE_CRON` | `0 4 * * *` | nightly incremental at 04:00 | | `SCAN_FULL_SWEEP_CRON` | `0 3 * * 0` | weekly full sweep, Sunday 03:00 | @@ -810,14 +1011,17 @@ All via environment variables, so the Portainer stack is the single source of tr | `SCORE_STALE_HORIZON_DAYS` | `730` | | | `SCORE_AGE_HORIZON_DAYS` | `1095` | | | `SCORE_POPULAR_AT` | `3` | | +| `SCORE_REJECTED_AT` | `2` | abandoned views for a full `rejection` score | | `SCORE_GRACE_DAYS` | `30` | | | `SCORE_RECENT_DAYS` | `90` | | | `TZ` | `America/Regina` | so cron times mean what they look like | | `LOG_LEVEL` | `INFO` | | -`PLEX_TOKEN` is a secret and must not be committed. The repo ships `.env.example` with -placeholders, and `.env` is gitignored. (Worth noting given the SOAP-password-in-history -problem on the Mythica repo — this one starts clean and stays clean.) +`PLEX_TOKEN` and `TAUTULLI_API_KEY` are secrets and must not be committed. The repo ships +`.env.example` with placeholders, and `.env` is gitignored. (Worth noting given the +SOAP-password-in-history problem on the Mythica repo — this one starts clean and stays +clean.) Neither credential is ever written to the database, the logs, or an API response; +`/api/v1/settings` redacts both. --- @@ -828,12 +1032,30 @@ problem on the Mythica repo — this one starts clean and stays clean.) Single image, `python:3.12-slim` base, multi-stage so build deps don't ship. Runs as a non-root user. One volume: `/data`. One port: 8080. -### 11.2 Portainer stack on Nox +### 11.2 Getting the image onto Nox + +The image is built off-box and delivered to Nox's Portainer rather than built there. Two +workable routes, in order of preference: + +1. **Registry** — build, tag `registry.hallsworth.ca/mediashelf:` (or Forgejo's + own container registry, which this Forgejo version supports), push, and have the + Portainer stack pull it. This is the route worth setting up once: redeploys become a + version bump in the stack, and rollback is pulling the previous tag. +2. **Image upload** — `docker save mediashelf: | gzip` and load it on Nox via + Portainer's *Images → Import*. No registry needed, fine for the first deploy or two, + but it makes rollback manual and versions easy to lose track of. + +Either way the image is **tagged with a real version**, never deployed as bare `latest`. +The Mythica stack had to be torn down and recreated because a `PUT` update kept serving +old code; immutable version tags are how that failure mode is avoided rather than +worked around. + +### 11.3 Portainer stack on Nox ```yaml services: mediashelf: - image: mediashelf:latest + image: registry.hallsworth.ca/mediashelf:0.1.0 # never bare :latest, see 11.2 container_name: mediashelf restart: unless-stopped ports: @@ -844,6 +1066,9 @@ services: - MEDIASHELF_SECRET_KEY=${MEDIASHELF_SECRET_KEY} - PLEX_BASE_URL=${PLEX_BASE_URL} - PLEX_TOKEN=${PLEX_TOKEN} + - TAUTULLI_BASE_URL=${TAUTULLI_BASE_URL} + - TAUTULLI_API_KEY=${TAUTULLI_API_KEY} + - HISTORY_SOURCE=auto - DATABASE_PATH=/data/mediashelf.db - SCAN_SCHEDULE_CRON=${SCAN_SCHEDULE_CRON} - TZ=America/Regina @@ -866,7 +1091,7 @@ Published through Nginx Proxy Manager at `mediashelf.hallsworth.ca`, upstream `192.168.1.77:8085`. Note the host-port → container-port mapping is 8085 → 8080; NPM's upstream must point at the **host** port. -### 11.3 The gunicorn / scheduler constraint +### 11.4 The gunicorn / scheduler constraint APScheduler runs in-process. With `--workers 2`, both workers would start a scheduler and two scans would race for the SQLite write lock. Mitigation: the scheduler starts only in @@ -874,12 +1099,13 @@ the worker that successfully acquires an exclusive advisory lock on a lockfile i at boot. The `scan_lock` table is the second line of defence. Both are needed; either alone has a race window. -### 11.4 Backup +### 11.5 Backup The database is one file. A weekly `sqlite3 .backup` to the volume, keeping four, is -enough — everything in it is reconstructible from Plex except accumulated `watch_event` -rows beyond Plex's own history retention (§4.5), which is precisely the part worth -backing up. +enough — everything in it is reconstructible from Plex and Tautulli, so the real value +is avoiding a multi-hour re-ingest rather than protecting irreplaceable data. The one +genuinely irreplaceable thing is the saved-view definitions, which are small and worth +exporting to the repo as JSON once they stabilise. --- @@ -915,7 +1141,8 @@ a misconfigured proxy rule. | Layer | Approach | |---|---| -| Provider | Recorded Plex JSON fixtures (`responses`/`vcr.py`) covering a movie section, a show section with multi-season shows, multi-version movies, split parts, and a history page. No live server needed in CI. | +| Provider | Recorded JSON fixtures (`responses`/`vcr.py`) for both sources: Plex covering a movie section, a show section with multi-season shows, multi-version movies, split parts; Tautulli covering `get_history` pages, `get_users`, `get_library_media_info`, and an error-shaped 200 response. No live server needed in CI. | +| History semantics | Disposition classification at threshold boundaries; session merging across the window; a `pre_history` item scoring differently from a truly-never-watched one; Plex-fallback rows renormalizing the weights correctly. | | Normalization | Table-driven tests: odd cases like an item with no `Part`, a season with a missing episode, unicode titles, zero-byte parts. | | Ingest | In-memory SQLite; assert idempotency (running the same scan twice changes nothing), assert `missing` transitions, assert history watermarking doesn't skip or duplicate events. | | Scoring | Unit tests on known inputs; **property test asserting the SQL and Python implementations agree** (§6.4); tests that grace rules clamp correctly. | @@ -965,17 +1192,26 @@ class MediaProvider(Protocol): def server_info(self) -> ServerInfo: ... def libraries(self) -> list[Library]: ... def items(self, library: Library) -> Iterator[Item]: ... # yields normalized items - def watch_events(self, since: int | None) -> Iterator[WatchEvent]: ... - def accounts(self) -> list[Account]: ... def refresh_library(self, library: Library) -> None: ... # v2 only + +class HistoryProvider(Protocol): + def server_info(self) -> ServerInfo: ... # for the pms_identifier match, §4.11 + def accounts(self) -> list[Account]: ... + def watch_events(self, since: int | None) -> Iterator[WatchEvent]: ... + def coverage(self) -> Coverage: ... # earliest/latest event, count + @property + def has_completion_data(self) -> bool: ... # False for PlexHistoryProvider ``` Jellyfin and Emby share an API lineage, so one `JellyfinProvider` will likely cover both with a capability flag or two. The mapping is close: `Items` with `Fields=Path,MediaSources,DateCreated`, `UserData.PlayCount` and `LastPlayedDate` for -watch data, `/Users` for accounts. The notable difference is that Jellyfin's watch data -is genuinely per-user and must be fetched per user and merged, rather than read from one -global history endpoint. +watch data, `/Users` for accounts. Two differences matter. Their watch data is genuinely +per-user and must be fetched per user and merged, rather than read from one global history +endpoint. And there is no Tautulli equivalent, so `has_completion_data` is `False` and the +`rejection` component drops out — the same degraded path the Plex fallback already +exercises, which is a good reason for that path to exist and be tested from day one +rather than bolted on later. The `provider` table already carries `kind` and the schema already keys everything on `(provider_id, provider_item_id)`, so multiple servers can coexist in one database — the @@ -989,24 +1225,41 @@ door is open to a combined view across Plex and Jellyfin without a migration. and invisible source of waste. - **Trend tracking** — MediaShelf already snapshots on a schedule; a chart of library size over time and a projection of when the vault fills up is nearly free. -- **Tautulli import** — if Tautulli is running, its history database predates Plex's own - retention window and would deepen the watch record considerably. +- **Direct Tautulli database read** — Tautulli's SQLite file could be read directly + instead of paging its API, which would make the first full history ingest dramatically + faster. Only worth doing if the API ingest proves slow, and it couples MediaShelf to + Tautulli's internal schema, so the API stays the default. +- **Per-user reclaim views** — the data to answer "what does only one person watch" + is already stored; it just has no UI in v1. - **Notifications** — a monthly "here's what's gone cold" summary. --- ## 15. Open questions -1. **Plex history retention on Loki.** If it's short, the first ingest captures less than - hoped and `watch_event` becomes valuable immediately. Worth checking before the first - scan, and worth knowing whether Tautulli has been running (§14.3). -2. **Multiple Plex libraries of the same kind** — are there several movie sections (e.g. - Movies, Kids, Documentaries)? The design handles it, but the seed views and default - charts get more useful if the real shape is known. -3. **Path structure on the vaults** — knowing the actual roots would let the grid group by - physical vault, which matters when the goal is freeing a *specific* array rather than - space in general. -4. **Where does the image get built?** Nox running Portainer suggests either a build from - a git URL in the stack, or pushing to a registry. Worth settling before the first - deploy. -5. **Is Tautulli running anywhere?** It changes the watch-history story materially. +### Resolved + +- ~~Is Tautulli running anywhere?~~ **Yes** — 192.168.1.100:8181, on Isis. Promoted to + the primary watch-history source; see §4.8–4.11. +- ~~Plex history retention on Loki~~ — no longer on the critical path, since Tautulli + keeps its own independent record. +- ~~Where does the image get built?~~ Built off-box, delivered to Nox's Portainer (§11.2). + +### Still open + +1. **When was Tautulli installed?** This sets `history_coverage_since` and therefore how + many older items get the `pre_history` flag. If it went in recently, a large slice of + the library will look never-watched on the first report and the flag is doing real + work; if it has been running for years, it barely matters. A single + `get_history&order_dir=asc&length=1` answers it. +2. **Is *Group Successive Play History* on in Tautulli's settings?** MediaShelf requests + `grouping=0` and does its own merging (§4.9), so it should not matter — but confirming + the setting on the first run rules out a whole class of double-counting bug. +3. **Multiple Plex libraries of the same kind** — are there several movie sections (Movies, + Kids, Documentaries)? The design handles it, but the seed views and default charts get + more useful if the real shape is known. +4. **Path structure on the vaults** — knowing the actual roots would let the grid group by + physical vault, which matters when the goal is freeing a *specific* array (vault 2) + rather than space in general. +5. **Registry or image upload?** (§11.2) Setting up a registry is the better long-term + answer but is a piece of infrastructure that doesn't exist yet.