Jess Hallsworth
58c2883492
The keep_all capability stays at library level, but no library is marked by default. KEEP_ALL_LIBRARIES defaults to empty rather than naming Family Videos, and the design says plainly that every keep in the system got there because a person put it there. A keep the application invented would be indistinguishable in the database from one the user made, which undermines the audit trail the Kept view exists to provide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVbG48GAXfCZatcmX123Ra
1669 lines
82 KiB
Markdown
1669 lines
82 KiB
Markdown
# MediaShelf — Software Design
|
||
|
||
**Status:** Draft v1.3 · **Author:** Claude (spec) for Jess · **Date:** 2026-09-07
|
||
|
||
*Revision 1.1 — Tautulli (192.168.1.100:8181) confirmed present and promoted to the
|
||
primary watch-history source; Plex's own history demoted to fallback. Deployment path
|
||
settled: image built in CI/locally and pushed to Nox's Portainer.*
|
||
|
||
*Revision 1.3 — keep marks (§6.6) generalised from the earlier protected-libraries idea:
|
||
one `keep` concept applied at library, show, season or movie level, keyed on content guid
|
||
rather than Plex rating key so marks survive library rebuilds, kept items hidden from the
|
||
grid by default, and refused outright by v2 deletion.*
|
||
|
||
*Revision 1.2 — **validated against the live servers.** The measured library (§2.1) is
|
||
65.7 TB across 24 libraries with 60 users and 87,640 logged plays. Five corrections
|
||
followed: Tautulli's per-item table cannot supply TV sizes (§4.11), both Tautulli-derived
|
||
score weights recalibrated against real distributions (§6.1), cross-library duplicates
|
||
promoted to v1 (§6.5), keep marks added (§6.6), and confident-vs-uncertain
|
||
reclaim separated as distinct seed views (§7.2).*
|
||
|
||
---
|
||
|
||
## 1. Overview
|
||
|
||
MediaShelf is a self-hosted web application that reads a Plex Media Server library,
|
||
builds a local analytical snapshot of every item in it, and gives a human a fast way
|
||
to answer one question:
|
||
|
||
> *Which of these thousands of files are worth keeping, and which are burning disk
|
||
> for nothing?*
|
||
|
||
It does this by joining together facts Plex already knows but never shows side by
|
||
side — when a file was added, how large it is, where it lives on disk, which library
|
||
it belongs to, how many times it has been played, and when it was last played — and
|
||
exposing them as a sortable, filterable, chartable grid with a tunable "reclaim
|
||
score" that ranks deletion candidates.
|
||
|
||
**v1 is read-only.** MediaShelf will never delete, move, or modify a file or a Plex
|
||
record in this version. Its output is a report: an on-screen ranked list, saved rule
|
||
sets, and CSV export. Actual deletion is designed for in §14 but deliberately not
|
||
built yet, so the scanner and scoring model can be trusted before anything
|
||
destructive is wired up.
|
||
|
||
### 1.1 Goals
|
||
|
||
- Full-library ingest from Plex over its HTTP API, with no agent installed on the Plex host.
|
||
- Watch history from **Tautulli**, which has been logging every play since it was
|
||
installed and knows *how much* of each item was actually watched — not just that a
|
||
play started.
|
||
- A durable local snapshot so the UI is instant and Plex isn't hammered on every click.
|
||
- Movies at item level; TV rolled up to **season** level.
|
||
- Sort/filter/visualize on: date added, last watched, watch count, size on disk, file path, library.
|
||
- A weighted **reclaim score** with live sliders, so "what should I delete" is one sort click.
|
||
- **Saved views** — named, re-runnable rule sets like *unwatched, older than 2 years, over 10 GB*.
|
||
- CSV export of any view.
|
||
- A provider abstraction clean enough that Emby and Jellyfin drop in later without
|
||
touching the scoring, storage, or UI layers.
|
||
|
||
### 1.2 Non-goals (v1)
|
||
|
||
- Deleting, moving, or trashing files. Not in v1. See §14.1.
|
||
- Writing anything back to Plex (no rating changes, no collection edits, no unmatch/refresh).
|
||
- Authentication or multi-user accounts — this is a LAN-only, single-admin tool. See §12.
|
||
- Music, photo, and "other video" libraries. Movie and Show sections only.
|
||
- Direct filesystem access. MediaShelf never mounts the media shares. All size and path
|
||
data comes from the Plex API. See §4.4.
|
||
- Transcode/quality analysis, duplicate detection, subtitle management.
|
||
|
||
### 1.3 Decisions already made
|
||
|
||
| Question | Decision |
|
||
|---|---|
|
||
| Deletion in v1 | No — report only |
|
||
| Stack | Python + Flask, SQLite |
|
||
| Deployment | Portainer stack on **Nox** (192.168.1.77), published via Nginx Proxy Manager |
|
||
| Auth | None; LAN-only, single admin view, watch data aggregated across all Plex accounts |
|
||
| TV granularity | Season |
|
||
| Size/path source | Plex API only |
|
||
| Watch-history source | **Tautulli** primary, Plex session history as fallback |
|
||
| v1 output | Sortable grid + filters + charts, tunable reclaim score, saved rule sets |
|
||
| Image delivery | Built outside Nox, pushed to Nox's Portainer |
|
||
| Implementation | Design doc only for now; code to follow |
|
||
|
||
---
|
||
|
||
## 2. Context
|
||
|
||
The Plex server is **Loki** (192.168.1.10, Dell R710, Ubuntu 16.04, Plex 1.43.3) with
|
||
roughly 90 TB of RAID-6 storage across a Dell MD1000 vault. A second vault is powered
|
||
down awaiting drive upgrades, so reclaiming space on the live vault has real value —
|
||
this tool exists because buying disks is the more expensive alternative.
|
||
|
||
MediaShelf runs on **Nox**, not on Loki. Loki's job is serving media; MediaShelf is a
|
||
polite HTTP client that talks to it across the LAN and keeps its own database locally.
|
||
This also means MediaShelf survives Loki reboots, Plex upgrades, and the eventual
|
||
migration of services onto Enki.
|
||
|
||
### 2.1 The library, measured
|
||
|
||
Everything below was measured against the live servers on 2026-09-07 with
|
||
`tools/probe.py` and `tools/reclaim_preview.py` (§13.1). These are not estimates.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Total size | **65.7 TB** |
|
||
| Libraries | **24** — 16 movie sections, 8 show sections |
|
||
| Movies | 2,930 across 16 sections |
|
||
| TV | 44,455 episodes → **2,807 seasons** across 664 shows |
|
||
| Grid rows at season granularity | **5,727** |
|
||
| Files (parts) | ~47,900 |
|
||
| Multi-file items | 640, concentrated in Documentary (338 of 2,511 parts) |
|
||
| Storage roots | `/mnt/titan4` 40.6 TB · `/mnt/titan3` 24.3 TB · `/mnt/titan1` 849 GB |
|
||
| Tautulli history | 87,640 plays since **2025-03-08**, 24,204 distinct items, 60 users |
|
||
| Never played | **27.4 TB (41.7%)** — of which 6.5 TB confident, 20.9 TB uncertain |
|
||
|
||
Four things here materially changed the design.
|
||
|
||
**This is not a household server.** 60 distinct users appear in the history. Every
|
||
component calibrated for a family of three or four had to be recalibrated (§6.1), and
|
||
"who else watches this" becomes a real signal rather than a rounding error.
|
||
|
||
**80% of the library predates Tautulli.** Coverage begins 2025-03-08, but content goes
|
||
back to 2000. So of the 27.4 TB never played, only 6.5 TB is *confidently* unwatched —
|
||
added while Tautulli was watching and never touched since. The other 20.9 TB might have
|
||
been watched in 2019 and nobody can tell. The `pre_history` flag (§4.11) is therefore not
|
||
an edge case, it is the majority state, and the UI must make the distinction impossible
|
||
to miss.
|
||
|
||
**24 libraries, not two.** Several are genre splits of the same kind of content (Horror,
|
||
Comedy, War, Superhero, Classic), and *Movies* / *4K Movies* frequently hold the same
|
||
film twice. Cross-library duplicate detection moves from "later candidate" to a real v1
|
||
concern (§6.5).
|
||
|
||
**Some content must never be a deletion candidate, and no data can tell you which.**
|
||
*Family Videos* is 63 items and 38 GB of what looks like home video, 86% of it "never
|
||
played" and therefore ranked highly by a naive reclaim score. The same applies for reasons
|
||
the tool cannot see at all: content kept deliberately in case someone wants it later, or
|
||
held at 4K because re-acquiring it is expensive. Keep marks are a first-class concept
|
||
(§6.6) — and because the tool cannot infer any of this, it does not try: nothing is kept
|
||
until a person says so.
|
||
|
||
---
|
||
|
||
## 3. Architecture
|
||
|
||
```
|
||
┌──────────────────────────── Nox (192.168.1.77) ────────────────────────────┐
|
||
│ │
|
||
│ ┌─────────────────── mediashelf container ───────────────────┐ │
|
||
│ │ │ │
|
||
│ │ Flask app (gunicorn, 2 workers) │ │
|
||
│ │ ├── / → UI (server-rendered shell + JS) │ │
|
||
│ │ ├── /api/* → JSON API │ │
|
||
│ │ └── /healthz │ │
|
||
│ │ │ │
|
||
│ │ APScheduler (in-process, 1 worker only) │ │
|
||
│ │ └── nightly scan job │ │
|
||
│ │ │ │
|
||
│ │ Ingestion pipeline │ │
|
||
│ │ PlexProvider ──▶ normalize ──▶ upsert ──▶ rollup │ │
|
||
│ │ │ │
|
||
│ │ SQLite /data/mediashelf.db (WAL) │ │
|
||
│ └────────────────────────────┬───────────────────────────────┘ │
|
||
│ │ volume: /data │
|
||
└──────────┬─────────────────────┴───────────────────────────────────────────┘
|
||
│ │
|
||
│ HTTP :32400 + X-Plex-Token │ HTTP :8181 + apikey
|
||
▼ ▼
|
||
Loki (192.168.1.10) Isis (192.168.1.100:8181)
|
||
Plex Media Server Tautulli
|
||
· libraries, items, parts · every play since install
|
||
· size, path, added_at · per-user, per-item, % complete
|
||
· session history (fallback)
|
||
```
|
||
|
||
### 3.1 Component responsibilities
|
||
|
||
**`mediashelf.providers`** — the pluggable source layer. Defines two protocols:
|
||
`MediaProvider` (libraries, items, parts) implemented by `PlexProvider`, and
|
||
`HistoryProvider` (watch events) implemented by both `TautulliHistoryProvider` and
|
||
`PlexHistoryProvider`. Everything above this layer speaks only in MediaShelf's own
|
||
normalized dataclasses. Splitting the two protocols is what lets watch history come
|
||
from a different box than the library itself — which, given Tautulli lives on Isis and
|
||
Plex on Loki, is not hypothetical. It is also the seam Emby/Jellyfin slot into (§14.2).
|
||
|
||
**`mediashelf.ingest`** — orchestrates a scan: pull sections, pull items, pull history,
|
||
normalize, upsert into SQLite inside a transaction, recompute season rollups, mark
|
||
vanished items, write a `scan` row.
|
||
|
||
**`mediashelf.scoring`** — pure functions. Takes a row of metrics plus a weight profile,
|
||
returns a 0–100 reclaim score and the component breakdown. No I/O, fully unit-testable.
|
||
|
||
**`mediashelf.rules`** — evaluates saved-view rule sets into SQL `WHERE` fragments via a
|
||
whitelisted field/operator grammar. Never string-interpolates user input into SQL.
|
||
|
||
**`mediashelf.api`** — Flask blueprints returning JSON.
|
||
|
||
**`mediashelf.web`** — one server-rendered HTML shell; the grid, charts, and sliders are
|
||
client-side JS against `/api`.
|
||
|
||
### 3.2 Why Flask + SQLite
|
||
|
||
The dataset is small in database terms. A library of ~5,000 movies and ~2,000 seasons
|
||
across ~60,000 episodes is a few tens of megabytes. SQLite in WAL mode handles this with
|
||
sub-10 ms queries on indexed columns, needs no second container, and backs up by copying
|
||
one file. Flask matches the pattern already running for the Mythica portal on Nox, so
|
||
deployment, logging, and reverse-proxy conventions carry over unchanged.
|
||
|
||
The one real constraint SQLite imposes is a single writer. Since the only writer is the
|
||
scan job, and scans run one at a time behind a lock, this is not a limitation. It does
|
||
mean **gunicorn must run the scheduler in exactly one worker** — see §11.3.
|
||
|
||
---
|
||
|
||
## 4. Data sources
|
||
|
||
MediaShelf reads from two servers. **Plex on Loki** is the authority on what exists —
|
||
libraries, items, files, sizes, paths, added dates. **Tautulli on Isis** is the authority
|
||
on what has been watched. Sections 4.1–4.7 cover Plex; 4.8–4.11 cover Tautulli.
|
||
|
||
### 4.1 Plex connection
|
||
|
||
MediaShelf authenticates with a Plex server token (`X-Plex-Token`) supplied via
|
||
environment variable. It talks directly to `http://192.168.1.10:32400` on the LAN — no
|
||
plex.tv round trip, no account login, no dependency on Plex's cloud being up.
|
||
|
||
All requests carry standard client identification headers so the connection is
|
||
identifiable in Plex's own logs:
|
||
|
||
```
|
||
X-Plex-Token: <token>
|
||
X-Plex-Client-Identifier: mediashelf-<stable-uuid>
|
||
X-Plex-Product: MediaShelf
|
||
X-Plex-Version: <app version>
|
||
Accept: application/json
|
||
```
|
||
|
||
Plex returns JSON when `Accept: application/json` is set, which avoids XML parsing
|
||
entirely. Every response body is wrapped in a `MediaContainer` object.
|
||
|
||
### 4.2 Endpoints used
|
||
|
||
| Purpose | Endpoint |
|
||
|---|---|
|
||
| Server identity / version | `GET /identity` |
|
||
| List libraries | `GET /library/sections` |
|
||
| Library detail + on-disk locations | `GET /library/sections/{id}` |
|
||
| All movies in a section | `GET /library/sections/{id}/all?type=1` |
|
||
| All episodes in a show section | `GET /library/sections/{id}/all?type=4` |
|
||
| All seasons in a show section | `GET /library/sections/{id}/all?type=3` |
|
||
| Global watch history | `GET /status/sessions/history/all` |
|
||
| Accounts (for history attribution) | `GET /accounts` |
|
||
|
||
Pagination on every list endpoint uses `X-Plex-Container-Start` and
|
||
`X-Plex-Container-Size` headers (or the equivalent query params). MediaShelf pages at
|
||
**500 items** per request; the `MediaContainer.totalSize` field on the first response
|
||
gives the loop its bound.
|
||
|
||
### 4.3 The fields that matter
|
||
|
||
From each `Video` element on `/library/sections/{id}/all`:
|
||
|
||
| Plex attribute | Meaning | MediaShelf column |
|
||
|---|---|---|
|
||
| `ratingKey` | stable per-server item id | `provider_item_id` |
|
||
| `guid` | cross-server content identity | `guid` |
|
||
| `type` | `movie` / `episode` / `season` / `show` | `kind` |
|
||
| `title`, `year` | display | `title`, `year` |
|
||
| `addedAt` | epoch seconds, when Plex first saw the file | `added_at` |
|
||
| `updatedAt` | epoch seconds, last metadata change | `updated_at` |
|
||
| `duration` | milliseconds | `duration_ms` |
|
||
| `viewCount` | plays — **token-account scoped**, see §4.5 | (advisory only) |
|
||
| `lastViewedAt` | epoch seconds — token-account scoped | (advisory only) |
|
||
| `librarySectionID` | owning library | `library_id` |
|
||
| `grandparentRatingKey`, `grandparentTitle` | (episodes) the show | `show_id`, `show_title` |
|
||
| `parentRatingKey`, `parentIndex` | (episodes) the season and its number | `season_id`, `season_number` |
|
||
|
||
From the nested `Media[].Part[]` elements:
|
||
|
||
| Plex attribute | MediaShelf column |
|
||
|---|---|
|
||
| `Part.file` | `file_path` — absolute path on the Plex host |
|
||
| `Part.size` | `size_bytes` |
|
||
| `Part.container` | `container` |
|
||
| `Media.videoResolution`, `videoCodec`, `audioCodec`, `bitrate` | `resolution`, `video_codec`, `audio_codec`, `bitrate` |
|
||
|
||
An item can have several `Media` entries (multiple versions) each with several `Part`
|
||
entries (split files). MediaShelf stores **every part as its own row** in `media_part`
|
||
and treats an item's size as the sum of its parts. This matters: a movie with a 4K and a
|
||
1080p version is one item occupying two files, and a reclaim report that reported only
|
||
the first part would understate the win.
|
||
|
||
### 4.4 Why Plex-only, and what it costs
|
||
|
||
MediaShelf does not mount the media shares. That keeps the container trivially portable,
|
||
removes any possibility of it touching a file by accident, and means it works from
|
||
anywhere that can reach port 32400.
|
||
|
||
The accepted trade-offs:
|
||
|
||
- **Size is Plex's number, not the filesystem's.** Plex records `Part.size` at scan
|
||
time. If a file is replaced outside Plex and Plex hasn't rescanned, the number is
|
||
stale. In practice Plex's own scanner keeps this current; the design accepts drift.
|
||
- **Orphans are invisible.** A file on disk that Plex has never indexed will not appear
|
||
in MediaShelf at all. Finding those needs a filesystem walk, which is explicitly out
|
||
of scope (see §14.3 for the optional-mount future).
|
||
- **Paths are Loki-relative.** `file_path` is the path as the Plex server sees it. It is
|
||
displayed and exported for a human's benefit, not resolved by MediaShelf.
|
||
|
||
### 4.5 Why Plex is not the watch-data source
|
||
|
||
Two problems make Plex's own watch data unfit to build a reclaim tool on.
|
||
|
||
**It is scoped to one account.** `viewCount` and `lastViewedAt` on a library item reflect
|
||
only the account that owns the token. Plays by home users, managed users, and shared users
|
||
are invisible. For a household library this understates watches badly, and a tool that
|
||
recommends deleting "unwatched" content someone else watches weekly is worse than useless.
|
||
|
||
**It only knows that a play started.** `/status/sessions/history/all` records a playback
|
||
event. It does not record whether the viewer watched the whole thing or bailed after four
|
||
minutes. Those two events mean opposite things for a deletion decision, and Plex reports
|
||
them identically.
|
||
|
||
`/status/sessions/history/all` solves the first problem — it is server-wide across all
|
||
accounts — so it remains the **fallback** source, and §4.11 covers when it is used. But
|
||
Tautulli solves both, so Tautulli is the primary.
|
||
|
||
### 4.6 Plex scan strategy
|
||
|
||
**Full scan** — walks every section, every item, every part, and the complete history.
|
||
Run on first launch and on demand. Expect a few minutes for a large library.
|
||
|
||
**Incremental scan** — the nightly default. Two cheap tricks:
|
||
|
||
1. Items: request the section sorted by `addedAt:desc` and stop paging once the page's
|
||
oldest `addedAt` predates the last successful scan. This catches additions. To catch
|
||
*removals* and metadata edits, a full item sweep still runs weekly (configurable) —
|
||
an item silently vanishing from Plex is exactly the event that must not be missed.
|
||
2. History: request `/status/sessions/history/all` sorted by `viewedAt:desc` and stop at
|
||
the last-seen `viewedAt` watermark.
|
||
|
||
**Concurrency** — one scan at a time, enforced by a row in a `scan_lock` table plus an
|
||
in-process lock. A scan that dies leaves a stale lock; locks older than
|
||
`SCAN_LOCK_TIMEOUT` (default 2 h) are broken automatically and the orphaned scan row is
|
||
marked `failed`.
|
||
|
||
**Politeness** — requests are issued serially with a small configurable delay
|
||
(`PLEX_REQUEST_DELAY_MS`, default 0) and a hard timeout. Loki is a 2010-vintage R710;
|
||
the ingest is designed to be slow and steady rather than parallel and aggressive.
|
||
|
||
### 4.7 Failure handling
|
||
|
||
| Failure | Behaviour |
|
||
|---|---|
|
||
| Plex unreachable | Scan marked `failed` with the error; UI keeps serving the last good snapshot with a staleness banner |
|
||
| Token rejected (401) | Scan fails fast, prominent UI banner, no retry loop |
|
||
| Section disappears | Its items are marked `missing` rather than deleted (§5.4) |
|
||
| Malformed / partial item | Item skipped, logged, counted in `scan.warning_count`; the scan still succeeds |
|
||
| Timeout mid-page | Up to 3 retries with exponential backoff, then the scan fails |
|
||
|
||
### 4.8 Tautulli — the watch-history source
|
||
|
||
Tautulli runs at `http://192.168.1.100:8181` (on Isis) and has been logging every
|
||
playback on Loki since the day it was installed. **It stays LAN-only and is not published
|
||
externally** — which costs nothing here, since MediaShelf runs on Nox on the same network.
|
||
The practical consequence is that design validation happens on the LAN: see
|
||
`tools/probe.py` (§13.1). Its database is independent of Plex's:
|
||
clearing Plex's history, or Plex pruning its own, does not touch it. That makes it a
|
||
deeper and more durable record than anything Plex exposes.
|
||
|
||
Its API is a single endpoint with a `cmd` parameter:
|
||
|
||
```
|
||
GET http://192.168.1.100:8181/api/v2?apikey=<key>&cmd=<command>&<params>
|
||
```
|
||
|
||
Every response is `{"response": {"result": "success"|"error", "message": ..., "data": ...}}`.
|
||
A transport-level 200 with `result: "error"` in the body is the normal failure mode, so
|
||
**the result field must be checked on every call** — treating HTTP 200 as success is the
|
||
mistake this API invites.
|
||
|
||
| Purpose | Command |
|
||
|---|---|
|
||
| Reachability / version | `cmd=get_server_info`, `cmd=arnold` (returns a quote; a cheap liveness ping) |
|
||
| Which Plex server it watches | `cmd=get_server_info` → `pms_identifier` |
|
||
| Users | `cmd=get_users` |
|
||
| Libraries | `cmd=get_libraries` |
|
||
| **Watch history** | `cmd=get_history` |
|
||
| Per-item size / play count (cross-check) | `cmd=get_library_media_info` |
|
||
|
||
### 4.9 Ingesting history
|
||
|
||
```
|
||
cmd=get_history
|
||
&grouping=0 # do NOT collapse successive plays — we want raw events
|
||
&order_column=date
|
||
&order_dir=desc
|
||
&start=<offset>
|
||
&length=1000
|
||
&after=<YYYY-MM-DD> # incremental watermark
|
||
```
|
||
|
||
`grouping=0` matters. Tautulli's *Group Successive Play History* setting merges a paused-
|
||
and-resumed play into one row for display purposes. For counting distinct viewings that
|
||
is what we want — but the grouping should be MediaShelf's decision, applied consistently,
|
||
not a mirror of however that setting happens to be configured on Isis. So MediaShelf pulls
|
||
ungrouped events and does its own session collapsing: **events for the same item by the
|
||
same user whose start times fall within `SESSION_MERGE_WINDOW` (default 6 hours) count as
|
||
one viewing.**
|
||
|
||
The fields MediaShelf keeps from each history row:
|
||
|
||
| Tautulli field | Use |
|
||
|---|---|
|
||
| `row_id` | idempotency key for upsert |
|
||
| `reference_id` | Tautulli's own grouping id, kept for cross-checking |
|
||
| `date` / `started` / `stopped` | epoch seconds |
|
||
| `rating_key` | the leaf item — movie or episode |
|
||
| `parent_rating_key`, `grandparent_rating_key` | season and show, for rollup |
|
||
| `user_id`, `user`, `friendly_name` | who watched |
|
||
| `media_type` | `movie` / `episode` |
|
||
| `percent_complete` | **how much was actually watched** |
|
||
| `watched_status` | Tautulli's own verdict against its configured threshold |
|
||
| `play_duration`, `paused_counter` | real time spent, minus pauses |
|
||
| `platform`, `player` | stored, unused in v1 |
|
||
|
||
### 4.10 What "watched" means
|
||
|
||
This is the part that most changes the quality of the output, so MediaShelf defines it
|
||
itself rather than inheriting Tautulli's threshold setting:
|
||
|
||
| Derived value | Rule |
|
||
|---|---|
|
||
| **completed view** | `percent_complete >= COMPLETION_THRESHOLD` (default 85) |
|
||
| **partial view** | `percent_complete` between `ABANDON_CEILING` (default 15) and the completion threshold |
|
||
| **abandoned view** | `percent_complete < ABANDON_CEILING` — started and bailed |
|
||
|
||
MediaShelf uses `percent_complete` rather than `watched_status` as the primary signal
|
||
because `watched_status` depends on how the threshold is configured on Isis, and that
|
||
configuration can change without warning. `watched_status` is stored anyway and shown in
|
||
the UI as a cross-check; a systematic disagreement between the two is worth seeing.
|
||
|
||
The rollups per item become:
|
||
|
||
- `watch_count` — **completed** views only
|
||
- `partial_count`, `abandoned_count`
|
||
- `last_watched_at` — most recent completed view
|
||
- `last_touched_at` — most recent view of any kind, including abandoned
|
||
- `first_watched_at`, `distinct_watcher_count`
|
||
- `avg_percent_complete`
|
||
|
||
**An abandoned view is evidence for deletion, not against it.** Three people started a
|
||
film and all three quit twenty minutes in: that is a stronger signal to delete than a film
|
||
nobody ever opened, because the household has now actively rejected it. Plex's history
|
||
cannot express this at all, and it is the single biggest reason to prefer Tautulli. The
|
||
scoring model uses it (§6.1, `rejection` component).
|
||
|
||
### 4.11 Fallback, coverage, and cross-checks
|
||
|
||
**When Tautulli is unavailable**, MediaShelf falls back to `PlexHistoryProvider` and
|
||
`/status/sessions/history/all`. Events ingested that way are marked `source = 'plex'` and
|
||
carry no `percent_complete`, so they count as completed views (the only assumption
|
||
available) and are excluded from the `rejection` component. The UI shows a banner
|
||
explaining the score is running in degraded mode, because a silently degraded score is a
|
||
score that gets trusted when it shouldn't be.
|
||
|
||
**Tautulli's coverage horizon.** Tautulli only knows about plays since it was installed.
|
||
Anything watched before that is invisible to it — and those items will look never-watched.
|
||
On first ingest MediaShelf records `history_coverage_since` = the oldest event Tautulli
|
||
holds, and:
|
||
|
||
- Items with `added_at` earlier than that date get a `pre_history` flag.
|
||
- The UI shows a badge on those rows and a one-line explanation on the dashboard.
|
||
- The `staleness` score component is **capped** for `pre_history` items, so a 2009 movie
|
||
that was watched in 2015 and never since doesn't get scored as though it had never been
|
||
watched at all.
|
||
|
||
**Cross-check with `get_library_media_info` — movie libraries only.** Tautulli maintains
|
||
its own per-item table with `file_size`, `added_at`, `play_count`, and `last_played`.
|
||
For **movie** sections this is a useful second opinion: MediaShelf pulls it once per scan
|
||
per library and records size disagreements against Plex as scan warnings, not errors.
|
||
|
||
For **show** sections it is unusable and must not be called. Measured against the live
|
||
server (2026-09-07): `get_library_media_info` on a show section returns one row per
|
||
*show*, with `file_size: 0` on every row, and passing `section_type=episode`, `season`,
|
||
or `show` changes nothing — all four return the same 169 show-level rows with zero sizes.
|
||
A `refresh=true` rebuild might populate them, but that triggers an expensive scan on
|
||
Tautulli and was deliberately not attempted.
|
||
|
||
The consequence is architectural and worth stating plainly: **Plex is the only size
|
||
authority for TV**, which is 49 of the library's 66 TB. Tautulli contributes watch data
|
||
and nothing else for show libraries. Any future code tempted to use Tautulli's table as a
|
||
convenient size source will silently produce zeroes for three-quarters of the library.
|
||
|
||
**Joining to Plex.** Both systems use the same Plex `rating_key`, so the join is direct —
|
||
no title or path matching, no fuzzy logic. If `get_server_info.pms_identifier` does not
|
||
match the `machineIdentifier` from Plex's `/identity`, Tautulli is watching a *different*
|
||
Plex server and MediaShelf refuses to use it rather than joining nonsense data together.
|
||
That check runs at the start of every scan.
|
||
|
||
---
|
||
|
||
## 5. Data model
|
||
|
||
SQLite, WAL mode, foreign keys on. Migrations are plain numbered SQL files applied at
|
||
startup, tracked in a `schema_version` table — no ORM migration framework.
|
||
|
||
### 5.1 Tables
|
||
|
||
```sql
|
||
-- One row per configured media server. v1 has exactly one, but the column exists
|
||
-- so Emby/Jellyfin can be added without a migration.
|
||
CREATE TABLE provider (
|
||
id INTEGER PRIMARY KEY,
|
||
kind TEXT NOT NULL, -- 'plex' | 'emby' | 'jellyfin'
|
||
name TEXT NOT NULL,
|
||
base_url TEXT NOT NULL,
|
||
server_id TEXT, -- Plex machineIdentifier
|
||
version TEXT,
|
||
last_scan_id INTEGER,
|
||
created_at INTEGER NOT NULL
|
||
);
|
||
|
||
CREATE TABLE library (
|
||
id INTEGER PRIMARY KEY,
|
||
provider_id INTEGER NOT NULL REFERENCES provider(id),
|
||
provider_key TEXT NOT NULL, -- Plex section id
|
||
title TEXT NOT NULL,
|
||
kind TEXT NOT NULL, -- 'movie' | 'show'
|
||
locations TEXT, -- JSON array of on-disk roots
|
||
keep_all INTEGER NOT NULL DEFAULT 0, -- library-level keep rule (§6.6)
|
||
scanned_at INTEGER,
|
||
UNIQUE (provider_id, provider_key)
|
||
);
|
||
|
||
-- Explicit human judgements that the reclaim score must not override (§6.6).
|
||
-- Keyed on CONTENT identity, never on rating_key, which Plex reassigns.
|
||
CREATE TABLE keep_mark (
|
||
id INTEGER PRIMARY KEY,
|
||
scope TEXT NOT NULL, -- 'show' | 'season' | 'movie'
|
||
mode TEXT NOT NULL, -- 'keep' | 'exclude'
|
||
library_id INTEGER NOT NULL REFERENCES library(id),
|
||
guid TEXT NOT NULL, -- movie guid, or the SHOW's guid
|
||
season_number INTEGER, -- set only for scope='season'
|
||
-- convenience only; allowed to go stale, never matched on
|
||
provider_item_id TEXT,
|
||
label TEXT NOT NULL, -- human-readable title at marking time
|
||
note TEXT, -- "might watch someday", "expensive to re-rip"
|
||
created_at INTEGER NOT NULL,
|
||
updated_at INTEGER NOT NULL,
|
||
last_matched_scan_id INTEGER, -- NULL after a scan => orphaned
|
||
UNIQUE (scope, library_id, guid, season_number)
|
||
);
|
||
|
||
-- The unit of analysis. One row per movie, and one row per SEASON.
|
||
-- Shows themselves get a row too (kind='show') for grouping and display,
|
||
-- but carry no size of their own.
|
||
CREATE TABLE media_item (
|
||
id INTEGER PRIMARY KEY,
|
||
provider_id INTEGER NOT NULL REFERENCES provider(id),
|
||
library_id INTEGER NOT NULL REFERENCES library(id),
|
||
provider_item_id TEXT NOT NULL, -- Plex ratingKey
|
||
kind TEXT NOT NULL, -- 'movie' | 'show' | 'season'
|
||
guid TEXT,
|
||
title TEXT NOT NULL,
|
||
sort_title TEXT,
|
||
year INTEGER,
|
||
-- season/show linkage
|
||
parent_id INTEGER REFERENCES media_item(id), -- season -> show
|
||
season_number INTEGER,
|
||
-- provenance
|
||
added_at INTEGER, -- epoch s; for a season, MIN over episodes
|
||
updated_at INTEGER,
|
||
-- rollups (denormalized, recomputed each scan)
|
||
episode_count INTEGER NOT NULL DEFAULT 0,
|
||
size_bytes INTEGER NOT NULL DEFAULT 0,
|
||
duration_ms INTEGER NOT NULL DEFAULT 0,
|
||
part_count INTEGER NOT NULL DEFAULT 0,
|
||
primary_path TEXT, -- representative file path for display
|
||
resolution TEXT,
|
||
video_codec TEXT,
|
||
-- watch rollups (from watch_event)
|
||
watch_count INTEGER NOT NULL DEFAULT 0, -- COMPLETED views only
|
||
partial_count INTEGER NOT NULL DEFAULT 0,
|
||
abandoned_count INTEGER NOT NULL DEFAULT 0,
|
||
last_watched_at INTEGER, -- last completed view
|
||
last_touched_at INTEGER, -- last view of any kind
|
||
first_watched_at INTEGER,
|
||
distinct_watcher_count INTEGER NOT NULL DEFAULT 0,
|
||
avg_percent_complete REAL,
|
||
pre_history INTEGER NOT NULL DEFAULT 0, -- added before history coverage
|
||
-- keep state, recomputed each scan and on every keep change (§6.6)
|
||
kept INTEGER NOT NULL DEFAULT 0,
|
||
kept_via TEXT, -- 'library' | 'show' | 'season' | 'movie'
|
||
kept_mark_id INTEGER REFERENCES keep_mark(id),
|
||
provider_view_count INTEGER NOT NULL DEFAULT 0, -- advisory, token-scoped
|
||
-- lifecycle
|
||
status TEXT NOT NULL DEFAULT 'present', -- 'present' | 'missing'
|
||
first_seen_scan_id INTEGER,
|
||
last_seen_scan_id INTEGER,
|
||
UNIQUE (provider_id, provider_item_id)
|
||
);
|
||
|
||
-- Episodes are ingested (they carry the size and the history), but they are NOT
|
||
-- the unit of analysis. They exist so seasons can be rolled up accurately and so
|
||
-- a season can be expanded in the UI.
|
||
CREATE TABLE episode (
|
||
id INTEGER PRIMARY KEY,
|
||
season_item_id INTEGER NOT NULL REFERENCES media_item(id) ON DELETE CASCADE,
|
||
provider_item_id TEXT NOT NULL,
|
||
episode_number INTEGER,
|
||
title TEXT,
|
||
added_at INTEGER,
|
||
duration_ms INTEGER,
|
||
size_bytes INTEGER NOT NULL DEFAULT 0,
|
||
watch_count INTEGER NOT NULL DEFAULT 0,
|
||
last_watched_at INTEGER,
|
||
status TEXT NOT NULL DEFAULT 'present',
|
||
UNIQUE (provider_item_id)
|
||
);
|
||
|
||
-- Every physical file. A movie with two versions has two rows.
|
||
CREATE TABLE media_part (
|
||
id INTEGER PRIMARY KEY,
|
||
media_item_id INTEGER REFERENCES media_item(id) ON DELETE CASCADE,
|
||
episode_id INTEGER REFERENCES episode(id) ON DELETE CASCADE,
|
||
provider_part_id TEXT,
|
||
file_path TEXT NOT NULL,
|
||
size_bytes INTEGER NOT NULL DEFAULT 0,
|
||
container TEXT,
|
||
resolution TEXT,
|
||
video_codec TEXT,
|
||
audio_codec TEXT,
|
||
bitrate INTEGER,
|
||
CHECK ((media_item_id IS NULL) != (episode_id IS NULL))
|
||
);
|
||
|
||
-- One row per playback event, server-wide, all accounts.
|
||
-- Primary source is Tautulli; 'plex' rows are degraded fallback events (§4.11).
|
||
CREATE TABLE watch_event (
|
||
id INTEGER PRIMARY KEY,
|
||
provider_id INTEGER NOT NULL REFERENCES provider(id),
|
||
source TEXT NOT NULL, -- 'tautulli' | 'plex'
|
||
source_row_id TEXT NOT NULL, -- Tautulli row_id, or Plex historyKey
|
||
reference_id TEXT, -- Tautulli's own grouping id
|
||
provider_item_id TEXT NOT NULL, -- the leaf: movie or episode ratingKey
|
||
account_id TEXT,
|
||
viewed_at INTEGER NOT NULL, -- 'date' / 'started'
|
||
stopped_at INTEGER,
|
||
play_duration_s INTEGER,
|
||
paused_counter_s INTEGER,
|
||
percent_complete INTEGER, -- NULL for source='plex'
|
||
watched_status REAL, -- Tautulli's verdict, advisory
|
||
disposition TEXT NOT NULL, -- 'completed' | 'partial' | 'abandoned'
|
||
session_id TEXT, -- MediaShelf's own merge key (§4.9)
|
||
media_type TEXT,
|
||
platform TEXT,
|
||
UNIQUE (provider_id, source, source_row_id)
|
||
);
|
||
|
||
CREATE TABLE account (
|
||
id INTEGER PRIMARY KEY,
|
||
provider_id INTEGER NOT NULL REFERENCES provider(id),
|
||
account_id TEXT NOT NULL, -- Tautulli user_id / Plex accountID
|
||
name TEXT,
|
||
friendly_name TEXT,
|
||
UNIQUE (provider_id, account_id)
|
||
);
|
||
|
||
-- What the history record actually covers, so 'never watched' can be
|
||
-- distinguished from 'watched before Tautulli existed' (§4.11).
|
||
CREATE TABLE history_coverage (
|
||
id INTEGER PRIMARY KEY,
|
||
provider_id INTEGER NOT NULL REFERENCES provider(id),
|
||
source TEXT NOT NULL,
|
||
earliest_event_at INTEGER,
|
||
latest_event_at INTEGER,
|
||
event_count INTEGER NOT NULL DEFAULT 0,
|
||
updated_at INTEGER NOT NULL,
|
||
UNIQUE (provider_id, source)
|
||
);
|
||
|
||
CREATE TABLE scan (
|
||
id INTEGER PRIMARY KEY,
|
||
provider_id INTEGER NOT NULL REFERENCES provider(id),
|
||
mode TEXT NOT NULL, -- 'full' | 'incremental'
|
||
trigger TEXT NOT NULL, -- 'manual' | 'schedule' | 'startup'
|
||
status TEXT NOT NULL, -- 'running' | 'succeeded' | 'failed'
|
||
started_at INTEGER NOT NULL,
|
||
finished_at INTEGER,
|
||
items_seen INTEGER DEFAULT 0,
|
||
items_added INTEGER DEFAULT 0,
|
||
items_updated INTEGER DEFAULT 0,
|
||
items_missing INTEGER DEFAULT 0,
|
||
events_added INTEGER DEFAULT 0,
|
||
warning_count INTEGER DEFAULT 0,
|
||
error TEXT
|
||
);
|
||
|
||
CREATE TABLE saved_view (
|
||
id INTEGER PRIMARY KEY,
|
||
name TEXT NOT NULL UNIQUE,
|
||
description TEXT,
|
||
rules TEXT NOT NULL, -- JSON rule group, see §7
|
||
sort TEXT, -- JSON: [{field, dir}]
|
||
columns TEXT, -- JSON array of column keys
|
||
weights TEXT, -- JSON weight profile, see §6
|
||
created_at INTEGER NOT NULL,
|
||
updated_at INTEGER NOT NULL
|
||
);
|
||
|
||
CREATE TABLE setting (
|
||
key TEXT PRIMARY KEY,
|
||
value TEXT NOT NULL
|
||
);
|
||
```
|
||
|
||
### 5.2 Indexes
|
||
|
||
```sql
|
||
CREATE INDEX ix_item_library ON media_item(library_id, kind, status);
|
||
CREATE INDEX ix_item_added ON media_item(added_at);
|
||
CREATE INDEX ix_item_lastwatch ON media_item(last_watched_at);
|
||
CREATE INDEX ix_item_size ON media_item(size_bytes);
|
||
CREATE INDEX ix_item_watchcount ON media_item(watch_count);
|
||
CREATE INDEX ix_item_parent ON media_item(parent_id);
|
||
CREATE INDEX ix_item_guid ON media_item(guid); -- duplicate grouping, §6.5
|
||
CREATE INDEX ix_item_kept ON media_item(kept, library_id);
|
||
CREATE INDEX ix_keep_lookup ON keep_mark(library_id, guid, season_number);
|
||
CREATE INDEX ix_episode_season ON episode(season_item_id);
|
||
CREATE INDEX ix_part_item ON media_part(media_item_id);
|
||
CREATE INDEX ix_part_episode ON media_part(episode_id);
|
||
CREATE INDEX ix_event_item ON watch_event(provider_item_id);
|
||
CREATE INDEX ix_event_viewed ON watch_event(viewed_at);
|
||
CREATE INDEX ix_event_disp ON watch_event(provider_item_id, disposition);
|
||
CREATE INDEX ix_event_account ON watch_event(account_id, viewed_at);
|
||
```
|
||
|
||
A full-text index over `title` uses SQLite FTS5 (`media_item_fts`) kept in sync by
|
||
triggers, so the grid's search box is instant even at 100k rows.
|
||
|
||
### 5.3 Season rollup rules
|
||
|
||
After items and history are ingested, a rollup pass recomputes each season row:
|
||
|
||
| Season field | Rule |
|
||
|---|---|
|
||
| `size_bytes` | `SUM(episode.size_bytes)` |
|
||
| `episode_count` | `COUNT(episode)` |
|
||
| `duration_ms` | `SUM(episode.duration_ms)` |
|
||
| `added_at` | `MIN(episode.added_at)` — when the season first landed |
|
||
| `watch_count` | `SUM(episode.watch_count)` — completed episode views |
|
||
| `abandoned_count` | `SUM(episode.abandoned_count)` |
|
||
| `last_watched_at` | `MAX(episode.last_watched_at)` |
|
||
| `last_touched_at` | `MAX(episode.last_touched_at)` |
|
||
| `primary_path` | the common directory prefix of its episodes' paths |
|
||
|
||
A show row rolls up the same way from its seasons, for display and for the "delete the
|
||
whole show" case.
|
||
|
||
The `watch_count` choice deserves a note: summing episode plays means a 24-episode season
|
||
watched once has `watch_count = 24`, while a movie watched once has `watch_count = 1`.
|
||
Comparing those raw numbers across kinds is meaningless. So the scoring model
|
||
(§6) uses a **normalized completion figure** for TV — `watch_count / episode_count` —
|
||
rather than the raw sum, and the grid displays both (`24 plays · 1.0× season`).
|
||
|
||
### 5.4 Missing items
|
||
|
||
Nothing is ever hard-deleted by a scan. When a full sweep completes and an item that was
|
||
present is no longer returned by Plex, its `status` flips to `missing` and
|
||
`last_seen_scan_id` freezes. Missing items are hidden from the default grid but remain
|
||
queryable — they're the record of what left the library and when, which is exactly what
|
||
you want the first time something disappears unexpectedly.
|
||
|
||
---
|
||
|
||
## 6. The reclaim score
|
||
|
||
The score answers: *how much do I gain by deleting this, weighed against how likely I am
|
||
to want it?* It is a weighted sum of normalized components, each in `[0, 1]`, scaled to
|
||
0–100. Higher means "better candidate for deletion".
|
||
|
||
### 6.1 Components
|
||
|
||
| Component | Definition | Rationale |
|
||
|---|---|---|
|
||
| **`size`** | `log10(1 + size_bytes) / log10(1 + max_size_in_library)` | Log scale — the difference between 2 GB and 4 GB matters more than between 60 GB and 62 GB |
|
||
| **`staleness`** | `min(days_since_last_watch / STALE_HORIZON, 1)`, where a never-watched item scores `1.0` | How long since anyone cared |
|
||
| **`unpopularity`** | `1 - min(normalized_watches / POPULAR_AT, 1)` where `normalized_watches` = `watch_count` for movies, `watch_count / episode_count` for seasons | Rewatched things are keepers |
|
||
| **`age`** | `min(days_since_added / AGE_HORIZON, 1)` | Something added last week deserves a chance |
|
||
| **`solitude`** | `1 - min(distinct_watcher_count / SOLITUDE_AT, 1)` | Content almost nobody watches is safer to cut |
|
||
| **`rejection`** | `min(abandoned_count / REJECTED_AT, 1)`, zero when `watch_count > 0` | Started and quit is an active verdict — see below |
|
||
|
||
`STALE_HORIZON` defaults to 730 days, `AGE_HORIZON` to 1095 days, `POPULAR_AT` to 3,
|
||
`REJECTED_AT` to 2, `SOLITUDE_AT` to 3.
|
||
|
||
Both of the Tautulli-dependent components were calibrated against real measurements
|
||
rather than guessed, and both moved:
|
||
|
||
**`solitude` — divisor 3 confirmed, weight raised.** The original divisor assumed a
|
||
household. With 60 users the concern was that it would saturate. It does not: of 24,204
|
||
watched items, **61.7% were watched by exactly one user and 20.0% by two**, with a long
|
||
thin tail. Three distinct watchers already puts an item in the top ~8% of the library, so
|
||
a divisor of 3 is the right shape. What changed is that on a 60-user server "only one
|
||
person ever watched this" is a genuinely informative statement rather than noise, so the
|
||
weight goes up from 0.04 to 0.10.
|
||
|
||
**`rejection` — real but smaller than hoped, weight cut.** Across all 87,640 plays the
|
||
split is **76.3% finished · 18.8% partial · 4.9% abandoned**. Abandonment is a real
|
||
signal but a thin one, so its weight drops from 0.12 to 0.06. It stays in the model
|
||
because when it does fire it is unusually decisive — an item several people started and
|
||
none finished is a much stronger delete case than the score's other components can
|
||
express — but it should not be pushing the ranking around on 5% of the evidence.
|
||
|
||
The larger surprise is the **18.8% partial** band. That is four times the abandoned share
|
||
and it is genuinely ambiguous: someone 60% through an episode may have been interrupted
|
||
and intending to return, or may have quietly given up. v1 deliberately does not score it
|
||
in either direction; it is stored, displayed, and filterable, and the first month of
|
||
looking at real rows should settle whether it deserves a component of its own.
|
||
|
||
The `rejection` component only exists because Tautulli supplies `percent_complete`
|
||
(§4.10). It is deliberately zeroed the moment anyone completes a view, so a film that was
|
||
abandoned twice and then watched through is not penalised. When history runs in
|
||
Plex-fallback mode this component is unavailable and drops out of the weighting
|
||
entirely (§4.11).
|
||
|
||
### 6.2 The formula
|
||
|
||
```
|
||
score = 100 × Σ(wᵢ · componentᵢ) / Σ(wᵢ)
|
||
```
|
||
|
||
Default weight profile:
|
||
|
||
```json
|
||
{ "size": 0.28, "staleness": 0.24, "unpopularity": 0.22,
|
||
"solitude": 0.10, "age": 0.10, "rejection": 0.06 }
|
||
```
|
||
|
||
The denominator sums only the weights of components that are **available** for that row.
|
||
When history is in Plex-fallback mode, `rejection` is dropped and the remaining weights
|
||
renormalize, rather than every item silently scoring lower because one component is
|
||
pinned at zero. The same applies per-row: a `pre_history` item has its `staleness`
|
||
capped (§4.11), and the UI shows which components contributed.
|
||
|
||
### 6.3 The grace rule
|
||
|
||
Two hard overrides, applied after scoring, because a purely numeric model will
|
||
eventually recommend something obviously wrong:
|
||
|
||
- **New-arrival grace** — an item with `added_at` inside `GRACE_DAYS` (default 30) is
|
||
clamped to score 0 and flagged `grace: new`. You haven't had a chance to watch it yet.
|
||
- **Recent-watch grace** — an item watched inside `RECENT_DAYS` (default 90) is clamped
|
||
to a maximum of 25 and flagged `grace: recent`.
|
||
|
||
Both are toggleable per view, and the UI shows why an item was clamped rather than
|
||
silently hiding it.
|
||
|
||
### 6.4 Where it's computed
|
||
|
||
In SQL, as a generated expression in the query, not as a stored column. Weights change
|
||
every time a slider moves; storing the score would mean recomputing 100k rows per drag.
|
||
The normalization constants (`max_size_in_library`, horizons) are computed once per
|
||
request in a CTE. At this row count the whole thing runs in single-digit milliseconds.
|
||
|
||
The same function is implemented in Python in `mediashelf.scoring` for unit tests and CSV
|
||
export, with a property test asserting the SQL and Python implementations agree to within
|
||
0.01 on a generated corpus. Two implementations of the same formula is a real risk, and
|
||
that test is the mitigation.
|
||
|
||
### 6.5 Cross-library duplicates
|
||
|
||
With 16 movie sections, the same film is often held more than once — most obviously
|
||
*Movies* vs *4K Movies*, but genre sections overlap too. Two copies of a film are 100%
|
||
redundant storage, and neither copy looks like a deletion candidate on its own: each has
|
||
its own watch history, and plays are split between them, so both score as moderately
|
||
watched.
|
||
|
||
MediaShelf detects this using the Plex `guid`, which identifies the *content* rather than
|
||
the file, and is the same across sections. Items sharing a `guid` are grouped, and the
|
||
group is surfaced with:
|
||
|
||
- total redundant bytes (group size minus its largest member)
|
||
- which copy is played and which is not — the common case is that everyone watches the
|
||
4K copy and the 1080p copy has not been touched in a year
|
||
- resolution and bitrate of each copy, so "keep the best one" is a visible decision
|
||
|
||
This is a **report**, not an automatic recommendation. Deciding that a 4K copy supersedes
|
||
a 1080p one depends on what plays back well on the household's devices, which MediaShelf
|
||
has no way to know. It surfaces the pairs and their sizes and lets a human choose.
|
||
|
||
A `duplicates` flag on the item and a *Duplicate groups* view are the v1 surface. This was
|
||
originally deferred to §14.3; the measured library shape moved it forward.
|
||
|
||
### 6.6 Keep marks
|
||
|
||
**A reclaim score measures what is unwatched, not what is valuable.** Those two things
|
||
correlate for ordinary purchased or downloaded content and are close to unrelated
|
||
everywhere else. *Family Videos* is 38 GB of irreplaceable home video that is 86% never
|
||
played and scores as a perfect deletion target. *4K Movies* is 72.9% never played and is
|
||
the most expensive content in the library to re-acquire. And of the 1,337 never-played
|
||
seasons, an unknown number are simply things kept deliberately, on purpose, in case
|
||
someone wants them later.
|
||
|
||
No amount of watch data can infer any of that. It has to be stated by a human, and once
|
||
stated the score must not be able to override it. That is what a **keep mark** is.
|
||
|
||
#### Scopes
|
||
|
||
One kind of mark, applied at four levels:
|
||
|
||
| Scope | Granularity |
|
||
|---|---|
|
||
| `library` | a whole section — *keep everything in 4K Movies* |
|
||
| `show` | a series and all its seasons, present and future |
|
||
| `season` | one season of one series |
|
||
| `movie` | one film |
|
||
|
||
Libraries are kept by a **rule** (`library.keep_all`), so anything added to that library
|
||
later is kept automatically. Everything below the library level is marked **explicitly**,
|
||
per item — deliberate acts on specific things, each with a timestamp and an optional note.
|
||
|
||
#### Precedence
|
||
|
||
Most specific wins, and an explicit mark can point in either direction:
|
||
|
||
```
|
||
movie/season explicit 'exclude' → NOT kept (overrides an inherited library keep)
|
||
movie/season explicit 'keep' → kept
|
||
show 'keep' → kept
|
||
library keep_all → kept
|
||
otherwise → not kept
|
||
```
|
||
|
||
The `exclude` direction exists so that *"keep all of 4K Movies except this one"* is
|
||
expressible. Without it a library rule is a trap: the only way out would be to turn the
|
||
whole rule off and re-mark 313 films by hand.
|
||
|
||
Each item carries a computed `kept` flag and a `kept_via` field naming which mark applied,
|
||
recomputed at the end of every scan and immediately on any keep change. `kept_via` matters
|
||
in the UI — "kept because the whole library is kept" and "kept because you marked this
|
||
season in March" are different facts and lead to different actions.
|
||
|
||
#### Identity, and why it is not the rating key
|
||
|
||
Plex `ratingKey` values are **not stable**. A library rebuild, an unmatch-and-rematch, or
|
||
a section removed and re-added can reassign them. A keep list stored against rating keys
|
||
would silently detach from the content it was protecting, and in v2 the consequence of
|
||
that is deleting something explicitly marked keep. That is the worst failure this
|
||
application could have.
|
||
|
||
So keep marks are stored against **content identity** instead:
|
||
|
||
| Scope | Key |
|
||
|---|---|
|
||
| `library` | the library row |
|
||
| `movie` | `(library_id, guid)` |
|
||
| `show` | `(library_id, guid)` |
|
||
| `season` | `(library_id, show_guid, season_number)` |
|
||
|
||
Plex's `guid` identifies the *content*, not the file, and survives rebuilds. Seasons are
|
||
keyed by their show's guid plus the season number rather than a season guid, so the scheme
|
||
does not depend on Plex exposing season-level guids at all.
|
||
|
||
**Marks are scoped per library on purpose.** Because *Movies* and *4K Movies* hold the
|
||
same films under the same `guid` (§6.5), an unscoped mark would keep both copies at once —
|
||
so keeping the 4K version would silently protect the 1080p one you were trying to remove.
|
||
Scoping by `(library_id, guid)` keeps the two copies independently markable, which is
|
||
exactly what the duplicate-groups view needs.
|
||
|
||
`provider_item_id` is stored alongside as a convenience for linking, and is allowed to go
|
||
stale. It is never the thing a mark is matched on.
|
||
|
||
#### Orphans
|
||
|
||
If a scan completes and a keep mark matched nothing, it is flagged **orphaned** rather
|
||
than deleted. The content may have been removed from Plex, renamed, rematched to a
|
||
different guid, or moved to another library. The Kept view lists orphans separately with
|
||
their label and note as recorded at marking time, so a keep that has come adrift is
|
||
visible rather than silent. Nothing removes a keep mark except an explicit human action.
|
||
|
||
#### Effect
|
||
|
||
- Kept items are **hidden from the grid by default**, with a *Show kept* toggle.
|
||
- They are excluded from every candidate view, from CSV exports of those views, and from
|
||
the reclaim totals — which is why those totals are always reported three ways (§9.1).
|
||
- Their score is still computed and shown when kept items are displayed, so *"this scores
|
||
91 and I'm keeping it anyway"* stays a visible, revisitable decision.
|
||
- In v2, a kept item **cannot be quarantined**. The deletion path refuses it at the
|
||
guardrail, before any confirmation dialog. Removing the keep mark is a separate,
|
||
deliberate act — deletion never offers to do it for you as part of the same click.
|
||
|
||
**Nothing ships kept.** No library is pre-marked, in seed data or in configuration — every
|
||
keep in the system got there because a person put it there. The `KEEP_ALL_LIBRARIES`
|
||
environment variable exists so a library *can* be marked from configuration when that is
|
||
convenient, but its default is empty and it stays empty unless someone sets it.
|
||
|
||
That is a deliberate choice rather than an omission. A keep mark's value is that it
|
||
records a human judgement; one the application invented on the user's behalf would be
|
||
indistinguishable in the database from one they actually made, and would undermine the
|
||
audit trail that makes the Kept view worth reading.
|
||
|
||
#### The number that keeps this honest
|
||
|
||
A keep list that grows quietly will eventually swallow the library and leave the reclaim
|
||
report meaningless without anyone noticing. So the dashboard always shows the split:
|
||
|
||
```
|
||
27.4 TB never played · 8.1 TB kept · 19.3 TB available
|
||
```
|
||
|
||
If the middle number climbs toward the first, the tool is telling you it has stopped
|
||
being useful — which is worth knowing, and is invisible if kept items are simply
|
||
subtracted out.
|
||
|
||
---
|
||
|
||
## 7. Saved views and rules
|
||
|
||
A saved view bundles four things: a rule set, a sort, a column selection, and a weight
|
||
profile. Naming one — *"Big unwatched movies"* — makes it a one-click, re-runnable
|
||
report that re-evaluates against fresh data every time it's opened.
|
||
|
||
### 7.1 Rule grammar
|
||
|
||
Rules are a nested JSON group, evaluated into parameterized SQL:
|
||
|
||
```json
|
||
{
|
||
"op": "and",
|
||
"rules": [
|
||
{ "field": "kind", "op": "in", "value": ["movie"] },
|
||
{ "field": "size_bytes", "op": "gte", "value": 10737418240 },
|
||
{ "field": "watch_count", "op": "eq", "value": 0 },
|
||
{ "field": "added_at", "op": "older_than_days", "value": 730 },
|
||
{ "op": "or", "rules": [
|
||
{ "field": "library_id", "op": "in", "value": [1, 4] },
|
||
{ "field": "resolution", "op": "eq", "value": "4k" }
|
||
]}
|
||
]
|
||
}
|
||
```
|
||
|
||
Fields are drawn from a whitelist mapping field name → column and type. Operators are a
|
||
fixed set: `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `in`, `not_in`, `contains`,
|
||
`starts_with`, `is_null`, `is_not_null`, `older_than_days`, `newer_than_days`,
|
||
`never` (for watch fields). The whitelisted fields include the completion metrics —
|
||
`watch_count`, `partial_count`, `abandoned_count`, `avg_percent_complete`,
|
||
`last_touched_at`, `pre_history` — plus `kept` and `kept_via`. Anything outside the
|
||
whitelist is a 400. Values are always
|
||
bound parameters. There is no SQL string interpolation anywhere in this path.
|
||
|
||
### 7.2 Seed views shipped by default
|
||
|
||
| Name | Rule |
|
||
|---|---|
|
||
| Never watched, large | `watch_count = 0 AND size_bytes >= 8 GB` |
|
||
| Cold storage | `last_watched_at older than 3 years` |
|
||
| One-and-done movies | `kind = movie AND watch_count = 1 AND last_watched_at older than 2 years` |
|
||
| Abandoned seasons | `kind = season AND watch_count/episode_count < 0.3 AND added_at older than 1 year` |
|
||
| Tried and rejected | `abandoned_count >= 2 AND watch_count = 0` — the household said no |
|
||
| Bailed in the first 15 min | `avg_percent_complete < 15 AND watch_count = 0 AND size_bytes >= 4 GB` |
|
||
| Biggest 100 | sorted by `size_bytes desc`, no filter |
|
||
| Recently added | `added_at newer than 30 days` — the sanity check, not a delete list |
|
||
| **Confident reclaim** | `watch_count = 0 AND pre_history = false` — added while Tautulli was watching and never played. On the measured library this is 6.5 TB and is the list to act on first |
|
||
| **Uncertain reclaim** | `watch_count = 0 AND pre_history = true` — 20.9 TB that *might* have been watched before 2025-03-08. Needs human judgement, not a bulk action |
|
||
| Duplicate groups | items sharing a `guid` across libraries (§6.5) |
|
||
| Kept | `kept = true` — what you have decided to keep, and what it costs (§6.6) |
|
||
|
||
Every view except *Kept* excludes kept items implicitly; `kept` is a whitelisted rule
|
||
field, so a view can opt back in explicitly when that is the point.
|
||
|
||
---
|
||
|
||
## 8. HTTP API
|
||
|
||
All JSON, all under `/api/v1`. No auth in v1 (§12).
|
||
|
||
### 8.1 Library data
|
||
|
||
```
|
||
GET /api/v1/items
|
||
```
|
||
|
||
Query parameters:
|
||
|
||
| Param | Meaning |
|
||
|---|---|
|
||
| `library_id` | repeatable filter |
|
||
| `kind` | `movie` / `season` / `show` |
|
||
| `q` | FTS search over title |
|
||
| `rules` | URL-encoded JSON rule group (§7.1) |
|
||
| `view_id` | apply a saved view instead of ad-hoc rules |
|
||
| `sort` | e.g. `reclaim_score:desc`, `size_bytes:desc` |
|
||
| `weights` | URL-encoded JSON weight profile |
|
||
| `include_missing` | default `false` |
|
||
| `include_kept` | default `false` — kept items are hidden unless asked for (§6.6) |
|
||
| `page`, `page_size` | page_size max 500 |
|
||
|
||
Response:
|
||
|
||
```json
|
||
{
|
||
"total": 4821,
|
||
"page": 1,
|
||
"page_size": 100,
|
||
"aggregate": {
|
||
"total_size_bytes": 41234567890123,
|
||
"item_count": 4821,
|
||
"kept_bytes": 8906044184985,
|
||
"available_bytes": 21228456789012
|
||
},
|
||
"items": [
|
||
{
|
||
"id": 1183,
|
||
"kind": "movie",
|
||
"title": "Example Film",
|
||
"year": 2011,
|
||
"library": { "id": 1, "title": "Movies" },
|
||
"size_bytes": 32212254720,
|
||
"added_at": 1490000000,
|
||
"last_watched_at": null,
|
||
"last_touched_at": 1712000000,
|
||
"watch_count": 0,
|
||
"partial_count": 0,
|
||
"abandoned_count": 3,
|
||
"avg_percent_complete": 8.4,
|
||
"distinct_watcher_count": 3,
|
||
"pre_history": false,
|
||
"episode_count": null,
|
||
"primary_path": "/mnt/vault2/movies/Example Film (2011)/Example Film (2011).mkv",
|
||
"part_count": 1,
|
||
"resolution": "4k",
|
||
"kept": false,
|
||
"kept_via": null,
|
||
"reclaim_score": 91.4,
|
||
"reclaim_components": {
|
||
"size": 0.97, "staleness": 1.0, "unpopularity": 1.0,
|
||
"rejection": 1.0, "age": 1.0, "solitude": 0.0
|
||
},
|
||
"grace": null,
|
||
"flags": ["rejected"]
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
Other endpoints:
|
||
|
||
```
|
||
GET /api/v1/items/{id} full detail incl. every part and, for seasons, episodes
|
||
GET /api/v1/libraries list with item counts and total size
|
||
GET /api/v1/accounts Plex accounts seen in history
|
||
GET /api/v1/stats/overview headline numbers for the dashboard
|
||
GET /api/v1/stats/size-by-library
|
||
GET /api/v1/stats/added-over-time?bucket=month
|
||
GET /api/v1/stats/watch-distribution
|
||
GET /api/v1/stats/size-vs-lastwatched scatter data, downsampled server-side
|
||
GET /api/v1/stats/completion finished / abandoned / never-opened, by size
|
||
GET /api/v1/sources per-source status: reachable?, coverage window,
|
||
event count, active history source
|
||
```
|
||
|
||
### 8.2 Keep marks
|
||
|
||
```
|
||
GET /api/v1/keeps all marks, with resolved item counts and bytes
|
||
GET /api/v1/keeps/orphans marks that matched nothing on the last scan
|
||
POST /api/v1/keeps { scope, library_id, item_id | guid,
|
||
season_number?, mode, note? }
|
||
PATCH /api/v1/keeps/{id} edit the note
|
||
DELETE /api/v1/keeps/{id} un-keep (the only way a mark is ever removed)
|
||
POST /api/v1/keeps/bulk { item_ids: [...], mode, note? }
|
||
PUT /api/v1/libraries/{id}/keep_all { keep_all: true|false }
|
||
```
|
||
|
||
`POST /keeps` accepts either an `item_id` (the UI's normal path — the server resolves it
|
||
to the durable `(library_id, guid, season_number)` key) or those key parts directly.
|
||
`POST /keeps/bulk` is what makes marking tractable: with 1,337 never-played seasons,
|
||
keeping a batch has to be one action on a multi-selection, not 1,337 clicks.
|
||
|
||
Every keep response includes `resolved_items` and `resolved_bytes`, so the cost of a mark
|
||
is visible at the moment it is made rather than discovered later in a total.
|
||
|
||
### 8.3 Saved views
|
||
|
||
```
|
||
GET /api/v1/views
|
||
POST /api/v1/views
|
||
GET /api/v1/views/{id}
|
||
PUT /api/v1/views/{id}
|
||
DELETE /api/v1/views/{id}
|
||
```
|
||
|
||
### 8.4 Scans
|
||
|
||
```
|
||
GET /api/v1/scans history, newest first
|
||
GET /api/v1/scans/current null or the running scan with progress
|
||
POST /api/v1/scans { "mode": "full" | "incremental" } → 202 or 409 if one is running
|
||
```
|
||
|
||
Scan progress is polled at `/api/v1/scans/current` every 2 s while one is running. No
|
||
websockets — polling one row is cheaper than the complexity.
|
||
|
||
### 8.5 Export
|
||
|
||
```
|
||
GET /api/v1/export.csv?<same params as /items>
|
||
```
|
||
|
||
Streams the full result set — not just the current page — as CSV with the currently
|
||
selected columns plus the reclaim score and its components. This is the v1 deliverable:
|
||
the artifact you take away and act on.
|
||
|
||
### 8.6 Health
|
||
|
||
```
|
||
GET /healthz → { "status": "ok", "db": "ok",
|
||
"plex": "ok"|"unreachable",
|
||
"tautulli": "ok"|"unreachable"|"disabled",
|
||
"history_source": "tautulli"|"plex",
|
||
"last_scan_at": 1757000000, "stale": false }
|
||
```
|
||
|
||
---
|
||
|
||
## 9. Web UI
|
||
|
||
Server-rendered Flask shell, vanilla JS plus two small libraries: a virtualized table
|
||
(TanStack Table core or Tabulator) and a charting library. No build step, no npm in the
|
||
container — assets are vendored. This keeps the container small and the deploy identical
|
||
to the Mythica pattern.
|
||
|
||
### 9.1 Screens
|
||
|
||
**Dashboard** — the landing page. Headline tiles (total items, total size, size of
|
||
never-watched content, size of content untouched for 2+ years, last scan time), plus the
|
||
three-way reclaim split that keeps the keep list honest (§6.6):
|
||
|
||
```
|
||
NEVER PLAYED 27.4 TB · KEPT 8.1 TB · AVAILABLE 19.3 TB
|
||
```
|
||
|
||
and the charts:
|
||
|
||
- *Size by library* — horizontal bar.
|
||
- *Added over time* — monthly bars, stacked by library. Shows acquisition habits.
|
||
- *Watched vs unwatched by size* — the "how much of this shelf has anyone ever touched"
|
||
chart, which is usually the one that motivates the whole exercise. Split three ways
|
||
now that completion data exists: **finished · started and abandoned · never opened**.
|
||
The middle band is the interesting one and is invisible without Tautulli.
|
||
- *Size vs. last watched* — scatter, log-size on Y, days-since-watch on X. Points in the
|
||
upper right are the reclaim targets, and clicking a region filters the grid to it.
|
||
|
||
**Library grid** — the main workspace. A virtualized table, every column sortable, with:
|
||
|
||
- A filter rail: library checkboxes, kind, size range, added range, last-watched range,
|
||
watch-count range, resolution, and a free-text title search.
|
||
- A weight panel with five sliders and a live-updating `reclaim_score` column.
|
||
- Multi-select via checkboxes, shift-click ranges, and select-all-matching-filter.
|
||
In v1 selection drives the aggregate footer ("**312 items selected · 4.1 TB**"), the
|
||
CSV export, and **bulk keep**. The buttons that will eventually delete are present but
|
||
disabled, with a tooltip explaining v1 is report-only — so the workflow is proven
|
||
before it's armed.
|
||
- A **keep toggle** on every row, and *Keep selected* on a multi-selection with an
|
||
optional shared note. Kept rows are hidden by default; a *Show kept* switch in the
|
||
filter rail brings them back, badged with why they are kept (`library` / `show` /
|
||
`season` / `movie`) and still showing their score.
|
||
- Row expansion: a season expands to its episodes; any row expands to its file parts
|
||
with full paths.
|
||
|
||
**Kept items** — the audit surface for §6.6. Every mark with its scope, label, note, when
|
||
it was made, and how many items and bytes it currently resolves to, sorted by size so the
|
||
expensive keeps are obvious. Library-level rules are listed separately from explicit
|
||
marks, and **orphaned marks get their own section** — a keep that no longer matches
|
||
anything is a signal that content was removed, renamed, or rematched, and it should be
|
||
seen rather than silently carried forever. Un-keeping is done from here or from the grid;
|
||
nothing else ever removes a mark.
|
||
|
||
**Saved views** — list, create, edit, duplicate, run. Creating one captures the current
|
||
filter, sort, columns, and weights, so the flow is *explore, then name what you found*.
|
||
|
||
**Item detail** — everything known: all parts with paths and sizes, a per-account watch
|
||
timeline showing each play with how far it got (a row of 8%, 12%, 6% tells the story at a
|
||
glance), and the score breakdown showing each component's contribution.
|
||
|
||
**Scans** — history table, per-scan counts and duration, a "Scan now" button, live
|
||
progress.
|
||
|
||
**Settings** — connection status, scan schedule, score defaults, horizons, grace periods.
|
||
|
||
### 9.2 Interaction principles
|
||
|
||
- Every number that represents bytes is displayed in both human units and, on hover, the
|
||
exact byte count. Reclaim decisions turn on real numbers.
|
||
- The reclaim score column always shows its components on hover. A score you can't
|
||
interrogate is a score you won't trust, and won't act on.
|
||
- Filters live in the URL query string, so any view is linkable and back/forward works.
|
||
- Nothing in v1 is destructive, and the UI says so plainly in the header rather than
|
||
leaving the user to wonder.
|
||
- Keeping something is one click and un-keeping is one click, but neither ever happens as
|
||
a side effect of another action. A keep mark is a stated human judgement, so only a
|
||
human statement creates or removes it.
|
||
|
||
---
|
||
|
||
## 10. Configuration
|
||
|
||
All via environment variables, so the Portainer stack is the single source of truth.
|
||
|
||
| Variable | Default | Notes |
|
||
|---|---|---|
|
||
| `MEDIASHELF_SECRET_KEY` | — | required, Flask session signing |
|
||
| `PLEX_BASE_URL` | — | required, e.g. `http://192.168.1.10:32400` |
|
||
| `PLEX_TOKEN` | — | required, secret |
|
||
| `PLEX_VERIFY_SSL` | `true` | |
|
||
| `PLEX_TIMEOUT_S` | `30` | |
|
||
| `PLEX_PAGE_SIZE` | `500` | |
|
||
| `PLEX_REQUEST_DELAY_MS` | `0` | throttle for Loki's sake |
|
||
| `TAUTULLI_BASE_URL` | — | e.g. `http://192.168.1.100:8181` — unset disables Tautulli |
|
||
| `TAUTULLI_API_KEY` | — | secret |
|
||
| `TAUTULLI_TIMEOUT_S` | `30` | |
|
||
| `TAUTULLI_PAGE_SIZE` | `1000` | `length` on `get_history` |
|
||
| `HISTORY_SOURCE` | `auto` | `auto` \| `tautulli` \| `plex` — `auto` prefers Tautulli, falls back |
|
||
| `SESSION_MERGE_WINDOW_H` | `6` | events merged into one viewing (§4.9) |
|
||
| `COMPLETION_THRESHOLD` | `85` | percent complete counting as a watch |
|
||
| `ABANDON_CEILING` | `15` | percent complete below which a play is "abandoned" |
|
||
| `DATABASE_PATH` | `/data/mediashelf.db` | on the mounted volume |
|
||
| `SCAN_SCHEDULE_CRON` | `0 4 * * *` | nightly incremental at 04:00 |
|
||
| `SCAN_FULL_SWEEP_CRON` | `0 3 * * 0` | weekly full sweep, Sunday 03:00 |
|
||
| `SCAN_ON_STARTUP` | `false` | |
|
||
| `SCAN_LOCK_TIMEOUT_S` | `7200` | |
|
||
| `SCORE_STALE_HORIZON_DAYS` | `730` | |
|
||
| `SCORE_AGE_HORIZON_DAYS` | `1095` | |
|
||
| `SCORE_POPULAR_AT` | `3` | |
|
||
| `SCORE_REJECTED_AT` | `2` | abandoned views for a full `rejection` score |
|
||
| `SCORE_SOLITUDE_AT` | `3` | distinct watchers before `solitude` reaches zero |
|
||
| `KEEP_ALL_LIBRARIES` | *(empty)* | optional comma-separated library names to mark `keep_all` on first run. **Empty by default** — no library is ever kept unless someone says so (§6.6) |
|
||
| `SCORE_GRACE_DAYS` | `30` | |
|
||
| `SCORE_RECENT_DAYS` | `90` | |
|
||
| `TZ` | `America/Regina` | so cron times mean what they look like |
|
||
| `LOG_LEVEL` | `INFO` | |
|
||
|
||
`PLEX_TOKEN` and `TAUTULLI_API_KEY` are secrets and must not be committed. The repo ships
|
||
`.env.example` with placeholders, and `.env` is gitignored. (Worth noting given the
|
||
SOAP-password-in-history problem on the Mythica repo — this one starts clean and stays
|
||
clean.) Neither credential is ever written to the database, the logs, or an API response;
|
||
`/api/v1/settings` redacts both.
|
||
|
||
---
|
||
|
||
## 11. Deployment
|
||
|
||
### 11.1 Container
|
||
|
||
Single image, `python:3.12-slim` base, multi-stage so build deps don't ship. Runs as a
|
||
non-root user. One volume: `/data`. One port: 8080.
|
||
|
||
### 11.2 Getting the image onto Nox
|
||
|
||
The image is built off-box and delivered to Nox's Portainer rather than built there. Two
|
||
workable routes, in order of preference:
|
||
|
||
1. **Registry** — build, tag `registry.hallsworth.ca/mediashelf:<version>` (or Forgejo's
|
||
own container registry, which this Forgejo version supports), push, and have the
|
||
Portainer stack pull it. This is the route worth setting up once: redeploys become a
|
||
version bump in the stack, and rollback is pulling the previous tag.
|
||
2. **Image upload** — `docker save mediashelf:<version> | gzip` and load it on Nox via
|
||
Portainer's *Images → Import*. No registry needed, fine for the first deploy or two,
|
||
but it makes rollback manual and versions easy to lose track of.
|
||
|
||
Either way the image is **tagged with a real version**, never deployed as bare `latest`.
|
||
The Mythica stack had to be torn down and recreated because a `PUT` update kept serving
|
||
old code; immutable version tags are how that failure mode is avoided rather than
|
||
worked around.
|
||
|
||
### 11.3 Portainer stack on Nox
|
||
|
||
```yaml
|
||
services:
|
||
mediashelf:
|
||
image: registry.hallsworth.ca/mediashelf:0.1.0 # never bare :latest, see 11.2
|
||
container_name: mediashelf
|
||
restart: unless-stopped
|
||
ports:
|
||
- "8085:8080"
|
||
volumes:
|
||
- mediashelf_data:/data
|
||
environment:
|
||
- MEDIASHELF_SECRET_KEY=${MEDIASHELF_SECRET_KEY}
|
||
- PLEX_BASE_URL=${PLEX_BASE_URL}
|
||
- PLEX_TOKEN=${PLEX_TOKEN}
|
||
- TAUTULLI_BASE_URL=${TAUTULLI_BASE_URL}
|
||
- TAUTULLI_API_KEY=${TAUTULLI_API_KEY}
|
||
- HISTORY_SOURCE=auto
|
||
- DATABASE_PATH=/data/mediashelf.db
|
||
- SCAN_SCHEDULE_CRON=${SCAN_SCHEDULE_CRON}
|
||
- TZ=America/Regina
|
||
- LOG_LEVEL=INFO
|
||
healthcheck:
|
||
test: ["CMD", "python", "-c", "import urllib.request;urllib.request.urlopen('http://localhost:8080/healthz')"]
|
||
interval: 60s
|
||
timeout: 10s
|
||
retries: 3
|
||
|
||
volumes:
|
||
mediashelf_data:
|
||
```
|
||
|
||
**Every variable used in the stack must be listed under `environment:` here, not only set
|
||
in Portainer's UI.** A variable set in Portainer that isn't declared in the compose file
|
||
silently does nothing — the same trap that bit `PLAYERMAP_SHOW_GMS` on the Mythica stack.
|
||
|
||
Published through Nginx Proxy Manager at `mediashelf.hallsworth.ca`, upstream
|
||
`192.168.1.77:8085`. Note the host-port → container-port mapping is 8085 → 8080; NPM's
|
||
upstream must point at the **host** port.
|
||
|
||
### 11.4 The gunicorn / scheduler constraint
|
||
|
||
APScheduler runs in-process. With `--workers 2`, both workers would start a scheduler and
|
||
two scans would race for the SQLite write lock. Mitigation: the scheduler starts only in
|
||
the worker that successfully acquires an exclusive advisory lock on a lockfile in `/data`
|
||
at boot. The `scan_lock` table is the second line of defence. Both are needed; either
|
||
alone has a race window.
|
||
|
||
### 11.5 Backup
|
||
|
||
The database is one file. A weekly `sqlite3 .backup` to the volume, keeping four, is
|
||
enough for the bulk of it — items, parts and watch events are all reconstructible from
|
||
Plex and Tautulli, so backing those up only saves a re-ingest.
|
||
|
||
Two things in the database are **not** reconstructible from anything: the saved-view
|
||
definitions and, far more importantly, the **keep marks**. Those encode human judgements
|
||
that exist nowhere else, and losing them means either re-marking hundreds of items by
|
||
hand or — worse, in v2 — deleting something that was explicitly protected.
|
||
|
||
So keeps and views get their own treatment beyond the database backup: a nightly export
|
||
to `/data/keeps.json` and `/data/views.json`, human-readable, containing the durable keys
|
||
(`library`, `guid`, `season_number`) rather than rating keys, plus labels and notes.
|
||
These are small, diffable, and worth committing to the repo. Restoring them into an empty
|
||
database is an import of one file, and because they key on content identity rather than
|
||
rating keys, they restore correctly even into a rebuilt Plex library.
|
||
|
||
---
|
||
|
||
## 12. Security
|
||
|
||
MediaShelf is LAN-only with no authentication, by decision. That is a reasonable posture
|
||
for a read-only tool behind a firewall, and it is a bad posture the moment v2 adds
|
||
deletion. This section records what that means now and what changes later.
|
||
|
||
**v1 posture:**
|
||
|
||
- No auth, no sessions, no user records.
|
||
- Bind to the LAN; if published through NPM, restrict by source IP at the proxy or keep
|
||
the hostname internal-only. It should not be reachable from the internet.
|
||
- `PLEX_TOKEN` lives in the environment only, never in the database, never in logs, never
|
||
in an API response. The `/api/v1/settings` endpoint redacts it.
|
||
- No user input reaches SQL except through the whitelisted rule grammar (§7.1) with bound
|
||
parameters.
|
||
- All rendered strings are escaped; file paths from Plex are treated as untrusted display
|
||
data.
|
||
- CSRF is not a concern in v1 because nothing mutating exists beyond scan triggering and
|
||
saved views. Both use `POST` with a `SameSite=Strict` token anyway, because retrofitting
|
||
CSRF protection after deletion exists is exactly the kind of thing that gets skipped.
|
||
|
||
**Required before v2 ships deletion:** authentication is not optional. A single admin
|
||
password with a signed session, rate-limited login, and an audit log is the minimum. An
|
||
unauthenticated endpoint that deletes files on a 90 TB array is an accident waiting for
|
||
a misconfigured proxy rule.
|
||
|
||
---
|
||
|
||
## 13. Testing
|
||
|
||
| Layer | Approach |
|
||
|---|---|
|
||
| Provider | Recorded JSON fixtures (`responses`/`vcr.py`) for both sources: Plex covering a movie section, a show section with multi-season shows, multi-version movies, split parts; Tautulli covering `get_history` pages, `get_users`, `get_library_media_info`, and an error-shaped 200 response. No live server needed in CI. |
|
||
| History semantics | Disposition classification at threshold boundaries; session merging across the window; a `pre_history` item scoring differently from a truly-never-watched one; Plex-fallback rows renormalizing the weights correctly. |
|
||
| Keep marks | Every precedence combination in §6.6, especially `exclude` overriding a library rule. **Rating-key churn: a mark must still resolve after every `ratingKey` in the library is reassigned** — the single most important test in the suite, since its failure mode in v2 is deleting protected content. Same-guid items in two libraries mark independently. Orphan detection fires when content disappears and does not fire on a rescan that merely renumbers. Export/import round-trips into an empty database. |
|
||
| Normalization | Table-driven tests: odd cases like an item with no `Part`, a season with a missing episode, unicode titles, zero-byte parts. |
|
||
| Ingest | In-memory SQLite; assert idempotency (running the same scan twice changes nothing), assert `missing` transitions, assert history watermarking doesn't skip or duplicate events. |
|
||
| Scoring | Unit tests on known inputs; **property test asserting the SQL and Python implementations agree** (§6.4); tests that grace rules clamp correctly. |
|
||
| Rules | Every operator round-trips to correct SQL; injection attempts in field names and values are rejected with 400. |
|
||
| API | Flask test client, schema assertions on every response shape. |
|
||
| Smoke | A `make smoke` target that runs a real scan against a live Plex and prints the top 20 reclaim candidates — the human sanity check that no unit test replaces. |
|
||
|
||
The single most important test is idempotency: a scanner that double-counts sizes or
|
||
duplicates history events produces a report that looks plausible and is wrong, which is
|
||
worse than one that crashes.
|
||
|
||
### 13.1 Validating the design before writing the app
|
||
|
||
Both Plex and Tautulli are LAN-only, so nothing outside the network can check this design
|
||
against real data. `tools/probe.py` closes that gap: a dependency-free, **read-only**
|
||
script run from anywhere on the LAN that issues GET requests only and answers the open
|
||
questions in §15 directly —
|
||
|
||
- Tautulli's coverage horizon, and therefore how much of the library gets `pre_history`
|
||
- the finished / partial / abandoned split across a sample of real plays, which is the
|
||
assumption the `rejection` component rests on
|
||
- whether successive-play grouping is actually happening, so the session-merge logic
|
||
(§4.9) can be checked against reality
|
||
- real library shapes: movie counts, season counts, total sizes, multi-version items
|
||
- path roots and their sizes, which decides whether grouping by physical vault is worth
|
||
building
|
||
|
||
It also does the `pms_identifier` cross-check from §4.11 and reports whether it passes.
|
||
Its output is safe to share — credentials are redacted from every message, including
|
||
error messages, which is verified by test. `--dump` writes a full item inventory to JSON
|
||
for offline analysis, and prints the twenty largest never-watched items as a first taste
|
||
of what the real report will look like.
|
||
|
||
`tools/reclaim_preview.py` goes further: it joins Plex sizes to Tautulli plays exactly as
|
||
MediaShelf will, and reports never-played bytes per library split into the confident and
|
||
uncertain pools, TV broken out at season granularity, and the watcher-count distribution.
|
||
This is what produced the calibration in §2.1 and §6.1.
|
||
|
||
`tools/mockserver.py` mocks both APIs so the probe can be exercised without touching a
|
||
live server; it is a test fixture, not part of the application.
|
||
|
||
Both scripts were run against the live servers on 2026-09-07 from a LAN host, and their
|
||
findings are folded into this document. Re-running them after any significant library
|
||
change is the cheapest way to check whether the score's constants still fit.
|
||
|
||
---
|
||
|
||
## 14. Roadmap
|
||
|
||
### 14.1 v2 — Deletion
|
||
|
||
The reason v1 is report-only is that the scoring model needs to earn trust before it
|
||
gets to remove anything. When it does, the design is **two-stage quarantine, not
|
||
delete**:
|
||
|
||
1. **Select** items in the UI, review the manifest (every file path, every byte, total).
|
||
2. **Quarantine** — MediaShelf records an intent and moves the files to a trash directory
|
||
on the same filesystem (a rename, so it's instant and atomic regardless of size),
|
||
then triggers a Plex library refresh so the entries disappear from Plex.
|
||
3. **Purge** — after a configurable hold (default 14 days) the quarantined files are
|
||
permanently removed, either automatically or by an explicit click. Until then,
|
||
restore is a rename back.
|
||
|
||
This requires the one thing v1 deliberately avoids: filesystem access to the media
|
||
shares. The v2 container mounts them read-write and gains a path-mapping layer
|
||
translating Plex's paths (`/mnt/titan4/...` as Loki sees them) into container paths.
|
||
Guardrails, in the order they fire:
|
||
|
||
1. **Keep marks are refused first** (§6.6). A kept item cannot be quarantined at all —
|
||
not with a confirmation, not with a force flag. Removing the mark is a separate,
|
||
deliberate act taken beforehand, and the deletion flow never offers to do it inline.
|
||
This is the guardrail that matters most, because it is the one protecting decisions a
|
||
human actually made.
|
||
2. A configured **allowlist of path prefixes** MediaShelf may touch, and a refusal to act
|
||
on any path outside them.
|
||
3. A **dry run** on every destructive operation, listing exactly what would happen, with
|
||
explicit confirmation required.
|
||
4. An **immutable audit log** of every quarantine and purge — what, when, how big, what
|
||
the score was that recommended it, and whether a keep mark was removed in the days
|
||
before. That last field is worth having: "un-kept on Tuesday, deleted on Wednesday" is
|
||
the pattern you want to be able to find after the fact.
|
||
|
||
A scan that runs between marking and deleting must not be able to change a keep
|
||
resolution underneath an in-flight operation, so quarantine re-resolves the keep state at
|
||
the moment of execution rather than trusting what the UI displayed.
|
||
|
||
Deletion also makes authentication mandatory (§12) and makes a dry-run mode
|
||
non-negotiable: every destructive operation gets a preview that lists exactly what would
|
||
happen and requires explicit confirmation.
|
||
|
||
### 14.2 v3 — Emby and Jellyfin
|
||
|
||
The `MediaProvider` protocol is the whole preparation:
|
||
|
||
```python
|
||
class MediaProvider(Protocol):
|
||
def server_info(self) -> ServerInfo: ...
|
||
def libraries(self) -> list[Library]: ...
|
||
def items(self, library: Library) -> Iterator[Item]: ... # yields normalized items
|
||
def refresh_library(self, library: Library) -> None: ... # v2 only
|
||
|
||
class HistoryProvider(Protocol):
|
||
def server_info(self) -> ServerInfo: ... # for the pms_identifier match, §4.11
|
||
def accounts(self) -> list[Account]: ...
|
||
def watch_events(self, since: int | None) -> Iterator[WatchEvent]: ...
|
||
def coverage(self) -> Coverage: ... # earliest/latest event, count
|
||
@property
|
||
def has_completion_data(self) -> bool: ... # False for PlexHistoryProvider
|
||
```
|
||
|
||
Jellyfin and Emby share an API lineage, so one `JellyfinProvider` will likely cover both
|
||
with a capability flag or two. The mapping is close: `Items` with
|
||
`Fields=Path,MediaSources,DateCreated`, `UserData.PlayCount` and `LastPlayedDate` for
|
||
watch data, `/Users` for accounts. Two differences matter. Their watch data is genuinely
|
||
per-user and must be fetched per user and merged, rather than read from one global history
|
||
endpoint. And there is no Tautulli equivalent, so `has_completion_data` is `False` and the
|
||
`rejection` component drops out — the same degraded path the Plex fallback already
|
||
exercises, which is a good reason for that path to exist and be tested from day one
|
||
rather than bolted on later.
|
||
|
||
The `provider` table already carries `kind` and the schema already keys everything on
|
||
`(provider_id, provider_item_id)`, so multiple servers can coexist in one database — the
|
||
door is open to a combined view across Plex and Jellyfin without a migration.
|
||
|
||
### 14.3 Later candidates
|
||
|
||
- **Optional filesystem stat** — if the shares happen to be mounted, verify Plex's sizes
|
||
against reality and surface orphans Plex never indexed.
|
||
- **Duplicate and version analysis** — the same title held at 4K and 1080p is a common
|
||
and invisible source of waste.
|
||
- **Trend tracking** — MediaShelf already snapshots on a schedule; a chart of library
|
||
size over time and a projection of when the vault fills up is nearly free.
|
||
- **Direct Tautulli database read** — Tautulli's SQLite file could be read directly
|
||
instead of paging its API. Measured: 87,640 rows page in about 40 seconds at 10,000 per
|
||
request, which is fine for a nightly job, so this stays unnecessary. It would also
|
||
couple MediaShelf to Tautulli's internal schema.
|
||
- **Per-user reclaim views** — the data to answer "what does only one person watch"
|
||
is already stored; it just has no UI in v1.
|
||
- **Notifications** — a monthly "here's what's gone cold" summary.
|
||
|
||
---
|
||
|
||
## 15. Open questions
|
||
|
||
### Resolved
|
||
|
||
- ~~Is Tautulli running anywhere?~~ **Yes** — 192.168.1.100:8181, on Isis. Promoted to
|
||
the primary watch-history source; see §4.8–4.11.
|
||
- ~~Plex history retention on Loki~~ — no longer on the critical path, since Tautulli
|
||
keeps its own independent record.
|
||
- ~~Where does the image get built?~~ Built off-box, delivered to Nox's Portainer (§11.2).
|
||
|
||
### Still open
|
||
|
||
### Resolved by measurement (2026-09-07)
|
||
|
||
Questions 1–4 were answered by running `tools/probe.py` and `tools/reclaim_preview.py`
|
||
against the live servers. The findings are in §2.1 and changed the design in five places:
|
||
the Tautulli size cross-check (§4.11), both Tautulli-dependent score weights (§6.1),
|
||
cross-library duplicates (§6.5), keep marks (§6.6), and two new seed views (§7.2).
|
||
|
||
- ~~When was Tautulli installed?~~ **2025-03-08**, 547 days of coverage — but the library
|
||
goes back to 2000, so 80% of it predates that. `pre_history` is the majority state.
|
||
- ~~Is successive-play grouping on?~~ Immaterial — MediaShelf requests `grouping=0` and
|
||
merges itself. Confirmed working against 87,640 real rows.
|
||
- ~~Multiple libraries of the same kind?~~ **24 libraries**, 16 of them movie sections.
|
||
- ~~Path structure?~~ **titan4 / titan3 / titan1**, and grouping by root is worth building.
|
||
|
||
### Still open
|
||
|
||
1. **Registry or image upload?** (§11.2) A registry is the better long-term answer but is
|
||
infrastructure that doesn't exist yet. A preference, not a measurement.
|
||
2. **Which libraries, if any, should be kept wholesale?** Deliberately left unanswered —
|
||
nothing ships kept (§6.6), and this is a judgement to make against a real report rather
|
||
than in advance. The candidates the measurements suggest are *Family Videos*
|
||
(irreplaceable home video), *4K Movies* (72.9% never played and expensive to
|
||
re-acquire, though marking it removes 5 TB from consideration permanently), and
|
||
*Sports*, which looks like personal capture. All three are equally well served by
|
||
explicit marks on the items that actually matter.
|
||
3. **What should the 18.8% partial-view band mean?** Four times the abandoned share and
|
||
genuinely ambiguous (§6.1). v1 stores and displays it without scoring it; a month of
|
||
looking at real rows should settle whether it earns a component.
|
||
4. **Is the 20.9 TB uncertain pool worth attacking at all?** It is three times the size of
|
||
the confident pool, and the honest answer is that nobody can tell from data whether
|
||
those items were watched in 2019. Options are to leave it to human judgement, to
|
||
import older history if any exists elsewhere, or to accept that content untouched for
|
||
18+ months is stale enough regardless of what happened before. Worth deciding
|
||
deliberately rather than by default.
|