MediaShelf/docs/design.md

MediaShelf — Software Design

Status: Draft v1.2 · Author: Claude (spec) for Jess · Date: 2026-09-07

Revision 1.1 — Tautulli (192.168.1.100:8181) confirmed present and promoted to the primary watch-history source; Plex's own history demoted to fallback. Deployment path settled: image built in CI/locally and pushed to Nox's Portainer.

Revision 1.2 — validated against the live servers. The measured library (§2.1) is 65.7 TB across 24 libraries with 60 users and 87,640 logged plays. Five corrections followed: Tautulli's per-item table cannot supply TV sizes (§4.11), both Tautulli-derived score weights recalibrated against real distributions (§6.1), cross-library duplicates promoted to v1 (§6.5), protected libraries added (§6.6), and confident-vs-uncertain reclaim separated as distinct seed views (§7.2).

---

1. Overview

MediaShelf is a self-hosted web application that reads a Plex Media Server library, builds a local analytical snapshot of every item in it, and gives a human a fast way to answer one question:

Which of these thousands of files are worth keeping, and which are burning disk for nothing?

It does this by joining together facts Plex already knows but never shows side by side — when a file was added, how large it is, where it lives on disk, which library it belongs to, how many times it has been played, and when it was last played — and exposing them as a sortable, filterable, chartable grid with a tunable "reclaim score" that ranks deletion candidates.

v1 is read-only. MediaShelf will never delete, move, or modify a file or a Plex record in this version. Its output is a report: an on-screen ranked list, saved rule sets, and CSV export. Actual deletion is designed for in §14 but deliberately not built yet, so the scanner and scoring model can be trusted before anything destructive is wired up.

1.1 Goals

• Full-library ingest from Plex over its HTTP API, with no agent installed on the Plex host.
• Watch history from Tautulli, which has been logging every play since it was installed and knows how much of each item was actually watched — not just that a play started.
• A durable local snapshot so the UI is instant and Plex isn't hammered on every click.
• Movies at item level; TV rolled up to season level.
• Sort/filter/visualize on: date added, last watched, watch count, size on disk, file path, library.
• A weighted reclaim score with live sliders, so "what should I delete" is one sort click.
• Saved views — named, re-runnable rule sets like unwatched, older than 2 years, over 10 GB.
• CSV export of any view.
• A provider abstraction clean enough that Emby and Jellyfin drop in later without touching the scoring, storage, or UI layers.

1.2 Non-goals (v1)

• Deleting, moving, or trashing files. Not in v1. See §14.1.
• Writing anything back to Plex (no rating changes, no collection edits, no unmatch/refresh).
• Authentication or multi-user accounts — this is a LAN-only, single-admin tool. See §12.
• Music, photo, and "other video" libraries. Movie and Show sections only.
• Direct filesystem access. MediaShelf never mounts the media shares. All size and path data comes from the Plex API. See §4.4.
• Transcode/quality analysis, duplicate detection, subtitle management.

1.3 Decisions already made

Question	Decision
Deletion in v1	No — report only
Stack	Python + Flask, SQLite
Deployment	Portainer stack on Nox (192.168.1.77), published via Nginx Proxy Manager
Auth	None; LAN-only, single admin view, watch data aggregated across all Plex accounts
TV granularity	Season
Size/path source	Plex API only
Watch-history source	Tautulli primary, Plex session history as fallback
v1 output	Sortable grid + filters + charts, tunable reclaim score, saved rule sets
Image delivery	Built outside Nox, pushed to Nox's Portainer
Implementation	Design doc only for now; code to follow

---

2. Context

The Plex server is Loki (192.168.1.10, Dell R710, Ubuntu 16.04, Plex 1.43.3) with roughly 90 TB of RAID-6 storage across a Dell MD1000 vault. A second vault is powered down awaiting drive upgrades, so reclaiming space on the live vault has real value — this tool exists because buying disks is the more expensive alternative.

MediaShelf runs on Nox, not on Loki. Loki's job is serving media; MediaShelf is a polite HTTP client that talks to it across the LAN and keeps its own database locally. This also means MediaShelf survives Loki reboots, Plex upgrades, and the eventual migration of services onto Enki.

2.1 The library, measured

Everything below was measured against the live servers on 2026-09-07 with tools/probe.py and tools/reclaim_preview.py (§13.1). These are not estimates.

Total size	65.7 TB
Libraries	24 — 16 movie sections, 8 show sections
Movies	2,930 across 16 sections
TV	44,455 episodes → 2,807 seasons across 664 shows
Grid rows at season granularity	5,727
Files (parts)	~47,900
Multi-file items	640, concentrated in Documentary (338 of 2,511 parts)
Storage roots	/mnt/titan4 40.6 TB · /mnt/titan3 24.3 TB · /mnt/titan1 849 GB
Tautulli history	87,640 plays since 2025-03-08, 24,204 distinct items, 60 users
Never played	27.4 TB (41.7%) — of which 6.5 TB confident, 20.9 TB uncertain

Four things here materially changed the design.

This is not a household server. 60 distinct users appear in the history. Every component calibrated for a family of three or four had to be recalibrated (§6.1), and "who else watches this" becomes a real signal rather than a rounding error.

80% of the library predates Tautulli. Coverage begins 2025-03-08, but content goes back to 2000. So of the 27.4 TB never played, only 6.5 TB is confidently unwatched — added while Tautulli was watching and never touched since. The other 20.9 TB might have been watched in 2019 and nobody can tell. The pre_history flag (§4.11) is therefore not an edge case, it is the majority state, and the UI must make the distinction impossible to miss.

24 libraries, not two. Several are genre splits of the same kind of content (Horror, Comedy, War, Superhero, Classic), and Movies / 4K Movies frequently hold the same film twice. Cross-library duplicate detection moves from "later candidate" to a real v1 concern (§6.5).

Some libraries must never be deletion candidates. Family Videos is 63 items and 38 GB of irreplaceable home video, 86% of it "never played". A naive reclaim score ranks it highly. Protected libraries are now a first-class concept (§6.6).

---

3. Architecture

┌──────────────────────────── Nox (192.168.1.77) ────────────────────────────┐
│                                                                            │
│   ┌─────────────────── mediashelf container ───────────────────┐           │
│   │                                                            │           │
│   │   Flask app (gunicorn, 2 workers)                          │           │
│   │     ├── /            → UI (server-rendered shell + JS)     │           │
│   │     ├── /api/*       → JSON API                            │           │
│   │     └── /healthz                                           │           │
│   │                                                            │           │
│   │   APScheduler (in-process, 1 worker only)                  │           │
│   │     └── nightly scan job                                   │           │
│   │                                                            │           │
│   │   Ingestion pipeline                                       │           │
│   │     PlexProvider ──▶ normalize ──▶ upsert ──▶ rollup       │           │
│   │                                                            │           │
│   │   SQLite  /data/mediashelf.db  (WAL)                       │           │
│   └────────────────────────────┬───────────────────────────────┘           │
│                                │ volume: /data                             │
└──────────┬─────────────────────┴───────────────────────────────────────────┘
           │                                        │
           │ HTTP :32400 + X-Plex-Token             │ HTTP :8181 + apikey
           ▼                                        ▼
  Loki (192.168.1.10)                     Isis (192.168.1.100:8181)
  Plex Media Server                       Tautulli
  · libraries, items, parts               · every play since install
  · size, path, added_at                  · per-user, per-item, % complete
  · session history (fallback)

3.1 Component responsibilities

mediashelf.providers — the pluggable source layer. Defines two protocols: MediaProvider (libraries, items, parts) implemented by PlexProvider, and HistoryProvider (watch events) implemented by both TautulliHistoryProvider and PlexHistoryProvider. Everything above this layer speaks only in MediaShelf's own normalized dataclasses. Splitting the two protocols is what lets watch history come from a different box than the library itself — which, given Tautulli lives on Isis and Plex on Loki, is not hypothetical. It is also the seam Emby/Jellyfin slot into (§14.2).

mediashelf.ingest — orchestrates a scan: pull sections, pull items, pull history, normalize, upsert into SQLite inside a transaction, recompute season rollups, mark vanished items, write a scan row.

mediashelf.scoring — pure functions. Takes a row of metrics plus a weight profile, returns a 0–100 reclaim score and the component breakdown. No I/O, fully unit-testable.

mediashelf.rules — evaluates saved-view rule sets into SQL WHERE fragments via a whitelisted field/operator grammar. Never string-interpolates user input into SQL.

mediashelf.api — Flask blueprints returning JSON.

mediashelf.web — one server-rendered HTML shell; the grid, charts, and sliders are client-side JS against /api.

3.2 Why Flask + SQLite

The dataset is small in database terms. A library of ~5,000 movies and ~2,000 seasons across ~60,000 episodes is a few tens of megabytes. SQLite in WAL mode handles this with sub-10 ms queries on indexed columns, needs no second container, and backs up by copying one file. Flask matches the pattern already running for the Mythica portal on Nox, so deployment, logging, and reverse-proxy conventions carry over unchanged.

The one real constraint SQLite imposes is a single writer. Since the only writer is the scan job, and scans run one at a time behind a lock, this is not a limitation. It does mean gunicorn must run the scheduler in exactly one worker — see §11.3.

---

4. Data sources

MediaShelf reads from two servers. Plex on Loki is the authority on what exists — libraries, items, files, sizes, paths, added dates. Tautulli on Isis is the authority on what has been watched. Sections 4.1–4.7 cover Plex; 4.8–4.11 cover Tautulli.

4.1 Plex connection

MediaShelf authenticates with a Plex server token (X-Plex-Token) supplied via environment variable. It talks directly to http://192.168.1.10:32400 on the LAN — no plex.tv round trip, no account login, no dependency on Plex's cloud being up.

All requests carry standard client identification headers so the connection is identifiable in Plex's own logs:

X-Plex-Token: <token>
X-Plex-Client-Identifier: mediashelf-<stable-uuid>
X-Plex-Product: MediaShelf
X-Plex-Version: <app version>
Accept: application/json

Plex returns JSON when Accept: application/json is set, which avoids XML parsing entirely. Every response body is wrapped in a MediaContainer object.

4.2 Endpoints used

Purpose	Endpoint
Server identity / version	GET /identity
List libraries	GET /library/sections
Library detail + on-disk locations	GET /library/sections/{id}
All movies in a section	GET /library/sections/{id}/all?type=1
All episodes in a show section	GET /library/sections/{id}/all?type=4
All seasons in a show section	GET /library/sections/{id}/all?type=3
Global watch history	GET /status/sessions/history/all
Accounts (for history attribution)	GET /accounts

Pagination on every list endpoint uses X-Plex-Container-Start and X-Plex-Container-Size headers (or the equivalent query params). MediaShelf pages at 500 items per request; the MediaContainer.totalSize field on the first response gives the loop its bound.

4.3 The fields that matter

From each Video element on /library/sections/{id}/all:

Plex attribute	Meaning	MediaShelf column
ratingKey	stable per-server item id	provider_item_id
guid	cross-server content identity	guid
type	movie / episode / season / show	kind
title, year	display	title, year
addedAt	epoch seconds, when Plex first saw the file	added_at
updatedAt	epoch seconds, last metadata change	updated_at
duration	milliseconds	duration_ms
viewCount	plays — token-account scoped, see §4.5	(advisory only)
lastViewedAt	epoch seconds — token-account scoped	(advisory only)
librarySectionID	owning library	library_id
grandparentRatingKey, grandparentTitle	(episodes) the show	show_id, show_title
parentRatingKey, parentIndex	(episodes) the season and its number	season_id, season_number

From the nested Media[].Part[] elements:

Plex attribute	MediaShelf column
Part.file	file_path — absolute path on the Plex host
Part.size	size_bytes
Part.container	container
Media.videoResolution, videoCodec, audioCodec, bitrate	resolution, video_codec, audio_codec, bitrate

An item can have several Media entries (multiple versions) each with several Part entries (split files). MediaShelf stores every part as its own row in media_part and treats an item's size as the sum of its parts. This matters: a movie with a 4K and a 1080p version is one item occupying two files, and a reclaim report that reported only the first part would understate the win.

4.4 Why Plex-only, and what it costs

MediaShelf does not mount the media shares. That keeps the container trivially portable, removes any possibility of it touching a file by accident, and means it works from anywhere that can reach port 32400.

The accepted trade-offs:

• Size is Plex's number, not the filesystem's. Plex records Part.size at scan time. If a file is replaced outside Plex and Plex hasn't rescanned, the number is stale. In practice Plex's own scanner keeps this current; the design accepts drift.
• Orphans are invisible. A file on disk that Plex has never indexed will not appear in MediaShelf at all. Finding those needs a filesystem walk, which is explicitly out of scope (see §14.3 for the optional-mount future).
• Paths are Loki-relative. file_path is the path as the Plex server sees it. It is displayed and exported for a human's benefit, not resolved by MediaShelf.

4.5 Why Plex is not the watch-data source

Two problems make Plex's own watch data unfit to build a reclaim tool on.

It is scoped to one account. viewCount and lastViewedAt on a library item reflect only the account that owns the token. Plays by home users, managed users, and shared users are invisible. For a household library this understates watches badly, and a tool that recommends deleting "unwatched" content someone else watches weekly is worse than useless.

It only knows that a play started. /status/sessions/history/all records a playback event. It does not record whether the viewer watched the whole thing or bailed after four minutes. Those two events mean opposite things for a deletion decision, and Plex reports them identically.

/status/sessions/history/all solves the first problem — it is server-wide across all accounts — so it remains the fallback source, and §4.11 covers when it is used. But Tautulli solves both, so Tautulli is the primary.

4.6 Plex scan strategy

Full scan — walks every section, every item, every part, and the complete history. Run on first launch and on demand. Expect a few minutes for a large library.

Incremental scan — the nightly default. Two cheap tricks:

1. Items: request the section sorted by addedAt:desc and stop paging once the page's oldest addedAt predates the last successful scan. This catches additions. To catch removals and metadata edits, a full item sweep still runs weekly (configurable) — an item silently vanishing from Plex is exactly the event that must not be missed.
2. History: request /status/sessions/history/all sorted by viewedAt:desc and stop at the last-seen viewedAt watermark.

Concurrency — one scan at a time, enforced by a row in a scan_lock table plus an in-process lock. A scan that dies leaves a stale lock; locks older than SCAN_LOCK_TIMEOUT (default 2 h) are broken automatically and the orphaned scan row is marked failed.

Politeness — requests are issued serially with a small configurable delay (PLEX_REQUEST_DELAY_MS, default 0) and a hard timeout. Loki is a 2010-vintage R710; the ingest is designed to be slow and steady rather than parallel and aggressive.

4.7 Failure handling

Failure	Behaviour
Plex unreachable	Scan marked failed with the error; UI keeps serving the last good snapshot with a staleness banner
Token rejected (401)	Scan fails fast, prominent UI banner, no retry loop
Section disappears	Its items are marked missing rather than deleted (§5.4)
Malformed / partial item	Item skipped, logged, counted in scan.warning_count; the scan still succeeds
Timeout mid-page	Up to 3 retries with exponential backoff, then the scan fails

4.8 Tautulli — the watch-history source

Tautulli runs at http://192.168.1.100:8181 (on Isis) and has been logging every playback on Loki since the day it was installed. It stays LAN-only and is not published externally — which costs nothing here, since MediaShelf runs on Nox on the same network. The practical consequence is that design validation happens on the LAN: see tools/probe.py (§13.1). Its database is independent of Plex's: clearing Plex's history, or Plex pruning its own, does not touch it. That makes it a deeper and more durable record than anything Plex exposes.

Its API is a single endpoint with a cmd parameter:

GET http://192.168.1.100:8181/api/v2?apikey=<key>&cmd=<command>&<params>

Every response is {"response": {"result": "success"|"error", "message": ..., "data": ...}}. A transport-level 200 with result: "error" in the body is the normal failure mode, so the result field must be checked on every call — treating HTTP 200 as success is the mistake this API invites.

Purpose	Command
Reachability / version	cmd=get_server_info, cmd=arnold (returns a quote; a cheap liveness ping)
Which Plex server it watches	cmd=get_server_info → pms_identifier
Users	cmd=get_users
Libraries	cmd=get_libraries
Watch history	cmd=get_history
Per-item size / play count (cross-check)	cmd=get_library_media_info

4.9 Ingesting history

cmd=get_history
  &grouping=0          # do NOT collapse successive plays — we want raw events
  &order_column=date
  &order_dir=desc
  &start=<offset>
  &length=1000
  &after=<YYYY-MM-DD>  # incremental watermark

grouping=0 matters. Tautulli's Group Successive Play History setting merges a paused- and-resumed play into one row for display purposes. For counting distinct viewings that is what we want — but the grouping should be MediaShelf's decision, applied consistently, not a mirror of however that setting happens to be configured on Isis. So MediaShelf pulls ungrouped events and does its own session collapsing: events for the same item by the same user whose start times fall within SESSION_MERGE_WINDOW (default 6 hours) count as one viewing.

The fields MediaShelf keeps from each history row:

Tautulli field	Use
row_id	idempotency key for upsert
reference_id	Tautulli's own grouping id, kept for cross-checking
date / started / stopped	epoch seconds
rating_key	the leaf item — movie or episode
parent_rating_key, grandparent_rating_key	season and show, for rollup
user_id, user, friendly_name	who watched
media_type	movie / episode
percent_complete	how much was actually watched
watched_status	Tautulli's own verdict against its configured threshold
play_duration, paused_counter	real time spent, minus pauses
platform, player	stored, unused in v1

4.10 What "watched" means

This is the part that most changes the quality of the output, so MediaShelf defines it itself rather than inheriting Tautulli's threshold setting:

Derived value	Rule
completed view	percent_complete >= COMPLETION_THRESHOLD (default 85)
partial view	percent_complete between ABANDON_CEILING (default 15) and the completion threshold
abandoned view	percent_complete < ABANDON_CEILING — started and bailed

MediaShelf uses percent_complete rather than watched_status as the primary signal because watched_status depends on how the threshold is configured on Isis, and that configuration can change without warning. watched_status is stored anyway and shown in the UI as a cross-check; a systematic disagreement between the two is worth seeing.

The rollups per item become:

• watch_count — completed views only
• partial_count, abandoned_count
• last_watched_at — most recent completed view
• last_touched_at — most recent view of any kind, including abandoned
• first_watched_at, distinct_watcher_count
• avg_percent_complete

An abandoned view is evidence for deletion, not against it. Three people started a film and all three quit twenty minutes in: that is a stronger signal to delete than a film nobody ever opened, because the household has now actively rejected it. Plex's history cannot express this at all, and it is the single biggest reason to prefer Tautulli. The scoring model uses it (§6.1, rejection component).

4.11 Fallback, coverage, and cross-checks

When Tautulli is unavailable, MediaShelf falls back to PlexHistoryProvider and /status/sessions/history/all. Events ingested that way are marked source = 'plex' and carry no percent_complete, so they count as completed views (the only assumption available) and are excluded from the rejection component. The UI shows a banner explaining the score is running in degraded mode, because a silently degraded score is a score that gets trusted when it shouldn't be.

Tautulli's coverage horizon. Tautulli only knows about plays since it was installed. Anything watched before that is invisible to it — and those items will look never-watched. On first ingest MediaShelf records history_coverage_since = the oldest event Tautulli holds, and:

• Items with added_at earlier than that date get a pre_history flag.
• The UI shows a badge on those rows and a one-line explanation on the dashboard.
• The staleness score component is capped for pre_history items, so a 2009 movie that was watched in 2015 and never since doesn't get scored as though it had never been watched at all.

Cross-check with get_library_media_info — movie libraries only. Tautulli maintains its own per-item table with file_size, added_at, play_count, and last_played. For movie sections this is a useful second opinion: MediaShelf pulls it once per scan per library and records size disagreements against Plex as scan warnings, not errors.

For show sections it is unusable and must not be called. Measured against the live server (2026-09-07): get_library_media_info on a show section returns one row per show, with file_size: 0 on every row, and passing section_type=episode, season, or show changes nothing — all four return the same 169 show-level rows with zero sizes. A refresh=true rebuild might populate them, but that triggers an expensive scan on Tautulli and was deliberately not attempted.

The consequence is architectural and worth stating plainly: Plex is the only size authority for TV, which is 49 of the library's 66 TB. Tautulli contributes watch data and nothing else for show libraries. Any future code tempted to use Tautulli's table as a convenient size source will silently produce zeroes for three-quarters of the library.

Joining to Plex. Both systems use the same Plex rating_key, so the join is direct — no title or path matching, no fuzzy logic. If get_server_info.pms_identifier does not match the machineIdentifier from Plex's /identity, Tautulli is watching a different Plex server and MediaShelf refuses to use it rather than joining nonsense data together. That check runs at the start of every scan.

---

5. Data model

SQLite, WAL mode, foreign keys on. Migrations are plain numbered SQL files applied at startup, tracked in a schema_version table — no ORM migration framework.

5.1 Tables

-- One row per configured media server. v1 has exactly one, but the column exists
-- so Emby/Jellyfin can be added without a migration.
CREATE TABLE provider (
    id              INTEGER PRIMARY KEY,
    kind            TEXT NOT NULL,          -- 'plex' | 'emby' | 'jellyfin'
    name            TEXT NOT NULL,
    base_url        TEXT NOT NULL,
    server_id       TEXT,                   -- Plex machineIdentifier
    version         TEXT,
    last_scan_id    INTEGER,
    created_at      INTEGER NOT NULL
);

CREATE TABLE library (
    id              INTEGER PRIMARY KEY,
    provider_id     INTEGER NOT NULL REFERENCES provider(id),
    provider_key    TEXT NOT NULL,          -- Plex section id
    title           TEXT NOT NULL,
    kind            TEXT NOT NULL,          -- 'movie' | 'show'
    locations       TEXT,                   -- JSON array of on-disk roots
    protected       INTEGER NOT NULL DEFAULT 0,  -- never a delete candidate (§6.6)
    scanned_at      INTEGER,
    UNIQUE (provider_id, provider_key)
);

-- The unit of analysis. One row per movie, and one row per SEASON.
-- Shows themselves get a row too (kind='show') for grouping and display,
-- but carry no size of their own.
CREATE TABLE media_item (
    id                  INTEGER PRIMARY KEY,
    provider_id         INTEGER NOT NULL REFERENCES provider(id),
    library_id          INTEGER NOT NULL REFERENCES library(id),
    provider_item_id    TEXT NOT NULL,      -- Plex ratingKey
    kind                TEXT NOT NULL,      -- 'movie' | 'show' | 'season'
    guid                TEXT,
    title               TEXT NOT NULL,
    sort_title          TEXT,
    year                INTEGER,
    -- season/show linkage
    parent_id           INTEGER REFERENCES media_item(id),   -- season -> show
    season_number       INTEGER,
    -- provenance
    added_at            INTEGER,            -- epoch s; for a season, MIN over episodes
    updated_at          INTEGER,
    -- rollups (denormalized, recomputed each scan)
    episode_count       INTEGER NOT NULL DEFAULT 0,
    size_bytes          INTEGER NOT NULL DEFAULT 0,
    duration_ms         INTEGER NOT NULL DEFAULT 0,
    part_count          INTEGER NOT NULL DEFAULT 0,
    primary_path        TEXT,               -- representative file path for display
    resolution          TEXT,
    video_codec         TEXT,
    -- watch rollups (from watch_event)
    watch_count             INTEGER NOT NULL DEFAULT 0,  -- COMPLETED views only
    partial_count           INTEGER NOT NULL DEFAULT 0,
    abandoned_count         INTEGER NOT NULL DEFAULT 0,
    last_watched_at         INTEGER,                     -- last completed view
    last_touched_at         INTEGER,                     -- last view of any kind
    first_watched_at        INTEGER,
    distinct_watcher_count  INTEGER NOT NULL DEFAULT 0,
    avg_percent_complete    REAL,
    pre_history             INTEGER NOT NULL DEFAULT 0,  -- added before history coverage
    provider_view_count     INTEGER NOT NULL DEFAULT 0,  -- advisory, token-scoped
    -- lifecycle
    status              TEXT NOT NULL DEFAULT 'present',  -- 'present' | 'missing'
    first_seen_scan_id  INTEGER,
    last_seen_scan_id   INTEGER,
    UNIQUE (provider_id, provider_item_id)
);

-- Episodes are ingested (they carry the size and the history), but they are NOT
-- the unit of analysis. They exist so seasons can be rolled up accurately and so
-- a season can be expanded in the UI.
CREATE TABLE episode (
    id                  INTEGER PRIMARY KEY,
    season_item_id      INTEGER NOT NULL REFERENCES media_item(id) ON DELETE CASCADE,
    provider_item_id    TEXT NOT NULL,
    episode_number      INTEGER,
    title               TEXT,
    added_at            INTEGER,
    duration_ms         INTEGER,
    size_bytes          INTEGER NOT NULL DEFAULT 0,
    watch_count         INTEGER NOT NULL DEFAULT 0,
    last_watched_at     INTEGER,
    status              TEXT NOT NULL DEFAULT 'present',
    UNIQUE (provider_item_id)
);

-- Every physical file. A movie with two versions has two rows.
CREATE TABLE media_part (
    id                  INTEGER PRIMARY KEY,
    media_item_id       INTEGER REFERENCES media_item(id) ON DELETE CASCADE,
    episode_id          INTEGER REFERENCES episode(id) ON DELETE CASCADE,
    provider_part_id    TEXT,
    file_path           TEXT NOT NULL,
    size_bytes          INTEGER NOT NULL DEFAULT 0,
    container           TEXT,
    resolution          TEXT,
    video_codec         TEXT,
    audio_codec         TEXT,
    bitrate             INTEGER,
    CHECK ((media_item_id IS NULL) != (episode_id IS NULL))
);

-- One row per playback event, server-wide, all accounts.
-- Primary source is Tautulli; 'plex' rows are degraded fallback events (§4.11).
CREATE TABLE watch_event (
    id                  INTEGER PRIMARY KEY,
    provider_id         INTEGER NOT NULL REFERENCES provider(id),
    source              TEXT NOT NULL,      -- 'tautulli' | 'plex'
    source_row_id       TEXT NOT NULL,      -- Tautulli row_id, or Plex historyKey
    reference_id        TEXT,               -- Tautulli's own grouping id
    provider_item_id    TEXT NOT NULL,      -- the leaf: movie or episode ratingKey
    account_id          TEXT,
    viewed_at           INTEGER NOT NULL,   -- 'date' / 'started'
    stopped_at          INTEGER,
    play_duration_s     INTEGER,
    paused_counter_s    INTEGER,
    percent_complete    INTEGER,            -- NULL for source='plex'
    watched_status      REAL,               -- Tautulli's verdict, advisory
    disposition         TEXT NOT NULL,      -- 'completed' | 'partial' | 'abandoned'
    session_id          TEXT,               -- MediaShelf's own merge key (§4.9)
    media_type          TEXT,
    platform            TEXT,
    UNIQUE (provider_id, source, source_row_id)
);

CREATE TABLE account (
    id              INTEGER PRIMARY KEY,
    provider_id     INTEGER NOT NULL REFERENCES provider(id),
    account_id      TEXT NOT NULL,          -- Tautulli user_id / Plex accountID
    name            TEXT,
    friendly_name   TEXT,
    UNIQUE (provider_id, account_id)
);

-- What the history record actually covers, so 'never watched' can be
-- distinguished from 'watched before Tautulli existed' (§4.11).
CREATE TABLE history_coverage (
    id                  INTEGER PRIMARY KEY,
    provider_id         INTEGER NOT NULL REFERENCES provider(id),
    source              TEXT NOT NULL,
    earliest_event_at   INTEGER,
    latest_event_at     INTEGER,
    event_count         INTEGER NOT NULL DEFAULT 0,
    updated_at          INTEGER NOT NULL,
    UNIQUE (provider_id, source)
);

CREATE TABLE scan (
    id                  INTEGER PRIMARY KEY,
    provider_id         INTEGER NOT NULL REFERENCES provider(id),
    mode                TEXT NOT NULL,      -- 'full' | 'incremental'
    trigger             TEXT NOT NULL,      -- 'manual' | 'schedule' | 'startup'
    status              TEXT NOT NULL,      -- 'running' | 'succeeded' | 'failed'
    started_at          INTEGER NOT NULL,
    finished_at         INTEGER,
    items_seen          INTEGER DEFAULT 0,
    items_added         INTEGER DEFAULT 0,
    items_updated       INTEGER DEFAULT 0,
    items_missing       INTEGER DEFAULT 0,
    events_added        INTEGER DEFAULT 0,
    warning_count       INTEGER DEFAULT 0,
    error               TEXT
);

CREATE TABLE saved_view (
    id              INTEGER PRIMARY KEY,
    name            TEXT NOT NULL UNIQUE,
    description     TEXT,
    rules           TEXT NOT NULL,          -- JSON rule group, see §7
    sort            TEXT,                   -- JSON: [{field, dir}]
    columns         TEXT,                   -- JSON array of column keys
    weights         TEXT,                   -- JSON weight profile, see §6
    created_at      INTEGER NOT NULL,
    updated_at      INTEGER NOT NULL
);

CREATE TABLE setting (
    key             TEXT PRIMARY KEY,
    value           TEXT NOT NULL
);

5.2 Indexes

CREATE INDEX ix_item_library     ON media_item(library_id, kind, status);
CREATE INDEX ix_item_added       ON media_item(added_at);
CREATE INDEX ix_item_lastwatch   ON media_item(last_watched_at);
CREATE INDEX ix_item_size        ON media_item(size_bytes);
CREATE INDEX ix_item_watchcount  ON media_item(watch_count);
CREATE INDEX ix_item_parent      ON media_item(parent_id);
CREATE INDEX ix_item_guid        ON media_item(guid);   -- duplicate grouping, §6.5
CREATE INDEX ix_episode_season   ON episode(season_item_id);
CREATE INDEX ix_part_item        ON media_part(media_item_id);
CREATE INDEX ix_part_episode     ON media_part(episode_id);
CREATE INDEX ix_event_item       ON watch_event(provider_item_id);
CREATE INDEX ix_event_viewed     ON watch_event(viewed_at);
CREATE INDEX ix_event_disp       ON watch_event(provider_item_id, disposition);
CREATE INDEX ix_event_account    ON watch_event(account_id, viewed_at);

A full-text index over title uses SQLite FTS5 (media_item_fts) kept in sync by triggers, so the grid's search box is instant even at 100k rows.

5.3 Season rollup rules

After items and history are ingested, a rollup pass recomputes each season row:

Season field	Rule
size_bytes	SUM(episode.size_bytes)
episode_count	COUNT(episode)
duration_ms	SUM(episode.duration_ms)
added_at	MIN(episode.added_at) — when the season first landed
watch_count	SUM(episode.watch_count) — completed episode views
abandoned_count	SUM(episode.abandoned_count)
last_watched_at	MAX(episode.last_watched_at)
last_touched_at	MAX(episode.last_touched_at)
primary_path	the common directory prefix of its episodes' paths

A show row rolls up the same way from its seasons, for display and for the "delete the whole show" case.

The watch_count choice deserves a note: summing episode plays means a 24-episode season watched once has watch_count = 24, while a movie watched once has watch_count = 1. Comparing those raw numbers across kinds is meaningless. So the scoring model (§6) uses a normalized completion figure for TV — watch_count / episode_count — rather than the raw sum, and the grid displays both (24 plays · 1.0× season).

5.4 Missing items

Nothing is ever hard-deleted by a scan. When a full sweep completes and an item that was present is no longer returned by Plex, its status flips to missing and last_seen_scan_id freezes. Missing items are hidden from the default grid but remain queryable — they're the record of what left the library and when, which is exactly what you want the first time something disappears unexpectedly.

---

6. The reclaim score

The score answers: how much do I gain by deleting this, weighed against how likely I am to want it? It is a weighted sum of normalized components, each in [0, 1], scaled to 0–100. Higher means "better candidate for deletion".

6.1 Components

Component	Definition	Rationale
size	log10(1 + size_bytes) / log10(1 + max_size_in_library)	Log scale — the difference between 2 GB and 4 GB matters more than between 60 GB and 62 GB
staleness	min(days_since_last_watch / STALE_HORIZON, 1), where a never-watched item scores 1.0	How long since anyone cared
unpopularity	1 - min(normalized_watches / POPULAR_AT, 1) where normalized_watches = watch_count for movies, watch_count / episode_count for seasons	Rewatched things are keepers
age	min(days_since_added / AGE_HORIZON, 1)	Something added last week deserves a chance
solitude	1 - min(distinct_watcher_count / SOLITUDE_AT, 1)	Content almost nobody watches is safer to cut
rejection	min(abandoned_count / REJECTED_AT, 1), zero when watch_count > 0	Started and quit is an active verdict — see below

STALE_HORIZON defaults to 730 days, AGE_HORIZON to 1095 days, POPULAR_AT to 3, REJECTED_AT to 2, SOLITUDE_AT to 3.

Both of the Tautulli-dependent components were calibrated against real measurements rather than guessed, and both moved:

solitude — divisor 3 confirmed, weight raised. The original divisor assumed a household. With 60 users the concern was that it would saturate. It does not: of 24,204 watched items, 61.7% were watched by exactly one user and 20.0% by two, with a long thin tail. Three distinct watchers already puts an item in the top ~8% of the library, so a divisor of 3 is the right shape. What changed is that on a 60-user server "only one person ever watched this" is a genuinely informative statement rather than noise, so the weight goes up from 0.04 to 0.10.

rejection — real but smaller than hoped, weight cut. Across all 87,640 plays the split is 76.3% finished · 18.8% partial · 4.9% abandoned. Abandonment is a real signal but a thin one, so its weight drops from 0.12 to 0.06. It stays in the model because when it does fire it is unusually decisive — an item several people started and none finished is a much stronger delete case than the score's other components can express — but it should not be pushing the ranking around on 5% of the evidence.

The larger surprise is the 18.8% partial band. That is four times the abandoned share and it is genuinely ambiguous: someone 60% through an episode may have been interrupted and intending to return, or may have quietly given up. v1 deliberately does not score it in either direction; it is stored, displayed, and filterable, and the first month of looking at real rows should settle whether it deserves a component of its own.

The rejection component only exists because Tautulli supplies percent_complete (§4.10). It is deliberately zeroed the moment anyone completes a view, so a film that was abandoned twice and then watched through is not penalised. When history runs in Plex-fallback mode this component is unavailable and drops out of the weighting entirely (§4.11).

6.2 The formula

score = 100 × Σ(wᵢ · componentᵢ) / Σ(wᵢ)

Default weight profile:

{ "size": 0.28, "staleness": 0.24, "unpopularity": 0.22,
  "solitude": 0.10, "age": 0.10, "rejection": 0.06 }

The denominator sums only the weights of components that are available for that row. When history is in Plex-fallback mode, rejection is dropped and the remaining weights renormalize, rather than every item silently scoring lower because one component is pinned at zero. The same applies per-row: a pre_history item has its staleness capped (§4.11), and the UI shows which components contributed.

6.3 The grace rule

Two hard overrides, applied after scoring, because a purely numeric model will eventually recommend something obviously wrong:

• New-arrival grace — an item with added_at inside GRACE_DAYS (default 30) is clamped to score 0 and flagged grace: new. You haven't had a chance to watch it yet.
• Recent-watch grace — an item watched inside RECENT_DAYS (default 90) is clamped to a maximum of 25 and flagged grace: recent.

Both are toggleable per view, and the UI shows why an item was clamped rather than silently hiding it.

6.4 Where it's computed

In SQL, as a generated expression in the query, not as a stored column. Weights change every time a slider moves; storing the score would mean recomputing 100k rows per drag. The normalization constants (max_size_in_library, horizons) are computed once per request in a CTE. At this row count the whole thing runs in single-digit milliseconds.

The same function is implemented in Python in mediashelf.scoring for unit tests and CSV export, with a property test asserting the SQL and Python implementations agree to within 0.01 on a generated corpus. Two implementations of the same formula is a real risk, and that test is the mitigation.

6.5 Cross-library duplicates

With 16 movie sections, the same film is often held more than once — most obviously Movies vs 4K Movies, but genre sections overlap too. Two copies of a film are 100% redundant storage, and neither copy looks like a deletion candidate on its own: each has its own watch history, and plays are split between them, so both score as moderately watched.

MediaShelf detects this using the Plex guid, which identifies the content rather than the file, and is the same across sections. Items sharing a guid are grouped, and the group is surfaced with:

• total redundant bytes (group size minus its largest member)
• which copy is played and which is not — the common case is that everyone watches the 4K copy and the 1080p copy has not been touched in a year
• resolution and bitrate of each copy, so "keep the best one" is a visible decision

This is a report, not an automatic recommendation. Deciding that a 4K copy supersedes a 1080p one depends on what plays back well on the household's devices, which MediaShelf has no way to know. It surfaces the pairs and their sizes and lets a human choose.

A duplicates flag on the item and a Duplicate groups view are the v1 surface. This was originally deferred to §14.3; the measured library shape moved it forward.

6.6 Protected libraries

Family Videos is 63 items, 38 GB, and 86% "never played". Every component of the reclaim score ranks it as an excellent deletion candidate. It is also irreplaceable home video — the one category in the entire library where a wrong deletion cannot be undone by re-acquiring the file.

So libraries can be marked protected, and MediaShelf applies that at three levels:

1. Protected items are excluded from the reclaim score entirely — they show a — rather than a number, so they can never sort to the top of a delete list.
2. They are excluded from CSV exports of candidate views by default.
3. In v2, when deletion exists, they are refused outright: the path allowlist (§14.1) will not include a protected library's roots, so a deletion request against one fails at the guardrail rather than at the confirmation dialog.

Family Videos ships protected by default, based on the measured library. The setting is per-library and editable; Sports and any other personal-capture library are worth the same treatment.

The general principle is worth stating, because it applies beyond this one library: a reclaim score measures what is unwatched, not what is valuable. Those correlate for purchased and downloaded content and are close to unrelated for anything the household made itself. The score cannot tell the difference, so a human marks the boundary once and the tool respects it permanently.

---

7. Saved views and rules

A saved view bundles four things: a rule set, a sort, a column selection, and a weight profile. Naming one — "Big unwatched movies" — makes it a one-click, re-runnable report that re-evaluates against fresh data every time it's opened.

7.1 Rule grammar

Rules are a nested JSON group, evaluated into parameterized SQL:

{
  "op": "and",
  "rules": [
    { "field": "kind",            "op": "in",     "value": ["movie"] },
    { "field": "size_bytes",      "op": "gte",    "value": 10737418240 },
    { "field": "watch_count",     "op": "eq",     "value": 0 },
    { "field": "added_at",        "op": "older_than_days", "value": 730 },
    { "op": "or", "rules": [
        { "field": "library_id",  "op": "in",     "value": [1, 4] },
        { "field": "resolution",  "op": "eq",     "value": "4k" }
    ]}
  ]
}

Fields are drawn from a whitelist mapping field name → column and type. Operators are a fixed set: eq, ne, lt, lte, gt, gte, in, not_in, contains, starts_with, is_null, is_not_null, older_than_days, newer_than_days, never (for watch fields). The whitelisted fields include the completion metrics — watch_count, partial_count, abandoned_count, avg_percent_complete, last_touched_at, pre_history. Anything outside the whitelist is a 400. Values are always bound parameters. There is no SQL string interpolation anywhere in this path.

7.2 Seed views shipped by default

Name	Rule
Never watched, large	watch_count = 0 AND size_bytes >= 8 GB
Cold storage	last_watched_at older than 3 years
One-and-done movies	kind = movie AND watch_count = 1 AND last_watched_at older than 2 years
Abandoned seasons	kind = season AND watch_count/episode_count < 0.3 AND added_at older than 1 year
Tried and rejected	abandoned_count >= 2 AND watch_count = 0 — the household said no
Bailed in the first 15 min	avg_percent_complete < 15 AND watch_count = 0 AND size_bytes >= 4 GB
Biggest 100	sorted by size_bytes desc, no filter
Recently added	added_at newer than 30 days — the sanity check, not a delete list
Confident reclaim	watch_count = 0 AND pre_history = false — added while Tautulli was watching and never played. On the measured library this is 6.5 TB and is the list to act on first
Uncertain reclaim	watch_count = 0 AND pre_history = true — 20.9 TB that might have been watched before 2025-03-08. Needs human judgement, not a bulk action
Duplicate groups	items sharing a guid across libraries (§6.5)

---

8. HTTP API

All JSON, all under /api/v1. No auth in v1 (§12).

8.1 Library data

GET /api/v1/items

Query parameters:

Param	Meaning
library_id	repeatable filter
kind	movie / season / show
q	FTS search over title
rules	URL-encoded JSON rule group (§7.1)
view_id	apply a saved view instead of ad-hoc rules
sort	e.g. reclaim_score:desc, size_bytes:desc
weights	URL-encoded JSON weight profile
include_missing	default false
page, page_size	page_size max 500

Response:

{
  "total": 4821,
  "page": 1,
  "page_size": 100,
  "aggregate": { "total_size_bytes": 41234567890123, "item_count": 4821 },
  "items": [
    {
      "id": 1183,
      "kind": "movie",
      "title": "Example Film",
      "year": 2011,
      "library": { "id": 1, "title": "Movies" },
      "size_bytes": 32212254720,
      "added_at": 1490000000,
      "last_watched_at": null,
      "last_touched_at": 1712000000,
      "watch_count": 0,
      "partial_count": 0,
      "abandoned_count": 3,
      "avg_percent_complete": 8.4,
      "distinct_watcher_count": 3,
      "pre_history": false,
      "episode_count": null,
      "primary_path": "/mnt/vault2/movies/Example Film (2011)/Example Film (2011).mkv",
      "part_count": 1,
      "resolution": "4k",
      "reclaim_score": 91.4,
      "reclaim_components": {
        "size": 0.97, "staleness": 1.0, "unpopularity": 1.0,
        "rejection": 1.0, "age": 1.0, "solitude": 0.0
      },
      "grace": null,
      "flags": ["rejected"]
    }
  ]
}

Other endpoints:

GET  /api/v1/items/{id}            full detail incl. every part and, for seasons, episodes
GET  /api/v1/libraries             list with item counts and total size
GET  /api/v1/accounts              Plex accounts seen in history
GET  /api/v1/stats/overview        headline numbers for the dashboard
GET  /api/v1/stats/size-by-library
GET  /api/v1/stats/added-over-time?bucket=month
GET  /api/v1/stats/watch-distribution
GET  /api/v1/stats/size-vs-lastwatched   scatter data, downsampled server-side
GET  /api/v1/stats/completion            finished / abandoned / never-opened, by size
GET  /api/v1/sources                     per-source status: reachable?, coverage window,
                                         event count, active history source

8.2 Saved views

GET    /api/v1/views
POST   /api/v1/views
GET    /api/v1/views/{id}
PUT    /api/v1/views/{id}
DELETE /api/v1/views/{id}

8.3 Scans

GET  /api/v1/scans                 history, newest first
GET  /api/v1/scans/current         null or the running scan with progress
POST /api/v1/scans                 { "mode": "full" | "incremental" } → 202 or 409 if one is running

Scan progress is polled at /api/v1/scans/current every 2 s while one is running. No websockets — polling one row is cheaper than the complexity.

8.4 Export

GET /api/v1/export.csv?<same params as /items>

Streams the full result set — not just the current page — as CSV with the currently selected columns plus the reclaim score and its components. This is the v1 deliverable: the artifact you take away and act on.

8.5 Health

GET /healthz    → { "status": "ok", "db": "ok",
                    "plex": "ok"|"unreachable",
                    "tautulli": "ok"|"unreachable"|"disabled",
                    "history_source": "tautulli"|"plex",
                    "last_scan_at": 1757000000, "stale": false }

---

9. Web UI

Server-rendered Flask shell, vanilla JS plus two small libraries: a virtualized table (TanStack Table core or Tabulator) and a charting library. No build step, no npm in the container — assets are vendored. This keeps the container small and the deploy identical to the Mythica pattern.

9.1 Screens

Dashboard — the landing page. Headline tiles (total items, total size, size of never-watched content, size of content untouched for 2+ years, last scan time), plus:

• Size by library — horizontal bar.
• Added over time — monthly bars, stacked by library. Shows acquisition habits.
• Watched vs unwatched by size — the "how much of this shelf has anyone ever touched" chart, which is usually the one that motivates the whole exercise. Split three ways now that completion data exists: finished · started and abandoned · never opened. The middle band is the interesting one and is invisible without Tautulli.
• Size vs. last watched — scatter, log-size on Y, days-since-watch on X. Points in the upper right are the reclaim targets, and clicking a region filters the grid to it.

Library grid — the main workspace. A virtualized table, every column sortable, with:

• A filter rail: library checkboxes, kind, size range, added range, last-watched range, watch-count range, resolution, and a free-text title search.
• A weight panel with five sliders and a live-updating reclaim_score column.
• Multi-select via checkboxes, shift-click ranges, and select-all-matching-filter. In v1 selection drives the aggregate footer ("312 items selected · 4.1 TB") and the CSV export. The buttons that will eventually delete are present but disabled, with a tooltip explaining v1 is report-only — so the workflow is proven before it's armed.
• Row expansion: a season expands to its episodes; any row expands to its file parts with full paths.

Saved views — list, create, edit, duplicate, run. Creating one captures the current filter, sort, columns, and weights, so the flow is explore, then name what you found.

Item detail — everything known: all parts with paths and sizes, a per-account watch timeline showing each play with how far it got (a row of 8%, 12%, 6% tells the story at a glance), and the score breakdown showing each component's contribution.

Scans — history table, per-scan counts and duration, a "Scan now" button, live progress.

Settings — connection status, scan schedule, score defaults, horizons, grace periods.

9.2 Interaction principles

• Every number that represents bytes is displayed in both human units and, on hover, the exact byte count. Reclaim decisions turn on real numbers.
• The reclaim score column always shows its components on hover. A score you can't interrogate is a score you won't trust, and won't act on.
• Filters live in the URL query string, so any view is linkable and back/forward works.
• Nothing in v1 is destructive, and the UI says so plainly in the header rather than leaving the user to wonder.

---

10. Configuration

All via environment variables, so the Portainer stack is the single source of truth.

Variable	Default	Notes
MEDIASHELF_SECRET_KEY	—	required, Flask session signing
PLEX_BASE_URL	—	required, e.g. http://192.168.1.10:32400
PLEX_TOKEN	—	required, secret
PLEX_VERIFY_SSL	true
PLEX_TIMEOUT_S	30
PLEX_PAGE_SIZE	500
PLEX_REQUEST_DELAY_MS	0	throttle for Loki's sake
TAUTULLI_BASE_URL	—	e.g. http://192.168.1.100:8181 — unset disables Tautulli
TAUTULLI_API_KEY	—	secret
TAUTULLI_TIMEOUT_S	30
TAUTULLI_PAGE_SIZE	1000	length on get_history
HISTORY_SOURCE	auto	auto | tautulli | plex — auto prefers Tautulli, falls back
SESSION_MERGE_WINDOW_H	6	events merged into one viewing (§4.9)
COMPLETION_THRESHOLD	85	percent complete counting as a watch
ABANDON_CEILING	15	percent complete below which a play is "abandoned"
DATABASE_PATH	/data/mediashelf.db	on the mounted volume
SCAN_SCHEDULE_CRON	0 4 * * *	nightly incremental at 04:00
SCAN_FULL_SWEEP_CRON	0 3 * * 0	weekly full sweep, Sunday 03:00
SCAN_ON_STARTUP	false
SCAN_LOCK_TIMEOUT_S	7200
SCORE_STALE_HORIZON_DAYS	730
SCORE_AGE_HORIZON_DAYS	1095
SCORE_POPULAR_AT	3
SCORE_REJECTED_AT	2	abandoned views for a full rejection score
SCORE_SOLITUDE_AT	3	distinct watchers before solitude reaches zero
PROTECTED_LIBRARIES	Family Videos	comma-separated; never scored (§6.6)
SCORE_GRACE_DAYS	30
SCORE_RECENT_DAYS	90
TZ	America/Regina	so cron times mean what they look like
LOG_LEVEL	INFO

PLEX_TOKEN and TAUTULLI_API_KEY are secrets and must not be committed. The repo ships .env.example with placeholders, and .env is gitignored. (Worth noting given the SOAP-password-in-history problem on the Mythica repo — this one starts clean and stays clean.) Neither credential is ever written to the database, the logs, or an API response; /api/v1/settings redacts both.

---

11. Deployment

11.1 Container

Single image, python:3.12-slim base, multi-stage so build deps don't ship. Runs as a non-root user. One volume: /data. One port: 8080.

11.2 Getting the image onto Nox

The image is built off-box and delivered to Nox's Portainer rather than built there. Two workable routes, in order of preference:

1. Registry — build, tag registry.hallsworth.ca/mediashelf:<version> (or Forgejo's own container registry, which this Forgejo version supports), push, and have the Portainer stack pull it. This is the route worth setting up once: redeploys become a version bump in the stack, and rollback is pulling the previous tag.
2. Image upload — docker save mediashelf:<version> | gzip and load it on Nox via Portainer's Images → Import. No registry needed, fine for the first deploy or two, but it makes rollback manual and versions easy to lose track of.

Either way the image is tagged with a real version, never deployed as bare latest. The Mythica stack had to be torn down and recreated because a PUT update kept serving old code; immutable version tags are how that failure mode is avoided rather than worked around.

11.3 Portainer stack on Nox

services:
  mediashelf:
    image: registry.hallsworth.ca/mediashelf:0.1.0   # never bare :latest, see 11.2
    container_name: mediashelf
    restart: unless-stopped
    ports:
      - "8085:8080"
    volumes:
      - mediashelf_data:/data
    environment:
      - MEDIASHELF_SECRET_KEY=${MEDIASHELF_SECRET_KEY}
      - PLEX_BASE_URL=${PLEX_BASE_URL}
      - PLEX_TOKEN=${PLEX_TOKEN}
      - TAUTULLI_BASE_URL=${TAUTULLI_BASE_URL}
      - TAUTULLI_API_KEY=${TAUTULLI_API_KEY}
      - HISTORY_SOURCE=auto
      - DATABASE_PATH=/data/mediashelf.db
      - SCAN_SCHEDULE_CRON=${SCAN_SCHEDULE_CRON}
      - TZ=America/Regina
      - LOG_LEVEL=INFO
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request;urllib.request.urlopen('http://localhost:8080/healthz')"]
      interval: 60s
      timeout: 10s
      retries: 3

volumes:
  mediashelf_data:

Every variable used in the stack must be listed under environment: here, not only set in Portainer's UI. A variable set in Portainer that isn't declared in the compose file silently does nothing — the same trap that bit PLAYERMAP_SHOW_GMS on the Mythica stack.

Published through Nginx Proxy Manager at mediashelf.hallsworth.ca, upstream 192.168.1.77:8085. Note the host-port → container-port mapping is 8085 → 8080; NPM's upstream must point at the host port.

11.4 The gunicorn / scheduler constraint

APScheduler runs in-process. With --workers 2, both workers would start a scheduler and two scans would race for the SQLite write lock. Mitigation: the scheduler starts only in the worker that successfully acquires an exclusive advisory lock on a lockfile in /data at boot. The scan_lock table is the second line of defence. Both are needed; either alone has a race window.

11.5 Backup

The database is one file. A weekly sqlite3 .backup to the volume, keeping four, is enough — everything in it is reconstructible from Plex and Tautulli, so the real value is avoiding a multi-hour re-ingest rather than protecting irreplaceable data. The one genuinely irreplaceable thing is the saved-view definitions, which are small and worth exporting to the repo as JSON once they stabilise.

---

12. Security

MediaShelf is LAN-only with no authentication, by decision. That is a reasonable posture for a read-only tool behind a firewall, and it is a bad posture the moment v2 adds deletion. This section records what that means now and what changes later.

v1 posture:

• No auth, no sessions, no user records.
• Bind to the LAN; if published through NPM, restrict by source IP at the proxy or keep the hostname internal-only. It should not be reachable from the internet.
• PLEX_TOKEN lives in the environment only, never in the database, never in logs, never in an API response. The /api/v1/settings endpoint redacts it.
• No user input reaches SQL except through the whitelisted rule grammar (§7.1) with bound parameters.
• All rendered strings are escaped; file paths from Plex are treated as untrusted display data.
• CSRF is not a concern in v1 because nothing mutating exists beyond scan triggering and saved views. Both use POST with a SameSite=Strict token anyway, because retrofitting CSRF protection after deletion exists is exactly the kind of thing that gets skipped.

Required before v2 ships deletion: authentication is not optional. A single admin password with a signed session, rate-limited login, and an audit log is the minimum. An unauthenticated endpoint that deletes files on a 90 TB array is an accident waiting for a misconfigured proxy rule.

---

13. Testing

Layer	Approach
Provider	Recorded JSON fixtures (responses/vcr.py) for both sources: Plex covering a movie section, a show section with multi-season shows, multi-version movies, split parts; Tautulli covering get_history pages, get_users, get_library_media_info, and an error-shaped 200 response. No live server needed in CI.
History semantics	Disposition classification at threshold boundaries; session merging across the window; a pre_history item scoring differently from a truly-never-watched one; Plex-fallback rows renormalizing the weights correctly.
Normalization	Table-driven tests: odd cases like an item with no Part, a season with a missing episode, unicode titles, zero-byte parts.
Ingest	In-memory SQLite; assert idempotency (running the same scan twice changes nothing), assert missing transitions, assert history watermarking doesn't skip or duplicate events.
Scoring	Unit tests on known inputs; property test asserting the SQL and Python implementations agree (§6.4); tests that grace rules clamp correctly.
Rules	Every operator round-trips to correct SQL; injection attempts in field names and values are rejected with 400.
API	Flask test client, schema assertions on every response shape.
Smoke	A make smoke target that runs a real scan against a live Plex and prints the top 20 reclaim candidates — the human sanity check that no unit test replaces.

The single most important test is idempotency: a scanner that double-counts sizes or duplicates history events produces a report that looks plausible and is wrong, which is worse than one that crashes.

13.1 Validating the design before writing the app

Both Plex and Tautulli are LAN-only, so nothing outside the network can check this design against real data. tools/probe.py closes that gap: a dependency-free, read-only script run from anywhere on the LAN that issues GET requests only and answers the open questions in §15 directly —

• Tautulli's coverage horizon, and therefore how much of the library gets pre_history
• the finished / partial / abandoned split across a sample of real plays, which is the assumption the rejection component rests on
• whether successive-play grouping is actually happening, so the session-merge logic (§4.9) can be checked against reality
• real library shapes: movie counts, season counts, total sizes, multi-version items
• path roots and their sizes, which decides whether grouping by physical vault is worth building

It also does the pms_identifier cross-check from §4.11 and reports whether it passes. Its output is safe to share — credentials are redacted from every message, including error messages, which is verified by test. --dump writes a full item inventory to JSON for offline analysis, and prints the twenty largest never-watched items as a first taste of what the real report will look like.

tools/reclaim_preview.py goes further: it joins Plex sizes to Tautulli plays exactly as MediaShelf will, and reports never-played bytes per library split into the confident and uncertain pools, TV broken out at season granularity, and the watcher-count distribution. This is what produced the calibration in §2.1 and §6.1.

tools/mockserver.py mocks both APIs so the probe can be exercised without touching a live server; it is a test fixture, not part of the application.

Both scripts were run against the live servers on 2026-09-07 from a LAN host, and their findings are folded into this document. Re-running them after any significant library change is the cheapest way to check whether the score's constants still fit.

---

14. Roadmap

14.1 v2 — Deletion

The reason v1 is report-only is that the scoring model needs to earn trust before it gets to remove anything. When it does, the design is two-stage quarantine, not delete:

1. Select items in the UI, review the manifest (every file path, every byte, total).
2. Quarantine — MediaShelf records an intent and moves the files to a trash directory on the same filesystem (a rename, so it's instant and atomic regardless of size), then triggers a Plex library refresh so the entries disappear from Plex.
3. Purge — after a configurable hold (default 14 days) the quarantined files are permanently removed, either automatically or by an explicit click. Until then, restore is a rename back.

This requires the one thing v1 deliberately avoids: filesystem access to the media shares. The v2 container mounts them read-write and gains a path-mapping layer translating Plex's paths (/mnt/vault2/... as Loki sees them) into container paths. Two guardrails: a configured allowlist of path prefixes MediaShelf may touch, and a refusal to act on any path outside them. Plus an immutable audit log of every quarantine and purge — what, when, how big, and what the score was that recommended it.

Deletion also makes authentication mandatory (§12) and makes a dry-run mode non-negotiable: every destructive operation gets a preview that lists exactly what would happen and requires explicit confirmation.

14.2 v3 — Emby and Jellyfin

The MediaProvider protocol is the whole preparation:

class MediaProvider(Protocol):
    def server_info(self) -> ServerInfo: ...
    def libraries(self) -> list[Library]: ...
    def items(self, library: Library) -> Iterator[Item]: ...     # yields normalized items
    def refresh_library(self, library: Library) -> None: ...     # v2 only

class HistoryProvider(Protocol):
    def server_info(self) -> ServerInfo: ...      # for the pms_identifier match, §4.11
    def accounts(self) -> list[Account]: ...
    def watch_events(self, since: int | None) -> Iterator[WatchEvent]: ...
    def coverage(self) -> Coverage: ...           # earliest/latest event, count
    @property
    def has_completion_data(self) -> bool: ...    # False for PlexHistoryProvider

Jellyfin and Emby share an API lineage, so one JellyfinProvider will likely cover both with a capability flag or two. The mapping is close: Items with Fields=Path,MediaSources,DateCreated, UserData.PlayCount and LastPlayedDate for watch data, /Users for accounts. Two differences matter. Their watch data is genuinely per-user and must be fetched per user and merged, rather than read from one global history endpoint. And there is no Tautulli equivalent, so has_completion_data is False and the rejection component drops out — the same degraded path the Plex fallback already exercises, which is a good reason for that path to exist and be tested from day one rather than bolted on later.

The provider table already carries kind and the schema already keys everything on (provider_id, provider_item_id), so multiple servers can coexist in one database — the door is open to a combined view across Plex and Jellyfin without a migration.

14.3 Later candidates

• Optional filesystem stat — if the shares happen to be mounted, verify Plex's sizes against reality and surface orphans Plex never indexed.
• Duplicate and version analysis — the same title held at 4K and 1080p is a common and invisible source of waste.
• Trend tracking — MediaShelf already snapshots on a schedule; a chart of library size over time and a projection of when the vault fills up is nearly free.
• Direct Tautulli database read — Tautulli's SQLite file could be read directly instead of paging its API. Measured: 87,640 rows page in about 40 seconds at 10,000 per request, which is fine for a nightly job, so this stays unnecessary. It would also couple MediaShelf to Tautulli's internal schema.
• Per-user reclaim views — the data to answer "what does only one person watch" is already stored; it just has no UI in v1.
• Notifications — a monthly "here's what's gone cold" summary.

---

15. Open questions

Resolved

• Is Tautulli running anywhere? Yes — 192.168.1.100:8181, on Isis. Promoted to the primary watch-history source; see §4.8–4.11.
• Plex history retention on Loki — no longer on the critical path, since Tautulli keeps its own independent record.
• Where does the image get built? Built off-box, delivered to Nox's Portainer (§11.2).

Still open

Resolved by measurement (2026-09-07)

Questions 1–4 were answered by running tools/probe.py and tools/reclaim_preview.py against the live servers. The findings are in §2.1 and changed the design in five places: the Tautulli size cross-check (§4.11), both Tautulli-dependent score weights (§6.1), cross-library duplicates (§6.5), protected libraries (§6.6), and two new seed views (§7.2).

• When was Tautulli installed? 2025-03-08, 547 days of coverage — but the library goes back to 2000, so 80% of it predates that. pre_history is the majority state.
• Is successive-play grouping on? Immaterial — MediaShelf requests grouping=0 and merges itself. Confirmed working against 87,640 real rows.
• Multiple libraries of the same kind? 24 libraries, 16 of them movie sections.
• Path structure? titan4 / titan3 / titan1, and grouping by root is worth building.

Still open

1. Registry or image upload? (§11.2) A registry is the better long-term answer but is infrastructure that doesn't exist yet. A preference, not a measurement.
2. Which other libraries should be protected? Family Videos is unambiguous. Sports (2 UFC items) is probably personal capture too. Anything else self-recorded needs marking before the first real report is trusted (§6.6).
3. What should the 18.8% partial-view band mean? Four times the abandoned share and genuinely ambiguous (§6.1). v1 stores and displays it without scoring it; a month of looking at real rows should settle whether it earns a component.
4. Is the 20.9 TB uncertain pool worth attacking at all? It is three times the size of the confident pool, and the honest answer is that nobody can tell from data whether those items were watched in 2019. Options are to leave it to human judgement, to import older history if any exists elsewhere, or to accept that content untouched for 18+ months is stale enough regardless of what happened before. Worth deciding deliberately rather than by default.