MediaShelf/docs/design.md

# MediaShelf — Software Design

**Status:** Draft v1 · **Author:** Claude (spec) for Jess · **Date:** 2026-09-07

---

## 1. Overview

MediaShelf is a self-hosted web application that reads a Plex Media Server library,
builds a local analytical snapshot of every item in it, and gives a human a fast way
to answer one question:

> *Which of these thousands of files are worth keeping, and which are burning disk
> for nothing?*

It does this by joining together facts Plex already knows but never shows side by
side — when a file was added, how large it is, where it lives on disk, which library
it belongs to, how many times it has been played, and when it was last played — and
exposing them as a sortable, filterable, chartable grid with a tunable "reclaim
score" that ranks deletion candidates.

**v1 is read-only.** MediaShelf will never delete, move, or modify a file or a Plex
record in this version. Its output is a report: an on-screen ranked list, saved rule
sets, and CSV export. Actual deletion is designed for in §14 but deliberately not
built yet, so the scanner and scoring model can be trusted before anything
destructive is wired up.

### 1.1 Goals

- Full-library ingest from Plex over its HTTP API, with no agent installed on the Plex host.
- A durable local snapshot so the UI is instant and Plex isn't hammered on every click.
- Movies at item level; TV rolled up to **season** level.
- Sort/filter/visualize on: date added, last watched, watch count, size on disk, file path, library.
- A weighted **reclaim score** with live sliders, so "what should I delete" is one sort click.
- **Saved views** — named, re-runnable rule sets like *unwatched, older than 2 years, over 10 GB*.
- CSV export of any view.
- A provider abstraction clean enough that Emby and Jellyfin drop in later without
  touching the scoring, storage, or UI layers.

### 1.2 Non-goals (v1)

- Deleting, moving, or trashing files. Not in v1. See §14.1.
- Writing anything back to Plex (no rating changes, no collection edits, no unmatch/refresh).
- Authentication or multi-user accounts — this is a LAN-only, single-admin tool. See §12.
- Music, photo, and "other video" libraries. Movie and Show sections only.
- Direct filesystem access. MediaShelf never mounts the media shares. All size and path
  data comes from the Plex API. See §4.4.
- Transcode/quality analysis, duplicate detection, subtitle management.

### 1.3 Decisions already made

| Question | Decision |
|---|---|
| Deletion in v1 | No — report only |
| Stack | Python + Flask, SQLite |
| Deployment | Portainer stack on **Nox** (192.168.1.77), published via Nginx Proxy Manager |
| Auth | None; LAN-only, single admin view, watch data aggregated across all Plex accounts |
| TV granularity | Season |
| Size/path source | Plex API only |
| v1 output | Sortable grid + filters + charts, tunable reclaim score, saved rule sets |
| Implementation | Design doc only for now; code to follow |

---

## 2. Context

The Plex server is **Loki** (192.168.1.10, Dell R710, Ubuntu 16.04) with roughly 90 TB
of RAID-6 storage across a Dell MD1000 vault. A second vault is powered down awaiting
drive upgrades, so reclaiming space on the live vault has real value — this tool exists
because buying disks is the more expensive alternative.

MediaShelf runs on **Nox**, not on Loki. Loki's job is serving media; MediaShelf is a
polite HTTP client that talks to it across the LAN and keeps its own database locally.
This also means MediaShelf survives Loki reboots, Plex upgrades, and the eventual
migration of services onto Enki.

---

## 3. Architecture

```
┌──────────────────────────── Nox (192.168.1.77) ────────────────────────────┐
│                                                                            │
│   ┌─────────────────── mediashelf container ───────────────────┐           │
│   │                                                            │           │
│   │   Flask app (gunicorn, 2 workers)                          │           │
│   │     ├── /            → UI (server-rendered shell + JS)     │           │
│   │     ├── /api/*       → JSON API                            │           │
│   │     └── /healthz                                           │           │
│   │                                                            │           │
│   │   APScheduler (in-process, 1 worker only)                  │           │
│   │     └── nightly scan job                                   │           │
│   │                                                            │           │
│   │   Ingestion pipeline                                       │           │
│   │     PlexProvider ──▶ normalize ──▶ upsert ──▶ rollup       │           │
│   │                                                            │           │
│   │   SQLite  /data/mediashelf.db  (WAL)                       │           │
│   └────────────────────────────┬───────────────────────────────┘           │
│                                │ volume: /data                             │
└────────────────────────────────┼───────────────────────────────────────────┘
                                 │ HTTP :32400 + X-Plex-Token
                                 ▼
                    Loki (192.168.1.10) — Plex Media Server
```

### 3.1 Component responsibilities

**`mediashelf.providers`** — the pluggable source layer. Defines the `MediaProvider`
protocol and ships one implementation, `PlexProvider`. Everything above this layer
speaks only in MediaShelf's own normalized dataclasses, never in Plex XML. This is the
single seam that Emby/Jellyfin support slots into (§14.2).

**`mediashelf.ingest`** — orchestrates a scan: pull sections, pull items, pull history,
normalize, upsert into SQLite inside a transaction, recompute season rollups, mark
vanished items, write a `scan` row.

**`mediashelf.scoring`** — pure functions. Takes a row of metrics plus a weight profile,
returns a 0–100 reclaim score and the component breakdown. No I/O, fully unit-testable.

**`mediashelf.rules`** — evaluates saved-view rule sets into SQL `WHERE` fragments via a
whitelisted field/operator grammar. Never string-interpolates user input into SQL.

**`mediashelf.api`** — Flask blueprints returning JSON.

**`mediashelf.web`** — one server-rendered HTML shell; the grid, charts, and sliders are
client-side JS against `/api`.

### 3.2 Why Flask + SQLite

The dataset is small in database terms. A library of ~5,000 movies and ~2,000 seasons
across ~60,000 episodes is a few tens of megabytes. SQLite in WAL mode handles this with
sub-10 ms queries on indexed columns, needs no second container, and backs up by copying
one file. Flask matches the pattern already running for the Mythica portal on Nox, so
deployment, logging, and reverse-proxy conventions carry over unchanged.

The one real constraint SQLite imposes is a single writer. Since the only writer is the
scan job, and scans run one at a time behind a lock, this is not a limitation. It does
mean **gunicorn must run the scheduler in exactly one worker** — see §11.3.

---

## 4. Plex integration

### 4.1 Connection

MediaShelf authenticates with a Plex server token (`X-Plex-Token`) supplied via
environment variable. It talks directly to `http://192.168.1.10:32400` on the LAN — no
plex.tv round trip, no account login, no dependency on Plex's cloud being up.

All requests carry standard client identification headers so the connection is
identifiable in Plex's own logs:

```
X-Plex-Token: <token>
X-Plex-Client-Identifier: mediashelf-<stable-uuid>
X-Plex-Product: MediaShelf
X-Plex-Version: <app version>
Accept: application/json
```

Plex returns JSON when `Accept: application/json` is set, which avoids XML parsing
entirely. Every response body is wrapped in a `MediaContainer` object.

### 4.2 Endpoints used

| Purpose | Endpoint |
|---|---|
| Server identity / version | `GET /identity` |
| List libraries | `GET /library/sections` |
| Library detail + on-disk locations | `GET /library/sections/{id}` |
| All movies in a section | `GET /library/sections/{id}/all?type=1` |
| All episodes in a show section | `GET /library/sections/{id}/all?type=4` |
| All seasons in a show section | `GET /library/sections/{id}/all?type=3` |
| Global watch history | `GET /status/sessions/history/all` |
| Accounts (for history attribution) | `GET /accounts` |

Pagination on every list endpoint uses `X-Plex-Container-Start` and
`X-Plex-Container-Size` headers (or the equivalent query params). MediaShelf pages at
**500 items** per request; the `MediaContainer.totalSize` field on the first response
gives the loop its bound.

### 4.3 The fields that matter

From each `Video` element on `/library/sections/{id}/all`:

| Plex attribute | Meaning | MediaShelf column |
|---|---|---|
| `ratingKey` | stable per-server item id | `provider_item_id` |
| `guid` | cross-server content identity | `guid` |
| `type` | `movie` / `episode` / `season` / `show` | `kind` |
| `title`, `year` | display | `title`, `year` |
| `addedAt` | epoch seconds, when Plex first saw the file | `added_at` |
| `updatedAt` | epoch seconds, last metadata change | `updated_at` |
| `duration` | milliseconds | `duration_ms` |
| `viewCount` | plays — **token-account scoped**, see §4.5 | (advisory only) |
| `lastViewedAt` | epoch seconds — token-account scoped | (advisory only) |
| `librarySectionID` | owning library | `library_id` |
| `grandparentRatingKey`, `grandparentTitle` | (episodes) the show | `show_id`, `show_title` |
| `parentRatingKey`, `parentIndex` | (episodes) the season and its number | `season_id`, `season_number` |

From the nested `Media[].Part[]` elements:

| Plex attribute | MediaShelf column |
|---|---|
| `Part.file` | `file_path` — absolute path on the Plex host |
| `Part.size` | `size_bytes` |
| `Part.container` | `container` |
| `Media.videoResolution`, `videoCodec`, `audioCodec`, `bitrate` | `resolution`, `video_codec`, `audio_codec`, `bitrate` |

An item can have several `Media` entries (multiple versions) each with several `Part`
entries (split files). MediaShelf stores **every part as its own row** in `media_part`
and treats an item's size as the sum of its parts. This matters: a movie with a 4K and a
1080p version is one item occupying two files, and a reclaim report that reported only
the first part would understate the win.

### 4.4 Why Plex-only, and what it costs

MediaShelf does not mount the media shares. That keeps the container trivially portable,
removes any possibility of it touching a file by accident, and means it works from
anywhere that can reach port 32400.

The accepted trade-offs:

- **Size is Plex's number, not the filesystem's.** Plex records `Part.size` at scan
  time. If a file is replaced outside Plex and Plex hasn't rescanned, the number is
  stale. In practice Plex's own scanner keeps this current; the design accepts drift.
- **Orphans are invisible.** A file on disk that Plex has never indexed will not appear
  in MediaShelf at all. Finding those needs a filesystem walk, which is explicitly out
  of scope (see §14.3 for the optional-mount future).
- **Paths are Loki-relative.** `file_path` is the path as the Plex server sees it. It is
  displayed and exported for a human's benefit, not resolved by MediaShelf.

### 4.5 Watch data: the important subtlety

`viewCount` and `lastViewedAt` on a library item are **scoped to the account that owns
the token**. If Jess's token is used, plays by home users, managed users, and shared
users are not counted. For a household library this understates watches badly, and a
reclaim tool that deletes "unwatched" content someone else has been watching is worse
than useless.

So MediaShelf treats `/status/sessions/history/all` as the source of truth for watch
data. That endpoint returns one row per playback event, server-wide, across all
accounts, each with:

- `historyKey`, `ratingKey`, `librarySectionID`
- `viewedAt` (epoch seconds)
- `accountID` (which Plex user)
- `type`, `title`, and for episodes the parent/grandparent identifiers

MediaShelf ingests this into a `watch_event` table and derives, per item:

- `watch_count` — number of distinct playback events
- `last_watched_at` — max `viewedAt`
- `first_watched_at` — min `viewedAt`
- `distinct_watcher_count` — count of distinct `accountID`

The single-admin UI shows the aggregate. The per-account data is stored anyway, because
it's free to keep and it's what the "who actually watches this" question will need later.

**Caveat to verify during the first spike:** Plex prunes session history according to the
server's *Media Playback → Settings → Empty trash / history retention* configuration, and
some servers cap it. If Loki's retention window turns out to be shorter than the useful
analysis window, MediaShelf's own `watch_event` table becomes the long-term record —
which it already is, since it accumulates across scans and never deletes rows. The first
full ingest establishes the baseline; every scan after that only needs events newer than
the last one seen.

The item-level `viewCount` is still stored as an advisory field, and the UI shows a small
warning badge when `viewCount > 0` but no history events exist for that item — that's the
signature of a watch that happened before history retention began.

### 4.6 Scan strategy

**Full scan** — walks every section, every item, every part, and the complete history.
Run on first launch and on demand. Expect a few minutes for a large library.

**Incremental scan** — the nightly default. Two cheap tricks:

1. Items: request the section sorted by `addedAt:desc` and stop paging once the page's
   oldest `addedAt` predates the last successful scan. This catches additions. To catch
   *removals* and metadata edits, a full item sweep still runs weekly (configurable) —
   an item silently vanishing from Plex is exactly the event that must not be missed.
2. History: request `/status/sessions/history/all` sorted by `viewedAt:desc` and stop at
   the last-seen `viewedAt` watermark.

**Concurrency** — one scan at a time, enforced by a row in a `scan_lock` table plus an
in-process lock. A scan that dies leaves a stale lock; locks older than
`SCAN_LOCK_TIMEOUT` (default 2 h) are broken automatically and the orphaned scan row is
marked `failed`.

**Politeness** — requests are issued serially with a small configurable delay
(`PLEX_REQUEST_DELAY_MS`, default 0) and a hard timeout. Loki is a 2010-vintage R710;
the ingest is designed to be slow and steady rather than parallel and aggressive.

### 4.7 Failure handling

| Failure | Behaviour |
|---|---|
| Plex unreachable | Scan marked `failed` with the error; UI keeps serving the last good snapshot with a staleness banner |
| Token rejected (401) | Scan fails fast, prominent UI banner, no retry loop |
| Section disappears | Its items are marked `missing` rather than deleted (§5.4) |
| Malformed / partial item | Item skipped, logged, counted in `scan.warning_count`; the scan still succeeds |
| Timeout mid-page | Up to 3 retries with exponential backoff, then the scan fails |

---

## 5. Data model

SQLite, WAL mode, foreign keys on. Migrations are plain numbered SQL files applied at
startup, tracked in a `schema_version` table — no ORM migration framework.

### 5.1 Tables

```sql
-- One row per configured media server. v1 has exactly one, but the column exists
-- so Emby/Jellyfin can be added without a migration.
CREATE TABLE provider (
    id              INTEGER PRIMARY KEY,
    kind            TEXT NOT NULL,          -- 'plex' | 'emby' | 'jellyfin'
    name            TEXT NOT NULL,
    base_url        TEXT NOT NULL,
    server_id       TEXT,                   -- Plex machineIdentifier
    version         TEXT,
    last_scan_id    INTEGER,
    created_at      INTEGER NOT NULL
);

CREATE TABLE library (
    id              INTEGER PRIMARY KEY,
    provider_id     INTEGER NOT NULL REFERENCES provider(id),
    provider_key    TEXT NOT NULL,          -- Plex section id
    title           TEXT NOT NULL,
    kind            TEXT NOT NULL,          -- 'movie' | 'show'
    locations       TEXT,                   -- JSON array of on-disk roots
    scanned_at      INTEGER,
    UNIQUE (provider_id, provider_key)
);

-- The unit of analysis. One row per movie, and one row per SEASON.
-- Shows themselves get a row too (kind='show') for grouping and display,
-- but carry no size of their own.
CREATE TABLE media_item (
    id                  INTEGER PRIMARY KEY,
    provider_id         INTEGER NOT NULL REFERENCES provider(id),
    library_id          INTEGER NOT NULL REFERENCES library(id),
    provider_item_id    TEXT NOT NULL,      -- Plex ratingKey
    kind                TEXT NOT NULL,      -- 'movie' | 'show' | 'season'
    guid                TEXT,
    title               TEXT NOT NULL,
    sort_title          TEXT,
    year                INTEGER,
    -- season/show linkage
    parent_id           INTEGER REFERENCES media_item(id),   -- season -> show
    season_number       INTEGER,
    -- provenance
    added_at            INTEGER,            -- epoch s; for a season, MIN over episodes
    updated_at          INTEGER,
    -- rollups (denormalized, recomputed each scan)
    episode_count       INTEGER NOT NULL DEFAULT 0,
    size_bytes          INTEGER NOT NULL DEFAULT 0,
    duration_ms         INTEGER NOT NULL DEFAULT 0,
    part_count          INTEGER NOT NULL DEFAULT 0,
    primary_path        TEXT,               -- representative file path for display
    resolution          TEXT,
    video_codec         TEXT,
    -- watch rollups (from watch_event)
    watch_count             INTEGER NOT NULL DEFAULT 0,
    last_watched_at         INTEGER,
    first_watched_at        INTEGER,
    distinct_watcher_count  INTEGER NOT NULL DEFAULT 0,
    provider_view_count     INTEGER NOT NULL DEFAULT 0,  -- advisory, token-scoped
    -- lifecycle
    status              TEXT NOT NULL DEFAULT 'present',  -- 'present' | 'missing'
    first_seen_scan_id  INTEGER,
    last_seen_scan_id   INTEGER,
    UNIQUE (provider_id, provider_item_id)
);

-- Episodes are ingested (they carry the size and the history), but they are NOT
-- the unit of analysis. They exist so seasons can be rolled up accurately and so
-- a season can be expanded in the UI.
CREATE TABLE episode (
    id                  INTEGER PRIMARY KEY,
    season_item_id      INTEGER NOT NULL REFERENCES media_item(id) ON DELETE CASCADE,
    provider_item_id    TEXT NOT NULL,
    episode_number      INTEGER,
    title               TEXT,
    added_at            INTEGER,
    duration_ms         INTEGER,
    size_bytes          INTEGER NOT NULL DEFAULT 0,
    watch_count         INTEGER NOT NULL DEFAULT 0,
    last_watched_at     INTEGER,
    status              TEXT NOT NULL DEFAULT 'present',
    UNIQUE (provider_item_id)
);

-- Every physical file. A movie with two versions has two rows.
CREATE TABLE media_part (
    id                  INTEGER PRIMARY KEY,
    media_item_id       INTEGER REFERENCES media_item(id) ON DELETE CASCADE,
    episode_id          INTEGER REFERENCES episode(id) ON DELETE CASCADE,
    provider_part_id    TEXT,
    file_path           TEXT NOT NULL,
    size_bytes          INTEGER NOT NULL DEFAULT 0,
    container           TEXT,
    resolution          TEXT,
    video_codec         TEXT,
    audio_codec         TEXT,
    bitrate             INTEGER,
    CHECK ((media_item_id IS NULL) != (episode_id IS NULL))
);

-- One row per playback event, server-wide, all accounts.
CREATE TABLE watch_event (
    id                  INTEGER PRIMARY KEY,
    provider_id         INTEGER NOT NULL REFERENCES provider(id),
    history_key         TEXT NOT NULL,
    provider_item_id    TEXT NOT NULL,      -- the leaf: movie or episode ratingKey
    account_id          TEXT,
    viewed_at           INTEGER NOT NULL,
    UNIQUE (provider_id, history_key)
);

CREATE TABLE account (
    id              INTEGER PRIMARY KEY,
    provider_id     INTEGER NOT NULL REFERENCES provider(id),
    account_id      TEXT NOT NULL,
    name            TEXT,
    UNIQUE (provider_id, account_id)
);

CREATE TABLE scan (
    id                  INTEGER PRIMARY KEY,
    provider_id         INTEGER NOT NULL REFERENCES provider(id),
    mode                TEXT NOT NULL,      -- 'full' | 'incremental'
    trigger             TEXT NOT NULL,      -- 'manual' | 'schedule' | 'startup'
    status              TEXT NOT NULL,      -- 'running' | 'succeeded' | 'failed'
    started_at          INTEGER NOT NULL,
    finished_at         INTEGER,
    items_seen          INTEGER DEFAULT 0,
    items_added         INTEGER DEFAULT 0,
    items_updated       INTEGER DEFAULT 0,
    items_missing       INTEGER DEFAULT 0,
    events_added        INTEGER DEFAULT 0,
    warning_count       INTEGER DEFAULT 0,
    error               TEXT
);

CREATE TABLE saved_view (
    id              INTEGER PRIMARY KEY,
    name            TEXT NOT NULL UNIQUE,
    description     TEXT,
    rules           TEXT NOT NULL,          -- JSON rule group, see §7
    sort            TEXT,                   -- JSON: [{field, dir}]
    columns         TEXT,                   -- JSON array of column keys
    weights         TEXT,                   -- JSON weight profile, see §6
    created_at      INTEGER NOT NULL,
    updated_at      INTEGER NOT NULL
);

CREATE TABLE setting (
    key             TEXT PRIMARY KEY,
    value           TEXT NOT NULL
);
```

### 5.2 Indexes

```sql
CREATE INDEX ix_item_library     ON media_item(library_id, kind, status);
CREATE INDEX ix_item_added       ON media_item(added_at);
CREATE INDEX ix_item_lastwatch   ON media_item(last_watched_at);
CREATE INDEX ix_item_size        ON media_item(size_bytes);
CREATE INDEX ix_item_watchcount  ON media_item(watch_count);
CREATE INDEX ix_item_parent      ON media_item(parent_id);
CREATE INDEX ix_episode_season   ON episode(season_item_id);
CREATE INDEX ix_part_item        ON media_part(media_item_id);
CREATE INDEX ix_part_episode     ON media_part(episode_id);
CREATE INDEX ix_event_item       ON watch_event(provider_item_id);
CREATE INDEX ix_event_viewed     ON watch_event(viewed_at);
```

A full-text index over `title` uses SQLite FTS5 (`media_item_fts`) kept in sync by
triggers, so the grid's search box is instant even at 100k rows.

### 5.3 Season rollup rules

After items and history are ingested, a rollup pass recomputes each season row:

| Season field | Rule |
|---|---|
| `size_bytes` | `SUM(episode.size_bytes)` |
| `episode_count` | `COUNT(episode)` |
| `duration_ms` | `SUM(episode.duration_ms)` |
| `added_at` | `MIN(episode.added_at)` — when the season first landed |
| `watch_count` | `SUM(episode.watch_count)` |
| `last_watched_at` | `MAX(episode.last_watched_at)` |
| `primary_path` | the common directory prefix of its episodes' paths |

A show row rolls up the same way from its seasons, for display and for the "delete the
whole show" case.

The `watch_count` choice deserves a note: summing episode plays means a 24-episode season
watched once has `watch_count = 24`, while a movie watched once has `watch_count = 1`.
Comparing those raw numbers across kinds is meaningless. So the scoring model
(§6) uses a **normalized completion figure** for TV — `watch_count / episode_count` —
rather than the raw sum, and the grid displays both (`24 plays · 1.0× season`).

### 5.4 Missing items

Nothing is ever hard-deleted by a scan. When a full sweep completes and an item that was
present is no longer returned by Plex, its `status` flips to `missing` and
`last_seen_scan_id` freezes. Missing items are hidden from the default grid but remain
queryable — they're the record of what left the library and when, which is exactly what
you want the first time something disappears unexpectedly.

---

## 6. The reclaim score

The score answers: *how much do I gain by deleting this, weighed against how likely I am
to want it?* It is a weighted sum of normalized components, each in `[0, 1]`, scaled to
0–100. Higher means "better candidate for deletion".

### 6.1 Components

| Component | Definition | Rationale |
|---|---|---|
| **`size`** | `log10(1 + size_bytes) / log10(1 + max_size_in_library)` | Log scale — the difference between 2 GB and 4 GB matters more than between 60 GB and 62 GB |
| **`staleness`** | `min(days_since_last_watch / STALE_HORIZON, 1)`, where a never-watched item scores `1.0` | How long since anyone cared |
| **`unpopularity`** | `1 - min(normalized_watches / POPULAR_AT, 1)` where `normalized_watches` = `watch_count` for movies, `watch_count / episode_count` for seasons | Rewatched things are keepers |
| **`age`** | `min(days_since_added / AGE_HORIZON, 1)` | Something added last week deserves a chance |
| **`solitude`** | `1 - min(distinct_watcher_count / 3, 1)` | Content only one household member ever touched is safer to cut |

`STALE_HORIZON` defaults to 730 days, `AGE_HORIZON` to 1095 days, `POPULAR_AT` to 3.

### 6.2 The formula

```
score = 100 × ( w_size        · size
              + w_staleness   · staleness
              + w_unpopularity· unpopularity
              + w_age         · age
              + w_solitude    · solitude )
        / (w_size + w_staleness + w_unpopularity + w_age + w_solitude)
```

Default weight profile:

```json
{ "size": 0.30, "staleness": 0.30, "unpopularity": 0.25, "age": 0.10, "solitude": 0.05 }
```

### 6.3 The grace rule

Two hard overrides, applied after scoring, because a purely numeric model will
eventually recommend something obviously wrong:

- **New-arrival grace** — an item with `added_at` inside `GRACE_DAYS` (default 30) is
  clamped to score 0 and flagged `grace: new`. You haven't had a chance to watch it yet.
- **Recent-watch grace** — an item watched inside `RECENT_DAYS` (default 90) is clamped
  to a maximum of 25 and flagged `grace: recent`.

Both are toggleable per view, and the UI shows why an item was clamped rather than
silently hiding it.

### 6.4 Where it's computed

In SQL, as a generated expression in the query, not as a stored column. Weights change
every time a slider moves; storing the score would mean recomputing 100k rows per drag.
The normalization constants (`max_size_in_library`, horizons) are computed once per
request in a CTE. At this row count the whole thing runs in single-digit milliseconds.

The same function is implemented in Python in `mediashelf.scoring` for unit tests and CSV
export, with a property test asserting the SQL and Python implementations agree to within
0.01 on a generated corpus. Two implementations of the same formula is a real risk, and
that test is the mitigation.

---

## 7. Saved views and rules

A saved view bundles four things: a rule set, a sort, a column selection, and a weight
profile. Naming one — *"Big unwatched movies"* — makes it a one-click, re-runnable
report that re-evaluates against fresh data every time it's opened.

### 7.1 Rule grammar

Rules are a nested JSON group, evaluated into parameterized SQL:

```json
{
  "op": "and",
  "rules": [
    { "field": "kind",            "op": "in",     "value": ["movie"] },
    { "field": "size_bytes",      "op": "gte",    "value": 10737418240 },
    { "field": "watch_count",     "op": "eq",     "value": 0 },
    { "field": "added_at",        "op": "older_than_days", "value": 730 },
    { "op": "or", "rules": [
        { "field": "library_id",  "op": "in",     "value": [1, 4] },
        { "field": "resolution",  "op": "eq",     "value": "4k" }
    ]}
  ]
}
```

Fields are drawn from a whitelist mapping field name → column and type. Operators are a
fixed set: `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `in`, `not_in`, `contains`,
`starts_with`, `is_null`, `is_not_null`, `older_than_days`, `newer_than_days`,
`never` (for watch fields). Anything outside the whitelist is a 400. Values are always
bound parameters. There is no SQL string interpolation anywhere in this path.

### 7.2 Seed views shipped by default

| Name | Rule |
|---|---|
| Never watched, large | `watch_count = 0 AND size_bytes >= 8 GB` |
| Cold storage | `last_watched_at older than 3 years` |
| One-and-done movies | `kind = movie AND watch_count = 1 AND last_watched_at older than 2 years` |
| Abandoned seasons | `kind = season AND watch_count/episode_count < 0.3 AND added_at older than 1 year` |
| Biggest 100 | sorted by `size_bytes desc`, no filter |
| Recently added | `added_at newer than 30 days` — the sanity check, not a delete list |

---

## 8. HTTP API

All JSON, all under `/api/v1`. No auth in v1 (§12).

### 8.1 Library data

```
GET /api/v1/items
```

Query parameters:

| Param | Meaning |
|---|---|
| `library_id` | repeatable filter |
| `kind` | `movie` / `season` / `show` |
| `q` | FTS search over title |
| `rules` | URL-encoded JSON rule group (§7.1) |
| `view_id` | apply a saved view instead of ad-hoc rules |
| `sort` | e.g. `reclaim_score:desc`, `size_bytes:desc` |
| `weights` | URL-encoded JSON weight profile |
| `include_missing` | default `false` |
| `page`, `page_size` | page_size max 500 |

Response:

```json
{
  "total": 4821,
  "page": 1,
  "page_size": 100,
  "aggregate": { "total_size_bytes": 41234567890123, "item_count": 4821 },
  "items": [
    {
      "id": 1183,
      "kind": "movie",
      "title": "Example Film",
      "year": 2011,
      "library": { "id": 1, "title": "Movies" },
      "size_bytes": 32212254720,
      "added_at": 1490000000,
      "last_watched_at": null,
      "watch_count": 0,
      "distinct_watcher_count": 0,
      "episode_count": null,
      "primary_path": "/mnt/vault2/movies/Example Film (2011)/Example Film (2011).mkv",
      "part_count": 1,
      "resolution": "4k",
      "reclaim_score": 91.4,
      "reclaim_components": {
        "size": 0.97, "staleness": 1.0, "unpopularity": 1.0, "age": 1.0, "solitude": 1.0
      },
      "grace": null,
      "flags": ["history_gap"]
    }
  ]
}
```

Other endpoints:

```
GET  /api/v1/items/{id}            full detail incl. every part and, for seasons, episodes
GET  /api/v1/libraries             list with item counts and total size
GET  /api/v1/accounts              Plex accounts seen in history
GET  /api/v1/stats/overview        headline numbers for the dashboard
GET  /api/v1/stats/size-by-library
GET  /api/v1/stats/added-over-time?bucket=month
GET  /api/v1/stats/watch-distribution
GET  /api/v1/stats/size-vs-lastwatched   scatter data, downsampled server-side
```

### 8.2 Saved views

```
GET    /api/v1/views
POST   /api/v1/views
GET    /api/v1/views/{id}
PUT    /api/v1/views/{id}
DELETE /api/v1/views/{id}
```

### 8.3 Scans

```
GET  /api/v1/scans                 history, newest first
GET  /api/v1/scans/current         null or the running scan with progress
POST /api/v1/scans                 { "mode": "full" | "incremental" } → 202 or 409 if one is running
```

Scan progress is polled at `/api/v1/scans/current` every 2 s while one is running. No
websockets — polling one row is cheaper than the complexity.

### 8.4 Export

```
GET /api/v1/export.csv?<same params as /items>
```

Streams the full result set — not just the current page — as CSV with the currently
selected columns plus the reclaim score and its components. This is the v1 deliverable:
the artifact you take away and act on.

### 8.5 Health

```
GET /healthz    → { "status": "ok", "db": "ok", "plex": "ok"|"unreachable",
                    "last_scan_at": 1757000000, "stale": false }
```

---

## 9. Web UI

Server-rendered Flask shell, vanilla JS plus two small libraries: a virtualized table
(TanStack Table core or Tabulator) and a charting library. No build step, no npm in the
container — assets are vendored. This keeps the container small and the deploy identical
to the Mythica pattern.

### 9.1 Screens

**Dashboard** — the landing page. Headline tiles (total items, total size, size of
never-watched content, size of content untouched for 2+ years, last scan time), plus:

- *Size by library* — horizontal bar.
- *Added over time* — monthly bars, stacked by library. Shows acquisition habits.
- *Watched vs unwatched by size* — the "how much of this shelf has anyone ever touched"
  chart, which is usually the one that motivates the whole exercise.
- *Size vs. last watched* — scatter, log-size on Y, days-since-watch on X. Points in the
  upper right are the reclaim targets, and clicking a region filters the grid to it.

**Library grid** — the main workspace. A virtualized table, every column sortable, with:

- A filter rail: library checkboxes, kind, size range, added range, last-watched range,
  watch-count range, resolution, and a free-text title search.
- A weight panel with five sliders and a live-updating `reclaim_score` column.
- Multi-select via checkboxes, shift-click ranges, and select-all-matching-filter.
  In v1 selection drives the aggregate footer ("**312 items selected · 4.1 TB**") and
  the CSV export. The buttons that will eventually delete are present but disabled, with
  a tooltip explaining v1 is report-only — so the workflow is proven before it's armed.
- Row expansion: a season expands to its episodes; any row expands to its file parts
  with full paths.

**Saved views** — list, create, edit, duplicate, run. Creating one captures the current
filter, sort, columns, and weights, so the flow is *explore, then name what you found*.

**Item detail** — everything known: all parts with paths and sizes, per-account watch
history timeline, score breakdown showing each component's contribution.

**Scans** — history table, per-scan counts and duration, a "Scan now" button, live
progress.

**Settings** — connection status, scan schedule, score defaults, horizons, grace periods.

### 9.2 Interaction principles

- Every number that represents bytes is displayed in both human units and, on hover, the
  exact byte count. Reclaim decisions turn on real numbers.
- The reclaim score column always shows its components on hover. A score you can't
  interrogate is a score you won't trust, and won't act on.
- Filters live in the URL query string, so any view is linkable and back/forward works.
- Nothing in v1 is destructive, and the UI says so plainly in the header rather than
  leaving the user to wonder.

---

## 10. Configuration

All via environment variables, so the Portainer stack is the single source of truth.

| Variable | Default | Notes |
|---|---|---|
| `MEDIASHELF_SECRET_KEY` | — | required, Flask session signing |
| `PLEX_BASE_URL` | — | required, e.g. `http://192.168.1.10:32400` |
| `PLEX_TOKEN` | — | required, secret |
| `PLEX_VERIFY_SSL` | `true` | |
| `PLEX_TIMEOUT_S` | `30` | |
| `PLEX_PAGE_SIZE` | `500` | |
| `PLEX_REQUEST_DELAY_MS` | `0` | throttle for Loki's sake |
| `DATABASE_PATH` | `/data/mediashelf.db` | on the mounted volume |
| `SCAN_SCHEDULE_CRON` | `0 4 * * *` | nightly incremental at 04:00 |
| `SCAN_FULL_SWEEP_CRON` | `0 3 * * 0` | weekly full sweep, Sunday 03:00 |
| `SCAN_ON_STARTUP` | `false` | |
| `SCAN_LOCK_TIMEOUT_S` | `7200` | |
| `SCORE_STALE_HORIZON_DAYS` | `730` | |
| `SCORE_AGE_HORIZON_DAYS` | `1095` | |
| `SCORE_POPULAR_AT` | `3` | |
| `SCORE_GRACE_DAYS` | `30` | |
| `SCORE_RECENT_DAYS` | `90` | |
| `TZ` | `America/Regina` | so cron times mean what they look like |
| `LOG_LEVEL` | `INFO` | |

`PLEX_TOKEN` is a secret and must not be committed. The repo ships `.env.example` with
placeholders, and `.env` is gitignored. (Worth noting given the SOAP-password-in-history
problem on the Mythica repo — this one starts clean and stays clean.)

---

## 11. Deployment

### 11.1 Container

Single image, `python:3.12-slim` base, multi-stage so build deps don't ship. Runs as a
non-root user. One volume: `/data`. One port: 8080.

### 11.2 Portainer stack on Nox

```yaml
services:
  mediashelf:
    image: mediashelf:latest
    container_name: mediashelf
    restart: unless-stopped
    ports:
      - "8085:8080"
    volumes:
      - mediashelf_data:/data
    environment:
      - MEDIASHELF_SECRET_KEY=${MEDIASHELF_SECRET_KEY}
      - PLEX_BASE_URL=${PLEX_BASE_URL}
      - PLEX_TOKEN=${PLEX_TOKEN}
      - DATABASE_PATH=/data/mediashelf.db
      - SCAN_SCHEDULE_CRON=${SCAN_SCHEDULE_CRON}
      - TZ=America/Regina
      - LOG_LEVEL=INFO
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request;urllib.request.urlopen('http://localhost:8080/healthz')"]
      interval: 60s
      timeout: 10s
      retries: 3

volumes:
  mediashelf_data:
```

**Every variable used in the stack must be listed under `environment:` here, not only set
in Portainer's UI.** A variable set in Portainer that isn't declared in the compose file
silently does nothing — the same trap that bit `PLAYERMAP_SHOW_GMS` on the Mythica stack.

Published through Nginx Proxy Manager at `mediashelf.hallsworth.ca`, upstream
`192.168.1.77:8085`. Note the host-port → container-port mapping is 8085 → 8080; NPM's
upstream must point at the **host** port.

### 11.3 The gunicorn / scheduler constraint

APScheduler runs in-process. With `--workers 2`, both workers would start a scheduler and
two scans would race for the SQLite write lock. Mitigation: the scheduler starts only in
the worker that successfully acquires an exclusive advisory lock on a lockfile in `/data`
at boot. The `scan_lock` table is the second line of defence. Both are needed; either
alone has a race window.

### 11.4 Backup

The database is one file. A weekly `sqlite3 .backup` to the volume, keeping four, is
enough — everything in it is reconstructible from Plex except accumulated `watch_event`
rows beyond Plex's own history retention (§4.5), which is precisely the part worth
backing up.

---

## 12. Security

MediaShelf is LAN-only with no authentication, by decision. That is a reasonable posture
for a read-only tool behind a firewall, and it is a bad posture the moment v2 adds
deletion. This section records what that means now and what changes later.

**v1 posture:**

- No auth, no sessions, no user records.
- Bind to the LAN; if published through NPM, restrict by source IP at the proxy or keep
  the hostname internal-only. It should not be reachable from the internet.
- `PLEX_TOKEN` lives in the environment only, never in the database, never in logs, never
  in an API response. The `/api/v1/settings` endpoint redacts it.
- No user input reaches SQL except through the whitelisted rule grammar (§7.1) with bound
  parameters.
- All rendered strings are escaped; file paths from Plex are treated as untrusted display
  data.
- CSRF is not a concern in v1 because nothing mutating exists beyond scan triggering and
  saved views. Both use `POST` with a `SameSite=Strict` token anyway, because retrofitting
  CSRF protection after deletion exists is exactly the kind of thing that gets skipped.

**Required before v2 ships deletion:** authentication is not optional. A single admin
password with a signed session, rate-limited login, and an audit log is the minimum. An
unauthenticated endpoint that deletes files on a 90 TB array is an accident waiting for
a misconfigured proxy rule.

---

## 13. Testing

| Layer | Approach |
|---|---|
| Provider | Recorded Plex JSON fixtures (`responses`/`vcr.py`) covering a movie section, a show section with multi-season shows, multi-version movies, split parts, and a history page. No live server needed in CI. |
| Normalization | Table-driven tests: odd cases like an item with no `Part`, a season with a missing episode, unicode titles, zero-byte parts. |
| Ingest | In-memory SQLite; assert idempotency (running the same scan twice changes nothing), assert `missing` transitions, assert history watermarking doesn't skip or duplicate events. |
| Scoring | Unit tests on known inputs; **property test asserting the SQL and Python implementations agree** (§6.4); tests that grace rules clamp correctly. |
| Rules | Every operator round-trips to correct SQL; injection attempts in field names and values are rejected with 400. |
| API | Flask test client, schema assertions on every response shape. |
| Smoke | A `make smoke` target that runs a real scan against a live Plex and prints the top 20 reclaim candidates — the human sanity check that no unit test replaces. |

The single most important test is idempotency: a scanner that double-counts sizes or
duplicates history events produces a report that looks plausible and is wrong, which is
worse than one that crashes.

---

## 14. Roadmap

### 14.1 v2 — Deletion

The reason v1 is report-only is that the scoring model needs to earn trust before it
gets to remove anything. When it does, the design is **two-stage quarantine, not
delete**:

1. **Select** items in the UI, review the manifest (every file path, every byte, total).
2. **Quarantine** — MediaShelf records an intent and moves the files to a trash directory
   on the same filesystem (a rename, so it's instant and atomic regardless of size),
   then triggers a Plex library refresh so the entries disappear from Plex.
3. **Purge** — after a configurable hold (default 14 days) the quarantined files are
   permanently removed, either automatically or by an explicit click. Until then,
   restore is a rename back.

This requires the one thing v1 deliberately avoids: filesystem access to the media
shares. The v2 container mounts them read-write and gains a path-mapping layer
translating Plex's paths (`/mnt/vault2/...` as Loki sees them) into container paths. Two
guardrails: a configured allowlist of path prefixes MediaShelf may touch, and a refusal
to act on any path outside them. Plus an immutable audit log of every quarantine and
purge — what, when, how big, and what the score was that recommended it.

Deletion also makes authentication mandatory (§12) and makes a dry-run mode
non-negotiable: every destructive operation gets a preview that lists exactly what would
happen and requires explicit confirmation.

### 14.2 v3 — Emby and Jellyfin

The `MediaProvider` protocol is the whole preparation:

```python
class MediaProvider(Protocol):
    def server_info(self) -> ServerInfo: ...
    def libraries(self) -> list[Library]: ...
    def items(self, library: Library) -> Iterator[Item]: ...     # yields normalized items
    def watch_events(self, since: int | None) -> Iterator[WatchEvent]: ...
    def accounts(self) -> list[Account]: ...
    def refresh_library(self, library: Library) -> None: ...     # v2 only
```

Jellyfin and Emby share an API lineage, so one `JellyfinProvider` will likely cover both
with a capability flag or two. The mapping is close: `Items` with
`Fields=Path,MediaSources,DateCreated`, `UserData.PlayCount` and `LastPlayedDate` for
watch data, `/Users` for accounts. The notable difference is that Jellyfin's watch data
is genuinely per-user and must be fetched per user and merged, rather than read from one
global history endpoint.

The `provider` table already carries `kind` and the schema already keys everything on
`(provider_id, provider_item_id)`, so multiple servers can coexist in one database — the
door is open to a combined view across Plex and Jellyfin without a migration.

### 14.3 Later candidates

- **Optional filesystem stat** — if the shares happen to be mounted, verify Plex's sizes
  against reality and surface orphans Plex never indexed.
- **Duplicate and version analysis** — the same title held at 4K and 1080p is a common
  and invisible source of waste.
- **Trend tracking** — MediaShelf already snapshots on a schedule; a chart of library
  size over time and a projection of when the vault fills up is nearly free.
- **Tautulli import** — if Tautulli is running, its history database predates Plex's own
  retention window and would deepen the watch record considerably.
- **Notifications** — a monthly "here's what's gone cold" summary.

---

## 15. Open questions

1. **Plex history retention on Loki.** If it's short, the first ingest captures less than
   hoped and `watch_event` becomes valuable immediately. Worth checking before the first
   scan, and worth knowing whether Tautulli has been running (§14.3).
2. **Multiple Plex libraries of the same kind** — are there several movie sections (e.g.
   Movies, Kids, Documentaries)? The design handles it, but the seed views and default
   charts get more useful if the real shape is known.
3. **Path structure on the vaults** — knowing the actual roots would let the grid group by
   physical vault, which matters when the goal is freeing a *specific* array rather than
   space in general.
4. **Where does the image get built?** Nox running Portainer suggests either a build from
   a git URL in the stack, or pushing to a registry. Worth settling before the first
   deploy.
5. **Is Tautulli running anywhere?** It changes the watch-history story materially.