yrtria/MediaShelf
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Promote Tautulli to the primary watch-history source

Browse source 
Tautulli is running at 192.168.1.100:8181, so watch data now comes from
get_history rather than Plex's session history. This matters beyond
convenience: Plex records that a play started, Tautulli records how far it
got. A film three people abandoned after five minutes and a film nobody ever
opened are opposite signals for a deletion decision, and Plex reports them
identically.

- Adds sections 4.8-4.11: Tautulli ingest, completion semantics, fallback,
  coverage horizon, and the pms_identifier cross-check
- Splits MediaProvider into MediaProvider + HistoryProvider so library data
  and watch data can come from different servers
- watch_event gains percent_complete, disposition, source, and session
  merging; media_item gains partial/abandoned counts and pre_history
- New 'rejection' score component, with weights renormalizing when a
  component is unavailable rather than silently scoring everything lower
- Deployment: image built off-box and pushed to Nox's Portainer, versioned
  tags rather than :latest
- Closes three open questions, opens two smaller ones

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GVbG48GAXfCZatcmX123Ra
This commit is contained in:
Jess Hallsworth 2026-09-07 04:51:41 +00:00
parent fdcddc8149
commit 74dfc4dab8
No known key found for this signature in database
3 changed files with 380 additions and 109 deletions
Download patch file Download diff file Expand all files Collapse all files

453
docs/design.md
View file

@ -1,6 +1,10 @@
# MediaShelf — Software Design
**Status:** Draft v1 · **Author:** Claude (spec) for Jess · **Date:** 2026-09-07
**Status:** Draft v1.1 · **Author:** Claude (spec) for Jess · **Date:** 2026-09-07
*Revision 1.1 — Tautulli (192.168.1.100:8181) confirmed present and promoted to the
primary watch-history source; Plex's own history demoted to fallback. Deployment path
settled: image built in CI/locally and pushed to Nox's Portainer.*
---
@ -28,6 +32,9 @@ destructive is wired up.
### 1.1 Goals
- Full-library ingest from Plex over its HTTP API, with no agent installed on the Plex host.
- Watch history from **Tautulli**, which has been logging every play since it was
installed and knows *how much* of each item was actually watched — not just that a
play started.
- A durable local snapshot so the UI is instant and Plex isn't hammered on every click.
- Movies at item level; TV rolled up to **season** level.
- Sort/filter/visualize on: date added, last watched, watch count, size on disk, file path, library.
@ -57,7 +64,9 @@ destructive is wired up.
| Auth | None; LAN-only, single admin view, watch data aggregated across all Plex accounts |
| TV granularity | Season |
| Size/path source | Plex API only |
| Watch-history source | **Tautulli** primary, Plex session history as fallback |
| v1 output | Sortable grid + filters + charts, tunable reclaim score, saved rule sets |
| Image delivery | Built outside Nox, pushed to Nox's Portainer |
| Implementation | Design doc only for now; code to follow |
---
@ -97,18 +106,26 @@ migration of services onto Enki.
│ │ SQLite /data/mediashelf.db (WAL) │ │
│ └────────────────────────────┬───────────────────────────────┘ │
│ │ volume: /data │
└────────────────────────────────┼───────────────────────────────────────────┘
│ HTTP :32400 + X-Plex-Token
▼
Loki (192.168.1.10) — Plex Media Server
└──────────┬─────────────────────┴───────────────────────────────────────────┘
│ │
│ HTTP :32400 + X-Plex-Token │ HTTP :8181 + apikey
▼ ▼
Loki (192.168.1.10) Isis (192.168.1.100:8181)
Plex Media Server Tautulli
· libraries, items, parts · every play since install
· size, path, added_at · per-user, per-item, % complete
· session history (fallback)
```
### 3.1 Component responsibilities
**`mediashelf.providers`** — the pluggable source layer. Defines the `MediaProvider`
protocol and ships one implementation, `PlexProvider`. Everything above this layer
speaks only in MediaShelf's own normalized dataclasses, never in Plex XML. This is the
single seam that Emby/Jellyfin support slots into (§14.2).
**`mediashelf.providers`** — the pluggable source layer. Defines two protocols:
`MediaProvider` (libraries, items, parts) implemented by `PlexProvider`, and
`HistoryProvider` (watch events) implemented by both `TautulliHistoryProvider` and
`PlexHistoryProvider`. Everything above this layer speaks only in MediaShelf's own
normalized dataclasses. Splitting the two protocols is what lets watch history come
from a different box than the library itself — which, given Tautulli lives on Isis and
Plex on Loki, is not hypothetical. It is also the seam Emby/Jellyfin slot into (§14.2).
**`mediashelf.ingest`** — orchestrates a scan: pull sections, pull items, pull history,
normalize, upsert into SQLite inside a transaction, recompute season rollups, mark
@ -139,9 +156,13 @@ mean **gunicorn must run the scheduler in exactly one worker** — see §11.3.
---
## 4. Plex integration
## 4. Data sources
### 4.1 Connection
MediaShelf reads from two servers. **Plex on Loki** is the authority on what exists —
libraries, items, files, sizes, paths, added dates. **Tautulli on Isis** is the authority
on what has been watched. Sections 4.1–4.7 cover Plex; 4.8–4.11 cover Tautulli.
### 4.1 Plex connection
MediaShelf authenticates with a Plex server token (`X-Plex-Token`) supplied via
environment variable. It talks directly to `http://192.168.1.10:32400` on the LAN — no
@ -230,46 +251,25 @@ The accepted trade-offs:
- **Paths are Loki-relative.** `file_path` is the path as the Plex server sees it. It is
displayed and exported for a human's benefit, not resolved by MediaShelf.
### 4.5 Watch data: the important subtlety
### 4.5 Why Plex is not the watch-data source
`viewCount` and `lastViewedAt` on a library item are **scoped to the account that owns
the token**. If Jess's token is used, plays by home users, managed users, and shared
users are not counted. For a household library this understates watches badly, and a
reclaim tool that deletes "unwatched" content someone else has been watching is worse
than useless.
Two problems make Plex's own watch data unfit to build a reclaim tool on.
So MediaShelf treats `/status/sessions/history/all` as the source of truth for watch
data. That endpoint returns one row per playback event, server-wide, across all
accounts, each with:
**It is scoped to one account.** `viewCount` and `lastViewedAt` on a library item reflect
only the account that owns the token. Plays by home users, managed users, and shared users
are invisible. For a household library this understates watches badly, and a tool that
recommends deleting "unwatched" content someone else watches weekly is worse than useless.
- `historyKey`, `ratingKey`, `librarySectionID`
- `viewedAt` (epoch seconds)
- `accountID` (which Plex user)
- `type`, `title`, and for episodes the parent/grandparent identifiers
**It only knows that a play started.** `/status/sessions/history/all` records a playback
event. It does not record whether the viewer watched the whole thing or bailed after four
minutes. Those two events mean opposite things for a deletion decision, and Plex reports
them identically.
MediaShelf ingests this into a `watch_event` table and derives, per item:
`/status/sessions/history/all` solves the first problem — it is server-wide across all
accounts — so it remains the **fallback** source, and §4.11 covers when it is used. But
Tautulli solves both, so Tautulli is the primary.
- `watch_count` — number of distinct playback events
- `last_watched_at` — max `viewedAt`
- `first_watched_at` — min `viewedAt`
- `distinct_watcher_count` — count of distinct `accountID`
The single-admin UI shows the aggregate. The per-account data is stored anyway, because
it's free to keep and it's what the "who actually watches this" question will need later.
**Caveat to verify during the first spike:** Plex prunes session history according to the
server's *Media Playback → Settings → Empty trash / history retention* configuration, and
some servers cap it. If Loki's retention window turns out to be shorter than the useful
analysis window, MediaShelf's own `watch_event` table becomes the long-term record —
which it already is, since it accumulates across scans and never deletes rows. The first
full ingest establishes the baseline; every scan after that only needs events newer than
the last one seen.
The item-level `viewCount` is still stored as an advisory field, and the UI shows a small
warning badge when `viewCount > 0` but no history events exist for that item — that's the
signature of a watch that happened before history retention began.
### 4.6 Scan strategy
### 4.6 Plex scan strategy
**Full scan** — walks every section, every item, every part, and the complete history.
Run on first launch and on demand. Expect a few minutes for a large library.
@ -302,6 +302,133 @@ the ingest is designed to be slow and steady rather than parallel and aggressive
| Malformed / partial item | Item skipped, logged, counted in `scan.warning_count`; the scan still succeeds |
| Timeout mid-page | Up to 3 retries with exponential backoff, then the scan fails |
### 4.8 Tautulli — the watch-history source
Tautulli runs at `http://192.168.1.100:8181` (on Isis) and has been logging every
playback on Loki since the day it was installed. Its database is independent of Plex's:
clearing Plex's history, or Plex pruning its own, does not touch it. That makes it a
deeper and more durable record than anything Plex exposes.
Its API is a single endpoint with a `cmd` parameter:
```
GET http://192.168.1.100:8181/api/v2?apikey=<key>&cmd=<command>&<params>
```
Every response is `{"response": {"result": "success"|"error", "message": ..., "data": ...}}`.
A transport-level 200 with `result: "error"` in the body is the normal failure mode, so
**the result field must be checked on every call** — treating HTTP 200 as success is the
mistake this API invites.
| Purpose | Command |
|---|---|
| Reachability / version | `cmd=get_server_info`, `cmd=arnold` (returns a quote; a cheap liveness ping) |
| Which Plex server it watches | `cmd=get_server_info` → `pms_identifier` |
| Users | `cmd=get_users` |
| Libraries | `cmd=get_libraries` |
| **Watch history** | `cmd=get_history` |
| Per-item size / play count (cross-check) | `cmd=get_library_media_info` |
### 4.9 Ingesting history
```
cmd=get_history
&grouping=0 # do NOT collapse successive plays — we want raw events
&order_column=date
&order_dir=desc
&start=<offset>
&length=1000
&after=<YYYY-MM-DD> # incremental watermark
```
`grouping=0` matters. Tautulli's *Group Successive Play History* setting merges a paused-
and-resumed play into one row for display purposes. For counting distinct viewings that
is what we want — but the grouping should be MediaShelf's decision, applied consistently,
not a mirror of however that setting happens to be configured on Isis. So MediaShelf pulls
ungrouped events and does its own session collapsing: **events for the same item by the
same user whose start times fall within `SESSION_MERGE_WINDOW` (default 6 hours) count as
one viewing.**
The fields MediaShelf keeps from each history row:
| Tautulli field | Use |
|---|---|
| `row_id` | idempotency key for upsert |
| `reference_id` | Tautulli's own grouping id, kept for cross-checking |
| `date` / `started` / `stopped` | epoch seconds |
| `rating_key` | the leaf item — movie or episode |
| `parent_rating_key`, `grandparent_rating_key` | season and show, for rollup |
| `user_id`, `user`, `friendly_name` | who watched |
| `media_type` | `movie` / `episode` |
| `percent_complete` | **how much was actually watched** |
| `watched_status` | Tautulli's own verdict against its configured threshold |
| `play_duration`, `paused_counter` | real time spent, minus pauses |
| `platform`, `player` | stored, unused in v1 |
### 4.10 What "watched" means
This is the part that most changes the quality of the output, so MediaShelf defines it
itself rather than inheriting Tautulli's threshold setting:
| Derived value | Rule |
|---|---|
| **completed view** | `percent_complete >= COMPLETION_THRESHOLD` (default 85) |
| **partial view** | `percent_complete` between `ABANDON_CEILING` (default 15) and the completion threshold |
| **abandoned view** | `percent_complete < ABANDON_CEILING` — started and bailed |
MediaShelf uses `percent_complete` rather than `watched_status` as the primary signal
because `watched_status` depends on how the threshold is configured on Isis, and that
configuration can change without warning. `watched_status` is stored anyway and shown in
the UI as a cross-check; a systematic disagreement between the two is worth seeing.
The rollups per item become:
- `watch_count` — **completed** views only
- `partial_count`, `abandoned_count`
- `last_watched_at` — most recent completed view
- `last_touched_at` — most recent view of any kind, including abandoned
- `first_watched_at`, `distinct_watcher_count`
- `avg_percent_complete`
**An abandoned view is evidence for deletion, not against it.** Three people started a
film and all three quit twenty minutes in: that is a stronger signal to delete than a film
nobody ever opened, because the household has now actively rejected it. Plex's history
cannot express this at all, and it is the single biggest reason to prefer Tautulli. The
scoring model uses it (§6.1, `rejection` component).
### 4.11 Fallback, coverage, and cross-checks
**When Tautulli is unavailable**, MediaShelf falls back to `PlexHistoryProvider` and
`/status/sessions/history/all`. Events ingested that way are marked `source = 'plex'` and
carry no `percent_complete`, so they count as completed views (the only assumption
available) and are excluded from the `rejection` component. The UI shows a banner
explaining the score is running in degraded mode, because a silently degraded score is a
score that gets trusted when it shouldn't be.
**Tautulli's coverage horizon.** Tautulli only knows about plays since it was installed.
Anything watched before that is invisible to it — and those items will look never-watched.
On first ingest MediaShelf records `history_coverage_since` = the oldest event Tautulli
holds, and:
- Items with `added_at` earlier than that date get a `pre_history` flag.
- The UI shows a badge on those rows and a one-line explanation on the dashboard.
- The `staleness` score component is **capped** for `pre_history` items, so a 2009 movie
that was watched in 2015 and never since doesn't get scored as though it had never been
watched at all.
**Cross-check with `get_library_media_info`.** Tautulli maintains its own per-item table
with `file_size`, `added_at`, `play_count`, and `last_played`. MediaShelf pulls this once
per scan per library and compares it to what Plex reported. Disagreements are recorded on
the scan as warnings, not errors. This is cheap insurance: two independent views of the
same library disagreeing about a file's size is exactly the sort of thing that means a
scan is stale somewhere, and it is much better found by a scan than by a deletion.
**Joining to Plex.** Both systems use the same Plex `rating_key`, so the join is direct —
no title or path matching, no fuzzy logic. If `get_server_info.pms_identifier` does not
match the `machineIdentifier` from Plex's `/identity`, Tautulli is watching a *different*
Plex server and MediaShelf refuses to use it rather than joining nonsense data together.
That check runs at the start of every scan.
---
## 5. Data model
@ -364,10 +491,15 @@ CREATE TABLE media_item (
resolution TEXT,
video_codec TEXT,
-- watch rollups (from watch_event)
watch_count INTEGER NOT NULL DEFAULT 0,
last_watched_at INTEGER,
watch_count INTEGER NOT NULL DEFAULT 0, -- COMPLETED views only
partial_count INTEGER NOT NULL DEFAULT 0,
abandoned_count INTEGER NOT NULL DEFAULT 0,
last_watched_at INTEGER, -- last completed view
last_touched_at INTEGER, -- last view of any kind
first_watched_at INTEGER,
distinct_watcher_count INTEGER NOT NULL DEFAULT 0,
avg_percent_complete REAL,
pre_history INTEGER NOT NULL DEFAULT 0, -- added before history coverage
provider_view_count INTEGER NOT NULL DEFAULT 0, -- advisory, token-scoped
-- lifecycle
status TEXT NOT NULL DEFAULT 'present', -- 'present' | 'missing'
@ -411,24 +543,50 @@ CREATE TABLE media_part (
);
-- One row per playback event, server-wide, all accounts.
-- Primary source is Tautulli; 'plex' rows are degraded fallback events (§4.11).
CREATE TABLE watch_event (
id INTEGER PRIMARY KEY,
provider_id INTEGER NOT NULL REFERENCES provider(id),
history_key TEXT NOT NULL,
source TEXT NOT NULL, -- 'tautulli' | 'plex'
source_row_id TEXT NOT NULL, -- Tautulli row_id, or Plex historyKey
reference_id TEXT, -- Tautulli's own grouping id
provider_item_id TEXT NOT NULL, -- the leaf: movie or episode ratingKey
account_id TEXT,
viewed_at INTEGER NOT NULL,
UNIQUE (provider_id, history_key)
viewed_at INTEGER NOT NULL, -- 'date' / 'started'
stopped_at INTEGER,
play_duration_s INTEGER,
paused_counter_s INTEGER,
percent_complete INTEGER, -- NULL for source='plex'
watched_status REAL, -- Tautulli's verdict, advisory
disposition TEXT NOT NULL, -- 'completed' | 'partial' | 'abandoned'
session_id TEXT, -- MediaShelf's own merge key (§4.9)
media_type TEXT,
platform TEXT,
UNIQUE (provider_id, source, source_row_id)
);
CREATE TABLE account (
id INTEGER PRIMARY KEY,
provider_id INTEGER NOT NULL REFERENCES provider(id),
account_id TEXT NOT NULL,
account_id TEXT NOT NULL, -- Tautulli user_id / Plex accountID
name TEXT,
friendly_name TEXT,
UNIQUE (provider_id, account_id)
);
-- What the history record actually covers, so 'never watched' can be
-- distinguished from 'watched before Tautulli existed' (§4.11).
CREATE TABLE history_coverage (
id INTEGER PRIMARY KEY,
provider_id INTEGER NOT NULL REFERENCES provider(id),
source TEXT NOT NULL,
earliest_event_at INTEGER,
latest_event_at INTEGER,
event_count INTEGER NOT NULL DEFAULT 0,
updated_at INTEGER NOT NULL,
UNIQUE (provider_id, source)
);
CREATE TABLE scan (
id INTEGER PRIMARY KEY,
provider_id INTEGER NOT NULL REFERENCES provider(id),
@ -478,6 +636,8 @@ CREATE INDEX ix_part_item ON media_part(media_item_id);
CREATE INDEX ix_part_episode ON media_part(episode_id);
CREATE INDEX ix_event_item ON watch_event(provider_item_id);
CREATE INDEX ix_event_viewed ON watch_event(viewed_at);
CREATE INDEX ix_event_disp ON watch_event(provider_item_id, disposition);
CREATE INDEX ix_event_account ON watch_event(account_id, viewed_at);
```
A full-text index over `title` uses SQLite FTS5 (`media_item_fts`) kept in sync by
@ -493,8 +653,10 @@ After items and history are ingested, a rollup pass recomputes each season row:
| `episode_count` | `COUNT(episode)` |
| `duration_ms` | `SUM(episode.duration_ms)` |
| `added_at` | `MIN(episode.added_at)` — when the season first landed |
| `watch_count` | `SUM(episode.watch_count)` |
| `watch_count` | `SUM(episode.watch_count)` — completed episode views |
| `abandoned_count` | `SUM(episode.abandoned_count)` |
| `last_watched_at` | `MAX(episode.last_watched_at)` |
| `last_touched_at` | `MAX(episode.last_touched_at)` |
| `primary_path` | the common directory prefix of its episodes' paths |
A show row rolls up the same way from its seasons, for display and for the "delete the
@ -531,26 +693,38 @@ to want it?* It is a weighted sum of normalized components, each in `[0, 1]`, sc
| **`unpopularity`** | `1 - min(normalized_watches / POPULAR_AT, 1)` where `normalized_watches` = `watch_count` for movies, `watch_count / episode_count` for seasons | Rewatched things are keepers |
| **`age`** | `min(days_since_added / AGE_HORIZON, 1)` | Something added last week deserves a chance |
| **`solitude`** | `1 - min(distinct_watcher_count / 3, 1)` | Content only one household member ever touched is safer to cut |
| **`rejection`** | `min(abandoned_count / REJECTED_AT, 1)`, zero when `watch_count > 0` | Started and quit is an active verdict — see below |
`STALE_HORIZON` defaults to 730 days, `AGE_HORIZON` to 1095 days, `POPULAR_AT` to 3.
`STALE_HORIZON` defaults to 730 days, `AGE_HORIZON` to 1095 days, `POPULAR_AT` to 3,
`REJECTED_AT` to 2.
The `rejection` component only exists because Tautulli supplies `percent_complete`
(§4.10). It captures the case a purely count-based model gets backwards: an item three
people started and all abandoned looks "watched" to Plex and looks like a keeper to a
naive score, when in fact the household has tried it and said no. It is deliberately
zeroed the moment anyone completes a view, so a film that was abandoned twice and then
watched through is not penalised. When history is running in Plex-fallback mode this
component is unavailable and drops out of the weighting entirely (§4.11).
### 6.2 The formula
```
score = 100 × ( w_size · size
+ w_staleness · staleness
+ w_unpopularity· unpopularity
+ w_age · age
+ w_solitude · solitude )
/ (w_size + w_staleness + w_unpopularity + w_age + w_solitude)
score = 100 × Σ(wᵢ · componentᵢ) / Σ(wᵢ)
```
Default weight profile:
```json
{ "size": 0.30, "staleness": 0.30, "unpopularity": 0.25, "age": 0.10, "solitude": 0.05 }
{ "size": 0.28, "staleness": 0.26, "unpopularity": 0.22,
"rejection": 0.12, "age": 0.08, "solitude": 0.04 }
```
The denominator sums only the weights of components that are **available** for that row.
When history is in Plex-fallback mode, `rejection` is dropped and the remaining weights
renormalize, rather than every item silently scoring lower because one component is
pinned at zero. The same applies per-row: a `pre_history` item has its `staleness`
capped (§4.11), and the UI shows which components contributed.
### 6.3 The grace rule
Two hard overrides, applied after scoring, because a purely numeric model will
@ -607,7 +781,9 @@ Rules are a nested JSON group, evaluated into parameterized SQL:
Fields are drawn from a whitelist mapping field name → column and type. Operators are a
fixed set: `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `in`, `not_in`, `contains`,
`starts_with`, `is_null`, `is_not_null`, `older_than_days`, `newer_than_days`,
`never` (for watch fields). Anything outside the whitelist is a 400. Values are always
`never` (for watch fields). The whitelisted fields include the completion metrics —
`watch_count`, `partial_count`, `abandoned_count`, `avg_percent_complete`,
`last_touched_at`, `pre_history`. Anything outside the whitelist is a 400. Values are always
bound parameters. There is no SQL string interpolation anywhere in this path.
### 7.2 Seed views shipped by default
@ -618,6 +794,8 @@ bound parameters. There is no SQL string interpolation anywhere in this path.
| Cold storage | `last_watched_at older than 3 years` |
| One-and-done movies | `kind = movie AND watch_count = 1 AND last_watched_at older than 2 years` |
| Abandoned seasons | `kind = season AND watch_count/episode_count < 0.3 AND added_at older than 1 year` |
| Tried and rejected | `abandoned_count >= 2 AND watch_count = 0` — the household said no |
| Bailed in the first 15 min | `avg_percent_complete < 15 AND watch_count = 0 AND size_bytes >= 4 GB` |
| Biggest 100 | sorted by `size_bytes desc`, no filter |
| Recently added | `added_at newer than 30 days` — the sanity check, not a delete list |
@ -665,18 +843,24 @@ Response:
"size_bytes": 32212254720,
"added_at": 1490000000,
"last_watched_at": null,
"last_touched_at": 1712000000,
"watch_count": 0,
"distinct_watcher_count": 0,
"partial_count": 0,
"abandoned_count": 3,
"avg_percent_complete": 8.4,
"distinct_watcher_count": 3,
"pre_history": false,
"episode_count": null,
"primary_path": "/mnt/vault2/movies/Example Film (2011)/Example Film (2011).mkv",
"part_count": 1,
"resolution": "4k",
"reclaim_score": 91.4,
"reclaim_components": {
"size": 0.97, "staleness": 1.0, "unpopularity": 1.0, "age": 1.0, "solitude": 1.0
"size": 0.97, "staleness": 1.0, "unpopularity": 1.0,
"rejection": 1.0, "age": 1.0, "solitude": 0.0
},
"grace": null,
"flags": ["history_gap"]
"flags": ["rejected"]
}
]
}
@ -693,6 +877,9 @@ GET /api/v1/stats/size-by-library
GET /api/v1/stats/added-over-time?bucket=month
GET /api/v1/stats/watch-distribution
GET /api/v1/stats/size-vs-lastwatched scatter data, downsampled server-side
GET /api/v1/stats/completion finished / abandoned / never-opened, by size
GET /api/v1/sources per-source status: reachable?, coverage window,
event count, active history source
```
### 8.2 Saved views
@ -729,7 +916,10 @@ the artifact you take away and act on.
### 8.5 Health
```
GET /healthz → { "status": "ok", "db": "ok", "plex": "ok"|"unreachable",
GET /healthz → { "status": "ok", "db": "ok",
"plex": "ok"|"unreachable",
"tautulli": "ok"|"unreachable"|"disabled",
"history_source": "tautulli"|"plex",
"last_scan_at": 1757000000, "stale": false }
```
@ -750,7 +940,9 @@ never-watched content, size of content untouched for 2+ years, last scan time),
- *Size by library* — horizontal bar.
- *Added over time* — monthly bars, stacked by library. Shows acquisition habits.
- *Watched vs unwatched by size* — the "how much of this shelf has anyone ever touched"
chart, which is usually the one that motivates the whole exercise.
chart, which is usually the one that motivates the whole exercise. Split three ways
now that completion data exists: **finished · started and abandoned · never opened**.
The middle band is the interesting one and is invisible without Tautulli.
- *Size vs. last watched* — scatter, log-size on Y, days-since-watch on X. Points in the
upper right are the reclaim targets, and clicking a region filters the grid to it.
@ -769,8 +961,9 @@ never-watched content, size of content untouched for 2+ years, last scan time),
**Saved views** — list, create, edit, duplicate, run. Creating one captures the current
filter, sort, columns, and weights, so the flow is *explore, then name what you found*.
**Item detail** — everything known: all parts with paths and sizes, per-account watch
history timeline, score breakdown showing each component's contribution.
**Item detail** — everything known: all parts with paths and sizes, a per-account watch
timeline showing each play with how far it got (a row of 8%, 12%, 6% tells the story at a
glance), and the score breakdown showing each component's contribution.
**Scans** — history table, per-scan counts and duration, a "Scan now" button, live
progress.
@ -802,6 +995,14 @@ All via environment variables, so the Portainer stack is the single source of tr
| `PLEX_TIMEOUT_S` | `30` | |
| `PLEX_PAGE_SIZE` | `500` | |
| `PLEX_REQUEST_DELAY_MS` | `0` | throttle for Loki's sake |
| `TAUTULLI_BASE_URL` | — | e.g. `http://192.168.1.100:8181` — unset disables Tautulli |
| `TAUTULLI_API_KEY` | — | secret |
| `TAUTULLI_TIMEOUT_S` | `30` | |
| `TAUTULLI_PAGE_SIZE` | `1000` | `length` on `get_history` |
| `HISTORY_SOURCE` | `auto` | `auto` \| `tautulli` \| `plex` — `auto` prefers Tautulli, falls back |
| `SESSION_MERGE_WINDOW_H` | `6` | events merged into one viewing (§4.9) |
| `COMPLETION_THRESHOLD` | `85` | percent complete counting as a watch |
| `ABANDON_CEILING` | `15` | percent complete below which a play is "abandoned" |
| `DATABASE_PATH` | `/data/mediashelf.db` | on the mounted volume |
| `SCAN_SCHEDULE_CRON` | `0 4 * * *` | nightly incremental at 04:00 |
| `SCAN_FULL_SWEEP_CRON` | `0 3 * * 0` | weekly full sweep, Sunday 03:00 |
@ -810,14 +1011,17 @@ All via environment variables, so the Portainer stack is the single source of tr
| `SCORE_STALE_HORIZON_DAYS` | `730` | |
| `SCORE_AGE_HORIZON_DAYS` | `1095` | |
| `SCORE_POPULAR_AT` | `3` | |
| `SCORE_REJECTED_AT` | `2` | abandoned views for a full `rejection` score |
| `SCORE_GRACE_DAYS` | `30` | |
| `SCORE_RECENT_DAYS` | `90` | |
| `TZ` | `America/Regina` | so cron times mean what they look like |
| `LOG_LEVEL` | `INFO` | |
`PLEX_TOKEN` is a secret and must not be committed. The repo ships `.env.example` with
placeholders, and `.env` is gitignored. (Worth noting given the SOAP-password-in-history
problem on the Mythica repo — this one starts clean and stays clean.)
`PLEX_TOKEN` and `TAUTULLI_API_KEY` are secrets and must not be committed. The repo ships
`.env.example` with placeholders, and `.env` is gitignored. (Worth noting given the
SOAP-password-in-history problem on the Mythica repo — this one starts clean and stays
clean.) Neither credential is ever written to the database, the logs, or an API response;
`/api/v1/settings` redacts both.
---
@ -828,12 +1032,30 @@ problem on the Mythica repo — this one starts clean and stays clean.)
Single image, `python:3.12-slim` base, multi-stage so build deps don't ship. Runs as a
non-root user. One volume: `/data`. One port: 8080.
### 11.2 Portainer stack on Nox
### 11.2 Getting the image onto Nox
The image is built off-box and delivered to Nox's Portainer rather than built there. Two
workable routes, in order of preference:
1. **Registry** — build, tag `registry.hallsworth.ca/mediashelf:<version>` (or Forgejo's
own container registry, which this Forgejo version supports), push, and have the
Portainer stack pull it. This is the route worth setting up once: redeploys become a
version bump in the stack, and rollback is pulling the previous tag.
2. **Image upload** — `docker save mediashelf:<version> | gzip` and load it on Nox via
Portainer's *Images → Import*. No registry needed, fine for the first deploy or two,
but it makes rollback manual and versions easy to lose track of.
Either way the image is **tagged with a real version**, never deployed as bare `latest`.
The Mythica stack had to be torn down and recreated because a `PUT` update kept serving
old code; immutable version tags are how that failure mode is avoided rather than
worked around.
### 11.3 Portainer stack on Nox
```yaml
services:
mediashelf:
image: mediashelf:latest
image: registry.hallsworth.ca/mediashelf:0.1.0 # never bare :latest, see 11.2
container_name: mediashelf
restart: unless-stopped
ports:
@ -844,6 +1066,9 @@ services:
- MEDIASHELF_SECRET_KEY=${MEDIASHELF_SECRET_KEY}
- PLEX_BASE_URL=${PLEX_BASE_URL}
- PLEX_TOKEN=${PLEX_TOKEN}
- TAUTULLI_BASE_URL=${TAUTULLI_BASE_URL}
- TAUTULLI_API_KEY=${TAUTULLI_API_KEY}
- HISTORY_SOURCE=auto
- DATABASE_PATH=/data/mediashelf.db
- SCAN_SCHEDULE_CRON=${SCAN_SCHEDULE_CRON}
- TZ=America/Regina
@ -866,7 +1091,7 @@ Published through Nginx Proxy Manager at `mediashelf.hallsworth.ca`, upstream
`192.168.1.77:8085`. Note the host-port → container-port mapping is 8085 → 8080; NPM's
upstream must point at the **host** port.
### 11.3 The gunicorn / scheduler constraint
### 11.4 The gunicorn / scheduler constraint
APScheduler runs in-process. With `--workers 2`, both workers would start a scheduler and
two scans would race for the SQLite write lock. Mitigation: the scheduler starts only in
@ -874,12 +1099,13 @@ the worker that successfully acquires an exclusive advisory lock on a lockfile i
at boot. The `scan_lock` table is the second line of defence. Both are needed; either
alone has a race window.
### 11.4 Backup
### 11.5 Backup
The database is one file. A weekly `sqlite3 .backup` to the volume, keeping four, is
enough — everything in it is reconstructible from Plex except accumulated `watch_event`
rows beyond Plex's own history retention (§4.5), which is precisely the part worth
backing up.
enough — everything in it is reconstructible from Plex and Tautulli, so the real value
is avoiding a multi-hour re-ingest rather than protecting irreplaceable data. The one
genuinely irreplaceable thing is the saved-view definitions, which are small and worth
exporting to the repo as JSON once they stabilise.
---
@ -915,7 +1141,8 @@ a misconfigured proxy rule.
| Layer | Approach |
|---|---|
| Provider | Recorded Plex JSON fixtures (`responses`/`vcr.py`) covering a movie section, a show section with multi-season shows, multi-version movies, split parts, and a history page. No live server needed in CI. |
| Provider | Recorded JSON fixtures (`responses`/`vcr.py`) for both sources: Plex covering a movie section, a show section with multi-season shows, multi-version movies, split parts; Tautulli covering `get_history` pages, `get_users`, `get_library_media_info`, and an error-shaped 200 response. No live server needed in CI. |
| History semantics | Disposition classification at threshold boundaries; session merging across the window; a `pre_history` item scoring differently from a truly-never-watched one; Plex-fallback rows renormalizing the weights correctly. |
| Normalization | Table-driven tests: odd cases like an item with no `Part`, a season with a missing episode, unicode titles, zero-byte parts. |
| Ingest | In-memory SQLite; assert idempotency (running the same scan twice changes nothing), assert `missing` transitions, assert history watermarking doesn't skip or duplicate events. |
| Scoring | Unit tests on known inputs; **property test asserting the SQL and Python implementations agree** (§6.4); tests that grace rules clamp correctly. |
@ -965,17 +1192,26 @@ class MediaProvider(Protocol):
def server_info(self) -> ServerInfo: ...
def libraries(self) -> list[Library]: ...
def items(self, library: Library) -> Iterator[Item]: ... # yields normalized items
def watch_events(self, since: int | None) -> Iterator[WatchEvent]: ...
def accounts(self) -> list[Account]: ...
def refresh_library(self, library: Library) -> None: ... # v2 only
class HistoryProvider(Protocol):
def server_info(self) -> ServerInfo: ... # for the pms_identifier match, §4.11
def accounts(self) -> list[Account]: ...
def watch_events(self, since: int | None) -> Iterator[WatchEvent]: ...
def coverage(self) -> Coverage: ... # earliest/latest event, count
@property
def has_completion_data(self) -> bool: ... # False for PlexHistoryProvider
```
Jellyfin and Emby share an API lineage, so one `JellyfinProvider` will likely cover both
with a capability flag or two. The mapping is close: `Items` with
`Fields=Path,MediaSources,DateCreated`, `UserData.PlayCount` and `LastPlayedDate` for
watch data, `/Users` for accounts. The notable difference is that Jellyfin's watch data
is genuinely per-user and must be fetched per user and merged, rather than read from one
global history endpoint.
watch data, `/Users` for accounts. Two differences matter. Their watch data is genuinely
per-user and must be fetched per user and merged, rather than read from one global history
endpoint. And there is no Tautulli equivalent, so `has_completion_data` is `False` and the
`rejection` component drops out — the same degraded path the Plex fallback already
exercises, which is a good reason for that path to exist and be tested from day one
rather than bolted on later.
The `provider` table already carries `kind` and the schema already keys everything on
`(provider_id, provider_item_id)`, so multiple servers can coexist in one database — the
@ -989,24 +1225,41 @@ door is open to a combined view across Plex and Jellyfin without a migration.
and invisible source of waste.
- **Trend tracking** — MediaShelf already snapshots on a schedule; a chart of library
size over time and a projection of when the vault fills up is nearly free.
- **Tautulli import** — if Tautulli is running, its history database predates Plex's own
retention window and would deepen the watch record considerably.
- **Direct Tautulli database read** — Tautulli's SQLite file could be read directly
instead of paging its API, which would make the first full history ingest dramatically
faster. Only worth doing if the API ingest proves slow, and it couples MediaShelf to
Tautulli's internal schema, so the API stays the default.
- **Per-user reclaim views** — the data to answer "what does only one person watch"
is already stored; it just has no UI in v1.
- **Notifications** — a monthly "here's what's gone cold" summary.
---
## 15. Open questions
1. **Plex history retention on Loki.** If it's short, the first ingest captures less than
hoped and `watch_event` becomes valuable immediately. Worth checking before the first
scan, and worth knowing whether Tautulli has been running (§14.3).
2. **Multiple Plex libraries of the same kind** — are there several movie sections (e.g.
Movies, Kids, Documentaries)? The design handles it, but the seed views and default
charts get more useful if the real shape is known.
3. **Path structure on the vaults** — knowing the actual roots would let the grid group by
physical vault, which matters when the goal is freeing a *specific* array rather than
space in general.
4. **Where does the image get built?** Nox running Portainer suggests either a build from
a git URL in the stack, or pushing to a registry. Worth settling before the first
deploy.
5. **Is Tautulli running anywhere?** It changes the watch-history story materially.
### Resolved
- ~~Is Tautulli running anywhere?~~ **Yes** — 192.168.1.100:8181, on Isis. Promoted to
the primary watch-history source; see §4.8–4.11.
- ~~Plex history retention on Loki~~ — no longer on the critical path, since Tautulli
keeps its own independent record.
- ~~Where does the image get built?~~ Built off-box, delivered to Nox's Portainer (§11.2).
### Still open
1. **When was Tautulli installed?** This sets `history_coverage_since` and therefore how
many older items get the `pre_history` flag. If it went in recently, a large slice of
the library will look never-watched on the first report and the flag is doing real
work; if it has been running for years, it barely matters. A single
`get_history&order_dir=asc&length=1` answers it.
2. **Is *Group Successive Play History* on in Tautulli's settings?** MediaShelf requests
`grouping=0` and does its own merging (§4.9), so it should not matter — but confirming
the setting on the first run rules out a whole class of double-counting bug.
3. **Multiple Plex libraries of the same kind** — are there several movie sections (Movies,
Kids, Documentaries)? The design handles it, but the seed views and default charts get
more useful if the real shape is known.
4. **Path structure on the vaults** — knowing the actual roots would let the grid group by
physical vault, which matters when the goal is freeing a *specific* array (vault 2)
rather than space in general.
5. **Registry or image upload?** (§11.2) Setting up a registry is the better long-term
answer but is a piece of infrastructure that doesn't exist yet.