Add a read-only LAN probe for validating the design
Tautulli stays local-network-only, so nothing off the LAN can check this design against real data. tools/probe.py closes that gap from the inside: GET requests only, standard library only, credentials redacted from all output including error messages. It answers open questions 1-4 in one run — Tautulli's coverage horizon, the finished/partial/abandoned split across real plays, whether successive plays are being grouped, library shapes and sizes, multi-version items, and path roots by size. It also runs the pms_identifier cross-check from section 4.11. tools/mockserver.py mocks both APIs so the probe is testable without a live server. Verified against it: the happy path, Tautulli absent, Tautulli unreachable, Tautulli erroring, Plex unreachable, and an identifier mismatch. Credential redaction confirmed in every error path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVbG48GAXfCZatcmX123Ra
This commit is contained in:
parent
74dfc4dab8
commit
2c1033e4a7
4 changed files with 626 additions and 5 deletions
23
README.md
23
README.md
|
|
@ -46,6 +46,29 @@ Portainer stack behind Nginx Proxy Manager.
|
||||||
allowlist, and an audit log
|
allowlist, and an audit log
|
||||||
- **v3** — Emby and Jellyfin support behind the existing `MediaProvider` abstraction
|
- **v3** — Emby and Jellyfin support behind the existing `MediaProvider` abstraction
|
||||||
|
|
||||||
|
## Validating the design first
|
||||||
|
|
||||||
|
Plex and Tautulli are both LAN-only, so `tools/probe.py` exists to check this design
|
||||||
|
against real data from inside the network. It is **read-only** — GET requests only,
|
||||||
|
nothing is modified — and has no dependencies beyond the standard library.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PLEX_BASE_URL=http://192.168.1.10:32400
|
||||||
|
export PLEX_TOKEN=...
|
||||||
|
export TAUTULLI_BASE_URL=http://192.168.1.100:8181
|
||||||
|
export TAUTULLI_API_KEY=...
|
||||||
|
|
||||||
|
python3 tools/probe.py # summary to stdout
|
||||||
|
python3 tools/probe.py --dump inventory.json # plus a full item inventory
|
||||||
|
```
|
||||||
|
|
||||||
|
It reports Tautulli's coverage horizon, the finished/abandoned/never-opened split across
|
||||||
|
real plays, library shapes and sizes, path roots, multi-version items, and whether
|
||||||
|
Tautulli is actually watching the Plex server you think it is. Credentials are redacted
|
||||||
|
from all output including error messages, so the result is safe to paste anywhere.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- [`docs/design.md`](docs/design.md) — the full software design
|
- [`docs/design.md`](docs/design.md) — the full software design
|
||||||
|
- [`tools/probe.py`](tools/probe.py) — read-only reconnaissance script
|
||||||
|
- [`tools/mockserver.py`](tools/mockserver.py) — mock Plex/Tautulli for testing the probe
|
||||||
|
|
|
||||||
|
|
@ -305,7 +305,10 @@ the ingest is designed to be slow and steady rather than parallel and aggressive
|
||||||
### 4.8 Tautulli — the watch-history source
|
### 4.8 Tautulli — the watch-history source
|
||||||
|
|
||||||
Tautulli runs at `http://192.168.1.100:8181` (on Isis) and has been logging every
|
Tautulli runs at `http://192.168.1.100:8181` (on Isis) and has been logging every
|
||||||
playback on Loki since the day it was installed. Its database is independent of Plex's:
|
playback on Loki since the day it was installed. **It stays LAN-only and is not published
|
||||||
|
externally** — which costs nothing here, since MediaShelf runs on Nox on the same network.
|
||||||
|
The practical consequence is that design validation happens on the LAN: see
|
||||||
|
`tools/probe.py` (§13.1). Its database is independent of Plex's:
|
||||||
clearing Plex's history, or Plex pruning its own, does not touch it. That makes it a
|
clearing Plex's history, or Plex pruning its own, does not touch it. That makes it a
|
||||||
deeper and more durable record than anything Plex exposes.
|
deeper and more durable record than anything Plex exposes.
|
||||||
|
|
||||||
|
|
@ -1154,6 +1157,31 @@ The single most important test is idempotency: a scanner that double-counts size
|
||||||
duplicates history events produces a report that looks plausible and is wrong, which is
|
duplicates history events produces a report that looks plausible and is wrong, which is
|
||||||
worse than one that crashes.
|
worse than one that crashes.
|
||||||
|
|
||||||
|
### 13.1 Validating the design before writing the app
|
||||||
|
|
||||||
|
Both Plex and Tautulli are LAN-only, so nothing outside the network can check this design
|
||||||
|
against real data. `tools/probe.py` closes that gap: a dependency-free, **read-only**
|
||||||
|
script run from anywhere on the LAN that issues GET requests only and answers the open
|
||||||
|
questions in §15 directly —
|
||||||
|
|
||||||
|
- Tautulli's coverage horizon, and therefore how much of the library gets `pre_history`
|
||||||
|
- the finished / partial / abandoned split across a sample of real plays, which is the
|
||||||
|
assumption the `rejection` component rests on
|
||||||
|
- whether successive-play grouping is actually happening, so the session-merge logic
|
||||||
|
(§4.9) can be checked against reality
|
||||||
|
- real library shapes: movie counts, season counts, total sizes, multi-version items
|
||||||
|
- path roots and their sizes, which decides whether grouping by physical vault is worth
|
||||||
|
building
|
||||||
|
|
||||||
|
It also does the `pms_identifier` cross-check from §4.11 and reports whether it passes.
|
||||||
|
Its output is safe to share — credentials are redacted from every message, including
|
||||||
|
error messages, which is verified by test. `--dump` writes a full item inventory to JSON
|
||||||
|
for offline analysis, and prints the twenty largest never-watched items as a first taste
|
||||||
|
of what the real report will look like.
|
||||||
|
|
||||||
|
`tools/mockserver.py` mocks both APIs so the probe can be exercised without touching a
|
||||||
|
live server; it is a test fixture, not part of the application.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 14. Roadmap
|
## 14. Roadmap
|
||||||
|
|
@ -1247,14 +1275,15 @@ door is open to a combined view across Plex and Jellyfin without a migration.
|
||||||
|
|
||||||
### Still open
|
### Still open
|
||||||
|
|
||||||
|
**Questions 1–4 are all answered by one run of `tools/probe.py` on the LAN (§13.1).**
|
||||||
|
|
||||||
1. **When was Tautulli installed?** This sets `history_coverage_since` and therefore how
|
1. **When was Tautulli installed?** This sets `history_coverage_since` and therefore how
|
||||||
many older items get the `pre_history` flag. If it went in recently, a large slice of
|
many older items get the `pre_history` flag. If it went in recently, a large slice of
|
||||||
the library will look never-watched on the first report and the flag is doing real
|
the library will look never-watched on the first report and the flag is doing real
|
||||||
work; if it has been running for years, it barely matters. A single
|
work; if it has been running for years, it barely matters.
|
||||||
`get_history&order_dir=asc&length=1` answers it.
|
|
||||||
2. **Is *Group Successive Play History* on in Tautulli's settings?** MediaShelf requests
|
2. **Is *Group Successive Play History* on in Tautulli's settings?** MediaShelf requests
|
||||||
`grouping=0` and does its own merging (§4.9), so it should not matter — but confirming
|
`grouping=0` and does its own merging (§4.9), so it should not matter — but confirming
|
||||||
the setting on the first run rules out a whole class of double-counting bug.
|
it on the first run rules out a whole class of double-counting bug.
|
||||||
3. **Multiple Plex libraries of the same kind** — are there several movie sections (Movies,
|
3. **Multiple Plex libraries of the same kind** — are there several movie sections (Movies,
|
||||||
Kids, Documentaries)? The design handles it, but the seed views and default charts get
|
Kids, Documentaries)? The design handles it, but the seed views and default charts get
|
||||||
more useful if the real shape is known.
|
more useful if the real shape is known.
|
||||||
|
|
@ -1262,4 +1291,5 @@ door is open to a combined view across Plex and Jellyfin without a migration.
|
||||||
physical vault, which matters when the goal is freeing a *specific* array (vault 2)
|
physical vault, which matters when the goal is freeing a *specific* array (vault 2)
|
||||||
rather than space in general.
|
rather than space in general.
|
||||||
5. **Registry or image upload?** (§11.2) Setting up a registry is the better long-term
|
5. **Registry or image upload?** (§11.2) Setting up a registry is the better long-term
|
||||||
answer but is a piece of infrastructure that doesn't exist yet.
|
answer but is a piece of infrastructure that doesn't exist yet. Not answerable by the
|
||||||
|
probe — this one is a preference.
|
||||||
|
|
|
||||||
119
tools/mockserver.py
Normal file
119
tools/mockserver.py
Normal file
|
|
@ -0,0 +1,119 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Throwaway mock of the Plex + Tautulli endpoints probe.py uses, for testing it
|
||||||
|
without a live server. Not part of the application."""
|
||||||
|
import json
|
||||||
|
import random
|
||||||
|
import time
|
||||||
|
import urllib.parse
|
||||||
|
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||||
|
|
||||||
|
NOW = int(time.time())
|
||||||
|
random.seed(7)
|
||||||
|
|
||||||
|
MOVIES = []
|
||||||
|
for i in range(120):
|
||||||
|
parts = [{"file": "/mnt/vault2/movies/Film %d/Film %d.mkv" % (i, i),
|
||||||
|
"size": random.randint(2, 60) * 10**9, "container": "mkv"}]
|
||||||
|
if i % 17 == 0: # some multi-version items
|
||||||
|
parts.append({"file": "/mnt/vault1/movies4k/Film %d/Film %d.4k.mkv" % (i, i),
|
||||||
|
"size": random.randint(40, 90) * 10**9, "container": "mkv"})
|
||||||
|
MOVIES.append({
|
||||||
|
"ratingKey": str(1000 + i), "type": "movie", "title": "Film %d" % i,
|
||||||
|
"year": 1990 + i % 35, "addedAt": NOW - random.randint(30, 3000) * 86400,
|
||||||
|
"viewCount": random.choice([0, 0, 0, 1, 2]),
|
||||||
|
"Media": [{"videoResolution": "1080", "Part": parts}],
|
||||||
|
})
|
||||||
|
|
||||||
|
EPISODES = []
|
||||||
|
for s in range(8):
|
||||||
|
for se in range(1, 4):
|
||||||
|
for ep in range(1, 11):
|
||||||
|
EPISODES.append({
|
||||||
|
"ratingKey": str(50000 + len(EPISODES)), "type": "episode",
|
||||||
|
"title": "Ep %d" % ep, "index": ep,
|
||||||
|
"grandparentRatingKey": str(900 + s), "grandparentTitle": "Show %d" % s,
|
||||||
|
"parentRatingKey": str(9000 + s * 10 + se), "parentIndex": se,
|
||||||
|
"addedAt": NOW - random.randint(30, 2000) * 86400,
|
||||||
|
"viewCount": random.choice([0, 0, 1]),
|
||||||
|
"Media": [{"Part": [{"file": "/mnt/vault2/tv/Show %d/S%02d/E%02d.mkv" % (s, se, ep),
|
||||||
|
"size": random.randint(1, 4) * 10**9}]}],
|
||||||
|
})
|
||||||
|
|
||||||
|
HISTORY = []
|
||||||
|
for i in range(500):
|
||||||
|
HISTORY.append({
|
||||||
|
"row_id": i, "reference_id": i // 2, "date": NOW - random.randint(1, 900) * 86400,
|
||||||
|
"rating_key": str(random.choice([m["ratingKey"] for m in MOVIES])),
|
||||||
|
"user_id": random.choice([1, 2, 3]), "user": "u%d" % random.choice([1, 2, 3]),
|
||||||
|
"friendly_name": random.choice(["Jess", "Sam", "Guest"]),
|
||||||
|
"media_type": "movie",
|
||||||
|
"percent_complete": random.choice([3, 8, 12, 45, 60, 92, 97, 100]),
|
||||||
|
"watched_status": random.choice([0, 0.5, 1]),
|
||||||
|
"play_duration": random.randint(60, 9000), "paused_counter": 0,
|
||||||
|
})
|
||||||
|
HISTORY.sort(key=lambda h: h["date"])
|
||||||
|
|
||||||
|
|
||||||
|
def page(items, q):
|
||||||
|
start = int(q.get("X-Plex-Container-Start", ["0"])[0])
|
||||||
|
size = int(q.get("X-Plex-Container-Size", [str(len(items))])[0])
|
||||||
|
return items[start:start + size], len(items)
|
||||||
|
|
||||||
|
|
||||||
|
class H(BaseHTTPRequestHandler):
|
||||||
|
def log_message(self, *a):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _send(self, obj):
|
||||||
|
body = json.dumps(obj).encode()
|
||||||
|
self.send_response(200)
|
||||||
|
self.send_header("Content-Type", "application/json")
|
||||||
|
self.send_header("Content-Length", str(len(body)))
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(body)
|
||||||
|
|
||||||
|
def do_GET(self):
|
||||||
|
u = urllib.parse.urlsplit(self.path)
|
||||||
|
q = urllib.parse.parse_qs(u.query)
|
||||||
|
|
||||||
|
if u.path == "/identity":
|
||||||
|
return self._send({"MediaContainer": {"machineIdentifier": "abc123def456",
|
||||||
|
"version": "1.41.0.1234"}})
|
||||||
|
if u.path == "/library/sections":
|
||||||
|
return self._send({"MediaContainer": {"Directory": [
|
||||||
|
{"key": "1", "type": "movie", "title": "Movies",
|
||||||
|
"Location": [{"path": "/mnt/vault2/movies"}]},
|
||||||
|
{"key": "2", "type": "show", "title": "TV Shows",
|
||||||
|
"Location": [{"path": "/mnt/vault2/tv"}]},
|
||||||
|
{"key": "3", "type": "artist", "title": "Music",
|
||||||
|
"Location": [{"path": "/mnt/vault2/music"}]},
|
||||||
|
]}})
|
||||||
|
if u.path.startswith("/library/sections/") and u.path.endswith("/all"):
|
||||||
|
key = u.path.split("/")[3]
|
||||||
|
src = MOVIES if key == "1" else EPISODES
|
||||||
|
batch, total = page(src, q)
|
||||||
|
return self._send({"MediaContainer": {"Metadata": batch, "totalSize": total,
|
||||||
|
"size": len(batch)}})
|
||||||
|
if u.path == "/api/v2":
|
||||||
|
cmd = q.get("cmd", [""])[0]
|
||||||
|
if cmd == "get_server_info":
|
||||||
|
return self._send({"response": {"result": "success", "data": {
|
||||||
|
"pms_identifier": "abc123def456", "pms_name": "Loki",
|
||||||
|
"pms_ip": "192.168.1.10", "pms_port": 32400}}})
|
||||||
|
if cmd == "get_history":
|
||||||
|
d = q.get("order_dir", ["desc"])[0]
|
||||||
|
rows = HISTORY if d == "asc" else list(reversed(HISTORY))
|
||||||
|
length = int(q.get("length", ["25"])[0])
|
||||||
|
return self._send({"response": {"result": "success", "data": {
|
||||||
|
"data": rows[:length], "recordsFiltered": len(HISTORY),
|
||||||
|
"recordsTotal": len(HISTORY)}}})
|
||||||
|
if cmd == "get_users":
|
||||||
|
return self._send({"response": {"result": "success", "data": []}})
|
||||||
|
return self._send({"response": {"result": "error",
|
||||||
|
"message": "unknown cmd", "data": None}})
|
||||||
|
self.send_response(404)
|
||||||
|
self.end_headers()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
HTTPServer(("127.0.0.1", 8899), H).serve_forever()
|
||||||
449
tools/probe.py
Executable file
449
tools/probe.py
Executable file
|
|
@ -0,0 +1,449 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
MediaShelf probe — read-only reconnaissance against Plex and Tautulli.
|
||||||
|
|
||||||
|
Run this on the LAN (anywhere that can reach both servers). It answers the open
|
||||||
|
questions in docs/design.md against real data, so the design can be validated
|
||||||
|
without exposing either service outside the network.
|
||||||
|
|
||||||
|
It is READ-ONLY. It issues GET requests only. It never deletes, modifies, or
|
||||||
|
writes anything to either server, and it never prints your credentials.
|
||||||
|
|
||||||
|
Standard library only — no pip install needed. Python 3.8+.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
|
||||||
|
export PLEX_BASE_URL=http://192.168.1.10:32400
|
||||||
|
export PLEX_TOKEN=...
|
||||||
|
export TAUTULLI_BASE_URL=http://192.168.1.100:8181
|
||||||
|
export TAUTULLI_API_KEY=...
|
||||||
|
python3 probe.py
|
||||||
|
|
||||||
|
# or, to also write the full item inventory out for offline analysis:
|
||||||
|
python3 probe.py --dump inventory.json
|
||||||
|
|
||||||
|
Everything it prints is safe to share.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
|
||||||
|
TIMEOUT = 30
|
||||||
|
PAGE = 500
|
||||||
|
HIST_PAGE = 1000
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────── plumbing ────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
class Fail(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _get(url, headers=None):
|
||||||
|
req = urllib.request.Request(url, headers=headers or {}, method="GET")
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||||
|
return json.loads(r.read().decode("utf-8", "replace"))
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
raise Fail("HTTP %s from %s" % (e.code, _safe(url)))
|
||||||
|
except urllib.error.URLError as e:
|
||||||
|
raise Fail("cannot reach %s (%s)" % (_safe(url), e.reason))
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
raise Fail("non-JSON response from %s" % _safe(url))
|
||||||
|
|
||||||
|
|
||||||
|
def _safe(url):
|
||||||
|
"""Strip credentials out of a URL before it appears in any output."""
|
||||||
|
parts = urllib.parse.urlsplit(url)
|
||||||
|
q = urllib.parse.parse_qsl(parts.query)
|
||||||
|
q = [(k, "***" if k.lower() in ("apikey", "x-plex-token") else v) for k, v in q]
|
||||||
|
return urllib.parse.urlunsplit(
|
||||||
|
(parts.scheme, parts.netloc, parts.path, urllib.parse.urlencode(q), "")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def human(n):
|
||||||
|
if n is None:
|
||||||
|
return "?"
|
||||||
|
for unit in ("B", "KB", "MB", "GB", "TB", "PB"):
|
||||||
|
if abs(n) < 1024:
|
||||||
|
return "%.1f %s" % (n, unit)
|
||||||
|
n /= 1024.0
|
||||||
|
return "%.1f EB" % n
|
||||||
|
|
||||||
|
|
||||||
|
def ymd(ts):
|
||||||
|
if not ts:
|
||||||
|
return "—"
|
||||||
|
return time.strftime("%Y-%m-%d", time.localtime(int(ts)))
|
||||||
|
|
||||||
|
|
||||||
|
def days_since(ts):
|
||||||
|
if not ts:
|
||||||
|
return None
|
||||||
|
return int((time.time() - int(ts)) / 86400)
|
||||||
|
|
||||||
|
|
||||||
|
# ──────────────────────────────── plex ───────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
class Plex:
|
||||||
|
def __init__(self, base, token):
|
||||||
|
self.base = base.rstrip("/")
|
||||||
|
self.headers = {
|
||||||
|
"X-Plex-Token": token,
|
||||||
|
"Accept": "application/json",
|
||||||
|
"X-Plex-Product": "MediaShelf-probe",
|
||||||
|
"X-Plex-Client-Identifier": "mediashelf-probe",
|
||||||
|
}
|
||||||
|
|
||||||
|
def get(self, path, **params):
|
||||||
|
url = self.base + path
|
||||||
|
if params:
|
||||||
|
url += "?" + urllib.parse.urlencode(params)
|
||||||
|
return _get(url, self.headers).get("MediaContainer", {})
|
||||||
|
|
||||||
|
def identity(self):
|
||||||
|
return self.get("/identity")
|
||||||
|
|
||||||
|
def sections(self):
|
||||||
|
return self.get("/library/sections").get("Directory", [])
|
||||||
|
|
||||||
|
def section_detail(self, key):
|
||||||
|
return self.get("/library/sections/" + str(key))
|
||||||
|
|
||||||
|
def items(self, key, libtype):
|
||||||
|
"""Page through a section, yielding raw Video/Directory dicts."""
|
||||||
|
start = 0
|
||||||
|
while True:
|
||||||
|
mc = self.get(
|
||||||
|
"/library/sections/%s/all" % key,
|
||||||
|
type=libtype,
|
||||||
|
**{
|
||||||
|
"X-Plex-Container-Start": start,
|
||||||
|
"X-Plex-Container-Size": PAGE,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
batch = mc.get("Metadata", []) or []
|
||||||
|
if not batch:
|
||||||
|
return
|
||||||
|
for row in batch:
|
||||||
|
yield row
|
||||||
|
start += len(batch)
|
||||||
|
total = mc.get("totalSize") or mc.get("size") or 0
|
||||||
|
if start >= int(total):
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
|
def item_size(row):
|
||||||
|
total = 0
|
||||||
|
parts = 0
|
||||||
|
for media in row.get("Media", []) or []:
|
||||||
|
for part in media.get("Part", []) or []:
|
||||||
|
total += int(part.get("size") or 0)
|
||||||
|
parts += 1
|
||||||
|
return total, parts
|
||||||
|
|
||||||
|
|
||||||
|
def item_paths(row):
|
||||||
|
out = []
|
||||||
|
for media in row.get("Media", []) or []:
|
||||||
|
for part in media.get("Part", []) or []:
|
||||||
|
if part.get("file"):
|
||||||
|
out.append(part["file"])
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# ────────────────────────────── tautulli ─────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
class Tautulli:
|
||||||
|
def __init__(self, base, key):
|
||||||
|
self.base = base.rstrip("/") + "/api/v2"
|
||||||
|
self.key = key
|
||||||
|
|
||||||
|
def cmd(self, command, **params):
|
||||||
|
params["apikey"] = self.key
|
||||||
|
params["cmd"] = command
|
||||||
|
url = self.base + "?" + urllib.parse.urlencode(params)
|
||||||
|
body = _get(url)
|
||||||
|
resp = body.get("response") or {}
|
||||||
|
if resp.get("result") != "success":
|
||||||
|
raise Fail(
|
||||||
|
"Tautulli cmd=%s returned %s: %s"
|
||||||
|
% (command, resp.get("result"), resp.get("message"))
|
||||||
|
)
|
||||||
|
return resp.get("data")
|
||||||
|
|
||||||
|
def server_info(self):
|
||||||
|
return self.cmd("get_server_info") or {}
|
||||||
|
|
||||||
|
def users(self):
|
||||||
|
return self.cmd("get_users") or []
|
||||||
|
|
||||||
|
def history_edge(self, direction):
|
||||||
|
"""Oldest (asc) or newest (desc) single history row."""
|
||||||
|
d = self.cmd(
|
||||||
|
"get_history", order_column="date", order_dir=direction,
|
||||||
|
grouping=0, start=0, length=1,
|
||||||
|
) or {}
|
||||||
|
rows = d.get("data") or []
|
||||||
|
return (rows[0] if rows else None), d.get("recordsFiltered") or d.get("recordsTotal") or 0
|
||||||
|
|
||||||
|
def history_sample(self, length=2000):
|
||||||
|
d = self.cmd(
|
||||||
|
"get_history", order_column="date", order_dir="desc",
|
||||||
|
grouping=0, start=0, length=length,
|
||||||
|
) or {}
|
||||||
|
return d.get("data") or []
|
||||||
|
|
||||||
|
|
||||||
|
# ───────────────────────────────── report ────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def section(title):
|
||||||
|
print("\n" + title)
|
||||||
|
print("─" * len(title))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description="Read-only Plex + Tautulli probe for MediaShelf.")
|
||||||
|
ap.add_argument("--dump", metavar="FILE", help="write the full item inventory to FILE as JSON")
|
||||||
|
ap.add_argument("--history-sample", type=int, default=2000,
|
||||||
|
help="how many recent history rows to sample (default 2000)")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
plex_url = os.environ.get("PLEX_BASE_URL")
|
||||||
|
plex_token = os.environ.get("PLEX_TOKEN")
|
||||||
|
taut_url = os.environ.get("TAUTULLI_BASE_URL")
|
||||||
|
taut_key = os.environ.get("TAUTULLI_API_KEY")
|
||||||
|
|
||||||
|
if not plex_url or not plex_token:
|
||||||
|
sys.exit("PLEX_BASE_URL and PLEX_TOKEN must be set.")
|
||||||
|
|
||||||
|
print("MediaShelf probe — read-only. No data is modified.")
|
||||||
|
print("Run at %s" % time.strftime("%Y-%m-%d %H:%M:%S"))
|
||||||
|
|
||||||
|
plex = Plex(plex_url, plex_token)
|
||||||
|
taut = Tautulli(taut_url, taut_key) if (taut_url and taut_key) else None
|
||||||
|
|
||||||
|
# ── identity and the cross-check ───────────────────────────────────
|
||||||
|
section("Servers")
|
||||||
|
try:
|
||||||
|
ident = plex.identity()
|
||||||
|
pms_id = ident.get("machineIdentifier")
|
||||||
|
print("Plex : version %s, machineIdentifier %s…"
|
||||||
|
% (ident.get("version"), (pms_id or "")[:8]))
|
||||||
|
except Fail as e:
|
||||||
|
sys.exit("Plex unreachable: %s" % e)
|
||||||
|
|
||||||
|
taut_ok = False
|
||||||
|
if taut:
|
||||||
|
try:
|
||||||
|
si = taut.server_info()
|
||||||
|
taut_ok = True
|
||||||
|
match = si.get("pms_identifier") == pms_id
|
||||||
|
print("Tautulli : watching %s (%s:%s)"
|
||||||
|
% (si.get("pms_name"), si.get("pms_ip"), si.get("pms_port")))
|
||||||
|
print(" pms_identifier match: %s"
|
||||||
|
% ("YES" if match else "NO — Tautulli is watching a DIFFERENT Plex server"))
|
||||||
|
if not match:
|
||||||
|
print(" ** MediaShelf would refuse to join this history data. **")
|
||||||
|
except Fail as e:
|
||||||
|
print("Tautulli : UNREACHABLE — %s" % e)
|
||||||
|
else:
|
||||||
|
print("Tautulli : not configured (set TAUTULLI_BASE_URL and TAUTULLI_API_KEY)")
|
||||||
|
|
||||||
|
# ── question 1: coverage horizon ───────────────────────────────────
|
||||||
|
coverage_since = None
|
||||||
|
if taut_ok:
|
||||||
|
section("Q1 · Tautulli coverage horizon")
|
||||||
|
try:
|
||||||
|
oldest, total = taut.history_edge("asc")
|
||||||
|
newest, _ = taut.history_edge("desc")
|
||||||
|
if oldest:
|
||||||
|
coverage_since = int(oldest.get("date") or 0)
|
||||||
|
print("Oldest logged play : %s (%s days ago)"
|
||||||
|
% (ymd(coverage_since), days_since(coverage_since)))
|
||||||
|
print("Newest logged play : %s" % ymd(newest.get("date") if newest else None))
|
||||||
|
print("Total history rows : %s" % f"{int(total):,}")
|
||||||
|
print("\n-> Anything added before %s that was watched before that date"
|
||||||
|
% ymd(coverage_since))
|
||||||
|
print(" will look never-watched. MediaShelf flags these 'pre_history'.")
|
||||||
|
else:
|
||||||
|
print("No history rows found — Tautulli has logged nothing yet.")
|
||||||
|
except Fail as e:
|
||||||
|
print("Could not read history: %s" % e)
|
||||||
|
|
||||||
|
# ── question 2: grouping sanity + user spread ──────────────────────
|
||||||
|
if taut_ok:
|
||||||
|
section("Q2 · History shape (sample)")
|
||||||
|
try:
|
||||||
|
rows = taut.history_sample(args.history_sample)
|
||||||
|
print("Sampled %s most recent rows." % f"{len(rows):,}")
|
||||||
|
if rows:
|
||||||
|
users = Counter(r.get("friendly_name") or r.get("user") for r in rows)
|
||||||
|
print("\nDistinct users in sample: %d" % len(users))
|
||||||
|
for name, n in users.most_common(10):
|
||||||
|
print(" %-24s %6d plays" % (str(name)[:24], n))
|
||||||
|
|
||||||
|
pcs = [int(r.get("percent_complete") or 0) for r in rows]
|
||||||
|
finished = sum(1 for p in pcs if p >= 85)
|
||||||
|
abandoned = sum(1 for p in pcs if p < 15)
|
||||||
|
partial = len(pcs) - finished - abandoned
|
||||||
|
print("\nCompletion breakdown of sampled plays:")
|
||||||
|
print(" finished (>=85%%) : %6d (%.1f%%)" % (finished, 100.0 * finished / len(pcs)))
|
||||||
|
print(" partial (15-85%%): %6d (%.1f%%)" % (partial, 100.0 * partial / len(pcs)))
|
||||||
|
print(" abandoned (<15%%) : %6d (%.1f%%)" % (abandoned, 100.0 * abandoned / len(pcs)))
|
||||||
|
print("\n-> The 'abandoned' share is the signal Plex cannot provide.")
|
||||||
|
|
||||||
|
refs = Counter(r.get("reference_id") for r in rows if r.get("reference_id"))
|
||||||
|
dupes = sum(1 for c in refs.values() if c > 1)
|
||||||
|
print("\nRows sharing a reference_id: %d groups" % dupes)
|
||||||
|
print("-> Non-zero is expected and fine: it means successive plays exist")
|
||||||
|
print(" and MediaShelf's own session merging has real work to do.")
|
||||||
|
except Fail as e:
|
||||||
|
print("Could not sample history: %s" % e)
|
||||||
|
|
||||||
|
# ── questions 3 & 4: library shape and paths ───────────────────────
|
||||||
|
section("Q3/Q4 · Libraries, sizes and path roots")
|
||||||
|
try:
|
||||||
|
secs = plex.sections()
|
||||||
|
except Fail as e:
|
||||||
|
sys.exit("Could not list sections: %s" % e)
|
||||||
|
|
||||||
|
inventory = []
|
||||||
|
grand_total = 0
|
||||||
|
grand_count = 0
|
||||||
|
root_bytes = defaultdict(int)
|
||||||
|
|
||||||
|
for s in secs:
|
||||||
|
stype = s.get("type")
|
||||||
|
if stype not in ("movie", "show"):
|
||||||
|
print("\n%-24s %-8s (skipped — not movie/show)" % (s.get("title"), stype))
|
||||||
|
continue
|
||||||
|
|
||||||
|
key = s.get("key")
|
||||||
|
locs = [d.get("path") for d in (s.get("Location") or [])]
|
||||||
|
libtype = 1 if stype == "movie" else 4 # movie | episode
|
||||||
|
|
||||||
|
n = 0
|
||||||
|
size = 0
|
||||||
|
parts = 0
|
||||||
|
oldest_add = None
|
||||||
|
multi_version = 0
|
||||||
|
shows = set()
|
||||||
|
seasons = set()
|
||||||
|
|
||||||
|
try:
|
||||||
|
for row in plex.items(key, libtype):
|
||||||
|
sz, pc = item_size(row)
|
||||||
|
n += 1
|
||||||
|
size += sz
|
||||||
|
parts += pc
|
||||||
|
if pc > 1:
|
||||||
|
multi_version += 1
|
||||||
|
added = int(row.get("addedAt") or 0)
|
||||||
|
if added and (oldest_add is None or added < oldest_add):
|
||||||
|
oldest_add = added
|
||||||
|
if stype == "show":
|
||||||
|
shows.add(row.get("grandparentRatingKey"))
|
||||||
|
seasons.add(row.get("parentRatingKey"))
|
||||||
|
|
||||||
|
for p in item_paths(row):
|
||||||
|
# bucket by the first two path components, e.g. /mnt/vault2
|
||||||
|
bits = p.split("/")
|
||||||
|
root = "/".join(bits[:3]) if len(bits) > 3 else p
|
||||||
|
root_bytes[root] += sz if pc == 1 else sz // max(pc, 1)
|
||||||
|
|
||||||
|
if args.dump is not None:
|
||||||
|
inventory.append({
|
||||||
|
"section": s.get("title"),
|
||||||
|
"type": stype,
|
||||||
|
"ratingKey": row.get("ratingKey"),
|
||||||
|
"title": row.get("title"),
|
||||||
|
"grandparentTitle": row.get("grandparentTitle"),
|
||||||
|
"parentIndex": row.get("parentIndex"),
|
||||||
|
"year": row.get("year"),
|
||||||
|
"addedAt": added,
|
||||||
|
"viewCount": int(row.get("viewCount") or 0),
|
||||||
|
"lastViewedAt": int(row.get("lastViewedAt") or 0) or None,
|
||||||
|
"size": sz,
|
||||||
|
"parts": pc,
|
||||||
|
"paths": item_paths(row),
|
||||||
|
})
|
||||||
|
except Fail as e:
|
||||||
|
print("\n%-24s ERROR: %s" % (s.get("title"), e))
|
||||||
|
continue
|
||||||
|
|
||||||
|
grand_total += size
|
||||||
|
grand_count += n
|
||||||
|
|
||||||
|
print("\n%s [%s]" % (s.get("title"), stype))
|
||||||
|
if stype == "movie":
|
||||||
|
print(" movies : %s" % f"{n:,}")
|
||||||
|
else:
|
||||||
|
print(" episodes : %s" % f"{n:,}")
|
||||||
|
print(" seasons : %s (MediaShelf's unit of analysis)"
|
||||||
|
% f"{len(seasons):,}")
|
||||||
|
print(" shows : %s" % f"{len(shows):,}")
|
||||||
|
print(" total size : %s" % human(size))
|
||||||
|
print(" files (parts) : %s" % f"{parts:,}")
|
||||||
|
if multi_version:
|
||||||
|
print(" multi-file items : %s <- these are the ones a naive scan undercounts"
|
||||||
|
% f"{multi_version:,}")
|
||||||
|
print(" oldest addedAt : %s" % ymd(oldest_add))
|
||||||
|
for loc in locs:
|
||||||
|
print(" root : %s" % loc)
|
||||||
|
|
||||||
|
section("Totals")
|
||||||
|
print("Items scanned : %s" % f"{grand_count:,}")
|
||||||
|
print("Total size : %s" % human(grand_total))
|
||||||
|
if root_bytes:
|
||||||
|
print("\nApproximate size by path root:")
|
||||||
|
for root, b in sorted(root_bytes.items(), key=lambda kv: -kv[1]):
|
||||||
|
print(" %-40s %s" % (root[:40], human(b)))
|
||||||
|
print("\n-> If these map to separate arrays, MediaShelf can group by vault,")
|
||||||
|
print(" which matters when the goal is freeing one specific array.")
|
||||||
|
|
||||||
|
# ── a first taste of the actual output ─────────────────────────────
|
||||||
|
if args.dump is not None and inventory:
|
||||||
|
with open(args.dump, "w") as fh:
|
||||||
|
json.dump(inventory, fh)
|
||||||
|
print("\nWrote %s items to %s" % (f"{len(inventory):,}", args.dump))
|
||||||
|
|
||||||
|
section("Preview · 20 largest items Plex thinks were never watched")
|
||||||
|
print("(Plex's viewCount is token-scoped, so treat this as indicative only —")
|
||||||
|
print(" the real report uses Tautulli. This is a sanity check on sizes.)\n")
|
||||||
|
never = [i for i in inventory if not i["viewCount"]]
|
||||||
|
never.sort(key=lambda i: -i["size"])
|
||||||
|
for i in never[:20]:
|
||||||
|
label = i["title"]
|
||||||
|
if i.get("grandparentTitle"):
|
||||||
|
label = "%s — S%s %s" % (i["grandparentTitle"], i.get("parentIndex"), i["title"])
|
||||||
|
print(" %10s added %s %s" % (human(i["size"]), ymd(i["addedAt"]), label[:60]))
|
||||||
|
reclaimable = sum(i["size"] for i in never)
|
||||||
|
print("\n %s across %s items Plex records no views for."
|
||||||
|
% (human(reclaimable), f"{len(never):,}"))
|
||||||
|
|
||||||
|
section("Done")
|
||||||
|
print("Nothing was modified. Paste this output back to review the design against it.")
|
||||||
|
if coverage_since:
|
||||||
|
print("Key number: Tautulli coverage begins %s." % ymd(coverage_since))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
sys.exit("\nInterrupted.")
|
||||||
|
except Fail as e:
|
||||||
|
sys.exit("Failed: %s" % e)
|
||||||
Loading…
Add table
Add a link
Reference in a new issue