Add inline comments and update README for MusicGPT workflow

README.md

@@ -14,13 +14,14 @@ The MusicGPT fork automates song generation: instead of manually creating songs
 2. [Customer-facing pages](#customer-facing-pages)
 3. [Operator / admin pages](#operator--admin-pages)
 4. [Status flow](#status-flow)
-5. [Settings page explained](#settings-page-explained)
+5. [MusicGPT workflow](#musicgpt-workflow)
-6. [Docker installation](#docker-installation)
+6. [Settings page explained](#settings-page-explained)
-7. [Environment variables](#environment-variables)
+7. [Docker installation](#docker-installation)
-8. [File layout](#file-layout)
+8. [Environment variables](#environment-variables)
-9. [Local development](#local-development)
+9. [File layout](#file-layout)
-10. [Common troubleshooting](#common-troubleshooting)
+10. [Local development](#local-development)
-11. [License / ownership](#license--ownership)
+11. [Common troubleshooting](#common-troubleshooting)
+12. [License / ownership](#license--ownership)
 
 ---
 
@@ -29,7 +30,7 @@ The MusicGPT fork automates song generation: instead of manually creating songs
 1. A visitor fills out a short form at `/request`.
 2. The operator reviews the request in the admin dashboard and generates a prompt for MusicGPT (manually or via the `/api/prompt` Hermes callback).
 3. The operator saves the title, style, and lyrics to the request.
-4. The operator clicks **Generate A/B with MusicGPT**, selects a model, and waits for the webhook to complete. MusicGPT produces two MP3 versions, WAVs, and an album cover.
+4. The operator clicks **Generate A/B with MusicGPT**, selects a model, and waits for MusicGPT to finish. MusicGPT produces two MP3 versions, optional WAVs, and an album cover.
 5. The operator sends a preview email; the customer visits their private player page, listens to both versions, and either approves one/both or requests changes.
 6. After the customer approves and pays, the operator records the payment reference and delivers the selected MP3(s) — optionally with WAV files and the album cover — by email.
 7. Optional STEMS can be generated for an extra fee and delivered via a share link.
@@ -75,11 +76,11 @@ The selected pronouns are stored in the `pronouns` column and included in confir
 | Page | Path | Purpose |
 |------|------|---------|
 | Login | `/admin/login` | Simple session-based login. Password comes from the `ADMIN_PASSWORD` environment variable. |
-| Dashboard | `/admin` | Main queue. Filter by status and auto-refresh at a configurable interval. |
+| Dashboard | `/admin` | Main queue. Filter by status and auto-refresh at a configurable interval. Includes a **Refresh MusicGPT Status** button to manually poll all in-flight tasks. |
-| Request detail | `/admin/request/<id>` | Full control of one request: edit customer info, save prompt, copy Hermes callback, queue MusicGPT generation, view status/cost, view revision history, upload MP3 overrides, send preview, record payment, deliver files (MP3/WAV/cover), add operator notes, cancel generation, and generate STEMS. |
+| Request detail | `/admin/request/<id>` | Full control of one request: edit customer info, save prompt, copy Hermes callback, queue MusicGPT generation, poll a single MusicGPT task, view status/cost, view revision history, upload MP3 overrides, send preview, record payment, deliver files (MP3/WAV/cover), add operator notes, cancel generation, and generate STEMS. |
 | Pricing | `/admin/pricing` | Configure fixed prices and up to 5 custom items. Also shows running MusicGPT cost totals in USD. |
 | Sales | `/admin/sales` | Report of all delivered requests with customer details and Square payment references. |
-| Settings | `/admin/settings` | Database health, backups, SMTP config, MP3 metadata defaults, revision limit, auto-refresh interval, kiosk mode, booth open/closed switch, Hermes API key management, MusicGPT webhook URL, album cover delivery toggle, and system reset. |
+| Settings | `/admin/settings` | Database health, backups, SMTP config, MP3 metadata defaults, revision limit, auto-refresh interval, kiosk mode, booth open/closed switch, Hermes API key management, MusicGPT webhook URL, album cover delivery toggle, automatic MusicGPT polling toggle, and system reset. |
 | Reset | `/admin/reset` | Clears all requests and uploaded files. Requires admin password confirmation. |
 
 ---
@@ -103,6 +104,56 @@ pending → prompt_ready → songs_uploaded → awaiting_payment → paid → de
 
 ---
 
+## MusicGPT workflow
+
+### Supported models
+
+The available MusicGPT models are limited by the API key. The deployed key supports `v6` and `v6-pro`; `v6-pro` is the default. The model dropdown on `/admin/request/<id>` only shows models returned by the MusicGPT API for the configured key.
+
+### Generating songs
+
+1. On `/admin/request/<id>`, enter or paste the **Title**, **Style**, and **Lyrics**.
+2. Click **Save Prompt**.
+3. Choose a model and whether to include WAV files.
+4. Click **Generate A/B with MusicGPT**.
+
+The app queues two conversions (Version A and Version B) with MusicGPT. It stores the task ID, both conversion IDs, the webhook URL, and an estimated cost.
+
+### Webhook
+
+MusicGPT POSTs to `/api/musicgpt/webhook` when each conversion completes. The handler:
+
+- Matches the payload to the request by `task_id` and `conversion_id`.
+- Updates the request status and cost.
+- Downloads album cover, MP3s, and WAVs when the payload includes URLs.
+- Webhooks often arrive before audio URLs are ready; missing files are filled in by polling.
+
+### Polling
+
+Three ways to poll MusicGPT:
+
+| Method | How | When to use |
+|--------|-----|-------------|
+| Automatic | Background job hits `/admin/musicgpt/autopoll` every 3 minutes | Set-and-forget; enabled by default and can be turned off in `/admin/settings`. |
+| Dashboard refresh | **Refresh MusicGPT Status** button on `/admin` | Manual check of all in-flight tasks. |
+| Per-request | **Poll MusicGPT** button on `/admin/request/<id>` | Force a single request to update now. |
+
+All polling paths call `musicgpt_poll_status()` and run `download_musicgpt_outputs()` when the API reports `COMPLETED`, so MP3s/WAVs arrive without further operator action once they are available.
+
+### Cost tracking
+
+The `/admin/pricing` page shows aggregate MusicGPT costs in USD. The per-request cost is recorded from the webhook/poll payload (`conversion_cost`) when MusicGPT provides it; otherwise the queued estimate is shown. Stems are tracked separately by `stems_cost`.
+
+### WAV delivery
+
+A per-request checkbox **Include WAV files with delivery** stores `deliver_wav=1`. When enabled, WAV files are downloaded alongside MP3s and attached to the delivery email.
+
+### Album cover
+
+A global toggle in `/admin/settings` controls whether the generated album cover is attached to delivery emails. The cover is always downloaded and shown on the admin request page.
+
+---
+
 ## Settings page explained
 
 The `/admin/settings` page is split into functional sections:
@@ -112,8 +163,7 @@ The `/admin/settings` page is split into functional sections:
 
 ### Hermes API key
 - Displays whether a key is configured.
-- **Regenerate API Key** creates a new random key stored in runtime settings.
+- The key is used by the `/api/prompt/<id>` callback.
-- The key is used by the `/api/prompt/<id>` callback and by the `/api/key-test` diagnostic endpoint.
 - Copy this key into your Hermes skill or AI assistant config.
 
 ### Customer revision limit
@@ -134,8 +184,14 @@ The `/admin/settings` page is split into functional sections:
 - **Send Test Email** verifies the configuration.
 
 ### MP3 metadata defaults
-- Artist, album, year, and comment tags applied automatically to uploaded MP3s.
+- Artist, album, year, and comment tags applied automatically to downloaded MP3s.
-- The title tag is taken from the saved Suno prompt.
+- The title tag is taken from the saved Suno/MusicGPT prompt title.
+
+### MusicGPT settings
+- **API key configured** indicator.
+- **Webhook URL** — shown for reference; sent per-conversion with every generation request.
+- **Album cover delivery** — attach cover to delivery emails.
+- **Automatic MusicGPT polling** — enable/disable the background poll that downloads ready files without operator action.
 
 ### Database maintenance
 - **Health Check** — verifies all expected tables and columns exist.
@@ -176,7 +232,7 @@ python init_db.py
 ```
 
 7. Point your reverse proxy at the host port you chose (default `127.0.0.1:8000`).
-8. Visit `/admin/settings`, confirm the MusicGPT webhook URL, and configure the MusicGPT account webhook endpoint.
+8. Visit `/admin/settings`, confirm the MusicGPT webhook URL, and configure the MusicGPT account webhook endpoint to the same URL.
 9. Copy the Hermes API key to your Hermes skill / AI assistant.
 10. Print or display a QR code pointing to `https://your-domain/request`.
 
@@ -196,10 +252,10 @@ Persistent volumes keep the database and uploads safe across redeploys.
 
 | Variable | Required | Default | Description |
 |----------|----------|---------|-------------|
-| `APP_SECRET_KEY` | Yes | — | Long random string for Flask sessions and for encrypting stored settings such as the SMTP password and legacy stored API key. |
+| `APP_SECRET_KEY` | Yes | — | Long random string for Flask sessions and for encrypting stored settings such as the SMTP password. |
 | `ADMIN_PASSWORD` | Yes | — | Password used to log in to `/admin`. |
 | `SMTP_PASS` | Yes | — | Password for the SMTP account used to send customer emails. |
-| `PUBLIC_BASE_URL` | Yes | — | Public HTTPS URL of the booth, e.g. `https://booth.dionysismedia.ca`. Used in player links, emails, and callback URLs. |
+| `PUBLIC_BASE_URL` | Yes | — | Public HTTPS URL of the booth, e.g. `https://music.dionysismedia.ca`. Used in player links, emails, callback URLs, and the MusicGPT webhook URL. |
 | `SMTP_HOST` | No | `mailroot8.namespro.ca` | SMTP server hostname. |
 | `SMTP_PORT` | No | `465` | SMTP server port. |
 | `SMTP_USER` | No | `ai@hallsworth.ca` | SMTP username. |
@@ -221,8 +277,9 @@ Persistent volumes keep the database and uploads safe across redeploys.
 
 | File | Purpose |
 |------|---------|
-| `app.py` | Flask routes, helpers, email layer, runtime settings, MP3 tagging, rate limiting, database maintenance, Hermes callback, kiosk, pricing, and sales report. |
+| `app.py` | Flask routes, runtime settings, email layer, MP3 tagging, rate limiting, database maintenance, Hermes callback, kiosk, pricing, sales report, MusicGPT webhooks, polling, and file downloads. |
 | `config.py` | Environment-variable based configuration with sensible defaults. |
+| `helpers.py` | MusicGPT API client, polling, downloads, email helpers, MP3 metadata, settings persistence, and utility functions. |
 | `models.py` | SQLite schema, CRUD helpers, and revision history. |
 | `init_db.py` | Standalone script to create or migrate the database. |
 | `templates/` | Jinja2 templates for customer pages, admin pages, and kiosk display. |
@@ -242,7 +299,7 @@ cd /home/jess/workspace/booth-musicgpt
 python3 -m venv .venv
 .venv/bin/pip install -r requirements.txt
 cp .env.example .env
-# Edit .env and set APP_SECRET_KEY, ADMIN_PASSWORD, SMTP_PASS, PUBLIC_BASE_URL
+# Edit .env and set APP_SECRET_KEY, ADMIN_PASSWORD, SMTP_PASS, PUBLIC_BASE_URL, MUSICGPT_API_KEY
 .venv/bin/python init_db.py
 .venv/bin/python -m flask --app app run --host=0.0.0.0
 ```
@@ -265,6 +322,8 @@ Visit:
 | Database schema mismatch | New column/table added but old DB not migrated. | Go to `/admin/settings` and click **Fix Database Schema**, or run `python init_db.py`. |
 | Kiosk shows old prices | Page auto-refreshes every 30s; check `/admin/pricing`. | Verify pricing values and redeploy if templates changed. |
 | `/api/prompt` returns 401 | Callback token expired or API key mismatch. | Copy a fresh callback URL from `/admin/request/<id>` and verify the key with `/api/key-test`. |
+| MusicGPT generation fails | API key lacks access to the selected model. | Only `v6` and `v6-pro` are supported by the current key; choose from the dropdown. |
+| Webhook arrives but no files download | MusicGPT webhook fires before audio URLs are ready. | Wait for the automatic background poll, click **Poll MusicGPT** on the request, or use **Refresh MusicGPT Status** on the dashboard. |
 
 ---
 

app.py

@@ -308,6 +308,10 @@ def musicgpt_webhook():
         req = dict(row)
 
         # Determine which version this webhook belongs to.
+        # MusicGPT sends a separate webhook for each conversion_id; version A is the
+        # first conversion queued and version B is the second. We store both IDs on
+        # the request so we can route the payload correctly and decide which file to
+        # overwrite when a later webhook arrives for the same version.
         version = None
         if req.get('musicgpt_conversion_id_1') and conversion_id == req['musicgpt_conversion_id_1']:
             version = 'A'
@@ -381,7 +385,15 @@ def musicgpt_webhook():
 
 @app.route('/admin/musicgpt/autopoll', methods=['POST'])
 def admin_musicgpt_autopoll():
-    """Internal endpoint used by the cron job. Only runs refresh if auto-poll is enabled."""
+    """
+    Internal endpoint used by the cron job for automatic MusicGPT polling.
+
+    Unlike the manual dashboard refresh endpoint, this route first checks the
+    'musicgpt_autopoll' runtime setting. If automatic polling is disabled it
+    returns immediately without touching the MusicGPT API, so operators can
+    turn background automation on/off from /admin/settings without stopping
+    the cron job.
+    """
     redir = require_admin()
     if redir:
         return redir
@@ -392,7 +404,12 @@ def admin_musicgpt_autopoll():
 
 @app.route('/admin/musicgpt/refresh', methods=['POST'])
 def admin_musicgpt_refresh():
-    """Manual dashboard action: poll all in-flight MusicGPT tasks and update statuses."""
+    """
+    Manual dashboard action: poll all in-flight MusicGPT tasks and update statuses.
+
+    This is intentionally separate from /admin/musicgpt/autopoll so operators can
+    always trigger a manual refresh even when automatic polling is disabled.
+    """
     redir = require_admin()
     if redir:
         return redir
@@ -423,7 +440,7 @@ def admin_musicgpt_refresh():
                 update_request(row['id'], **fields)
             updated += 1
         elif status in ('FAILED', 'ERROR'):
-            update_request(row['id'], musicgpt_status='FAILED', musicgpt_error=conversion.get('status_msg') or 'Polling reported failure')
+            update_request(row['id'], musicgpt_status='FAILED', musicgpt_error=(conversion.get('status_msg') if isinstance(conversion, dict) else None) or 'Polling reported failure')
             failed += 1
         elif status:
             update_request(row['id'], musicgpt_status=status)
@@ -434,7 +451,12 @@ def admin_musicgpt_refresh():
 
 @app.route('/admin/musicgpt/poll/<int:rid>', methods=['POST'])
 def admin_musicgpt_poll_request(rid):
-    """Poll a single MusicGPT task from the admin request page and download files if ready."""
+    """
+    Poll a single MusicGPT task from the admin request page and download files if ready.
+
+    This gives operators a way to force-update one request without waiting for the
+    dashboard refresh or the automatic background poll.
+    """
     redir = require_admin()
     if redir:
         return redir

config.py

@@ -13,6 +13,7 @@ Environment variables (defaults shown):
     - ADMIN_PASSWORD      plain-text password for /admin login
     - PUBLIC_BASE_URL     public URL customers use (e.g. https://booth.example.com)
     - SMTP_PASS           password for the SMTP account
+    - MUSICGPT_API_KEY    API key for MusicGPT generation and polling
   Optional:
     - BOOTH_NAME          name in customer text and emails (default Trollgorithm Theme Songs)
     - HOST_PORT           docker-compose host-side port mapping (default 127.0.0.1:8000)
@@ -27,6 +28,9 @@ Environment variables (defaults shown):
     - SMTP_PORT           outgoing mail server port (default 465)
     - SMTP_USER           SMTP login username (default ai@hallsworth.ca)
     - SMTP_FROM           From address for customer emails (default ai@hallsworth.ca)
+    - HERMES_API_KEY      API key for the /api/prompt callback (optional, can be set in settings)
+    - MUSICGPT_DEFAULT_MODEL  default model selected in the admin dropdown (default v6-pro)
+    - MUSICGPT_MODELS     comma-separated list of models available to the API key (default v6,v6-pro)
 """
 
 import os
@@ -101,7 +105,8 @@ class Config:
     # MusicGPT API key (env only; never stored in repo).
     MUSICGPT_API_KEY = os.environ.get('MUSICGPT_API_KEY', '')
 
-    # Available MusicGPT generation models.
+    # Available MusicGPT generation models. The deployed API key only supports
+    # v6 and v6-pro, so we keep the list explicit rather than making it dynamic.
     MUSICGPT_MODELS = ['v6', 'v6-pro']
     MUSICGPT_DEFAULT_MODEL = os.environ.get('MUSICGPT_DEFAULT_MODEL', 'v6-pro')
 

helpers.py

@@ -342,19 +342,19 @@ def get_musicgpt_api_key():
 
 
 def get_musicgpt_default_model():
-    """Return the configured default MusicGPT model."""
+    """Return the configured default MusicGPT model, falling back to a supported model."""
     default = current_app.config.get('MUSICGPT_DEFAULT_MODEL', 'v6-pro')
     models = get_musicgpt_models()
     return default if default in models else models[-1]
 
 
 def get_musicgpt_models():
-    """Return the list of supported MusicGPT models."""
+    """Return the list of supported MusicGPT models (filtered by API key capability)."""
     return list(current_app.config.get('MUSICGPT_MODELS', ['v6', 'v6-pro']))
 
 
 def build_musicgpt_webhook_url():
-    """Build the public webhook URL for MusicGPT async callbacks."""
+    """Build the public webhook URL for MusicGPT async callbacks from PUBLIC_BASE_URL."""
     base = current_app.config.get('PUBLIC_BASE_URL', '').rstrip('/')
     return f"{base}/api/musicgpt/webhook"
 
@@ -539,6 +539,11 @@ def set_musicgpt_autopoll_enabled(enabled):
     cfg['musicgpt_autopoll'] = bool(enabled)
     save_booth_settings(cfg)
 
+
+# ---------------------------------------------------------------------------
+# MusicGPT API client
+# ---------------------------------------------------------------------------
+
 MUSICGPT_API_BASE = "https://api.musicgpt.com/api/public"
 
 
@@ -587,6 +592,9 @@ def musicgpt_generate_request(rid, title, music_style, lyrics, gender=None, mode
 def musicgpt_poll_status(task_id):
     """
     Poll the MusicGPT API for a generation task status.
+
+    MusicGPT's /v1/byId endpoint returns the whole task record, including the
+    'conversion' object with the current status and any available audio URLs.
     Returns a dict with keys: status, message, conversion, or error.
     """
     url = f"{MUSICGPT_API_BASE}/v1/byId"
@@ -640,6 +648,12 @@ def _download_file(url, dest):
 def download_musicgpt_outputs(req, data, version=None):
     """
     Download the MP3 and WAV outputs from a completed MusicGPT webhook/poll payload.
+
+    MusicGPT returns audio URLs under different keys depending on whether the
+    payload is for one conversion or the combined task result. This function
+    normalizes those keys, downloads each available file to the request upload
+    directory, applies MP3 metadata tags, and updates the request row.
+
     Updates the request row with local paths and returns a dict of saved paths.
     `data` is the conversion dict from the API. For per-conversion webhooks,
     pass `version='A' or 'B'`. For combined payloads, version is auto-detected
@@ -650,7 +664,7 @@ def download_musicgpt_outputs(req, data, version=None):
     song_title = req.get("suno_title") or req.get("title")
     saved = {}
 
-    # Map version label to field names.
+    # Map version label to the database column names used to store local paths.
     def _fields(v):
         return ("song_a_path", "song_a_wav_path") if v == "A" else ("song_b_path", "song_b_wav_path")
 
@@ -674,6 +688,9 @@ def download_musicgpt_outputs(req, data, version=None):
         url_key = "conversion_path_1" if v == "A" else "conversion_path_2"
         wav_key = "conversion_path_wav_1" if v == "A" else "conversion_path_wav_2"
         # Also support per-conversion webhook keys without _1/_2 suffix.
+        # When MusicGPT sends one webhook per conversion it usually uses the plain
+        # 'conversion_path' / 'conversion_path_wav' keys; we still need to know
+        # whether to treat it as version A or B. The caller-provided version helps.
         if not (data.get(url_key) or data.get(wav_key)):
             if v == "A" or version == "A":
                 mp3_url = data.get("conversion_path")

models.py

@@ -10,6 +10,8 @@ Schema overview (see SCHEMA constant):
 - requests table stores customer data, generated prompts, file paths,
   approval state, email timestamps, payment reference, player token,
   vocal gender preference, revision count, revision note, and operator notes.
+- MusicGPT integration fields store task/conversion IDs, status, cost, error,
+  album cover URL, WAV paths, and stems task/cost/URL.
 - Revisions: when a customer requests changes, the current A/B MP3 files are
   renamed to archived "RevN-" copies and new versions are uploaded later.
 - operator_notes is an internal column for the booth team and is never shown
@@ -54,21 +56,21 @@ CREATE TABLE IF NOT EXISTS requests (
     operator_notes TEXT,
     stems_link TEXT,
     stems_interest INTEGER DEFAULT 0,
-    musicgpt_task_id TEXT,
+    musicgpt_task_id TEXT,           -- MusicGPT Music AI task ID shared by Version A and B
-    musicgpt_conversion_id_1 TEXT,
+    musicgpt_conversion_id_1 TEXT,   -- conversion ID for Version A
-    musicgpt_conversion_id_2 TEXT,
+    musicgpt_conversion_id_2 TEXT,   -- conversion ID for Version B
-    musicgpt_status TEXT,
+    musicgpt_status TEXT,            -- IN_QUEUE / IN_PROGRESS / COMPLETED / FAILED
-    musicgpt_cost REAL,
+    musicgpt_cost REAL,              -- API-reported cost in USD credits
-    musicgpt_error TEXT,
+    musicgpt_error TEXT,             -- error message from MusicGPT or polling
-    album_cover_url TEXT,
+    album_cover_url TEXT,            -- URL to generated album cover image
-    song_a_wav_path TEXT,
+    song_a_wav_path TEXT,            -- local path to Version A WAV (if deliver_wav)
-    song_b_wav_path TEXT,
+    song_b_wav_path TEXT,            -- local path to Version B WAV (if deliver_wav)
-    deliver_wav INTEGER DEFAULT 0,
+    deliver_wav INTEGER DEFAULT 0,     -- 1 if WAV files should be delivered with MP3s
-    stems_task_id TEXT,
+    stems_task_id TEXT,              -- MusicGPT Extraction task ID for optional stems
-    stems_status TEXT,
+    stems_status TEXT,               -- status of the Extraction job
-    stems_cost REAL,
+    stems_cost REAL,                 -- API-reported stems cost in USD credits
-    stems_url TEXT,
+    stems_url TEXT,                  -- download / share link for stems
-    stems_error TEXT
+    stems_error TEXT                 -- error message from stems job
 );
 
 CREATE INDEX IF NOT EXISTS idx_requests_status ON requests(status);