Files
Psychotoxical-psysonic/src-tauri/src/cover_cache/backfill_worker.rs
T
Psychotoxical b950d4704b feat(cover): artist artwork from fanart.tv (off by default) (#1137)
* feat(cover): add artist_artwork_lookup table + accessors

Image-scraper P0 (design-review §12): additive library-SQLite migration 013
plus get/upsert/clear-per-server accessors for the external artist-artwork
lookup (fanart.tv). Render never reads it; the on-demand cover ensure path
and the mbid_ambiguous 24h negative cache use it. server_id = serverIndexKey.

* feat(cover): add fanart.tv + getArtistInfo2 provider layer

Image-scraper P0 (§7/§19/§23): new cover_cache/external.rs — Rust-side
getArtistInfo2 tag-MBID resolution plus fanart.tv v3/music URL + first
artistbackground fetch (BYOK client_key sent in addition to the project
api_key per fanart.tv ToS, §22). Extract a shared build_subsonic_url helper
in fetch.rs (cover URL behaviour unchanged). Add a dedicated low-concurrency
fanart_http_sem so external HTTP never starves Navidrome (§26). URL builders
unit-tested; wired into ensure_inner next.

* feat(cover): wire fanart.tv external branch into ensure_inner

Image-scraper P0 (§16): on-demand artist `fanart` ensures try fanart.tv
before the Navidrome fallback. MBID resolved Rust-side via getArtistInfo2
(§23, tag MBID); on a miss it falls through WITHOUT a .fetch-failed marker so
Navidrome stays the display fallback (§28). External tiers are written as
{tier}-fanart.webp in the same entity dir (same cacheKind, §16) — 2000 + 512
(matryoshka §17); peek prefers them for the fanart surface. Dedicated
low-concurrency fanart lane (§26); .miss-fanart ~30min negative marker.
Additive IPC args (externalArtworkEnabled, surfaceKind), off by default and
gated by PSYSONIC_FANART_KEY — inert until a render surface opts in (P1).
Quality gate (§11), name->MusicBrainz (§19), and lookup-table writes are P1.

* feat(cover): fanart-first peek for the fanart surface

Image-scraper P0: for an artist `fanart` ensure, the early peek serves only
the external {tier}-fanart.webp tiers; if none exist yet it returns None so
ensure runs the external branch (fetch fanart) instead of short-circuiting on
a cached Navidrome tier. Realises "fanart prioritised" (§18) for the opt-in
surface; Navidrome stays the fallback inside the branch's miss path.

* chore(cover): dev-only artist-fanart spike helper

DEV-only window.psyFanartSpike(name) — resolves an artist by name and fires
the real cover_cache_ensure with externalArtworkEnabled+surfaceKind=fanart to
verify the P0 pipeline against a live server (with PSYSONIC_FANART_KEY set).
Not wired in production.

* feat(cover): §11 quality gate for the fanart surface

Before an external fanart fetch, check whether a Navidrome tier already on
disk is an HQ ~16:9 image (width >= 1280, aspect 1.6-2.0) and skip the fetch
if so — square artist portraits never satisfy it, so the common case still
fetches. Reads tier dimensions only (no full decode). Rust consts per the
design review. Unit-tested predicate.

* feat(cover): persist fanart resolution in artist_artwork_lookup (§12)

Wire the lookup table as both the MBID resolution cache and the negative
cache: a cached MBID skips the getArtistInfo2 round-trip; no_mbid/mbid_ambiguous
back off 24h and miss 30min from updated_at before re-querying. Writes
hit/miss/no_mbid with mbid/mbid_source/provider; transient network errors are
not cached. All store reads/writes run off the async executor via
spawn_blocking and no-op before login. Store reached via app.try_state::<LibraryRuntime>().

* feat(cover): compile-time fanart key fallback + album/name IPC args

A runtime PSYSONIC_FANART_KEY still wins (dev), else the key baked in at build
time via option_env! (release). Add additive artistName/albumTitle ensure args
as context for the §19 name->MusicBrainz fallback (inert until the render
passes them). Library backfill passes None.

* feat(settings): add External Artwork Scraper toggle under Integrations

Master toggle (themeStore, off by default per §20) in a new Integrations
subsection, alongside the other opt-in third-party categories. Contacts
fanart.tv only when enabled. i18n across all 9 locales.

* feat(cover): wire fanart background into the fullscreen player (§28)

New useArtistFanart hook resolves a fanart.tv 16:9 background via a dedicated
cover_cache_ensure (surfaceKind=fanart) — it bypasses the shared peek/disk-src
cache (the {tier}-fanart.webp surface is keyed differently) and reuses
coverDiskUrl for the asset URL. Fullscreen background priority is now
fanart -> Navidrome artist image (cover pipeline) -> album cover; the live
useFsArtistPortrait probe is deleted (§28). Additive ensure opts
(surfaceKind/artistName/albumTitle); externalArtworkEnabled is derived in
ensureArgsFromRef from the master toggle and restricted to the artist fanart
surface, so plain cover ensures are unaffected.

* feat(cover): generalize external surface to fanart + banner (§13)

surfaceKind='banner' fetches the fanart.tv musicbanner array -> {tier}-banner.webp
in the same entity dir; fanart stays the 16:9 artistbackground. The ensure
branch, peek, lookup rows (per-surface surface_kind), miss marker
(.miss-<surface>) and tier suffix are all surface-parameterised. The §11
quality gate stays fanart-only (the banner strip has its own aspect). Unit
test for the surface->fanart JSON key map.

* feat(artist): fanart banner on the artist-detail header (§13, Option B)

The artist-detail header gets an album-detail-style background layer: fanart.tv
banner (musicbanner) -> the 16:9 fanart background cropped to the strip ->
empty. Both via a shared useArtistExternalImage hook (useArtistBanner /
useArtistFanart); each fetches on demand and shares the Rust cache, so the
header and the fullscreen player warm each other's images. The header is its
own stacking context (isolation) so a z-index:-1 banner clips behind the avatar
+ meta with no content wrapper; the album-style framing (padding/radius/clip)
is applied only when a banner is shown, so the off-by-default case stays
pixel-identical.

* refactor(artist): reuse album-detail header structure for the fanart banner

The artist-detail header now uses the same album-detail-* container classes as
AlbumHeader (header/bg/overlay/content/hero) with the fanart banner as the
background; the Back button moves inside the header. A surgical
`artist-detail-bleed` cancels the artist page's .content-body padding so the
banner is full-bleed to the container edges, matching the album header exactly
instead of the earlier inset card. Reverts the experimental artist-specific bg
CSS.

* feat(cover): drop artist_artwork_lookup rows on clear-cover-cache (§12/B.4)

cover_cache_clear_server already removed the server's whole cover dir (so the
{tier}-fanart.webp / -banner.webp tiers + .miss-* markers go with it); also
clear the artist_artwork_lookup rows for that server (off-thread) so no stale
resolution state lingers. Automatic toggle-off purge deferred — turning the
toggle off already hides external artwork (render is gated), and explicit
cache-clear now cleans external state too.

* feat(cover): name->MusicBrainz album-confirmed MBID resolution (§19)

When getArtistInfo2 has no tag MBID and the ensure carries the artist name +
an album in context (fullscreen), one MusicBrainz release-search query resolves
the artist MBID: the primary artist across score>=90 releases wins, conflicting
ids -> mbid_ambiguous (24h backoff), none -> no_mbid. Sends the required
User-Agent; a single-permit musicbrainz_sem + >=1s spacing holds us under MB's
rate limit. mbid_source=musicbrainz persisted. Banner surface (no album
context) correctly skips this. Pure classify/escape helpers unit-tested.

* fix(cover): enable banner surface in ensureArgsFromRef

externalEnsureFields only set externalArtworkEnabled for surfaceKind 'fanart',
so the 'banner' surface never fired — the artist-detail header always fell back
to the fanart image instead of the fanart.tv musicbanner. Both external artist
surfaces (fanart/banner) now enable the external branch.

* feat(settings): optional BYOK personal fanart key field

Add an optional personal fanart.tv API key field to the External Artwork
Scraper block (shown when the toggle is on): a masked input, a saved/in-use
status line, and the simple note that it is sent in addition to the app key.
Persisted in themeStore and plumbed through cover_cache_ensure
(externalArtworkByok); Rust prefers the settings key, falling back to the
PSYSONIC_FANART_CLIENT_KEY dev env. i18n x9.

* fix(cover): resolve artist-page fanart image collision on navigation

The artist-detail header keyed its fanart/banner hooks on the route `id`,
which flips immediately on navigation while `artist`/`albums` refetch a beat
later. The mismatched ensure wrote the previous artist's image under the new
artist's id (e.g. Sepultura's image under Lordi's id).

- key on the loaded `artist.id`, not the route `id`, so id/name/album always
  describe the same artist
- pick the §19 album context from an album that actually belongs to this
  artist (`albums.find(a => a.artistId === artist.id)`), so a stale album can't
  run a mismatched name→MusicBrainz query or cache a wrong `no_mbid`
- reset `src` on every input change in `useArtistExternalImage` so a previous
  artist's image never lingers while the new one resolves

* fix(cover): strip trailing album qualifier before MusicBrainz lookup

Library titles like "Show No Mercy (2004 Remastered)" or "Album [Deluxe
Edition]" failed the §19 MusicBrainz release query, blocking name-confirmed
MBID resolution. `normalize_album_for_mb` strips a single trailing
parenthetical/bracketed qualifier; leading qualifiers (e.g. "(What's the
Story) Morning Glory?") are left intact. Unit-tested.

* fix(cover): don't cache no_mbid when album context is unavailable

The banner ensure could fire before the artist's albums loaded, with no album
in context. The old code cached `no_mbid` there and the 24h backoff then
blocked the later ensure that arrived *with* album context. Could-not-attempt
is not tried-and-failed: the no-album branch now returns without persisting.

* fix(cover): don't emit tier-ready for external fanart/banner surfaces

`try_external_fanart` emitted `cover:tier-ready` with the `{tier}-{surface}.webp`
path. That event is keyed by the canonical cover key (cacheKind/cacheEntityId/
tier, no surface), so the frontend `useCoverArtBridge` listener seeded the
Navidrome artist cover's disk-src cache with the external image — leaking
fanart/banner into the plain artist cover (avatar, fullscreen "navidrome-artist"
fallback) even with the scraper toggled off.

Remove the emit: the fanart/banner hooks read the path from the
`cover_cache_ensure` return value, so no event is needed. (No disk-level
overwrite — the suffixed files are never matched by `tier_exists`; this was
frontend disk-src-cache cross-contamination.)

* fix(cover): wait for the final external background before showing it, with fade-in

The fullscreen player and artist-detail header flashed several backgrounds in
sequence while the fanart resolved (upscaled album cover → Navidrome artist
image → fanart), and the artist header could show the fanart first and then
swap to the banner.

- the album cover is no longer a background source — it only feeds the
  foreground thumbnail
- the external-artwork hooks return `{ src, pending }` so callers can tell
  "still resolving" (hold back) from "resolved, no image" (fall back now)
- fullscreen background: scraper on → fanart, empty while it resolves, Navidrome
  artist image only on a confirmed miss; scraper off → Navidrome artist image
- artist header: the banner is preferred — nothing shows while it resolves
  (no fanart flash), fanart is the fallback only once the banner misses
- both backgrounds preload the chosen image and fade it in (`onLoad` plus a
  `ref` `complete` check so an already-cached image, whose load event can fire
  before React attaches the handler, still appears). The header fade is a
  scoped inline opacity so the shared `album-detail-bg` class is untouched.

* ci(release): pass PSYSONIC_FANART_KEY into the macOS + Linux builds

* refactor(cover): extract external-artwork ensure into its own module

Pure code move: the on-demand fanart/banner fetch, the quality gate, the
surface-aware peek and the lookup-table cache move from cover_cache/mod.rs
into cover_cache/external_ensure.rs. Behaviour unchanged; mod.rs 1877 -> 1488.

* chore(cover): remove dev-only fanart spike helper

The real render wiring now exercises the external ensure branch, so the
dev-only window.psyFanartSpike helper is redundant.

* feat(cover): purge external artwork on opt-out (B3)

New cover_cache_purge_external command: when the External Artwork toggle is
turned off, drop every fetched {tier}-{provider}.webp, .miss-{provider}
marker and artist_artwork_lookup row across all configured servers, leaving
the canonical Navidrome covers intact. Opting out now removes the
third-party-sourced data instead of just hiding it (design-review §9/§12/B.4).

* docs: changelog, credits and what's new for artist fanart (PR #1137)
2026-06-20 21:04:21 +02:00

712 lines
27 KiB
Rust

//! Library cover backfill — one background pass per wake (native, not webview timers).
use super::{state, CoverCacheEnsureArgs, CoverCacheState};
use psysonic_library::cover_backfill::{
clear_cover_fetch_failures, collect_cover_progress, count_distinct_cover_ids,
diff_missing_against_snapshot, fetch_all_catalog_rows, snapshot_cover_disk,
LIBRARY_COVER_CANONICAL_TIER,
};
use psysonic_library::payload::LibrarySyncProgressPayload;
use psysonic_library::repos::sync_state::SyncStateRepository;
use psysonic_library::LibraryRuntime;
use serde::{Deserialize, Serialize};
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use tauri::{AppHandle, Emitter, Listener, Manager};
use tokio::sync::{Mutex, Semaphore};
use super::{count_cached_cover_ids, dir_usage_for_server};
/// Default concurrent library downloads + encodes. Runtime-tunable via the
/// perf probe (`set_parallel`); the constant is only the startup value.
const LIBRARY_BACKFILL_PARALLEL_DEFAULT: usize = 2;
/// Bounds for the runtime knob — keep it sane so a stray value cannot DoS the
/// host or starve the audio path.
pub const LIBRARY_BACKFILL_PARALLEL_MIN: usize = 1;
pub const LIBRARY_BACKFILL_PARALLEL_MAX: usize = 16;
/// Raw catalog rows diffed per streaming chunk. Small enough that downloads
/// start almost immediately after the one-shot enumeration, large enough to
/// amortize the per-chunk `spawn_blocking` hop.
const SCAN_CHUNK_ROWS: usize = 512;
const SYNC_WAIT_MS: u64 = 5000;
/// Cadence of the in-pass progress ticker (drives the "offline & cache" menu and
/// the perf-probe overlay while a pass downloads). Only runs for the duration of
/// an active pass.
const PROGRESS_TICK_SECS: u64 = 3;
/// Minimum gap between `library:sync-idle`-driven passes. Each such pass runs the
/// idle-gate signature (a full cover-dir walk + DB count), so a chatty sync (e.g.
/// periodic delta syncs) must not make that walk fire every few seconds. Manual
/// runs and strategy-toggle wakes bypass this.
const SYNC_IDLE_COOLDOWN_MS: u64 = 60_000;
fn now_ms() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0)
}
#[derive(Clone)]
pub struct CoverBackfillSession {
pub server_index_key: String,
pub library_server_id: String,
pub username: String,
pub password: String,
}
/// Catalog signature captured when a full pass completes.
///
/// While it still matches, `library:sync-idle` must NOT re-trigger a rescan —
/// mirrors the analysis coordinator's `completed_total` gate. Deliberately the
/// **cheap** `COUNT(DISTINCT)` over the catalog only: a server change (track
/// add/remove shifts `total`) re-arms the next pass, but checking it never
/// touches the filesystem. A cover-cache clear leaves `total` unchanged, so the
/// clear commands re-arm the gate explicitly (`rearm_idle_gate`).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct CoverIdleSignature {
total: i64,
}
pub struct CoverBackfillWorker {
pub enabled: AtomicBool,
/// When true, the active pass yields so visible-route cover IPC is not starved.
pub ui_priority_hold: AtomicBool,
session: Mutex<Option<CoverBackfillSession>>,
cursor: Mutex<String>,
pass_running: AtomicBool,
backfill_http: Arc<Semaphore>,
/// Live download/encode concurrency for backfill passes. Mirrors the
/// `backfill_http` permit count and gates per-batch `ensure_one` tasks.
parallel: AtomicUsize,
/// Set when a pass found nothing pending; suppresses idle-driven rescans
/// until the catalog signature changes. `None` means "re-armed".
settled: Mutex<Option<CoverIdleSignature>>,
/// Epoch-ms of the last `sync-idle`-driven pass, to rate-limit the idle-gate
/// disk walk against chatty syncs. 0 = never.
last_sync_idle_ms: AtomicU64,
/// Live connect URL, resolved fresh per cover fetch rather than baked into
/// the worklist. The worklist holds URL-agnostic items; a LAN→public flip
/// just swaps this cell, so even the pass already in flight downloads its
/// remaining covers against the now-reachable endpoint.
base_url: std::sync::Mutex<String>,
/// A forced retry requested while a pass was already running (e.g. the
/// connect URL flipped LAN→public at boot). The in-flight pass already
/// adopts the new URL live, but the handful of covers it attempted against
/// the stale address need one more forced pass once it finishes.
rerun_pending: AtomicBool,
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct CoverBackfillPulseDto {
pub scheduled: u32,
pub exhausted: bool,
pub pending: i64,
pub done: i64,
pub total: i64,
pub status: String,
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct CoverBackfillRunDto {
pub started: bool,
}
#[derive(Debug, Deserialize)]
#[serde(rename_all = "camelCase")]
struct SyncIdlePayload {
server_id: String,
ok: bool,
}
impl CoverBackfillWorker {
pub fn new() -> Self {
Self {
enabled: AtomicBool::new(false),
ui_priority_hold: AtomicBool::new(false),
session: Mutex::new(None),
cursor: Mutex::new(String::new()),
pass_running: AtomicBool::new(false),
backfill_http: Arc::new(Semaphore::new(LIBRARY_BACKFILL_PARALLEL_DEFAULT)),
parallel: AtomicUsize::new(LIBRARY_BACKFILL_PARALLEL_DEFAULT),
settled: Mutex::new(None),
last_sync_idle_ms: AtomicU64::new(0),
base_url: std::sync::Mutex::new(String::new()),
rerun_pending: AtomicBool::new(false),
}
}
pub fn set_ui_priority_hold(&self, hold: bool) {
self.ui_priority_hold.store(hold, Ordering::Relaxed);
}
/// Re-arm the idle gate so the next opportunistic pass runs even though the
/// catalog `total` is unchanged — used after a cover-cache clear, which
/// drops files the cheap signature cannot see.
pub async fn rearm_idle_gate(&self) {
*self.settled.lock().await = None;
self.last_sync_idle_ms.store(0, Ordering::Relaxed);
}
/// Current backfill download/encode concurrency.
pub fn parallel(&self) -> usize {
self.parallel.load(Ordering::Relaxed).max(LIBRARY_BACKFILL_PARALLEL_MIN)
}
/// Retune backfill concurrency at runtime. Resizes the shared HTTP permit
/// pool to match (next batch picks up the new per-batch slot count). Returns
/// the clamped value actually applied.
pub fn set_parallel(&self, threads: usize) -> usize {
let next = threads.clamp(LIBRARY_BACKFILL_PARALLEL_MIN, LIBRARY_BACKFILL_PARALLEL_MAX);
let prev = self.parallel.swap(next, Ordering::SeqCst);
if next > prev {
self.backfill_http.add_permits(next - prev);
} else if next < prev {
// Shrinking: drain surplus permits as they free up so in-flight
// fetches finish but no new ones start beyond the new cap.
let sem = self.backfill_http.clone();
let surplus = prev - next;
tauri::async_runtime::spawn(async move {
for _ in 0..surplus {
if let Ok(permit) = sem.acquire().await {
permit.forget();
}
}
});
}
next
}
pub async fn set_session(
&self,
enabled: bool,
session: Option<CoverBackfillSession>,
base_url: String,
) {
self.enabled.store(enabled, Ordering::Relaxed);
*self.session.lock().await = session;
*self.base_url.lock().unwrap() = base_url;
// Server switch or enable/disable invalidates any settled state: re-arm
// so the next idle event runs a real pass for the new focus.
*self.settled.lock().await = None;
if !enabled {
*self.cursor.lock().await = String::new();
}
}
/// Current connect URL for backfill fetches. Read fresh per cover so a
/// LAN→public flip is honoured mid-pass without rebuilding the worklist.
pub fn base_url(&self) -> String {
self.base_url.lock().unwrap().clone()
}
/// Swap the live connect URL. Returns `true` when it actually changed, so the
/// caller can clear the now-stale fetch-failed backoff and kick a retry pass.
pub fn set_base_url(&self, url: String) -> bool {
let mut cell = self.base_url.lock().unwrap();
if *cell == url {
return false;
}
*cell = url;
true
}
pub async fn reset_cursor(&self) {
*self.cursor.lock().await = String::new();
}
/// Semaphore-backed library backfill HTTP slots (perf probe).
pub fn pipeline_http_stats(&self) -> (u32, u32, bool) {
let max = self.parallel() as u32;
let active = max.saturating_sub(self.backfill_http.available_permits() as u32);
let pass_running = self.pass_running.load(Ordering::Relaxed);
(max, active, pass_running)
}
}
fn sync_allows_cover_backfill(store: &psysonic_library::store::LibraryStore, server_id: &str) -> bool {
let repo = SyncStateRepository::new(store);
match repo.get_sync_phase(server_id, "") {
Ok(Some(phase)) => phase != "initial_sync" && phase != "probing",
_ => true,
}
}
fn session_matches_server(session: &CoverBackfillSession, server_id: &str) -> bool {
server_id == session.server_index_key || server_id == session.library_server_id
}
/// Backfill runs only while this session is still the configured focus (active
/// server). A connect-URL flip keeps the same `server_index_key` and is picked
/// up live via `worker.base_url()`, so it does not abort the pass — only a
/// server switch or disable does.
async fn session_still_focused(worker: &CoverBackfillWorker, expected: &CoverBackfillSession) -> bool {
if !worker.enabled.load(Ordering::Relaxed) {
return false;
}
worker
.session
.lock()
.await
.as_ref()
.is_some_and(|s| s.server_index_key == expected.server_index_key)
}
async fn progress_snapshot(
store: &psysonic_library::store::LibraryStore,
root: &std::path::Path,
library_server_id: &str,
server_index_key: &str,
) -> Result<(i64, i64, i64), String> {
let cached = count_cached_cover_ids(root, server_index_key);
let p = collect_cover_progress(store, library_server_id, root, server_index_key, cached)?;
Ok((p.done, p.total_distinct, p.pending))
}
async fn emit_library_progress(
app: &AppHandle,
session: &CoverBackfillSession,
done: i64,
total: i64,
pending: i64,
root: &std::path::Path,
) {
let (bytes, entry_count) = dir_usage_for_server(root, &session.server_index_key);
let _ = app.emit(
"cover:library-progress",
serde_json::json!({
"serverIndexKey": session.server_index_key,
"done": done,
"total": total,
"pending": pending,
"bytes": bytes,
"entryCount": entry_count,
}),
);
}
async fn ensure_one(
worker: &CoverBackfillWorker,
st: Arc<tokio::sync::Mutex<CoverCacheState>>,
http_sem: Arc<Semaphore>,
app: AppHandle,
session: CoverBackfillSession,
item: psysonic_library::cover_backfill::CoverBackfillItem,
) {
if worker.ui_priority_hold.load(Ordering::Relaxed) {
return;
}
let args = CoverCacheEnsureArgs {
server_index_key: session.server_index_key,
cache_kind: item.cache_kind,
cache_entity_id: item.cache_entity_id,
cover_art_id: item.fetch_cover_art_id,
tier: LIBRARY_COVER_CANONICAL_TIER,
rest_base_url: worker.base_url(),
username: session.username,
password: session.password,
library_bulk: true,
library_server_id: Some(session.library_server_id),
// Library backfill never touches external providers (§15).
external_artwork_enabled: false,
surface_kind: None,
artist_name: None,
album_title: None,
external_artwork_byok: None,
};
let _ = CoverCacheState::ensure_inner(&st, &app, &args, Some(http_sem)).await;
}
async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>, force: bool) {
if !worker.enabled.load(Ordering::Relaxed) {
return;
}
let session = worker.session.lock().await.clone();
let Some(session) = session else {
return;
};
let runtime = match app.try_state::<LibraryRuntime>() {
Some(r) => r,
None => return,
};
// Opportunistic triggers (wake on track change, sync-idle) skip the whole
// scan when a prior pass already settled and nothing changed — otherwise a
// library with permanently-unfetchable covers (404s) would re-scan on every
// wake forever. The manual "Run full pass now" sets `force` to bypass this.
if !force && cover_idle_gate_should_skip(&app, &worker, &session).await {
return;
}
while !sync_allows_cover_backfill(&runtime.store, &session.library_server_id) {
if !worker.enabled.load(Ordering::Relaxed) {
return;
}
tokio::time::sleep(Duration::from_millis(SYNC_WAIT_MS)).await;
}
let st = match state(&app) {
Ok(s) => s,
Err(_) => return,
};
let root = {
let guard = st.lock().await;
guard.root.clone()
};
let st_arc = st.clone();
worker.reset_cursor().await;
let http_sem = worker.backfill_http.clone();
// A forced pass is an explicit user retry: drop the `.fetch-failed` backoff
// markers and the settled gate so previously-404'd covers are attempted
// again. Opportunistic passes leave the markers in place (30-min TTL).
if force {
*worker.settled.lock().await = None;
let root2 = root.clone();
let index_key2 = session.server_index_key.clone();
let _ = tauri::async_runtime::spawn_blocking(move || {
clear_cover_fetch_failures(&root2, &index_key2)
})
.await;
}
// Two snapshots, taken ONCE per pass: the DB catalog (single GROUP BY) and
// the on-disk cover bucket (one directory walk). The delta = catalog minus
// disk, streamed in chunks below. No per-row `stat` on the filesystem and no
// re-scan loop — pure set math against the captured disk snapshot.
let (raw_rows, snapshot) = {
let store = runtime.store.clone();
let lib_id = session.library_server_id.clone();
let root_for_scan = root.clone();
let index_key = session.server_index_key.clone();
match tauri::async_runtime::spawn_blocking(move || {
let rows = fetch_all_catalog_rows(&store, &lib_id)?;
let snap = snapshot_cover_disk(&root_for_scan, &index_key);
Ok::<_, String>((rows, snap))
})
.await
{
Ok(Ok(pair)) => pair,
_ => (Vec::new(), Default::default()),
}
};
let snapshot = Arc::new(snapshot);
// Producer/consumer: a fixed pool of consumer tasks pulls misses off a
// bounded channel and downloads them continuously, while the producer scans
// the catalog in chunks and feeds misses in. This keeps the pool saturated
// even when misses are sparse across chunks — no per-chunk drain barrier.
// True concurrency stays governed by the resizable `http_sem` / encode
// semaphores inside `ensure_one`, so the threads slider still applies live.
let (tx, rx) =
tokio::sync::mpsc::channel::<psysonic_library::cover_backfill::CoverBackfillItem>(256);
let rx = Arc::new(Mutex::new(rx));
let mut consumers = tokio::task::JoinSet::new();
for _ in 0..LIBRARY_BACKFILL_PARALLEL_MAX {
let rx = rx.clone();
let st = st_arc.clone();
let http_sem = http_sem.clone();
let app = app.clone();
let session = session.clone();
let worker_arc = worker.clone();
consumers.spawn(async move {
loop {
// Bail the moment the strategy flips to lazy / focus changes, so a
// switch to "lazy" abandons the buffered backlog instead of
// draining the whole channel (mirrors the producer's check).
if !session_still_focused(&worker_arc, &session).await {
break;
}
let item = {
let mut guard = rx.lock().await;
guard.recv().await
};
let Some(item) = item else { break };
ensure_one(
worker_arc.as_ref(),
st.clone(),
http_sem.clone(),
app.clone(),
session.clone(),
item,
)
.await;
}
});
}
// Progress ticker: the producer finishes enumerating the worklist long before
// the consumers finish downloading it, so emitting only while feeding would
// freeze the "offline & cache" menu through the whole drain phase. Tick a
// periodic snapshot for the lifetime of the pass instead; aborted once the
// consumers drain (a final accurate emit happens at settle below).
let progress_ticker = {
let app = app.clone();
let store = runtime.store.clone();
let root = root.clone();
let session = session.clone();
let lib_id = session.library_server_id.clone();
let index_key = session.server_index_key.clone();
tauri::async_runtime::spawn(async move {
loop {
tokio::time::sleep(Duration::from_secs(PROGRESS_TICK_SECS)).await;
if let Ok((done, total, pending)) =
progress_snapshot(&store, &root, &lib_id, &index_key).await
{
emit_library_progress(&app, &session, done, total, pending, &root).await;
}
}
})
};
let mut rows_iter = raw_rows.into_iter();
let mut completed = false;
loop {
if !session_still_focused(&worker, &session).await {
break;
}
if worker.ui_priority_hold.load(Ordering::Relaxed) {
tokio::time::sleep(Duration::from_millis(200)).await;
continue;
}
let scan_chunk: Vec<_> = rows_iter.by_ref().take(SCAN_CHUNK_ROWS).collect();
if scan_chunk.is_empty() {
completed = true;
break;
}
// Diff this chunk against the captured snapshot off-thread (in-memory set
// math + DB expand only for rows not already cached) → misses to download.
let missing: Vec<_> = {
let store = runtime.store.clone();
let lib_id = session.library_server_id.clone();
let snapshot = snapshot.clone();
match tauri::async_runtime::spawn_blocking(move || {
diff_missing_against_snapshot(&store, &lib_id, &snapshot, scan_chunk)
})
.await
{
Ok(Ok(missing)) => missing,
_ => Vec::new(),
}
};
// Focus-aware feed: never park indefinitely on a full channel, or a
// switch to lazy (which stops the consumers) would deadlock the producer
// here. `try_send` + a short retry lets us re-check focus and bail.
let mut feed_closed = false;
'feed: for mut item in missing {
loop {
match tx.try_send(item) {
Ok(()) => break,
Err(tokio::sync::mpsc::error::TrySendError::Closed(_)) => {
feed_closed = true;
break 'feed;
}
Err(tokio::sync::mpsc::error::TrySendError::Full(returned)) => {
if !session_still_focused(&worker, &session).await {
feed_closed = true;
break 'feed;
}
item = returned;
tokio::time::sleep(Duration::from_millis(25)).await;
}
}
}
}
if feed_closed {
break;
}
}
// Close the channel so consumers drain the remaining backlog and exit.
drop(tx);
while consumers.join_next().await.is_some() {}
progress_ticker.abort();
// Only settle the idle gate on a natural finish (worklist drained), never on
// a session-switch break — that belongs to the previous focus.
if completed {
worker.cursor.lock().await.clear();
match progress_snapshot(
&runtime.store,
&root,
&session.library_server_id,
&session.server_index_key,
)
.await
{
Ok((done, total, pending)) => {
// Settle on a full scan regardless of `pending`: whatever is left
// is unfetchable for now (404s with a fresh `.fetch-failed`
// marker). The cheap `total` signature re-triggers a pass only if
// the server catalog changes; a cache clear re-arms via the clear
// command. This stops the wake storm on libraries whose covers
// can never reach 100%.
*worker.settled.lock().await = Some(CoverIdleSignature { total });
emit_library_progress(&app, &session, done, total, pending, &root).await;
}
Err(_) => {
*worker.settled.lock().await = None;
}
}
}
}
/// Start one full-catalog pass on the Tokio runtime (survives inactive webview).
/// `force` bypasses the idle gate and clears fetch-failed backoff (explicit user
/// retry); opportunistic callers (wake / sync-idle) pass `false`.
pub async fn try_schedule_full_pass(app: &AppHandle, force: bool) -> bool {
let worker = match app.try_state::<Arc<CoverBackfillWorker>>() {
Some(w) => w.inner().clone(),
None => return false,
};
if !worker.enabled.load(Ordering::Relaxed) {
return false;
}
if worker
.pass_running
.compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst)
.is_err()
{
// A pass is already running. It reads the connect URL live per cover, so
// any flip that landed mid-pass already applies to its remaining work.
// A forced retry (URL flip) still queues a rerun so the few covers the
// in-flight pass attempted against the stale address get re-fetched.
if force {
worker.rerun_pending.store(true, Ordering::SeqCst);
}
return false;
}
let app = app.clone();
tauri::async_runtime::spawn(async move {
run_full_pass(app.clone(), worker.clone(), force).await;
// Drain a forced rerun queued mid-pass (always forced: it bypasses the
// idle gate the just-finished pass re-armed and clears the stale backoff).
loop {
worker.pass_running.store(false, Ordering::SeqCst);
if !worker.rerun_pending.swap(false, Ordering::SeqCst)
|| !worker.enabled.load(Ordering::Relaxed)
{
break;
}
if worker
.pass_running
.compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst)
.is_err()
{
worker.rerun_pending.store(true, Ordering::SeqCst);
break;
}
run_full_pass(app.clone(), worker.clone(), true).await;
}
});
true
}
/// Cheap catalog signature for the idle gate: a single `COUNT(DISTINCT)` over
/// the cover catalog. No filesystem access — checking "did anything change on
/// the server?" must never walk the on-disk cover cache.
async fn current_cover_signature(
app: &AppHandle,
session: &CoverBackfillSession,
) -> Option<CoverIdleSignature> {
let runtime = app.try_state::<LibraryRuntime>()?;
let store = runtime.store.clone();
let lib_id = session.library_server_id.clone();
tauri::async_runtime::spawn_blocking(move || {
count_distinct_cover_ids(&store, &lib_id)
.ok()
.map(|total| CoverIdleSignature { total })
})
.await
.ok()
.flatten()
}
/// True when the previous pass settled with nothing pending and the catalog
/// still matches that signature — so an idle event need not rescan.
async fn cover_idle_gate_should_skip(app: &AppHandle, worker: &CoverBackfillWorker, session: &CoverBackfillSession) -> bool {
let Some(settled) = *worker.settled.lock().await else {
return false;
};
match current_cover_signature(app, session).await {
Some(current) => current == settled,
None => false,
}
}
fn on_sync_idle(app: &AppHandle, payload: SyncIdlePayload) {
if !payload.ok {
return;
}
let app = app.clone();
tauri::async_runtime::spawn(async move {
let worker = match app.try_state::<Arc<CoverBackfillWorker>>() {
Some(w) => w.inner().clone(),
None => return,
};
if !worker.enabled.load(Ordering::Relaxed) {
return;
}
let session = worker.session.lock().await.clone();
let Some(session) = session else {
return;
};
if !session_matches_server(&session, &payload.server_id) {
return;
}
// Rate-limit sync-idle passes: each runs the idle-gate disk walk, so a
// chatty sync must not trigger it every few seconds. The gate inside the
// pass still skips the actual rescan when nothing changed.
let now = now_ms();
let last = worker.last_sync_idle_ms.load(Ordering::Relaxed);
if last != 0 && now.saturating_sub(last) < SYNC_IDLE_COOLDOWN_MS {
return;
}
worker.last_sync_idle_ms.store(now, Ordering::Relaxed);
// Opportunistic: the gate (checked inside the pass) skips the rescan when
// a prior pass settled and nothing changed (mirrors the analysis gate).
let _ = try_schedule_full_pass(&app, false).await;
});
}
/// Listen for library sync completion in native code (not throttled with the webview).
pub fn setup_library_sync_idle_listener(app: &AppHandle) {
let app_handle = app.clone();
let _ = app.listen(LibrarySyncProgressPayload::IDLE_EVENT_NAME, move |event| {
let Ok(payload) = serde_json::from_str::<SyncIdlePayload>(event.payload()) else {
return;
};
on_sync_idle(&app_handle, payload);
});
}
/// Legacy single-step API (optional diagnostics).
pub async fn pulse_backfill(app: &AppHandle, _worker: &Arc<CoverBackfillWorker>) -> CoverBackfillPulseDto {
if try_schedule_full_pass(app, false).await {
return CoverBackfillPulseDto {
scheduled: 0,
exhausted: false,
pending: 0,
done: 0,
total: 0,
status: "active".into(),
};
}
CoverBackfillPulseDto {
scheduled: 0,
exhausted: true,
pending: 0,
done: 0,
total: 0,
status: "disabled".into(),
}
}