mirror of
https://github.com/Psychotoxical/psysonic.git
synced 2026-07-21 23:05:46 +00:00
b950d4704b
* feat(cover): add artist_artwork_lookup table + accessors
Image-scraper P0 (design-review §12): additive library-SQLite migration 013
plus get/upsert/clear-per-server accessors for the external artist-artwork
lookup (fanart.tv). Render never reads it; the on-demand cover ensure path
and the mbid_ambiguous 24h negative cache use it. server_id = serverIndexKey.
* feat(cover): add fanart.tv + getArtistInfo2 provider layer
Image-scraper P0 (§7/§19/§23): new cover_cache/external.rs — Rust-side
getArtistInfo2 tag-MBID resolution plus fanart.tv v3/music URL + first
artistbackground fetch (BYOK client_key sent in addition to the project
api_key per fanart.tv ToS, §22). Extract a shared build_subsonic_url helper
in fetch.rs (cover URL behaviour unchanged). Add a dedicated low-concurrency
fanart_http_sem so external HTTP never starves Navidrome (§26). URL builders
unit-tested; wired into ensure_inner next.
* feat(cover): wire fanart.tv external branch into ensure_inner
Image-scraper P0 (§16): on-demand artist `fanart` ensures try fanart.tv
before the Navidrome fallback. MBID resolved Rust-side via getArtistInfo2
(§23, tag MBID); on a miss it falls through WITHOUT a .fetch-failed marker so
Navidrome stays the display fallback (§28). External tiers are written as
{tier}-fanart.webp in the same entity dir (same cacheKind, §16) — 2000 + 512
(matryoshka §17); peek prefers them for the fanart surface. Dedicated
low-concurrency fanart lane (§26); .miss-fanart ~30min negative marker.
Additive IPC args (externalArtworkEnabled, surfaceKind), off by default and
gated by PSYSONIC_FANART_KEY — inert until a render surface opts in (P1).
Quality gate (§11), name->MusicBrainz (§19), and lookup-table writes are P1.
* feat(cover): fanart-first peek for the fanart surface
Image-scraper P0: for an artist `fanart` ensure, the early peek serves only
the external {tier}-fanart.webp tiers; if none exist yet it returns None so
ensure runs the external branch (fetch fanart) instead of short-circuiting on
a cached Navidrome tier. Realises "fanart prioritised" (§18) for the opt-in
surface; Navidrome stays the fallback inside the branch's miss path.
* chore(cover): dev-only artist-fanart spike helper
DEV-only window.psyFanartSpike(name) — resolves an artist by name and fires
the real cover_cache_ensure with externalArtworkEnabled+surfaceKind=fanart to
verify the P0 pipeline against a live server (with PSYSONIC_FANART_KEY set).
Not wired in production.
* feat(cover): §11 quality gate for the fanart surface
Before an external fanart fetch, check whether a Navidrome tier already on
disk is an HQ ~16:9 image (width >= 1280, aspect 1.6-2.0) and skip the fetch
if so — square artist portraits never satisfy it, so the common case still
fetches. Reads tier dimensions only (no full decode). Rust consts per the
design review. Unit-tested predicate.
* feat(cover): persist fanart resolution in artist_artwork_lookup (§12)
Wire the lookup table as both the MBID resolution cache and the negative
cache: a cached MBID skips the getArtistInfo2 round-trip; no_mbid/mbid_ambiguous
back off 24h and miss 30min from updated_at before re-querying. Writes
hit/miss/no_mbid with mbid/mbid_source/provider; transient network errors are
not cached. All store reads/writes run off the async executor via
spawn_blocking and no-op before login. Store reached via app.try_state::<LibraryRuntime>().
* feat(cover): compile-time fanart key fallback + album/name IPC args
A runtime PSYSONIC_FANART_KEY still wins (dev), else the key baked in at build
time via option_env! (release). Add additive artistName/albumTitle ensure args
as context for the §19 name->MusicBrainz fallback (inert until the render
passes them). Library backfill passes None.
* feat(settings): add External Artwork Scraper toggle under Integrations
Master toggle (themeStore, off by default per §20) in a new Integrations
subsection, alongside the other opt-in third-party categories. Contacts
fanart.tv only when enabled. i18n across all 9 locales.
* feat(cover): wire fanart background into the fullscreen player (§28)
New useArtistFanart hook resolves a fanart.tv 16:9 background via a dedicated
cover_cache_ensure (surfaceKind=fanart) — it bypasses the shared peek/disk-src
cache (the {tier}-fanart.webp surface is keyed differently) and reuses
coverDiskUrl for the asset URL. Fullscreen background priority is now
fanart -> Navidrome artist image (cover pipeline) -> album cover; the live
useFsArtistPortrait probe is deleted (§28). Additive ensure opts
(surfaceKind/artistName/albumTitle); externalArtworkEnabled is derived in
ensureArgsFromRef from the master toggle and restricted to the artist fanart
surface, so plain cover ensures are unaffected.
* feat(cover): generalize external surface to fanart + banner (§13)
surfaceKind='banner' fetches the fanart.tv musicbanner array -> {tier}-banner.webp
in the same entity dir; fanart stays the 16:9 artistbackground. The ensure
branch, peek, lookup rows (per-surface surface_kind), miss marker
(.miss-<surface>) and tier suffix are all surface-parameterised. The §11
quality gate stays fanart-only (the banner strip has its own aspect). Unit
test for the surface->fanart JSON key map.
* feat(artist): fanart banner on the artist-detail header (§13, Option B)
The artist-detail header gets an album-detail-style background layer: fanart.tv
banner (musicbanner) -> the 16:9 fanart background cropped to the strip ->
empty. Both via a shared useArtistExternalImage hook (useArtistBanner /
useArtistFanart); each fetches on demand and shares the Rust cache, so the
header and the fullscreen player warm each other's images. The header is its
own stacking context (isolation) so a z-index:-1 banner clips behind the avatar
+ meta with no content wrapper; the album-style framing (padding/radius/clip)
is applied only when a banner is shown, so the off-by-default case stays
pixel-identical.
* refactor(artist): reuse album-detail header structure for the fanart banner
The artist-detail header now uses the same album-detail-* container classes as
AlbumHeader (header/bg/overlay/content/hero) with the fanart banner as the
background; the Back button moves inside the header. A surgical
`artist-detail-bleed` cancels the artist page's .content-body padding so the
banner is full-bleed to the container edges, matching the album header exactly
instead of the earlier inset card. Reverts the experimental artist-specific bg
CSS.
* feat(cover): drop artist_artwork_lookup rows on clear-cover-cache (§12/B.4)
cover_cache_clear_server already removed the server's whole cover dir (so the
{tier}-fanart.webp / -banner.webp tiers + .miss-* markers go with it); also
clear the artist_artwork_lookup rows for that server (off-thread) so no stale
resolution state lingers. Automatic toggle-off purge deferred — turning the
toggle off already hides external artwork (render is gated), and explicit
cache-clear now cleans external state too.
* feat(cover): name->MusicBrainz album-confirmed MBID resolution (§19)
When getArtistInfo2 has no tag MBID and the ensure carries the artist name +
an album in context (fullscreen), one MusicBrainz release-search query resolves
the artist MBID: the primary artist across score>=90 releases wins, conflicting
ids -> mbid_ambiguous (24h backoff), none -> no_mbid. Sends the required
User-Agent; a single-permit musicbrainz_sem + >=1s spacing holds us under MB's
rate limit. mbid_source=musicbrainz persisted. Banner surface (no album
context) correctly skips this. Pure classify/escape helpers unit-tested.
* fix(cover): enable banner surface in ensureArgsFromRef
externalEnsureFields only set externalArtworkEnabled for surfaceKind 'fanart',
so the 'banner' surface never fired — the artist-detail header always fell back
to the fanart image instead of the fanart.tv musicbanner. Both external artist
surfaces (fanart/banner) now enable the external branch.
* feat(settings): optional BYOK personal fanart key field
Add an optional personal fanart.tv API key field to the External Artwork
Scraper block (shown when the toggle is on): a masked input, a saved/in-use
status line, and the simple note that it is sent in addition to the app key.
Persisted in themeStore and plumbed through cover_cache_ensure
(externalArtworkByok); Rust prefers the settings key, falling back to the
PSYSONIC_FANART_CLIENT_KEY dev env. i18n x9.
* fix(cover): resolve artist-page fanart image collision on navigation
The artist-detail header keyed its fanart/banner hooks on the route `id`,
which flips immediately on navigation while `artist`/`albums` refetch a beat
later. The mismatched ensure wrote the previous artist's image under the new
artist's id (e.g. Sepultura's image under Lordi's id).
- key on the loaded `artist.id`, not the route `id`, so id/name/album always
describe the same artist
- pick the §19 album context from an album that actually belongs to this
artist (`albums.find(a => a.artistId === artist.id)`), so a stale album can't
run a mismatched name→MusicBrainz query or cache a wrong `no_mbid`
- reset `src` on every input change in `useArtistExternalImage` so a previous
artist's image never lingers while the new one resolves
* fix(cover): strip trailing album qualifier before MusicBrainz lookup
Library titles like "Show No Mercy (2004 Remastered)" or "Album [Deluxe
Edition]" failed the §19 MusicBrainz release query, blocking name-confirmed
MBID resolution. `normalize_album_for_mb` strips a single trailing
parenthetical/bracketed qualifier; leading qualifiers (e.g. "(What's the
Story) Morning Glory?") are left intact. Unit-tested.
* fix(cover): don't cache no_mbid when album context is unavailable
The banner ensure could fire before the artist's albums loaded, with no album
in context. The old code cached `no_mbid` there and the 24h backoff then
blocked the later ensure that arrived *with* album context. Could-not-attempt
is not tried-and-failed: the no-album branch now returns without persisting.
* fix(cover): don't emit tier-ready for external fanart/banner surfaces
`try_external_fanart` emitted `cover:tier-ready` with the `{tier}-{surface}.webp`
path. That event is keyed by the canonical cover key (cacheKind/cacheEntityId/
tier, no surface), so the frontend `useCoverArtBridge` listener seeded the
Navidrome artist cover's disk-src cache with the external image — leaking
fanart/banner into the plain artist cover (avatar, fullscreen "navidrome-artist"
fallback) even with the scraper toggled off.
Remove the emit: the fanart/banner hooks read the path from the
`cover_cache_ensure` return value, so no event is needed. (No disk-level
overwrite — the suffixed files are never matched by `tier_exists`; this was
frontend disk-src-cache cross-contamination.)
* fix(cover): wait for the final external background before showing it, with fade-in
The fullscreen player and artist-detail header flashed several backgrounds in
sequence while the fanart resolved (upscaled album cover → Navidrome artist
image → fanart), and the artist header could show the fanart first and then
swap to the banner.
- the album cover is no longer a background source — it only feeds the
foreground thumbnail
- the external-artwork hooks return `{ src, pending }` so callers can tell
"still resolving" (hold back) from "resolved, no image" (fall back now)
- fullscreen background: scraper on → fanart, empty while it resolves, Navidrome
artist image only on a confirmed miss; scraper off → Navidrome artist image
- artist header: the banner is preferred — nothing shows while it resolves
(no fanart flash), fanart is the fallback only once the banner misses
- both backgrounds preload the chosen image and fade it in (`onLoad` plus a
`ref` `complete` check so an already-cached image, whose load event can fire
before React attaches the handler, still appears). The header fade is a
scoped inline opacity so the shared `album-detail-bg` class is untouched.
* ci(release): pass PSYSONIC_FANART_KEY into the macOS + Linux builds
* refactor(cover): extract external-artwork ensure into its own module
Pure code move: the on-demand fanart/banner fetch, the quality gate, the
surface-aware peek and the lookup-table cache move from cover_cache/mod.rs
into cover_cache/external_ensure.rs. Behaviour unchanged; mod.rs 1877 -> 1488.
* chore(cover): remove dev-only fanart spike helper
The real render wiring now exercises the external ensure branch, so the
dev-only window.psyFanartSpike helper is redundant.
* feat(cover): purge external artwork on opt-out (B3)
New cover_cache_purge_external command: when the External Artwork toggle is
turned off, drop every fetched {tier}-{provider}.webp, .miss-{provider}
marker and artist_artwork_lookup row across all configured servers, leaving
the canonical Navidrome covers intact. Opting out now removes the
third-party-sourced data instead of just hiding it (design-review §9/§12/B.4).
* docs: changelog, credits and what's new for artist fanart (PR #1137)
712 lines
27 KiB
Rust
712 lines
27 KiB
Rust
//! Library cover backfill — one background pass per wake (native, not webview timers).
|
|
|
|
use super::{state, CoverCacheEnsureArgs, CoverCacheState};
|
|
use psysonic_library::cover_backfill::{
|
|
clear_cover_fetch_failures, collect_cover_progress, count_distinct_cover_ids,
|
|
diff_missing_against_snapshot, fetch_all_catalog_rows, snapshot_cover_disk,
|
|
LIBRARY_COVER_CANONICAL_TIER,
|
|
};
|
|
use psysonic_library::payload::LibrarySyncProgressPayload;
|
|
use psysonic_library::repos::sync_state::SyncStateRepository;
|
|
use psysonic_library::LibraryRuntime;
|
|
use serde::{Deserialize, Serialize};
|
|
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
|
|
use std::sync::Arc;
|
|
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
|
use tauri::{AppHandle, Emitter, Listener, Manager};
|
|
use tokio::sync::{Mutex, Semaphore};
|
|
|
|
use super::{count_cached_cover_ids, dir_usage_for_server};
|
|
|
|
/// Default concurrent library downloads + encodes. Runtime-tunable via the
|
|
/// perf probe (`set_parallel`); the constant is only the startup value.
|
|
const LIBRARY_BACKFILL_PARALLEL_DEFAULT: usize = 2;
|
|
/// Bounds for the runtime knob — keep it sane so a stray value cannot DoS the
|
|
/// host or starve the audio path.
|
|
pub const LIBRARY_BACKFILL_PARALLEL_MIN: usize = 1;
|
|
pub const LIBRARY_BACKFILL_PARALLEL_MAX: usize = 16;
|
|
/// Raw catalog rows diffed per streaming chunk. Small enough that downloads
|
|
/// start almost immediately after the one-shot enumeration, large enough to
|
|
/// amortize the per-chunk `spawn_blocking` hop.
|
|
const SCAN_CHUNK_ROWS: usize = 512;
|
|
const SYNC_WAIT_MS: u64 = 5000;
|
|
/// Cadence of the in-pass progress ticker (drives the "offline & cache" menu and
|
|
/// the perf-probe overlay while a pass downloads). Only runs for the duration of
|
|
/// an active pass.
|
|
const PROGRESS_TICK_SECS: u64 = 3;
|
|
/// Minimum gap between `library:sync-idle`-driven passes. Each such pass runs the
|
|
/// idle-gate signature (a full cover-dir walk + DB count), so a chatty sync (e.g.
|
|
/// periodic delta syncs) must not make that walk fire every few seconds. Manual
|
|
/// runs and strategy-toggle wakes bypass this.
|
|
const SYNC_IDLE_COOLDOWN_MS: u64 = 60_000;
|
|
|
|
fn now_ms() -> u64 {
|
|
SystemTime::now()
|
|
.duration_since(UNIX_EPOCH)
|
|
.map(|d| d.as_millis() as u64)
|
|
.unwrap_or(0)
|
|
}
|
|
|
|
#[derive(Clone)]
|
|
pub struct CoverBackfillSession {
|
|
pub server_index_key: String,
|
|
pub library_server_id: String,
|
|
pub username: String,
|
|
pub password: String,
|
|
}
|
|
|
|
/// Catalog signature captured when a full pass completes.
|
|
///
|
|
/// While it still matches, `library:sync-idle` must NOT re-trigger a rescan —
|
|
/// mirrors the analysis coordinator's `completed_total` gate. Deliberately the
|
|
/// **cheap** `COUNT(DISTINCT)` over the catalog only: a server change (track
|
|
/// add/remove shifts `total`) re-arms the next pass, but checking it never
|
|
/// touches the filesystem. A cover-cache clear leaves `total` unchanged, so the
|
|
/// clear commands re-arm the gate explicitly (`rearm_idle_gate`).
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
struct CoverIdleSignature {
|
|
total: i64,
|
|
}
|
|
|
|
pub struct CoverBackfillWorker {
|
|
pub enabled: AtomicBool,
|
|
/// When true, the active pass yields so visible-route cover IPC is not starved.
|
|
pub ui_priority_hold: AtomicBool,
|
|
session: Mutex<Option<CoverBackfillSession>>,
|
|
cursor: Mutex<String>,
|
|
pass_running: AtomicBool,
|
|
backfill_http: Arc<Semaphore>,
|
|
/// Live download/encode concurrency for backfill passes. Mirrors the
|
|
/// `backfill_http` permit count and gates per-batch `ensure_one` tasks.
|
|
parallel: AtomicUsize,
|
|
/// Set when a pass found nothing pending; suppresses idle-driven rescans
|
|
/// until the catalog signature changes. `None` means "re-armed".
|
|
settled: Mutex<Option<CoverIdleSignature>>,
|
|
/// Epoch-ms of the last `sync-idle`-driven pass, to rate-limit the idle-gate
|
|
/// disk walk against chatty syncs. 0 = never.
|
|
last_sync_idle_ms: AtomicU64,
|
|
/// Live connect URL, resolved fresh per cover fetch rather than baked into
|
|
/// the worklist. The worklist holds URL-agnostic items; a LAN→public flip
|
|
/// just swaps this cell, so even the pass already in flight downloads its
|
|
/// remaining covers against the now-reachable endpoint.
|
|
base_url: std::sync::Mutex<String>,
|
|
/// A forced retry requested while a pass was already running (e.g. the
|
|
/// connect URL flipped LAN→public at boot). The in-flight pass already
|
|
/// adopts the new URL live, but the handful of covers it attempted against
|
|
/// the stale address need one more forced pass once it finishes.
|
|
rerun_pending: AtomicBool,
|
|
}
|
|
|
|
#[derive(Debug, Clone, Serialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
pub struct CoverBackfillPulseDto {
|
|
pub scheduled: u32,
|
|
pub exhausted: bool,
|
|
pub pending: i64,
|
|
pub done: i64,
|
|
pub total: i64,
|
|
pub status: String,
|
|
}
|
|
|
|
#[derive(Debug, Clone, Serialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
pub struct CoverBackfillRunDto {
|
|
pub started: bool,
|
|
}
|
|
|
|
#[derive(Debug, Deserialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
struct SyncIdlePayload {
|
|
server_id: String,
|
|
ok: bool,
|
|
}
|
|
|
|
impl CoverBackfillWorker {
|
|
pub fn new() -> Self {
|
|
Self {
|
|
enabled: AtomicBool::new(false),
|
|
ui_priority_hold: AtomicBool::new(false),
|
|
session: Mutex::new(None),
|
|
cursor: Mutex::new(String::new()),
|
|
pass_running: AtomicBool::new(false),
|
|
backfill_http: Arc::new(Semaphore::new(LIBRARY_BACKFILL_PARALLEL_DEFAULT)),
|
|
parallel: AtomicUsize::new(LIBRARY_BACKFILL_PARALLEL_DEFAULT),
|
|
settled: Mutex::new(None),
|
|
last_sync_idle_ms: AtomicU64::new(0),
|
|
base_url: std::sync::Mutex::new(String::new()),
|
|
rerun_pending: AtomicBool::new(false),
|
|
}
|
|
}
|
|
|
|
pub fn set_ui_priority_hold(&self, hold: bool) {
|
|
self.ui_priority_hold.store(hold, Ordering::Relaxed);
|
|
}
|
|
|
|
/// Re-arm the idle gate so the next opportunistic pass runs even though the
|
|
/// catalog `total` is unchanged — used after a cover-cache clear, which
|
|
/// drops files the cheap signature cannot see.
|
|
pub async fn rearm_idle_gate(&self) {
|
|
*self.settled.lock().await = None;
|
|
self.last_sync_idle_ms.store(0, Ordering::Relaxed);
|
|
}
|
|
|
|
/// Current backfill download/encode concurrency.
|
|
pub fn parallel(&self) -> usize {
|
|
self.parallel.load(Ordering::Relaxed).max(LIBRARY_BACKFILL_PARALLEL_MIN)
|
|
}
|
|
|
|
/// Retune backfill concurrency at runtime. Resizes the shared HTTP permit
|
|
/// pool to match (next batch picks up the new per-batch slot count). Returns
|
|
/// the clamped value actually applied.
|
|
pub fn set_parallel(&self, threads: usize) -> usize {
|
|
let next = threads.clamp(LIBRARY_BACKFILL_PARALLEL_MIN, LIBRARY_BACKFILL_PARALLEL_MAX);
|
|
let prev = self.parallel.swap(next, Ordering::SeqCst);
|
|
if next > prev {
|
|
self.backfill_http.add_permits(next - prev);
|
|
} else if next < prev {
|
|
// Shrinking: drain surplus permits as they free up so in-flight
|
|
// fetches finish but no new ones start beyond the new cap.
|
|
let sem = self.backfill_http.clone();
|
|
let surplus = prev - next;
|
|
tauri::async_runtime::spawn(async move {
|
|
for _ in 0..surplus {
|
|
if let Ok(permit) = sem.acquire().await {
|
|
permit.forget();
|
|
}
|
|
}
|
|
});
|
|
}
|
|
next
|
|
}
|
|
|
|
pub async fn set_session(
|
|
&self,
|
|
enabled: bool,
|
|
session: Option<CoverBackfillSession>,
|
|
base_url: String,
|
|
) {
|
|
self.enabled.store(enabled, Ordering::Relaxed);
|
|
*self.session.lock().await = session;
|
|
*self.base_url.lock().unwrap() = base_url;
|
|
// Server switch or enable/disable invalidates any settled state: re-arm
|
|
// so the next idle event runs a real pass for the new focus.
|
|
*self.settled.lock().await = None;
|
|
if !enabled {
|
|
*self.cursor.lock().await = String::new();
|
|
}
|
|
}
|
|
|
|
/// Current connect URL for backfill fetches. Read fresh per cover so a
|
|
/// LAN→public flip is honoured mid-pass without rebuilding the worklist.
|
|
pub fn base_url(&self) -> String {
|
|
self.base_url.lock().unwrap().clone()
|
|
}
|
|
|
|
/// Swap the live connect URL. Returns `true` when it actually changed, so the
|
|
/// caller can clear the now-stale fetch-failed backoff and kick a retry pass.
|
|
pub fn set_base_url(&self, url: String) -> bool {
|
|
let mut cell = self.base_url.lock().unwrap();
|
|
if *cell == url {
|
|
return false;
|
|
}
|
|
*cell = url;
|
|
true
|
|
}
|
|
|
|
pub async fn reset_cursor(&self) {
|
|
*self.cursor.lock().await = String::new();
|
|
}
|
|
|
|
/// Semaphore-backed library backfill HTTP slots (perf probe).
|
|
pub fn pipeline_http_stats(&self) -> (u32, u32, bool) {
|
|
let max = self.parallel() as u32;
|
|
let active = max.saturating_sub(self.backfill_http.available_permits() as u32);
|
|
let pass_running = self.pass_running.load(Ordering::Relaxed);
|
|
(max, active, pass_running)
|
|
}
|
|
}
|
|
|
|
fn sync_allows_cover_backfill(store: &psysonic_library::store::LibraryStore, server_id: &str) -> bool {
|
|
let repo = SyncStateRepository::new(store);
|
|
match repo.get_sync_phase(server_id, "") {
|
|
Ok(Some(phase)) => phase != "initial_sync" && phase != "probing",
|
|
_ => true,
|
|
}
|
|
}
|
|
|
|
fn session_matches_server(session: &CoverBackfillSession, server_id: &str) -> bool {
|
|
server_id == session.server_index_key || server_id == session.library_server_id
|
|
}
|
|
|
|
/// Backfill runs only while this session is still the configured focus (active
|
|
/// server). A connect-URL flip keeps the same `server_index_key` and is picked
|
|
/// up live via `worker.base_url()`, so it does not abort the pass — only a
|
|
/// server switch or disable does.
|
|
async fn session_still_focused(worker: &CoverBackfillWorker, expected: &CoverBackfillSession) -> bool {
|
|
if !worker.enabled.load(Ordering::Relaxed) {
|
|
return false;
|
|
}
|
|
worker
|
|
.session
|
|
.lock()
|
|
.await
|
|
.as_ref()
|
|
.is_some_and(|s| s.server_index_key == expected.server_index_key)
|
|
}
|
|
|
|
async fn progress_snapshot(
|
|
store: &psysonic_library::store::LibraryStore,
|
|
root: &std::path::Path,
|
|
library_server_id: &str,
|
|
server_index_key: &str,
|
|
) -> Result<(i64, i64, i64), String> {
|
|
let cached = count_cached_cover_ids(root, server_index_key);
|
|
let p = collect_cover_progress(store, library_server_id, root, server_index_key, cached)?;
|
|
Ok((p.done, p.total_distinct, p.pending))
|
|
}
|
|
|
|
async fn emit_library_progress(
|
|
app: &AppHandle,
|
|
session: &CoverBackfillSession,
|
|
done: i64,
|
|
total: i64,
|
|
pending: i64,
|
|
root: &std::path::Path,
|
|
) {
|
|
let (bytes, entry_count) = dir_usage_for_server(root, &session.server_index_key);
|
|
let _ = app.emit(
|
|
"cover:library-progress",
|
|
serde_json::json!({
|
|
"serverIndexKey": session.server_index_key,
|
|
"done": done,
|
|
"total": total,
|
|
"pending": pending,
|
|
"bytes": bytes,
|
|
"entryCount": entry_count,
|
|
}),
|
|
);
|
|
}
|
|
|
|
async fn ensure_one(
|
|
worker: &CoverBackfillWorker,
|
|
st: Arc<tokio::sync::Mutex<CoverCacheState>>,
|
|
http_sem: Arc<Semaphore>,
|
|
app: AppHandle,
|
|
session: CoverBackfillSession,
|
|
item: psysonic_library::cover_backfill::CoverBackfillItem,
|
|
) {
|
|
if worker.ui_priority_hold.load(Ordering::Relaxed) {
|
|
return;
|
|
}
|
|
let args = CoverCacheEnsureArgs {
|
|
server_index_key: session.server_index_key,
|
|
cache_kind: item.cache_kind,
|
|
cache_entity_id: item.cache_entity_id,
|
|
cover_art_id: item.fetch_cover_art_id,
|
|
tier: LIBRARY_COVER_CANONICAL_TIER,
|
|
rest_base_url: worker.base_url(),
|
|
username: session.username,
|
|
password: session.password,
|
|
library_bulk: true,
|
|
library_server_id: Some(session.library_server_id),
|
|
// Library backfill never touches external providers (§15).
|
|
external_artwork_enabled: false,
|
|
surface_kind: None,
|
|
artist_name: None,
|
|
album_title: None,
|
|
external_artwork_byok: None,
|
|
};
|
|
let _ = CoverCacheState::ensure_inner(&st, &app, &args, Some(http_sem)).await;
|
|
}
|
|
|
|
async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>, force: bool) {
|
|
if !worker.enabled.load(Ordering::Relaxed) {
|
|
return;
|
|
}
|
|
let session = worker.session.lock().await.clone();
|
|
let Some(session) = session else {
|
|
return;
|
|
};
|
|
|
|
let runtime = match app.try_state::<LibraryRuntime>() {
|
|
Some(r) => r,
|
|
None => return,
|
|
};
|
|
|
|
// Opportunistic triggers (wake on track change, sync-idle) skip the whole
|
|
// scan when a prior pass already settled and nothing changed — otherwise a
|
|
// library with permanently-unfetchable covers (404s) would re-scan on every
|
|
// wake forever. The manual "Run full pass now" sets `force` to bypass this.
|
|
if !force && cover_idle_gate_should_skip(&app, &worker, &session).await {
|
|
return;
|
|
}
|
|
|
|
while !sync_allows_cover_backfill(&runtime.store, &session.library_server_id) {
|
|
if !worker.enabled.load(Ordering::Relaxed) {
|
|
return;
|
|
}
|
|
tokio::time::sleep(Duration::from_millis(SYNC_WAIT_MS)).await;
|
|
}
|
|
|
|
let st = match state(&app) {
|
|
Ok(s) => s,
|
|
Err(_) => return,
|
|
};
|
|
let root = {
|
|
let guard = st.lock().await;
|
|
guard.root.clone()
|
|
};
|
|
let st_arc = st.clone();
|
|
|
|
worker.reset_cursor().await;
|
|
let http_sem = worker.backfill_http.clone();
|
|
|
|
// A forced pass is an explicit user retry: drop the `.fetch-failed` backoff
|
|
// markers and the settled gate so previously-404'd covers are attempted
|
|
// again. Opportunistic passes leave the markers in place (30-min TTL).
|
|
if force {
|
|
*worker.settled.lock().await = None;
|
|
let root2 = root.clone();
|
|
let index_key2 = session.server_index_key.clone();
|
|
let _ = tauri::async_runtime::spawn_blocking(move || {
|
|
clear_cover_fetch_failures(&root2, &index_key2)
|
|
})
|
|
.await;
|
|
}
|
|
|
|
// Two snapshots, taken ONCE per pass: the DB catalog (single GROUP BY) and
|
|
// the on-disk cover bucket (one directory walk). The delta = catalog minus
|
|
// disk, streamed in chunks below. No per-row `stat` on the filesystem and no
|
|
// re-scan loop — pure set math against the captured disk snapshot.
|
|
let (raw_rows, snapshot) = {
|
|
let store = runtime.store.clone();
|
|
let lib_id = session.library_server_id.clone();
|
|
let root_for_scan = root.clone();
|
|
let index_key = session.server_index_key.clone();
|
|
match tauri::async_runtime::spawn_blocking(move || {
|
|
let rows = fetch_all_catalog_rows(&store, &lib_id)?;
|
|
let snap = snapshot_cover_disk(&root_for_scan, &index_key);
|
|
Ok::<_, String>((rows, snap))
|
|
})
|
|
.await
|
|
{
|
|
Ok(Ok(pair)) => pair,
|
|
_ => (Vec::new(), Default::default()),
|
|
}
|
|
};
|
|
let snapshot = Arc::new(snapshot);
|
|
|
|
// Producer/consumer: a fixed pool of consumer tasks pulls misses off a
|
|
// bounded channel and downloads them continuously, while the producer scans
|
|
// the catalog in chunks and feeds misses in. This keeps the pool saturated
|
|
// even when misses are sparse across chunks — no per-chunk drain barrier.
|
|
// True concurrency stays governed by the resizable `http_sem` / encode
|
|
// semaphores inside `ensure_one`, so the threads slider still applies live.
|
|
let (tx, rx) =
|
|
tokio::sync::mpsc::channel::<psysonic_library::cover_backfill::CoverBackfillItem>(256);
|
|
let rx = Arc::new(Mutex::new(rx));
|
|
let mut consumers = tokio::task::JoinSet::new();
|
|
for _ in 0..LIBRARY_BACKFILL_PARALLEL_MAX {
|
|
let rx = rx.clone();
|
|
let st = st_arc.clone();
|
|
let http_sem = http_sem.clone();
|
|
let app = app.clone();
|
|
let session = session.clone();
|
|
let worker_arc = worker.clone();
|
|
consumers.spawn(async move {
|
|
loop {
|
|
// Bail the moment the strategy flips to lazy / focus changes, so a
|
|
// switch to "lazy" abandons the buffered backlog instead of
|
|
// draining the whole channel (mirrors the producer's check).
|
|
if !session_still_focused(&worker_arc, &session).await {
|
|
break;
|
|
}
|
|
let item = {
|
|
let mut guard = rx.lock().await;
|
|
guard.recv().await
|
|
};
|
|
let Some(item) = item else { break };
|
|
ensure_one(
|
|
worker_arc.as_ref(),
|
|
st.clone(),
|
|
http_sem.clone(),
|
|
app.clone(),
|
|
session.clone(),
|
|
item,
|
|
)
|
|
.await;
|
|
}
|
|
});
|
|
}
|
|
|
|
// Progress ticker: the producer finishes enumerating the worklist long before
|
|
// the consumers finish downloading it, so emitting only while feeding would
|
|
// freeze the "offline & cache" menu through the whole drain phase. Tick a
|
|
// periodic snapshot for the lifetime of the pass instead; aborted once the
|
|
// consumers drain (a final accurate emit happens at settle below).
|
|
let progress_ticker = {
|
|
let app = app.clone();
|
|
let store = runtime.store.clone();
|
|
let root = root.clone();
|
|
let session = session.clone();
|
|
let lib_id = session.library_server_id.clone();
|
|
let index_key = session.server_index_key.clone();
|
|
tauri::async_runtime::spawn(async move {
|
|
loop {
|
|
tokio::time::sleep(Duration::from_secs(PROGRESS_TICK_SECS)).await;
|
|
if let Ok((done, total, pending)) =
|
|
progress_snapshot(&store, &root, &lib_id, &index_key).await
|
|
{
|
|
emit_library_progress(&app, &session, done, total, pending, &root).await;
|
|
}
|
|
}
|
|
})
|
|
};
|
|
|
|
let mut rows_iter = raw_rows.into_iter();
|
|
let mut completed = false;
|
|
loop {
|
|
if !session_still_focused(&worker, &session).await {
|
|
break;
|
|
}
|
|
if worker.ui_priority_hold.load(Ordering::Relaxed) {
|
|
tokio::time::sleep(Duration::from_millis(200)).await;
|
|
continue;
|
|
}
|
|
|
|
let scan_chunk: Vec<_> = rows_iter.by_ref().take(SCAN_CHUNK_ROWS).collect();
|
|
if scan_chunk.is_empty() {
|
|
completed = true;
|
|
break;
|
|
}
|
|
|
|
// Diff this chunk against the captured snapshot off-thread (in-memory set
|
|
// math + DB expand only for rows not already cached) → misses to download.
|
|
let missing: Vec<_> = {
|
|
let store = runtime.store.clone();
|
|
let lib_id = session.library_server_id.clone();
|
|
let snapshot = snapshot.clone();
|
|
match tauri::async_runtime::spawn_blocking(move || {
|
|
diff_missing_against_snapshot(&store, &lib_id, &snapshot, scan_chunk)
|
|
})
|
|
.await
|
|
{
|
|
Ok(Ok(missing)) => missing,
|
|
_ => Vec::new(),
|
|
}
|
|
};
|
|
|
|
// Focus-aware feed: never park indefinitely on a full channel, or a
|
|
// switch to lazy (which stops the consumers) would deadlock the producer
|
|
// here. `try_send` + a short retry lets us re-check focus and bail.
|
|
let mut feed_closed = false;
|
|
'feed: for mut item in missing {
|
|
loop {
|
|
match tx.try_send(item) {
|
|
Ok(()) => break,
|
|
Err(tokio::sync::mpsc::error::TrySendError::Closed(_)) => {
|
|
feed_closed = true;
|
|
break 'feed;
|
|
}
|
|
Err(tokio::sync::mpsc::error::TrySendError::Full(returned)) => {
|
|
if !session_still_focused(&worker, &session).await {
|
|
feed_closed = true;
|
|
break 'feed;
|
|
}
|
|
item = returned;
|
|
tokio::time::sleep(Duration::from_millis(25)).await;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
if feed_closed {
|
|
break;
|
|
}
|
|
}
|
|
|
|
// Close the channel so consumers drain the remaining backlog and exit.
|
|
drop(tx);
|
|
while consumers.join_next().await.is_some() {}
|
|
progress_ticker.abort();
|
|
|
|
// Only settle the idle gate on a natural finish (worklist drained), never on
|
|
// a session-switch break — that belongs to the previous focus.
|
|
if completed {
|
|
worker.cursor.lock().await.clear();
|
|
match progress_snapshot(
|
|
&runtime.store,
|
|
&root,
|
|
&session.library_server_id,
|
|
&session.server_index_key,
|
|
)
|
|
.await
|
|
{
|
|
Ok((done, total, pending)) => {
|
|
// Settle on a full scan regardless of `pending`: whatever is left
|
|
// is unfetchable for now (404s with a fresh `.fetch-failed`
|
|
// marker). The cheap `total` signature re-triggers a pass only if
|
|
// the server catalog changes; a cache clear re-arms via the clear
|
|
// command. This stops the wake storm on libraries whose covers
|
|
// can never reach 100%.
|
|
*worker.settled.lock().await = Some(CoverIdleSignature { total });
|
|
emit_library_progress(&app, &session, done, total, pending, &root).await;
|
|
}
|
|
Err(_) => {
|
|
*worker.settled.lock().await = None;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Start one full-catalog pass on the Tokio runtime (survives inactive webview).
|
|
/// `force` bypasses the idle gate and clears fetch-failed backoff (explicit user
|
|
/// retry); opportunistic callers (wake / sync-idle) pass `false`.
|
|
pub async fn try_schedule_full_pass(app: &AppHandle, force: bool) -> bool {
|
|
let worker = match app.try_state::<Arc<CoverBackfillWorker>>() {
|
|
Some(w) => w.inner().clone(),
|
|
None => return false,
|
|
};
|
|
if !worker.enabled.load(Ordering::Relaxed) {
|
|
return false;
|
|
}
|
|
if worker
|
|
.pass_running
|
|
.compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst)
|
|
.is_err()
|
|
{
|
|
// A pass is already running. It reads the connect URL live per cover, so
|
|
// any flip that landed mid-pass already applies to its remaining work.
|
|
// A forced retry (URL flip) still queues a rerun so the few covers the
|
|
// in-flight pass attempted against the stale address get re-fetched.
|
|
if force {
|
|
worker.rerun_pending.store(true, Ordering::SeqCst);
|
|
}
|
|
return false;
|
|
}
|
|
|
|
let app = app.clone();
|
|
tauri::async_runtime::spawn(async move {
|
|
run_full_pass(app.clone(), worker.clone(), force).await;
|
|
// Drain a forced rerun queued mid-pass (always forced: it bypasses the
|
|
// idle gate the just-finished pass re-armed and clears the stale backoff).
|
|
loop {
|
|
worker.pass_running.store(false, Ordering::SeqCst);
|
|
if !worker.rerun_pending.swap(false, Ordering::SeqCst)
|
|
|| !worker.enabled.load(Ordering::Relaxed)
|
|
{
|
|
break;
|
|
}
|
|
if worker
|
|
.pass_running
|
|
.compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst)
|
|
.is_err()
|
|
{
|
|
worker.rerun_pending.store(true, Ordering::SeqCst);
|
|
break;
|
|
}
|
|
run_full_pass(app.clone(), worker.clone(), true).await;
|
|
}
|
|
});
|
|
true
|
|
}
|
|
|
|
/// Cheap catalog signature for the idle gate: a single `COUNT(DISTINCT)` over
|
|
/// the cover catalog. No filesystem access — checking "did anything change on
|
|
/// the server?" must never walk the on-disk cover cache.
|
|
async fn current_cover_signature(
|
|
app: &AppHandle,
|
|
session: &CoverBackfillSession,
|
|
) -> Option<CoverIdleSignature> {
|
|
let runtime = app.try_state::<LibraryRuntime>()?;
|
|
let store = runtime.store.clone();
|
|
let lib_id = session.library_server_id.clone();
|
|
tauri::async_runtime::spawn_blocking(move || {
|
|
count_distinct_cover_ids(&store, &lib_id)
|
|
.ok()
|
|
.map(|total| CoverIdleSignature { total })
|
|
})
|
|
.await
|
|
.ok()
|
|
.flatten()
|
|
}
|
|
|
|
/// True when the previous pass settled with nothing pending and the catalog
|
|
/// still matches that signature — so an idle event need not rescan.
|
|
async fn cover_idle_gate_should_skip(app: &AppHandle, worker: &CoverBackfillWorker, session: &CoverBackfillSession) -> bool {
|
|
let Some(settled) = *worker.settled.lock().await else {
|
|
return false;
|
|
};
|
|
match current_cover_signature(app, session).await {
|
|
Some(current) => current == settled,
|
|
None => false,
|
|
}
|
|
}
|
|
|
|
fn on_sync_idle(app: &AppHandle, payload: SyncIdlePayload) {
|
|
if !payload.ok {
|
|
return;
|
|
}
|
|
let app = app.clone();
|
|
tauri::async_runtime::spawn(async move {
|
|
let worker = match app.try_state::<Arc<CoverBackfillWorker>>() {
|
|
Some(w) => w.inner().clone(),
|
|
None => return,
|
|
};
|
|
if !worker.enabled.load(Ordering::Relaxed) {
|
|
return;
|
|
}
|
|
let session = worker.session.lock().await.clone();
|
|
let Some(session) = session else {
|
|
return;
|
|
};
|
|
if !session_matches_server(&session, &payload.server_id) {
|
|
return;
|
|
}
|
|
// Rate-limit sync-idle passes: each runs the idle-gate disk walk, so a
|
|
// chatty sync must not trigger it every few seconds. The gate inside the
|
|
// pass still skips the actual rescan when nothing changed.
|
|
let now = now_ms();
|
|
let last = worker.last_sync_idle_ms.load(Ordering::Relaxed);
|
|
if last != 0 && now.saturating_sub(last) < SYNC_IDLE_COOLDOWN_MS {
|
|
return;
|
|
}
|
|
worker.last_sync_idle_ms.store(now, Ordering::Relaxed);
|
|
// Opportunistic: the gate (checked inside the pass) skips the rescan when
|
|
// a prior pass settled and nothing changed (mirrors the analysis gate).
|
|
let _ = try_schedule_full_pass(&app, false).await;
|
|
});
|
|
}
|
|
|
|
/// Listen for library sync completion in native code (not throttled with the webview).
|
|
pub fn setup_library_sync_idle_listener(app: &AppHandle) {
|
|
let app_handle = app.clone();
|
|
let _ = app.listen(LibrarySyncProgressPayload::IDLE_EVENT_NAME, move |event| {
|
|
let Ok(payload) = serde_json::from_str::<SyncIdlePayload>(event.payload()) else {
|
|
return;
|
|
};
|
|
on_sync_idle(&app_handle, payload);
|
|
});
|
|
}
|
|
|
|
/// Legacy single-step API (optional diagnostics).
|
|
pub async fn pulse_backfill(app: &AppHandle, _worker: &Arc<CoverBackfillWorker>) -> CoverBackfillPulseDto {
|
|
if try_schedule_full_pass(app, false).await {
|
|
return CoverBackfillPulseDto {
|
|
scheduled: 0,
|
|
exhausted: false,
|
|
pending: 0,
|
|
done: 0,
|
|
total: 0,
|
|
status: "active".into(),
|
|
};
|
|
}
|
|
CoverBackfillPulseDto {
|
|
scheduled: 0,
|
|
exhausted: true,
|
|
pending: 0,
|
|
done: 0,
|
|
total: 0,
|
|
status: "disabled".into(),
|
|
}
|
|
}
|