mirror of
https://github.com/Psychotoxical/psysonic.git
synced 2026-07-21 23:05:46 +00:00
fix(cover-backfill): kill idle CPU spin and offline-cache menu re-walks (#943)
* fix(cover-backfill): snapshot-diff worklist and live-tunable parallelism Aggressive cover backfill pegged one tokio worker at ~100% on large, fully-synced libraries while the download queues stayed empty. - Take two snapshots once per pass — the DB catalog (single GROUP BY) and the on-disk cover bucket (one directory walk) — and download the set-difference. No per-row `stat` syscalls and no re-scan loop; the empty cache case (heavy backfill) costs zero per-item disk hits. - Replace the front-loaded enumeration with a producer/consumer pipeline: the producer streams the catalog in chunks and feeds misses into a bounded channel; a fixed consumer pool keeps the download/encode pools saturated. - Make cover backfill parallelism runtime-tunable from the Performance Probe (threads slider + "Run full pass now"); HTTP download and CPU encode semaphores resize live. Not surfaced in app settings. - Add a "nothing changed" idle gate (catalog signature) so a settled pass is not re-run on every library:sync-idle, mirroring the analysis worker. - Cancel promptly on switch to lazy: consumers bail on enabled/focus change and the producer feeds via try_send so a full channel cannot deadlock. - Drop the per-item recursive disk walk from the ensure hot path. * fix(cover-backfill): cheap idle gate, settle on 404s, transient retries Follow-up to the snapshot-diff backfill: stop the periodic CPU spikes and the 89%-plateau wake storm on libraries whose covers can never reach 100%. - Idle gate is now disk-free: compare only the catalog COUNT(DISTINCT) instead of walking ~all cover dirs on every sync-idle. "Did the server change?" never touches the filesystem. Clear-cache commands re-arm the gate (rearm_idle_gate) since a clear leaves the catalog total unchanged, and the settings UI wakes the active server after a clear. - Settle the gate on any completed pass regardless of pending: remaining items are unfetchable-for-now (404), so the wake/sync-idle storm stops once the fetchable set is exhausted. - Stop auto-clearing .fetch-failed markers every pass (it defeated the 30-min backoff and re-attempted 404s forever). The manual "Run full pass now" sends force=true to clear them and retry; wake/sync-idle/configure stay opportunistic. - Rate-limit sync-idle passes (60s cooldown) as defence against chatty syncs. - Retry cover downloads up to 3x with backoff on transient failures (5xx / 429 / network), but never on a real 4xx so missing covers don't hammer the server. * fix(cover-cache): stop re-walking cover dirs from offline & cache menu The settings cover-cache section polled disk usage + progress every 15s for every server, each call doing a full recursive walk of the per-server cover directory. On a fully populated cache this caused periodic CPU spikes whenever that menu was open. - mod.rs: add a 10s TTL memo around the per-server cover dir walk (cached_dir_usage_for_server), shared by cover_cache_stats_server and library_cover_progress; invalidate on clear (per-server and clear-all). - CoverCacheStrategySection: recompute on entry only; rely on the cover:library-progress and cover:cache-cleared events for live updates; drop the per-cover cover:tier-ready refresh storm; turn the 15s loop into a 5-minute safety net. * fix(cover-backfill): keep emitting progress during the whole pass The producer finishes enumerating the worklist long before the consumer pool finishes downloading it, so progress was only emitted while feeding the channel — the "offline & cache" menu and overlay then froze through the entire drain phase. Replace the per-chunk emit with a 3s progress ticker that runs for the lifetime of the pass and is aborted once the consumers drain (final accurate emit still happens at settle). * docs(changelog): record cover-backfill idle-CPU fix (PR #943) Add [1.47.0] Fixed + Changed entries and a settingsCredits line for the cover-backfill idle CPU / offline & cache menu work.
This commit is contained in:
cucadmuh
GitHub
@@ -359,6 +359,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
* Frontend and Rust `coverage` jobs no longer carry `continue-on-error`; listed hot-path files must stay at ≥70% line coverage or the PR fails.
|
||||
|
||||
### Cover backfill — live-tunable parallelism and pipeline
|
||||
|
||||
**By [@cucadmuh](https://github.com/cucadmuh), PR [#943](https://github.com/Psychotoxical/psysonic/pull/943)**
|
||||
|
||||
* Cover backfill runs through a producer/consumer pipeline (bounded channel + fixed consumer pool) that stays saturated and bails promptly on a switch to **lazy** instead of draining the whole backlog.
|
||||
* **Performance Probe** gains a runtime cover-thread control (`library_cover_backfill_set_parallel`) that resizes the HTTP/encode pools live; "Run full pass now" forces a pass and clears fetch-failed backoff.
|
||||
* Clearing the active server's cover cache re-arms the idle gate and wakes the worker, and in-pass progress is emitted on a ticker so the offline & cache view keeps counting through the whole scan.
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -645,7 +653,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
* Pausing a track and resuming later inflated the listening time in **Statistics → Player stats** — the whole paused span was billed as if the track had been playing.
|
||||
* Root cause: the session's tick baseline froze on pause, so the first progress tick after resume measured against the pre-pause timestamp. It now settles the played segment on pause and rebaselines on resume.
|
||||
|
||||
### Cover backfill — idle CPU spin and offline & cache menu spikes
|
||||
|
||||
**By [@cucadmuh](https://github.com/cucadmuh), PR [#943](https://github.com/Psychotoxical/psysonic/pull/943)**
|
||||
|
||||
* **Aggressive** cover backfill could pin a `tokio-runtime-worker` near 100% CPU when effectively idle: it had no "nothing changed, don't rescan" gate. Added a cheap disk-free idle signature (`COUNT(DISTINCT)` covers) with a `sync-idle` cooldown; the gate settles on a completed pass even when some covers are unfetchable (404), so libraries that never reach 100% no longer trigger a wake storm.
|
||||
* Worklist is now built from a single DB `GROUP BY` plus one cover-dir snapshot and diffed in memory — no per-row `stat`, no per-batch rescan loop — so increasing parallelism actually saturates the pipeline.
|
||||
* **Settings → offline & cache** caused periodic CPU spikes: the section re-walked each server's full cover directory every 15s. The per-server walk is now memoized with a short TTL and reused by the stats/progress commands; the menu recomputes on entry and via progress/cache-cleared events, with a 5-minute safety poll instead of a tight 15s loop.
|
||||
* Transient download failures (network / 5xx / 429) retry up to 3× with exponential backoff; permanent 4xx settle without re-scanning.
|
||||
|
||||
## [1.46.0] - 2026-05-18
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
//! Catalog rows come from SQLite (`album` / `artist` tables) with explicit `kind`.
|
||||
//! On-disk paths — `psysonic_core::cover_cache_layout`.
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use psysonic_core::cover_cache_layout::{self, is_fetch_only_cover_id};
|
||||
@@ -13,7 +14,12 @@ use crate::cover_resolve::{
|
||||
use crate::store::LibraryStore;
|
||||
|
||||
const DEFAULT_BATCH: u32 = 32;
|
||||
const MAX_BATCH: u32 = 48;
|
||||
/// Upper bound on items collected per `collect_cover_backfill_batch` call. The
|
||||
/// catalog scan (`fetch_catalog_page` GROUP BY over the cover UNION) is O(catalog)
|
||||
/// per page, so larger batches amortize that cost across more downloads and keep
|
||||
/// the backfill download pool fed. Per-call page count stays bounded by
|
||||
/// `MAX_SCAN_PAGES`, so a sparse backlog cannot inflate scan cost here.
|
||||
const MAX_BATCH: u32 = 256;
|
||||
const SCAN_PAGE: i64 = 256;
|
||||
const MAX_SCAN_PAGES: usize = 16;
|
||||
|
||||
@@ -224,6 +230,152 @@ fn fetch_catalog_page(
|
||||
})
|
||||
}
|
||||
|
||||
/// All distinct cover catalog rows in ONE query (no cursor pagination, so the
|
||||
/// expensive `GROUP BY` over the cover UNION runs once per pass instead of once
|
||||
/// per page). Caller expands + disk-diffs the result.
|
||||
pub fn fetch_all_catalog_rows(
|
||||
store: &LibraryStore,
|
||||
library_server_id: &str,
|
||||
) -> Result<Vec<CoverBackfillItem>, String> {
|
||||
store.with_read_conn(|conn| {
|
||||
let sql = format!(
|
||||
"SELECT kind, id, MAX(fetch_id) AS fetch_id
|
||||
FROM ({COVER_CATALOG_SUBQUERY})
|
||||
GROUP BY kind, id
|
||||
ORDER BY kind ASC, id ASC"
|
||||
);
|
||||
let mut stmt = conn.prepare(&sql)?;
|
||||
let rows = stmt
|
||||
.query_map(rusqlite::params![library_server_id], |row| {
|
||||
let kind: String = row.get(0)?;
|
||||
let id: String = row.get(1)?;
|
||||
let fetch_id: String = row.get(2)?;
|
||||
Ok(CoverBackfillItem {
|
||||
cache_kind: kind,
|
||||
cache_entity_id: id,
|
||||
fetch_cover_art_id: fetch_id,
|
||||
})
|
||||
})?
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
Ok(rows
|
||||
.into_iter()
|
||||
.filter(|item| {
|
||||
!item.cache_entity_id.is_empty() && !is_fetch_only_cover_id(&item.cache_entity_id)
|
||||
})
|
||||
.collect())
|
||||
})
|
||||
}
|
||||
|
||||
/// One-pass on-disk snapshot of a server's cover bucket: which entities already
|
||||
/// have the canonical tier, and which carry a recent `.fetch-failed` marker.
|
||||
///
|
||||
/// Built once per pass so the catalog diff is pure in-memory set math — no
|
||||
/// per-row `stat` syscalls hammering the filesystem on every album/artist. Keys
|
||||
/// are `(kind, sanitized_entity_id)` to match on-disk directory names.
|
||||
#[derive(Debug, Default)]
|
||||
pub struct CoverDiskSnapshot {
|
||||
present: HashSet<(String, String)>,
|
||||
failed: HashSet<(String, String)>,
|
||||
}
|
||||
|
||||
impl CoverDiskSnapshot {
|
||||
fn key(kind: &str, entity_id: &str) -> (String, String) {
|
||||
(
|
||||
kind.to_string(),
|
||||
cover_cache_layout::sanitize_path_segment(entity_id),
|
||||
)
|
||||
}
|
||||
|
||||
/// Canonical tier already on disk for this entity.
|
||||
pub fn is_cached(&self, kind: &str, entity_id: &str) -> bool {
|
||||
self.present.contains(&Self::key(kind, entity_id))
|
||||
}
|
||||
|
||||
/// Recent `.fetch-failed` marker — skip so slots go to fetchable art.
|
||||
pub fn is_recently_failed(&self, kind: &str, entity_id: &str) -> bool {
|
||||
self.failed.contains(&Self::key(kind, entity_id))
|
||||
}
|
||||
}
|
||||
|
||||
/// Walk the server's cover bucket once (`album/` and `artist/`) and record the
|
||||
/// cached plus recently-failed entities. Cheap exactly when it matters most: an
|
||||
/// empty cache yields an empty `read_dir`, so the heavy backfill diff costs zero
|
||||
/// per-item `stat`s instead of one (or more) per catalog row.
|
||||
pub fn snapshot_cover_disk(cover_root: &Path, server_index_key: &str) -> CoverDiskSnapshot {
|
||||
let server_dir = cover_cache_layout::cover_server_dir(cover_root, server_index_key);
|
||||
let mut snap = CoverDiskSnapshot::default();
|
||||
for kind in cover_cache_layout::SEGMENT_KINDS {
|
||||
let kind_dir = server_dir.join(kind);
|
||||
let Ok(entries) = std::fs::read_dir(&kind_dir) else {
|
||||
continue;
|
||||
};
|
||||
for ent in entries.flatten() {
|
||||
let path = ent.path();
|
||||
let Some(name) = path.file_name().and_then(|n| n.to_str()).map(str::to_string) else {
|
||||
continue;
|
||||
};
|
||||
if !path.is_dir() {
|
||||
continue;
|
||||
}
|
||||
let key = (kind.to_string(), name);
|
||||
if cover_cache_layout::entity_dir_has_canonical_tier(&path) {
|
||||
snap.present.insert(key.clone());
|
||||
}
|
||||
if cover_fetch_recently_failed(&path) {
|
||||
snap.failed.insert(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
snap
|
||||
}
|
||||
|
||||
/// Diff catalog rows against a pre-built disk snapshot → the subset still needing
|
||||
/// a download. No filesystem access here: the snapshot already captured disk
|
||||
/// state once. Rows whose raw id is cached/failed are skipped without expanding;
|
||||
/// the rest get `expand_backfill_items` (DB) to resolve multi-disc `mf-*` /
|
||||
/// artist entities, which are then diffed against the same snapshot.
|
||||
pub fn diff_missing_against_snapshot(
|
||||
store: &LibraryStore,
|
||||
library_server_id: &str,
|
||||
snapshot: &CoverDiskSnapshot,
|
||||
rows: Vec<CoverBackfillItem>,
|
||||
) -> Result<Vec<CoverBackfillItem>, String> {
|
||||
let mut out = Vec::new();
|
||||
for row in rows {
|
||||
if snapshot.is_cached(&row.cache_kind, &row.cache_entity_id)
|
||||
|| snapshot.is_recently_failed(&row.cache_kind, &row.cache_entity_id)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
for normalized in expand_backfill_items(store, library_server_id, row)? {
|
||||
if normalized.cache_entity_id.is_empty() {
|
||||
continue;
|
||||
}
|
||||
if snapshot.is_cached(&normalized.cache_kind, &normalized.cache_entity_id)
|
||||
|| snapshot.is_recently_failed(&normalized.cache_kind, &normalized.cache_entity_id)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
out.push(normalized);
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// One-shot worklist of every cover target still missing its canonical tier:
|
||||
/// DB catalog snapshot minus the on-disk snapshot. The worker streams the diff
|
||||
/// in chunks against a shared snapshot; tests use this whole-catalog form.
|
||||
pub fn collect_missing_cover_targets(
|
||||
store: &LibraryStore,
|
||||
library_server_id: &str,
|
||||
cover_root: &Path,
|
||||
server_index_key: &str,
|
||||
) -> Result<Vec<CoverBackfillItem>, String> {
|
||||
let rows = fetch_all_catalog_rows(store, library_server_id)?;
|
||||
let snapshot = snapshot_cover_disk(cover_root, server_index_key);
|
||||
diff_missing_against_snapshot(store, library_server_id, &snapshot, rows)
|
||||
}
|
||||
|
||||
pub fn count_distinct_cover_ids(store: &LibraryStore, library_server_id: &str) -> Result<i64, String> {
|
||||
store.with_read_conn(|conn| {
|
||||
let sql = format!(
|
||||
@@ -535,6 +687,29 @@ mod tests {
|
||||
let _ = std::fs::remove_dir_all(root.join(host));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn collect_missing_excludes_cached_includes_missing() {
|
||||
let store = LibraryStore::open_in_memory();
|
||||
seed_track(&store, "srv", "tr1", "al-have", None);
|
||||
seed_track(&store, "srv", "tr2", "al-need", None);
|
||||
let root = std::env::temp_dir().join("psysonic-missing-targets-test");
|
||||
let host = "srv-host";
|
||||
let have_dir = cover_cache_layout::cover_dir(&root, host, "album", "al-have");
|
||||
std::fs::create_dir_all(&have_dir).unwrap();
|
||||
std::fs::write(
|
||||
have_dir.join(format!("{LIBRARY_COVER_CANONICAL_TIER}.webp")),
|
||||
b"x",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let missing = collect_missing_cover_targets(&store, "srv", &root, host).unwrap();
|
||||
let ids: Vec<_> = missing.iter().map(|i| i.cache_entity_id.as_str()).collect();
|
||||
assert!(ids.contains(&"al-need"), "missing cover should be queued");
|
||||
assert!(!ids.contains(&"al-have"), "cached cover must be skipped");
|
||||
|
||||
let _ = std::fs::remove_dir_all(root.join(host));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn backfill_includes_per_disc_mf_when_discs_differ() {
|
||||
let store = LibraryStore::open_in_memory();
|
||||
|
||||
@@ -2,27 +2,50 @@
|
||||
|
||||
use super::{state, CoverCacheEnsureArgs, CoverCacheState};
|
||||
use psysonic_library::cover_backfill::{
|
||||
clear_cover_fetch_failures, collect_cover_backfill_batch, collect_cover_progress,
|
||||
LibraryCoverBackfillBatchDto, LIBRARY_COVER_CANONICAL_TIER,
|
||||
clear_cover_fetch_failures, collect_cover_progress, count_distinct_cover_ids,
|
||||
diff_missing_against_snapshot, fetch_all_catalog_rows, snapshot_cover_disk,
|
||||
LIBRARY_COVER_CANONICAL_TIER,
|
||||
};
|
||||
use psysonic_library::payload::LibrarySyncProgressPayload;
|
||||
use psysonic_library::repos::sync_state::SyncStateRepository;
|
||||
use psysonic_library::LibraryRuntime;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
use tauri::{AppHandle, Emitter, Listener, Manager};
|
||||
use tokio::sync::{Mutex, Semaphore};
|
||||
|
||||
use super::{count_cached_cover_ids, dir_usage_for_server};
|
||||
|
||||
/// Concurrent library downloads + encodes (hard cap — avoids saturating all CPU cores).
|
||||
const LIBRARY_BACKFILL_PARALLEL: usize = 2;
|
||||
const BATCH_SIZE: u32 = 24;
|
||||
const PENDING_RESTART_THRESHOLD: i64 = 32;
|
||||
/// Default concurrent library downloads + encodes. Runtime-tunable via the
|
||||
/// perf probe (`set_parallel`); the constant is only the startup value.
|
||||
const LIBRARY_BACKFILL_PARALLEL_DEFAULT: usize = 2;
|
||||
/// Bounds for the runtime knob — keep it sane so a stray value cannot DoS the
|
||||
/// host or starve the audio path.
|
||||
pub const LIBRARY_BACKFILL_PARALLEL_MIN: usize = 1;
|
||||
pub const LIBRARY_BACKFILL_PARALLEL_MAX: usize = 16;
|
||||
/// Raw catalog rows diffed per streaming chunk. Small enough that downloads
|
||||
/// start almost immediately after the one-shot enumeration, large enough to
|
||||
/// amortize the per-chunk `spawn_blocking` hop.
|
||||
const SCAN_CHUNK_ROWS: usize = 512;
|
||||
const SYNC_WAIT_MS: u64 = 5000;
|
||||
const PROGRESS_EVERY_BATCHES: u32 = 8;
|
||||
/// Cadence of the in-pass progress ticker (drives the "offline & cache" menu and
|
||||
/// the perf-probe overlay while a pass downloads). Only runs for the duration of
|
||||
/// an active pass.
|
||||
const PROGRESS_TICK_SECS: u64 = 3;
|
||||
/// Minimum gap between `library:sync-idle`-driven passes. Each such pass runs the
|
||||
/// idle-gate signature (a full cover-dir walk + DB count), so a chatty sync (e.g.
|
||||
/// periodic delta syncs) must not make that walk fire every few seconds. Manual
|
||||
/// runs and strategy-toggle wakes bypass this.
|
||||
const SYNC_IDLE_COOLDOWN_MS: u64 = 60_000;
|
||||
|
||||
fn now_ms() -> u64 {
|
||||
SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_millis() as u64)
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct CoverBackfillSession {
|
||||
@@ -33,6 +56,19 @@ pub struct CoverBackfillSession {
|
||||
pub password: String,
|
||||
}
|
||||
|
||||
/// Catalog signature captured when a full pass completes.
|
||||
///
|
||||
/// While it still matches, `library:sync-idle` must NOT re-trigger a rescan —
|
||||
/// mirrors the analysis coordinator's `completed_total` gate. Deliberately the
|
||||
/// **cheap** `COUNT(DISTINCT)` over the catalog only: a server change (track
|
||||
/// add/remove shifts `total`) re-arms the next pass, but checking it never
|
||||
/// touches the filesystem. A cover-cache clear leaves `total` unchanged, so the
|
||||
/// clear commands re-arm the gate explicitly (`rearm_idle_gate`).
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
struct CoverIdleSignature {
|
||||
total: i64,
|
||||
}
|
||||
|
||||
pub struct CoverBackfillWorker {
|
||||
pub enabled: AtomicBool,
|
||||
/// When true, the active pass yields so visible-route cover IPC is not starved.
|
||||
@@ -41,6 +77,15 @@ pub struct CoverBackfillWorker {
|
||||
cursor: Mutex<String>,
|
||||
pass_running: AtomicBool,
|
||||
backfill_http: Arc<Semaphore>,
|
||||
/// Live download/encode concurrency for backfill passes. Mirrors the
|
||||
/// `backfill_http` permit count and gates per-batch `ensure_one` tasks.
|
||||
parallel: AtomicUsize,
|
||||
/// Set when a pass found nothing pending; suppresses idle-driven rescans
|
||||
/// until the catalog signature changes. `None` means "re-armed".
|
||||
settled: Mutex<Option<CoverIdleSignature>>,
|
||||
/// Epoch-ms of the last `sync-idle`-driven pass, to rate-limit the idle-gate
|
||||
/// disk walk against chatty syncs. 0 = never.
|
||||
last_sync_idle_ms: AtomicU64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
@@ -75,7 +120,10 @@ impl CoverBackfillWorker {
|
||||
session: Mutex::new(None),
|
||||
cursor: Mutex::new(String::new()),
|
||||
pass_running: AtomicBool::new(false),
|
||||
backfill_http: Arc::new(Semaphore::new(LIBRARY_BACKFILL_PARALLEL)),
|
||||
backfill_http: Arc::new(Semaphore::new(LIBRARY_BACKFILL_PARALLEL_DEFAULT)),
|
||||
parallel: AtomicUsize::new(LIBRARY_BACKFILL_PARALLEL_DEFAULT),
|
||||
settled: Mutex::new(None),
|
||||
last_sync_idle_ms: AtomicU64::new(0),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -83,9 +131,49 @@ impl CoverBackfillWorker {
|
||||
self.ui_priority_hold.store(hold, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Re-arm the idle gate so the next opportunistic pass runs even though the
|
||||
/// catalog `total` is unchanged — used after a cover-cache clear, which
|
||||
/// drops files the cheap signature cannot see.
|
||||
pub async fn rearm_idle_gate(&self) {
|
||||
*self.settled.lock().await = None;
|
||||
self.last_sync_idle_ms.store(0, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Current backfill download/encode concurrency.
|
||||
pub fn parallel(&self) -> usize {
|
||||
self.parallel.load(Ordering::Relaxed).max(LIBRARY_BACKFILL_PARALLEL_MIN)
|
||||
}
|
||||
|
||||
/// Retune backfill concurrency at runtime. Resizes the shared HTTP permit
|
||||
/// pool to match (next batch picks up the new per-batch slot count). Returns
|
||||
/// the clamped value actually applied.
|
||||
pub fn set_parallel(&self, threads: usize) -> usize {
|
||||
let next = threads.clamp(LIBRARY_BACKFILL_PARALLEL_MIN, LIBRARY_BACKFILL_PARALLEL_MAX);
|
||||
let prev = self.parallel.swap(next, Ordering::SeqCst);
|
||||
if next > prev {
|
||||
self.backfill_http.add_permits(next - prev);
|
||||
} else if next < prev {
|
||||
// Shrinking: drain surplus permits as they free up so in-flight
|
||||
// fetches finish but no new ones start beyond the new cap.
|
||||
let sem = self.backfill_http.clone();
|
||||
let surplus = prev - next;
|
||||
tauri::async_runtime::spawn(async move {
|
||||
for _ in 0..surplus {
|
||||
if let Ok(permit) = sem.acquire().await {
|
||||
permit.forget();
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
next
|
||||
}
|
||||
|
||||
pub async fn set_session(&self, enabled: bool, session: Option<CoverBackfillSession>) {
|
||||
self.enabled.store(enabled, Ordering::Relaxed);
|
||||
*self.session.lock().await = session;
|
||||
// Server switch or enable/disable invalidates any settled state: re-arm
|
||||
// so the next idle event runs a real pass for the new focus.
|
||||
*self.settled.lock().await = None;
|
||||
if !enabled {
|
||||
*self.cursor.lock().await = String::new();
|
||||
}
|
||||
@@ -97,7 +185,7 @@ impl CoverBackfillWorker {
|
||||
|
||||
/// Semaphore-backed library backfill HTTP slots (perf probe).
|
||||
pub fn pipeline_http_stats(&self) -> (u32, u32, bool) {
|
||||
let max = LIBRARY_BACKFILL_PARALLEL as u32;
|
||||
let max = self.parallel() as u32;
|
||||
let active = max.saturating_sub(self.backfill_http.available_permits() as u32);
|
||||
let pass_running = self.pass_running.load(Ordering::Relaxed);
|
||||
(max, active, pass_running)
|
||||
@@ -187,7 +275,7 @@ async fn ensure_one(
|
||||
let _ = CoverCacheState::ensure_inner(&st, &app, &args, Some(http_sem)).await;
|
||||
}
|
||||
|
||||
async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>) {
|
||||
async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>, force: bool) {
|
||||
if !worker.enabled.load(Ordering::Relaxed) {
|
||||
return;
|
||||
}
|
||||
@@ -201,6 +289,14 @@ async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>) {
|
||||
None => return,
|
||||
};
|
||||
|
||||
// Opportunistic triggers (wake on track change, sync-idle) skip the whole
|
||||
// scan when a prior pass already settled and nothing changed — otherwise a
|
||||
// library with permanently-unfetchable covers (404s) would re-scan on every
|
||||
// wake forever. The manual "Run full pass now" sets `force` to bypass this.
|
||||
if !force && cover_idle_gate_should_skip(&app, &worker, &session).await {
|
||||
return;
|
||||
}
|
||||
|
||||
while !sync_allows_cover_backfill(&runtime.store, &session.library_server_id) {
|
||||
if !worker.enabled.load(Ordering::Relaxed) {
|
||||
return;
|
||||
@@ -220,125 +316,208 @@ async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>) {
|
||||
|
||||
worker.reset_cursor().await;
|
||||
let http_sem = worker.backfill_http.clone();
|
||||
let mut batch_count = 0u32;
|
||||
|
||||
// A forced pass is an explicit user retry: drop the `.fetch-failed` backoff
|
||||
// markers and the settled gate so previously-404'd covers are attempted
|
||||
// again. Opportunistic passes leave the markers in place (30-min TTL).
|
||||
if force {
|
||||
*worker.settled.lock().await = None;
|
||||
let root2 = root.clone();
|
||||
let index_key2 = session.server_index_key.clone();
|
||||
let _ = tauri::async_runtime::spawn_blocking(move || {
|
||||
clear_cover_fetch_failures(&root2, &index_key2)
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
// Two snapshots, taken ONCE per pass: the DB catalog (single GROUP BY) and
|
||||
// the on-disk cover bucket (one directory walk). The delta = catalog minus
|
||||
// disk, streamed in chunks below. No per-row `stat` on the filesystem and no
|
||||
// re-scan loop — pure set math against the captured disk snapshot.
|
||||
let (raw_rows, snapshot) = {
|
||||
let store = runtime.store.clone();
|
||||
let lib_id = session.library_server_id.clone();
|
||||
let root_for_scan = root.clone();
|
||||
let index_key = session.server_index_key.clone();
|
||||
match tauri::async_runtime::spawn_blocking(move || {
|
||||
let rows = fetch_all_catalog_rows(&store, &lib_id)?;
|
||||
let snap = snapshot_cover_disk(&root_for_scan, &index_key);
|
||||
Ok::<_, String>((rows, snap))
|
||||
})
|
||||
.await
|
||||
{
|
||||
Ok(Ok(pair)) => pair,
|
||||
_ => (Vec::new(), Default::default()),
|
||||
}
|
||||
};
|
||||
let snapshot = Arc::new(snapshot);
|
||||
|
||||
// Producer/consumer: a fixed pool of consumer tasks pulls misses off a
|
||||
// bounded channel and downloads them continuously, while the producer scans
|
||||
// the catalog in chunks and feeds misses in. This keeps the pool saturated
|
||||
// even when misses are sparse across chunks — no per-chunk drain barrier.
|
||||
// True concurrency stays governed by the resizable `http_sem` / encode
|
||||
// semaphores inside `ensure_one`, so the threads slider still applies live.
|
||||
let (tx, rx) =
|
||||
tokio::sync::mpsc::channel::<psysonic_library::cover_backfill::CoverBackfillItem>(256);
|
||||
let rx = Arc::new(Mutex::new(rx));
|
||||
let mut consumers = tokio::task::JoinSet::new();
|
||||
for _ in 0..LIBRARY_BACKFILL_PARALLEL_MAX {
|
||||
let rx = rx.clone();
|
||||
let st = st_arc.clone();
|
||||
let http_sem = http_sem.clone();
|
||||
let app = app.clone();
|
||||
let session = session.clone();
|
||||
let worker_arc = worker.clone();
|
||||
consumers.spawn(async move {
|
||||
loop {
|
||||
// Bail the moment the strategy flips to lazy / focus changes, so a
|
||||
// switch to "lazy" abandons the buffered backlog instead of
|
||||
// draining the whole channel (mirrors the producer's check).
|
||||
if !session_still_focused(&worker_arc, &session).await {
|
||||
break;
|
||||
}
|
||||
let item = {
|
||||
let mut guard = rx.lock().await;
|
||||
guard.recv().await
|
||||
};
|
||||
let Some(item) = item else { break };
|
||||
ensure_one(
|
||||
worker_arc.as_ref(),
|
||||
st.clone(),
|
||||
http_sem.clone(),
|
||||
app.clone(),
|
||||
session.clone(),
|
||||
item,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// Progress ticker: the producer finishes enumerating the worklist long before
|
||||
// the consumers finish downloading it, so emitting only while feeding would
|
||||
// freeze the "offline & cache" menu through the whole drain phase. Tick a
|
||||
// periodic snapshot for the lifetime of the pass instead; aborted once the
|
||||
// consumers drain (a final accurate emit happens at settle below).
|
||||
let progress_ticker = {
|
||||
let app = app.clone();
|
||||
let store = runtime.store.clone();
|
||||
let root = root.clone();
|
||||
let session = session.clone();
|
||||
let lib_id = session.library_server_id.clone();
|
||||
let index_key = session.server_index_key.clone();
|
||||
tauri::async_runtime::spawn(async move {
|
||||
loop {
|
||||
tokio::time::sleep(Duration::from_secs(PROGRESS_TICK_SECS)).await;
|
||||
if let Ok((done, total, pending)) =
|
||||
progress_snapshot(&store, &root, &lib_id, &index_key).await
|
||||
{
|
||||
emit_library_progress(&app, &session, done, total, pending, &root).await;
|
||||
}
|
||||
}
|
||||
})
|
||||
};
|
||||
|
||||
let mut rows_iter = raw_rows.into_iter();
|
||||
let mut completed = false;
|
||||
loop {
|
||||
if !session_still_focused(&worker, &session).await {
|
||||
break;
|
||||
}
|
||||
|
||||
if worker.ui_priority_hold.load(Ordering::Relaxed) {
|
||||
tokio::time::sleep(Duration::from_millis(200)).await;
|
||||
continue;
|
||||
}
|
||||
|
||||
let cursor = worker.cursor.lock().await.clone();
|
||||
let cursor_opt = if cursor.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(cursor)
|
||||
};
|
||||
let store = runtime.store.clone();
|
||||
let lib_id = session.library_server_id.clone();
|
||||
let index_key = session.server_index_key.clone();
|
||||
let root_for_batch = root.clone();
|
||||
let scan_chunk: Vec<_> = rows_iter.by_ref().take(SCAN_CHUNK_ROWS).collect();
|
||||
if scan_chunk.is_empty() {
|
||||
completed = true;
|
||||
break;
|
||||
}
|
||||
|
||||
let batch: Option<LibraryCoverBackfillBatchDto> =
|
||||
// Diff this chunk against the captured snapshot off-thread (in-memory set
|
||||
// math + DB expand only for rows not already cached) → misses to download.
|
||||
let missing: Vec<_> = {
|
||||
let store = runtime.store.clone();
|
||||
let lib_id = session.library_server_id.clone();
|
||||
let snapshot = snapshot.clone();
|
||||
match tauri::async_runtime::spawn_blocking(move || {
|
||||
collect_cover_backfill_batch(
|
||||
&store,
|
||||
&lib_id,
|
||||
&root_for_batch,
|
||||
&index_key,
|
||||
cursor_opt.as_deref(),
|
||||
Some(BATCH_SIZE),
|
||||
)
|
||||
diff_missing_against_snapshot(&store, &lib_id, &snapshot, scan_chunk)
|
||||
})
|
||||
.await
|
||||
{
|
||||
Ok(Ok(b)) => Some(b),
|
||||
_ => None,
|
||||
};
|
||||
|
||||
let Some(batch) = batch else {
|
||||
break;
|
||||
Ok(Ok(missing)) => missing,
|
||||
_ => Vec::new(),
|
||||
}
|
||||
};
|
||||
|
||||
batch_count += 1;
|
||||
if !session_still_focused(&worker, &session).await {
|
||||
break;
|
||||
}
|
||||
let items = batch.items.clone();
|
||||
let mut paused_for_ui_priority = false;
|
||||
let batch_slots = Arc::new(Semaphore::new(LIBRARY_BACKFILL_PARALLEL));
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for item in items {
|
||||
if worker.ui_priority_hold.load(Ordering::Relaxed) {
|
||||
paused_for_ui_priority = true;
|
||||
break;
|
||||
}
|
||||
let st = st_arc.clone();
|
||||
let http_sem = http_sem.clone();
|
||||
let app = app.clone();
|
||||
let session = session.clone();
|
||||
let worker_arc = worker.clone();
|
||||
let batch_slots = batch_slots.clone();
|
||||
set.spawn(async move {
|
||||
let Ok(_slot) = batch_slots.acquire().await else {
|
||||
return;
|
||||
};
|
||||
ensure_one(worker_arc.as_ref(), st, http_sem, app, session, item).await;
|
||||
});
|
||||
}
|
||||
while set.join_next().await.is_some() {}
|
||||
if paused_for_ui_priority || worker.ui_priority_hold.load(Ordering::Relaxed) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if batch_count.is_multiple_of(PROGRESS_EVERY_BATCHES) {
|
||||
if let Ok((done, total, pending)) = progress_snapshot(
|
||||
&runtime.store,
|
||||
&root,
|
||||
&session.library_server_id,
|
||||
&session.server_index_key,
|
||||
)
|
||||
.await
|
||||
{
|
||||
emit_library_progress(&app, &session, done, total, pending, &root).await;
|
||||
}
|
||||
}
|
||||
|
||||
if batch.exhausted {
|
||||
worker.cursor.lock().await.clear();
|
||||
if let Ok((done, total, pending)) = progress_snapshot(
|
||||
&runtime.store,
|
||||
&root,
|
||||
&session.library_server_id,
|
||||
&session.server_index_key,
|
||||
)
|
||||
.await
|
||||
{
|
||||
if pending > PENDING_RESTART_THRESHOLD {
|
||||
let root3 = root.clone();
|
||||
let index_key3 = session.server_index_key.clone();
|
||||
let _ = tauri::async_runtime::spawn_blocking(move || {
|
||||
clear_cover_fetch_failures(&root3, &index_key3)
|
||||
})
|
||||
.await;
|
||||
// Focus-aware feed: never park indefinitely on a full channel, or a
|
||||
// switch to lazy (which stops the consumers) would deadlock the producer
|
||||
// here. `try_send` + a short retry lets us re-check focus and bail.
|
||||
let mut feed_closed = false;
|
||||
'feed: for mut item in missing {
|
||||
loop {
|
||||
match tx.try_send(item) {
|
||||
Ok(()) => break,
|
||||
Err(tokio::sync::mpsc::error::TrySendError::Closed(_)) => {
|
||||
feed_closed = true;
|
||||
break 'feed;
|
||||
}
|
||||
Err(tokio::sync::mpsc::error::TrySendError::Full(returned)) => {
|
||||
if !session_still_focused(&worker, &session).await {
|
||||
feed_closed = true;
|
||||
break 'feed;
|
||||
}
|
||||
item = returned;
|
||||
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||
}
|
||||
}
|
||||
emit_library_progress(&app, &session, done, total, pending, &root).await;
|
||||
}
|
||||
}
|
||||
if feed_closed {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(next) = batch.next_cursor {
|
||||
*worker.cursor.lock().await = next;
|
||||
// Close the channel so consumers drain the remaining backlog and exit.
|
||||
drop(tx);
|
||||
while consumers.join_next().await.is_some() {}
|
||||
progress_ticker.abort();
|
||||
|
||||
// Only settle the idle gate on a natural finish (worklist drained), never on
|
||||
// a session-switch break — that belongs to the previous focus.
|
||||
if completed {
|
||||
worker.cursor.lock().await.clear();
|
||||
match progress_snapshot(
|
||||
&runtime.store,
|
||||
&root,
|
||||
&session.library_server_id,
|
||||
&session.server_index_key,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok((done, total, pending)) => {
|
||||
// Settle on a full scan regardless of `pending`: whatever is left
|
||||
// is unfetchable for now (404s with a fresh `.fetch-failed`
|
||||
// marker). The cheap `total` signature re-triggers a pass only if
|
||||
// the server catalog changes; a cache clear re-arms via the clear
|
||||
// command. This stops the wake storm on libraries whose covers
|
||||
// can never reach 100%.
|
||||
*worker.settled.lock().await = Some(CoverIdleSignature { total });
|
||||
emit_library_progress(&app, &session, done, total, pending, &root).await;
|
||||
}
|
||||
Err(_) => {
|
||||
*worker.settled.lock().await = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Start one full-catalog pass on the Tokio runtime (survives inactive webview).
|
||||
pub async fn try_schedule_full_pass(app: &AppHandle) -> bool {
|
||||
/// `force` bypasses the idle gate and clears fetch-failed backoff (explicit user
|
||||
/// retry); opportunistic callers (wake / sync-idle) pass `false`.
|
||||
pub async fn try_schedule_full_pass(app: &AppHandle, force: bool) -> bool {
|
||||
let worker = match app.try_state::<Arc<CoverBackfillWorker>>() {
|
||||
Some(w) => w.inner().clone(),
|
||||
None => return false,
|
||||
@@ -356,12 +535,44 @@ pub async fn try_schedule_full_pass(app: &AppHandle) -> bool {
|
||||
|
||||
let app = app.clone();
|
||||
tauri::async_runtime::spawn(async move {
|
||||
run_full_pass(app, worker.clone()).await;
|
||||
run_full_pass(app, worker.clone(), force).await;
|
||||
worker.pass_running.store(false, Ordering::SeqCst);
|
||||
});
|
||||
true
|
||||
}
|
||||
|
||||
/// Cheap catalog signature for the idle gate: a single `COUNT(DISTINCT)` over
|
||||
/// the cover catalog. No filesystem access — checking "did anything change on
|
||||
/// the server?" must never walk the on-disk cover cache.
|
||||
async fn current_cover_signature(
|
||||
app: &AppHandle,
|
||||
session: &CoverBackfillSession,
|
||||
) -> Option<CoverIdleSignature> {
|
||||
let runtime = app.try_state::<LibraryRuntime>()?;
|
||||
let store = runtime.store.clone();
|
||||
let lib_id = session.library_server_id.clone();
|
||||
tauri::async_runtime::spawn_blocking(move || {
|
||||
count_distinct_cover_ids(&store, &lib_id)
|
||||
.ok()
|
||||
.map(|total| CoverIdleSignature { total })
|
||||
})
|
||||
.await
|
||||
.ok()
|
||||
.flatten()
|
||||
}
|
||||
|
||||
/// True when the previous pass settled with nothing pending and the catalog
|
||||
/// still matches that signature — so an idle event need not rescan.
|
||||
async fn cover_idle_gate_should_skip(app: &AppHandle, worker: &CoverBackfillWorker, session: &CoverBackfillSession) -> bool {
|
||||
let Some(settled) = *worker.settled.lock().await else {
|
||||
return false;
|
||||
};
|
||||
match current_cover_signature(app, session).await {
|
||||
Some(current) => current == settled,
|
||||
None => false,
|
||||
}
|
||||
}
|
||||
|
||||
fn on_sync_idle(app: &AppHandle, payload: SyncIdlePayload) {
|
||||
if !payload.ok {
|
||||
return;
|
||||
@@ -382,7 +593,18 @@ fn on_sync_idle(app: &AppHandle, payload: SyncIdlePayload) {
|
||||
if !session_matches_server(&session, &payload.server_id) {
|
||||
return;
|
||||
}
|
||||
let _ = try_schedule_full_pass(&app).await;
|
||||
// Rate-limit sync-idle passes: each runs the idle-gate disk walk, so a
|
||||
// chatty sync must not trigger it every few seconds. The gate inside the
|
||||
// pass still skips the actual rescan when nothing changed.
|
||||
let now = now_ms();
|
||||
let last = worker.last_sync_idle_ms.load(Ordering::Relaxed);
|
||||
if last != 0 && now.saturating_sub(last) < SYNC_IDLE_COOLDOWN_MS {
|
||||
return;
|
||||
}
|
||||
worker.last_sync_idle_ms.store(now, Ordering::Relaxed);
|
||||
// Opportunistic: the gate (checked inside the pass) skips the rescan when
|
||||
// a prior pass settled and nothing changed (mirrors the analysis gate).
|
||||
let _ = try_schedule_full_pass(&app, false).await;
|
||||
});
|
||||
}
|
||||
|
||||
@@ -399,7 +621,7 @@ pub fn setup_library_sync_idle_listener(app: &AppHandle) {
|
||||
|
||||
/// Legacy single-step API (optional diagnostics).
|
||||
pub async fn pulse_backfill(app: &AppHandle, _worker: &Arc<CoverBackfillWorker>) -> CoverBackfillPulseDto {
|
||||
if try_schedule_full_pass(app).await {
|
||||
if try_schedule_full_pass(app, false).await {
|
||||
return CoverBackfillPulseDto {
|
||||
scheduled: 0,
|
||||
exhausted: false,
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
use reqwest::Client;
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
use url::Url;
|
||||
|
||||
const SUBSONIC_CLIENT: &str = "Psysonic";
|
||||
|
||||
/// Total cover fetch attempts (1 initial + retries). A busy server can answer
|
||||
/// covers with 5xx/429/timeouts under our own backfill load, so a couple of
|
||||
/// backed-off retries recover those without a permanent `.fetch-failed` marker.
|
||||
const COVER_FETCH_ATTEMPTS: usize = 3;
|
||||
/// Base backoff between attempts (grows linearly: 1×, 2×, …).
|
||||
const COVER_FETCH_BACKOFF_MS: u64 = 400;
|
||||
|
||||
fn random_salt() -> String {
|
||||
let nanos = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
@@ -46,16 +53,54 @@ pub fn build_cover_art_url(
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_cover_bytes(client: &Client, url: &str) -> Result<Vec<u8>, String> {
|
||||
let resp = client
|
||||
.get(url)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| e.to_string())?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(format!("cover HTTP {}", resp.status()));
|
||||
/// Outcome of a single fetch attempt: transient errors are worth retrying,
|
||||
/// permanent ones (a real 4xx like 404 — the cover simply does not exist) are
|
||||
/// not, so we never hammer the server for genuinely-missing art.
|
||||
enum FetchAttempt {
|
||||
Ok(Vec<u8>),
|
||||
Transient(String),
|
||||
Permanent(String),
|
||||
}
|
||||
|
||||
async fn fetch_cover_once(client: &Client, url: &str) -> FetchAttempt {
|
||||
let resp = match client.get(url).send().await {
|
||||
Ok(r) => r,
|
||||
// Connection reset / timeout / DNS — transient under server load.
|
||||
Err(e) => return FetchAttempt::Transient(e.to_string()),
|
||||
};
|
||||
let status = resp.status();
|
||||
if status.is_success() {
|
||||
return match resp.bytes().await {
|
||||
Ok(b) => FetchAttempt::Ok(b.to_vec()),
|
||||
Err(e) => FetchAttempt::Transient(e.to_string()),
|
||||
};
|
||||
}
|
||||
resp.bytes().await.map(|b| b.to_vec()).map_err(|e| e.to_string())
|
||||
let msg = format!("cover HTTP {status}");
|
||||
if status.is_server_error() || status == reqwest::StatusCode::TOO_MANY_REQUESTS {
|
||||
FetchAttempt::Transient(msg)
|
||||
} else {
|
||||
FetchAttempt::Permanent(msg)
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_cover_bytes(client: &Client, url: &str) -> Result<Vec<u8>, String> {
|
||||
let mut last_err = String::from("cover fetch failed");
|
||||
for attempt in 0..COVER_FETCH_ATTEMPTS {
|
||||
match fetch_cover_once(client, url).await {
|
||||
FetchAttempt::Ok(bytes) => return Ok(bytes),
|
||||
FetchAttempt::Permanent(e) => return Err(e),
|
||||
FetchAttempt::Transient(e) => {
|
||||
last_err = e;
|
||||
if attempt + 1 < COVER_FETCH_ATTEMPTS {
|
||||
tokio::time::sleep(Duration::from_millis(
|
||||
COVER_FETCH_BACKOFF_MS * (attempt as u64 + 1),
|
||||
))
|
||||
.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(last_err)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -24,6 +24,7 @@ use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use std::io::Cursor;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use tokio::sync::{Mutex, Semaphore};
|
||||
@@ -79,10 +80,10 @@ pub(crate) fn cover_pipeline_queue_stats(
|
||||
http_active: sem_active(&cache.http_sem, COVER_HTTP_CONCURRENCY as u32),
|
||||
cpu_ui_max: COVER_CPU_UI_CONCURRENCY as u32,
|
||||
cpu_ui_active: sem_active(&cache.cover_cpu_ui_sem, COVER_CPU_UI_CONCURRENCY as u32),
|
||||
cpu_backfill_max: COVER_CPU_BACKFILL_CONCURRENCY as u32,
|
||||
cpu_backfill_max: cache.cover_backfill_cpu_parallel() as u32,
|
||||
cpu_backfill_active: sem_active(
|
||||
&cache.cover_cpu_backfill_sem,
|
||||
COVER_CPU_BACKFILL_CONCURRENCY as u32,
|
||||
cache.cover_backfill_cpu_parallel() as u32,
|
||||
),
|
||||
library_backfill_http_max,
|
||||
library_backfill_http_active,
|
||||
@@ -117,7 +118,10 @@ const COVER_HTTP_CONCURRENCY: usize = 16;
|
||||
/// UI-visible decode + WebP encode (grid, hero, player) — not shared with library backfill.
|
||||
const COVER_CPU_UI_CONCURRENCY: usize = 2;
|
||||
/// Library backfill encode ladder — separate pool so bulk warm-up cannot starve the webview.
|
||||
/// Default only; runtime-tunable from the perf probe via `set_backfill_cpu_parallel`.
|
||||
const COVER_CPU_BACKFILL_CONCURRENCY: usize = 2;
|
||||
/// Upper bound for the runtime encode-pool knob (matches the worker cap).
|
||||
const COVER_CPU_BACKFILL_MAX: usize = 16;
|
||||
|
||||
pub struct CoverCacheState {
|
||||
pub root: PathBuf,
|
||||
@@ -128,6 +132,9 @@ pub struct CoverCacheState {
|
||||
pub http_sem: Arc<Semaphore>,
|
||||
pub cover_cpu_ui_sem: Arc<Semaphore>,
|
||||
pub cover_cpu_backfill_sem: Arc<Semaphore>,
|
||||
/// Live permit count of `cover_cpu_backfill_sem` (the semaphore itself only
|
||||
/// exposes *available* permits, not the configured ceiling).
|
||||
cover_cpu_backfill_max: AtomicUsize,
|
||||
}
|
||||
|
||||
impl CoverCacheState {
|
||||
@@ -147,9 +154,35 @@ impl CoverCacheState {
|
||||
http_sem: Arc::new(Semaphore::new(COVER_HTTP_CONCURRENCY)),
|
||||
cover_cpu_ui_sem: Arc::new(Semaphore::new(COVER_CPU_UI_CONCURRENCY)),
|
||||
cover_cpu_backfill_sem: Arc::new(Semaphore::new(COVER_CPU_BACKFILL_CONCURRENCY)),
|
||||
cover_cpu_backfill_max: AtomicUsize::new(COVER_CPU_BACKFILL_CONCURRENCY),
|
||||
})
|
||||
}
|
||||
|
||||
/// Current configured ceiling of the backfill encode pool.
|
||||
pub fn cover_backfill_cpu_parallel(&self) -> usize {
|
||||
self.cover_cpu_backfill_max.load(Ordering::Relaxed).max(1)
|
||||
}
|
||||
|
||||
/// Retune the backfill encode pool to match the worker's download
|
||||
/// concurrency. Grows/shrinks the semaphore permits in place.
|
||||
pub fn set_backfill_cpu_parallel(&self, threads: usize) {
|
||||
let next = threads.clamp(1, COVER_CPU_BACKFILL_MAX);
|
||||
let prev = self.cover_cpu_backfill_max.swap(next, Ordering::SeqCst);
|
||||
if next > prev {
|
||||
self.cover_cpu_backfill_sem.add_permits(next - prev);
|
||||
} else if next < prev {
|
||||
let sem = self.cover_cpu_backfill_sem.clone();
|
||||
let surplus = prev - next;
|
||||
tauri::async_runtime::spawn(async move {
|
||||
for _ in 0..surplus {
|
||||
if let Ok(permit) = sem.acquire().await {
|
||||
permit.forget();
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
fn cpu_sem_for(&self, library_bulk: bool) -> Arc<Semaphore> {
|
||||
if library_bulk {
|
||||
self.cover_cpu_backfill_sem.clone()
|
||||
@@ -162,11 +195,6 @@ impl CoverCacheState {
|
||||
("ok".into(), true)
|
||||
}
|
||||
|
||||
fn pressure(&self) -> (String, bool) {
|
||||
let (bytes, _) = dir_usage_at_root(&self.root);
|
||||
self.pressure_from_bytes(bytes)
|
||||
}
|
||||
|
||||
pub(crate) async fn ensure_inner(
|
||||
state: &Arc<Mutex<CoverCacheState>>,
|
||||
app: &AppHandle,
|
||||
@@ -183,7 +211,11 @@ impl CoverCacheState {
|
||||
});
|
||||
}
|
||||
|
||||
let (_, auto_dl) = this.pressure();
|
||||
// Cheap, no-IO gate. Previously this ran a full recursive disk walk of
|
||||
// the entire cover cache (`pressure()` → `dir_usage_at_root`) on every
|
||||
// ensure, under the global state lock — serializing the whole backfill
|
||||
// pool onto filesystem stat work. The walked bytes were then discarded.
|
||||
let (_, auto_dl) = this.pressure_from_bytes(0);
|
||||
if !auto_dl && args.tier != 2000 {
|
||||
return Ok(CoverCacheEnsureResult {
|
||||
hit: false,
|
||||
@@ -453,6 +485,42 @@ pub(crate) fn dir_usage_for_server(root: &Path, server_index_key: &str) -> (u64,
|
||||
server_cover_disk_usage(&cover_server_dir(root, server_index_key))
|
||||
}
|
||||
|
||||
/// TTL-memoized per-server cover dir walk. The "offline & cache" settings menu
|
||||
/// polls byte usage + cached count every few seconds for every server; on a full
|
||||
/// cache that is several full directory walks per tick. Reuse a recent walk so we
|
||||
/// don't re-stat thousands of files when nothing has changed. Active backfill still
|
||||
/// pushes live numbers through the `cover:library-progress` event, so a short TTL
|
||||
/// only de-dupes the idle polling, it does not hide real progress.
|
||||
const DIR_USAGE_CACHE_TTL: Duration = Duration::from_secs(10);
|
||||
|
||||
type DirUsageCache = std::sync::Mutex<HashMap<String, (std::time::Instant, (u64, u64))>>;
|
||||
|
||||
fn dir_usage_cache() -> &'static DirUsageCache {
|
||||
static CACHE: std::sync::OnceLock<DirUsageCache> = std::sync::OnceLock::new();
|
||||
CACHE.get_or_init(|| std::sync::Mutex::new(HashMap::new()))
|
||||
}
|
||||
|
||||
pub(crate) fn cached_dir_usage_for_server(root: &Path, server_index_key: &str) -> (u64, u64) {
|
||||
if let Ok(map) = dir_usage_cache().lock() {
|
||||
if let Some((at, value)) = map.get(server_index_key) {
|
||||
if at.elapsed() < DIR_USAGE_CACHE_TTL {
|
||||
return *value;
|
||||
}
|
||||
}
|
||||
}
|
||||
let value = dir_usage_for_server(root, server_index_key);
|
||||
if let Ok(mut map) = dir_usage_cache().lock() {
|
||||
map.insert(server_index_key.to_string(), (std::time::Instant::now(), value));
|
||||
}
|
||||
value
|
||||
}
|
||||
|
||||
pub(crate) fn invalidate_dir_usage_cache(server_index_key: &str) {
|
||||
if let Ok(mut map) = dir_usage_cache().lock() {
|
||||
map.remove(server_index_key);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn dir_usage_at_root(root: &Path) -> (u64, u64) {
|
||||
cover_root_disk_usage(root)
|
||||
}
|
||||
@@ -512,9 +580,12 @@ pub fn init_cover_cache(app: &AppHandle) -> Result<(), String> {
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
pub async fn library_cover_backfill_run_full_pass(app: AppHandle) -> Result<CoverBackfillRunDto, String> {
|
||||
pub async fn library_cover_backfill_run_full_pass(
|
||||
app: AppHandle,
|
||||
force: Option<bool>,
|
||||
) -> Result<CoverBackfillRunDto, String> {
|
||||
Ok(CoverBackfillRunDto {
|
||||
started: try_schedule_full_pass(&app).await,
|
||||
started: try_schedule_full_pass(&app, force.unwrap_or(false)).await,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -548,6 +619,24 @@ pub async fn library_cover_backfill_set_ui_priority(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Perf-probe tuning knob: set how many threads cover backfill uses (download
|
||||
/// + encode pools move together). Not exposed in app Settings by design.
|
||||
/// Returns the clamped value actually applied.
|
||||
#[tauri::command]
|
||||
pub async fn library_cover_backfill_set_parallel(
|
||||
app: AppHandle,
|
||||
threads: usize,
|
||||
) -> Result<u32, String> {
|
||||
let worker = app
|
||||
.try_state::<Arc<CoverBackfillWorker>>()
|
||||
.ok_or_else(|| "cover backfill worker not initialized".to_string())?;
|
||||
let applied = worker.set_parallel(threads);
|
||||
if let Ok(cache) = state(&app) {
|
||||
cache.lock().await.set_backfill_cpu_parallel(applied);
|
||||
}
|
||||
Ok(applied as u32)
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
pub async fn library_cover_backfill_configure(
|
||||
app: AppHandle,
|
||||
@@ -576,7 +665,7 @@ pub async fn library_cover_backfill_configure(
|
||||
.set_session(enabled && session.is_some(), session)
|
||||
.await;
|
||||
if enabled {
|
||||
let _ = try_schedule_full_pass(&app).await;
|
||||
let _ = try_schedule_full_pass(&app, false).await;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -706,7 +795,7 @@ pub async fn cover_cache_stats_server(
|
||||
) -> Result<CoverCacheStatsDto, String> {
|
||||
let st = state(&app)?;
|
||||
let guard = st.lock().await;
|
||||
let (bytes, entry_count) = dir_usage_for_server(&guard.root, &server_index_key);
|
||||
let (bytes, entry_count) = cached_dir_usage_for_server(&guard.root, &server_index_key);
|
||||
let (pressure, auto_download_enabled) = guard.pressure_from_bytes(bytes);
|
||||
Ok(CoverCacheStatsDto {
|
||||
bytes,
|
||||
@@ -741,7 +830,13 @@ pub async fn cover_cache_clear_server(
|
||||
if path.is_dir() {
|
||||
std::fs::remove_dir_all(&path).map_err(|e| e.to_string())?;
|
||||
}
|
||||
invalidate_dir_usage_cache(&server_index_key);
|
||||
drop(guard);
|
||||
// Clearing drops files the cheap idle-gate signature can't see, so re-arm
|
||||
// the backfill worker — otherwise the next sync-idle would skip the rescan.
|
||||
if let Some(worker) = app.try_state::<Arc<CoverBackfillWorker>>() {
|
||||
worker.rearm_idle_gate().await;
|
||||
}
|
||||
let _ = app.emit(
|
||||
"cover:cache-cleared",
|
||||
serde_json::json!({ "serverIndexKey": server_index_key }),
|
||||
@@ -904,6 +999,12 @@ pub async fn cover_cache_clear(app: AppHandle) -> Result<(), String> {
|
||||
}
|
||||
}
|
||||
drop(guard);
|
||||
if let Ok(mut map) = dir_usage_cache().lock() {
|
||||
map.clear();
|
||||
}
|
||||
if let Some(worker) = app.try_state::<Arc<CoverBackfillWorker>>() {
|
||||
worker.rearm_idle_gate().await;
|
||||
}
|
||||
let _ = app.emit("cover:cache-cleared", serde_json::json!({}));
|
||||
Ok(())
|
||||
}
|
||||
@@ -956,7 +1057,7 @@ pub async fn library_cover_progress(
|
||||
let index_key = server_index_key.clone();
|
||||
let store = runtime.store.clone();
|
||||
tauri::async_runtime::spawn_blocking(move || {
|
||||
let cached_dirs = count_cached_cover_ids(&root, &index_key);
|
||||
let cached_dirs = cached_dir_usage_for_server(&root, &index_key).1 as i64;
|
||||
collect_cover_progress(
|
||||
&store,
|
||||
&library_server_id,
|
||||
|
||||
@@ -762,6 +762,7 @@ pub fn run() {
|
||||
cover_cache::library_cover_backfill_pulse,
|
||||
cover_cache::library_cover_backfill_reset_cursor,
|
||||
cover_cache::library_cover_backfill_set_ui_priority,
|
||||
cover_cache::library_cover_backfill_set_parallel,
|
||||
cover_cache::library_cover_backfill_run_full_pass,
|
||||
cover_cache::cover_revalidate_enqueue,
|
||||
cover_cache::cover_revalidate_tick,
|
||||
|
||||
+14
-3
@@ -223,9 +223,15 @@ export async function libraryCoverBackfillPulse(): Promise<CoverBackfillPulseRes
|
||||
return invoke<CoverBackfillPulseResult>('library_cover_backfill_pulse');
|
||||
}
|
||||
|
||||
/** Start one full-catalog pass on the native runtime (works when the window is inactive). */
|
||||
export async function libraryCoverBackfillRunFullPass(): Promise<{ started: boolean }> {
|
||||
return invoke<{ started: boolean }>('library_cover_backfill_run_full_pass');
|
||||
/**
|
||||
* Start one full-catalog pass on the native runtime (works when the window is inactive).
|
||||
* `force` bypasses the idle gate and clears the fetch-failed backoff so previously
|
||||
* unfetchable (404) covers are retried — used by the manual "Run full pass now".
|
||||
*/
|
||||
export async function libraryCoverBackfillRunFullPass(
|
||||
force = false,
|
||||
): Promise<{ started: boolean }> {
|
||||
return invoke<{ started: boolean }>('library_cover_backfill_run_full_pass', { force });
|
||||
}
|
||||
|
||||
export async function libraryCoverBackfillResetCursor(): Promise<void> {
|
||||
@@ -237,6 +243,11 @@ export async function libraryCoverBackfillSetUiPriority(hold: boolean): Promise<
|
||||
return invoke('library_cover_backfill_set_ui_priority', { hold });
|
||||
}
|
||||
|
||||
/** Perf-probe only: retune cover backfill threads (download + encode). Returns the clamped value applied. */
|
||||
export async function libraryCoverBackfillSetParallel(threads: number): Promise<number> {
|
||||
return invoke<number>('library_cover_backfill_set_parallel', { threads });
|
||||
}
|
||||
|
||||
export async function libraryCoverClearFetchFailures(serverIndexKey: string): Promise<number> {
|
||||
return invoke<number>('library_cover_clear_fetch_failures', { serverIndexKey });
|
||||
}
|
||||
|
||||
@@ -79,9 +79,14 @@ export default function CoverCacheStrategySection() {
|
||||
);
|
||||
}, [servers, refreshRow]);
|
||||
|
||||
// Recompute on entry only. Live updates during an active backfill arrive via the
|
||||
// `cover:library-progress` event (carries done/total/pending/bytes/entryCount), and
|
||||
// clearing the cache emits `cover:cache-cleared`. A slow 5-minute tick is just a
|
||||
// safety net for changes made outside this view (e.g. browsing-time caching); it is
|
||||
// not needed for correctness, so we avoid re-walking the cover dirs in a tight loop.
|
||||
useEffect(() => {
|
||||
refreshAll();
|
||||
const id = window.setInterval(refreshAll, 15_000);
|
||||
const id = window.setInterval(refreshAll, 300_000);
|
||||
return () => window.clearInterval(id);
|
||||
}, [refreshAll]);
|
||||
|
||||
@@ -128,9 +133,6 @@ export default function CoverCacheStrategySection() {
|
||||
refreshAll();
|
||||
}
|
||||
}));
|
||||
unsubs.push(await listen('cover:tier-ready', () => {
|
||||
refreshAll();
|
||||
}));
|
||||
})();
|
||||
return () => {
|
||||
for (const u of unsubs) u();
|
||||
@@ -173,6 +175,9 @@ export default function CoverCacheStrategySection() {
|
||||
await coverCacheClearServer(clearTarget.indexKey);
|
||||
clearDiskSrcCacheForServer(clearTarget.indexKey);
|
||||
await refreshRow(clearTarget.serverId, clearTarget.indexKey);
|
||||
if (clearTarget.serverId === activeServerId) {
|
||||
wakeLibraryCoverBackfill();
|
||||
}
|
||||
showToast(t('settings.coverCacheStrategyClearSuccess'), 4000, 'success');
|
||||
} catch {
|
||||
showToast(t('settings.coverCacheStrategyClearError'), 5000, 'error');
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import {
|
||||
coverGetPipelineQueueStats,
|
||||
libraryCoverBackfillResetCursor,
|
||||
libraryCoverBackfillRunFullPass,
|
||||
libraryCoverBackfillSetParallel,
|
||||
} from '../../../api/coverCache';
|
||||
|
||||
const COVER_THREADS_MIN = 1;
|
||||
const COVER_THREADS_MAX = 16;
|
||||
|
||||
/**
|
||||
* Perf-probe-only knob for cover backfill concurrency (download + encode pools
|
||||
* move together). Deliberately not surfaced in app Settings — it is a live
|
||||
* diagnostics/experiment control. The value is process-local and resets to the
|
||||
* backend default on app restart.
|
||||
*/
|
||||
export default function PerfCoverThreadsControl() {
|
||||
const [threads, setThreads] = useState<number | null>(null);
|
||||
const pendingRef = useRef(false);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
void coverGetPipelineQueueStats()
|
||||
.then(stats => {
|
||||
if (!cancelled) setThreads(stats.libraryBackfillHttpMax || COVER_THREADS_MIN);
|
||||
})
|
||||
.catch(() => {
|
||||
if (!cancelled) setThreads(COVER_THREADS_MIN);
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, []);
|
||||
|
||||
const apply = (next: number) => {
|
||||
setThreads(next);
|
||||
if (pendingRef.current) return;
|
||||
pendingRef.current = true;
|
||||
void libraryCoverBackfillSetParallel(next)
|
||||
.then(applied => setThreads(applied))
|
||||
.catch(() => {})
|
||||
.finally(() => {
|
||||
pendingRef.current = false;
|
||||
});
|
||||
};
|
||||
|
||||
const value = threads ?? COVER_THREADS_MIN;
|
||||
const [running, setRunning] = useState(false);
|
||||
|
||||
const runPass = () => {
|
||||
if (running) return;
|
||||
setRunning(true);
|
||||
void libraryCoverBackfillResetCursor()
|
||||
.then(() => libraryCoverBackfillRunFullPass(true))
|
||||
.catch(() => {})
|
||||
.finally(() => setRunning(false));
|
||||
};
|
||||
|
||||
return (
|
||||
<section className="perf-live-poll" aria-label="Cover backfill threads">
|
||||
<div className="perf-live-poll__title">Cover backfill</div>
|
||||
<label className="perf-live-poll__row">
|
||||
<span className="perf-live-poll__label">
|
||||
Threads
|
||||
{' '}
|
||||
<span className="perf-live-poll__value">{value}</span>
|
||||
</span>
|
||||
<input
|
||||
type="range"
|
||||
min={COVER_THREADS_MIN}
|
||||
max={COVER_THREADS_MAX}
|
||||
step={1}
|
||||
value={value}
|
||||
disabled={threads === null}
|
||||
onChange={e => apply(Number(e.target.value))}
|
||||
/>
|
||||
</label>
|
||||
<button
|
||||
type="button"
|
||||
className="perf-live-poll__run"
|
||||
onClick={runPass}
|
||||
disabled={running}
|
||||
>
|
||||
{running ? 'Starting…' : 'Run full pass now'}
|
||||
</button>
|
||||
<p className="perf-live-poll__hint">
|
||||
Download + encode concurrency for background cover warm-up. Higher = faster
|
||||
backfill but more CPU/network while it runs. Resets to default on restart.
|
||||
“lib …/N” in the cover overlay only fills when covers are actually missing —
|
||||
clear a server’s cover cache first, then run a pass.
|
||||
</p>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import PerfProbeMetricCard, { PerfProbeMetricSection } from './PerfProbeMetricCa
|
||||
import PerfOverlayAppearanceControls from './PerfOverlayAppearanceControls';
|
||||
import PerfOverlayModeControls from './PerfOverlayModeControls';
|
||||
import PerfLivePollControls from './PerfLivePollControls';
|
||||
import PerfCoverThreadsControl from './PerfCoverThreadsControl';
|
||||
|
||||
function memoryBarPct(rssKb: number, maxKb: number): number {
|
||||
if (maxKb <= 0) return 0;
|
||||
@@ -60,6 +61,7 @@ export default function SidebarPerfProbeMonitorTab() {
|
||||
<PerfOverlayModeControls />
|
||||
<PerfOverlayAppearanceControls />
|
||||
<PerfLivePollControls />
|
||||
<PerfCoverThreadsControl />
|
||||
<PerfProbeMetricSection title="Pipeline overlays" hint="Rust / UI queues">
|
||||
<PerfProbeMetricCard
|
||||
label="FPS"
|
||||
|
||||
@@ -144,6 +144,7 @@ const CONTRIBUTOR_ENTRIES = [
|
||||
'Genres: local index genre browse with Subsonic fallback, aligned counts cache, scroll restore, hold-to-shuffle play (PR #937)',
|
||||
'Live Search: scoped browse on Artists, Albums, New Releases, Tracks, and Composers — header badge, ghost restore, album title FTS, session stash (PR #938)',
|
||||
'Performance: idle Rust CPU — backfill coordinator park, probe overlay stability, throttled idle polls, lazy cover prefetch restore (PR #939)',
|
||||
'Cover backfill: disk-free idle gate, snapshot-diff worklist, live-tunable parallelism, transient-error retries, and memoized offline & cache stats to stop idle CPU spin (PR #943)',
|
||||
],
|
||||
},
|
||||
{
|
||||
|
||||
@@ -206,6 +206,27 @@
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.perf-live-poll__run {
|
||||
margin-top: 10px;
|
||||
align-self: flex-start;
|
||||
padding: 4px 10px;
|
||||
font-size: 11px;
|
||||
border-radius: 6px;
|
||||
border: 1px solid var(--border-subtle, color-mix(in srgb, var(--text-muted) 30%, transparent));
|
||||
background: color-mix(in srgb, var(--text-muted) 12%, transparent);
|
||||
color: var(--text-primary);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.perf-live-poll__run:disabled {
|
||||
opacity: 0.6;
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
.perf-live-poll__run:not(:disabled):hover {
|
||||
background: color-mix(in srgb, var(--text-muted) 20%, transparent);
|
||||
}
|
||||
|
||||
.perf-live-poll__hint {
|
||||
margin: 6px 0 0;
|
||||
font-size: 10px;
|
||||
|
||||
Reference in New Issue
Block a user