fix(cover-backfill): kill idle CPU spin and offline-cache menu re-walks (#943)

* fix(cover-backfill): snapshot-diff worklist and live-tunable parallelism

Aggressive cover backfill pegged one tokio worker at ~100% on large,
fully-synced libraries while the download queues stayed empty.

- Take two snapshots once per pass — the DB catalog (single GROUP BY) and
  the on-disk cover bucket (one directory walk) — and download the
  set-difference. No per-row `stat` syscalls and no re-scan loop; the empty
  cache case (heavy backfill) costs zero per-item disk hits.
- Replace the front-loaded enumeration with a producer/consumer pipeline:
  the producer streams the catalog in chunks and feeds misses into a bounded
  channel; a fixed consumer pool keeps the download/encode pools saturated.
- Make cover backfill parallelism runtime-tunable from the Performance Probe
  (threads slider + "Run full pass now"); HTTP download and CPU encode
  semaphores resize live. Not surfaced in app settings.
- Add a "nothing changed" idle gate (catalog signature) so a settled pass is
  not re-run on every library:sync-idle, mirroring the analysis worker.
- Cancel promptly on switch to lazy: consumers bail on enabled/focus change
  and the producer feeds via try_send so a full channel cannot deadlock.
- Drop the per-item recursive disk walk from the ensure hot path.

* fix(cover-backfill): cheap idle gate, settle on 404s, transient retries

Follow-up to the snapshot-diff backfill: stop the periodic CPU spikes and the
89%-plateau wake storm on libraries whose covers can never reach 100%.

- Idle gate is now disk-free: compare only the catalog COUNT(DISTINCT) instead
  of walking ~all cover dirs on every sync-idle. "Did the server change?" never
  touches the filesystem. Clear-cache commands re-arm the gate (rearm_idle_gate)
  since a clear leaves the catalog total unchanged, and the settings UI wakes the
  active server after a clear.
- Settle the gate on any completed pass regardless of pending: remaining items
  are unfetchable-for-now (404), so the wake/sync-idle storm stops once the
  fetchable set is exhausted.
- Stop auto-clearing .fetch-failed markers every pass (it defeated the 30-min
  backoff and re-attempted 404s forever). The manual "Run full pass now" sends
  force=true to clear them and retry; wake/sync-idle/configure stay opportunistic.
- Rate-limit sync-idle passes (60s cooldown) as defence against chatty syncs.
- Retry cover downloads up to 3x with backoff on transient failures (5xx / 429 /
  network), but never on a real 4xx so missing covers don't hammer the server.

* fix(cover-cache): stop re-walking cover dirs from offline & cache menu

The settings cover-cache section polled disk usage + progress every 15s for
every server, each call doing a full recursive walk of the per-server cover
directory. On a fully populated cache this caused periodic CPU spikes whenever
that menu was open.

- mod.rs: add a 10s TTL memo around the per-server cover dir walk
  (cached_dir_usage_for_server), shared by cover_cache_stats_server and
  library_cover_progress; invalidate on clear (per-server and clear-all).
- CoverCacheStrategySection: recompute on entry only; rely on the
  cover:library-progress and cover:cache-cleared events for live updates;
  drop the per-cover cover:tier-ready refresh storm; turn the 15s loop into a
  5-minute safety net.

* fix(cover-backfill): keep emitting progress during the whole pass

The producer finishes enumerating the worklist long before the consumer pool
finishes downloading it, so progress was only emitted while feeding the channel
— the "offline & cache" menu and overlay then froze through the entire drain
phase. Replace the per-chunk emit with a 3s progress ticker that runs for the
lifetime of the pass and is aborted once the consumers drain (final accurate
emit still happens at settle).

* docs(changelog): record cover-backfill idle-CPU fix (PR #943)

Add [1.47.0] Fixed + Changed entries and a settingsCredits line for the
cover-backfill idle CPU / offline & cache menu work.
This commit is contained in:
cucadmuh
2026-06-02 04:56:34 +03:00
committed by GitHub
parent 5e977cfd49
commit a63ba3c9cb
12 changed files with 832 additions and 138 deletions
+55 -10
View File
@@ -1,9 +1,16 @@
use reqwest::Client;
use std::time::{SystemTime, UNIX_EPOCH};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use url::Url;
const SUBSONIC_CLIENT: &str = "Psysonic";
/// Total cover fetch attempts (1 initial + retries). A busy server can answer
/// covers with 5xx/429/timeouts under our own backfill load, so a couple of
/// backed-off retries recover those without a permanent `.fetch-failed` marker.
const COVER_FETCH_ATTEMPTS: usize = 3;
/// Base backoff between attempts (grows linearly: 1×, 2×, …).
const COVER_FETCH_BACKOFF_MS: u64 = 400;
fn random_salt() -> String {
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
@@ -46,16 +53,54 @@ pub fn build_cover_art_url(
}
}
pub async fn fetch_cover_bytes(client: &Client, url: &str) -> Result<Vec<u8>, String> {
let resp = client
.get(url)
.send()
.await
.map_err(|e| e.to_string())?;
if !resp.status().is_success() {
return Err(format!("cover HTTP {}", resp.status()));
/// Outcome of a single fetch attempt: transient errors are worth retrying,
/// permanent ones (a real 4xx like 404 — the cover simply does not exist) are
/// not, so we never hammer the server for genuinely-missing art.
enum FetchAttempt {
Ok(Vec<u8>),
Transient(String),
Permanent(String),
}
async fn fetch_cover_once(client: &Client, url: &str) -> FetchAttempt {
let resp = match client.get(url).send().await {
Ok(r) => r,
// Connection reset / timeout / DNS — transient under server load.
Err(e) => return FetchAttempt::Transient(e.to_string()),
};
let status = resp.status();
if status.is_success() {
return match resp.bytes().await {
Ok(b) => FetchAttempt::Ok(b.to_vec()),
Err(e) => FetchAttempt::Transient(e.to_string()),
};
}
resp.bytes().await.map(|b| b.to_vec()).map_err(|e| e.to_string())
let msg = format!("cover HTTP {status}");
if status.is_server_error() || status == reqwest::StatusCode::TOO_MANY_REQUESTS {
FetchAttempt::Transient(msg)
} else {
FetchAttempt::Permanent(msg)
}
}
pub async fn fetch_cover_bytes(client: &Client, url: &str) -> Result<Vec<u8>, String> {
let mut last_err = String::from("cover fetch failed");
for attempt in 0..COVER_FETCH_ATTEMPTS {
match fetch_cover_once(client, url).await {
FetchAttempt::Ok(bytes) => return Ok(bytes),
FetchAttempt::Permanent(e) => return Err(e),
FetchAttempt::Transient(e) => {
last_err = e;
if attempt + 1 < COVER_FETCH_ATTEMPTS {
tokio::time::sleep(Duration::from_millis(
COVER_FETCH_BACKOFF_MS * (attempt as u64 + 1),
))
.await;
}
}
}
}
Err(last_err)
}
#[cfg(test)]