fix(cover-backfill): kill idle CPU spin and offline-cache menu re-walks (#943)

* fix(cover-backfill): snapshot-diff worklist and live-tunable parallelism

Aggressive cover backfill pegged one tokio worker at ~100% on large,
fully-synced libraries while the download queues stayed empty.

- Take two snapshots once per pass — the DB catalog (single GROUP BY) and
  the on-disk cover bucket (one directory walk) — and download the
  set-difference. No per-row `stat` syscalls and no re-scan loop; the empty
  cache case (heavy backfill) costs zero per-item disk hits.
- Replace the front-loaded enumeration with a producer/consumer pipeline:
  the producer streams the catalog in chunks and feeds misses into a bounded
  channel; a fixed consumer pool keeps the download/encode pools saturated.
- Make cover backfill parallelism runtime-tunable from the Performance Probe
  (threads slider + "Run full pass now"); HTTP download and CPU encode
  semaphores resize live. Not surfaced in app settings.
- Add a "nothing changed" idle gate (catalog signature) so a settled pass is
  not re-run on every library:sync-idle, mirroring the analysis worker.
- Cancel promptly on switch to lazy: consumers bail on enabled/focus change
  and the producer feeds via try_send so a full channel cannot deadlock.
- Drop the per-item recursive disk walk from the ensure hot path.

* fix(cover-backfill): cheap idle gate, settle on 404s, transient retries

Follow-up to the snapshot-diff backfill: stop the periodic CPU spikes and the
89%-plateau wake storm on libraries whose covers can never reach 100%.

- Idle gate is now disk-free: compare only the catalog COUNT(DISTINCT) instead
  of walking ~all cover dirs on every sync-idle. "Did the server change?" never
  touches the filesystem. Clear-cache commands re-arm the gate (rearm_idle_gate)
  since a clear leaves the catalog total unchanged, and the settings UI wakes the
  active server after a clear.
- Settle the gate on any completed pass regardless of pending: remaining items
  are unfetchable-for-now (404), so the wake/sync-idle storm stops once the
  fetchable set is exhausted.
- Stop auto-clearing .fetch-failed markers every pass (it defeated the 30-min
  backoff and re-attempted 404s forever). The manual "Run full pass now" sends
  force=true to clear them and retry; wake/sync-idle/configure stay opportunistic.
- Rate-limit sync-idle passes (60s cooldown) as defence against chatty syncs.
- Retry cover downloads up to 3x with backoff on transient failures (5xx / 429 /
  network), but never on a real 4xx so missing covers don't hammer the server.

* fix(cover-cache): stop re-walking cover dirs from offline & cache menu

The settings cover-cache section polled disk usage + progress every 15s for
every server, each call doing a full recursive walk of the per-server cover
directory. On a fully populated cache this caused periodic CPU spikes whenever
that menu was open.

- mod.rs: add a 10s TTL memo around the per-server cover dir walk
  (cached_dir_usage_for_server), shared by cover_cache_stats_server and
  library_cover_progress; invalidate on clear (per-server and clear-all).
- CoverCacheStrategySection: recompute on entry only; rely on the
  cover:library-progress and cover:cache-cleared events for live updates;
  drop the per-cover cover:tier-ready refresh storm; turn the 15s loop into a
  5-minute safety net.

* fix(cover-backfill): keep emitting progress during the whole pass

The producer finishes enumerating the worklist long before the consumer pool
finishes downloading it, so progress was only emitted while feeding the channel
— the "offline & cache" menu and overlay then froze through the entire drain
phase. Replace the per-chunk emit with a 3s progress ticker that runs for the
lifetime of the pass and is aborted once the consumers drain (final accurate
emit still happens at settle).

* docs(changelog): record cover-backfill idle-CPU fix (PR #943)

Add [1.47.0] Fixed + Changed entries and a settingsCredits line for the
cover-backfill idle CPU / offline & cache menu work.
This commit is contained in:
cucadmuh
2026-06-02 04:56:34 +03:00
committed by GitHub
parent 5e977cfd49
commit a63ba3c9cb
12 changed files with 832 additions and 138 deletions
+15
View File
@@ -359,6 +359,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* Frontend and Rust `coverage` jobs no longer carry `continue-on-error`; listed hot-path files must stay at ≥70% line coverage or the PR fails.
### Cover backfill — live-tunable parallelism and pipeline
**By [@cucadmuh](https://github.com/cucadmuh), PR [#943](https://github.com/Psychotoxical/psysonic/pull/943)**
* Cover backfill runs through a producer/consumer pipeline (bounded channel + fixed consumer pool) that stays saturated and bails promptly on a switch to **lazy** instead of draining the whole backlog.
* **Performance Probe** gains a runtime cover-thread control (`library_cover_backfill_set_parallel`) that resizes the HTTP/encode pools live; "Run full pass now" forces a pass and clears fetch-failed backoff.
* Clearing the active server's cover cache re-arms the idle gate and wakes the worker, and in-pass progress is emitted on a ticker so the offline & cache view keeps counting through the whole scan.
@@ -645,7 +653,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* Pausing a track and resuming later inflated the listening time in **Statistics → Player stats** — the whole paused span was billed as if the track had been playing.
* Root cause: the session's tick baseline froze on pause, so the first progress tick after resume measured against the pre-pause timestamp. It now settles the played segment on pause and rebaselines on resume.
### Cover backfill — idle CPU spin and offline & cache menu spikes
**By [@cucadmuh](https://github.com/cucadmuh), PR [#943](https://github.com/Psychotoxical/psysonic/pull/943)**
* **Aggressive** cover backfill could pin a `tokio-runtime-worker` near 100% CPU when effectively idle: it had no "nothing changed, don't rescan" gate. Added a cheap disk-free idle signature (`COUNT(DISTINCT)` covers) with a `sync-idle` cooldown; the gate settles on a completed pass even when some covers are unfetchable (404), so libraries that never reach 100% no longer trigger a wake storm.
* Worklist is now built from a single DB `GROUP BY` plus one cover-dir snapshot and diffed in memory — no per-row `stat`, no per-batch rescan loop — so increasing parallelism actually saturates the pipeline.
* **Settings → offline & cache** caused periodic CPU spikes: the section re-walked each server's full cover directory every 15s. The per-server walk is now memoized with a short TTL and reused by the stats/progress commands; the menu recomputes on entry and via progress/cache-cleared events, with a 5-minute safety poll instead of a tight 15s loop.
* Transient download failures (network / 5xx / 429) retry up to 3× with exponential backoff; permanent 4xx settle without re-scanning.
## [1.46.0] - 2026-05-18
@@ -3,6 +3,7 @@
//! Catalog rows come from SQLite (`album` / `artist` tables) with explicit `kind`.
//! On-disk paths — `psysonic_core::cover_cache_layout`.
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use psysonic_core::cover_cache_layout::{self, is_fetch_only_cover_id};
@@ -13,7 +14,12 @@ use crate::cover_resolve::{
use crate::store::LibraryStore;
const DEFAULT_BATCH: u32 = 32;
const MAX_BATCH: u32 = 48;
/// Upper bound on items collected per `collect_cover_backfill_batch` call. The
/// catalog scan (`fetch_catalog_page` GROUP BY over the cover UNION) is O(catalog)
/// per page, so larger batches amortize that cost across more downloads and keep
/// the backfill download pool fed. Per-call page count stays bounded by
/// `MAX_SCAN_PAGES`, so a sparse backlog cannot inflate scan cost here.
const MAX_BATCH: u32 = 256;
const SCAN_PAGE: i64 = 256;
const MAX_SCAN_PAGES: usize = 16;
@@ -224,6 +230,152 @@ fn fetch_catalog_page(
})
}
/// All distinct cover catalog rows in ONE query (no cursor pagination, so the
/// expensive `GROUP BY` over the cover UNION runs once per pass instead of once
/// per page). Caller expands + disk-diffs the result.
pub fn fetch_all_catalog_rows(
store: &LibraryStore,
library_server_id: &str,
) -> Result<Vec<CoverBackfillItem>, String> {
store.with_read_conn(|conn| {
let sql = format!(
"SELECT kind, id, MAX(fetch_id) AS fetch_id
FROM ({COVER_CATALOG_SUBQUERY})
GROUP BY kind, id
ORDER BY kind ASC, id ASC"
);
let mut stmt = conn.prepare(&sql)?;
let rows = stmt
.query_map(rusqlite::params![library_server_id], |row| {
let kind: String = row.get(0)?;
let id: String = row.get(1)?;
let fetch_id: String = row.get(2)?;
Ok(CoverBackfillItem {
cache_kind: kind,
cache_entity_id: id,
fetch_cover_art_id: fetch_id,
})
})?
.collect::<Result<Vec<_>, _>>()?;
Ok(rows
.into_iter()
.filter(|item| {
!item.cache_entity_id.is_empty() && !is_fetch_only_cover_id(&item.cache_entity_id)
})
.collect())
})
}
/// One-pass on-disk snapshot of a server's cover bucket: which entities already
/// have the canonical tier, and which carry a recent `.fetch-failed` marker.
///
/// Built once per pass so the catalog diff is pure in-memory set math — no
/// per-row `stat` syscalls hammering the filesystem on every album/artist. Keys
/// are `(kind, sanitized_entity_id)` to match on-disk directory names.
#[derive(Debug, Default)]
pub struct CoverDiskSnapshot {
present: HashSet<(String, String)>,
failed: HashSet<(String, String)>,
}
impl CoverDiskSnapshot {
fn key(kind: &str, entity_id: &str) -> (String, String) {
(
kind.to_string(),
cover_cache_layout::sanitize_path_segment(entity_id),
)
}
/// Canonical tier already on disk for this entity.
pub fn is_cached(&self, kind: &str, entity_id: &str) -> bool {
self.present.contains(&Self::key(kind, entity_id))
}
/// Recent `.fetch-failed` marker — skip so slots go to fetchable art.
pub fn is_recently_failed(&self, kind: &str, entity_id: &str) -> bool {
self.failed.contains(&Self::key(kind, entity_id))
}
}
/// Walk the server's cover bucket once (`album/` and `artist/`) and record the
/// cached plus recently-failed entities. Cheap exactly when it matters most: an
/// empty cache yields an empty `read_dir`, so the heavy backfill diff costs zero
/// per-item `stat`s instead of one (or more) per catalog row.
pub fn snapshot_cover_disk(cover_root: &Path, server_index_key: &str) -> CoverDiskSnapshot {
let server_dir = cover_cache_layout::cover_server_dir(cover_root, server_index_key);
let mut snap = CoverDiskSnapshot::default();
for kind in cover_cache_layout::SEGMENT_KINDS {
let kind_dir = server_dir.join(kind);
let Ok(entries) = std::fs::read_dir(&kind_dir) else {
continue;
};
for ent in entries.flatten() {
let path = ent.path();
let Some(name) = path.file_name().and_then(|n| n.to_str()).map(str::to_string) else {
continue;
};
if !path.is_dir() {
continue;
}
let key = (kind.to_string(), name);
if cover_cache_layout::entity_dir_has_canonical_tier(&path) {
snap.present.insert(key.clone());
}
if cover_fetch_recently_failed(&path) {
snap.failed.insert(key);
}
}
}
snap
}
/// Diff catalog rows against a pre-built disk snapshot → the subset still needing
/// a download. No filesystem access here: the snapshot already captured disk
/// state once. Rows whose raw id is cached/failed are skipped without expanding;
/// the rest get `expand_backfill_items` (DB) to resolve multi-disc `mf-*` /
/// artist entities, which are then diffed against the same snapshot.
pub fn diff_missing_against_snapshot(
store: &LibraryStore,
library_server_id: &str,
snapshot: &CoverDiskSnapshot,
rows: Vec<CoverBackfillItem>,
) -> Result<Vec<CoverBackfillItem>, String> {
let mut out = Vec::new();
for row in rows {
if snapshot.is_cached(&row.cache_kind, &row.cache_entity_id)
|| snapshot.is_recently_failed(&row.cache_kind, &row.cache_entity_id)
{
continue;
}
for normalized in expand_backfill_items(store, library_server_id, row)? {
if normalized.cache_entity_id.is_empty() {
continue;
}
if snapshot.is_cached(&normalized.cache_kind, &normalized.cache_entity_id)
|| snapshot.is_recently_failed(&normalized.cache_kind, &normalized.cache_entity_id)
{
continue;
}
out.push(normalized);
}
}
Ok(out)
}
/// One-shot worklist of every cover target still missing its canonical tier:
/// DB catalog snapshot minus the on-disk snapshot. The worker streams the diff
/// in chunks against a shared snapshot; tests use this whole-catalog form.
pub fn collect_missing_cover_targets(
store: &LibraryStore,
library_server_id: &str,
cover_root: &Path,
server_index_key: &str,
) -> Result<Vec<CoverBackfillItem>, String> {
let rows = fetch_all_catalog_rows(store, library_server_id)?;
let snapshot = snapshot_cover_disk(cover_root, server_index_key);
diff_missing_against_snapshot(store, library_server_id, &snapshot, rows)
}
pub fn count_distinct_cover_ids(store: &LibraryStore, library_server_id: &str) -> Result<i64, String> {
store.with_read_conn(|conn| {
let sql = format!(
@@ -535,6 +687,29 @@ mod tests {
let _ = std::fs::remove_dir_all(root.join(host));
}
#[test]
fn collect_missing_excludes_cached_includes_missing() {
let store = LibraryStore::open_in_memory();
seed_track(&store, "srv", "tr1", "al-have", None);
seed_track(&store, "srv", "tr2", "al-need", None);
let root = std::env::temp_dir().join("psysonic-missing-targets-test");
let host = "srv-host";
let have_dir = cover_cache_layout::cover_dir(&root, host, "album", "al-have");
std::fs::create_dir_all(&have_dir).unwrap();
std::fs::write(
have_dir.join(format!("{LIBRARY_COVER_CANONICAL_TIER}.webp")),
b"x",
)
.unwrap();
let missing = collect_missing_cover_targets(&store, "srv", &root, host).unwrap();
let ids: Vec<_> = missing.iter().map(|i| i.cache_entity_id.as_str()).collect();
assert!(ids.contains(&"al-need"), "missing cover should be queued");
assert!(!ids.contains(&"al-have"), "cached cover must be skipped");
let _ = std::fs::remove_dir_all(root.join(host));
}
#[test]
fn backfill_includes_per_disc_mf_when_discs_differ() {
let store = LibraryStore::open_in_memory();
+326 -104
View File
@@ -2,27 +2,50 @@
use super::{state, CoverCacheEnsureArgs, CoverCacheState};
use psysonic_library::cover_backfill::{
clear_cover_fetch_failures, collect_cover_backfill_batch, collect_cover_progress,
LibraryCoverBackfillBatchDto, LIBRARY_COVER_CANONICAL_TIER,
clear_cover_fetch_failures, collect_cover_progress, count_distinct_cover_ids,
diff_missing_against_snapshot, fetch_all_catalog_rows, snapshot_cover_disk,
LIBRARY_COVER_CANONICAL_TIER,
};
use psysonic_library::payload::LibrarySyncProgressPayload;
use psysonic_library::repos::sync_state::SyncStateRepository;
use psysonic_library::LibraryRuntime;
use serde::{Deserialize, Serialize};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Duration;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use tauri::{AppHandle, Emitter, Listener, Manager};
use tokio::sync::{Mutex, Semaphore};
use super::{count_cached_cover_ids, dir_usage_for_server};
/// Concurrent library downloads + encodes (hard cap — avoids saturating all CPU cores).
const LIBRARY_BACKFILL_PARALLEL: usize = 2;
const BATCH_SIZE: u32 = 24;
const PENDING_RESTART_THRESHOLD: i64 = 32;
/// Default concurrent library downloads + encodes. Runtime-tunable via the
/// perf probe (`set_parallel`); the constant is only the startup value.
const LIBRARY_BACKFILL_PARALLEL_DEFAULT: usize = 2;
/// Bounds for the runtime knob — keep it sane so a stray value cannot DoS the
/// host or starve the audio path.
pub const LIBRARY_BACKFILL_PARALLEL_MIN: usize = 1;
pub const LIBRARY_BACKFILL_PARALLEL_MAX: usize = 16;
/// Raw catalog rows diffed per streaming chunk. Small enough that downloads
/// start almost immediately after the one-shot enumeration, large enough to
/// amortize the per-chunk `spawn_blocking` hop.
const SCAN_CHUNK_ROWS: usize = 512;
const SYNC_WAIT_MS: u64 = 5000;
const PROGRESS_EVERY_BATCHES: u32 = 8;
/// Cadence of the in-pass progress ticker (drives the "offline & cache" menu and
/// the perf-probe overlay while a pass downloads). Only runs for the duration of
/// an active pass.
const PROGRESS_TICK_SECS: u64 = 3;
/// Minimum gap between `library:sync-idle`-driven passes. Each such pass runs the
/// idle-gate signature (a full cover-dir walk + DB count), so a chatty sync (e.g.
/// periodic delta syncs) must not make that walk fire every few seconds. Manual
/// runs and strategy-toggle wakes bypass this.
const SYNC_IDLE_COOLDOWN_MS: u64 = 60_000;
fn now_ms() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0)
}
#[derive(Clone)]
pub struct CoverBackfillSession {
@@ -33,6 +56,19 @@ pub struct CoverBackfillSession {
pub password: String,
}
/// Catalog signature captured when a full pass completes.
///
/// While it still matches, `library:sync-idle` must NOT re-trigger a rescan —
/// mirrors the analysis coordinator's `completed_total` gate. Deliberately the
/// **cheap** `COUNT(DISTINCT)` over the catalog only: a server change (track
/// add/remove shifts `total`) re-arms the next pass, but checking it never
/// touches the filesystem. A cover-cache clear leaves `total` unchanged, so the
/// clear commands re-arm the gate explicitly (`rearm_idle_gate`).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct CoverIdleSignature {
total: i64,
}
pub struct CoverBackfillWorker {
pub enabled: AtomicBool,
/// When true, the active pass yields so visible-route cover IPC is not starved.
@@ -41,6 +77,15 @@ pub struct CoverBackfillWorker {
cursor: Mutex<String>,
pass_running: AtomicBool,
backfill_http: Arc<Semaphore>,
/// Live download/encode concurrency for backfill passes. Mirrors the
/// `backfill_http` permit count and gates per-batch `ensure_one` tasks.
parallel: AtomicUsize,
/// Set when a pass found nothing pending; suppresses idle-driven rescans
/// until the catalog signature changes. `None` means "re-armed".
settled: Mutex<Option<CoverIdleSignature>>,
/// Epoch-ms of the last `sync-idle`-driven pass, to rate-limit the idle-gate
/// disk walk against chatty syncs. 0 = never.
last_sync_idle_ms: AtomicU64,
}
#[derive(Debug, Clone, Serialize)]
@@ -75,7 +120,10 @@ impl CoverBackfillWorker {
session: Mutex::new(None),
cursor: Mutex::new(String::new()),
pass_running: AtomicBool::new(false),
backfill_http: Arc::new(Semaphore::new(LIBRARY_BACKFILL_PARALLEL)),
backfill_http: Arc::new(Semaphore::new(LIBRARY_BACKFILL_PARALLEL_DEFAULT)),
parallel: AtomicUsize::new(LIBRARY_BACKFILL_PARALLEL_DEFAULT),
settled: Mutex::new(None),
last_sync_idle_ms: AtomicU64::new(0),
}
}
@@ -83,9 +131,49 @@ impl CoverBackfillWorker {
self.ui_priority_hold.store(hold, Ordering::Relaxed);
}
/// Re-arm the idle gate so the next opportunistic pass runs even though the
/// catalog `total` is unchanged — used after a cover-cache clear, which
/// drops files the cheap signature cannot see.
pub async fn rearm_idle_gate(&self) {
*self.settled.lock().await = None;
self.last_sync_idle_ms.store(0, Ordering::Relaxed);
}
/// Current backfill download/encode concurrency.
pub fn parallel(&self) -> usize {
self.parallel.load(Ordering::Relaxed).max(LIBRARY_BACKFILL_PARALLEL_MIN)
}
/// Retune backfill concurrency at runtime. Resizes the shared HTTP permit
/// pool to match (next batch picks up the new per-batch slot count). Returns
/// the clamped value actually applied.
pub fn set_parallel(&self, threads: usize) -> usize {
let next = threads.clamp(LIBRARY_BACKFILL_PARALLEL_MIN, LIBRARY_BACKFILL_PARALLEL_MAX);
let prev = self.parallel.swap(next, Ordering::SeqCst);
if next > prev {
self.backfill_http.add_permits(next - prev);
} else if next < prev {
// Shrinking: drain surplus permits as they free up so in-flight
// fetches finish but no new ones start beyond the new cap.
let sem = self.backfill_http.clone();
let surplus = prev - next;
tauri::async_runtime::spawn(async move {
for _ in 0..surplus {
if let Ok(permit) = sem.acquire().await {
permit.forget();
}
}
});
}
next
}
pub async fn set_session(&self, enabled: bool, session: Option<CoverBackfillSession>) {
self.enabled.store(enabled, Ordering::Relaxed);
*self.session.lock().await = session;
// Server switch or enable/disable invalidates any settled state: re-arm
// so the next idle event runs a real pass for the new focus.
*self.settled.lock().await = None;
if !enabled {
*self.cursor.lock().await = String::new();
}
@@ -97,7 +185,7 @@ impl CoverBackfillWorker {
/// Semaphore-backed library backfill HTTP slots (perf probe).
pub fn pipeline_http_stats(&self) -> (u32, u32, bool) {
let max = LIBRARY_BACKFILL_PARALLEL as u32;
let max = self.parallel() as u32;
let active = max.saturating_sub(self.backfill_http.available_permits() as u32);
let pass_running = self.pass_running.load(Ordering::Relaxed);
(max, active, pass_running)
@@ -187,7 +275,7 @@ async fn ensure_one(
let _ = CoverCacheState::ensure_inner(&st, &app, &args, Some(http_sem)).await;
}
async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>) {
async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>, force: bool) {
if !worker.enabled.load(Ordering::Relaxed) {
return;
}
@@ -201,6 +289,14 @@ async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>) {
None => return,
};
// Opportunistic triggers (wake on track change, sync-idle) skip the whole
// scan when a prior pass already settled and nothing changed — otherwise a
// library with permanently-unfetchable covers (404s) would re-scan on every
// wake forever. The manual "Run full pass now" sets `force` to bypass this.
if !force && cover_idle_gate_should_skip(&app, &worker, &session).await {
return;
}
while !sync_allows_cover_backfill(&runtime.store, &session.library_server_id) {
if !worker.enabled.load(Ordering::Relaxed) {
return;
@@ -220,125 +316,208 @@ async fn run_full_pass(app: AppHandle, worker: Arc<CoverBackfillWorker>) {
worker.reset_cursor().await;
let http_sem = worker.backfill_http.clone();
let mut batch_count = 0u32;
loop {
if !session_still_focused(&worker, &session).await {
break;
// A forced pass is an explicit user retry: drop the `.fetch-failed` backoff
// markers and the settled gate so previously-404'd covers are attempted
// again. Opportunistic passes leave the markers in place (30-min TTL).
if force {
*worker.settled.lock().await = None;
let root2 = root.clone();
let index_key2 = session.server_index_key.clone();
let _ = tauri::async_runtime::spawn_blocking(move || {
clear_cover_fetch_failures(&root2, &index_key2)
})
.await;
}
if worker.ui_priority_hold.load(Ordering::Relaxed) {
tokio::time::sleep(Duration::from_millis(200)).await;
continue;
}
let cursor = worker.cursor.lock().await.clone();
let cursor_opt = if cursor.is_empty() {
None
} else {
Some(cursor)
};
// Two snapshots, taken ONCE per pass: the DB catalog (single GROUP BY) and
// the on-disk cover bucket (one directory walk). The delta = catalog minus
// disk, streamed in chunks below. No per-row `stat` on the filesystem and no
// re-scan loop — pure set math against the captured disk snapshot.
let (raw_rows, snapshot) = {
let store = runtime.store.clone();
let lib_id = session.library_server_id.clone();
let root_for_scan = root.clone();
let index_key = session.server_index_key.clone();
let root_for_batch = root.clone();
let batch: Option<LibraryCoverBackfillBatchDto> =
match tauri::async_runtime::spawn_blocking(move || {
collect_cover_backfill_batch(
&store,
&lib_id,
&root_for_batch,
&index_key,
cursor_opt.as_deref(),
Some(BATCH_SIZE),
)
let rows = fetch_all_catalog_rows(&store, &lib_id)?;
let snap = snapshot_cover_disk(&root_for_scan, &index_key);
Ok::<_, String>((rows, snap))
})
.await
{
Ok(Ok(b)) => Some(b),
_ => None,
};
let Some(batch) = batch else {
break;
};
batch_count += 1;
if !session_still_focused(&worker, &session).await {
break;
}
let items = batch.items.clone();
let mut paused_for_ui_priority = false;
let batch_slots = Arc::new(Semaphore::new(LIBRARY_BACKFILL_PARALLEL));
let mut set = tokio::task::JoinSet::new();
for item in items {
if worker.ui_priority_hold.load(Ordering::Relaxed) {
paused_for_ui_priority = true;
break;
Ok(Ok(pair)) => pair,
_ => (Vec::new(), Default::default()),
}
};
let snapshot = Arc::new(snapshot);
// Producer/consumer: a fixed pool of consumer tasks pulls misses off a
// bounded channel and downloads them continuously, while the producer scans
// the catalog in chunks and feeds misses in. This keeps the pool saturated
// even when misses are sparse across chunks — no per-chunk drain barrier.
// True concurrency stays governed by the resizable `http_sem` / encode
// semaphores inside `ensure_one`, so the threads slider still applies live.
let (tx, rx) =
tokio::sync::mpsc::channel::<psysonic_library::cover_backfill::CoverBackfillItem>(256);
let rx = Arc::new(Mutex::new(rx));
let mut consumers = tokio::task::JoinSet::new();
for _ in 0..LIBRARY_BACKFILL_PARALLEL_MAX {
let rx = rx.clone();
let st = st_arc.clone();
let http_sem = http_sem.clone();
let app = app.clone();
let session = session.clone();
let worker_arc = worker.clone();
let batch_slots = batch_slots.clone();
set.spawn(async move {
let Ok(_slot) = batch_slots.acquire().await else {
return;
consumers.spawn(async move {
loop {
// Bail the moment the strategy flips to lazy / focus changes, so a
// switch to "lazy" abandons the buffered backlog instead of
// draining the whole channel (mirrors the producer's check).
if !session_still_focused(&worker_arc, &session).await {
break;
}
let item = {
let mut guard = rx.lock().await;
guard.recv().await
};
ensure_one(worker_arc.as_ref(), st, http_sem, app, session, item).await;
let Some(item) = item else { break };
ensure_one(
worker_arc.as_ref(),
st.clone(),
http_sem.clone(),
app.clone(),
session.clone(),
item,
)
.await;
}
});
}
while set.join_next().await.is_some() {}
if paused_for_ui_priority || worker.ui_priority_hold.load(Ordering::Relaxed) {
// Progress ticker: the producer finishes enumerating the worklist long before
// the consumers finish downloading it, so emitting only while feeding would
// freeze the "offline & cache" menu through the whole drain phase. Tick a
// periodic snapshot for the lifetime of the pass instead; aborted once the
// consumers drain (a final accurate emit happens at settle below).
let progress_ticker = {
let app = app.clone();
let store = runtime.store.clone();
let root = root.clone();
let session = session.clone();
let lib_id = session.library_server_id.clone();
let index_key = session.server_index_key.clone();
tauri::async_runtime::spawn(async move {
loop {
tokio::time::sleep(Duration::from_secs(PROGRESS_TICK_SECS)).await;
if let Ok((done, total, pending)) =
progress_snapshot(&store, &root, &lib_id, &index_key).await
{
emit_library_progress(&app, &session, done, total, pending, &root).await;
}
}
})
};
let mut rows_iter = raw_rows.into_iter();
let mut completed = false;
loop {
if !session_still_focused(&worker, &session).await {
break;
}
if worker.ui_priority_hold.load(Ordering::Relaxed) {
tokio::time::sleep(Duration::from_millis(200)).await;
continue;
}
if batch_count.is_multiple_of(PROGRESS_EVERY_BATCHES) {
if let Ok((done, total, pending)) = progress_snapshot(
&runtime.store,
&root,
&session.library_server_id,
&session.server_index_key,
)
.await
{
emit_library_progress(&app, &session, done, total, pending, &root).await;
}
}
if batch.exhausted {
worker.cursor.lock().await.clear();
if let Ok((done, total, pending)) = progress_snapshot(
&runtime.store,
&root,
&session.library_server_id,
&session.server_index_key,
)
.await
{
if pending > PENDING_RESTART_THRESHOLD {
let root3 = root.clone();
let index_key3 = session.server_index_key.clone();
let _ = tauri::async_runtime::spawn_blocking(move || {
clear_cover_fetch_failures(&root3, &index_key3)
})
.await;
}
emit_library_progress(&app, &session, done, total, pending, &root).await;
}
let scan_chunk: Vec<_> = rows_iter.by_ref().take(SCAN_CHUNK_ROWS).collect();
if scan_chunk.is_empty() {
completed = true;
break;
}
if let Some(next) = batch.next_cursor {
*worker.cursor.lock().await = next;
// Diff this chunk against the captured snapshot off-thread (in-memory set
// math + DB expand only for rows not already cached) → misses to download.
let missing: Vec<_> = {
let store = runtime.store.clone();
let lib_id = session.library_server_id.clone();
let snapshot = snapshot.clone();
match tauri::async_runtime::spawn_blocking(move || {
diff_missing_against_snapshot(&store, &lib_id, &snapshot, scan_chunk)
})
.await
{
Ok(Ok(missing)) => missing,
_ => Vec::new(),
}
};
// Focus-aware feed: never park indefinitely on a full channel, or a
// switch to lazy (which stops the consumers) would deadlock the producer
// here. `try_send` + a short retry lets us re-check focus and bail.
let mut feed_closed = false;
'feed: for mut item in missing {
loop {
match tx.try_send(item) {
Ok(()) => break,
Err(tokio::sync::mpsc::error::TrySendError::Closed(_)) => {
feed_closed = true;
break 'feed;
}
Err(tokio::sync::mpsc::error::TrySendError::Full(returned)) => {
if !session_still_focused(&worker, &session).await {
feed_closed = true;
break 'feed;
}
item = returned;
tokio::time::sleep(Duration::from_millis(25)).await;
}
}
}
}
if feed_closed {
break;
}
}
// Close the channel so consumers drain the remaining backlog and exit.
drop(tx);
while consumers.join_next().await.is_some() {}
progress_ticker.abort();
// Only settle the idle gate on a natural finish (worklist drained), never on
// a session-switch break — that belongs to the previous focus.
if completed {
worker.cursor.lock().await.clear();
match progress_snapshot(
&runtime.store,
&root,
&session.library_server_id,
&session.server_index_key,
)
.await
{
Ok((done, total, pending)) => {
// Settle on a full scan regardless of `pending`: whatever is left
// is unfetchable for now (404s with a fresh `.fetch-failed`
// marker). The cheap `total` signature re-triggers a pass only if
// the server catalog changes; a cache clear re-arms via the clear
// command. This stops the wake storm on libraries whose covers
// can never reach 100%.
*worker.settled.lock().await = Some(CoverIdleSignature { total });
emit_library_progress(&app, &session, done, total, pending, &root).await;
}
Err(_) => {
*worker.settled.lock().await = None;
}
}
}
}
/// Start one full-catalog pass on the Tokio runtime (survives inactive webview).
pub async fn try_schedule_full_pass(app: &AppHandle) -> bool {
/// `force` bypasses the idle gate and clears fetch-failed backoff (explicit user
/// retry); opportunistic callers (wake / sync-idle) pass `false`.
pub async fn try_schedule_full_pass(app: &AppHandle, force: bool) -> bool {
let worker = match app.try_state::<Arc<CoverBackfillWorker>>() {
Some(w) => w.inner().clone(),
None => return false,
@@ -356,12 +535,44 @@ pub async fn try_schedule_full_pass(app: &AppHandle) -> bool {
let app = app.clone();
tauri::async_runtime::spawn(async move {
run_full_pass(app, worker.clone()).await;
run_full_pass(app, worker.clone(), force).await;
worker.pass_running.store(false, Ordering::SeqCst);
});
true
}
/// Cheap catalog signature for the idle gate: a single `COUNT(DISTINCT)` over
/// the cover catalog. No filesystem access — checking "did anything change on
/// the server?" must never walk the on-disk cover cache.
async fn current_cover_signature(
app: &AppHandle,
session: &CoverBackfillSession,
) -> Option<CoverIdleSignature> {
let runtime = app.try_state::<LibraryRuntime>()?;
let store = runtime.store.clone();
let lib_id = session.library_server_id.clone();
tauri::async_runtime::spawn_blocking(move || {
count_distinct_cover_ids(&store, &lib_id)
.ok()
.map(|total| CoverIdleSignature { total })
})
.await
.ok()
.flatten()
}
/// True when the previous pass settled with nothing pending and the catalog
/// still matches that signature — so an idle event need not rescan.
async fn cover_idle_gate_should_skip(app: &AppHandle, worker: &CoverBackfillWorker, session: &CoverBackfillSession) -> bool {
let Some(settled) = *worker.settled.lock().await else {
return false;
};
match current_cover_signature(app, session).await {
Some(current) => current == settled,
None => false,
}
}
fn on_sync_idle(app: &AppHandle, payload: SyncIdlePayload) {
if !payload.ok {
return;
@@ -382,7 +593,18 @@ fn on_sync_idle(app: &AppHandle, payload: SyncIdlePayload) {
if !session_matches_server(&session, &payload.server_id) {
return;
}
let _ = try_schedule_full_pass(&app).await;
// Rate-limit sync-idle passes: each runs the idle-gate disk walk, so a
// chatty sync must not trigger it every few seconds. The gate inside the
// pass still skips the actual rescan when nothing changed.
let now = now_ms();
let last = worker.last_sync_idle_ms.load(Ordering::Relaxed);
if last != 0 && now.saturating_sub(last) < SYNC_IDLE_COOLDOWN_MS {
return;
}
worker.last_sync_idle_ms.store(now, Ordering::Relaxed);
// Opportunistic: the gate (checked inside the pass) skips the rescan when
// a prior pass settled and nothing changed (mirrors the analysis gate).
let _ = try_schedule_full_pass(&app, false).await;
});
}
@@ -399,7 +621,7 @@ pub fn setup_library_sync_idle_listener(app: &AppHandle) {
/// Legacy single-step API (optional diagnostics).
pub async fn pulse_backfill(app: &AppHandle, _worker: &Arc<CoverBackfillWorker>) -> CoverBackfillPulseDto {
if try_schedule_full_pass(app).await {
if try_schedule_full_pass(app, false).await {
return CoverBackfillPulseDto {
scheduled: 0,
exhausted: false,
+55 -10
View File
@@ -1,9 +1,16 @@
use reqwest::Client;
use std::time::{SystemTime, UNIX_EPOCH};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use url::Url;
const SUBSONIC_CLIENT: &str = "Psysonic";
/// Total cover fetch attempts (1 initial + retries). A busy server can answer
/// covers with 5xx/429/timeouts under our own backfill load, so a couple of
/// backed-off retries recover those without a permanent `.fetch-failed` marker.
const COVER_FETCH_ATTEMPTS: usize = 3;
/// Base backoff between attempts (grows linearly: 1×, 2×, …).
const COVER_FETCH_BACKOFF_MS: u64 = 400;
fn random_salt() -> String {
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
@@ -46,16 +53,54 @@ pub fn build_cover_art_url(
}
}
pub async fn fetch_cover_bytes(client: &Client, url: &str) -> Result<Vec<u8>, String> {
let resp = client
.get(url)
.send()
.await
.map_err(|e| e.to_string())?;
if !resp.status().is_success() {
return Err(format!("cover HTTP {}", resp.status()));
/// Outcome of a single fetch attempt: transient errors are worth retrying,
/// permanent ones (a real 4xx like 404 — the cover simply does not exist) are
/// not, so we never hammer the server for genuinely-missing art.
enum FetchAttempt {
Ok(Vec<u8>),
Transient(String),
Permanent(String),
}
async fn fetch_cover_once(client: &Client, url: &str) -> FetchAttempt {
let resp = match client.get(url).send().await {
Ok(r) => r,
// Connection reset / timeout / DNS — transient under server load.
Err(e) => return FetchAttempt::Transient(e.to_string()),
};
let status = resp.status();
if status.is_success() {
return match resp.bytes().await {
Ok(b) => FetchAttempt::Ok(b.to_vec()),
Err(e) => FetchAttempt::Transient(e.to_string()),
};
}
resp.bytes().await.map(|b| b.to_vec()).map_err(|e| e.to_string())
let msg = format!("cover HTTP {status}");
if status.is_server_error() || status == reqwest::StatusCode::TOO_MANY_REQUESTS {
FetchAttempt::Transient(msg)
} else {
FetchAttempt::Permanent(msg)
}
}
pub async fn fetch_cover_bytes(client: &Client, url: &str) -> Result<Vec<u8>, String> {
let mut last_err = String::from("cover fetch failed");
for attempt in 0..COVER_FETCH_ATTEMPTS {
match fetch_cover_once(client, url).await {
FetchAttempt::Ok(bytes) => return Ok(bytes),
FetchAttempt::Permanent(e) => return Err(e),
FetchAttempt::Transient(e) => {
last_err = e;
if attempt + 1 < COVER_FETCH_ATTEMPTS {
tokio::time::sleep(Duration::from_millis(
COVER_FETCH_BACKOFF_MS * (attempt as u64 + 1),
))
.await;
}
}
}
}
Err(last_err)
}
#[cfg(test)]
+114 -13
View File
@@ -24,6 +24,7 @@ use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::io::Cursor;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::{Mutex, Semaphore};
@@ -79,10 +80,10 @@ pub(crate) fn cover_pipeline_queue_stats(
http_active: sem_active(&cache.http_sem, COVER_HTTP_CONCURRENCY as u32),
cpu_ui_max: COVER_CPU_UI_CONCURRENCY as u32,
cpu_ui_active: sem_active(&cache.cover_cpu_ui_sem, COVER_CPU_UI_CONCURRENCY as u32),
cpu_backfill_max: COVER_CPU_BACKFILL_CONCURRENCY as u32,
cpu_backfill_max: cache.cover_backfill_cpu_parallel() as u32,
cpu_backfill_active: sem_active(
&cache.cover_cpu_backfill_sem,
COVER_CPU_BACKFILL_CONCURRENCY as u32,
cache.cover_backfill_cpu_parallel() as u32,
),
library_backfill_http_max,
library_backfill_http_active,
@@ -117,7 +118,10 @@ const COVER_HTTP_CONCURRENCY: usize = 16;
/// UI-visible decode + WebP encode (grid, hero, player) — not shared with library backfill.
const COVER_CPU_UI_CONCURRENCY: usize = 2;
/// Library backfill encode ladder — separate pool so bulk warm-up cannot starve the webview.
/// Default only; runtime-tunable from the perf probe via `set_backfill_cpu_parallel`.
const COVER_CPU_BACKFILL_CONCURRENCY: usize = 2;
/// Upper bound for the runtime encode-pool knob (matches the worker cap).
const COVER_CPU_BACKFILL_MAX: usize = 16;
pub struct CoverCacheState {
pub root: PathBuf,
@@ -128,6 +132,9 @@ pub struct CoverCacheState {
pub http_sem: Arc<Semaphore>,
pub cover_cpu_ui_sem: Arc<Semaphore>,
pub cover_cpu_backfill_sem: Arc<Semaphore>,
/// Live permit count of `cover_cpu_backfill_sem` (the semaphore itself only
/// exposes *available* permits, not the configured ceiling).
cover_cpu_backfill_max: AtomicUsize,
}
impl CoverCacheState {
@@ -147,9 +154,35 @@ impl CoverCacheState {
http_sem: Arc::new(Semaphore::new(COVER_HTTP_CONCURRENCY)),
cover_cpu_ui_sem: Arc::new(Semaphore::new(COVER_CPU_UI_CONCURRENCY)),
cover_cpu_backfill_sem: Arc::new(Semaphore::new(COVER_CPU_BACKFILL_CONCURRENCY)),
cover_cpu_backfill_max: AtomicUsize::new(COVER_CPU_BACKFILL_CONCURRENCY),
})
}
/// Current configured ceiling of the backfill encode pool.
pub fn cover_backfill_cpu_parallel(&self) -> usize {
self.cover_cpu_backfill_max.load(Ordering::Relaxed).max(1)
}
/// Retune the backfill encode pool to match the worker's download
/// concurrency. Grows/shrinks the semaphore permits in place.
pub fn set_backfill_cpu_parallel(&self, threads: usize) {
let next = threads.clamp(1, COVER_CPU_BACKFILL_MAX);
let prev = self.cover_cpu_backfill_max.swap(next, Ordering::SeqCst);
if next > prev {
self.cover_cpu_backfill_sem.add_permits(next - prev);
} else if next < prev {
let sem = self.cover_cpu_backfill_sem.clone();
let surplus = prev - next;
tauri::async_runtime::spawn(async move {
for _ in 0..surplus {
if let Ok(permit) = sem.acquire().await {
permit.forget();
}
}
});
}
}
fn cpu_sem_for(&self, library_bulk: bool) -> Arc<Semaphore> {
if library_bulk {
self.cover_cpu_backfill_sem.clone()
@@ -162,11 +195,6 @@ impl CoverCacheState {
("ok".into(), true)
}
fn pressure(&self) -> (String, bool) {
let (bytes, _) = dir_usage_at_root(&self.root);
self.pressure_from_bytes(bytes)
}
pub(crate) async fn ensure_inner(
state: &Arc<Mutex<CoverCacheState>>,
app: &AppHandle,
@@ -183,7 +211,11 @@ impl CoverCacheState {
});
}
let (_, auto_dl) = this.pressure();
// Cheap, no-IO gate. Previously this ran a full recursive disk walk of
// the entire cover cache (`pressure()` → `dir_usage_at_root`) on every
// ensure, under the global state lock — serializing the whole backfill
// pool onto filesystem stat work. The walked bytes were then discarded.
let (_, auto_dl) = this.pressure_from_bytes(0);
if !auto_dl && args.tier != 2000 {
return Ok(CoverCacheEnsureResult {
hit: false,
@@ -453,6 +485,42 @@ pub(crate) fn dir_usage_for_server(root: &Path, server_index_key: &str) -> (u64,
server_cover_disk_usage(&cover_server_dir(root, server_index_key))
}
/// TTL-memoized per-server cover dir walk. The "offline & cache" settings menu
/// polls byte usage + cached count every few seconds for every server; on a full
/// cache that is several full directory walks per tick. Reuse a recent walk so we
/// don't re-stat thousands of files when nothing has changed. Active backfill still
/// pushes live numbers through the `cover:library-progress` event, so a short TTL
/// only de-dupes the idle polling, it does not hide real progress.
const DIR_USAGE_CACHE_TTL: Duration = Duration::from_secs(10);
type DirUsageCache = std::sync::Mutex<HashMap<String, (std::time::Instant, (u64, u64))>>;
fn dir_usage_cache() -> &'static DirUsageCache {
static CACHE: std::sync::OnceLock<DirUsageCache> = std::sync::OnceLock::new();
CACHE.get_or_init(|| std::sync::Mutex::new(HashMap::new()))
}
pub(crate) fn cached_dir_usage_for_server(root: &Path, server_index_key: &str) -> (u64, u64) {
if let Ok(map) = dir_usage_cache().lock() {
if let Some((at, value)) = map.get(server_index_key) {
if at.elapsed() < DIR_USAGE_CACHE_TTL {
return *value;
}
}
}
let value = dir_usage_for_server(root, server_index_key);
if let Ok(mut map) = dir_usage_cache().lock() {
map.insert(server_index_key.to_string(), (std::time::Instant::now(), value));
}
value
}
pub(crate) fn invalidate_dir_usage_cache(server_index_key: &str) {
if let Ok(mut map) = dir_usage_cache().lock() {
map.remove(server_index_key);
}
}
pub(crate) fn dir_usage_at_root(root: &Path) -> (u64, u64) {
cover_root_disk_usage(root)
}
@@ -512,9 +580,12 @@ pub fn init_cover_cache(app: &AppHandle) -> Result<(), String> {
}
#[tauri::command]
pub async fn library_cover_backfill_run_full_pass(app: AppHandle) -> Result<CoverBackfillRunDto, String> {
pub async fn library_cover_backfill_run_full_pass(
app: AppHandle,
force: Option<bool>,
) -> Result<CoverBackfillRunDto, String> {
Ok(CoverBackfillRunDto {
started: try_schedule_full_pass(&app).await,
started: try_schedule_full_pass(&app, force.unwrap_or(false)).await,
})
}
@@ -548,6 +619,24 @@ pub async fn library_cover_backfill_set_ui_priority(
Ok(())
}
/// Perf-probe tuning knob: set how many threads cover backfill uses (download
/// + encode pools move together). Not exposed in app Settings by design.
/// Returns the clamped value actually applied.
#[tauri::command]
pub async fn library_cover_backfill_set_parallel(
app: AppHandle,
threads: usize,
) -> Result<u32, String> {
let worker = app
.try_state::<Arc<CoverBackfillWorker>>()
.ok_or_else(|| "cover backfill worker not initialized".to_string())?;
let applied = worker.set_parallel(threads);
if let Ok(cache) = state(&app) {
cache.lock().await.set_backfill_cpu_parallel(applied);
}
Ok(applied as u32)
}
#[tauri::command]
pub async fn library_cover_backfill_configure(
app: AppHandle,
@@ -576,7 +665,7 @@ pub async fn library_cover_backfill_configure(
.set_session(enabled && session.is_some(), session)
.await;
if enabled {
let _ = try_schedule_full_pass(&app).await;
let _ = try_schedule_full_pass(&app, false).await;
}
Ok(())
}
@@ -706,7 +795,7 @@ pub async fn cover_cache_stats_server(
) -> Result<CoverCacheStatsDto, String> {
let st = state(&app)?;
let guard = st.lock().await;
let (bytes, entry_count) = dir_usage_for_server(&guard.root, &server_index_key);
let (bytes, entry_count) = cached_dir_usage_for_server(&guard.root, &server_index_key);
let (pressure, auto_download_enabled) = guard.pressure_from_bytes(bytes);
Ok(CoverCacheStatsDto {
bytes,
@@ -741,7 +830,13 @@ pub async fn cover_cache_clear_server(
if path.is_dir() {
std::fs::remove_dir_all(&path).map_err(|e| e.to_string())?;
}
invalidate_dir_usage_cache(&server_index_key);
drop(guard);
// Clearing drops files the cheap idle-gate signature can't see, so re-arm
// the backfill worker — otherwise the next sync-idle would skip the rescan.
if let Some(worker) = app.try_state::<Arc<CoverBackfillWorker>>() {
worker.rearm_idle_gate().await;
}
let _ = app.emit(
"cover:cache-cleared",
serde_json::json!({ "serverIndexKey": server_index_key }),
@@ -904,6 +999,12 @@ pub async fn cover_cache_clear(app: AppHandle) -> Result<(), String> {
}
}
drop(guard);
if let Ok(mut map) = dir_usage_cache().lock() {
map.clear();
}
if let Some(worker) = app.try_state::<Arc<CoverBackfillWorker>>() {
worker.rearm_idle_gate().await;
}
let _ = app.emit("cover:cache-cleared", serde_json::json!({}));
Ok(())
}
@@ -956,7 +1057,7 @@ pub async fn library_cover_progress(
let index_key = server_index_key.clone();
let store = runtime.store.clone();
tauri::async_runtime::spawn_blocking(move || {
let cached_dirs = count_cached_cover_ids(&root, &index_key);
let cached_dirs = cached_dir_usage_for_server(&root, &index_key).1 as i64;
collect_cover_progress(
&store,
&library_server_id,
+1
View File
@@ -762,6 +762,7 @@ pub fn run() {
cover_cache::library_cover_backfill_pulse,
cover_cache::library_cover_backfill_reset_cursor,
cover_cache::library_cover_backfill_set_ui_priority,
cover_cache::library_cover_backfill_set_parallel,
cover_cache::library_cover_backfill_run_full_pass,
cover_cache::cover_revalidate_enqueue,
cover_cache::cover_revalidate_tick,
+14 -3
View File
@@ -223,9 +223,15 @@ export async function libraryCoverBackfillPulse(): Promise<CoverBackfillPulseRes
return invoke<CoverBackfillPulseResult>('library_cover_backfill_pulse');
}
/** Start one full-catalog pass on the native runtime (works when the window is inactive). */
export async function libraryCoverBackfillRunFullPass(): Promise<{ started: boolean }> {
return invoke<{ started: boolean }>('library_cover_backfill_run_full_pass');
/**
* Start one full-catalog pass on the native runtime (works when the window is inactive).
* `force` bypasses the idle gate and clears the fetch-failed backoff so previously
* unfetchable (404) covers are retried — used by the manual "Run full pass now".
*/
export async function libraryCoverBackfillRunFullPass(
force = false,
): Promise<{ started: boolean }> {
return invoke<{ started: boolean }>('library_cover_backfill_run_full_pass', { force });
}
export async function libraryCoverBackfillResetCursor(): Promise<void> {
@@ -237,6 +243,11 @@ export async function libraryCoverBackfillSetUiPriority(hold: boolean): Promise<
return invoke('library_cover_backfill_set_ui_priority', { hold });
}
/** Perf-probe only: retune cover backfill threads (download + encode). Returns the clamped value applied. */
export async function libraryCoverBackfillSetParallel(threads: number): Promise<number> {
return invoke<number>('library_cover_backfill_set_parallel', { threads });
}
export async function libraryCoverClearFetchFailures(serverIndexKey: string): Promise<number> {
return invoke<number>('library_cover_clear_fetch_failures', { serverIndexKey });
}
@@ -79,9 +79,14 @@ export default function CoverCacheStrategySection() {
);
}, [servers, refreshRow]);
// Recompute on entry only. Live updates during an active backfill arrive via the
// `cover:library-progress` event (carries done/total/pending/bytes/entryCount), and
// clearing the cache emits `cover:cache-cleared`. A slow 5-minute tick is just a
// safety net for changes made outside this view (e.g. browsing-time caching); it is
// not needed for correctness, so we avoid re-walking the cover dirs in a tight loop.
useEffect(() => {
refreshAll();
const id = window.setInterval(refreshAll, 15_000);
const id = window.setInterval(refreshAll, 300_000);
return () => window.clearInterval(id);
}, [refreshAll]);
@@ -128,9 +133,6 @@ export default function CoverCacheStrategySection() {
refreshAll();
}
}));
unsubs.push(await listen('cover:tier-ready', () => {
refreshAll();
}));
})();
return () => {
for (const u of unsubs) u();
@@ -173,6 +175,9 @@ export default function CoverCacheStrategySection() {
await coverCacheClearServer(clearTarget.indexKey);
clearDiskSrcCacheForServer(clearTarget.indexKey);
await refreshRow(clearTarget.serverId, clearTarget.indexKey);
if (clearTarget.serverId === activeServerId) {
wakeLibraryCoverBackfill();
}
showToast(t('settings.coverCacheStrategyClearSuccess'), 4000, 'success');
} catch {
showToast(t('settings.coverCacheStrategyClearError'), 5000, 'error');
@@ -0,0 +1,95 @@
import { useEffect, useRef, useState } from 'react';
import {
coverGetPipelineQueueStats,
libraryCoverBackfillResetCursor,
libraryCoverBackfillRunFullPass,
libraryCoverBackfillSetParallel,
} from '../../../api/coverCache';
const COVER_THREADS_MIN = 1;
const COVER_THREADS_MAX = 16;
/**
* Perf-probe-only knob for cover backfill concurrency (download + encode pools
* move together). Deliberately not surfaced in app Settings — it is a live
* diagnostics/experiment control. The value is process-local and resets to the
* backend default on app restart.
*/
export default function PerfCoverThreadsControl() {
const [threads, setThreads] = useState<number | null>(null);
const pendingRef = useRef(false);
useEffect(() => {
let cancelled = false;
void coverGetPipelineQueueStats()
.then(stats => {
if (!cancelled) setThreads(stats.libraryBackfillHttpMax || COVER_THREADS_MIN);
})
.catch(() => {
if (!cancelled) setThreads(COVER_THREADS_MIN);
});
return () => {
cancelled = true;
};
}, []);
const apply = (next: number) => {
setThreads(next);
if (pendingRef.current) return;
pendingRef.current = true;
void libraryCoverBackfillSetParallel(next)
.then(applied => setThreads(applied))
.catch(() => {})
.finally(() => {
pendingRef.current = false;
});
};
const value = threads ?? COVER_THREADS_MIN;
const [running, setRunning] = useState(false);
const runPass = () => {
if (running) return;
setRunning(true);
void libraryCoverBackfillResetCursor()
.then(() => libraryCoverBackfillRunFullPass(true))
.catch(() => {})
.finally(() => setRunning(false));
};
return (
<section className="perf-live-poll" aria-label="Cover backfill threads">
<div className="perf-live-poll__title">Cover backfill</div>
<label className="perf-live-poll__row">
<span className="perf-live-poll__label">
Threads
{' '}
<span className="perf-live-poll__value">{value}</span>
</span>
<input
type="range"
min={COVER_THREADS_MIN}
max={COVER_THREADS_MAX}
step={1}
value={value}
disabled={threads === null}
onChange={e => apply(Number(e.target.value))}
/>
</label>
<button
type="button"
className="perf-live-poll__run"
onClick={runPass}
disabled={running}
>
{running ? 'Starting…' : 'Run full pass now'}
</button>
<p className="perf-live-poll__hint">
Download + encode concurrency for background cover warm-up. Higher = faster
backfill but more CPU/network while it runs. Resets to default on restart.
lib /N in the cover overlay only fills when covers are actually missing
clear a servers cover cache first, then run a pass.
</p>
</section>
);
}
@@ -12,6 +12,7 @@ import PerfProbeMetricCard, { PerfProbeMetricSection } from './PerfProbeMetricCa
import PerfOverlayAppearanceControls from './PerfOverlayAppearanceControls';
import PerfOverlayModeControls from './PerfOverlayModeControls';
import PerfLivePollControls from './PerfLivePollControls';
import PerfCoverThreadsControl from './PerfCoverThreadsControl';
function memoryBarPct(rssKb: number, maxKb: number): number {
if (maxKb <= 0) return 0;
@@ -60,6 +61,7 @@ export default function SidebarPerfProbeMonitorTab() {
<PerfOverlayModeControls />
<PerfOverlayAppearanceControls />
<PerfLivePollControls />
<PerfCoverThreadsControl />
<PerfProbeMetricSection title="Pipeline overlays" hint="Rust / UI queues">
<PerfProbeMetricCard
label="FPS"
+1
View File
@@ -144,6 +144,7 @@ const CONTRIBUTOR_ENTRIES = [
'Genres: local index genre browse with Subsonic fallback, aligned counts cache, scroll restore, hold-to-shuffle play (PR #937)',
'Live Search: scoped browse on Artists, Albums, New Releases, Tracks, and Composers — header badge, ghost restore, album title FTS, session stash (PR #938)',
'Performance: idle Rust CPU — backfill coordinator park, probe overlay stability, throttled idle polls, lazy cover prefetch restore (PR #939)',
'Cover backfill: disk-free idle gate, snapshot-diff worklist, live-tunable parallelism, transient-error retries, and memoized offline & cache stats to stop idle CPU spin (PR #943)',
],
},
{
+21
View File
@@ -206,6 +206,27 @@
color: var(--text-primary);
}
.perf-live-poll__run {
margin-top: 10px;
align-self: flex-start;
padding: 4px 10px;
font-size: 11px;
border-radius: 6px;
border: 1px solid var(--border-subtle, color-mix(in srgb, var(--text-muted) 30%, transparent));
background: color-mix(in srgb, var(--text-muted) 12%, transparent);
color: var(--text-primary);
cursor: pointer;
}
.perf-live-poll__run:disabled {
opacity: 0.6;
cursor: default;
}
.perf-live-poll__run:not(:disabled):hover {
background: color-mix(in srgb, var(--text-muted) 20%, transparent);
}
.perf-live-poll__hint {
margin: 6px 0 0;
font-size: 10px;