Client fetch storm: SPA polls every endpoint ~5/sec in bursts #396
Labels
No labels
actions
bug
cli
duplicate
enhancement
fork
forum
git storage
help wanted
insights
invalid
issues
moderation
oidc
ownership transfer
packages
pr/merge protection rules
projects
pull requests
question
releases
sponsorships
tags
webhooks
wiki
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
crueber/walhub#396
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found during the #394 investigation (see container logs 2026-09-12 17:15-17:19Z). With the settings page open, the browser issues ~5 GET/sec across ALL endpoints (access, summary, social, me, orgs, tasks, notifications...) in bursts lasting tens of seconds — ~200 requests/endpoint in 30 minutes from a single tab. useData TTL revalidation is 5s, so this is not TTL behavior; suspect an SSE-frame invalidation storm (task progress frames?) or an effect loop. Not the bounce cause (every response observed was state-coherent), but it hammers the server and drowns the access log. Fix: find the loop (tasks poller? collab invalidation per frame?), restore sensible cadence, add a client fetch-rate guard/test.
If this is an issue, that's fine. I think it still needs to make all those calls. But it really does need to show the correct visibility.
Fix open as PR #402 (branch fix/issue-396): #402 — TTL-aware SSE invalidation flush + tasks busy cadence 1.5s→5s + fetch-rate guard test (8 tests). No backend change, no new deps. Not merging per instructions.
REVIEW PR #402 (fix/issue-396, client fetch storm) — verified in scratch worktree, no browser (node tests + reasoning, as scoped).
(1) DIAGNOSIS SOUND. Reproduced the measurement headless: with data.js reverted to main, the new guard tests fail 5/8 — spread replay of 30 frames/macrotask costs 30 refetches on fresh keys (TTL fully bypassed, microtask coalescing cannot batch cross-task frames), and one post-TTL frame costs 2 GETs on a sha key (explicit +
*-prefix double generation). Post-fix all 8 pass. The flush fix bounds it: prefix-expansion dedup (one generation/entry/flush) + TTL-gate per key means sustained frame rates decay to TTL cadence. Claim 'burst tests fail 5/8 pre-fix' verified exactly.(2) invalidate() STAYS EAGER — no #41 regression. web/src/lib/data.js:277-284 untouched; mutation path bypasses scheduleInvalidate entirely (sole caller of scheduleInvalidate is invalidateCollab, data.js:418). Pinned by the mutation-eager test (double invalidate() on a fresh key = 2 refetches).
(3) STALENESS BOUND = the key's own TTL (5s typical; perms/milestones/releases/social 30s; release 60s; assignables 300s) — the same windows the read path already accepts via the 08 §6 table, so no new freshness contract. Narrow race noted (not blocking): a frame arriving during an in-flight fetch is skipped while the pre-frame body commits → staleness ≤ in-flight + TTL. Pre-fix that frame superseded the fetch; acceptable tradeoff, bounded. Same-tab mutations unaffected (eager path). Cross-tab/other-user mutations lag ≤ TTL.
(4) IMMUTABLE WINDOWS NEVER SSE-REFETCH — CORRECT. events:/diff:/prcommits: are Infinity in the TTL table (collab.js:8-28); the gate skips them once settled (Infinity fresh window) but still fetches unseeded entries (at==0 falls through to invalidate). Timelines append frames directly; diff/prcommits are sha-addressed immutable. No conflict with the fsck re-audit path (uses invalidatePrefix→invalidate(), the eager path, not scheduleInvalidate).
(5) TASKS 1.5s→5s — P7-COMPLIANT. P7 (features primitives: tasks+SSE, no polling loops for collab state) is untouched: collab updates still arrive via SSE frames. The overlay polls the tasks LIST — pre-existing sanctioned shape (recursive setTimeout chain, Repo.jsx:419, never setInterval; idle 15s unchanged) — and per-task progress still streams via SSE attach (repo.task(id, onEvent)). Only the busy cadence relaxes on the hottest endpoint; pill percentages stay live at 5s, fine for minutes-long maintenance/follow tasks. Reduces traffic; directionally compliant.
(6) GUARD TESTS MEANINGFUL. fetch-rate-guard.test.js: sync/spread/mixed bursts → zero refetches on fresh keys; post-TTL exactly-once (incl. prefix+explicit dedup pin); immutable pin; #41 eager pin; unknown/uncached no-op pin; BUSY/IDLE cadence + setTimeout-chain source pins. Budgets assert fetches/key/window with counting fetchers on the real prefetchData→invalidateCollab path.
(7) DOCS/DEPS/BACKEND. 12_web_ui §2.9 + 08 §4 + both 'Decisions & deviations' entries updated in the same change (law 12 ✓). No package.json change, no new imports (law 1 ✓). No backend change — correct: server emission is event-driven, storm was client-side fan-out.
SMALL FIX PUSHED (
d1aa97c): ttlForKey comment claimed unlisted prefixes 'all revalidate at 5s in their useData seeds' — wrong for collaborators: (Access.jsx:291, seeds at TTL.perms/30s; fail-fresh direction, behavior safe) and overview:/ops: have no seeds at all. Comment corrected; behavior untouched.TESTS: full node suite 796 tests / 793 pass / 0 fail / 3 skip (smoke skips, server absent) + vite build green, all in scratch worktree with the fix (incl.
d1aa97c). Note: 2 smoke tests fail in THIS environment only because a stray non-walhub service answers :8080/healthz 200 + / 401 — environmental, unrelated (suite is green with WALHUB_TEST_WEB_BASE_URL pointed at a dead port). No browser drive (scoped out; DOM untouched — data.js/Repo.jsx constant only). Main worktree left clean/read-only.RECOMMENDATION: ready to merge.
Fixed by PR #402 (review clean + one comment fix by reviewer; diagnosis reproduced, #41 intact, staleness bounded, guard break-verified), merged. Closing.