Tree fetch 404 toast on repos whose backend is healthy #203

New issue

Closed

opened 2026-09-08 17:01:16 +00:00 by crueber · 3 comments

crueber commented

2026-09-08 17:01:16 +00:00

Owner

Opening some repos shows "not found: " toast on the tree fetch

Opening repos (e.g. o/r) shows an error toast with key SHA:<sha>:TREE: (empty path) and body not found: <sha>, where <sha> is the repo's CURRENT head sha.

Evidence gathered (could not reproduce server-side — backend is healthy)

GET /o/r/api → 200, head 2dd90b80… (same sha as the toast), on BOTH 127.0.0.1:8080 and https://hub.packden.us.
GET /o/r/api/tree/main and /o/r/api/tree/<full-sha> → 200 on both lanes, both hosts.
Client key built at web/src/lib/data.js:339 (sha:<sha>:tree:<path>, CSS-uppercased in the tray chip); SDK builds /tree/{rev}[/{path}] correctly.
NO server code produces exactly not found: <sha> (nearest: tree %s:%s not found in bind_wal.go:301, repository not found: in wal/types.go, object not found: in store).
Tree GETs carry Cache-Control: private, max-age=31536000, immutable — a poisoned disk-cache entry replays failures (the #41 pattern); the stack was recreated ~10× today, giving ample windows for transient failures to get cached.

Fix

Reproduce LIVE in a browser (open the repo, watch the network tab): capture the actual failing request (URL, status, body, served-from-cache?) and fix the real cause. Prime suspects, in order: (1) poisoned immutable-cache entry replaying a transient failure (fix: no-store/error-path cache discipline on sha-addressed GETs, cache-busting or entry invalidation); (2) a resolve-then-fetch race serving a sha the tree path can't see; (3) proxy/host-specific path mangling on hub.packden.us. Do NOT "fix" by hiding the toast — the fetch genuinely fails in the user's browser.

Acceptance criteria

Root cause proven with a captured failing request (URL + status + body + cache state in the PR).
Repo opens clean with no toast, both themes, zero console errors; no new deps.
node --test green (+ Go gates if backend touched).

# Opening some repos shows "not found: <sha>" toast on the tree fetch Opening repos (e.g. `o/r`) shows an error toast with key `SHA:<sha>:TREE:` (empty path) and body `not found: <sha>`, where `<sha>` is the repo's CURRENT head sha. ## Evidence gathered (could not reproduce server-side — backend is healthy) - `GET /o/r/api` → 200, head `2dd90b80…` (same sha as the toast), on BOTH `127.0.0.1:8080` and `https://hub.packden.us`. - `GET /o/r/api/tree/main` and `/o/r/api/tree/<full-sha>` → 200 on both lanes, both hosts. - Client key built at `web/src/lib/data.js:339` (`sha:<sha>:tree:<path>`, CSS-uppercased in the tray chip); SDK builds `/tree/{rev}[/{path}]` correctly. - NO server code produces exactly `not found: <sha>` (nearest: `tree %s:%s not found` in `bind_wal.go:301`, `repository not found: ` in `wal/types.go`, `object not found: ` in store). - Tree GETs carry `Cache-Control: private, max-age=31536000, immutable` — a poisoned disk-cache entry replays failures (the #41 pattern); the stack was recreated ~10× today, giving ample windows for transient failures to get cached. ## Fix Reproduce LIVE in a browser (open the repo, watch the network tab): capture the actual failing request (URL, status, body, served-from-cache?) and fix the real cause. Prime suspects, in order: (1) poisoned immutable-cache entry replaying a transient failure (fix: no-store/error-path cache discipline on sha-addressed GETs, cache-busting or entry invalidation); (2) a resolve-then-fetch race serving a sha the tree path can't see; (3) proxy/host-specific path mangling on hub.packden.us. Do NOT "fix" by hiding the toast — the fetch genuinely fails in the user's browser. ## Acceptance criteria - [ ] Root cause proven with a captured failing request (URL + status + body + cache state in the PR). - [ ] Repo opens clean with no toast, both themes, zero console errors; no new deps. - [ ] `node --test` green (+ Go gates if backend touched).

crueber referenced this issue from a commit

2026-09-08 17:36:22 +00:00

Fix #203: sha-addressed resolve retries past a cold serving copy

crueber referenced this issue from a pull request that will close it,

2026-09-08 17:36:29 +00:00

Fix #203: tree-fetch failure behind healthy backend #204

crueber commented

2026-09-08 17:36:33 +00:00

Author

Owner

Root cause found and fixed in PR #204 (branch fix/issue-203, not merged).

It is NOT cache poisoning — every failing request goes to the network and the 404s carry no Cache-Control. Captured live via CDP on a cold server: GET /acme/demo/api/tree/8de1c9a5… → 404 not found: 8de1c9a5… (51 bytes, fromDiskCache=false) while GET …/api/resolve → 200 with the same sha and GET …/api/tree/main → 200 (and healed the copy).

Mechanism: on a cold serving copy (fresh restart/recreate — refs synced, objects/pack empty), the tree handler re-resolves the full head sha via git rev-parse in the empty copy → miss → 404 from the Resolve fallback arm (bind_wal.go, via notFoundOr). The UI's resolve→sha flow never issues a ref-named request, so nothing ever triggers the serve sync that would materialize the packs — every visit re-toasts. That is also why your curls looked healthy: tree/main heals, tree/<sha> does not.

Fix: Resolve serve-syncs once and retries rev-parse on a full-sha miss before 404ing. Post-fix browser proof on a cold server: clean open, tree renders, no toast, zero console errors, both themes.

Root cause found and fixed in PR #204 (branch fix/issue-203, not merged). It is NOT cache poisoning — every failing request goes to the network and the 404s carry no Cache-Control. Captured live via CDP on a cold server: `GET /acme/demo/api/tree/8de1c9a5…` → 404 `not found: 8de1c9a5…` (51 bytes, fromDiskCache=false) while `GET …/api/resolve` → 200 with the same sha and `GET …/api/tree/main` → 200 (and healed the copy). Mechanism: on a cold serving copy (fresh restart/recreate — refs synced, objects/pack empty), the tree handler re-resolves the full head sha via `git rev-parse` in the empty copy → miss → 404 from the Resolve fallback arm (bind_wal.go, via notFoundOr). The UI's resolve→sha flow never issues a ref-named request, so nothing ever triggers the serve sync that would materialize the packs — every visit re-toasts. That is also why your curls looked healthy: `tree/main` heals, `tree/<sha>` does not. Fix: Resolve serve-syncs once and retries rev-parse on a full-sha miss before 404ing. Post-fix browser proof on a cold server: clean open, tree renders, no toast, zero console errors, both themes.

crueber referenced this issue

2026-09-08 17:36:39 +00:00

Push path writes pack- prefixed manifest checksums (bucket contract violation + duplicate packs) #205

crueber commented

2026-09-08 17:39:28 +00:00

Author

Owner

PR #204 review (scratch worktree /tmp/pr204 @ 2ba62a1; main worktree untouched, no browser re-drive per instructions — relying on author's CDP capture in #203 comment-1763):

PASS law 6 / hot path: retry in internal/api/bind_wal.go:213 is gated on sha == '' && revIsFullSHA(segs[0]) — ref-named hits return at :194-205 before rev-parse, sha hits return with sha != '' so the block is unreachable. Zero added store round trips on hit. The extra Sync(Serve) runs only on the failure path (doc-allowed).
PASS bounded: sync-once-then-retry-once, no loop. Sync error swallowed to fall through to the original 404 (:222, if serr == nil); revParse never returns non-nil err (recipes.go:51-57 returns '',nil on miss) so no error-masking path. No retry storm possible.
PASS 404 shape: genuine misses keep exact not found: <seg> at both Resolve and HTTP layers — asserted in TestWalResolveColdCopyRetriesServe case 1 (bind_wal_test.go).
PASS concurrency: walView holds no locks across the call; engine.Sync owns syncMu/packMu per doc 05 §5.2 (comment cites it). Two concurrent cold resolves may duplicate the serve materialize — idempotent, acceptable; open path itself is single-flight (wal/registry.go:28).
PASS imports: no new imports (body-only change; import block unchanged).
PASS gates (scratch worktree): gofmt clean, go vet clean, go test -race ./internal/api/ PASS incl. new TestWalResolveColdCopyRetriesServe, package coverage 95.4% >= 95% gate.
PASS docs: 07_api.md §9.3 step-3 + Decisions entries accurately describe the behavior (full-sha-only, once, hot path unchanged, exact 404 shape, live CDP proof ruling out cache poisoning).
PASS #205 out of scope: push-path pack- manifest checksums (write path) vs this read-path Resolve retry (local-fixture test, no manifest involvement) — genuinely separate, not load-bearing.

No fixes pushed — nothing to fix. MERGE RECOMMENDATION: ready to merge.

PR #204 review (scratch worktree /tmp/pr204 @ 2ba62a1; main worktree untouched, no browser re-drive per instructions — relying on author's CDP capture in #203 comment-1763): PASS law 6 / hot path: retry in internal/api/bind_wal.go:213 is gated on `sha == '' && revIsFullSHA(segs[0])` — ref-named hits return at :194-205 before rev-parse, sha hits return with sha != '' so the block is unreachable. Zero added store round trips on hit. The extra Sync(Serve) runs only on the failure path (doc-allowed). PASS bounded: sync-once-then-retry-once, no loop. Sync error swallowed to fall through to the original 404 (:222, `if serr == nil`); revParse never returns non-nil err (recipes.go:51-57 returns '',nil on miss) so no error-masking path. No retry storm possible. PASS 404 shape: genuine misses keep exact `not found: <seg>` at both Resolve and HTTP layers — asserted in TestWalResolveColdCopyRetriesServe case 1 (bind_wal_test.go). PASS concurrency: walView holds no locks across the call; engine.Sync owns syncMu/packMu per doc 05 §5.2 (comment cites it). Two concurrent cold resolves may duplicate the serve materialize — idempotent, acceptable; open path itself is single-flight (wal/registry.go:28). PASS imports: no new imports (body-only change; import block unchanged). PASS gates (scratch worktree): gofmt clean, go vet clean, go test -race ./internal/api/ PASS incl. new TestWalResolveColdCopyRetriesServe, package coverage 95.4% >= 95% gate. PASS docs: 07_api.md §9.3 step-3 + Decisions entries accurately describe the behavior (full-sha-only, once, hot path unchanged, exact 404 shape, live CDP proof ruling out cache poisoning). PASS #205 out of scope: push-path pack- manifest checksums (write path) vs this read-path Resolve retry (local-fixture test, no manifest involvement) — genuinely separate, not load-bearing. No fixes pushed — nothing to fix. MERGE RECOMMENDATION: ready to merge.

crueber closed this issue

2026-09-08 17:39:59 +00:00

crueber referenced this issue from a commit

2026-09-08 17:39:59 +00:00

Merge pull request 'Fix #203: tree-fetch failure behind healthy backend' (#204) from fix/issue-203 into main

crueber commented

2026-09-08 17:40:00 +00:00

Author

Owner

Fixed by PR #204 (review: hot path zero-cost, bounded retry, 404 shape preserved; 95.4% coverage), merged. Closing.

crueber added the

git storage