Push mirroring: push repos to an upstream (password/token/SSH-key auth, keypair generation) with on-push + optional scheduled sync #623

New issue

Closed

opened 2026-09-16 14:24:05 +00:00 by crueber · 1 comment

crueber commented

2026-09-16 14:24:05 +00:00

Owner

What's requested

Push mirroring for repositories: walhub should be able to push a repository's refs (and objects) to a remote upstream — so walhub can be the primary and a Forgejo/GitHub mirror or personal backup host receives the pushes. This complements the existing pull mirror (internal/mirror), which today only fetches FROM an upstream and refuses pushes (IsMirror in internal/mirror/mirror.go).

Configured post-hoc only, in repo Settings — never at create or import time (mirroring a repo's destination is a lifecycle decision made after the repo exists, matching how the pull mirror is configured).

Auth support (all three, per-mirror choice)

Username/password — standard HTTPS basic auth against the upstream.
Token — bearer-style token auth (Forgejo/GitHub access tokens) over HTTPS.
SSH key — deploy-key style SSH push. Two paths:
- User provides a keypair (paste/upload private key + known_hosts or accept-insecure toggle decision for the implementer).
- Walhub generates a keypair, shows the public key for the user to install on the upstream, and holds the private key as a repo secret.

Credentials must never be echoed back in full after save (write-only fields; show presence/last-4-style confirmation instead) and must never appear in last_result error strings (the existing mirror doc scrubs failures — keep that contract).

Sync triggers

On-push: after a successful push lands in walhub, enqueue a mirror-sync task for the push destination (existing (repo, kind) single-flight task machinery in internal/wal/tasks.go composes here; a task kind distinct from pull mirror-sync, e.g. mirror-push-sync).
Optional scheduled sync: same preset cron model the pull mirror uses (internal/mirror/mirror.go presetCrons / NextFire), reusing internal/bundle/cron.go. Default off — on-push only unless the user opts into a schedule.

Evidence / current state

internal/mirror/mirror.go — MirrorDoc (upstream_url, schedule, last_result, consecutive_failures) and IsMirror push-refusal probe; sidecar precedent at meta/mirror.json.
internal/mirror/sync.go — pull-direction sync; comment at line 434 notes server-side publish bypasses the push funnel.
Feature-surface map: mirrors are pull-only today (POST /api/v1/repos/mirrors, New.jsx mirror-create mode, summary.mirror projection).
No push-mirror code exists in the tree (grep for push-mirror concepts: zero hits).

Architecture notes

Reusable primitives: cron parser (internal/bundle/cron.go), (repo, kind) task single-flight (internal/wal/tasks.go), sidecar doc pattern (internal/mirror Load/store.MirrorKey), separate-cadence loop pattern (internal/maintain/follow.go).
New sidecar doc (e.g. meta/pushmirror.json) or an extended mirror doc — planner's call; keep pull and push mirror configs independent so either can exist alone.
Auth material needs a secrets location; reuse or extend the existing repo-sidecar precedent (access.json / meta/placeholder.json) — planner's call on storage shape and what redaction applies.
On-push trigger hooks into the post-push path; ensure server-side publish (which bypasses the push funnel, internal/mirror/sync.go:434) does NOT accidentally fan out push-mirror syncs.
Git plumbing: pushing to an upstream is git push --mirror-style ref+object transfer over the git transport; consider what happens with walhub's manifest-store refs vs forge git refs (refs live in the manifest store, not forge git refs — the sync must push what the store publishes, matching the pull direction's ref reconstruction).
Reusable routes seam: new top-level endpoints need all three route twins (internal/api/routes.go: template /api/v1/…, api-browser/v1, services/api).
Settings surface: new section in repo Settings (reuse the pull mirror's settings/create UI patterns in web/src/pages/).

Acceptance criteria

A repo can configure a push mirror post-hoc in Settings with upstream URL + one of: username/password, token, or SSH key (provided or walhub-generated keypair with public key shown).
Credentials are write-only in the UI/API and redacted from all error surfaces.
A successful push to the repo enqueues a push-mirror sync; the upstream receives the refs and objects.
Optional scheduled sync can be enabled with a preset schedule; default is on-push only.
Sync outcomes are visible in the UI (last result, next fire), mirroring the pull mirror's status surface.
Push-mirror state appears in the repo summary projection with ETag coverage (cached-summary mutability law).
Server-side publish does not trigger push-mirror sync fan-out.
Either pull mirror or push mirror can be configured independently; configuring one never requires the other.
Coverage gate stays green (make cover, 95% per-package floor) for new/changed packages.

## What's requested Push mirroring for repositories: walhub should be able to push a repository's refs (and objects) to a remote upstream — so walhub can be the primary and a Forgejo/GitHub mirror or personal backup host receives the pushes. This complements the existing pull mirror (`internal/mirror`), which today only fetches FROM an upstream and refuses pushes (`IsMirror` in `internal/mirror/mirror.go`). Configured post-hoc only, in repo Settings — never at create or import time (mirroring a repo's destination is a lifecycle decision made after the repo exists, matching how the pull mirror is configured). ## Auth support (all three, per-mirror choice) 1. **Username/password** — standard HTTPS basic auth against the upstream. 2. **Token** — bearer-style token auth (Forgejo/GitHub access tokens) over HTTPS. 3. **SSH key** — deploy-key style SSH push. Two paths: - **User provides a keypair** (paste/upload private key + known_hosts or accept-insecure toggle decision for the implementer). - **Walhub generates a keypair**, shows the public key for the user to install on the upstream, and holds the private key as a repo secret. Credentials must never be echoed back in full after save (write-only fields; show presence/last-4-style confirmation instead) and must never appear in `last_result` error strings (the existing mirror doc scrubs failures — keep that contract). ## Sync triggers - **On-push**: after a successful push lands in walhub, enqueue a mirror-sync task for the push destination (existing `(repo, kind)` single-flight task machinery in `internal/wal/tasks.go` composes here; a task kind distinct from pull `mirror-sync`, e.g. `mirror-push-sync`). - **Optional scheduled sync**: same preset cron model the pull mirror uses (`internal/mirror/mirror.go` `presetCrons` / `NextFire`), reusing `internal/bundle/cron.go`. Default off — on-push only unless the user opts into a schedule. ## Evidence / current state - `internal/mirror/mirror.go` — `MirrorDoc` (upstream_url, schedule, last_result, consecutive_failures) and `IsMirror` push-refusal probe; sidecar precedent at `meta/mirror.json`. - `internal/mirror/sync.go` — pull-direction sync; comment at line 434 notes server-side publish bypasses the push funnel. - Feature-surface map: mirrors are pull-only today (`POST /api/v1/repos/mirrors`, `New.jsx` mirror-create mode, `summary.mirror` projection). - No push-mirror code exists in the tree (grep for push-mirror concepts: zero hits). ## Architecture notes - Reusable primitives: cron parser (`internal/bundle/cron.go`), `(repo, kind)` task single-flight (`internal/wal/tasks.go`), sidecar doc pattern (`internal/mirror` Load/`store.MirrorKey`), separate-cadence loop pattern (`internal/maintain/follow.go`). - New sidecar doc (e.g. `meta/pushmirror.json`) or an extended mirror doc — planner's call; keep pull and push mirror configs independent so either can exist alone. - Auth material needs a secrets location; reuse or extend the existing repo-sidecar precedent (`access.json` / `meta/placeholder.json`) — planner's call on storage shape and what redaction applies. - On-push trigger hooks into the post-push path; ensure server-side publish (which bypasses the push funnel, `internal/mirror/sync.go:434`) does NOT accidentally fan out push-mirror syncs. - Git plumbing: pushing to an upstream is `git push --mirror`-style ref+object transfer over the git transport; consider what happens with walhub's manifest-store refs vs forge git refs (refs live in the manifest store, not forge git refs — the sync must push what the store publishes, matching the pull direction's ref reconstruction). - Reusable routes seam: new top-level endpoints need all three route twins (`internal/api/routes.go`: template `/api/v1/…`, `api-browser/v1`, `services/api`). - Settings surface: new section in repo Settings (reuse the pull mirror's settings/create UI patterns in `web/src/pages/`). ## Acceptance criteria - [ ] A repo can configure a push mirror post-hoc in Settings with upstream URL + one of: username/password, token, or SSH key (provided or walhub-generated keypair with public key shown). - [ ] Credentials are write-only in the UI/API and redacted from all error surfaces. - [ ] A successful push to the repo enqueues a push-mirror sync; the upstream receives the refs and objects. - [ ] Optional scheduled sync can be enabled with a preset schedule; default is on-push only. - [ ] Sync outcomes are visible in the UI (last result, next fire), mirroring the pull mirror's status surface. - [ ] Push-mirror state appears in the repo summary projection with ETag coverage (cached-summary mutability law). - [ ] Server-side publish does not trigger push-mirror sync fan-out. - [ ] Either pull mirror or push mirror can be configured independently; configuring one never requires the other. - [ ] Coverage gate stays green (`make cover`, 95% per-package floor) for new/changed packages.

crueber added the

enhancement

git storage

labels

2026-09-16 14:24:14 +00:00

crueber added this to the v1 milestone

2026-09-16 14:24:14 +00:00

crueber referenced this issue from a commit

2026-09-16 15:28:33 +00:00

Push mirroring with on-push fan-out + optional schedule (Forgejo #623)

crueber referenced this issue from a pull request that will close it,

2026-09-16 15:29:13 +00:00

Push mirroring with on-push fan-out + optional schedule (Fix #623) #624

crueber referenced this issue from a commit

2026-09-16 15:46:13 +00:00

Push-mirror review fixes (Forgejo #623): scoped refspecs, env-only credentials, keygen/update guards, ETag coverage

crueber referenced this issue

2026-09-16 15:46:47 +00:00

Push mirroring with on-push fan-out + optional schedule (Fix #623) #624

crueber referenced this issue

2026-09-16 15:47:34 +00:00

Push-mirror SSH host keys: persist accept-new trust + surface host-key status #625

crueber referenced this issue from a commit

2026-09-16 16:10:05 +00:00

Merge pull request 'Push mirroring with on-push fan-out + optional schedule (Fix #623)' (#624) from fix/issue-623 into main

crueber closed this issue

2026-09-16 16:10:05 +00:00

crueber commented

2026-09-16 16:10:14 +00:00

Author

Owner

Fixed by #624 (merged): push mirroring with on-push fan-out + optional schedule (password/token/SSH provided-or-generated via stdlib ed25519, git-push subprocess, sidecar secrets with redaction, summary ETag, independent pull/push). Review fixed pre-merge: scoped refspecs (no forge-ref leak), shell-injection via username, keygen/update guards, ETag gaps. Rendered verification attached on the PR (headless Chromium desktop + 390px, zero console errors; live file:// on-push sync confirmed). Coverage gates green. TOFU persistence tracked as #625.