I notice this while ago in Tutorial: Edge gateway-mt:
INFO startupcheck/startupcheck.go:45 checking "<sat_address>"
# will stuck at checking satellite step forever..
INFO httpserver/server.go:320 HTTP server started {"addr": "[::]:20010"}
INFO httpserver/server.go:320 HTTPS server started {"addr": "[::]:20011"}
Run into this again today, basically, it hang forever if satellite unavailable at startupcheck time, it a problem because the binary still running, but the service is in limbo state.
I ask Claude to debug, it tell me this is a deadlock:
The bug:
If a configured satellite is unreachable at startup, the check correctly times out (--server.startup-check.timeout, default 30s) — but the process then hangs forever instead of exiting. Never serves, never dies.
Cause: Peer.Run (pkg/server/server.go) uses errs2.Group, which waits for all goroutines and never cancels siblings on error. When the startup check fails, s.server.Run(ctx) returns, but s.processor.Run (accesslogs) keeps running — it only stops via processor.Close(), which is only called by Peer.Close(), which only runs after Peer.Run returns. Deadlock.
The fix:
Fix: use errgroup.WithContext instead, so a failing goroutine cancels the rest:
func (s *Peer) Run(ctx context.Context) (err error) {
defer mon.Task()(&ctx)(&err)
minioOnce.Do(func() { minio.StartMinio(!s.config.InsecureDisableTLS) })
g, ctx := errgroup.WithContext(ctx)
g.Go(func() error { <-ctx.Done(); return errs2.IgnoreCanceled(s.processor.Close()) })
g.Go(s.processor.Run)
g.Go(func() error { return s.server.Run(ctx) })
return g.Wait()
}
(Same pattern cmd/gateway-mt/main.go already uses one level up.) Note: processor.Close() could now fire twice — worth checking it’s idempotent.
I intent to post to github at first, but storj github issue tab is also in limbo state, so…