feat: solver exponential backoff

[PBillingsby]

Summary

This pull request makes the following changes:

• Add exponential backoff mechanism for connection retries
• Improve detection of specific WebSocket closure errors
• Add error propagation to ensure issues during WebSocket setup are logged
• Add mutexes (sync.Mutex) to synchronize access to the WebSocket connection during reconnection or ping operations
• Change the signature of ConnectWebSocket to return an error alongside the response channel
• Move ctx to start of params for SolverClient

Task/Issue reference

Closes: #429

Test plan

1. Modify pkg/http/websocket_server.go to force an error in HandleFunc:

r.HandleFunc(path, func(w http.ResponseWriter, r *http.Request) {
		http.Error(w, "Internal Server Error", http.StatusInternalServerError)
		return

2. Start the stack
3. Observe the job-creator terminal for exponential backoff logs

ERR pkg/http/websocket_client.go:99 > WebSocket connection failed: websocket: bad handshake
Reconnecting in 2.200 seconds...
2024-11-25T11:11:28-05:00 DBG pkg/http/websocket_client.go:99 > WebSocket connection connecting: ws://localhost:8080/api/v1/ws?&Type=JobCreator&ID=0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC
2024-11-25T11:11:28-05:00 ERR pkg/http/websocket_client.go:99 > WebSocket connection failed: websocket: bad handshake
Reconnecting in 3.640 seconds...

4. Cancel the solver service context to see the context cancellation handling:

2024-11-25T11:11:54-05:00 INF pkg/http/websocket_client.go:99 > Solver service appears to be down
2024-11-25T11:11:54-05:00 ERR pkg/solver/client.go:34 > Error in WebSocket connection. error="solver service unavailable: dial tcp [::1]:8080: connect: connection refused"
Error: solver service unavailable: dial tcp [::1]:8080: connect: connection refused

[bgins]

nit: Could we rename this to exponent?

[bgins]

Suggested change

	log.Info().Msg("Exiting readMessage loop due to context cancellation.")
	log.Debug().Msg("Exiting readMessage loop due to context cancellation.")

This detail is a bit internal, would suggest dropping the log level to debug.

[bgins]

At the moment, we only use this helper function to connect to the solver. But we might re-use it in other contexts, so we may want to reword these errors to something like "Websocket server closed connection".

[bgins]

Also, we may want to log errors we where we will attempt again at the warn level.

[bgins]

Will returning here cause us to give up on our connection attempts?