Add fire-and-forget async statement submission

[Ch4s3]

Summary

Adds fire-and-forget async statement submission so a caller can start a Snowflake statement, get its handle back without holding a connection pool slot for the statement's full duration, and retrieve its rows later.

Today Snowflex.Transport.Http polls Snowflake's 202 response to completion inside the GenServer.call that owns the checked-out connection (await_async_execution/3), so a slot stays occupied until the statement finishes. Snowflake's SQL API already supports async=true, which acknowledges with a statementHandle as soon as it accepts the statement — Snowflex just never set it.

{:ok, handle} = Snowflex.submit_async(MyRepo, "CALL rebuild_scenarios(?)", [scenario_id])

# fetch_result/3 answers {:ok, :running} until it finishes, so it can poll alone
case Snowflex.fetch_result(MyRepo, handle) do
  {:ok, :running} -> :still_running
  {:ok, %Snowflex.Result{rows: rows}} -> rows
  {:error, error} -> Logger.error(Exception.message(error))
end

# or, if you don't care about rows
{:ok, :succeeded} = Snowflex.statement_status(MyRepo, handle)
:ok = Snowflex.cancel_statement(MyRepo, handle)

Rows come back decoded exactly as Repo.query/3 decodes them, because fetch_result/3 reuses the existing gather_results/3 + format_response_body/1 machinery (including parallel multi-partition merging).

Design notes

Routed through the pool, not around it. Since async=true returns in milliseconds, going through the pool costs only the submission round-trip while keeping JWT refresh, retry/backoff, Finch pool routing, telemetry and log metadata.

Reuses the execute path. Snowflex.Query gains an :op field; Connection.handle_execute/4 dispatches on it. For status/fetch/cancel, :statement carries the handle so it lands in snowflex_statement log metadata — deliberate, so a stuck statement is greppable. Happy to split it into a separate field if you'd prefer.

Transport callbacks are optional, so existing custom transports still satisfy the behaviour with no change and no compile warnings; calling one that's missing returns a descriptive Snowflex.Error.

:current_statement is untouched by the async path — that field belongs to the cursor/streaming path, and writing it would corrupt an in-flight Repo.stream on the same connection.

Included bug fixes

Two pre-existing unit mismatches, both in code this PR builds on. I'd originally planned to split these out, but the first one directly undermines the feature — for a fire-and-forget statement the server-side timeout is the only bound, since nothing client-side waits on it.

• The statement request body sent timeout in milliseconds where Snowflake defines seconds, so the 45s default asked Snowflake to wait ~12.5 hours, effectively disabling the server-side statement timeout. Behaviour change: statements that previously ran unbounded will now be cancelled at the configured timeout. Called out in the CHANGELOG.
• JWT exp added a millisecond :token_lifetime to a second-precision timestamp, advertising ~7 days instead of the configured 10 minutes. Verified fixed: exp - iat = 600s.

Also: the JWT is now cached for its lifetime instead of being re-signed (PEM decode + RSA) on every request — verified 1 distinct token across 4 requests, previously 4.

Review round

I ran an adversarial review over the first two commits (9 finder angles, 5 verifiers, mutation testing). 03f7fcf fixes what it found. The ones worth calling out:

• Async ops inside Repo.checkout/2 silently corrupted the enclosing checkout. run_op/5 asked the pool for a second connection instead of reusing the held one; at pool_size: 1 that queued until the timeout fired and the outer checkout was force-disconnected — while still returning {:ok, handle}, so it looked fine. Reproduced at 5.0s + disconnect, now 0.07s clean. Fixed by reusing the checked-out connection the way stream_query/5 already did.
• Timeouts raised instead of returning errors. The new entry points lacked the try/catch execute_statement/4 has, so an uncaught exit escaped handle_execute and DBConnection re-raised it. Note declare/fetch also lack it, so execute_statement was the outlier — worth a follow-up to make all six consistent.
• Status polls could block ~187s against a 45s deadline, because receive_timeout equalled the call timeout and Req retried the GET. Retries are now off for status (Snowflake already answered with 202) and the call deadline has grace over the HTTP timeout.
• No way to read the rows — the original motivation for the whole change. Hence fetch_result/3.
• The pool-occupancy test passed for the wrong reason. Mutation testing proved that reintroducing the polling bug left its central assertion green, since submit_async is awaited before the second query starts. It now asserts on submit's own elapsed time, which does fail under that mutation.

Two candidate findings were refuted by testing and deliberately not "fixed": Code.ensure_loaded? is load-bearing (proved by purging the module) and its embedded-mode failure path is unreachable for a running module; and params: %{} vs no params is byte-identical on the wire (Finch normalizes "" and nil) — tidied to [] anyway.

Test plan

• mix test --exclude integration — 159 passed (25 async tests)
• mix compile --warnings-as-errors
• mix credo --strict — no issues
• mix format --check-formatted
• mix assay (dialyzer) — 0 errors
• Mutation-tested the pool guard: reintroducing polling fails the test on both the elapsed-time assertion and refute_received
• Empirically verified the checkout fix, the timeout path, JWT caching and exp
• Integration tests against a live Snowflake account — not run; the 68 failures in HttpRepoTest/ConnectionTest are credential-dependent and fail identically on master (verified by diffing failing test names — the sets match)

The statement-timeout fix in particular deserves a live check before release, since it changes how long Snowflake lets a statement run.

Note on the linked issue

Two API names in #197 don't exist: there is no Snowflex.query_async/2 (the real mechanism is the implicit 202 polling above), and no Snowflex.HttpClient.transport_options/1 — the function consumers actually use is Snowflex.Transport.Http.options/1, which is public and documented. So the issue's "coupled to Snowflex internals" concern is unfounded. I'll follow up on the issue to correct both.

Refs #197

[jeremyowensboggs]

I think this looks great!

[jeremyowensboggs]

We should consider adding retry: false here as well - to prevent a transient error from causing a re-submission. https://github.com/pepsico-ecommerce/galileo/blob/master/apps/api_services/lib/api_services/common/snowflake/async_statement.ex#L244

[mphfish]

Should we use Snowflake's requestId for that? Have callers supply one if they want idempotent requests?

Iirc Req does not automatically retry POST

[jeremyowensboggs]

hmm -yup you are correct - it's not on by default http://req.hexdocs.pm/0.7.2/Req.Steps.html#retry/1-request-options.

Withdrawn.

[Ch4s3]

So we're good?