Skip to content

Commit 03b3bf7

Browse files
committed
Add SessionPool.CheckHealthAsync connectivity probe
新增 SessionPool.CheckHealthAsync 连通性探针 IsOpen() is a lifecycle flag - it reports whether the caller has opened the pool and not yet closed it. Because the client keeps no heartbeat, it stays true after the server goes down, and users routinely mistake it for a health check, e.g. `if (pool.IsOpen()) return;` in a lazy-open guard, which then never rebuilds the pool. This adds the connectivity check that IsOpen() is mistaken for, without changing IsOpen() itself. IsOpen() 是生命周期标志,仅表示调用方是否已打开且尚未关闭连接池。由于客户端无心跳, 服务端断开后它仍为 true,用户普遍将其误当作健康检查(例如惰性打开守卫里的 `if (pool.IsOpen()) return;`),从而导致连接池永远不会被重建。本提交新增了 IsOpen() 被误认为具备的连通性检查能力,且不改变 IsOpen() 自身行为。 - Add SessionPool.CheckHealthAsync(CancellationToken), which issues one lightweight request on an idle pooled connection and returns a SessionPoolHealth snapshot carrying Status, AvailableClients, TotalPoolSize, FailedReconnections, Message and the underlying Error. 新增 SessionPool.CheckHealthAsync(CancellationToken):在空闲连接上发送一次轻量请求, 返回包含状态、可用连接数、池容量、重连失败次数、说明和底层异常的 SessionPoolHealth 快照。 - Add SessionPoolHealthStatus with four outcomes: NotOpen (no network call is made), Healthy, Degraded (open but no idle connection to probe with - a saturation signal, not a server verdict) and Unhealthy (a connection was available but the server did not answer). 新增 SessionPoolHealthStatus 四种结果:NotOpen(不发起网络调用)、Healthy、 Degraded(池已打开但无空闲连接可用于探测,表示饱和而非服务端故障)、 Unhealthy(有可用连接但服务端未响应)。 - Add ConcurrentClientQueue.TryTake so the probe never blocks. A health endpoint must not queue behind application load, so when every client is busy the probe returns Degraded immediately instead of waiting. 新增 ConcurrentClientQueue.TryTake 使探针永不阻塞:健康检查端点不应排在业务负载之后, 因此所有连接繁忙时立即返回 Degraded 而不是等待。 - The borrowed connection is always returned to the pool and a failed probe does not close it, so the existing reconnect-on-use path is unchanged. 借出的连接总会归还池中,探测失败也不会关闭它,因此既有的按需重连路径保持不变。 - Forward CheckHealthAsync from TableSessionPool with identical semantics. TableSessionPool 以相同语义转发 CheckHealthAsync。 - Add tests for the NotOpen path, non-blocking TryTake semantics and the health snapshot; document the API in docs/SessionPool_Exception_Handling.md and docs/API.md. 补充 NotOpen 路径、TryTake 非阻塞语义与健康快照的测试;在相关文档中说明该 API。
1 parent 178ebb0 commit 03b3bf7

7 files changed

Lines changed: 351 additions & 21 deletions

File tree

docs/API.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,8 @@ var tablet =
4444
| -------------- | ------------------------- | ------------------------ | ----------------------------- |
4545
| Open | bool | open session | session_pool.Open(false) |
4646
| Close | null | close session | session_pool.Close() |
47-
| IsOpen | null | check if the pool was opened and not yet closed by the caller. It is a lifecycle flag, **not** a connectivity probe: it stays `true` after the server goes down, because the client keeps no heartbeat and reconnects on demand instead. | session_pool.IsOpen() |
47+
| IsOpen | null | check whether the pool was opened and not yet closed by the caller. It is a lifecycle flag, **not** a connectivity probe: it stays `true` after the server goes down, because the client keeps no heartbeat and reconnects on demand instead. For connectivity use `CheckHealthAsync` | session_pool.IsOpen() |
48+
| CheckHealthAsync | CancellationToken=default | probe server connectivity on an idle pooled connection; returns a `SessionPoolHealth` snapshot | await session_pool.CheckHealthAsync() |
4849
| OpenDebugMode | LoggingConfiguration=null | open debug mode | session_pool.OpenDebugMode() |
4950
| CloseDebugMode | null | close debug mode | session_pool.CloseDebugMode() |
5051
| SetTimeZone | string | set time zone | session_pool.GetTimeZone() |

docs/SessionPool_Exception_Handling.md

Lines changed: 62 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -54,24 +54,62 @@ catch (SessionPoolDepletedException ex)
5454
}
5555
```
5656

57-
## `IsOpen()` is a lifecycle flag, not a health check
57+
## Checking connectivity: `CheckHealthAsync` vs `IsOpen`
5858

59-
`SessionPool.IsOpen()` reports whether **you** have opened the pool and not yet closed it. It is not a
60-
connectivity probe:
59+
`SessionPool.IsOpen()` reports whether **you** have opened the pool and not yet closed it. It is a
60+
lifecycle flag, not a connectivity probe:
6161

6262
- It becomes `true` after a successful `Open()` and only returns to `false` when you call `Close()`.
6363
- The client runs no heartbeat, so a server that goes down does **not** flip it back to `false`.
6464
Reconnection happens lazily, on the next operation.
6565

66-
This means the following common guard never re-opens the pool, because the flag stays `true` forever:
66+
This makes the following common guard a trap - it short-circuits forever, so the pool is never rebuilt:
6767

6868
```csharp
69-
// Anti-pattern: this short-circuits even while every connection is dead
69+
// Anti-pattern: IsOpen() stays true even while every connection is dead
7070
if (_pool != null && _pool.IsOpen()) return;
7171
```
7272

73-
To reason about actual availability, use the health metrics below, or simply let an operation throw
74-
`SessionPoolDepletedException` and handle it.
73+
Use `CheckHealthAsync` when you need to know whether the server is actually reachable. It issues one
74+
lightweight request on an idle pooled connection and returns a `SessionPoolHealth` snapshot:
75+
76+
```csharp
77+
var health = await sessionPool.CheckHealthAsync();
78+
79+
if (!health.IsHealthy)
80+
{
81+
Console.WriteLine(health); // e.g. "Unhealthy: The server did not answer the probe: ..."
82+
Console.WriteLine(health.Status); // NotOpen | Healthy | Degraded | Unhealthy
83+
Console.WriteLine(health.Error); // the underlying exception, when Status is Unhealthy
84+
}
85+
```
86+
87+
### Status values
88+
89+
| Status | Meaning |
90+
| ----------- | ------------------------------------------------------------------------------------------- |
91+
| `NotOpen` | The pool has not been opened yet, or has already been closed. No network call is attempted. |
92+
| `Healthy` | The server answered the probe. |
93+
| `Degraded` | The pool is open but no connection was idle to probe with. Says nothing about the server. |
94+
| `Unhealthy` | A connection was available but the server did not answer. See `Error` for the cause. |
95+
96+
### Behaviour worth knowing
97+
98+
- **The probe never blocks.** If every client is busy, it returns `Degraded` immediately instead of
99+
queueing behind ordinary work, so a health endpoint cannot be starved by application load. It is also
100+
unaffected by the pool wait timeout described below.
101+
- **The borrowed connection is always returned** to the pool, and a failed probe does not close it. The
102+
regular reconnect-on-use path still applies to the next operation.
103+
- **`Degraded` is not a server verdict.** Under high concurrency it simply means the pool was saturated at
104+
that instant. Treat a persistent `Degraded` as a sizing signal, not an outage.
105+
- **Bound the probe yourself if you need to.** Pass a cancellation token:
106+
107+
```csharp
108+
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2));
109+
var health = await sessionPool.CheckHealthAsync(cts.Token);
110+
```
111+
112+
`TableSessionPool` exposes the same `CheckHealthAsync` method with identical semantics.
75113

76114
## Pool Wait Timeout
77115

@@ -94,7 +132,9 @@ var sessionPool = new SessionPool.Builder()
94132

95133
When the wait budget is exhausted, the operation throws `SessionPoolDepletedException` with the reason
96134
`Connection pool is empty and wait time out(...ms)`. Raise `SetPoolWaitTimeoutInMs` if your workload
97-
legitimately queues behind long operations; lower it if you would rather fail fast and retry.
135+
legitimately queues behind long operations; lower it if you would rather fail fast and retry. The budget is
136+
a single deadline for the whole call: waiters woken by a returned connection that they lose the race for do
137+
not restart it.
98138

99139
> **Note:** before this setting existed, the wait budget was derived from the connection timeout and then
100140
> misinterpreted as seconds, which turned the 500 ms default into a ~41 minute block. If you are upgrading
@@ -333,6 +373,8 @@ var sessionPool = new SessionPool.Builder()
333373

334374
### Health Check Implementation
335375

376+
The simplest health check delegates to the built-in probe and only adds your own policy on top:
377+
336378
```csharp
337379
public class SessionPoolHealthCheck
338380
{
@@ -343,34 +385,35 @@ public class SessionPoolHealthCheck
343385
_pool = pool;
344386
}
345387

346-
public HealthStatus CheckHealth()
388+
public async Task<HealthStatus> CheckHealthAsync()
347389
{
348-
var availableRatio = (double)_pool.AvailableClients / _pool.TotalPoolSize;
390+
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2));
391+
var health = await _pool.CheckHealthAsync(cts.Token);
349392

350-
if (_pool.FailedReconnections > 10)
393+
if (health.Status == SessionPoolHealthStatus.Unhealthy)
351394
{
352395
return new HealthStatus
353396
{
354397
Status = "Critical",
355-
Message = $"High reconnection failures: {_pool.FailedReconnections}",
398+
Message = health.Message,
356399
Recommendation = "Check IoTDB server availability"
357400
};
358401
}
359402

360-
if (availableRatio < 0.25)
403+
if (health.Status == SessionPoolHealthStatus.Degraded)
361404
{
362405
return new HealthStatus
363406
{
364407
Status = "Warning",
365-
Message = $"Low available clients: {_pool.AvailableClients}/{_pool.TotalPoolSize}",
408+
Message = $"No idle connection to probe ({health.AvailableClients}/{health.TotalPoolSize} available)",
366409
Recommendation = "Consider increasing pool size"
367410
};
368411
}
369412

370413
return new HealthStatus
371414
{
372415
Status = "Healthy",
373-
Message = $"Pool healthy: {_pool.AvailableClients}/{_pool.TotalPoolSize} available"
416+
Message = health.Message
374417
};
375418
}
376419
}
@@ -383,6 +426,9 @@ public class HealthStatus
383426
}
384427
```
385428

429+
If you would rather not issue a network call on every scrape, the metrics below can be sampled on their
430+
own - just remember they describe the pool, not the server.
431+
386432
### Metrics Collection for Monitoring Systems
387433

388434
```csharp
@@ -559,7 +605,7 @@ public class ProductionSessionPoolManager
559605
The SessionPool exception handling and health monitoring features provide comprehensive tools for building robust IoTDB applications:
560606

561607
- Use `SessionPoolDepletedException` to understand and react to pool issues
562-
- Treat `IsOpen()` as a lifecycle flag, never as a connectivity check
608+
- Use `CheckHealthAsync` for connectivity checks; `IsOpen()` is a lifecycle flag only
563609
- Tune `SetPoolWaitTimeoutInMs` separately from `SetConnectionTimeoutInMs`
564610
- Monitor `AvailableClients`, `TotalPoolSize`, and `FailedReconnections`; read `UnrealizedCapacity` as
565611
capacity not yet materialized rather than as an outage signal

src/Apache.IoTDB/ConcurrentClientQueue.cs

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,6 @@ public void Return(Client client)
5858
public void AddRef() => Interlocked.Increment(ref _ref);
5959
public int GetRef() => Volatile.Read(ref _ref);
6060
public void RemoveRef() => Interlocked.Decrement(ref _ref);
61-
6261
/// <summary>
6362
/// The maximum time, in milliseconds, that <see cref="Take"/> waits for a client to be
6463
/// returned to the pool before throwing. Defaults to 10000 (10 seconds).
@@ -79,6 +78,12 @@ public int Timeout
7978
set => TimeoutInMs = value * 1000;
8079
}
8180

81+
/// <summary>
82+
/// Attempts to take a client without ever blocking. Returns false when no client is idle.
83+
/// Use this for probes and diagnostics, which must not queue behind ordinary work.
84+
/// </summary>
85+
public bool TryTake(out Client client) => ClientQueue.TryDequeue(out client);
86+
8287
public Client Take()
8388
{
8489
Client client = null;

src/Apache.IoTDB/SessionPool.cs

Lines changed: 53 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -471,12 +471,62 @@ public async Task<Client> Reconnect(Client originalClient = null, CancellationTo
471471
/// <remarks>
472472
/// This reflects the lifecycle of the pool object only - it is NOT a server-connectivity probe.
473473
/// The client performs no heartbeat, so a server going down does not flip this back to false;
474-
/// it stays true until <see cref="Close"/> is called. To reason about connectivity, use
475-
/// <see cref="AvailableClients"/>, <see cref="UnrealizedCapacity"/> and <see cref="FailedReconnections"/>,
476-
/// or simply let an operation throw <see cref="SessionPoolDepletedException"/>.
474+
/// it stays true until <see cref="Close"/> is called. Use <see cref="CheckHealthAsync"/> when you
475+
/// need to know whether the server is actually reachable, or read <see cref="AvailableClients"/>,
476+
/// <see cref="UnrealizedCapacity"/> and <see cref="FailedReconnections"/> for pool state.
477477
/// </remarks>
478478
public bool IsOpen() => !_isClose;
479479

480+
/// <summary>
481+
/// Probes server connectivity by issuing one lightweight request on an idle pooled connection,
482+
/// and returns a snapshot of the result. This is the connectivity check that <see cref="IsOpen"/>
483+
/// is often mistaken for.
484+
/// </summary>
485+
/// <remarks>
486+
/// The probe never blocks waiting for a connection: if every client is busy, the call returns
487+
/// <see cref="SessionPoolHealthStatus.Degraded"/> immediately rather than queueing behind ordinary
488+
/// work. The borrowed connection is always returned to the pool, and a failed probe does not close
489+
/// it - the regular reconnect-on-use path still applies to the next operation.
490+
/// Pass a cancellation token if you want to bound how long the probe may take.
491+
/// </remarks>
492+
public async Task<SessionPoolHealth> CheckHealthAsync(CancellationToken cancellationToken = default)
493+
{
494+
if (_isClose || _clients == null)
495+
{
496+
return new SessionPoolHealth(SessionPoolHealthStatus.NotOpen, 0, TotalPoolSize, FailedReconnections,
497+
"The pool has not been opened yet, or it has already been closed.");
498+
}
499+
500+
if (!_clients.TryTake(out var client))
501+
{
502+
return new SessionPoolHealth(SessionPoolHealthStatus.Degraded, 0, TotalPoolSize, FailedReconnections,
503+
"No idle connection was available to probe. The pool is either saturated by concurrent work or has lost its connections.");
504+
}
505+
506+
SessionPoolHealthStatus status;
507+
string message;
508+
Exception error = null;
509+
try
510+
{
511+
await client.ServiceClient.getTimeZoneAsync(client.SessionId, cancellationToken);
512+
status = SessionPoolHealthStatus.Healthy;
513+
message = "The server answered the probe.";
514+
}
515+
catch (Exception e)
516+
{
517+
status = SessionPoolHealthStatus.Unhealthy;
518+
message = $"The server did not answer the probe: {e.Message}";
519+
error = e;
520+
_logger?.LogWarning(e, "Health probe failed for session pool");
521+
}
522+
finally
523+
{
524+
_clients.Add(client);
525+
}
526+
527+
return new SessionPoolHealth(status, AvailableClients, TotalPoolSize, FailedReconnections, message, error);
528+
}
529+
480530
public async Task Close()
481531
{
482532
if (_isClose)
Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
using System;
21+
22+
namespace Apache.IoTDB
23+
{
24+
/// <summary>
25+
/// Outcome of a <see cref="SessionPool.CheckHealthAsync"/> probe.
26+
/// </summary>
27+
public enum SessionPoolHealthStatus
28+
{
29+
/// <summary>
30+
/// The pool has not been opened yet, or it has already been closed.
31+
/// </summary>
32+
NotOpen,
33+
34+
/// <summary>
35+
/// The server answered the probe on a pooled connection.
36+
/// </summary>
37+
Healthy,
38+
39+
/// <summary>
40+
/// The pool is open but could not be probed, because no connection was idle at that moment.
41+
/// This says nothing about the server: the pool may simply be saturated by concurrent work.
42+
/// </summary>
43+
Degraded,
44+
45+
/// <summary>
46+
/// A connection was available but the server did not answer the probe.
47+
/// </summary>
48+
Unhealthy
49+
}
50+
51+
/// <summary>
52+
/// A point-in-time snapshot of pool connectivity, returned by <see cref="SessionPool.CheckHealthAsync"/>.
53+
/// Unlike <see cref="SessionPool.IsOpen"/> - which only reports whether the caller has opened the pool -
54+
/// this reflects whether the server actually answered just now.
55+
/// </summary>
56+
public class SessionPoolHealth
57+
{
58+
/// <summary>
59+
/// The probe outcome.
60+
/// </summary>
61+
public SessionPoolHealthStatus Status { get; }
62+
63+
/// <summary>
64+
/// True only when <see cref="Status"/> is <see cref="SessionPoolHealthStatus.Healthy"/>.
65+
/// </summary>
66+
public bool IsHealthy => Status == SessionPoolHealthStatus.Healthy;
67+
68+
/// <summary>
69+
/// Idle clients in the pool at the time the snapshot was taken.
70+
/// </summary>
71+
public int AvailableClients { get; }
72+
73+
/// <summary>
74+
/// Configured maximum capacity of the pool.
75+
/// </summary>
76+
public int TotalPoolSize { get; }
77+
78+
/// <summary>
79+
/// Cumulative tally of reconnection failures since the pool was opened.
80+
/// </summary>
81+
public int FailedReconnections { get; }
82+
83+
/// <summary>
84+
/// Human-readable explanation of <see cref="Status"/>.
85+
/// </summary>
86+
public string Message { get; }
87+
88+
/// <summary>
89+
/// The exception that caused an <see cref="SessionPoolHealthStatus.Unhealthy"/> result, if any.
90+
/// </summary>
91+
public Exception Error { get; }
92+
93+
public SessionPoolHealth(SessionPoolHealthStatus status, int availableClients, int totalPoolSize,
94+
int failedReconnections, string message, Exception error = null)
95+
{
96+
Status = status;
97+
AvailableClients = availableClients;
98+
TotalPoolSize = totalPoolSize;
99+
FailedReconnections = failedReconnections;
100+
Message = message;
101+
Error = error;
102+
}
103+
104+
public override string ToString()
105+
=> $"{Status}: {Message} (available {AvailableClients}/{TotalPoolSize}, failed reconnections {FailedReconnections})";
106+
}
107+
}

src/Apache.IoTDB/TableSessionPool.cs

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,15 @@ public void OpenDebugMode(Action<ILoggingBuilder> configure)
7070
sessionPool.OpenDebugMode(configure);
7171
}
7272

73+
/// <summary>
74+
/// Probes server connectivity on an idle pooled connection. See
75+
/// <see cref="SessionPool.CheckHealthAsync"/> for the semantics.
76+
/// </summary>
77+
public async Task<SessionPoolHealth> CheckHealthAsync(CancellationToken cancellationToken = default)
78+
{
79+
return await sessionPool.CheckHealthAsync(cancellationToken);
80+
}
81+
7382
public async Task Close()
7483
{
7584
await sessionPool.Close();

0 commit comments

Comments
 (0)