Repository navigation
Conversation
AbstractStatefulConfigurable kept per-route configs in a HashMap. The map is written while routes are built (AbstractRateLimiter on FilterArgsEvent, GatewayFilterSpec.RequestRateLimiterSpec in the Java DSL) and read on every request (RedisRateLimiter#loadConfiguration, Bucket4jRateLimiter#loadConfiguration). CachingRouteLocator only synchronizes updateCache(), so route builds can run concurrently, e.g. when an application-triggered refresh overlaps the refresh on startup. Concurrent puts into an empty HashMap can silently drop entries while it resizes. A dropped entry makes loadConfiguration() throw "No Configuration found for route" for every request on that route until the next refresh. When the route also has a circuit breaker, the exception is counted as a failure and opens the breaker even though the backend is healthy. Use a ConcurrentHashMap instead. Keys are route ids (never null: CompositeRouteDefinitionLocator assigns one and Route requires one) and values are newConfig() instances, so the null restriction of ConcurrentHashMap does not affect existing callers. Signed-off-by: Jaeyeob Jung <61595394+Frankle97@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
AbstractStatefulConfigurablestores per-route configs (used byRedisRateLimiterandBucket4jRateLimiter) in a plainHashMap. It is written while routes are built (AbstractRateLimiter#onApplicationEvent,GatewayFilterSpec.RequestRateLimiterSpec#rateLimiter) and read on every request. If two route builds fill the map at the same time, entries can be silently lost during a resize, andRedisRateLimiter#loadConfigurationthen throwsNo Configuration found for route ...for every request on that route until the next refresh.Background
CachingRouteLocatorrebuilds routes on everyRefreshRoutesEvent, on the thread that published it, and onlyupdateCache()is synchronized. These events come from several independent sources: the startupContextRefreshedEvent, discovery heartbeats andInstanceRegisteredEvent,RefreshScopeRefreshedEvent,/actuator/gateway/refresh, or the application itself (e.g. a scheduled reload of routes from a database). When two of them overlap, route builds, and the writes to this map, run on two threads at once. The map is most exposed right after startup, while it is still empty and resizes as it is filled.We ran into this when an application-triggered refresh overlapped the startup refresh: on one instance, every rate-limited route kept failing until the next refresh.
Solution
Back the map with a
ConcurrentHashMap. Existing callers never usenullkeys or values (keys are route ids, values arenewConfig()instances), so behavior is otherwise unchanged.AbstractStatefulConfigurableTestspopulates the map from several threads and checks that no route id is lost.