Support dynamic query limit configuration#3532
Open
lbschanno wants to merge 16 commits into
Open
Conversation
Support the ability to dynamically update concurrent query limits while
the webservers are running. Add the class `QueryLimitReloader`. A
`QueryLimitReloader` will operate within the Zookeeper namespace
`QueryLimitConfig` to listen for events that should trigger a reload. A
reload will be triggered by the following:
- The creation or modification of the node `/path` with non-empty data.
- The creation, modification, or deletion of the node `/trigger`.
Upon a triggering event, the reloader will attempt to load a valid
`QueryLimitConfiguration` and supply it to all listeners, such as the
one registered by `QueryLimiter`. The reloader will attempt to
deserialize a `QueryLimiterConfiguration` from a file. It will look for
the filepath in the node `/path`. The target file may be JSON, XML, or
YAML, and may have the URI schemes `http:`, `https:`, `hdfs:`, `file:`,
or none.
After a reload attempt, the following nodes will be updated with the
status of the attempt:
- `/attempts/<serverIpAddress>/cause`: The triggering event for the
reload. This may be `PATH_NODE_CREATED`, `PATH_NODE_MODIFIED`,
`TRIGGER_NODE_CREATED`, `TRIGGER_NODE_MODIFIED`, or
`TRIGGER_NODE_DELETED`.
- `/attempts/<serverIpAddress>/status`: The final status of the reload
attempt. It may be one of the following:
- `SUCCESS`: A valid `QueryLimitConfiguration` was loaded and supplied
to all listeners.
- `LISTENER_ERROR`: A valid `QueryLimitConfiguration` was loaded, but
an exception was thrown when supplying it to one or more listeners.
- `RELOAD_ERROR`: A valid `QueryLimitConfiguration` could not be
loaded.
- `/attempts/<serverIpAddress>/time`: The time of the reload attempt in
ISO-8601 format.
- `/attempts/<serverIpAddress>/errors`: A node whose children will
contain brief descriptions of the errors that occurred. Exists only if
any errors occurred. The children's paths will have the format
`error_X` where X is a value from `0...N` where N is one less than the
total errors that were captured. The full stacktrace for the errors
will be captured in the logs.
When the `QueryLimiter` is supplied with a new configuration, it will
update its internal configuration, and rebuild its internal limit
providers.
Closes #3511
ivakegg
reviewed
May 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support the ability to dynamically update concurrent query limits while the webservers are running.
Add the class
ZkObjectPublisherthat provides the ability to trigger and publish updates of a configured class to all subscribers. See this README for full details.Integrate using a configured
ZkObjectPublisherwithin theQueryLimiterclass to update its internal configured limits whenever an update has been triggered.Closes #3511