Full static membership support - #168
Conversation
Fencing, unlike a regular revoke, doesn't drain the pipeline before the producer stops: messages already emitted downstream are still in processors and batchers, and they route through the allocator ETS tables via :partition_by. Clearing the tables on fence made those lookups raise ArgumentError and crash the batchers holding in-flight messages. Since a fenced producer never receives new assignments, the retained entries are harmless and die with the pipeline.
|
Hi @whatyouhide, I was on vacation last week. I'll check the PR this week. |
|
@slashmili thank you sounds good! For context, we've forked broadway_kafka since and have been using all of my open PRs (merged to our fork's |
| def resolve_offset(topic, partition, current_offset, offset_reset_policy, config) do | ||
| policy = offset_reset_policy_value(offset_reset_policy) | ||
|
|
||
| # This is only for testing. |
There was a problem hiding this comment.
what kind of testing you had in mind? is this still needed for testing?
| {:brod, "~> 3.16 or ~> 4.0"}, | ||
| # We need our brod fork until https://github.com/kafka4beam/brod/pull/669 | ||
| # (fenced static member support) is merged and released upstream. | ||
| {:brod, github: "knocklabs/brod", ref: "d16aa8cea37ad9aa8e8591aba0eb2ef814c93e6b"}, |
There was a problem hiding this comment.
if I remember correctly, adding this mean we can't publish this package to Hex until this git reference is removed.
I suggest for this PR to be on hold until the PR is merged back to the main stream
There was a problem hiding this comment.
This is not meant to be merged as is yes, needs to wait for that PR to be merged. Sorry, my bad for not calling that out!
2 participants
No description provided.