Reusing battle-tested tooling for WATER

[thedevyashsaini]

Hey mentors, I was hoping to give my 2 cents regarding the discussions and the alternate approach I had in mind.

After getting clarification from a previous discussion (referring to this comment: #2 (reply in thread))

At a high level, WATER seems to be something like this REGARDLESS of how different people and their proposals phrase it:

If we implemented the whole thing FROM SCRATCH in the most straightforward way, this is roughly what we would end up building.

Basically, it’s a pretty straightforward distributed orchestration setup - edge nodes register themselves, send telemetry via heartbeats, workflows get ingested at the controller, an allocation engine decides where they should run, and the dispatcher triggers execution on the selected node.

But looking at it more carefully, this clearly introduces a lot of moving parts. And the more I thought about it, the more it felt like we would basically be reinventing something very close to Kubernetes, just for a slightly different use case.

Considering that this is meant to support HEALTHCARE infrastructure, I feel that prioritizing battle-tested architecture over the “more straightforward” approach is important, since ignoring an edge case in distributed infrastructure can easily cause SERIOUS issues.

That led me to a second approach that seemed more stable and realistic -- building WATER as a control layer on top of k8s rather than implementing everything from scratch.

In that model-

• Edge or cloud nodes join a WATER k8s cluster

• Nodes can be separated via namespaces to represent privacy levels (for example hospital nodes = high privacy, cloud nodes = lower privacy)

• node-exporter and dcgm-exporter can expose CPU / memory / disk / GPU telemetry

• WATER ingests workflows, parses the requirements, and then schedules workloads using k8s primitives like -

  • namespaces

  • node selectors

  • affinity / anti-affinity rules

In other words, Kubernetes would handle the actual scheduling, while WATER acts more like an allocation middleware / control layer that translates workflows into k8s jobs.

This way we avoid reinventing a lot of infrastructure that k8s already handles very well.

If we needed some additional functionality on the nodes, we could potentially ship a small sidecar and wrap the join process in a CLI script that -

• installs the required sidecar
• runs kubeadm join

so a single command prepares the node and joins it to the WATER cluster.

However, this introduces the first potential issue.

Hospital machines (edge nodes) may be behind NAT or firewalls that block inbound connections. Even if we solve connectivity using something like VPC peering, there are still a few structural issues with running a standard k8s control plane across distant edge nodes -

• latency between control plane and nodes affecting status updates
• non-trivial control plane traffic over long distances
• edge network partitions potentially causing duplicate processing or inconsistent state

These issues could make a vanilla Kubernetes deployment difficult for this use case.

Because of that, I started looking into KubeEdge, which seems designed specifically for this kind of setup.

KubeEdge extends Kubernetes with CloudCore and EdgeCore, where edge nodes connect to the control plane via outbound connections (WebSocket / MQTT), which helps bypass firewall restrictions and network boundaries.

It also allows edge nodes to continue operating locally using a local database and MQTT even if the connection to the control plane drops.

So the architecture in that case becomes something like:

WATER controller layer → Kubernetes → KubeEdge (CloudCore + EdgeCore)

This seems to solve the network boundary problem quite nicely.

The remaining issue I see is this -

If some cloud nodes are already part of existing k8s clusters (for example CPU nodes in one cluster and GPU nodes in another), then joining them into a new WATER cluster is not straightforward.

kubelet and edgecore cannot coexist, and a kubelet cannot be configured to belong to two clusters simultaneously without obv some duct-tape solution.

So while KubeEdge + a WATER control layer currently seems like the most robust architecture to me, it relies on the assumption that the nodes we want to use are not already bound to existing clusters.

My current thinking is that even if they are, migrating them into a unified WATER cluster might still be a reasonable trade-off, because building a custom orchestration layer from scratch would mean re-solving a lot of reliability problems that systems like Kubernetes have already handled over many years.

So I mainly wanted clarification on two assumptions before going further with this direction -

• Should we assume that hospital nodes may be behind firewalls / NAT and therefore cannot accept inbound connections?
• Should we assume that cloud nodes might already belong to existing k8s clusters that we are NOT WILLING to modify, or is it acceptable for the infrastructure to be reorganized so that nodes join a WATER cluster?

This isn’t a proposal draft or anything like that - I mainly wanted to clarify these architectural constraints before going deeper into a specific implementation direction.

Any feedback or corrections on the assumptions above would be really helpful.

[pradeeban]

@thedevyashsaini at the earliest version of WATER (as we are literally an empty repository right now), we could make safe assumptions to ensure our project remains within the 350 hours of coding period.

Surely there will be firewalls in healthcare networks. We might find ways to handle this, perhaps making the healthcare networks to communicate first. But we may not need to handle all of those by this summer.

Just to be aware: this execution might be decentralized (no administratively centralized), rather than just being distributed. Something to be aware of when we make an architecture such as a Kubernetes-based deployment.

As a new project, we are open to alternative approaches and there is no solid right or wrong approaches as long as it works.

[thedevyashsaini]

thanks for the clarification, that helps

For the first point - I'll go ahead assuming that hospital nodes may be behind NAT / firewalls... since this is almost foundational to the architecture (nodes being able to communicate outward but not necessarily accept inbound connections), we'll plan to solve that in this first iteration itself through an agent establishing an outbound channel to the controller. It doesn't add too much complexity and keeps the rest of the system simpler to reason about.

we're comfortable working on this within the 350-hour scope as well - for context, we implemented the distributed benchmarking architecture for the BRC in about ~5 weeks, so handling this level of distributed orchestration should be manageable within the timeline.

The part I wanted to clarify more is the "decentralized execution" comment.

From my understanding, a truly decentralized execution model would look something like this:

• the resource allocation is handled by whatever entity wants to request the execution of a workflow.
• there is a central channel.
• each machine, whether it is an edge node (hospital one) or a cloud one, has a daemon running.
• this daemon periodically advertises its capabilities, current usage, etc.
• the one requesting the workflow execution assigns the piece to one of the nodes.
• whether it is completed, or the node gets disconnected, or any other point of failure occurs, the one requesting the execution has to take care of it.
• whether it wants to reschedule or do something else is handled by the requester.

Which essentially removes a central authority responsible for scheduling or monitoring execution [ very close to bitcoin mining architectures ]. But that kind of architecture introduces quite a few edge cases around failure handling, job tracking, duplicate execution, etc.

the other interpretation I had was that by "decentralized" you might mean multiple administrative domains, for example separate clusters or environments (hospital infra, cloud infra, etc.) that we don't fully control.

Even in that case though, if we follow the agent-based architecture we discussed earlier, nodes would still register with a central WATER controller, advertise their capabilities, and scheduling decisions would still happen through WATER.. So there is still effectively a central scheduling authority, even if execution happens across different environments.

So I wanted to clarify what exactly you meant by decentralized execution in this context - whether you meant removing the central scheduling authority entirely, or simply allowing execution environments that are not administratively centralized.

just want to make sure I'm interpreting that constraint correctly before we polish the architecture further.

[pradeeban]

"we're comfortable working on this within the 350-hour scope as well - for context, we implemented the distributed benchmarking architecture for the BRC in about ~5 weeks, so handling this level of distributed orchestration should be manageable within the timeline."

I do not know who the "we" here is. Google Summer of Code (GSoC) is an individual work. If you are representing a team or a company, rather than an individual, it will disqualify your proposal. You must be participating under your own capacity, rather than as part of a team of consultants or a company that built an architecture in 5 weeks. GSoC is more about mentoring and learning rather than delivering the project on time by a team.

"what exactly you meant by decentralized execution in this context - whether you meant removing the central scheduling authority entirely, or simply allowing execution environments that are not administratively centralized."

More of the latter. Consider frameworks such as P2P networks. Perhaps consortium blockchains? Not necessarily entirely removing the centralized scheduling entity. But allowing such decentralized capability.

Rather than simply limiting yourself to a cluster, consider it a cluster farm. Communication east/west bound, protected data sharing across the inter-domain clusters. I am not saying Kubernetes is not the right solution. Just elaborating it would take more than simply having a Kubernetes cluster.

When you have a proposal draft, please feel free to email it to all the listed mentors and me for additional comments or clarifications.

[thedevyashsaini]

I do not know who the "we" here is. Google Summer of Code (GSoC) is an individual work. If you are representing a team or a company, rather than an individual, it will disqualify your proposal. You must be participating under your own capacity, rather than as part of a team of consultants or a company that built an architecture in 5 weeks. GSoC is more about mentoring and learning rather than delivering the project on time by a team.

the "we" there was just out of habit when describing past work. I was referring to the system I built for the BRC benchmarking pipeline and didn't mean to imply a team, company, or consultancy involvement for GSoC. the proposal and the work for WATER would of course be done individually under the GSoC program

When you have a proposal draft, please feel free to email it to all the listed mentors and me for additional comments or clarifications.

I've also sent a proposal draft to all the listed mentors for the WATER project and would really appreciate any feedback or suggestions you might have on it.

[pradeeban]

the "we" there was just out of habit when describing past work. I was referring to the system I built for the BRC benchmarking pipeline and didn't mean to imply a team, company, or consultancy involvement for GSoC. the proposal and the work for WATER would of course be done individually under the GSoC program

Thanks for the clarification.

We will get back to you if we have additional questions or comments. Make sure to upload the proposal to the GSoC Portal when the application period opens. You can keep updating the proposal.