Skip to content

Rootless bridge: preserve source IPs via pesto/pasta#28478

Merged
Luap99 merged 3 commits into
containers:mainfrom
Honny1:pesto-support
May 21, 2026
Merged

Rootless bridge: preserve source IPs via pesto/pasta#28478
Luap99 merged 3 commits into
containers:mainfrom
Honny1:pesto-support

Conversation

@Honny1
Copy link
Copy Markdown
Member

@Honny1 Honny1 commented Apr 9, 2026

Require passta version:

This requires the pesto binary, available since passt-0^20260507.g1afd4ed.

Local Passt Setup + testing

# Build pasta with pesto support
git clone git://passt.top/passt 
cd passt
git checkout ec96f0124282 
b4 shazam https://archives.passt.top/passt-dev/20260506092241.1607480-1-sbrivio@redhat.com/
make 
sudo make install

cd ../podman
make binaries
make localintegration FOCUS_FILE=run_networking_test.go

TODO:


Add rootless_port_forwarder="pasta" option in containers.conf that switches rootless bridge port forwarding from rootlessport to pasta, preserving source IPs. The default remains rootlessport.

Problem

When running rootless containers on a bridge network (podman run -p 8080:80 --network mynet), the old rootlessport userspace TCP/UDP proxy destroyed source IP information. Every connection appeared to come from 127.0.0.1 inside the container, regardless of the actual client IP.

Solution

Pesto is a client tool for passta that dynamically updates port forwarding rules via a UNIX domain socket. Instead of proxying traffic in userspace (which loses source IPs), pesto configures pasta to forward at the kernel level using splice (localhost) or TAP (external traffic), preserving the original source IP.

How it works

Host -> pasta (splice/TAP) -> rootless netns -> netavark bridge DNAT -> container
  1. A shared pasta instance runs in the rootless network namespace with a control socket (-c pasta.sock)
  2. On container start: netavark sets up the bridge + nftables DNAT rules, then pesto --add registers ports in pasta's forwarding table
  3. On container stop: pesto --delete removes the container's ports, then netavark tears down bridge/DNAT
  4. On network connect/disconnect/reload: port forwarding state is updated accordingly

Key implementation details:

  • HostIP stripping in the netavark wrapper (convertNetOpts): when rootless_port_forwarder="pasta" is active, HostIP is stripped from port mappings before passing to netavark. Pesto handles host-side address binding; netavark's DNAT rules inside the rootless netns must not restrict on destination address since pasta's splice delivers traffic with a different destination than the user-specified HostIP
  • Dual-stack binding: when no explicit HostIP is set, pesto binds both 0.0.0.0 and [::] so IPv6 networks work out of the box
  • Config toggle: rootless_port_forwarder in [network] section of containers.conf selects between rootlessport (default) and pasta

Current limitations

  • Same port on different HostIPs (e.g. -p 127.0.0.1:8080:80 and -p 127.0.0.2:8080:80 on separate containers) does not work in pasta mode because HostIP stripping causes conflicting DNAT rules. This requires destination mapping support in pesto.

Fixes: https://redhat.atlassian.net/browse/RUN-2214
Fixes: #8193
Fixes: https://redhat.atlassian.net/browse/RUN-3587

Depends on: passt >= passt-0^20260507.g1afd4ed
Depends on: containers/container-libs#755
Depends on: #27828
Depends on: #28451

Checklist

Ensure you have completed the following checklist for your pull request to be reviewed:

  • Certify you wrote the patch or otherwise have the right to pass it on as an open-source patch by signing all
    commits. (git commit -s). (If needed, use git commit -s --amend). The author email must match
    the sign-off email address. See CONTRIBUTING.md
    for more information.
  • Referenced issues using Fixes: #00000 in commit message (if applicable)
  • Tests have been added/updated (or no tests are needed)
  • Documentation has been updated (or no documentation changes are needed)
  • All commits pass make validatepr (format/lint checks)
  • Release note entered in the section below (or None if no user-facing changes)

Does this PR introduce a user-facing change?

Add experimental rootless_port_forwarder="pasta" option in containers.conf. When enabled, rootless bridge network port forwarding uses pasta's kernel-level forwarding (via pesto) instead of rootlessport's userspace proxy, preserving the original client source IP inside containers. Requires a passt build with pesto support. The default remains rootlessport.

@Honny1 Honny1 changed the title Rootless bridge: preserve source IPs via pesto/pasta [WIP] Rootless bridge: preserve source IPs via pesto/pasta Apr 9, 2026
@packit-as-a-service
Copy link
Copy Markdown

[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore.

1 similar comment
@packit-as-a-service
Copy link
Copy Markdown

[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore.

@Honny1 Honny1 added rootless pasta pasta(1) bugs or features 6.0 Breaking changes for Podman 6.0 labels Apr 10, 2026
@github-actions github-actions Bot added the kind/api-change Change to remote API; merits scrutiny label Apr 10, 2026
@Honny1 Honny1 force-pushed the pesto-support branch 2 times, most recently from 6fc22c5 to 662445d Compare April 14, 2026 12:26
@Honny1 Honny1 added bloat_approved Approve a PR in which binary file size grows by over 50k labels Apr 14, 2026
@Honny1 Honny1 force-pushed the pesto-support branch 2 times, most recently from 292a7b8 to 9fc6f49 Compare April 23, 2026 16:33
@packit-as-a-service
Copy link
Copy Markdown

tmt tests failed for commit 9fc6f49. @lsm5, @psss, @thrix please check.

@Honny1 Honny1 force-pushed the pesto-support branch 2 times, most recently from 68bbedb to f511d44 Compare April 24, 2026 07:35
@packit-as-a-service
Copy link
Copy Markdown

[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore.

@packit-as-a-service
Copy link
Copy Markdown

tmt tests failed for commit f511d44. @lsm5, @psss, @thrix please check.

@Honny1 Honny1 force-pushed the pesto-support branch 4 times, most recently from e2ae082 to 4a4fbf6 Compare May 12, 2026 22:14
@Honny1
Copy link
Copy Markdown
Member Author

Honny1 commented May 13, 2026

PTAL @Luap99 @containers/podman-maintainers

I am not sure about the packit failure.

cc @sbrivio-rh

@sbrivio-rh
Copy link
Copy Markdown
Collaborator

I am not sure about the packit failure.

To me the Packit failure (https://dashboard.packit.dev/jobs/testing-farm/1544417 / https://artifacts.dev.testing-farm.io/177feab7-7054-4ff9-9cf3-2f9f3f9ac985/) looks like a failure of a relevant test. From the second link:

not ok 619 [500] podman network reload
# (from function `bail-now' in file ./helpers.bash, line 230,
#  from function `die' in file ./helpers.bash, line 967,
#  from function `run_podman' in file ./helpers.bash, line 608,
#  from function `_test_network_reload' in file ./500-networking.bats, line 332,
#  in test file ./500-networking.bats, line 434)
#   `_test_network_reload pasta' failed

...or am I missing something obvious? Should I try to figure that out / help with debugging? In general, let me know if / how I can help here.

@Honny1
Copy link
Copy Markdown
Member Author

Honny1 commented May 14, 2026

I am not sure about the packit failure.

To me the Packit failure (https://dashboard.packit.dev/jobs/testing-farm/1544417 / https://artifacts.dev.testing-farm.io/177feab7-7054-4ff9-9cf3-2f9f3f9ac985/) looks like a failure of a relevant test. From the second link:

not ok 619 [500] podman network reload
# (from function `bail-now' in file ./helpers.bash, line 230,
#  from function `die' in file ./helpers.bash, line 967,
#  from function `run_podman' in file ./helpers.bash, line 608,
#  from function `_test_network_reload' in file ./500-networking.bats, line 332,
#  in test file ./500-networking.bats, line 434)
#   `_test_network_reload pasta' failed

...or am I missing something obvious? Should I try to figure that out / help with debugging? In general, let me know if / how I can help here.

It gets a Permission Denied error on the Unix socket for pesto. I am not sure if it is a Packit issue or if it is caused by an SELinux policy. I wasn't able to reproduce it locally on my system.

@sbrivio-rh
Copy link
Copy Markdown
Collaborator

It gets a Permission Denied error on the Unix socket for pesto.

Oops, so, yes, I missed something obvious. :)

I introduced a new SELinux policy for pesto which explicitly allows pesto_exec_t to access pasta_t files, but I didn't test that with Podman (just between pesto and stand-alone pasta, pun entirely intended). I'll debug this (perhaps not today though).

@sbrivio-rh
Copy link
Copy Markdown
Collaborator

I introduced a new SELinux policy for pesto which explicitly allows pesto_exec_t to access pasta_t files, but I didn't test that with Podman (just between pesto and stand-alone pasta, pun entirely intended). I'll debug this (perhaps not today though).

Sorry for the delay. I actually couldn't spot anything wrong in the new profiles, and I'm fairly sure this is a SELinux issue (maybe a relabeling missing as the packages are installed on the CI system), but I looked at all the possible logs and I couldn't find audit logs (/var/log/audit/audit.log) or SELinux warnings anywhere.

Would you have a way to check / extract / find them? I could just spot an out-of-memory message in the kernel log but I don't think it has anything to do with this issue.

@Luap99
Copy link
Copy Markdown
Member

Luap99 commented May 19, 2026

Sorry for the delay. I actually couldn't spot anything wrong in the new profiles, and I'm fairly sure this is a SELinux issue (maybe a relabeling missing as the packages are installed on the CI system), but I looked at all the possible logs and I couldn't find audit logs (/var/log/audit/audit.log) or SELinux warnings anywhere.

we will need to port containers/buildah#6850 here I think for the tmt jobs to capture these logs

@Honny1
Copy link
Copy Markdown
Member Author

Honny1 commented May 19, 2026

Sorry for the delay. I actually couldn't spot anything wrong in the new profiles, and I'm fairly sure this is a SELinux issue (maybe a relabeling missing as the packages are installed on the CI system), but I looked at all the possible logs and I couldn't find audit logs (/var/log/audit/audit.log) or SELinux warnings anywhere.

we will need to port containers/buildah#6850 here I think for the tmt jobs to capture these logs

I created PR for that: #28737

@Honny1
Copy link
Copy Markdown
Member Author

Honny1 commented May 19, 2026

@Luap99 @sbrivio-rh There seems to be an issue with SELinux. Based on these logs: https://artifacts.dev.testing-farm.io/2e1dcd26-d5e6-484d-854a-233494c648e2/work-local-rootlessa_2kq5lh/plans/system/local-rootless/data/logs/

type=AVC msg=audit(1779199621.208:2040): avc:  denied  { create } for  pid=217062 comm="pasta.avx2" name="pasta.sock" scontext=unconfined_u:unconfined_r:pasta_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:ifconfig_var_run_t:s0 tclass=sock_file permissive=0

cc @lsm5

@sbrivio-rh
Copy link
Copy Markdown
Collaborator

@Luap99 @sbrivio-rh There seems to be an issue with SELinux. Based on these logs: https://artifacts.dev.testing-farm.io/2e1dcd26-d5e6-484d-854a-233494c648e2/work-local-rootlessa_2kq5lh/plans/system/local-rootless/data/logs/

type=AVC msg=audit(1779199621.208:2040): avc:  denied  { create } for  pid=217062 comm="pasta.avx2" name="pasta.sock" scontext=unconfined_u:unconfined_r:pasta_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:ifconfig_var_run_t:s0 tclass=sock_file permissive=0

Thanks! Probably my stuff then, I'll keep you posted. Do you need a workaround meanwhile (something subtler than setenforce 0')?

@Honny1
Copy link
Copy Markdown
Member Author

Honny1 commented May 19, 2026

Thanks! Probably my stuff then, I'll keep you posted. Do you need a workaround meanwhile (something subtler than setenforce 0')?

Thanks, taking care of that. I am not sure if I need a workaround.

@sbrivio-rh
Copy link
Copy Markdown
Collaborator

@Luap99 @sbrivio-rh There seems to be an issue with SELinux. Based on these logs: https://artifacts.dev.testing-farm.io/2e1dcd26-d5e6-484d-854a-233494c648e2/work-local-rootlessa_2kq5lh/plans/system/local-rootless/data/logs/

type=AVC msg=audit(1779199621.208:2040): avc:  denied  { create } for  pid=217062 comm="pasta.avx2" name="pasta.sock" scontext=unconfined_u:unconfined_r:pasta_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:ifconfig_var_run_t:s0 tclass=sock_file permissive=0

Fix posted: https://archives.passt.top/passt-dev/20260519173634.3551297-1-sbrivio@redhat.com/

I plan to make a new release rather soon including that. There are already a couple of regressions we just fixed (not touching Podman) or we might be able to fix soon, so a new release in a couple of days makes sense anyway.

Comment thread go.mod Outdated
Comment on lines +192 to +196
replace go.podman.io/common => github.com/containers/container-libs/common v0.67.2-0.20260518190429-32704ef88648

replace go.podman.io/image/v5 => github.com/containers/container-libs/image/v5 v5.0.0-20260518190429-32704ef88648

replace go.podman.io/storage => github.com/containers/container-libs/storage v0.0.0-20260518190429-32704ef88648
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not use replace, just go get go.podman.io/storage@main && go get go.podman.io/image/v5@main && go get go.podman.io/common@main && make vendor

also need to cleanup the commit description of the vendor commit

Comment thread rpm/podman.spec Outdated
Comment on lines +114 to +115
# pesto binary (ships with passt) is needed for rootless_port_forwarder="pasta"
Recommends: passt >= 0^20260507.g1afd4ed
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honny1 added 3 commits May 20, 2026 14:23
Used command: go get go.podman.io/storage@main && go get go.podman.io/image/v5@main &&  go get go.podman.io/common@main && make vendor

Signed-off-by: Jan Rodák <hony.com@seznam.cz>
Add rootless_port_forwarder="pasta" option that uses pesto to update
pasta's forwarding table via UNIX socket, preserving source IPs that
rootlessport's userspace proxy masks.

HostIP is stripped from port mappings in the netavark wrapper when
pasta forwarding is active because pesto handles host-side binding
while pasta's splice changes the destination IP that netavark DNAT
expects. Pesto binds both 0.0.0.0 and [::] for dual-stack support.

Fixes: https://redhat.atlassian.net/browse/RUN-2214
Fixes: containers#8193
Fixes: https://redhat.atlassian.net/browse/RUN-3587

Signed-off-by: Jan Rodák <hony.com@seznam.cz>
Signed-off-by: Jan Rodák <hony.com@seznam.cz>
@packit-as-a-service
Copy link
Copy Markdown

[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore.

Copy link
Copy Markdown
Member

@Luap99 Luap99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TomSweeneyRedHat
Copy link
Copy Markdown
Member

LGTM
It would be nice to get another set of eyeballs on this.

@Luap99 Luap99 merged commit 2d09c79 into containers:main May 21, 2026
89 of 90 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.0 Breaking changes for Podman 6.0 bloat_approved Approve a PR in which binary file size grows by over 50k kind/api-change Change to remote API; merits scrutiny pasta pasta(1) bugs or features rootless

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Alternate port_handler that keeps the source ip for user-defined rootless networks

4 participants