This document will walk through how to setup a nydus image service to work with containerd. It assumes that you already have containerd installed. If not, please refer to containerd documents on how to install and set it up.
- Get
nydus-image,nydusd,nydusify,nydusctlandnydus-overlayfsbinaries from release page.
sudo install -D -m 755 nydusd nydus-image nydusify nydusctl nydus-overlayfs /usr/bin- Get
containerd-nydus-grpc(nydus snapshotter) binary from nydus-snapshotter release page.
sudo install -D -m 755 containerd-nydus-grpc /usr/binNydus provides a containerd remote snapshotter containerd-nydus-grpc (nydus snapshotter) to prepare container rootfs with nydus formatted images.
- Prepare a
nydusdconfiguration to/etc/nydus/nydusd-config.fusedev.json:
$ sudo tee /etc/nydus/nydusd-config.fusedev.json > /dev/null << EOF
{
"device": {
"backend": {
"type": "registry",
"config": {
"scheme": "",
"skip_verify": true,
"timeout": 5,
"connect_timeout": 5,
"retry_limit": 4,
"auth": "YOUR_LOGIN_AUTH="
}
},
"cache": {
"type": "blobcache",
"config": {
"work_dir": "cache"
}
}
},
"mode": "direct",
"digest_validate": false,
"iostats_files": false,
"enable_xattr": true,
"fs_prefetch": {
"enable": true,
"threads_count": 4
}
}
EOFPlease refer to the nydusd doc to learn more options.
- The
device.backend.config.schemeis the URL scheme for the registry. Leave it empty for automatic detection, or specifyhttpsorhttpdepending on your registry server configuration. - The
device.backend.config.authis the base64 encodedusername:passwordrequired by nydusd to lazily pull image data from an authenticated registry. The nydus snapshotter will automatically read the authentication information from the$HOME/.docker/config.jsonconfiguration file. If you are using a registry that requires authentication, you should replaceYOUR_LOGIN_AUTH=with your own login information. - The
device.backend.config.skip_verifyallows you to skip the insecure https certificate checks for the registry, only set it totruewhen necessary. Note that enabling this option is a security risk for the connection to registry, so you should only use this when you are sure it is safe. - The
fs_prefetch.enableoption enables nydusd to prefetch image data in background, which can make container startup faster when it needs to read a large amount of image data. Set this tofalseif you don't need this functionality when it brings disk and network pressure.
- [Optional] Cleanup snapshotter environment:
Make sure the default nydus snapshotter root directory is clear.
sudo rm -rf /var/lib/containerd-nydus
- Start
containerd-nydus-grpc(nydus snapshotter): Optionally, a TOML based nydus-snapshotter configuration file can be provided by appending--config <CONFIG>when starting nydus-snapshotter if you want fine-grained control items. An example configuration file can be found here
sudo /usr/bin/containerd-nydus-grpc \
--nydusd-config /etc/nydus/nydusd-config.fusedev.json \
--log-to-stdoutNydus depends on two features of Containerd:
- Support remote snapshotter plugin
- Support passing annotations to remote snapshotter
To enable them, add below configuration items to your containerd configuration file (default path is /etc/containerd/config.toml):
[proxy_plugins]
[proxy_plugins.nydus]
type = "snapshot"
address = "/run/containerd-nydus/containerd-nydus-grpc.sock"When working with Kubernetes CRI, please change the default snapshotter to nydus and enable snapshot annotations like below:
For version 1 containerd config format:
[plugins.cri]
[plugins.cri.containerd]
snapshotter = "nydus"
disable_snapshot_annotations = falseFor version 2 containerd config format:
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "nydus"
disable_snapshot_annotations = falseThen restart containerd, e.g.:
sudo systemctl restart containerdTo make it easier to convert and run nydus images next, we can run a local registry service with docker:
sudo docker run -d --restart=always -p 5000:5000 registryNydus image can be created by converting from an existing OCI or docker v2 image stored in container registry or directly built from Dockerfile(with Buildkit)
Note: For private registry repo, please make sure you are authorized to pull and push the target registry. The basic method is to use docker pull and docker push to verify your access to the source or target registry.
sudo nydusify convert --source ubuntu --target localhost:5000/ubuntu-nydusFor more details about how to build nydus image, please refer to Nydusify conversion tool, Acceld conversion service or Nerdctl.
Nydus snapshotter has been supported by nerdctl(requires >= v0.22), we can lazily start container with it.
$ sudo nerdctl --snapshotter nydus run --rm -it localhost:5000/ubuntu-nydus:latest bashFor example, use the following nydus-sandbox.yaml and nydus-container.yaml
The nydus-sandbox.yaml looks like below:
metadata:
attempt: 1
name: nydus-sandbox
namespace: default
log_directory: /tmp
linux:
security_context:
namespace_options:
network: 2
annotations:
"io.containerd.osfeature": "nydus.remoteimage.v1"The nydus-container.yaml looks like below:
metadata:
name: nydus-container
image:
image: localhost:5000/ubuntu-nydus:latest
command:
- /bin/sleep
args:
- 600
log_path: container.1.logTo create a pod with the just converted nydus image:
$ sudo crictl pull localhost:5000/ubuntu-nydus:latest
$ pod=`sudo crictl runp nydus-sandbox.yaml`
$ container=`sudo crictl create $pod nydus-container.yaml nydus-sandbox.yaml`
$ sudo crictl start $container
$ sudo crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT POD ID
f4a6c6dc47e34 localhost:5000/ubuntu-nydus:latest 9 seconds ago Running nydus-container 0 21b91779d551eNydus is deeply integrated with Dragonfly P2P system, which can greatly reduce the network latency and the single point of network pressure for registry server, testing in the production environment shows that using Dragonfly can reduce network latency by more than 80%, to understand the performance test data and how to configure Nydus to use Dragonfly, please refer to the doc.