| copyright |
|
||
|---|---|---|---|
| lastupdated | 2017-08-15 |
{:new_window: target="_blank"} {:shortdesc: .shortdesc} {:screen: .screen} {:pre: .pre} {:table: .aria-labeledby="caption"} {:codeblock: .codeblock} {:tip: .tip} {:download: .download}
{: #cs_security}
You can use built-in security features for risk analysis and security protection. These features help you to protect your cluster infrastructure and network communication, isolate your compute resources, and ensure security compliance across your infrastructure components and container deployments. {: shortdesc}
 Built-in cluster security settings in {{site.data.keyword.containershort_notm}} |
|
|---|---|
| Kubernetes master | The Kubernetes master in each cluster is managed by IBM and is highly available. It includes {{site.data.keyword.containershort_notm}} security settings that ensure security compliance and secure communication to and from the worker nodes. Updates are performed by IBM as required. The dedicated Kubernetes master centrally controls and monitors all Kubernetes resources in the cluster. Based on the deployment requirements and capacity in the cluster, the Kubernetes master automatically schedules your containerized apps to deploy across available worker nodes. For more information, see [Kubernetes master security](#cs_security_master). |
| Worker node | Containers are deployed on worker nodes that are dedicated to a cluster and that ensure compute, network, and storage isolation for IBM customers. {{site.data.keyword.containershort_notm}} provides built-in security features to keep your worker nodes secure on the private and public network and to ensure worker node security compliance. For more information, see [Worker node security](#cs_security_worker). |
| Images | As the cluster admin, you can set up your own secure Docker image repository in {{site.data.keyword.registryshort_notm}} where you can store and share Docker images between your cluster users. To ensure safe container deployments, every image in your private registry is scanned by Vulnerability Advisor. Vulnerability Advisor is a component of {{site.data.keyword.registryshort_notm}} that scans for potential vulnerabilities, makes security recommendations, and provides instructions to resolve vulnerabilities. For more information, see [Image security in {{site.data.keyword.containershort_notm}}](#cs_security_deployment). |
{: #cs_security_master}
Review the built-in Kubernetes master security features to protect the Kubernetes master and to secure the cluster network communication. {: shortdesc}
- Fully managed and dedicated Kubernetes master
- Every Kubernetes cluster in {{site.data.keyword.containershort_notm}} is controlled by a dedicated Kubernetes master that is managed by IBM in an IBM-owned {{site.data.keyword.BluSoftlayer_full}} account. The Kubernetes master is set up with the following dedicated components that are not shared with other IBM customers.
- etcd data store: Stores all Kubernetes resources of a cluster, such as Services, Deployments, and Pods. Kubernetes ConfigMaps and Secrets are app data that are stored as key value pairs so that they can be used by an app that runs in a pod. Data in etcd is stored on an encrypted disk that is managed by IBM and is encrypted via TLS when sent to a pod to assure data protection and integrity.
- kube-apiserver: Serves as the main entry point for all requests from the worker node to the Kubernetes master. The kube-apiserver validates and processes requests and can read from and write to the etcd data store.
- kube-controller-manager: Responsible for monitoring replica sets, and creating corresponding pods to achieve the desired state.
- OpenVPN: {{site.data.keyword.containershort_notm}} specific component to provide secured network connectivity for all Kubernetes master to worker node communication.
- TLS secured network connectivity for all worker node to Kubernetes master communication
- To secure the network communication to the Kubernetes master, {{site.data.keyword.containershort_notm}} generates TLS certificates that encrypts the communication to and from the kube-apiserver and etcd data store components for every cluster. These certificates are never shared across clusters or across Kubernetes master components.
- OpenVPN secured network connectivity for all Kubernetes master to worker node communication
- Although Kubernetes secures the communication between the Kubernetes master and worker nodes by using the `https` protocol, no authentication is provided on the worker node by default. To secure this communication, {{site.data.keyword.containershort_notm}} automatically sets up an OpenVPN connection between the Kubernetes master and the worker node when the cluster is created.
- Continuous Kubernetes master network monitoring
- Every Kubernetes master is continuously monitored by IBM to control and remediate process level Denial-Of-Service (DOS) attacks.
- Kubernetes master node security compliance
- {{site.data.keyword.containershort_notm}} automatically scans every node where the Kubernetes master is deployed for vulnerabilities found in Kubernetes and OS-sepcific security fixes that need to be applied to assure master node protection. If vulnerabilities are found, {{site.data.keyword.containershort_notm}} automatically applies fixes and resolves vulnerabilities on behalf of the user.
{: #cs_security_worker}
Review the built-in worker node security features to protect the worker node environment and to assure resource, network and storage isolation. {: shortdesc}
- Compute, network and storage infrastructure isolation
- When you create a cluster, virtual machines are provisioned as worker nodes in the customer {{site.data.keyword.BluSoftlayer_notm}} account or in the dedicated {{site.data.keyword.BluSoftlayer_notm}} account by IBM. Worker nodes are dedicated to a cluster and do not host workloads of other clusters.
Every {{site.data.keyword.Bluemix_notm}} account is set up with {{site.data.keyword.BluSoftlayer_notm}} VLANs to assure quality network performance and isolation on the worker nodes.
To persist data in your cluster, you can provision dedicated NFS based file storage from {{site.data.keyword.BluSoftlayer_notm}} and leverage the built-in data security features of that platform. - Secured worker node set up
- Every worker node is set up with an Ubuntu operating system that cannot be changed by the user. To protect the operating system of the worker nodes from potential attacks, every worker node is configured with expert firewall settings that are enforced by Linux iptable rules.
All containers that run on Kubernetes are protected by predefined Calico network policy settings that are configured on every worker node during cluster creation. This set up assures secure network communication between worker nodes and pods. To further restrict the actions that a container can perform on the worker node, users can choose to configure [AppArmor policies ](https://kubernetes.io/docs/tutorials/clusters/apparmor/) on the worker nodes.
By default, SSH access for the root user is disabled on the worker node. If you want to install additional features on your worker node, you can use [Kubernetes daemon sets ](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset) for everything that you want to run on every worker node, or [Kubernetes jobs ](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) for any one-time action you must execute. - Kubernetes worker node security compliance
- IBM works with security advisory teams internally and externally to identify potential security compliance issues for worker nodes and continuously releases compliance updates and security patches to address any vulnerabilities that are found. Updates and security patches are automatically deployed to the operating system of the worker node by IBM. To do that, IBM maintains SSH access to the worker nodes.
Note: Some updates require a worker node reboot. However, IBM does not reboot worker nodes during the installation of updates or security patches. Users are advised to reboot worker nodes on a regular basis to ensure that the installation of updates and security patches can finish. - Support for SoftLayer network firewalls
- {{site.data.keyword.containershort_notm}} is compatible with all [{{site.data.keyword.BluSoftlayer_notm}} firewall offerings ](https://www.ibm.com/cloud-computing/bluemix/network-security). On {{site.data.keyword.Bluemix_notm}} Public, you can set up a firewall with custom network policies to provide dedicated network security for your cluster and to detect and remediate network intrusion. For example, you might choose to set up a [Vyatta ](https://knowledgelayer.softlayer.com/topic/vyatta-1) to act as your firewall and block unwanted traffic. When you set up a firewall, [you must also open up the required ports and IP addresses](#opening_ports) for each region so that the master and the worker nodes can communicate. On {{site.data.keyword.Bluemix_notm}} Dedicated, firewalls, DataPower, Fortigate, and DNS are already configured as part of the standard dedicated environment deployment.
- Keep services private or selectively expose services and apps to the public internet
- You can choose to keep your services and apps private and leverage the built-in security features described in this topic to assure secured communication between worker nodes and pods. To expose services and apps to the public internet, you can leverage the Ingress and load balancer support to securely make your services publicly available.
- Securely connect your worker nodes and apps to an on-premise data center
- You can set up a Vyatta Gateway Appliance or Fortigate Appliance to configure an IPSec VPN endpoint that connects your Kubernetes cluster with an on-premise data center. Over an encrypted tunnel, all services that run in your Kubernetes cluster can communicate securely with on-premise apps, such as user directories, databases, or mainframes. For more information, see [Connecting a cluster to an on-premise data center ](https://www.ibm.com/blogs/bluemix/2017/07/kubernetes-and-bluemix-container-based-workloads-part4/).
- Continuous monitoring and logging of cluster activity
- For standard clusters, all cluster-related events, such as adding a worker node, rolling update progress, or capacity usage information are logged and monitored by {{site.data.keyword.containershort_notm}} and sent to the IBM Monitoring and Logging Service.
{: #opening_ports}
When you set up a firewall for your worker nodes or customize the firewall settings in your {{site.data.keyword.BluSoftlayer_notm}} account, you must open certain ports and IP addresses so that the worker node and the Kubernetes master can communicate.
-
Note the public IP address for all your worker nodes in the cluster.
bx cs workers <cluster_name_or_id>{: pre}
-
In your firewall, allow the following connections to and from your worker nodes.
TCP port 443 FROM <each_worker_node_publicIP> TO registry.<region>.bluemix.net, apt.dockerproject.org
{: codeblock}
<ul><li>For OUTBOUND connectivity from your worker nodes, allow outgoing network traffic from the source worker node to the destination TCP/UDP port range 20000-32767 for `<each_worker_node_publicIP>`, and the following IP addresses and network groups:</br>
| Outbound IP addresses |
|
|---|---|
| ams03 | 169.50.169.110 |
| dal10 | 169.46.7.238 |
| dal12 | 169.47.70.10 |
| fra02 | 169.50.56.174 |
| lon02 | 159.122.242.78 |
| lon04 | 158.175.65.170 |
| syd01 | 168.1.8.195 |
| syd04 | 130.198.64.19 |
{: #cs_security_network_policies}
Every Kubernetes cluster is set up with a network plug-in that is called Calico. Default network policies are set up to secure the public network interface of every worker node. You can use Calico and native Kubernetes capabilities to configure more network policies for a cluster when you have unique security requirements. These network policies specify the network traffic that you want to allow or block to and from a pod in a cluster. {: shortdesc}
You can choose between Calico and native Kubernetes capabilities to create network policies for your cluster. You might use Kubernetes network policies to get started, but for more robust capabilities, use the Calico network policies.
- [Kubernetes network policies ](https://kubernetes.io/docs/concepts/services-networking/network-policies/): Some basic options are provided, such as specifying which pods can communicate with each other. Incoming network traffic for pods can be allowed or blocked for a protocol and port based on the labels and Kubernetes namespaces of the pod that is trying to connect to them.
These policies can be applied by using `kubectl` commands or the Kubernetes APIs. When these policies are applied, they are converted into Calico network policies and Calico enforces these policies. - [Calico network policies ](http://docs.projectcalico.org/v2.0/getting-started/kubernetes/tutorials/advanced-policy): These policies are a superset of the Kubernetes network policies and enhance the native Kubernetes capabilities with the following features.
- Allow or block network traffic on specific network interfaces, not only Kubernetes pod traffic.
- Allow or block incoming (ingress) and outgoing (egress) network traffic.
- Allow or block traffic that is based on a source or destination IP address or CIDR.
These policies are applied by using `calicoctl` commands. Calico enforces these policies, including any Kubernetes network policies that are converted to Calico policies, by setting up Linux iptables rules on the Kubernetes worker nodes. Iptables rules serve as a firewall for the worker node to define the characteristics that the network traffic must meet to be forwarded to the targeted resource.
{: #concept_nq1_2rn_4z}
When a cluster is created, default network policies are automatically set up for the public network interface of each worker node to limit incoming traffic for a worker node from the public internet. These policies do not affect pod to pod traffic and are set up to allow access to the Kubernetes nodeport, load balancer, and Ingress services.
Default policies are not applied to pods directly; they are applied to the public network interface of a worker node by using a Calico host endpoint 
. When a host endpoint is created in Calico, all traffic to and from that worker node's network interface is blocked, unless that traffic is allowed by a policy.
Note that a policy to allow SSH does not exist, so SSH access by way of the public network interface is blocked, as are all other ports that do not have a policy to open them. SSH access, and other access, is available on the private network interface of each worker node.
Important: Do not remove policies that are applied to a host endpoint unless you fully understand the policy and know that you do not need the traffic that is being allowed by the policy.
| Default policies for each cluster |
|
|---|---|
allow-all-outbound |
Allows all outbound traffic. |
allow-icmp |
Allows incoming icmp packets (pings). |
allow-kubelet-port |
Allows all incoming traffic to port 10250, which is the port that is used by the kubelet. This policy allows `kubectl logs` and `kubectl exec` to work properly in the Kubernetes cluster. |
allow-node-port-dnat |
Allows incoming nodeport, load balancer, and ingress service traffic to the pods that those services are exposing. Note the port that those services expose on the public interface does not have to be specified because Kubernetes uses destination network address translation (DNAT) to forward those service requests to the correct pods. That forwarding takes place before the host endpoint policies are applied in iptables. |
allow-sys-mgmt |
Allows incoming connections for specific {{site.data.keyword.BluSoftlayer_notm}} systems that are used to manage the worker nodes. |
allow-vrrp |
Allow vrrp packets, which are used to monitor and move virtual IP addresses between worker nodes. |
{: #adding_network_policies}
In most cases, the default policies do not need to be changed. Only advanced scenarios might require changes. If you find that you must make changes, install the Calico CLI and create your own network policies
Before you begin, complete the following steps.
- Install the {{site.data.keyword.containershort_notm}} and Kubernetes CLIs.
- Create a lite or standard cluster.
- Target the Kubernetes CLI to the cluster. Include the
--adminoption with thebx cs cluster-configcommand, which is used to download the certificates and permission files. This download also includes the keys for the Administrator rbac role, which you need to run Calico commands.
bx cs cluster-config <cluster_name>
{: pre}
To add network policies:
-
Install the Calico CLI.
-
Tip: If you are using Windows, install the Calico CLI in the same directory as the {{site.data.keyword.Bluemix_notm}} CLI. This setup saves you some filepath changes when you run commands later.
-
For OSX and Linux users, complete the following steps.
-
Move the executable file to the /usr/local/bin directory.
- Linux:
mv /<path_to_file>/calicoctl /usr/local/bin/calicoctl{: pre}
- OS X:
mv /<path_to_file>/calico-darwin-amd64 /usr/local/bin/calicoctl{: pre}
-
Convert the binary file to an executable.
chmod +x /usr/local/bin/calicoctl{: pre}
-
-
Verify that the
calicocommands ran properly by checking the Calico CLI client version.calicoctl version{: pre}
-
-
Configure the Calico CLI.
- For Linux and OS X, create the '/etc/calico' directory. For Windows, any directory can be used.
mkdir -p /etc/calico/{: pre}
-
Create a 'calicoctl.cfg' file.
- Linux and OS X:
sudo vi /etc/calico/calicoctl.cfg{: pre}
- Windows: Create the file with a text editor.
-
Enter the following information in the
calicoctl.cfgfile.
apiVersion: v1 kind: calicoApiConfig metadata: spec: etcdEndpoints: <ETCD_URL> etcdKeyFile: <CERTS_DIR>/admin-key.pem etcdCertFile: <CERTS_DIR>/admin.pem etcdCACertFile: <CERTS_DIR>/<ca-*pem_file>{: pre}
1. Retrieve the `<ETCD_URL>`. - Linux and OS X: ``` kubectl describe pod -n kube-system `kubectl get pod -n kube-system | grep calico-policy-controller | awk '{print $1}'` | grep ETCD_ENDPOINTS | awk '{print $2}' ``` {: pre} - Output example: ``` https://169.1.1.1:30001 ``` {: screen} - Windows: <ol> <li>Get a list of the pods in the kube-system namespace and locate the Calico controller pod. </br><pre class="codeblock"><code>kubectl get pod -n kube-system</code></pre></br>Example:</br><pre class="screen"><code>calico-policy-controller-1674857634-k2ckm</code></pre> <li>View the details of the Calico controller pod.</br> <pre class="codeblock"><code>kubectl describe pod -n kube-system calico-policy-controller-<ID></code></pre> <li>Locate the ETCD endpoints value. Example: <code>https://169.1.1.1:30001</code> </ol> 2. Retrieve the `<CERTS_DIR>`, the directory that the Kubernetes certificates are downloaded in. - Linux and OS X: ``` dirname $KUBECONFIG ``` {: pre} Output example: ``` /home/sysadmin/.bluemix/plugins/container-service/clusters/<cluster_name>-admin/ ``` {: screen} - Windows: ``` ECHO %KUBECONFIG% ``` {: pre} Output example: ``` C:/Users/<user>/.bluemix/plugins/container-service/<cluster_name>-admin/kube-config-prod-<location>-<cluster_name>.yml ``` {: screen} **Note**: To get the directory path, remove the file name `kube-config-prod-<location>-<cluster_name>.yml` from the end of the output. 3. Retrieve the <code>ca-*pem_file<code>. - Linux and OS X: ``` ls `dirname $KUBECONFIG` | grep ca-.*pem ``` {: pre} - Windows: <ol><li>Open the directory you retrieved in the last step.</br><pre class="codeblock"><code>C:\Users\<user>\.bluemix\plugins\container-service\<cluster_name>-admin\</code></pre> <li> Locate the <code>ca-*pem_file</code> file.</ol> 4. Verify that the Calico configuration is working correctly. - Linux and OS X: ``` calicoctl get nodes ``` {: pre} - Windows: ``` calicoctl get nodes --config=<path_to_>/calicoctl.cfg ``` {: pre} Output: ``` NAME kube-dal10-crc21191ee3997497ca90c8173bbdaf560-w1.cloud.ibm kube-dal10-crc21191ee3997497ca90c8173bbdaf560-w2.cloud.ibm kube-dal10-crc21191ee3997497ca90c8173bbdaf560-w3.cloud.ibm ``` {: screen} -
Examine the existing network policies.
- View the Calico host endpoint.
calicoctl get hostendpoint -o yaml{: pre}
- View all of the Calico and Kubernetes network policies that were created for the cluster. This list includes policies that might not be applied to any pods or hosts yet. For a network policy to be enforced, it must find a Kubernetes resource that matches the selector that was defined in the Calico network policy.
calicoctl get policy -o wide{: pre}
- View details for a network policy.
calicoctl get policy -o yaml <policy_name>{: pre}
- View the details of all network policies for the cluster.
calicoctl get policy -o yaml{: pre}
-
Create the Calico network policies to allow or block traffic.
-
Define your Calico network policy
by creating a configuration script (.yaml). These configuration files include the selectors that describe what pods, namespaces, or hosts that these policies apply to. Refer to these sample Calico policies to help you create your own. -
Apply the policies to the cluster.
- Linux and OS X:
calicoctl apply -f <policy_file_name.yaml>{: pre}
- Windows:
calicoctl apply -f <path_to_>/<policy_file_name.yaml> --config=<path_to_>/calicoctl.cfg{: pre}
-
{: #cs_security_deployment}
Manage the security and integrity of your images with built-in security features. {: shortdesc}
You can set up your own Docker image repository in a multi-tenant, highly available, and scalable private image registry that is hosted and managed by IBM to build, securely store, and share Docker images across cluster users.
When you use {{site.data.keyword.registryshort_notm}}, you can leverage the built-in security scanning that is provided by Vulnerability Advisor. Every image that is pushed to your namespace is automatically scanned for vulnerabilities against a database of known CentOS, Debian, Red Hat, and Ubuntu issues. If vulnerabilities are found, Vulnerability Advisor provides instructions for how to resolve them to assure image integrity and security.
To view the vulnerability assessment for your image:
- From the catalog, in the Containers section, select Container Registry.
- On the Private Repositories page, in the Repositories table, identify the image.
- In the Security Report column, click the status of the image to retrieve its vulnerability assessment.