Update procedure for setting up single node/multi-node cluster

docs/setup-guide/multi-node-dev-cluster.md

@@ -2,99 +2,147 @@
 
 It may be desired to setup a dev cluster having one or more worker nodes in order to play with comprehensive features of Arktos. The simple and easy-to-use arktos-up.sh feels short in this case; this doc describes the minimum effort to run a cluster having 1 master node and extra worker nodes joining after the master is up.
 
-This doc, as at the moment written, does not mandate which cni plugin to be used; it is up to the user. We have verified that Flannel works well with multi-tenancy Arktos cluster; the instructions laid out here are based on our experience with Flannel at GCP env.
+This doc, as at the moment written, does not mandate which cni plugin to be used; it is up to the user. We have verified that Flannel works well with multi-tenancy Arktos cluster; the instructions laid out here are based on our experience with Flannel at GCP env or AWS env with Ubuntu 18.04 x86 image.
 
-Assuming you have got the Arktos repo downloaded to your local disk and the current folder is at the root of the repo,
+Assuming you have got the Arktos repo downloaded to your local disk and the current folder (i.e ~/go/src/arktos/) is at the root of the repo, and 
 
-0. Make sure the following directories are empty. If not, clean them up.
+- Setup master node and workwer nodes based on [set up developer environment](setup-dev-env.md) to install needed packages (docker, make, gcc, jq and golang).
+- SSH/SCP should work from worker nodes to master node in order to copy the worker secret files from the master node (in AWS, use private key file of keypair to access to master/worker nodes).
+- Needed ports are opened between master node and worker nodes
+  * Allow access to kube-api port 6443 on master node from work nodes (in AWS, add this rule into inbound rules of security group for master node)
+  * Allow access to kubelet port 10251 and kube-proxy port 10255 on worker nodes from master node(in AWS, add this rule into inbound rules of security group for worker nodes)
+- On master node, the permisson of others for file /var/run/docker.sock should be readable and writable.
+  Here is output of running command 'ls -al'
+
+```bash
+srw-rw-rw- 1 root docker 0 Aug  3 22:15 /var/run/docker.sock
+```
+
+  Normally if the machine is rebooted, the permission of this file is changed to default permission below.
+
+```bash
+srw-rw---- 1 root docker 0 Aug  9 23:18 /var/run/docker.sock
+```
+
+  Please run the command to add the permission for 'others' using sudo
+```bash
+sudo chmod o+rw /var/run/docker.sock
+ls -al
+```
+
+
+
+0. Make sure the following directories are empty. If not, clean them up using sudo permisson
 ```
-/opt/cni/bin/
-/etc/cni/net.d/
+sudo rm -rf /opt/cni/bin/*
+sudo rm -rf /etc/cni/net.d/*
 ```
 
-1. bootstrap the cluster by starting the master node (no CNI plgin at first)
+1. bootstrap the cluster by starting the master node (no CNI plugin at first)
 ```bash
 export ARKTOS_NO_CNI_PREINSTALLED=y
+make clean
 ./hack/arktos-up.sh
 ```
 
-Note: arktos-up.sh should be stuck in "Waiting for node ready at api server" messages. Don't worry, the apiserver is already up at this point, just the master node status is not "Ready" as we have not installed the network plugin yet. 
+Note: arktos-up.sh should be stuck in "Waiting for node ready at api server" messages. Don't worry, the apiserver is already up at this point, just the master node status is not "Ready"  and pod/kube-dns in name space 'kube-system' is not in state of 'Running' as we have not installed the network plugin yet. 
 
-2. Open another terminal to the master node to install CNI plugin
+2. Open another terminal to the master node to install CNI plugin of flannel
 ```bash
 ./cluster/kubectl.sh apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
 ```
 
-After that, "arktos-up.sh" should get rid of "Waiting for node ready at api server" messages and be successfully started.
+After that, "arktos-up.sh" should get rid of "Waiting for node ready at api server" messages and be successfully started. You can check the status of all resources including pod/kube-dns and flannel network
 
-3. In the lab machine to be added as a worker node, ensure following worker secret files copied from the master node:
-If the worker node is a GCP instance  
 ```bash
-mkdir -p /tmp/arktos
-gcloud compute scp <master-node-instance>:/var/run/kubernetes/kubelet.kubeconfig /tmp/arktos/
-gcloud compute scp <master-node-instance>:/var/run/kubernetes/client-ca.crt /tmp/arktos/
+export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
+./cluster/kubectl.sh get all --all-namespaces
+ifconfig -a
+ip route
+sudo ls -alg /etc/cni/net.d/10-flannel.conflist
 ```
 
-If the worker node is an AWS EC2, you can download files /var/run/kubernetes/kubelet.kubeconfig and /var/run/kubernetes/client-ca.crt from the master node, and then upload to /tmp/arktos/ in the worker node.
+Note: Please reference the blog [Kubernetes: Flannel networking](https://blog.laputa.io/kubernetes-flannel-networking-6a1cb1f8ec7c) if you want to know how flannel network works in kubernetes.
 
-At the moment this doc is written, kube-proxy does not support multi-tenancy yet, and it won't be deployed in the cluster. After kube-proxy issue has been fixed (we will allocate resource to cope with it soon), probably we will also need to copy over kube-proxy related secret and artifacts.
+Note: CNI plugin of Calico is not supported because the resource 'EndpointSlices' is not supported by Arktos so far.
 
-As a temporary measure to accommodate Flannel pods to access api server through the kubernetes service IP (the notorious 10.0.0.1) at the absense of kube-proxy, we need to place a bit trick by running below command __at the worker node__, assuming the kupe-apiserver is listening at https://10.138.0.19:6443 (please substitute with the proper value of your own cluster):
+3. In the lab machine to be added as a worker node, ensure following worker secret files copied from the master node:
+If the worker node is a GCP instance  
 ```bash
-sudo iptables -t nat -A OUTPUT -p tcp -d 10.0.0.1 --dport 443 -j DNAT --to-destination 10.138.0.19:6443
+gcloud beta compute ssh --zone "us-west2-a" "<worker node machine name>"  --project "<gce project name>"
 ```
 
-NOTE: you need to re-run the above command once you restart the machine.
+If the worker node is an AWS EC2, you can download files /var/run/kubernetes/kubelet.kubeconfig and /var/run/kubernetes/client-ca.crt from the master node, and then upload to /tmp/arktos/ in the worker node.
 
-Please be advised that this is a temporary quirk only; after we have the full service support by proper kube-proxy or other means, we don't need it any more.
+```bash
+mkdir -p /tmp/arktos
+scp -i "<private key of keypair of master node>" ubuntu@<master-node-instance>:/var/run/kubernetes/kubelet.kubeconfig /tmp/arktos/kubelet.kubeconfig
+scp -i "<private key of keypair of master node>" ubuntu@<master-node-instance>:/var/run/kubernetes/client-ca.crt /tmp/arktos/client-ca.crt
+```
 
 4. start the worker node and register into cluster
 
-First Make sure the following directories in the worker node are empty. If not, clean them up.
+First make sure the following directories in the worker node are empty. If not, clean them up using sudopermission.
 ```
-/opt/cni/bin/
-/etc/cni/net.d/
+sudo rm -rf /opt/cni/bin/*
+sudo rm -rf /etc/cni/net.d/*
 ```
 
 Then at worker node, run following commands:
 ```bash
 export ARKTOS_NO_CNI_PREINSTALLED=y
+
+hostname -i  (to get <worker-ip>)
 export KUBELET_IP=<worker-ip>
+
+OR
+export KUBELET_IP=`hostname -i`
+
+echo $KUBELET_IP
+make clean
 ./hack/arktos-worker-up.sh
 ```
 
-After the script returns, go to master node terminal and run command "[arktos_repo]/cluster/kubectl.sh get nodes", you should see the work node is displayed and its status should be "Ready".
+After the script returns, go to master node terminal and run command "./cluster/kubectl.sh get nodes", you should see the work node is displayed and its status should be "Ready".
 
-5. label worker node as vm runtime capable (optional)
+But when you run command "./cluster/kubectl.sh get all --all-namespaces", you will see new pod of flannel 'pod/kube-flannel-ds-xxxxx' for worker node is not in Running state. If you check the log of this pod 'pod/kube-flannel-ds-xxxxx', you will see the following error.
 
-If you would like to allow this work node to run VM-based pods, please run below command at the master console:
 ```bash
-./cluster/kubectl.sh label node <worker-node-name> extraRuntime=virtlet
+I0803 22:50:37.646013       1 main.go:520] Determining IP address of default interface
+I0803 22:50:37.646394       1 main.go:533] Using interface with name eth0 and address 172.31.2.184
+I0803 22:50:37.646415       1 main.go:550] Defaulting external address to interface address (172.31.2.184)
+W0803 22:50:37.646432       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
+E0803 22:51:07.648172       1 main.go:251] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-vgftf': Get "https://10.0.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-xxxxx": dial tcp 10.0.0.1:443: i/o timeout
 ```
-The work-node-name is the new worker just added; its name can be found by ```./cluster/kubectl.sh get node```.
 
-You should be able to notice that node in READY state after a while; you can run container pods now:
+5. On worker node, add one more rule of Linux iptables to fix the above issue
+
+At the moment this doc is written, kube-proxy does not support multi-tenancy yet, and it won't be deployed in the cluster. After kube-proxy issue has been fixed (we will allocate resource to cope with it soon), probably we will also need to copy over kube-proxy related secret and artifacts.
+
+As a temporary measure to accommodate Flannel pods to access api server through the kubernetes service IP (the notorious 10.0.0.1) at the absense of kube-proxy, we need to place a bit trick by running below command __at the worker node__, assuming the kupe-apiserver is listening at https://10.138.0.19:6443 (please substitute with the proper value of your own cluster):
+
 ```bash
-./cluster/kubectl.sh run nginx --image=nginx --replicas=2
-./cluster/kubectl.sh get pod -o wide
+sudo iptables -t nat -A OUTPUT -p tcp -d 10.0.0.1 --dport 443 -j DNAT --to-destination 10.138.0.19:6443
+sudo iptables -t nat -L (for verification)
 ```
 
-To run a small cirros VM pod, you can use below yaml content:
-```yaml
-apiVersion: v1
-kind: Pod
-metadata:
-  name: cirros-vm
-spec:
-  virtualMachine:
-    name: vm
-    keyPairName: "foo"
-    image: download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
-    imagePullPolicy: IfNotPresent
-```
+NOTE: you need to re-run the above command once you restart the machine.
 
-# How to set up multiple partitioned apiservers
+Please be advised that this is a temporary quick only; after we have the full service support by proper kube-proxy or other means, we don't need it any more.
 
-It may be desired to setup a cluster having 2 or 3 apiservers in order to play with the scalability features of Arktos. It is simple and easy to implement it using arktos-up.sh, arktos-apiserver-partition.sh, and install-etcd.sh. The  doc describes the minimum effort to run a cluster having 2 apiservers (1 apiserver first, plus a apiserver who joins later).
+Then you should see pod 'pod/kube-flannel-ds-xxxxx' is in Running state after you run command "./cluster/kubectl.sh get all --all-namespaces". 
 
-Instructions are in [API Server Partition](arktos-apiserver-partition.md)
+6. Test whether the ngnix application can be deployed successfully
+
+NOTE: You need first run the following command to create clusterrolebinding 'system-node-role-bound' to bind the group 'system:nodes' to clusterrole 'system:node' so that the master node has corresponding permission to get secret for every namespace and transfer the secret to worker node during pod creation.
+
+```bash
+./cluster/kubectl.sh create clusterrolebinding system-node-role-bound --clusterrole=system:node --group=system:nodes
+./cluster/kubectl.sh get clusterrolebinding/system-node-role-bound -o yaml
+```
+
+Then you can run container pods for nginx now and see all pods should be in Running state.
+```bash
+./cluster/kubectl.sh run nginx --image=nginx --replicas=2
+./cluster/kubectl.sh get all -n default
+```

docs/setup-guide/setup-dev-env.md

@@ -1,28 +1,22 @@
-## Set up developer environment
+## Set up developer environment (updated on 2021-07-22)
 
-Note: tested on AWS EC2 Ubuntu 16.04 x86 image.
-
+Note: Tested on AWS EC2 Ubuntu 16.04/18.04/20.04 x86 image.
 
 ### Clone repo
 ```
 $ mkdir -p go/src/
 $ cd go/src/
-$ git clone https://github.com/futurewei-cloud/arktos
+$ git clone https://github.com/centaurusinfra/arktos
 ```
 
-Note: the following steps can be simplified by running hack/setup-dev-node.sh
-
-### Install Golang
+### Install needed packages (docker, make, gcc, jq and golang)
 ```
-$ sudo apt-get update
-# $ sudo apt-get -y upgrade // optional
-$ cd /tmp
-$ wget https://dl.google.com/go/go1.12.9.linux-amd64.tar.gz
-$ tar -xvf go1.12.9.linux-amd64.tar.gz
-$ sudo mv go /usr/local
-$ rm go1.12.9.linux-amd64.tar.gz
+$ cd arktos
+$ ./hack/setup-dev-node.sh
 ```
-Add the following lines to ~/.profile
+
+### Update your account's profile
+Add the following lines into the profile ~/.profile
 ```
 GOROOT=/usr/local/go
 GOPATH=$HOME/go
@@ -31,54 +25,5 @@ PATH=$GOPATH/bin:$GOROOT/bin:$PATH
 Update the current shell session
 ```
 $ source ~/.profile
-```
-
-### Install gcc and make. There might be an issue to build images. It can be fixed by running "git tag -a v2.7.4"
-```
-$ sudo apt install build-essential
-```
-
-### Install Docker
-```
-# sudo apt-get update -y -q
-
-# sudo apt-get install \
-    apt-transport-https \
-    ca-certificates \
-    curl \
-    gnupg-agent \
-    software-properties-common -y -q
-
-$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
-# sudo apt-key fingerprint 0EBFCD88
-
-# sudo add-apt-repository \
-   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
-   $(lsb_release -cs) \
-   stable"
-
-# sudo apt-get update -y -q
-# sudo apt-get install docker-ce docker-ce-cli containerd.io -y -q
-# sudo gpasswd -a $USER docker
-```
-
-### Install crictl
-```
-$ cd /tmp
-$ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.17.0/crictl-v1.17.0-linux-amd64.tar.gz
-# sudo tar zxvf crictl-v1.17.0-linux-amd64.tar.gz -C /usr/local/bin
-$ rm -f crictl-v1.17.0-linux-amd64.tar.gz
-
-$ touch /tmp/crictl.yaml
-$ echo runtime-endpoint: unix:///run/containerd/containerd.sock >> /tmp/crictl.yaml
-$ echo image-endpoint: unix:///run/containerd/containerd.sock >> /tmp/crictl.yaml
-$ echo timeout: 10 >> /tmp/crictl.yaml
-$ echo debug: true >> /tmp/crictl.yaml
-# sudo mv /tmp/crictl.yaml /etc/crictl.yaml
-
-# mkdir -p /etc/containerd
-# sudo rm -rf /etc/containerd/config.toml
-# sudo containerd config default > /tmp/config.toml
-# sudo mv /tmp/config.toml /etc/containerd/config.toml
-# sudo systemctl restart containerd
+$ echo $PATH
 ```

docs/setup-guide/single-node-dev-cluster.md

@@ -0,0 +1,44 @@
+# How to Setup a Dev Cluster of single node on AWS EC2 instance running Ubuntu 18.04, 16.04 x86 
+
+0. Pre-requisite - setup local development environment
+```
+  https://github.com/CentaurusInfra/arktos/blob/master/docs/setup-guide/setup-dev-env.md
+```
+
+1. Run script to create a single arktos cluster
+
+   Note: It is not support to start Arktos Cluster of single node on AWS EC2 instance running Ubuntu 20.04 x86.
+
+```bash
+   $ ./hack/arktos-up.sh
+```
+
+   Note: If your Dev Cluster of single node is on AWS EC2 instance running Ubuntu 18.04, after you run the command './hack/arktos-up.sh', you may experience the following error. Just simply remove the symbolic link and re-run the command './hack/arktos-up.sh'.
+
+         ln: failed to create symbolic link '/home/ubuntu/go/src/arktos/_output/bin': File exists 
+
+```bash
+   $ rm -i /home/ubuntu/go/src/arktos/_output/bin
+   $ ./hack/arktos-up.sh
+```
+
+
+2. Open another terminal to use arktos cluster
+```bash
+  $ export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
+
+  $ cluster/kubectl.sh
+
+Alternatively, you can write to the default kubeconfig:
+
+  $ export KUBERNETES_PROVIDER=local
+
+  $ cluster/kubectl.sh config set-cluster local --server=https://<hostname>:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt
+  $ cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt
+  $ cluster/kubectl.sh config set-context local --cluster=local --user=myself
+  $ cluster/kubectl.sh config use-context local
+  $ cluster/kubectl.sh config get-contexts
+  $ cluster/kubectl.sh
+  $ cluster/kubectl.sh get nodes
+  $ cluster/kubectl.sh get all --all-namespaces
+```

hack/setup-dev-node.sh

@@ -22,7 +22,7 @@ set -o pipefail
 
 echo "The script is to help install prerequisites of Arktos development environment"
 echo "on a fresh Linux installation."
-echo "It's been tested on Ubuntu 16.04 LTS and 18.04 LTS."
+echo "It's been tested on Ubuntu 16.04 LTS, 18.04 and 20.04 LTS."
 
 GOLANG_VERSION=${GOLANG_VERSION:-"1.13.9"}
 
@@ -42,5 +42,14 @@ wget https://dl.google.com/go/go${GOLANG_VERSION}.linux-amd64.tar.gz -P /tmp
 sudo tar -C /usr/local -xzf /tmp/go${GOLANG_VERSION}.linux-amd64.tar.gz
 
 echo "Done."
-echo "Please run and add 'export PATH=\$PATH:/usr/local/go/bin' into your shell profile."
+echo ""
+echo "Please add the following lines into your shell profile ~/.profile."
+echo "    GOROOT=/usr/local/go"
+echo "    GOPATH=\$HOME/go"
+echo "    export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin"
+echo ""
+echo "Update the current shell session."
+echo "    $ source ~/.profile"
+echo "    $ echo \$PATH"
+echo ""
 echo "You can proceed to run arktos-up.sh if you want to launch a single-node cluster."