集群部署2：搭建一个完整的Kubernetes集群

todo

一、准备工作

首先，准备机器。最好能有三台或三台以上的机器，虚拟机和物理机都行。这些机器最好能满足如下要求：

• 2核CPU、8GB内存；
• 30GB磁盘（注：磁盘上要有未使用的分区，否则就要多加一块为使用的磁盘）；
• 64位的Linux操作系统、3.10 及以上的内核版本；
• 内网互通；
• 能访问外网；
• 禁用防火墙和Swap；

然后，实际部署前我们先明确一下目标：

• 在所有节点上安装 Docker 和 kubeadm；
• 部署 Kubernetes Master；
• 部署容器网络插件；
• 部署 Kubernetes Worker；
• 部署 Dashboard 可视化插件；
• 部署容器存储插件。

二、安装Docker和Kubeadm

对于Docker和Kubeadm的安装，我们可以直接使用操作系统的包管理工具来安装，这里以ubuntu18.04为例。需要注意的是，由于国内访问国外资源被限制，因此，我将所有的网络资源替换为了国内站点。

2.1、安装Docker

# Import key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Create repo
sudo tee /etc/apt/sources.list.d/docker-ce.list << EOF
deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable
EOF

# Install Docker
sudo apt-get update && sudo apt-get install -y docker-ce=18.06.3~ce~3-0~ubuntu

# Config Docker
sudo mkdir -p /etc/docker

sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "dns": ["223.5.5.5", "223.6.6.6"],
  "registry-mirrors": ["https://jfb4kjfb.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# Start Docker
sudo systemctl daemon-reload

sudo systemctl restart docker.service
sudo systemctl enable docker.service
sudo usermod -aG docker $USER

2.2、安装Kubeadm

# Import key
curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -

# Create repo
sudo bash -c '
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
'

# Install
sudo apt-get update
sudo apt-get install -y kubernetes-cni=0.8.6-00 kubelet=1.16.14-00 kubeadm=1.16.14-00 kubectl=1.16.14-00

三、配置高可用环境

使用kubeadm工具，虽然可以通过部署多个master节点和worker节点来保证kubernetes相关组件的高可用，但是apiserver作为一个HTTP服务，仍需要通过负载均衡来提供统一的入口，从而实现真正意义上的高可用。对于apiserver的高可用，kubernetes社区提供了三种解决方案：

• 使用传统的部署方式，直接在操作系统上部署keepalived和haproxy
• 使用kubelet的static pod特性，通过容器部署keepalived和haproxy
• 使用kubelet的static pod特性，通过容器部署kube-vip

接下来，我们以第三种方案为例，构建一个高可用环境。在我的环境中，有三个master节点node1、node2和node3。需要注意的是，由于kube-vip采用的是raft算法，因此组建高可用集群时至少需要三个节点。

首先，在maste节点上生成kube-vip的配置文件，内容如下：

# node1
sudo mkdir -p /etc/kube-vip/
sudo tee /etc/kube-vip/config.yaml <<EOF
localPeer:
  id: node1
  address: 192.168.3.150
  port: 10000
remotePeers:
- id: node2
  address: 192.168.3.141
  port: 10000
- id: node3
  address: 192.168.3.146
  port: 10000
vip: 192.168.3.100
gratuitousARP: true
singleNode: false
startAsLeader: true
interface: eth0
loadBalancers:
- name: API Server Load Balancer
  type: tcp
  port: 64436
  bindToVip: false
  backends:
  - address: 192.168.3.150
    port: 6443
  - address: 192.168.3.141
    port: 6443
  - address: 192.168.3.146
    port: 6443
EOF

# node2
sudo mkdir -p /etc/kube-vip/
sudo tee /etc/kube-vip/config.yaml <<EOF
localPeer:
  id: node2
  address: 192.168.3.141
  port: 10000
remotePeers:
- id: node1
  address: 192.168.3.150
  port: 10000
- id: node3
  address: 192.168.3.146
  port: 10000
vip: 192.168.3.100
gratuitousARP: true
singleNode: false
startAsLeader: false
interface: eth0
loadBalancers:
- name: API Server Load Balancer
  type: tcp
  port: 64436
  bindToVip: false
  backends:
  - address: 192.168.3.150
    port: 6443
  - address: 192.168.3.141
    port: 6443
  - address: 192.168.3.146
    port: 6443
EOF

# node3
sudo mkdir -p /etc/kube-vip/
sudo tee /etc/kube-vip/config.yaml <<EOF
localPeer:
  id: node3
  address: 192.168.3.146
  port: 10000
remotePeers:
- id: node1
  address: 192.168.3.150
  port: 10000
- id: node2
  address: 192.168.3.141
  port: 10000
vip: 192.168.3.100
gratuitousARP: true
singleNode: false
startAsLeader: false
interface: eth0
loadBalancers:
- name: API Server Load Balancer
  type: tcp
  port: 64436
  bindToVip: false
  backends:
  - address: 192.168.3.150
    port: 6443
  - address: 192.168.3.141
    port: 6443
  - address: 192.168.3.146
    port: 6443
EOF

然后，在master节点上执行如下命令，生成kube-vip容器的资源描述性文件kube-vip.yaml：

# on all mater node
mkdir -p /etc/kubernetes/manifests
sudo docker run -it --rm plndr/kube-vip:0.1.1 /kube-vip sample manifest \
    | sed "s|plndr/kube-vip:'|plndr/kube-vip:0.1.1'|" \
    | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

这样，当我们初始化集群（执行kubeadm init）时，kubelet就会通过static pod机制运行kube-vip。

四、部署Kubernetes的Master节点

首先，我们在一台master节点上执行如下命令：

# 生成配置文件
mkdir -p ~/conf

cat > ~/conf/kubeadm-config.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.16.0
imageRepository: "registry.aliyuncs.com/google_containers"
apiServer:
  certSANs:
  - "192.168.3.100"
controlPlaneEndpoint: "192.168.3.100:64436"
networking:
  serviceSubnet: "10.244.0.0/16"
  podSubnet: "10.243.0.0/16"
EOF

# 预下载镜像
sudo kubeadm config images pull --config ~/conf/kubeadm-config.yaml

# 执行初始化
sudo kubeadm init --config ~/conf/kubeadm-config.yaml --upload-certs

如果看到类似如下的输出，就说明初始化成功了，否则，就是失败，此时可以根据提示信息进行排错：

[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [test1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.244.0.1 192.168.3.150 192.168.3.100 192.168.3.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [test1 localhost] and IPs [192.168.3.150 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [test1 localhost] and IPs [192.168.3.150 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 27.556250 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
becb9899b6f143931d1112da0b6c92e94f1bc694458d5d6b761b55c8c49cfd8a
[mark-control-plane] Marking the node test1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node test1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 8zsbwe.8hu68ikz58gw93os
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.3.100:64436 --token 8zsbwe.8hu68ikz58gw93os \
    --discovery-token-ca-cert-hash sha256:0c6a1f0591b3fdff6353e777f8ecf008c4fddeb6a48446369fe0b8dd65a3e55a \
    --control-plane --certificate-key becb9899b6f143931d1112da0b6c92e94f1bc694458d5d6b761b55c8c49cfd8a

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use 
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.3.100:64436 --token 8zsbwe.8hu68ikz58gw93os \
    --discovery-token-ca-cert-hash sha256:0c6a1f0591b3fdff6353e777f8ecf008c4fddeb6a48446369fe0b8dd65a3e55a

到这里，我们还无法直接使用集群，初始化操作完成后还需要执行如下操作：

mkdir -p $HOME/.kube
sudo cp -a /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

此时，如果我们查看当前集群中的节点数，会发现只有一个master角色的节点：

$ kubectl get node
NAME    STATUS     ROLES    AGE   VERSION
test1   NotReady   master   18m   v1.16.14

而如果想要将其它节点作为master角色加入到集群中，就需要在待加入的节点上执行如下命令：

# 生成配置文件
mkdir -p ~/conf

cat > ~/conf/kubeadm-config.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.16.0
imageRepository: "registry.aliyuncs.com/google_containers"
apiServer:
  certSANs:
  - "192.168.3.100"
controlPlaneEndpoint: "192.168.3.100:64436"
networking:
  serviceSubnet: "10.244.0.0/16"
  podSubnet: "10.243.0.0/16"
EOF

# 预下载镜像
sudo kubeadm config images pull --config ~/conf/kubeadm-config.yaml

# 加入集群
sudo kubeadm join 192.168.3.100:64436 --token 8zsbwe.8hu68ikz58gw93os \
  --discovery-token-ca-cert-hash sha256:0c6a1f0591b3fdff6353e777f8ecf008c4fddeb6a48446369fe0b8dd65a3e55a \
  --control-plane --certificate-key becb9899b6f143931d1112da0b6c92e94f1bc694458d5d6b761b55c8c49cfd8a \
  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests

如果看到类似如下的输出，就说明加入成功了，否则，就是失败，此时可以根据提示信息进行排错：

[preflight] Running pre-flight checks
	[WARNING DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [test2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.244.0.1 192.168.3.141 192.168.3.100 192.168.3.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [test2 localhost] and IPs [192.168.3.141 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [test2 localhost] and IPs [192.168.3.141 127.0.0.1 ::1]
[certs] Generating "front-proxy-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
[kubelet-check] Initial timeout of 40s passed.
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node test2 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node test2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

五、部署容器网络插件

得益于kubernetes一切皆容器的设计理念，部署网络插件非常简单，只需要执行一句kubectl apply命令即可，这里以weave插件为例：

# 获取flannel配置文件
wget -O ~/conf/kube-weave.yml https://github.com/weaveworks/weave/releases/download/latest_release/weave-daemonset-k8s-1.11.yaml

# 获取flannel镜像
awk '/image/{print "sudo docker pull " $2}' ~/conf/kube-weave.yml | bash

# 安装flannel网络
kubectl apply -f ~/conf/kube-weave.yml

六、部署Kubernetes的Worker节点

有了kubeadm后，worker节点的部署也变得非常简单，在worker节点上安装好docker和kubelet之后，执行初始化master节点时生成的join指令即可，如下：

sudo kubeadm join 192.168.104.100:64436 --token nhb7g9.mqe2do308eda9m6g \
    --discovery-token-ca-cert-hash sha256:d9e07db543a2deaecad8946d18a8c8a7a956d255bf30b0c29f1adb0d4237cdf7

如果看到类似如下的输出，就说明加入成功了，否则，就是失败，此时可以根据提示信息进行排错：

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

七、调整Master节点的运行机制

7.1、Taint/Toleration机制

在默认情况下，master节点是不允许运行用户Pod的。而Kubernetes之所以能做到这一点，依靠的是一个叫做Taint/Toleration的机制。

该机制的原理非常简单：一旦某个节点被加上了某个Taint，即被“打上了污点”，那么所有的Pod就都不能在这个节点上运行，因为Kubernetes的Pod都是有“洁癖”的。除非，Pod自己声明能“容忍”这个“污点”，即在资源描述文件中定义了Toleration，它才能在这个有“污点”的节点上运行。

7.2、给节点打“污点”

那么，如何给节点打“污点”呢？

要为节点打上“污点”也很简单，比如：

kubectl taint nodes node1 foo=bar:NoSchedule

这时，节点node1上就会增加一个键值对格式的Taint，即：foo=bar:NoSchedule。在这里foo就是键，bar:NoSchedule就是值，其中，值里面的NoSchedule表示这个Taint只会在调度新Pod时才会产生作用，而不会影响已经在node1上运行的Pod，哪怕它并没有定义能“容忍”的“污点”（Toleration）。

7.3、设置Pod能“容忍”的“污点”

那么，Pod又是如何定义Toleration的呢？

我们只要在Pod的.yaml文件中的spec部分，加入tolerations字段即可，如下：

apiVersion: v1
kind: Pod
...
spec:
  tolerations:
  - key: "foo"
    operator: "Equal"
    value: "bar"
    effect: "NoSchedule"

这个Toleration的含义是，该Pod能“容忍”所有键值对为foo=bar的Taint，其中operator: "Equal"表示等于操作。

7.4、查看节点的“污点”

现在，回到我们刚刚搭建的集群上来。此时，如果我们通过kubectl describe检查一下master节点的Taint字段，就会发现：

$ kubectl describe node node1 | grep Taint
Taints:             node-role.kubernetes.io/master:NoSchedule

master节点默认被加上了一个"污点"，这个“污点”的“键”是node-role.kubernetes.io/master，而没有提供“值”。

此时，如果我们想要将某个Pod运行在master节点上，就需要像下面这样描述Pod，用“Exists”操作符来声明，该Pod能够“容忍”所有以“node-role.kubernetes.io/master”为键的Taint：

apiVersion: v1
kind: Pod
...
spec:
  tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"

这样，这个Pod才能在master节点上运行。

7.5、删除节点的“污点”

当然，如果我们想要在master节点上运行所有的pod，只需要删除这个Taint即可：

kubectl taint nodes --all node-role.kubernetes.io/master-

如上，我们在“node-role.kubernetes.io/master”这个键后面加上了一个减号“-”,这表示删除所有以“node-role.kubernetes.io/master”为键额Taint。

八、部署Dashboard可视化插件

在Kubernetes社区中，提供了一个Dashboard项目，通过它我们可以通过Web界面来查看当前集群的各种信息，这个项目的部署也非常简单：

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml

部署完成之后，我们就可以查看 Dashboard 对应的 Pod 的状态了：

kubectl get pods -n kubernetes-dashboard

需要注意的是，由于 Dashboard 是一个 Web Server，很多人经常会在自己的公有云上无意地暴露 Dashboard 的端口，从而造成安全隐患。所以，1.7 版本之后的 Dashboard 项目部署完成后，默认只能通过 Proxy 的方式在本地访问。具体的操作，可以查看 Dashboard 项目的官方文档。

https://github.com/kubernetes/dashboard

而如果想从集群外访问这个 Dashboard 的话，可参考如下链接中提供的几种方法：

https://github.com/kubernetes/dashboard/blob/master/docs/user/accessing-dashboard/README.md

九、部署容器存储插件

在这里，我们还需要完成Kubernetes部署的最后一步：容器持久化存储。

很多时候我们需要用数据卷（Volume）把宿主机上的目录或者文件挂载进容器的 Mount Namespace 中，从而达到容器和宿主机共享这些目录或者文件的目的。容器里的应用，也就可以在这些数据卷中新建和写入文件。

容器最典型的特征之一就是：无状态。而容器的持久化存储，就是用来保存容器存储状态的重要手段：存储插件会在容器里挂载一个基于网络或者其他机制的远程数据卷，使得在容器里创建的文件，实际上是保存在远程存储服务器上，或者以分布式的方式保存在多个节点上，而与当前宿主机没有任何绑定关系。

这样，无论在其他哪个宿主机上启动新的容器，都可以请求挂载指定的持久化存储卷，从而访问到数据卷里保存的内容。这就是“持久化”的含义。

由于 Kubernetes 本身的松耦合设计，绝大多数存储项目，比如 Ceph、GlusterFS、NFS 等，都可以为 Kubernetes 提供持久化存储能力。接下来，我们将部署一个很重要的 Kubernetes 存储插件项目：Rook。

Rook 项目是一个基于 Ceph 的 Kubernetes 存储插件（它后期也在加入对更多存储实现的支持）。不过，不同于对 Ceph 的简单封装，Rook 在自己的实现中加入了水平扩展、迁移、灾难备份、监控等大量的企业级功能，使得这个项目变成了一个完整的、生产级别可用的容器存储插件。

得益于容器化技术，用几条指令，Rook 就可以把复杂的 Ceph 存储后端部署起来：

git clone --single-branch --branch v1.5.7 https://github.com/rook/rook.git
cd rook/cluster/examples/kubernetes/ceph

kubectl apply -f crds.yaml
kubectl apply -f common.yaml
kubectl apply -f operator.yaml

kubectl -n rook-ceph create secret generic rook-ceph-crash-collector-keyring

kubectl apply -f cluster.yaml

部署完成后，就可以看到 Rook 项目会将自己的 Pod 放置在由它自己管理的两个 Namespace 当中：

$ kubectl get pods -n rook-ceph-system
NAME                                  READY     STATUS    RESTARTS   AGE
rook-ceph-agent-7cv62                 1/1       Running   0          15s
rook-ceph-operator-78d498c68c-7fj72   1/1       Running   0          44s
rook-discover-2ctcv                   1/1       Running   0          15s

$ kubectl get pods -n rook-ceph
NAME                   READY     STATUS    RESTARTS   AGE
rook-ceph-mon0-kxnzh   1/1       Running   0          13s
rook-ceph-mon1-7dn2t   1/1       Running   0          2s

这样，一个基于 Rook 持久化存储集群就以容器的方式运行起来了，而接下来在 Kubernetes 项目上创建的所有 Pod 就能够通过 Persistent Volume（PV）和 Persistent Volume Claim（PVC）的方式，在容器里挂载由 Ceph 提供的数据卷了。而 Rook 项目，则会负责这些数据卷的生命周期管理、灾难备份等运维工作。

这时候，我们可能会有个疑问：为什么要选择 Rook 项目呢？

其实，是因为这个项目很有前途。如果去研究一下 Rook 项目的实现，就会发现它巧妙地依赖了 Kubernetes 提供的编排能力，合理的使用了很多诸如 Operator、CRD 等重要的扩展特性（这些特性我都会在后面的文章中逐一讲解到）。这使得 Rook 项目，成为了目前社区中基于 Kubernetes API 构建的最完善也最成熟的容器存储插件。

备注：其实，在很多时候，大家说的所谓“云原生”，就是“Kubernetes 原生”的意思。而像 Rook、Istio 这样的项目，正是贯彻这个思路的典范。