环境准备
| 节点名 |
IP |
角色 |
| node2 |
192.168.41.129 |
master |
| node3 |
192.168.41.130 |
slave |
| node4 |
192.168.41.131 |
slave |
系统配置
在安装之前，需要先做如下准备,修改hosts：
cat /etc/hosts
192.168.41.129 node2
192.168.41.130 node3
192.168.41.131 node4
禁用防火墙：
systemctl stop firewalld
systemctl disable firewalld
systemctl stop iptables
systemctl disable iptables
禁用SELINUX：
setenforce 0
vi /etc/selinux/config
SELINUX=disabled
#检查SELINUX状态
/usr/sbin/sestatus -v
SELinux status: disabled
创建/etc/sysctl.d/k8s.conf文件，添加如下内容：（vm.swappiness=0是swap相关配置，后面需要配置，这里统一加上）
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
执行命令使修改生效。
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
kube-proxy开启ipvs的前置条件
由于ipvs已经加入到了内核的主干，所以为kube-proxy开启ipvs的前提需要加载以下的内核模块：
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
在所有的Kubernetes节点node1和node2上执行以下脚本:
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。 使用
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
命令查看是否已经正确加载所需的内核模块。
接下来还需要确保各个节点上已经安装了ipset软件包
yum install ipset
为了便于查看ipvs的代理规则，最好安装一下管理工具ipvsadm 。
yum install ipvsadm
如果以上前提条件如果不满足，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式。
安装Docker
Kubernetes从1.6开始使用CRI(Container Runtime Interface)容器运行时接口。默认的容器运行时仍然是Docker，使用的是kubelet中内置dockershim CRI实现。
安装docker的yum源:
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
查看最新的Docker版本：
yum list docker-ce.x86_64 --showduplicates |sort -r
docker-ce.x86_64 3:18.09.7-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.6-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.5-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.4-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.3-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.2-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.3.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.2.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable
Kubernetes 1.15当前支持的docker版本列表是1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09。 这里在各节点安装docker的18.09.7版本。
yum makecache fast
yum install -y --setopt=obsoletes=0 \
docker-ce-18.09.7-3.el7
systemctl start docker
systemctl enable docker
确认一下iptables filter表中FOWARD链的默认策略(pllicy)为ACCEPT:
iptables -nvL
Chain INPUT (policy ACCEPT 263 packets, 19209 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
修改docker cgroup driver为systemd
根据文档CRI installation中的内容，对于使用systemd作为init system的Linux的发行版，使用systemd作为docker的cgroup driver可以确保服务器节点在资源紧张的情况更加稳定，因此这里修改各个节点上docker的cgroup driver为systemd。
创建或修改/etc/docker/daemon.json：
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
重启docker：
systemctl restart docker
docker info | grep Cgroup
Cgroup Driver: systemd
使用kubeadm部署Kubernetes
安装kubeadm和kubelet
下面在各节点安装kubeadm和kubelet：
cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
#由于国内网络不通问题，推荐使用以下阿里云镜像
cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum makecache fast
#默认可能安装的版本不一样
yum install -y kubelet kubeadm kubectl
#建议指定版本
yum install -y kubelet-1.15.0 kubeadm-1.15.0 kubectl-1.15.0
Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。 关闭系统的Swap方法如下:
swapoff -a
永久关闭swap：
修改 /etc/fstab 文件，注释掉 SWAP 的自动挂载，使用free -m确认swap已经关闭。
swappiness参数调整，修改/etc/sysctl.d/k8s.conf添加下面一行,前面已经统一加过了：
vm.swappiness=0
执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。
关闭swap可能会对其他服务产生影响，所以这里修改kubelet的配置去掉这个限制。
使用kubelet的启动参数–fail-swap-on=false去掉必须关闭Swap的限制，修改/etc/sysconfig/kubelet，加入：
KUBELET_EXTRA_ARGS=--fail-swap-on=false
使用kubeadm init初始化集群
在各节点开机启动kubelet服务：
systemctl enable kubelet.service
使用kubeadm config print init-defaults可以打印集群初始化默认的使用的配置：
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 1.2.3.4
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: node1
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.14.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
scheduler: {}
从默认的配置中可以看到，可以使用imageRepository定制在集群初始化时拉取k8s所需镜像的地址。基于默认配置定制出本次使用kubeadm初始化集群所需的配置文件kubeadm.yaml：
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.41.129
bindPort: 6443
nodeRegistration:
taints:
- effect: PreferNoSchedule
key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.15.0
networking:
podSubnet: 10.244.0.0/16
在开始初始化集群之前可以使用
kubeadm config images pull
预先在各个节点上拉取所k8s需要的docker镜像。
由于国内网络问题，k8s.gcr.io 资源获取不了,使用以下方法
#!/bin/bash
MY_REGISTRY=gcr.azk8s.cn/google-containers
## 拉取镜像
docker pull ${MY_REGISTRY}/kube-apiserver:v1.15.0
docker pull ${MY_REGISTRY}/kube-controller-manager:v1.15.0
docker pull ${MY_REGISTRY}/kube-scheduler:v1.15.0
docker pull ${MY_REGISTRY}/kube-proxy:v1.15.0
docker pull ${MY_REGISTRY}/pause:3.1
docker pull ${MY_REGISTRY}/etcd:3.3.10
docker pull ${MY_REGISTRY}/coredns:1.3.1
## 添加Tag
docker tag ${MY_REGISTRY}/kube-apiserver:v1.15.0 k8s.gcr.io/kube-apiserver:v1.15.0
docker tag ${MY_REGISTRY}/kube-controller-manager:v1.15.0 k8s.gcr.io/kube-controller-manager:v1.15.0
docker tag ${MY_REGISTRY}/kube-scheduler:v1.15.0 k8s.gcr.io/kube-scheduler:v1.15.0
docker tag ${MY_REGISTRY}/kube-proxy:v1.15.0 k8s.gcr.io/kube-proxy:v1.15.0
docker tag ${MY_REGISTRY}/pause:3.1 k8s.gcr.io/pause:3.1
docker tag ${MY_REGISTRY}/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag ${MY_REGISTRY}/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
#删除无用的镜像
docker images | grep ${MY_REGISTRY} | awk '{print "docker rmi " $1":"$2}' | sh -x
echo "end"
上面的所有操作可以在一个节点上面完成，然后对进行复制即可。
其它因为网络原因pull的镜像都可以同理先下载下来:
#1.先使用docker search命令查找
#2.选择一个下载快的镜像使用docker pull下载下来
#3.使用docker tag命令修改成程序需要的镜像
#4.使用docker rmi删除之前的镜像
#下载flannel网络镜像
docker pull xmlgrg/flannel:v0.11.0-amd64
docker tag docker.io/xmlgrg/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
docker rmi docker.io/xmlgrg/flannel:v0.11.0-amd64
#下载tiller
docker pull junolu/tiller:v2.14.3
docker tag docker.io/junolu/tiller:v2.14.3 gcr.io/kubernetes-helm/tiller:v2.14.3
docker rmi docker.io/junolu/tiller:v2.14.3
#下载ingress-nginx/defaultbackend-amd64
docker pull fungitive/defaultbackend-amd64
docker tag docker.io/fungitive/defaultbackend-amd64:latest k8s.gcr.io/defaultbackend-amd64:1.5
docker rmi docker.io/fungitive/defaultbackend-amd64:latest
#下载dashboard
docker.io/loveone/kubernetes-dashboard-amd64:v1.10.1
docker tag docker.io/loveone/kubernetes-dashboard-amd64:v1.10.1 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
docker rmi docker.io/loveone/kubernetes-dashboard-amd64:v1.10.1
接下来使用kubeadm初始化集群，选择node2作为Master Node，在node2上执行下面的命令：
kubeadm init --config kubeadm.yaml --ignore-preflight-errors=Swap
[init] Using Kubernetes version: v1.15.0
[preflight] Running pre-flight checks
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [node1 localhost] and IPs [192.168.99.11 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [node1 localhost] and IPs [192.168.99.11 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.99.11]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 26.004907 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node node1 as control-plane by adding the taints [node-role.kubernetes.io/master:PreferNoSchedule]
[bootstrap-token] Using token: 4qcl2f.gtl3h8e5kjltuo0r
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.99.11:6443 --token 4qcl2f.gtl3h8e5kjltuo0r \
--discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e
上面记录了完成的初始化输出的内容，根据输出的内容基本上可以看出手动初始化安装一个Kubernetes集群所需要的关键步骤。 其中有以下关键内容：
- [kubelet-start] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”
- [certs]生成相关的各种证书
- [kubeconfig]生成相关的kubeconfig文件
- [control-plane]使用/etc/kubernetes/manifests目录中的yaml文件创建apiserver、controller-manager、scheduler的静态pod
- [bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
- 最后给出了将节点加入集群的命令
kubeadm join 18.16.202.163:6443 --token jrts59.18pe12atfafgcxca \ --discovery-token-ca-cert-hash sha256:56d6c7d7b63a9109444ece68a1b155d8a9ac049ba57febab2c72d40d8ab7d426
下面的命令是配置常规用户如何使用kubectl访问集群：务必配上，不然无法使用kubectl命令！
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
集群初始化如果遇到问题，可以使用下面的命令进行清理：
kubeadm reset
遇到的问题：The connection to the server localhost:8080 was refused - did you specify the right host or port?
初始化的集群，kube-apiserver没有监听默认的http 8080端口。
查看kube-apiserver的监听端口可以看到只监听了https的6443端口:
netstat -nltp | grep apiserver
tcp6 0 0 :::6443 :::* LISTEN 9831/kube-apiserver
为了使用kubectl访问apiserver，在~/.bash_profile中追加下面的环境变量：
export KUBECONFIG=/etc/kubernetes/admin.conf
source ~/.bash_profile
此时kubectl命令在master node上就好用了.
安装Pod Network
使用flannel网络
接下来安装flannel network add-on：
mkdir -p ~/k8s/
cd ~/k8s
curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created
这里注意kube-flannel.yml这个文件里的flannel的镜像是0.11.0，quay.io/coreos/flannel:v0.11.0-amd64.
由于网络问题，flannel镜像可能pull失败，建议参考之前拉取镜像的方法，提前再各个节点拉好镜像！
如果Node有多个网卡的话，参考flannel issues 39701，目前需要在kube-flannel.yml中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上–iface=<iface-name>
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth1
# ......
使用kubectl get pod –all-namespaces -o wide确保所有的Pod都处于Running状态。
kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5c98db65d4-dr8lf 1/1 Running 0 52m
coredns-5c98db65d4-lp8dg 1/1 Running 0 52m
etcd-node1 1/1 Running 0 51m
kube-apiserver-node1 1/1 Running 0 51m
kube-controller-manager-node1 1/1 Running 0 51m
kube-flannel-ds-amd64-mm296 1/1 Running 0 44s
kube-proxy-kchkf 1/1 Running 0 52m
kube-scheduler-node1 1/1 Running 0 51m
问题，flannel一直在CrashLoopBackOff，日志如下
[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5c98db65d4-57zm5 0/1 ContainerCreating 0 7m46s
kube-system coredns-5c98db65d4-g8kqh 0/1 ContainerCreating 0 7m46s
kube-system etcd-master 1/1 Running 0 6m51s
kube-system kube-apiserver-master 1/1 Running 0 6m31s
kube-system kube-controller-manager-master 1/1 Running 0 6m29s
kube-system kube-flannel-ds-amd64-jthsp 0/1 CrashLoopBackOff 3 104s
[root@master ~]# kubectl logs kube-flannel-ds-amd64-jthsp -n kube-system
I1105 08:37:34.863407 1 main.go:527] Using interface with name eth0 and address 192.168.1.209
I1105 08:37:34.863461 1 main.go:544] Defaulting external address to interface address (192.168.1.209)
I1105 08:37:35.061919 1 kube.go:126] Waiting 10m0s for node controller to sync
I1105 08:37:35.166015 1 kube.go:309] Starting kube subnet manager
I1105 08:37:36.065963 1 kube.go:133] Node controller sync successful
I1105 08:37:36.065985 1 main.go:244] Created subnet manager: Kubernetes Subnet Manager - master
I1105 08:37:36.065989 1 main.go:247] Installing signal handlers
I1105 08:37:36.066082 1 main.go:386] Found network config - Backend type: vxlan
I1105 08:37:36.066143 1 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
E1105 08:37:36.066314 1 main.go:289] Error registering network: failed to acquire lease: node "master" pod cidr not assigned
I1105 08:37:36.066356 1 main.go:366] Stopping shutdownHandler...
解决办法:
# 修改
/etc/kubernetes/manifests/kube-controller-manager.yaml
# 增加参数
--allocate-node-cidrs=true
--cluster-cidr=10.244.0.0/16
# 然后重启kubelet
使用weave网络
虽然系统显示安装成功，但是master节点显示状态还是 NotReady，原因是需要安装网络插件，这边kubernetes官方提供了几款网络插件。。
kubectl apply -f https://git.io/weave-kube-1.6
NAME STATUS ROLES AGE VERSION
izj6c3vsekthy7xn94b5s6z Ready master 1h v1.10.3
大概过一会master节点就显示ready了，说明master已经准备就绪了.
测试集群DNS是否可用
kubectl run curl --image=radial/busyboxplus:curl -it
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
If you don't see a command prompt, try pressing enter.
[ root@curl-5cc7b478b6-r997p:/ ]$
进入后执行nslookup kubernetes.default确认解析正常:
nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
向Kubernetes集群中添加Node节点
下面将node3这个主机添加到Kubernetes集群中，在node3上执行:
kubeadm join 192.168.99.11:6443 --token 4qcl2f.gtl3h8e5kjltuo0r \
--discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e \
--ignore-preflight-errors=Swap
[preflight] Running pre-flight checks
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
node3加入集群后，下面在master节点上执行命令查看集群中的节点：
kubectl get node
NAME STATUS ROLES AGE VERSION
node2 Ready master 57m v1.15.0
node3 Ready 11s v1.15.0
如果只想搭建一个只有master节点的单机测试环境
单机版kubernetes为了运行Pod.需要删除主机上的Train.允许master执行Pod.
执行命令如下：
kubectl taint nodes --all node-role.kubernetes.io/master-
遇到的问题
执行上述命令一直卡在了
[preflight] Running pre-flight checks
- 问题可能的原因是master防火墙没有关掉，检查firewalld和iptables
- 加上 –v=2 命令具体看看
从集群中移除Node
在master节点上执行：
kubectl drain node2 --delete-local-data --force --ignore-daemonsets
kubectl delete node node2
在Node上执行：
kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/
kube-proxy开启ipvs
修改ConfigMap的kube-system/kube-proxy中的config.conf，mode: "ipvs"
kubectl edit cm kube-proxy -n kube-system
之后重启各个节点上的kube-proxy pod：
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'
kubectl get pod -n kube-system | grep kube-proxy
kube-proxy-7fsrg 1/1 Running 0 3s
kube-proxy-k8vhm 1/1 Running 0 9s
kubectl logs kube-proxy-7fsrg -n kube-system
I0703 04:42:33.308289 1 server_others.go:170] Using ipvs Proxier.
W0703 04:42:33.309074 1 proxier.go:401] IPVS scheduler not specified, use rr by default
I0703 04:42:33.309831 1 server.go:534] Version: v1.15.0
I0703 04:42:33.320088 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0703 04:42:33.320365 1 config.go:96] Starting endpoints config controller
I0703 04:42:33.320393 1 controller_utils.go:1029] Waiting for caches to sync for endpoints config controller
I0703 04:42:33.320455 1 config.go:187] Starting service config controller
I0703 04:42:33.320470 1 controller_utils.go:1029] Waiting for caches to sync for service config controller
I0703 04:42:33.420899 1 controller_utils.go:1036] Caches are synced for endpoints config controller
I0703 04:42:33.420969 1 controller_utils.go:1036] Caches are synced for service config controller
日志中打印出了Using ipvs Proxier，说明ipvs模式已经开启。
Kubernetes常用组件部署
越来越多的公司和团队开始使用Helm这个Kubernetes的包管理器，这里也将使用Helm安装Kubernetes的常用组件。
Helm的安装
Helm由客户端命helm令行工具和服务端tiller组成，Helm的安装十分简单。 下载helm命令行工具到master节点node1的/usr/local/bin下，这里下载的2.14.1版本.
网络问题下载不了，可以直接在网上搜个压缩包拷进去即可！
curl -O https://get.helm.sh/helm-v2.14.1-linux-amd64.tar.gz
tar -zxvf helm-v2.14.1-linux-amd64.tar.gz
cd linux-amd64/
cp helm /usr/local/bin/
为了安装服务端tiller，还需要在这台机器上配置好kubectl工具和kubeconfig文件，确保kubectl工具可以在这台机器上访问apiserver且正常使用。
因为Kubernetes APIServer开启了RBAC访问控制，所以需要创建tiller使用的service account: tiller并分配合适的角色给它。 详细内容可以查看helm文档中的Role-based Access Control。 这里简单起见直接分配cluster-admin这个集群内置的ClusterRole给它。创建helm-rbac.yaml文件：
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
kubectl create -f helm-rbac.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
接下来使用helm部署tiller:
helm init --service-account tiller --skip-refresh
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
tiller默认被部署在k8s集群中的kube-system这个namespace下：
kubectl get pod -n kube-system -l app=helm
NAME READY STATUS RESTARTS AGE
tiller-deploy-c4fd4cd68-dwkhv 1/1 Running 0 83s
helm version
Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
遇到的问题!!!!
tiller可能因为网络问题安装失败，可以使用以下命令重置
helm reset --force
然后安装tiller的时候指定image和stable url，这里用的阿里云的
helm init --service-account tiller --skip-refresh -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.14.3 --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
最后在node1上修改helm chart仓库的地址为azure提供的镜像地址：
helm repo add stable http://mirror.azure.cn/kubernetes/charts
"stable" has been added to your repositories
helm repo list
NAME URL
stable http://mirror.azure.cn/kubernetes/charts
local http://127.0.0.1:8879/charts
使用Helm部署Nginx Ingress
为了便于将集群中的服务暴露到集群外部，需要使用Ingress。接下来使用Helm将Nginx Ingress部署到Kubernetes上。
Nginx Ingress Controller被部署在Kubernetes的边缘节点上
我们将node1(192.168.99.11)做为边缘节点，打上Label：
kubectl label node node1 node-role.kubernetes.io/edge=
node/node1 labeled
kubectl get node
NAME STATUS ROLES AGE VERSION
node1 Ready edge,master 138m v1.15.0
node2 Ready 82m v1.15.0
stable/nginx-ingress chart的值文件ingress-nginx.yaml如下：
controller:
replicaCount: 1
hostNetwork: true
nodeSelector:
node-role.kubernetes.io/edge: ''
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nginx-ingress
- key: component
operator: In
values:
- controller
topologyKey: kubernetes.io/hostname
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: PreferNoSchedule
defaultBackend:
nodeSelector:
node-role.kubernetes.io/edge: ''
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: PreferNoSchedule
nginx ingress controller的副本数replicaCount为1，将被调度到node1这个边缘节点上。
这里并没有指定nginx ingress controller service的externalIPs，而是通过hostNetwork: true设置nginx ingress controller使用宿主机网络。
helm repo update
# helm安装指定chart版本 --version
helm install stable/nginx-ingress \
-n nginx-ingress \
--namespace ingress-nginx \
-f ingress-nginx.yaml --verion 1.27.1
kubectl get pod -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-controller-cc9b6d55b-pr8vr 1/1 Running 0 10m 192.168.99.11 node1
nginx-ingress-default-backend-cc888fd56-bf4h2 1/1 Running 0 10m 10.244.0.14 node1
如果访问http://192.168.99.11返回default backend，则部署完成。
遇到的问题!!!!
这里可能因为网络原因安装失败需要清理之前的nginx-ingress，建议提前下载好相关镜像。
使用kubectl describe pod [name] -n [namespace]看看默认拉取的是哪个版本的镜像，参考之前方法提前下载！
kubectl delete namespace/ingress-nginx
helm delete nginx-ingress
helm del --purge nginx-ingress
使用Helm部署dashboard
创建tls secret：
通过https进行访问必需要使用证书和密钥，在Kubernetes中可以通过配置一个加密凭证（TLS secret）来提供。创建一个自己签名的证书:
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout ./tls.key -out ./tls.crt -subj "/CN=192.168.1.210"
将会产生两个文件tls.key和tls.crt，你可以改成自己的文件名或放在特定的目录下。后面的192.168.1.210是我的服务器IP地址，你需要改成自己的。
安装tls secret:
将这两个文件的信息创建为一个Kubernetes的secret访问凭证，我将名称指定为test-com-tls-secret，这在后面的Ingress配置时将会用到。
如果你修改了这个名字，注意后面的配置yaml文件也需要同步修改。
kubectl -n kube-system create secret tls test-com-tls-secret --key ./tls.key --cert ./tls.crt
查看:
kubectl get secret -n kube-system |grep test
kubernetes-dashboard.yaml:
注意修改里面的secretName和hosts
image:
repository: k8s.gcr.io/kubernetes-dashboard-amd64
tag: v1.10.1
ingress:
enabled: true
hosts:
- k8s.test.com
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
tls:
- secretName: test-com-tls-secret
hosts:
- k8s.test.com
nodeSelector:
node-role.kubernetes.io/edge: ''
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: PreferNoSchedule
rbac:
clusterAdminRole: true
安装：
helm install stable/kubernetes-dashboard \
-n kubernetes-dashboard \
--namespace kube-system \
-f kubernetes-dashboard.yaml
查看：
kubectl -n kube-system get secret | grep kubernetes-dashboard-token
kubernetes-dashboard-token-pkm2s kubernetes.io/service-account-token 3 3m7s
kubectl describe -n kube-system secret/kubernetes-dashboard-token-pkm2s
Name: kubernetes-dashboard-token-pkm2s
Namespace: kube-system
Labels:
Annotations: kubernetes.io/service-account.name: kubernetes-dashboard
kubernetes.io/service-account.uid: 2f0781dd-156a-11e9-b0f0-080027bb7c43
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1025 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1wa20ycyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjJmMDc4MWRkLTE1NmEtMTFlOS1iMGYwLTA4MDAyN2JiN2M0MyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.24ad6ZgZMxdydpwlmYAiMxZ9VSIN7dDR7Q6-RLW0qC81ajXoQKHAyrEGpIonfld3gqbE0xO8nisskpmlkQra72-9X6sBPoByqIKyTsO83BQlME2sfOJemWD0HqzwSCjvSQa0x-bUlq9HgH2vEXzpFuSS6Svi7RbfzLXlEuggNoC4MfA4E2hF1OX_ml8iAKx-49y1BQQe5FGWyCyBSi1TD_-ZpVs44H5gIvsGK2kcvi0JT4oHXtWjjQBKLIWL7xxyRCSE4HmUZT2StIHnOwlX7IEIB0oBX4mPg2_xNGnqwcu-8OERU9IoqAAE2cZa0v3b5O2LMcJPrcxrVOukvRIumA
测试：
在dashboard的登录窗口使用上面的token登录，访问刚刚填写的hosts，如果是内网域名，需要在访问的那台机器上，在hosts文件中加入刚刚的域名和IP。
![k8s]()
使用Helm部署metrics-server
heapster从Kubernetes 1.12开始从Kubernetes各种安装脚本中移除。
Kubernetes推荐使用metrics-server。我们这里也使用helm来部署metrics-server。
metrics-server.yaml:
replicaCount: 1
image:
repository: hub.deri.org.cn/k8s/metrics-server-amd64
tag: v0.3.5
pullPolicy: IfNotPresent
args:
- --logtostderr
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
nodeSelector:
node-role.kubernetes.io/edge: ''
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: PreferNoSchedule
helm install stable/metrics-server \
-n metrics-server \
--namespace kube-system \
-f metrics-server.yaml
使用下面的命令可以获取到关于集群节点基本的指标信息：
kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
node1 650m 32% 1276Mi 73%
node2 73m 3% 527Mi 30%
kubectl top pod -n kube-system
NAME CPU(cores) MEMORY(bytes)
coredns-5c98db65d4-dr8lf 8m 7Mi
coredns-5c98db65d4-lp8dg 6m 8Mi
etcd-node1 44m 46Mi
kube-apiserver-node1 74m 295Mi
kube-controller-manager-node1 35m 50Mi
kube-flannel-ds-amd64-7lwm9 2m 8Mi
kube-flannel-ds-amd64-mm296 5m 9Mi
kube-proxy-7fsrg 1m 11Mi
kube-proxy-k8vhm 3m 11Mi
kube-scheduler-node1 8m 15Mi
kubernetes-dashboard-848b8dd798-c4sc2 2m 14Mi
metrics-server-8456fb6676-fwh2t 10m 19Mi
tiller-deploy-7bf78cdbf7-9q94c 1m 16Mi
参考链接
