Happywzy

enjoy coding...

0%

kubernetes-v1.15.0安装【使用kubeadm部署】

发表于 分类于

环境准备

节点名 IP 角色
node2 192.168.41.129 master
node3 192.168.41.130 slave
node4 192.168.41.131 slave

系统配置

在安装之前，需要先做如下准备,修改hosts：

cat /etc/hosts
192.168.41.129 node2
192.168.41.130 node3
192.168.41.131 node4

禁用防火墙：

systemctl stop firewalld
systemctl disable firewalld
 
systemctl stop iptables
systemctl disable iptables

禁用SELINUX：

setenforce 0

vi /etc/selinux/config
SELINUX=disabled

#检查SELINUX状态
/usr/sbin/sestatus -v
SELinux status: disabled

创建/etc/sysctl.d/k8s.conf文件，添加如下内容：（vm.swappiness=0是swap相关配置，后面需要配置，这里统一加上）

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0

执行命令使修改生效。

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

kube-proxy开启ipvs的前置条件

由于ipvs已经加入到了内核的主干，所以为kube-proxy开启ipvs的前提需要加载以下的内核模块：

ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4

在所有的Kubernetes节点node1和node2上执行以下脚本:

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。 使用

lsmod | grep -e ip_vs -e nf_conntrack_ipv4

命令查看是否已经正确加载所需的内核模块。

接下来还需要确保各个节点上已经安装了ipset软件包

yum install ipset

为了便于查看ipvs的代理规则，最好安装一下管理工具ipvsadm 。

yum install ipvsadm

如果以上前提条件如果不满足，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式。

安装Docker

Kubernetes从1.6开始使用CRI(Container Runtime Interface)容器运行时接口。默认的容器运行时仍然是Docker，使用的是kubelet中内置dockershim CRI实现。

安装docker的yum源:

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

查看最新的Docker版本：

yum list docker-ce.x86_64  --showduplicates |sort -r
docker-ce.x86_64            3:18.09.7-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.6-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.5-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.4-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.3-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.2-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.1-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.0-3.el7                     docker-ce-stable
docker-ce.x86_64            18.06.3.ce-3.el7                    docker-ce-stable
docker-ce.x86_64            18.06.2.ce-3.el7                    docker-ce-stable
docker-ce.x86_64            18.06.1.ce-3.el7                    docker-ce-stable
docker-ce.x86_64            18.06.0.ce-3.el7                    docker-ce-stable
docker-ce.x86_64            18.03.1.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            18.03.0.ce-1.el7.centos             docker-ce-stable

Kubernetes 1.15当前支持的docker版本列表是1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09。 这里在各节点安装docker的18.09.7版本。

yum makecache fast
 
yum install -y --setopt=obsoletes=0 \
  docker-ce-18.09.7-3.el7 
 
systemctl start docker
systemctl enable docker

确认一下iptables filter表中FOWARD链的默认策略(pllicy)为ACCEPT:

iptables -nvL
Chain INPUT (policy ACCEPT 263 packets, 19209 bytes)
 pkts bytes target     prot opt in     out     source               destination
 
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

修改docker cgroup driver为systemd

根据文档CRI installation中的内容，对于使用systemd作为init system的Linux的发行版，使用systemd作为docker的cgroup driver可以确保服务器节点在资源紧张的情况更加稳定，因此这里修改各个节点上docker的cgroup driver为systemd。

创建或修改/etc/docker/daemon.json：

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

重启docker：

systemctl restart docker
 
docker info | grep Cgroup
Cgroup Driver: systemd

使用kubeadm部署Kubernetes

安装kubeadm和kubelet

下面在各节点安装kubeadm和kubelet：

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
 
 
#由于国内网络不通问题，推荐使用以下阿里云镜像
cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum makecache fast
#默认可能安装的版本不一样
yum install -y kubelet kubeadm kubectl
#建议指定版本
yum install -y kubelet-1.15.0 kubeadm-1.15.0 kubectl-1.15.0

Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。 关闭系统的Swap方法如下:

swapoff -a

永久关闭swap：

修改 /etc/fstab 文件，注释掉 SWAP 的自动挂载，使用free -m确认swap已经关闭。

swappiness参数调整，修改/etc/sysctl.d/k8s.conf添加下面一行,前面已经统一加过了：

vm.swappiness=0

执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。

关闭swap可能会对其他服务产生影响，所以这里修改kubelet的配置去掉这个限制。
使用kubelet的启动参数–fail-swap-on=false去掉必须关闭Swap的限制，修改/etc/sysconfig/kubelet，加入：
KUBELET_EXTRA_ARGS=--fail-swap-on=false

使用kubeadm init初始化集群

在各节点开机启动kubelet服务：

systemctl enable kubelet.service

使用kubeadm config print init-defaults可以打印集群初始化默认的使用的配置：

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: node1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.14.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

从默认的配置中可以看到，可以使用imageRepository定制在集群初始化时拉取k8s所需镜像的地址。基于默认配置定制出本次使用kubeadm初始化集群所需的配置文件kubeadm.yaml：

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.41.129
  bindPort: 6443
nodeRegistration:
  taints:
  - effect: PreferNoSchedule
    key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.15.0
networking:
  podSubnet: 10.244.0.0/16

在开始初始化集群之前可以使用

kubeadm config images pull

预先在各个节点上拉取所k8s需要的docker镜像。

由于国内网络问题，k8s.gcr.io 资源获取不了,使用以下方法

#!/bin/bash
MY_REGISTRY=gcr.azk8s.cn/google-containers
 
## 拉取镜像
docker pull ${MY_REGISTRY}/kube-apiserver:v1.15.0
docker pull ${MY_REGISTRY}/kube-controller-manager:v1.15.0
docker pull ${MY_REGISTRY}/kube-scheduler:v1.15.0
docker pull ${MY_REGISTRY}/kube-proxy:v1.15.0
docker pull ${MY_REGISTRY}/pause:3.1
docker pull ${MY_REGISTRY}/etcd:3.3.10
docker pull ${MY_REGISTRY}/coredns:1.3.1
 
## 添加Tag
docker tag ${MY_REGISTRY}/kube-apiserver:v1.15.0 k8s.gcr.io/kube-apiserver:v1.15.0
docker tag ${MY_REGISTRY}/kube-controller-manager:v1.15.0 k8s.gcr.io/kube-controller-manager:v1.15.0
docker tag ${MY_REGISTRY}/kube-scheduler:v1.15.0 k8s.gcr.io/kube-scheduler:v1.15.0
docker tag ${MY_REGISTRY}/kube-proxy:v1.15.0 k8s.gcr.io/kube-proxy:v1.15.0
docker tag ${MY_REGISTRY}/pause:3.1 k8s.gcr.io/pause:3.1
docker tag ${MY_REGISTRY}/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag ${MY_REGISTRY}/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
 
#删除无用的镜像
docker images | grep ${MY_REGISTRY} | awk '{print "docker rmi "  $1":"$2}' | sh -x
 
echo "end"

上面的所有操作可以在一个节点上面完成，然后对进行复制即可。

其它因为网络原因pull的镜像都可以同理先下载下来:

#1.先使用docker search命令查找
 
#2.选择一个下载快的镜像使用docker pull下载下来
 
#3.使用docker tag命令修改成程序需要的镜像
 
#4.使用docker rmi删除之前的镜像
 
#下载flannel网络镜像
docker pull xmlgrg/flannel:v0.11.0-amd64
docker tag docker.io/xmlgrg/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
docker rmi docker.io/xmlgrg/flannel:v0.11.0-amd64
 
#下载tiller
docker pull junolu/tiller:v2.14.3
docker tag docker.io/junolu/tiller:v2.14.3 gcr.io/kubernetes-helm/tiller:v2.14.3
docker rmi docker.io/junolu/tiller:v2.14.3
 
#下载ingress-nginx/defaultbackend-amd64
docker pull fungitive/defaultbackend-amd64
docker tag docker.io/fungitive/defaultbackend-amd64:latest k8s.gcr.io/defaultbackend-amd64:1.5
docker rmi docker.io/fungitive/defaultbackend-amd64:latest
 
#下载dashboard
docker.io/loveone/kubernetes-dashboard-amd64:v1.10.1
docker tag docker.io/loveone/kubernetes-dashboard-amd64:v1.10.1 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
docker  rmi docker.io/loveone/kubernetes-dashboard-amd64:v1.10.1

接下来使用kubeadm初始化集群，选择node2作为Master Node，在node2上执行下面的命令：

kubeadm init --config kubeadm.yaml --ignore-preflight-errors=Swap
[init] Using Kubernetes version: v1.15.0
[preflight] Running pre-flight checks
    [WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [node1 localhost] and IPs [192.168.99.11 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [node1 localhost] and IPs [192.168.99.11 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.99.11]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 26.004907 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node node1 as control-plane by adding the taints [node-role.kubernetes.io/master:PreferNoSchedule]
[bootstrap-token] Using token: 4qcl2f.gtl3h8e5kjltuo0r
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
 
Your Kubernetes control-plane has initialized successfully!
 
To start using your cluster, you need to run the following as a regular user:
 
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
 
Then you can join any number of worker nodes by running the following on each as root:
 
kubeadm join 192.168.99.11:6443 --token 4qcl2f.gtl3h8e5kjltuo0r \
    --discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e

上面记录了完成的初始化输出的内容，根据输出的内容基本上可以看出手动初始化安装一个Kubernetes集群所需要的关键步骤。 其中有以下关键内容：

  • [kubelet-start] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”
  • [certs]生成相关的各种证书
  • [kubeconfig]生成相关的kubeconfig文件
  • [control-plane]使用/etc/kubernetes/manifests目录中的yaml文件创建apiserver、controller-manager、scheduler的静态pod
  • [bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
  • 最后给出了将节点加入集群的命令kubeadm join 18.16.202.163:6443 --token jrts59.18pe12atfafgcxca \ --discovery-token-ca-cert-hash sha256:56d6c7d7b63a9109444ece68a1b155d8a9ac049ba57febab2c72d40d8ab7d426

下面的命令是配置常规用户如何使用kubectl访问集群：务必配上，不然无法使用kubectl命令！

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

集群初始化如果遇到问题，可以使用下面的命令进行清理：

kubeadm reset

遇到的问题：The connection to the server localhost:8080 was refused - did you specify the right host or port?
初始化的集群，kube-apiserver没有监听默认的http 8080端口。

查看kube-apiserver的监听端口可以看到只监听了https的6443端口:

netstat -nltp | grep apiserver
tcp6       0      0 :::6443                 :::*                    LISTEN      9831/kube-apiserver

为了使用kubectl访问apiserver，在~/.bash_profile中追加下面的环境变量：

export KUBECONFIG=/etc/kubernetes/admin.conf
source ~/.bash_profile

此时kubectl命令在master node上就好用了.

安装Pod Network

使用flannel网络

接下来安装flannel network add-on：

mkdir -p ~/k8s/
cd ~/k8s
curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f  kube-flannel.yml
 
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

这里注意kube-flannel.yml这个文件里的flannel的镜像是0.11.0，quay.io/coreos/flannel:v0.11.0-amd64.
由于网络问题，flannel镜像可能pull失败，建议参考之前拉取镜像的方法，提前再各个节点拉好镜像！

如果Node有多个网卡的话，参考flannel issues 39701，目前需要在kube-flannel.yml中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上–iface=<iface-name>

containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.11.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=eth1
# ......

使用kubectl get pod –all-namespaces -o wide确保所有的Pod都处于Running状态。

kubectl get pod -n kube-system
NAME                            READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-dr8lf        1/1     Running   0          52m
coredns-5c98db65d4-lp8dg        1/1     Running   0          52m
etcd-node1                      1/1     Running   0          51m
kube-apiserver-node1            1/1     Running   0          51m
kube-controller-manager-node1   1/1     Running   0          51m
kube-flannel-ds-amd64-mm296     1/1     Running   0          44s
kube-proxy-kchkf                1/1     Running   0          52m
kube-scheduler-node1            1/1     Running   0          51m

问题，flannel一直在CrashLoopBackOff，日志如下

[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                             READY   STATUS              RESTARTS   AGE
kube-system   coredns-5c98db65d4-57zm5         0/1     ContainerCreating   0          7m46s
kube-system   coredns-5c98db65d4-g8kqh         0/1     ContainerCreating   0          7m46s
kube-system   etcd-master                      1/1     Running             0          6m51s
kube-system   kube-apiserver-master            1/1     Running             0          6m31s
kube-system   kube-controller-manager-master   1/1     Running             0          6m29s
kube-system   kube-flannel-ds-amd64-jthsp      0/1     CrashLoopBackOff    3          104s

[root@master ~]# kubectl logs kube-flannel-ds-amd64-jthsp -n kube-system
I1105 08:37:34.863407       1 main.go:527] Using interface with name eth0 and address 192.168.1.209
I1105 08:37:34.863461       1 main.go:544] Defaulting external address to interface address (192.168.1.209)
I1105 08:37:35.061919       1 kube.go:126] Waiting 10m0s for node controller to sync
I1105 08:37:35.166015       1 kube.go:309] Starting kube subnet manager
I1105 08:37:36.065963       1 kube.go:133] Node controller sync successful
I1105 08:37:36.065985       1 main.go:244] Created subnet manager: Kubernetes Subnet Manager - master
I1105 08:37:36.065989       1 main.go:247] Installing signal handlers
I1105 08:37:36.066082       1 main.go:386] Found network config - Backend type: vxlan
I1105 08:37:36.066143       1 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
E1105 08:37:36.066314       1 main.go:289] Error registering network: failed to acquire lease: node "master" pod cidr not assigned
I1105 08:37:36.066356       1 main.go:366] Stopping shutdownHandler...

解决办法:

# 修改 
/etc/kubernetes/manifests/kube-controller-manager.yaml
# 增加参数
--allocate-node-cidrs=true
--cluster-cidr=10.244.0.0/16
# 然后重启kubelet

使用weave网络

虽然系统显示安装成功，但是master节点显示状态还是 NotReady，原因是需要安装网络插件，这边kubernetes官方提供了几款网络插件。。

kubectl apply -f https://git.io/weave-kube-1.6
NAME                      STATUS    ROLES     AGE       VERSION
izj6c3vsekthy7xn94b5s6z   Ready     master    1h        v1.10.3

大概过一会master节点就显示ready了，说明master已经准备就绪了.

测试集群DNS是否可用

kubectl run curl --image=radial/busyboxplus:curl -it
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
If you don't see a command prompt, try pressing enter.
[ root@curl-5cc7b478b6-r997p:/ ]$

进入后执行nslookup kubernetes.default确认解析正常:

nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
 
Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

向Kubernetes集群中添加Node节点

下面将node3这个主机添加到Kubernetes集群中，在node3上执行:

kubeadm join 192.168.99.11:6443 --token 4qcl2f.gtl3h8e5kjltuo0r \
    --discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e \
 --ignore-preflight-errors=Swap
 
[preflight] Running pre-flight checks
    [WARNING Swap]: running with swap on is not supported. Please disable swap
    [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
 
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
 
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

node3加入集群后，下面在master节点上执行命令查看集群中的节点：

kubectl get node
NAME    STATUS   ROLES    AGE   VERSION
node2   Ready    master   57m   v1.15.0
node3   Ready       11s   v1.15.0

如果只想搭建一个只有master节点的单机测试环境

单机版kubernetes为了运行Pod.需要删除主机上的Train.允许master执行Pod.
执行命令如下：

kubectl taint nodes --all node-role.kubernetes.io/master-

遇到的问题

执行上述命令一直卡在了
[preflight] Running pre-flight checks

  • 问题可能的原因是master防火墙没有关掉，检查firewalld和iptables
  • 加上 –v=2 命令具体看看

从集群中移除Node

在master节点上执行：

kubectl drain node2 --delete-local-data --force --ignore-daemonsets
kubectl delete node node2

在Node上执行：

kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/

kube-proxy开启ipvs

修改ConfigMap的kube-system/kube-proxy中的config.conf，mode: "ipvs"

kubectl edit cm kube-proxy -n kube-system

之后重启各个节点上的kube-proxy pod：

kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'

kubectl get pod -n kube-system | grep kube-proxy
kube-proxy-7fsrg                1/1     Running   0          3s
kube-proxy-k8vhm                1/1     Running   0          9s
 
kubectl logs kube-proxy-7fsrg  -n kube-system
I0703 04:42:33.308289       1 server_others.go:170] Using ipvs Proxier.
W0703 04:42:33.309074       1 proxier.go:401] IPVS scheduler not specified, use rr by default
I0703 04:42:33.309831       1 server.go:534] Version: v1.15.0
I0703 04:42:33.320088       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0703 04:42:33.320365       1 config.go:96] Starting endpoints config controller
I0703 04:42:33.320393       1 controller_utils.go:1029] Waiting for caches to sync for endpoints config controller
I0703 04:42:33.320455       1 config.go:187] Starting service config controller
I0703 04:42:33.320470       1 controller_utils.go:1029] Waiting for caches to sync for service config controller
I0703 04:42:33.420899       1 controller_utils.go:1036] Caches are synced for endpoints config controller
I0703 04:42:33.420969       1 controller_utils.go:1036] Caches are synced for service config controller

日志中打印出了Using ipvs Proxier，说明ipvs模式已经开启。

Kubernetes常用组件部署

越来越多的公司和团队开始使用Helm这个Kubernetes的包管理器，这里也将使用Helm安装Kubernetes的常用组件。

Helm的安装

Helm由客户端命helm令行工具和服务端tiller组成，Helm的安装十分简单。 下载helm命令行工具到master节点node1的/usr/local/bin下，这里下载的2.14.1版本.

网络问题下载不了，可以直接在网上搜个压缩包拷进去即可！

curl -O https://get.helm.sh/helm-v2.14.1-linux-amd64.tar.gz
tar -zxvf helm-v2.14.1-linux-amd64.tar.gz
cd linux-amd64/
cp helm /usr/local/bin/

为了安装服务端tiller，还需要在这台机器上配置好kubectl工具和kubeconfig文件，确保kubectl工具可以在这台机器上访问apiserver且正常使用。

因为Kubernetes APIServer开启了RBAC访问控制，所以需要创建tiller使用的service account: tiller并分配合适的角色给它。 详细内容可以查看helm文档中的Role-based Access Control。 这里简单起见直接分配cluster-admin这个集群内置的ClusterRole给它。创建helm-rbac.yaml文件：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

kubectl create -f helm-rbac.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created

接下来使用helm部署tiller:

helm init --service-account tiller --skip-refresh
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
 
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
 
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!

tiller默认被部署在k8s集群中的kube-system这个namespace下：

kubectl get pod -n kube-system -l app=helm
NAME                            READY   STATUS    RESTARTS   AGE
tiller-deploy-c4fd4cd68-dwkhv   1/1     Running   0          83s

helm version
Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}

遇到的问题!!!!

tiller可能因为网络问题安装失败，可以使用以下命令重置

helm reset --force

然后安装tiller的时候指定image和stable url，这里用的阿里云的

helm init --service-account tiller --skip-refresh -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.14.3 --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

最后在node1上修改helm chart仓库的地址为azure提供的镜像地址：

helm repo add stable http://mirror.azure.cn/kubernetes/charts
"stable" has been added to your repositories
 
helm repo list
NAME    URL                                     
stable  http://mirror.azure.cn/kubernetes/charts
local   http://127.0.0.1:8879/charts

使用Helm部署Nginx Ingress

为了便于将集群中的服务暴露到集群外部，需要使用Ingress。接下来使用Helm将Nginx Ingress部署到Kubernetes上。

Nginx Ingress Controller被部署在Kubernetes的边缘节点上

我们将node1(192.168.99.11)做为边缘节点，打上Label：

kubectl label node node1 node-role.kubernetes.io/edge=
node/node1 labeled
 
 
kubectl get node
NAME    STATUS   ROLES         AGE    VERSION
node1   Ready    edge,master   138m   v1.15.0
node2   Ready            82m    v1.15.0

stable/nginx-ingress chart的值文件ingress-nginx.yaml如下：

controller:
  replicaCount: 1
  hostNetwork: true
  nodeSelector:
    node-role.kubernetes.io/edge: ''
  affinity:
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: PreferNoSchedule
defaultBackend:
  nodeSelector:
    node-role.kubernetes.io/edge: ''
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: PreferNoSchedule

nginx ingress controller的副本数replicaCount为1，将被调度到node1这个边缘节点上。

这里并没有指定nginx ingress controller service的externalIPs，而是通过hostNetwork: true设置nginx ingress controller使用宿主机网络。

helm repo update
 
# helm安装指定chart版本 --version
helm install stable/nginx-ingress \
-n nginx-ingress \
--namespace ingress-nginx  \
-f ingress-nginx.yaml --verion 1.27.1

kubectl get pod -n ingress-nginx -o wide
NAME                                            READY   STATUS    RESTARTS   AGE   IP              NODE    NOMINATED NODE   READINESS GATES
nginx-ingress-controller-cc9b6d55b-pr8vr        1/1     Running   0          10m   192.168.99.11   node1              
nginx-ingress-default-backend-cc888fd56-bf4h2   1/1     Running   0          10m   10.244.0.14     node1

如果访问http://192.168.99.11返回default backend，则部署完成。

遇到的问题!!!!

这里可能因为网络原因安装失败需要清理之前的nginx-ingress，建议提前下载好相关镜像。
使用kubectl describe pod [name] -n [namespace]看看默认拉取的是哪个版本的镜像，参考之前方法提前下载！

kubectl delete namespace/ingress-nginx
helm delete nginx-ingress
helm del --purge nginx-ingress

使用Helm部署dashboard

创建tls secret：

通过https进行访问必需要使用证书和密钥，在Kubernetes中可以通过配置一个加密凭证（TLS secret）来提供。创建一个自己签名的证书:

openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout ./tls.key -out ./tls.crt -subj "/CN=192.168.1.210"

将会产生两个文件tls.key和tls.crt，你可以改成自己的文件名或放在特定的目录下。后面的192.168.1.210是我的服务器IP地址，你需要改成自己的。

安装tls secret:

将这两个文件的信息创建为一个Kubernetes的secret访问凭证，我将名称指定为test-com-tls-secret，这在后面的Ingress配置时将会用到。

如果你修改了这个名字，注意后面的配置yaml文件也需要同步修改。

kubectl -n kube-system create secret tls test-com-tls-secret --key ./tls.key --cert ./tls.crt

查看:

kubectl get secret -n kube-system |grep test

kubernetes-dashboard.yaml:

注意修改里面的secretName和hosts

image:
  repository: k8s.gcr.io/kubernetes-dashboard-amd64
  tag: v1.10.1
ingress:
  enabled: true
  hosts: 
    - k8s.test.com
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  tls:
    - secretName: test-com-tls-secret
      hosts:
      - k8s.test.com
nodeSelector:
    node-role.kubernetes.io/edge: ''
tolerations:
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: PreferNoSchedule
rbac:
  clusterAdminRole: true

安装：

helm install stable/kubernetes-dashboard \
-n kubernetes-dashboard \
--namespace kube-system  \
-f kubernetes-dashboard.yaml

查看：

kubectl -n kube-system get secret | grep kubernetes-dashboard-token
kubernetes-dashboard-token-pkm2s                 kubernetes.io/service-account-token   3      3m7s
 
kubectl describe -n kube-system secret/kubernetes-dashboard-token-pkm2s
Name:         kubernetes-dashboard-token-pkm2s
Namespace:    kube-system
Labels:       
Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard
              kubernetes.io/service-account.uid: 2f0781dd-156a-11e9-b0f0-080027bb7c43
 
Type:  kubernetes.io/service-account-token
 
Data
====
ca.crt:     1025 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1wa20ycyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjJmMDc4MWRkLTE1NmEtMTFlOS1iMGYwLTA4MDAyN2JiN2M0MyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.24ad6ZgZMxdydpwlmYAiMxZ9VSIN7dDR7Q6-RLW0qC81ajXoQKHAyrEGpIonfld3gqbE0xO8nisskpmlkQra72-9X6sBPoByqIKyTsO83BQlME2sfOJemWD0HqzwSCjvSQa0x-bUlq9HgH2vEXzpFuSS6Svi7RbfzLXlEuggNoC4MfA4E2hF1OX_ml8iAKx-49y1BQQe5FGWyCyBSi1TD_-ZpVs44H5gIvsGK2kcvi0JT4oHXtWjjQBKLIWL7xxyRCSE4HmUZT2StIHnOwlX7IEIB0oBX4mPg2_xNGnqwcu-8OERU9IoqAAE2cZa0v3b5O2LMcJPrcxrVOukvRIumA

测试：

在dashboard的登录窗口使用上面的token登录，访问刚刚填写的hosts，如果是内网域名，需要在访问的那台机器上，在hosts文件中加入刚刚的域名和IP。
k8s

使用Helm部署metrics-server

heapster从Kubernetes 1.12开始从Kubernetes各种安装脚本中移除。

Kubernetes推荐使用metrics-server。我们这里也使用helm来部署metrics-server。

metrics-server.yaml:

replicaCount: 1
image:
    repository: hub.deri.org.cn/k8s/metrics-server-amd64
    tag: v0.3.5
    pullPolicy: IfNotPresent
args:
- --logtostderr
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
nodeSelector:
    node-role.kubernetes.io/edge: ''
tolerations:
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: PreferNoSchedule

helm install stable/metrics-server \
-n metrics-server \
--namespace kube-system \
-f metrics-server.yaml

使用下面的命令可以获取到关于集群节点基本的指标信息：

kubectl top node
NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node1   650m         32%    1276Mi          73%
node2   73m          3%     527Mi           30%

kubectl top pod -n kube-system
NAME                                    CPU(cores)   MEMORY(bytes)   
coredns-5c98db65d4-dr8lf                8m           7Mi             
coredns-5c98db65d4-lp8dg                6m           8Mi             
etcd-node1                              44m          46Mi            
kube-apiserver-node1                    74m          295Mi           
kube-controller-manager-node1           35m          50Mi            
kube-flannel-ds-amd64-7lwm9             2m           8Mi             
kube-flannel-ds-amd64-mm296             5m           9Mi             
kube-proxy-7fsrg                        1m           11Mi            
kube-proxy-k8vhm                        3m           11Mi            
kube-scheduler-node1                    8m           15Mi            
kubernetes-dashboard-848b8dd798-c4sc2   2m           14Mi            
metrics-server-8456fb6676-fwh2t         10m          19Mi            
tiller-deploy-7bf78cdbf7-9q94c          1m           16Mi

参考链接