Happywzy

istio代理接口503报错排查分析

发表于 2022-02-22 分类于 istio ， k8s

获取istio-proxy所有配置

kubectl -n test exec auhss-v1-769dc687ff-479r4 -c istio-proxy -- curl http://localhost:15000/config_dump > test.json

配置istio-proxy日志级别为debug

# -n 名空间
# server-v1-6dfd6b9846-c2ch6 容器名
kubectl -n mgec  exec server-v1-6dfd6b9846-c2ch6  -c istio-proxy -- curl -XPOST -s -o /dev/null http://localhost:15000/logging?level=debug
# 通过pod注解方式设置日志级别
template:
    metadata:
      annotations:
        sidecar.istio.io/logLevel: debug

查看Prometheus收集的istio指标

istio_requests_total{response_flags="UC", response_code="503", reporter="destination"}

Kubernetes存储Longhorn

发表于 2022-01-21 分类于 k8s

安装环境要求

# 运行这个命令检查环境
curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.2.3/scripts/environment_check.sh | bash
# 所有节点安装必要依赖
yum install iscsi-initiator-utils
yum install nfs-utils
yum install jq -y
# 使用helm安装Longhorn
helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace
kubectl -n longhorn-system get pod

查看charts默认配置：helm inspect values longhorn/longhorn
修改默认值：helm install longhorn longhorn/longhorn –namespace longhorn-system –create-namespace –set persistence.defaultClassReplicaCount=1

使用

直接在Longhorn-UI上操作使用

设置备份URL

longhorn备份PV目前支持s3和nfs，打开UI选择Setting，选择General，下拉选择Backup Target，填写NFS服务器地址，格式nfs://172.16.100.13:/data/nfs.

一般在机房或者云上使用ECS自建Kubernetes集群是无法使用LoadBalancer类型的Service。因为Kubernetes本身没有为裸机群集提供网络负载均衡器的实现。自建的Kubernetes集群暴露让外网访问，目前只能使用NodePort或Ingress等的方法进行服务暴露.
有没有方法，让自建的Kubernetes集群也能使用LoadBalancer类型的Service？这里使用OPENELB实现这个功能.

OPENELB部署

参考链接

OPENELB使用

参考链接

两种模式

Layer 2 模式
BGP 模式

华为云使用

网络控制台 -> 子网 -> 选择自己的子网 -> IP地址管理 -> 申请虚拟IP地址.
申请好虚拟IP地址后,就可以在OPENELB创建EIP池时填上申请的虚拟IP.

参考Layer2模式为service分配了IP后,在华为云上还只能在内网访问,如果想公网访问还需要为这个虚拟IP绑定公网IP并绑定服务器网卡.

nfs-client安装

发表于 2021-12-30 分类于 k8s

说明

nfs-client的chart已经迁移到https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/，但是该chart对应的镜像都放在k8s.gcr.io，国内不方便下载.

本文继续使用以前的chart.

安装步骤

添加repo

helm repo add stable https://charts.helm.sh/stable

更新repo

helm repo update

安装nfs-client

# 这里nfs server参考centos nfs安装
helm install nfs  stable/nfs-client-provisioner  --set nfs.server=172.16.100.13 --set nfs.path=/data/nfs

设置nfs为默认存储

[root@node1 ~]# kubectl get sc
NAME         PROVISIONER                                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client   cluster.local/nfs-nfs-client-provisioner   Delete          Immediate           true                   30s
[root@node1 ~]# kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/nfs-client patched
[root@node1 ~]# kubectl get sc
NAME                   PROVISIONER                                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   cluster.local/nfs-nfs-client-provisioner   Delete          Immediate           true                   118s

遇到的问题

k8s-v1.21.5版本安装完nfs-client，创建pvc会一直在pending状态,kubectl logs nfs-client-provisioner后台有报错

unexpected error getting claim reference: selfLink was empty, can't make reference

临时解决方法：
编辑/etc/kubernetes/manifests/kube-apiserver.yaml,在这里：

spec:
  containers:
  - command:
    - kube-apiserver

添加这一行：
- --feature-gates=RemoveSelfLink=false

然后应用它，即可

kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

解决方案
参考issue，升级nfs-client-provisioner镜像到v4.0以上版本.

# 一个可用的版本
easzlab/nfs-subdir-external-provisioner:v4.0.1

helm版本回退

发表于 2021-12-08 分类于 k8s

[root@master ~]# helm list -n  istio-system
NAME           	NAMESPACE   	REVISION	UPDATED                                	STATUS  	CHART                 	APP VERSION
jaeger-operator	istio-system	7       	2021-11-18 15:49:10.783153039 +0800 CST	deployed	jaeger-operator-2.26.0	1.27.0     
kiali-operator 	istio-system	4       	2021-11-18 15:36:24.945838387 +0800 CST	deployed	kiali-operator-1.38.1 	v1.38.1    
[root@master ~]# helm history -n istio-system jaeger-operator
REVISION	UPDATED                 	STATUS    	CHART                 	APP VERSION	DESCRIPTION    
1       	Wed Aug 25 13:39:38 2021	superseded	jaeger-operator-2.14.2	1.17.1     	Install complete              
2       	Mon Nov 15 11:11:58 2021	superseded	jaeger-operator-2.26.0	1.27.0     	Upgrade complete     
3       	Thu Nov 18 15:37:13 2021	superseded	jaeger-operator-2.14.2	1.17.1     	Rollback to 1      
[root@master ~]#  helm rollback -n istio-system jaeger-operator 2
Rollback was a success! Happy Helming!

centos系统盘扩容

发表于 2021-12-08 分类于 centos

查看当前系统盘信息

[root@master ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G     0  7.8G   0% /dev/shm
tmpfs                    7.8G  7.4M  7.8G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root   44G   32G   13G  72% /    # 注意这里只有13G了
/dev/sda1               1014M  168M  847M  17% /boot
tmpfs                    1.6G     0  1.6G   0% /run/user/0

查看未分配的存储

[root@master ~]# lsblk -l
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  200G  0 disk  #还有150G未分配
sda1          8:1    0    1G  0 part /boot
sda2          8:2    0   49G  0 part 
centos-root 253:0    0   44G  0 lvm  /
centos-swap 253:1    0    5G  0 lvm  
sr0          11:0    1 1024M  0 rom  
[root@master ~]# fdisk -l
Disk /dev/sda: 214.7 GB, 214748364800 bytes, 419430400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00039057
Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   104857599    51379200   8e  Linux LVM

对新增加的磁盘进行分区

[root@master ~]# fdisk /dev/sda 
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
# 1
Command (m for help): n
Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended
# 2
Select (default p): p
Partition number (3,4, default 3): 
First sector (104857600-419430399, default 104857600): 
Using default value 104857600
Last sector, +sectors or +size{K,M,G} (104857600-419430399, default 419430399): 
Using default value 419430399
Partition 3 of type Linux and of size 150 GiB is set
# 3
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

分区完后重启
分区完之后无法通过lsbfk等命令查看到,但是fdisk -l可以看到

[root@master ~]# fdisk -l
Disk /dev/sda: 214.7 GB, 214748364800 bytes, 419430400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00039057
Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   104857599    51379200   8e  Linux LVM
/dev/sda3       104857600   419430399   157286400   83  Linux

创建物理卷

[root@master ~]# pvcreate /dev/sda3
  Physical volume "/dev/sda3" successfully created.

查看物理卷信息

[root@master ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               centos
  PV Size               <49.00 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              12543
  Free PE               0
  Allocated PE          12543
  PV UUID               1BfiJn-wrKX-mDQa-Z43E-l3XT-KEki-A9tgAj
  "/dev/sda3" is a new physical volume of "150.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sda3
  VG Name               # 这里是空的
  PV Size               150.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               AKLFX6-6vhY-t38N-0FjV-r0WR-NXFG-3fafid
[root@master ~]# vgdisplay
  --- Volume group ---
  VG Name               centos #这里是系统盘的VG名
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <49.00 GiB
  PE Size               4.00 MiB
  Total PE              12543
  Alloc PE / Size       12543 / <49.00 GiB
  Free  PE / Size       0 / 0   
  VG UUID               TWyVcI-YK3A-SAhs-DQWd-0C6n-1PAZ-KumTJW

将新增加的分区/dev/sda3加入到根目录分区centos

[root@master ~]# vgextend centos /dev/sda3
  Volume group "centos" successfully extended

重新查看卷组信息

[root@master ~]# vgdisplay
  --- Volume group ---
  VG Name               centos
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               198.99 GiB
  PE Size               4.00 MiB
  Total PE              50942
  Alloc PE / Size       12543 / <49.00 GiB
  Free  PE / Size       38399 / <150.00 GiB #还有150G未分配
  VG UUID               TWyVcI-YK3A-SAhs-DQWd-0C6n-1PAZ-KumTJW

进行卷扩容

[root@master ~]# lvextend -l +100%FREE /dev/mapper/centos-root
  Size of logical volume centos/root changed from <44.00 GiB (11263 extents) to 193.99 GiB (49662 extents).
  Logical volume centos/root successfully resized.

调整卷分区大小

[root@master ~]# xfs_growfs /dev/mapper/centos-root
meta-data=/dev/mapper/centos-root isize=512    agcount=4, agsize=2883328 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=11533312, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=5631, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 11533312 to 50853888

最后再查看系统盘大小

[root@master ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G     0  7.8G   0% /dev/shm
tmpfs                    7.8G  5.0M  7.8G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root  194G   26G  169G  13% / #已经完成扩容
/dev/sda1               1014M  168M  847M  17% /boot
tmpfs                    1.6G     0  1.6G   0% /run/user/0