背景
helm 安装jhub时超时
lynx@n149-136-019:~/magellan_docker/magellan_k8s/k8s_dev/k8s_jhub$ helm upgrade --install jhub jupyterhub/jupyterhub \
> --namespace jhub \
> --version=0.9.0 \
> --values k8s_config.yaml
Release "jhub" does not exist. Installing it now.
Error: timed out waiting for the condition
查看
kubectl get pod --all-namespaces
lynx@n149-136-019:/root$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
jhub hook-image-awaiter-nxf8v 1/1 Running 0 5m43s
jhub hook-image-puller-5ljjj 0/1 Init:0/2 0 5m51s
jhub hook-image-puller-j95bf 0/1 Init:0/2 0 5m51s
jhub hook-image-puller-nblf5 0/1 Init:0/2 0 5m51s
jhub hook-image-puller-wk2vp 1/1 Running 0 5m51s
kube-system coredns-66bff467f8-kx4ck 1/1 Running 0 117m
kube-system coredns-66bff467f8-v9wcb 1/1 Running 0 117m
kube-system etcd-n149-136-019 1/1 Running 0 117m
kube-system kube-apiserver-n149-136-019 1/1 Running 0 117m
kube-system kube-controller-manager-n149-136-019 1/1 Running 0 117m
kube-system kube-flannel-ds-4dsqt 1/1 Running 0 79m
kube-system kube-flannel-ds-5g2p5 1/1 Running 0 102m
kube-system kube-flannel-ds-6ldvm 1/1 Running 0 80m
kube-system kube-flannel-ds-9ssh9 1/1 Running 0 78m
kube-system kube-flannel-ds-sxhxd 1/1 Running 0 81m
kube-system kube-proxy-6nhc8 0/1 ImagePullBackOff 0 79m
kube-system kube-proxy-8nqfc 0/1 ImagePullBackOff 0 81m
kube-system kube-proxy-l55gh 0/1 ImagePullBackOff 0 78m
kube-system kube-proxy-m27q7 0/1 ImagePullBackOff 0 80m
kube-system kube-proxy-qq899 1/1 Running 0 117m
kube-system kube-scheduler-n149-136-019 1/1 Running 0 117m
kube-system tiller-deploy-5d58456765-dpkbm 1/1 Running 0 16m
kubernetes-dashboard dashboard-metrics-scraper-6b4884c9d5-nspnm 1/1 Running 0 78m
kubernetes-dashboard kubernetes-dashboard-7b544877d5-tzdxg 0/1 ContainerCreating 0 78m
定位错误
kubectl describe pod hook-image-puller-5ljjj -n jhub
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m59s default-scheduler Successfully assigned jhub/hook-image-puller-5ljjj to n130-072-042
Warning FailedCreatePodSandBox 6m56s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2bf2fb6f9344934744ee9c87465ca210278af14eb26d433043eca0822cfb92de" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
Warning FailedCreatePodSandBox 6m53s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7d161f41f7829e44140cacd8faf6f857fa48d31a3db14607a1c0e40748eaf30c" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
Warning FailedCreatePodSandBox 6m51s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c820f8456ce718645d757cff8092cab3ee0b44f36b2f05450a40f41b43e41221" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
Warning FailedCreatePodSandBox 6m49s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "21d1ac2870454f2ffdb1c1dc838753c32af41e53989cf45c48612eca714aa589" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
Warning FailedCreatePodSandBox 6m46s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "be549b17fcec87e6861ed0b1e36e6642c6f2f66addfa96d3d8ddee56c47fe36a" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
Warning FailedCreatePodSandBox 6m44s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "365d8545d30979b1c3cc27518873fbfe403395f39c0d26cf07c3ae234eb921ec" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
Warning FailedCreatePodSandBox 6m42s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "47f79483aebfa9a86355841d98951c1fed0e882f6b403169066ca657a58e0af5" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
Warning FailedCreatePodSandBox 6m40s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7e65c5ff153c2bda95db271fae704660e49a9700b366d684a986da101aa81e8f" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
Warning FailedCreatePodSandBox 6m38s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a93392dcc3ea288db5db2d4851076e3bf3378280788f5623919707b0da1e37fc" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
Normal SandboxChanged 6m31s (x12 over 6m56s) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 113s (x120 over 6m36s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9c7388a6ce257bed5bb7ad4218e14a53f43a5086978f42878e69746c98667335" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
原因
几个node节点加入了多次
期间没有没有重启到docker，导致原先的网络与现有的网络冲突
解决
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
systemctl restart docker
提供CDN加速