背景

helm 安裝jhub時超時

lynx@n149-136-019:~/magellan_docker/magellan_k8s/k8s_dev/k8s_jhub$ helm upgrade --install jhub jupyterhub/jupyterhub \
>   --namespace jhub  \
>   --version=0.9.0 \
>   --values k8s_config.yaml
Release "jhub" does not exist. Installing it now.
Error: timed out waiting for the condition

查看

kubectl get pod --all-namespaces

lynx@n149-136-019:/root$ kubectl get pod --all-namespaces
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
jhub                   hook-image-awaiter-nxf8v                     1/1     Running             0          5m43s
jhub                   hook-image-puller-5ljjj                      0/1     Init:0/2            0          5m51s
jhub                   hook-image-puller-j95bf                      0/1     Init:0/2            0          5m51s
jhub                   hook-image-puller-nblf5                      0/1     Init:0/2            0          5m51s
jhub                   hook-image-puller-wk2vp                      1/1     Running             0          5m51s
kube-system            coredns-66bff467f8-kx4ck                     1/1     Running             0          117m
kube-system            coredns-66bff467f8-v9wcb                     1/1     Running             0          117m
kube-system            etcd-n149-136-019                            1/1     Running             0          117m
kube-system            kube-apiserver-n149-136-019                  1/1     Running             0          117m
kube-system            kube-controller-manager-n149-136-019         1/1     Running             0          117m
kube-system            kube-flannel-ds-4dsqt                        1/1     Running             0          79m
kube-system            kube-flannel-ds-5g2p5                        1/1     Running             0          102m
kube-system            kube-flannel-ds-6ldvm                        1/1     Running             0          80m
kube-system            kube-flannel-ds-9ssh9                        1/1     Running             0          78m
kube-system            kube-flannel-ds-sxhxd                        1/1     Running             0          81m
kube-system            kube-proxy-6nhc8                             0/1     ImagePullBackOff    0          79m
kube-system            kube-proxy-8nqfc                             0/1     ImagePullBackOff    0          81m
kube-system            kube-proxy-l55gh                             0/1     ImagePullBackOff    0          78m
kube-system            kube-proxy-m27q7                             0/1     ImagePullBackOff    0          80m
kube-system            kube-proxy-qq899                             1/1     Running             0          117m
kube-system            kube-scheduler-n149-136-019                  1/1     Running             0          117m
kube-system            tiller-deploy-5d58456765-dpkbm               1/1     Running             0          16m
kubernetes-dashboard   dashboard-metrics-scraper-6b4884c9d5-nspnm   1/1     Running             0          78m
kubernetes-dashboard   kubernetes-dashboard-7b544877d5-tzdxg        0/1     ContainerCreating   0          78m

定位錯誤

• 查看日志

kubectl describe pod hook-image-puller-5ljjj -n jhub

Events:
  Type     Reason                  Age                     From               Message
  ----     ------                  ----                    ----               -------
  Normal   Scheduled               6m59s                   default-scheduler  Successfully assigned jhub/hook-image-puller-5ljjj to n130-072-042
  Warning  FailedCreatePodSandBox  6m56s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2bf2fb6f9344934744ee9c87465ca210278af14eb26d433043eca0822cfb92de" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
  Warning  FailedCreatePodSandBox  6m53s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7d161f41f7829e44140cacd8faf6f857fa48d31a3db14607a1c0e40748eaf30c" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
  Warning  FailedCreatePodSandBox  6m51s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c820f8456ce718645d757cff8092cab3ee0b44f36b2f05450a40f41b43e41221" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
  Warning  FailedCreatePodSandBox  6m49s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "21d1ac2870454f2ffdb1c1dc838753c32af41e53989cf45c48612eca714aa589" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
  Warning  FailedCreatePodSandBox  6m46s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "be549b17fcec87e6861ed0b1e36e6642c6f2f66addfa96d3d8ddee56c47fe36a" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
  Warning  FailedCreatePodSandBox  6m44s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "365d8545d30979b1c3cc27518873fbfe403395f39c0d26cf07c3ae234eb921ec" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
  Warning  FailedCreatePodSandBox  6m42s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "47f79483aebfa9a86355841d98951c1fed0e882f6b403169066ca657a58e0af5" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
  Warning  FailedCreatePodSandBox  6m40s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7e65c5ff153c2bda95db271fae704660e49a9700b366d684a986da101aa81e8f" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
  Warning  FailedCreatePodSandBox  6m38s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a93392dcc3ea288db5db2d4851076e3bf3378280788f5623919707b0da1e37fc" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24
  Normal   SandboxChanged          6m31s (x12 over 6m56s)  kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  113s (x120 over 6m36s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9c7388a6ce257bed5bb7ad4218e14a53f43a5086978f42878e69746c98667335" network for pod "hook-image-puller-5ljjj": networkPlugin cni failed to set up pod "hook-image-puller-5ljjj_jhub" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24

原因

幾個node節點加入了多次
期間沒有沒有重啟到docker，導致原先的網絡與現有的網絡沖突

解決

ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1

systemctl restart docker