docker client running fail when try to scale out using kubespray

이슈사항

kubespray를 이용한 worker node scaleout시에 실패가 발생되는 현상
아래와 같은 docker client의 버전 이슈가 발생됨.

RUNNING HANDLER [container-engine/docker : Docker | wait for docker] ******************************************************************************************************************************************************************
Wednesday 20 May 2020  00:08:17 +0000 (0:00:01.264)       0:05:48.031 ********* 
FAILED - RETRYING: Docker | wait for docker (20 retries left).
FAILED - RETRYING: Docker | wait for docker (19 retries left).
FAILED - RETRYING: Docker | wait for docker (18 retries left).
FAILED - RETRYING: Docker | wait for docker (17 retries left).
FAILED - RETRYING: Docker | wait for docker (16 retries left).
FAILED - RETRYING: Docker | wait for docker (15 retries left).
FAILED - RETRYING: Docker | wait for docker (14 retries left).
FAILED - RETRYING: Docker | wait for docker (13 retries left).
FAILED - RETRYING: Docker | wait for docker (12 retries left).
FAILED - RETRYING: Docker | wait for docker (11 retries left).
FAILED - RETRYING: Docker | wait for docker (10 retries left).
FAILED - RETRYING: Docker | wait for docker (9 retries left).
FAILED - RETRYING: Docker | wait for docker (8 retries left).
FAILED - RETRYING: Docker | wait for docker (7 retries left).
FAILED - RETRYING: Docker | wait for docker (6 retries left).
FAILED - RETRYING: Docker | wait for docker (5 retries left).
FAILED - RETRYING: Docker | wait for docker (4 retries left).
FAILED - RETRYING: Docker | wait for docker (3 retries left).
FAILED - RETRYING: Docker | wait for docker (2 retries left).
FAILED - RETRYING: Docker | wait for docker (1 retries left).
fatal: [node5]: FAILED! => {"attempts": 20, "changed": true, "cmd": ["/usr/bin/docker", "images"], "delta": "0:00:00.098703", "end": "2020-05-20 00:08:45.699406", "msg": "non-zero return code", "rc": 1, "start": "2020-05-20 00:08:45.600703", "stderr": "Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39", "stderr_lines": ["Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39"], "stdout": "", "stdout_lines": []}

실제 이슈 사항은 아래처럼 docker client 버전의 문제

[root@worker_new ~]# docker version
Client: Docker Engine - Community
 Version:           19.03.9
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        9d988398e7
 Built:             Fri May 15 00:25:27 2020
 OS/Arch:           linux/amd64
 Experimental:      false
Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39

이슈환경

[root@worker_new ~]# rpm -qa | grep docker
docker-ce-cli-19.03.9-3.el7.x86_64
docker-ce-18.09.7-3.el7.x86_64

정상 동작하는 환경

[root@worker_old ~]# rpm -qa | grep docker
docker-ce-18.09.7-3.el7.x86_64
docker-ce-cli-19.03.8-3.el7.x86_64

해결방법

아래와 같이 2020.05.19일 관련 이슈 논의가 있었고 개선작업이 진행중

• https://github.com/kubernetes-sigs/kubespray/issues/6160

하여 현재 해결은 downgrade를 수행해야 하는 상황

[root@worker_new ~]# yum downgrade docker-ce-cli
Resolving Dependencies
--> Running transaction check
---> Package docker-ce-cli.x86_64 1:19.03.8-3.el7 will be a downgrade
---> Package docker-ce-cli.x86_64 1:19.03.9-3.el7 will be erased

위와 같은 버전을 확인하고 19.03.8로 downgrade하여 사용하여야 이슈가 재현되지 않음.