티스토리 뷰

Cloud/Private Cloud

occur QEMU error when volume attach

jacobbaek Jacob_baek 2017. 3. 14. 16:01

아래의 경우는 ceph2.1에 packstack 환경으로 일반적이지 않을수 있으나 참고할 필요는 있다.


OpenStack과 Ceph를 연동한 환경에서 아래와 같이 볼륨 attach시에 error 가발생하는 경우가 있다.

실제에러는 cinder-volume log에서 발생된다.

instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf] Attaching volume 8b4088ae-eb34-4076-bea0-7ad5c250d5de to /dev/vdb

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [req-a5e2fc3f-551e-4bb4-8217-8ed28d0706ce 06b90330d8e24091b0a06c121176c882 4f1ed33b6f22471596cf499fb87eae5a - - -] [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf] Failed to attach volume at mountpoint: /dev/vdb

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf] Traceback (most recent call last):

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     guest.attach_device(conf, persistent=True, live=live)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     result = proxy_call(self._autowrap, f, *args, **kwargs)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     rv = execute(f, *args, **kwargs)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     six.reraise(c, e, tb)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     rv = meth(*args, **kwargs)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf] libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could not be initialized

2017-02-15 12:41:02.757 31185 ERROR nova.virt.libvirt.driver [instance: cc8e51fb-830e-4e25-8434-06fcc3deeaaf]


이와 함께 libvirt 로그 상에는 다음과 같은 에러로그가 출력된다.

error reading header from volume-8b4088ae-eb34-4076-bea0-7ad5c250d5de: Function not implemented

error reading header from volume-3cb07ffc-3b6f-4ded-841c-42d9a30ff1e5: Function not implemented

error reading header from volume-8b4088ae-eb34-4076-bea0-7ad5c250d5de: Function not implemented

error reading header from volume-8b4088ae-eb34-4076-bea0-7ad5c250d5de: Function not implemented

error reading header from volume-8b4088ae-eb34-4076-bea0-7ad5c250d5de: Function not implemented


당시 ceph의 volume 정보를 확인해보면 format이 2 이고 feature가 layering 으로 되어 있음을 확인할 수 있다.

[root@com02 ~]# rbd info volumes/volume-c46fbf21-babf-47d7-95a1-f43ffc1322a0

rbd image 'volume-c46fbf21-babf-47d7-95a1-f43ffc1322a0':

        size 10240 MB in 2560 objects

        order 22 (4096 kB objects)

        block_name_prefix: rbd_data.10c282e9b53db

        format: 2

        features: layering

        flags:


해결방법은 다음 설정을 ceph.conf 상에 추가한다.

실제 아래 설정은 nova-compute 및 cinder가 동작하는 노드에 수정하면 된다.

  [client]

  rbd default features = 1


참고로 rbd feature의 number는 다음과 같은 뜻을 의미한다.

* 1 : layering

* 2 : striping 

* 4 : exclusive-lock

* 8 : object-map 

- 참조 : https://www.sebastien-han.fr/blog/2015/07/06/ceph-enable-the-object-map-feature/





댓글
댓글쓰기 폼