autoscaling with heat

Heat

heat_template_version 을 최초 선언한다.

(http://docs.openstack.org/developer/heat/template_guide/hot_spec.html#hot-spec-template-version)

- 2013-05-23

- 2014-10-16

- 2015-04-30

- 2016-10-14 (newton version)


description 은 옵션이다.

resources 는 템플릿에서 중요한 섹션으로 다른 구성요소가 어디에 정의되어 있는지를 알려주기 때문이다.


아래 링크는 샘플 hot template를 생성하는 방법을 가르쳐준다.

http://docs.openstack.org/developer/heat/template_guide/hot_guide.html#writing-a-hello-world-hot-template





아래 링크는 hot template 의 구성과 관련된 설명을 해준다.

http://getcloudify.org/2015/05/20/openstack-summit-vancouver-cloud-network-orchestration-automation-heat-scaling.html

우선 아래의 heat template 샘플을 clone 받는다.

https://github.com/openstack/heat-templates/

stack 정보를 확인할 수 있다.

[stack@director9 ~]$ heat stack-list

WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead

+--------------------------------------+-------------+-----------------+---------------------+--------------+

| id                                   | stack_name  | stack_status    | creation_time       | updated_time |

+--------------------------------------+-------------+-----------------+---------------------+--------------+

| 258d7acb-71a8-4a07-ab3b-79e400939a88 | AutoScaling | CREATE_COMPLETE | 2017-03-23T03:27:51 | None         |

+--------------------------------------+-------------+-----------------+---------------------+--------------+

stack에 속한 resource를 확인할 수 있다.

[stack@director9 ~]$ heat resource-list 258d7acb-71a8-4a07-ab3b-79e400939a88

WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead

+-----------------------------+--------------------------------------+----------------------------+-----------------+---------------------+

| resource_name               | physical_resource_id                 | resource_type              | resource_status | updated_time        |

+-----------------------------+--------------------------------------+----------------------------+-----------------+---------------------+

| asg                         | 969546dc-1df4-49b2-9e8a-516c647aff88 | OS::Heat::AutoScalingGroup | CREATE_COMPLETE | 2017-03-23T03:27:51 |

| cpu_alarm_high              | b323ed3b-b68e-42da-aca4-ceaea7a4c0ae | OS::Ceilometer::Alarm      | CREATE_COMPLETE | 2017-03-23T03:27:51 |

| cpu_alarm_low               | d0877642-4cba-4a47-a08f-789f7cafb986 | OS::Ceilometer::Alarm      | CREATE_COMPLETE | 2017-03-23T03:27:51 |

| web_server_scaledown_policy | 09ae805d44954c88886f9292f538043d     | OS::Heat::ScalingPolicy    | CREATE_COMPLETE | 2017-03-23T03:27:51 |

| web_server_scaleup_policy   | eca299d167be467da534fe5603eb2078     | OS::Heat::ScalingPolicy    | CREATE_COMPLETE | 2017-03-23T03:27:51 |

+-----------------------------+--------------------------------------+----------------------------+-----------------+---------------------+

아래 command와 같이 cpu_alarm_low에 대한 resource 정보를 확인할 수 있다.

[stack@director9 ~]$ heat resource-show AutoScaling cpu_alarm_low

Ceilometer

heat는 austoscaling을 위한 ceilometer를 통한 report를 받게 된다. ceilometer는 다음과 같은 database scheme를 가진다.

항목	내용 및 결과
meter	아래와 같이 meter 정보는 각 resource에 대한 collect할 정보에 대한 속성값들이라 볼수 있다. [stack@director9 ~]$ ceilometer meter-list +---------------------------------+------------+-----------+-----------------------------------------------------------------------+----------------------------------+----------------------------------+ | Name                            | Type       | Unit      | Resource ID                                                           | User ID                          | Project ID                       | +---------------------------------+------------+-----------+-----------------------------------------------------------------------+----------------------------------+----------------------------------+ | network.outgoing.packets.rate   | gauge      | packet/s  | instance-00000003-1842fe5c-06ca-4d90-bb09-580162d2d386-tapbba66ada-b9 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 | | storage.containers.objects      | gauge      | object    | 9d81639eedb34d4e9d2b84dba410a648/glance                               | None                             | 9d81639eedb34d4e9d2b84dba410a648 | | storage.containers.objects.size | gauge      | B         | 9d81639eedb34d4e9d2b84dba410a648/glance                               | None                             | 9d81639eedb34d4e9d2b84dba410a648 | | storage.objects                 | gauge      | object    | 97265feaf29b42fdba6ee91a1e1e332b                                      | None                             | 97265feaf29b42fdba6ee91a1e1e332b |
Sample	아래와 같이 sample 정보는 각 meter에 매칭되는 실제 측정된 결과라 볼수 있다. (가장 최신 결과) [stack@director9 ~]$ ceilometer sample-list +--------------------------------------+-----------------------------------------------------------------------+---------------------------------+------------+----------------+-----------+----------------------------+ | ID                                   | Resource ID                                                           | Name                            | Type       | Volume         | Unit      | Timestamp                  | +--------------------------------------+-----------------------------------------------------------------------+---------------------------------+------------+----------------+-----------+----------------------------+ | 0d21650c-0f8b-11e7-b602-109836a4a7d7 | instance-00000018-425717c0-210c-47f8-8f9b-287b3a8d691a-tap59a952fd-e5 | network.incoming.packets        | cumulative | 39530.0        | packet    | 2017-03-23T05:39:15.325000 | | 0d21d8ac-0f8b-11e7-a2a2-109836a4a8af | instance-00000018-425717c0-210c-47f8-8f9b-287b3a8d691a-tap59a952fd-e5 | network.incoming.packets.rate   | gauge      | 0.185188717489 | packet/s  | 2017-03-23T05:39:15.325000 |
Statistic	sample의 조합으로 지정된 시간과 종료시간까지의 조합결과이다. [stack@director9 ~]$ ceilometer statistics -m cpu_util +--------+----------------------------+----------------------------+---------------+---------------+--------------+---------------+-------+----------+----------------------------+----------------------------+ | Period | Period Start               | Period End                 | Max           | Min           | Avg          | Sum           | Count | Duration | Duration Start             | Duration End               | +--------+----------------------------+----------------------------+---------------+---------------+--------------+---------------+-------+----------+----------------------------+----------------------------+ | 0      | 2017-03-21T08:27:50.399000 | 2017-03-23T05:41:26.909000 | 2.98667071122 | 1.36348273784 | 2.5257167207 | 1560.89293339 | 618   | 94326.49 | 2017-03-22T03:29:20.419000 | 2017-03-23T05:41:26.909000 | +--------+----------------------------+----------------------------+---------------+---------------+--------------+---------------+-------+----------+----------------------------+----------------------------+
alarm	지정한 alarm의 조건에 맞는 경우 state를 변경한다. [stack@director9 ~]$ ceilometer alarm-list +--------------------------------------+-----------------------------------------+-------------------+----------+---------+------------+-------------------------------------+------------------+ | Alarm ID                             | Name                                    | State             | Severity | Enabled | Continuous | Alarm condition                     | Time constraints | +--------------------------------------+-----------------------------------------+-------------------+----------+---------+------------+-------------------------------------+------------------+ | b323ed3b-b68e-42da-aca4-ceaea7a4c0ae | AutoScaling-cpu_alarm_high-cgpadcqwscwl | Ok | low      | True    | True       | avg(cpu_util) > 50.0 during 1 x 30s | None             | | d0877642-4cba-4a47-a08f-789f7cafb986 | AutoScaling-cpu_alarm_low-awedwernqaqy  | Ok | low      | True    | True       | avg(cpu_util) < 15.0 during 1 x 10s | None             | +--------------------------------------+-----------------------------------------+-------------------+----------+---------+------------+-------------------------------------+------------------+

아래 링크는 위 개념을 활용하여 autoscaling을 수행하는 방법이다.

- http://superuser.openstack.org/articles/simple-auto-scaling-environment-with-heat/

- https://keithtenzer.com/2015/09/02/auto-scaling-instances-with-openstack/

최종적으로 ceilometer로 확인시 statistic에 stack uuid로 정보가 출력되고 alarm이 ok 상태가 되어야 한다. 

아래와 같이 state가 insufficient 일 경우 무언가 부족한 상황으로 service daemon 및 기타 설정들을 다시 확인해봐야 한다.

[stack@director9 ~]$ ceilometer alarm-list

+--------------------------------------+-----------------------------------------+-------------------+----------+---------+------------+-------------------------------------+------------------+

| Alarm ID                             | Name                                    | State             | Severity | Enabled | Continuous | Alarm condition                     | Time constraints |

+--------------------------------------+-----------------------------------------+-------------------+----------+---------+------------+-------------------------------------+------------------+

| b323ed3b-b68e-42da-aca4-ceaea7a4c0ae | AutoScaling-cpu_alarm_high-cgpadcqwscwl | insufficient data | low      | True    | True       | avg(cpu_util) > 50.0 during 1 x 30s | None             |

| d0877642-4cba-4a47-a08f-789f7cafb986 | AutoScaling-cpu_alarm_low-awedwernqaqy  | insufficient data | low      | True    | True       | avg(cpu_util) < 15.0 during 1 x 10s | None             |

+--------------------------------------+-----------------------------------------+-------------------+----------+---------+------------+-------------------------------------+------------------+

아래의 meter 정보를 확인한다. q option을 통해 확인하고자 하는 instance resource id를 입력한다.

[stack@director9 ~]$ ceilometer meter-list -q resource=f6b57fc5-924c-43df-bef6-54efbbc434c4

+-------------------------------+------------+-----------+--------------------------------------+----------------------------------+----------------------------------+

| Name                          | Type       | Unit      | Resource ID                          | User ID                          | Project ID                       |

+-------------------------------+------------+-----------+--------------------------------------+----------------------------------+----------------------------------+

| compute.instance.booting.time | gauge      | sec       | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| cpu                           | cumulative | ns        | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| cpu.delta                     | delta      | ns        | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| cpu_util                      | gauge      | %         | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.allocation               | gauge      | B         | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.capacity                 | gauge      | B         | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.ephemeral.size           | gauge      | GB        | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.read.bytes               | cumulative | B         | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.read.bytes.rate          | gauge      | B/s       | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.read.requests            | cumulative | request   | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.read.requests.rate       | gauge      | request/s | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.root.size                | gauge      | GB        | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.usage                    | gauge      | B         | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.write.bytes              | cumulative | B         | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.write.bytes.rate         | gauge      | B/s       | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.write.requests           | cumulative | request   | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| disk.write.requests.rate      | gauge      | request/s | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| instance                      | gauge      | instance  | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| memory                        | gauge      | MB        | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| memory.resident               | gauge      | MB        | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

| vcpus                         | gauge      | vcpu      | f6b57fc5-924c-43df-bef6-54efbbc434c4 | b01fd9629bf84320ab281626800c8aee | b65dc5453eb145d0a613c4ec5ce30742 |

+-------------------------------+------------+-----------+--------------------------------------+----------------------------------+----------------------------------+

참고로 ceilomter에 의한 report interval은 pipeline.yaml 혹은 ceilometer.conf에 설정된 evaluation_interval 중 큰 값으로 선택된다. (mitaka 버전에서 확인결과 evaluation_interval은 default option으로 제공되지 않는다.)

[stack@director9 ~]$ ceilometer sample-list -m cpu_util -q resource=f6b57fc5-924c-43df-bef6-54efbbc434c4

+--------------------------------------+----------+-------+---------------+------+----------------------------+

| Resource ID                          | Name     | Type  | Volume        | Unit | Timestamp                  |

+--------------------------------------+----------+-------+---------------+------+----------------------------+

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.42849708842 | %    | 2017-03-23T05:58:54.876000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.40887003605 | %    | 2017-03-23T05:58:45.883000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.40286910852 | %    | 2017-03-23T05:58:06.057000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.39910951343 | %    | 2017-03-23T05:57:46.272000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.41421267047 | %    | 2017-03-23T05:57:05.111000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.39007913127 | %    | 2017-03-23T05:56:59.060000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.43842228514 | %    | 2017-03-23T05:56:05.106000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.4082989591  | %    | 2017-03-23T05:55:59.061000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.39102281432 | %    | 2017-03-23T05:55:05.256000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.39961821381 | %    | 2017-03-23T05:54:59.061000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.38433651083 | %    | 2017-03-23T05:54:05.118000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.38115319507 | %    | 2017-03-23T05:53:59.058000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.40812641894 | %    | 2017-03-23T05:53:05.118000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.39811386847 | %    | 2017-03-23T05:52:59.090000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.40056434729 | %    | 2017-03-23T05:52:05.088000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.40901767978 | %    | 2017-03-23T05:51:59.065000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.40013548711 | %    | 2017-03-23T05:51:05.127000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.38453228409 | %    | 2017-03-23T05:50:59.064000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.41060015616 | %    | 2017-03-23T05:50:05.255000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.39947233405 | %    | 2017-03-23T05:49:59.068000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.41242676716 | %    | 2017-03-23T05:49:05.119000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.41744536915 | %    | 2017-03-23T05:48:59.067000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.40055598978 | %    | 2017-03-23T05:48:05.097000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.41990064816 | %    | 2017-03-23T05:47:59.068000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.39463930127 | %    | 2017-03-23T05:47:05.191000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.38899133986 | %    | 2017-03-23T05:46:59.095000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.41621225565 | %    | 2017-03-23T05:46:05.099000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.41258347191 | %    | 2017-03-23T05:45:59.076000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.41662645022 | %    | 2017-03-23T05:45:05.088000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.40917362699 | %    | 2017-03-23T05:44:59.077000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.41328571709 | %    | 2017-03-23T05:44:05.500000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.42669200669 | %    | 2017-03-23T05:43:59.083000 |

| f6b57fc5-924c-43df-bef6-54efbbc434c4 | cpu_util | gauge | 1.40435347196 | %    | 2017-03-23T05:43:05.112000 |

+--------------------------------------+----------+-------+---------------+------+----------------------------+

autoscaling을 수행한 경우 아래의 command를 통해 ceilomter로 부터 수집되는 정보의 평균값을 확인할 수 있다.

# -q 옵션은 query의 약자로 grep과 같은 역할을 수행해준다.

# 아래는 stack list 중 autoscaling으로 생성한 stack의 uuid를 사용한 query이다.

# 해당 stack_id는 ceilomter alarm-show 를 통해서도 확인이 가능하다.

ceilometer statistics -m cpu_util -q metadata.user_metadata.stack=d8ab28cc-f880-4f47-854d-eefd08492eec

## 위 command로 결과가 나오지 않는 경우 yaml파일내 parameter를 잘못 사용한 경우일수도 있다.

## 아래와 같이 40초 단위의 평균(avg)치를 아래 command로 확인할수도 있다.

ceilometer statistics -m cpu_util -q metadata.user_metadata.stack=05338d88-7ea4-4c7a-ae2f-075e156f089b -p 40 -a avg 

## 실제 scale up & down은 cooldown value(예로 60이라면)에 따라 2 min 까지 소요될수 있다.

Director를 통해 배포된 경우 openstack-ceilometer-api-clone을 재구동한다.

# controller nodes

pcs resource restart openstack-ceilometer-api-clone

# compute nodes

systemctl restart openstack-ceilometer-compute 

systemctl restart openstack-ceilometer-polling # 해당 service는 필수적이지 않다. polling을 써야하는경우 사용

아래 구조를 이해하고 autoscaling을 수행하면 좀더 이해가 쉽다.

https://docs.openstack.org/developer/ceilometer/architecture.html

telemetry 관련 상세 정보 (gnocchi, aodh, ceilometer 관련 정보) 

- http://www.sparkmycloud.com/blog/telemetry-service-in-openstack/

참고로 Loadbalancer를 version 2를 사용하는 경우 기존 heat template으로 사용이 어렵다.

- https://github.com/tigerlinux/tigerlinux-extra-recipes/tree/master/recipes/openstack/autoscaling-with-lbaasv2

를 참고하여 사용한다.

추가로 instance name rule을 생성하려 했으나 autoscaling group resource에서는 기능 제공이 안되고 있다.

https://ask.openstack.org/en/question/68164/can-we-assign-instance-names-to-autoscaling-servers/

ceilometer에 의해 데이터가 쌓이면 query가 느려지는 문제가 발생될 수 있다.

기본설정이 -1로 무한대이기에 이값을 적절히 변경한다.

(이를 설정하지 않고 수동으로 제거하기 위해서는 mongodb에 직접 query하여 삭제해야한다.

- https://ask.openstack.org/en/question/31771/how-to-clean-up-old-ceilometer-data/

[database]

#

# From ceilometer

#

# Number of seconds that samples are kept in the database for (<= 0 means

# forever). (integer value)

# Deprecated group/name - [database]/time_to_live

#metering_time_to_live = -1

metering_time_to_live= 86400

# Number of seconds that events are kept in the database for (<= 0 means

# forever). (integer value)

#event_time_to_live = -1

event_time_to_live=86400

참고사이트

- heat template의 구조와 사용법에 간략히 설명한다.

  - https://developer.rackspace.com/blog/openstack-orchestration-in-depth-part-1-introduction-to-heat/

  - https://developer.rackspace.com/blog/openstack-orchestration-in-depth-part-2-single-instance-deployments/

  - https://developer.rackspace.com/blog/openstack-orchestration-in-depth-part-3-multi-instance-deployments/

  - https://developer.rackspace.com/blog/openstack-orchestration-in-depth-part-4-scaling/

- autoscaling 관련 사용법설명 문서  

  - http://cs.utdallas.edu/wp-content/uploads/2015/09/AutoScaling.pdf