Kolla
Kolla Preparations
When using overlay configuration files at environments/kolla/files/overlays
,
the following overlay files need to be synchronized. Customer specific changes
to the current files need to be copied manually to the new overlay configuration
files.
HAProxy Preparations
When updating to Rocky - Synchronize haproxy.cfg to
environments/kolla/files/overlays/haproxy/haproxy.cfg
.When updating to Stein - Synchronize haproxy_main.cfg to
environments/kolla/files/overlays/haproxy/haproxy_main.cfg
The old
environments/kolla/files/overlays/haproxy/haproxy.cfg
need to be deleted.
Horizon Preparations
The file
environments/kolla/files/overlays/horizon/local_settings.j2
need to be removed.Copy custom_local_settings to
environments/kolla/files/overlays/horizon/custom_local_settings
.
Backup MariaDB
Backup MariaDB databases before the upgrade on one of the control nodes:
The password for MariaDB can be found in the file environments/kolla/secrets.yml
in the variable
database_password
.
docker exec -t mariadb innobackupex -u root -p DATABASE_PASSWORD /tmp/mariadb
docker cp mariadb:/tmp/mariadb $PWD/mariadb
See MariaDB backup for further details.
Upgrading from Ocata to Pike
Upgrading Docker
Note
This task is only necessary on Ubuntu 16.04 because the python-docker
package is outdated.
When encountering the following error message, Docker need to be upgraded!
fatal: [20-10.betacloud.xyz]: FAILED! => {"changed": true, "failed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_Lrxpgg/ansible_module_kolla_docker.py\", line 804, in main\\n dw = DockerWorker(module)\\n File \"/tmp/ansible_Lrxpgg/ansible_module_kolla_docker.py\", line 218, in __init__\\n self.dc = get_docker_client()(**options)\\n File \"/tmp/ansible_Lrxpgg/ansible_module_kolla_docker.py\", line 201, in get_docker_client\\n return docker.APIClient\\nAttributeError: \\'module\\' object has no attribute \\'APIClient\\'\\n'"}
See Docker on how to upgrade Docker.
Inventory
New host groups need to be added to file
inventory/hosts
:
##########################################################
# environment: kolla
[redis:children]
control
# neutron
[neutron-bgp-dragent:children]
network
[openvswitch:children]
network
compute
Configuration
Mistral: Redis is now required by default. In file
environments/kolla/configuration.yml
the deployment need to be enabled by adding parameterenable_redis: "yes"
.enable_redis: yes
In file
environments/kolla/secrets.yml
new parameterredis_master_password
need to be set.pwgen -1 32 aevooVaeceeh2aisaeRah2ufieHee7oh ansible-vault edit environments/kolla/secrets.yml
redis_master_password: aevooVaeceeh2aisaeRah2ufieHee7oh
Ceilometer: The Ceilometer API was dropped. All
ceilometer / metering
endpoints from Keystone (openstack endpoint list) and theceilometer-api
host group need to be removed.
Notes
Ceilometer: After the upgrade, the
ceilometer_api
container and image need to be removed from all control nodes, as well as the configuration directory/etc/kolla/ceilometer-api
.
Upgrading from Pike to Queens
RabbitMQ Configuration
RabbitMQ: New parameter
rabbitmq_monitoring_password
need to be added toenvironments/kolla/secrets.yml
pwgen -1 32
we7oey4wifeilieK9ii1uighiraJoWoo
ansible-vault edit environments/kolla/secrets.yml
rabbitmq_monitoring_password: we7oey4wifeilieK9ii1uighiraJoWoo
Upgrading from Queens to Rocky
Upgrade Queens to Rocky Inventory
New host groups need to be added to file
inventory/hosts
:
##########################################################
# environment: kolla
# neutron
[neutron-infoblox-ipam-agent:children]
network
[ironic-neutron-agent:children]
network
Upgrade Configuration
In file
environments/kolla/configuration.yml
, if set, changeserial
tokolla_serial
.In file
environments/kolla/files/overlays
add the gnocci ceph keyfile and configuration file to thegnocchi
directory, The ceph keyfiles and configuration files in thegnocchi-metricd
,gnocchi-statsd
, andgnocchi-api
directories can be removed.The
glance_registry
containers on the control nodes can be removed, the service was deprecated in Queens and will be removed in Stein release.docker stop glance_registry docker rm glance_registry
Elasticsearch
Upgrading Elasticsearch might fail, because it still has pending operations when
trying to perform a synced flush. Normally it does not matter if some
logging data is lost while upgrading Elasticsearch, therefore
osism-kolla deploy elasticsearch
can be used instead of
osism-kolla upgrade elasticsearch
, to stop all operations on the cluster
immediately before restarting.
Kibana
The Elasticsearch index .kibana
might have to be deleted (or updated) after
the Upgrade. See Logging.
Dashboards and saved searches in Kibana, will be lost after deleting the index.
Upgrading from Rocky to Stein
Upgarde Rocky to Stein Inventory
New host group need to be added to file
inventory/hosts
:
# neutron
[neutron-metering-agent:children]
neutron
MariaDB (Upgrade R2S)
Backups are possible beginning with Stein release
configure MariaDB Backup in
environments/kolla/configuration.yml
enable_mariabackup: "yes"
set MariaDB Backup Password in
environments/kolla/secrets.yml
mariadb_backup_database_password: password
Glance (Upgrade R2S)
The location of file
environments/kolla/files/overlays/glance-api/ceph.client.glance.keyring
has moved toenvironments/kolla/files/overlays/glance/ceph.client.glance.keyring
.
HAProxy (Upgrade R2S)
In file
environments/kolla/configuration.yml
setkolla_enable_tls_internal: "no"
.File
environments/kolla/files/overlays/haproxy/haproxy.cfg
is no longer used and can be deleted.File
environments/kolla/files/overlays/haproxy/haproxy_main.cfg
need to added with the following content:
#jinja2: lstrip_blocks: True
global
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
log {{ syslog_server }}:{{ syslog_udp_port }} {{ syslog_haproxy_facility }}
maxconn {{ haproxy_max_connections }}
nbproc {{ haproxy_processes }}
{% if (haproxy_processes | int > 1) and (haproxy_process_cpu_map | bool) %}
{% for cpu_idx in range(0, haproxy_processes) %}
cpu-map {{ cpu_idx + 1 }} {{ cpu_idx }}
{% endfor %}
{% endif %}
stats socket /var/lib/kolla/haproxy/haproxy.sock group kolla mode 660
{% if kolla_enable_tls_external | bool or kolla_enable_tls_internal | bool %}
ssl-default-bind-ciphers DEFAULT:!MEDIUM:!3DES
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
tune.ssl.default-dh-param 4096
{% endif %}
defaults
log global
option redispatch
retries 3
timeout http-request {{ haproxy_http_request_timeout }}
timeout queue {{ haproxy_queue_timeout }}
timeout connect {{ haproxy_connect_timeout }}
timeout client {{ haproxy_client_timeout }}
timeout server {{ haproxy_server_timeout }}
timeout check {{ haproxy_check_timeout }}
balance {{ haproxy_defaults_balance }}
maxconn {{ haproxy_defaults_max_connections }}
listen stats
bind {{ api_interface_address }}:{{ haproxy_stats_port }}
mode http
stats enable
stats uri /
stats refresh 15s
stats realm Haproxy\ Stats
stats auth {{ haproxy_user }}:{{ haproxy_password }}
frontend status
bind {{ api_interface_address }}:{{ haproxy_monitor_port }}
{% if api_interface_address != kolla_internal_vip_address %}
bind {{ kolla_internal_vip_address }}:{{ haproxy_monitor_port }}
{% endif %}
mode http
monitor-uri /
# OSISM specific configuration
listen ceph_dashboard
option httpchk
http-check expect status 200
bind {{ kolla_internal_vip_address }}:8140
{% for host in groups['ceph-mgr'] %}
server {{ hostvars[host]['ansible_hostname'] }} {{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}:7000 check inter 2000 rise 2 fall 5
{% endfor %}
listen ceph_prometheus
bind {{ kolla_internal_vip_address }}:9283
{% for host in groups['ceph-mgr'] %}
server {{ hostvars[host]['ansible_hostname'] }} {{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}:9283 check inter 2000 rise 2 fall 5
{% endfor %}
# customer specific configuration
Running the upgrade
When using the osism-kolla upgrade
command, the currently running container
is shut down. Next the Docker image for the upgraded version is pulled, and
finally the new container is started. It might be advisable to first pull the
Docker image and then run osism-kolla upgrade
. See
Deploying Openstack Services for how to use osism-kolla pull
.
Gathering Ansible facts
osism-generic facts
Upgrade Common
osism-kolla upgrade common
Upgrade HAProxy
osism-kolla upgrade loadbalancer
Upgrade Logging
osism-kolla upgrade elasticsearch
osism-kolla upgrade kibana
Upgrade Infrastructure
osism-kolla upgrade memcached
osism-kolla upgrade mariadb
osism-kolla upgrade rabbitmq
osism-kolla upgrade redis
osism-kolla upgrade openvswitch
Upgrade Storage (optional)
osism-kolla upgrade iscsi
osism-kolla upgrade multipathd
Upgrade OpenStack Services
osism-kolla upgrade keystone
osism-kolla upgrade horizon
osism-kolla upgrade glance
osism-kolla upgrade cinder
osism-kolla upgrade neutron
osism-kolla upgrade heat
osism-kolla upgrade placement # beginning from Stein release
Upgrade Nova
Upgrade nova on the control nodes first:
osism-kolla upgrade nova -l control
Upgrade nova on the compute nodes:
osism-kolla upgrade nova -l compute
If additional optional services are deployed in your environment, run the upgrade for those services as well:
osism-kolla upgrade SERVICE_NAME
After the upgrade
Fix Horizon
After the upgrade the cache need to be cleaned and regenerated. Run the following command on all control nodes:
docker exec -it horizon rm /var/lib/kolla/.local_settings.md5sum.txt
docker restart horizon
Fix Elasticsearch
After the ugprade of Elasticsearch, the shard allocation need to be enabled.
curl -X PUT "http://api-int.osism.local:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}
'
Remove old Docker images
Verify none of the old images is running anymore.
docker ps --filter=label=io.osism.openstack=queens
Remove old version images.
docker rmi $(docker image ls --quiet --filter=label=io.osism.openstack=queens)