Kolla Preparations

When using overlay configuration files at environments/kolla/files/overlays, the following overlay files need to be synchronized. Customer specific changes to the current files need to be copied manually to the new overlay configuration files.

HAProxy Preparations

  • When updating to Rocky - Synchronize haproxy.cfg to


  • When updating to Stein - Synchronize haproxy_main.cfg to


    The old environments/kolla/files/overlays/haproxy/haproxy.cfg need to be deleted.

Horizon Preparations

  • The file environments/kolla/files/overlays/horizon/local_settings.j2 need to be removed.

  • Copy custom_local_settings to


Backup MariaDB

Backup MariaDB databases before the upgrade on one of the control nodes:

The password for MariaDB can be found in the file environments/kolla/secrets.yml in the variable database_password.

docker exec -t mariadb innobackupex -u root -p DATABASE_PASSWORD /tmp/mariadb
docker cp mariadb:/tmp/mariadb $PWD/mariadb

See MariaDB backup for further details.

Running the upgrade

When using the osism-kolla upgrade command, the currently running container is shut down. Next the Docker image for the upgraded version is pulled, and finally the new container is started. It might be advisable to first pull the Docker image and then run osism-kolla upgrade. See Deploying Openstack Services for how to use osism-kolla pull.

Gathering Ansible facts

osism-generic facts

Upgrade Common

osism-kolla upgrade common

Upgrade HAProxy

osism-kolla upgrade loadbalancer

Upgrade Logging

osism-kolla upgrade elasticsearch
osism-kolla upgrade kibana

Upgrade Infrastructure

osism-kolla upgrade memcached
osism-kolla upgrade mariadb
osism-kolla upgrade rabbitmq
osism-kolla upgrade redis
osism-kolla upgrade openvswitch

Upgrade Storage (optional)

osism-kolla upgrade iscsi
osism-kolla upgrade multipathd

Upgrade OpenStack Services

osism-kolla upgrade keystone
osism-kolla upgrade horizon
osism-kolla upgrade glance
osism-kolla upgrade cinder
osism-kolla upgrade neutron
osism-kolla upgrade heat
osism-kolla upgrade placement # beginning from Stein release

Upgrade Nova

  • Upgrade nova on the control nodes first:

osism-kolla upgrade nova -l control
  • Upgrade nova on the compute nodes:

osism-kolla upgrade nova -l compute

If additional optional services are deployed in your environment, run the upgrade for those services as well:

osism-kolla upgrade SERVICE_NAME

After the upgrade

Fix Horizon

  • After the upgrade the cache need to be cleaned and regenerated. Run the following command on all control nodes:

docker exec -it horizon rm /var/lib/kolla/.local_settings.md5sum.txt
docker restart horizon

Fix Elasticsearch

  • After the ugprade of Elasticsearch, the shard allocation need to be enabled.

curl -X PUT "http://api-int.osism.local:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
  "persistent": {
    "cluster.routing.allocation.enable": null

Remove old Docker images

Verify none of the old images is running anymore.

docker ps --filter=label=de.osism.release.openstack=victoria

Remove old version images.

docker rmi $(docker image ls --quiet --filter=label=de.osism.release.openstack=victoria)