Hi list, When i am trying to upgrade hosts from 3.5.6 to 3.6.7 i get these errors from agent.log: MainThread::ERROR::2017-01-24 15:17:58,981::upgrade::207::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Unable to find HE conf volume MainThread::INFO::2017-01-24 15:17:58,981::upgrade::262::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_create_shared_conf_volume) Creating hosted-engine configuration volume on the shared storage domain MainThread::DEBUG::2017-01-24 15:17:59,079::heconflib::358::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image) {'status': {'message': 'OK', 'code': 0}, 'uuid': 'df95e3fd-3715-4525-8955-abcc8cb24865'} MainThread::DEBUG::2017-01-24 15:17:59,080::heconflib::372::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image) Created configuration volume OK, request was: - image: e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc - volume: 9636163f-d634-4c28-b2bc-f84b5a90a17e MainThread::DEBUG::2017-01-24 15:17:59,080::heconflib::283::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(task_wait) Waiting for existing tasks to complete MainThread::DEBUG::2017-01-24 15:18:00,137::heconflib::283::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(task_wait) Waiting for existing tasks to complete MainThread::DEBUG::2017-01-24 15:18:00,278::heconflib::379::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image) configuration volume: prepareImage MainThread::DEBUG::2017-01-24 15:18:00,350::heconflib::387::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image) {'status': {'message': "Volume does not exist: ('9636163f-d634-4c28-b2bc-f84b5a90a17e',)", 'code': 201} from vdsm.log: df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24 15:17:59,187::task::752::Storage.TaskManager.Task::(_save) Task=`df95e3fd-3715-4525-8955-abcc8cb24865`::_save: orig /rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955- abcc8cb24865 temp /rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-abcc8cb24865.temp df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24 15:17:59,201::task::752::Storage.TaskManager.Task::(_save) Task=`df95e3fd-3715-4525-8955-abcc8cb24865`::_save: orig /rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955- abcc8cb24865 temp /rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-abcc8cb24865.temp df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24 15:17:59,217::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-79 /usr/bin/sudo -n /usr/sbin/lvm lvcreate --config ' devices { preferred_names = ["^*/dev/mapper/*"] ignore_suspended_devices=1 write_cac he_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/2347b42194dacd3656c9ce900aea3dee2|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --autobackup n --contiguous n --size 1024m --addtag OVIRT_VOL_INITIALIZING --name 9636163f-d634-4c28-b2bc-f84b5a90a17e 8fa5a242-fcb4-454d-aaa1-23f8dd92373b (cwd None) Thread-28556::DEBUG::2017-01-24 15:17:59,289::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list 0-79 /usr/sbin/tc qdisc show (cwd None) df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24 15:17:59,301::lvm::290::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan -- cache).\n Volume group "8fa5a242-fcb4-454d-aaa1-23f8dd92373b" has insufficient free space (4 extents): 8 required.\n'; <rc> = 5 df95e3fd-3715-4525-8955-abcc8cb24865::ERROR::2017-01-24 15:17:59,303::volume::485::Storage.Volume::(create) Failed to create volume /rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/8fa5a242-fcb4-454d-aaa1-23f8dd92373b/images/e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc/96361 63f-d634-4c28-b2bc-f84b5a90a17e: Cannot create Logical Volume: ('8fa5a242-fcb4-454d-aaa1-23f8dd92373b', '9636163f-d634-4c28-b2bc-f84b5a90a17e') df95e3fd-3715-4525-8955-abcc8cb24865::ERROR::2017-01-24 15:17:59,304::volume::521::Storage.Volume::(create) Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/volume.py", line 482, in create initialSize=initialSize) File "/usr/share/vdsm/storage/blockVolume.py", line 133, in _create initialTag=TAG_VOL_UNINIT) File "/usr/share/vdsm/storage/lvm.py", line 1096, in createLV raise se.CannotCreateLogicalVolume(vgName, lvName) CannotCreateLogicalVolume: Cannot create Logical Volume: ('8fa5a242-fcb4-454d-aaa1-23f8dd92373b', '9636163f-d634-4c28-b2bc-f84b5a90a17e') df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24 15:17:59,304::resourceManager::619::Storage.ResourceManager::(releaseResource) Trying to release resource '8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc' df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24 15:17:59,304::resourceManager::638::Storage.ResourceManager::(releaseResource) Released resource '8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc' (0 active users) df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24 15:17:59,305::resourceManager::644::Storage.ResourceManager::(releaseResource) Resource '8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc' is free, finding out if anyone is waiting for it. df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24 15:17:59,305::resourceManager::652::Storage.ResourceManager::(releaseResource) No one is waiting for resource '8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc', Clearing records. df95e3fd-3715-4525-8955-abcc8cb24865::ERROR::2017-01-24 15:17:59,305::task::866::Storage.TaskManager.Task::(_setError) Task=`df95e3fd-3715-4525-8955-abcc8cb24865`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/storage/task.py", line 332, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 1934, in createVolume initialSize=initialSize) File "/usr/share/vdsm/storage/sd.py", line 488, in createVolume initialSize=initialSize) File "/usr/share/vdsm/storage/volume.py", line 482, in create initialSize=initialSize) File "/usr/share/vdsm/storage/blockVolume.py", line 133, in _create initialTag=TAG_VOL_UNINIT) File "/usr/share/vdsm/storage/lvm.py", line 1096, in createLV raise se.CannotCreateLogicalVolume(vgName, lvName) CannotCreateLogicalVolume: Cannot create Logical Volume: ('8fa5a242-fcb4-454d-aaa1-23f8dd92373b', '9636163f-d634-4c28-b2bc-f84b5a90a17e') It seems that there is not enough disk space on the engine LUN to create the configuration volume: # vgs VG #PV #LV #SN Attr VSize VFree 8fa5a242-fcb4-454d-aaa1-23f8dd92373b 1 9 0 wz--n- 104.62g 512.00m What can we do to fix this?