<div dir="ltr"><div><div><div><div>Hi list!<br>We have a problem upgrading hosted engine nodes (hypervisors) from 3.5.6 to 3.6.7 we get these errors:<br><br></div>from agent.log:<br>MainThread::ERROR::2017-01-24
15:17:58,981::upgrade::207::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there)
Unable to find HE conf volume
<br>MainThread::<a class="gmail-moz-txt-link-freetext">INFO::2017-01-24</a>
15:17:58,981::upgrade::262::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_create_shared_conf_volume)
Creating hosted-engine configuration volume on the shared storage domain
<br>MainThread::DEBUG::2017-01-24
15:17:59,079::heconflib::358::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image)
{'status': {'message': 'OK', 'code': 0}, 'uuid':
'df95e3fd-3715-4525-8955-abcc8cb24865'}
<br>MainThread::DEBUG::2017-01-24
15:17:59,080::heconflib::372::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image)
Created configuration volume OK, request was:
<br>- image: e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc
<br>- volume: 9636163f-d634-4c28-b2bc-f84b5a90a17e
<br>MainThread::DEBUG::2017-01-24
15:17:59,080::heconflib::283::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(task_wait)
Waiting for existing tasks to complete
<br>MainThread::DEBUG::2017-01-24
15:18:00,137::heconflib::283::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(task_wait)
Waiting for existing tasks to complete
<br>MainThread::DEBUG::2017-01-24
15:18:00,278::heconflib::379::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image)
configuration volume: prepareImage
<br>MainThread::DEBUG::2017-01-24
15:18:00,350::heconflib::387::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image)
{'status': {'message': "Volume does not exist:
('9636163f-d634-4c28-b2bc-f84b5a90a17e',)", 'code': 201}<br><br></div>from vdsm.log:<br>df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,187::task::752::Storage.TaskManager.Task::(_save)
Task=`df95e3fd-3715-4525-8955-abcc8cb24865`::_save: orig
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-
<br>abcc8cb24865 temp
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-abcc8cb24865.temp
<br>df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,201::task::752::Storage.TaskManager.Task::(_save)
Task=`df95e3fd-3715-4525-8955-abcc8cb24865`::_save: orig
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-
<br>abcc8cb24865 temp
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-abcc8cb24865.temp
<br>df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,217::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-79 /usr/bin/sudo -n /usr/sbin/lvm lvcreate --config '
devices { preferred_names = ["^<i class="gmail-moz-txt-slash"><span class="gmail-moz-txt-tag">/</span>dev/mapper<span class="gmail-moz-txt-tag">/</span></i>"] ignore_suspended_devices=1
write_cac
<br>he_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/2347b42194dacd3656c9ce900aea3dee2|'\'', '\''r|.*|'\''
] } global { locking_type=1 prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50
retain_days = 0 } ' --autobackup
<br> n --contiguous n --size 1024m --addtag OVIRT_VOL_INITIALIZING --name
9636163f-d634-4c28-b2bc-f84b5a90a17e
8fa5a242-fcb4-454d-aaa1-23f8dd92373b (cwd None)
<br>Thread-28556::DEBUG::2017-01-24
15:17:59,289::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list
0-79 /usr/sbin/tc qdisc show (cwd None)
<br>df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,301::lvm::290::Storage.Misc.excCmd::(cmd) FAILED: <err> = '
WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
WARNING: To avoid corruption, rescan devices to make changes visible
(pvscan --
<br>cache).\n Volume group "8fa5a242-fcb4-454d-aaa1-23f8dd92373b" has
insufficient free space (4 extents): 8 required.\n'; <rc> = 5
<br>df95e3fd-3715-4525-8955-abcc8cb24865::ERROR::2017-01-24
15:17:59,303::volume::485::Storage.Volume::(create) Failed to create
volume
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/8fa5a242-fcb4-454d-aaa1-23f8dd92373b/images/e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc/96361
<br>63f-d634-4c28-b2bc-f84b5a90a17e: Cannot create Logical Volume:
('8fa5a242-fcb4-454d-aaa1-23f8dd92373b',
'9636163f-d634-4c28-b2bc-f84b5a90a17e')
<br>df95e3fd-3715-4525-8955-abcc8cb24865::ERROR::2017-01-24
15:17:59,304::volume::521::Storage.Volume::(create) Unexpected error
<br>Traceback (most recent call last):
<br> File "/usr/share/vdsm/storage/volume.py", line 482, in create
<br> initialSize=initialSize)
<br> File "/usr/share/vdsm/storage/blockVolume.py", line 133, in _create
<br> initialTag=TAG_VOL_UNINIT)
<br> File "/usr/share/vdsm/storage/lvm.py", line 1096, in createLV
<br> raise se.CannotCreateLogicalVolume(vgName, lvName)
<br>CannotCreateLogicalVolume: Cannot create Logical Volume:
('8fa5a242-fcb4-454d-aaa1-23f8dd92373b',
'9636163f-d634-4c28-b2bc-f84b5a90a17e')
<br>df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,304::resourceManager::619::Storage.ResourceManager::(releaseResource)
Trying to release resource
'8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc'
<br>df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,304::resourceManager::638::Storage.ResourceManager::(releaseResource)
Released resource
'8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc'
(0 active users)
<br>df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,305::resourceManager::644::Storage.ResourceManager::(releaseResource)
Resource
'8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc'
is free, finding out if anyone is waiting for
<br> it.
<br>df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,305::resourceManager::652::Storage.ResourceManager::(releaseResource)
No one is waiting for resource
'8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc',
Clearing records.
<br>df95e3fd-3715-4525-8955-abcc8cb24865::ERROR::2017-01-24
15:17:59,305::task::866::Storage.TaskManager.Task::(_setError)
Task=`df95e3fd-3715-4525-8955-abcc8cb24865`::Unexpected error
<br>Traceback (most recent call last):
<br> File "/usr/share/vdsm/storage/task.py", line 873, in _run
<br> return fn(*args, **kargs)
<br> File "/usr/share/vdsm/storage/task.py", line 332, in run
<br> return self.cmd(*self.argslist, **self.argsdict)
<br> File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper
<br> return method(self, *args, **kwargs)
<br> File "/usr/share/vdsm/storage/sp.py", line 1934, in createVolume
<br> initialSize=initialSize)
<br> File "/usr/share/vdsm/storage/sd.py", line 488, in createVolume
<br> initialSize=initialSize)
<br> File "/usr/share/vdsm/storage/volume.py", line 482, in create
<br> initialSize=initialSize)
<br> File "/usr/share/vdsm/storage/blockVolume.py", line 133, in _create
<br> initialTag=TAG_VOL_UNINIT)
<br> File "/usr/share/vdsm/storage/lvm.py", line 1096, in createLV
<br> raise se.CannotCreateLogicalVolume(vgName, lvName)
<br>CannotCreateLogicalVolume: Cannot create Logical Volume:
('8fa5a242-fcb4-454d-aaa1-23f8dd92373b',
'9636163f-d634-4c28-b2bc-f84b5a90a17e')
<br>
<br></div><div><br><br><br>It seems that there is not enough disk space on the engine LUN to create the configuration volume:<br># vgs<br> VG #PV #LV #SN Attr VSize VFree <br> 8fa5a242-fcb4-454d-aaa1-23f8dd92373b 1 9 0 wz--n- 104.62g 512.00m<br><br><br><br></div></div><div><div><div><div>What can we do to fix this?<br><br></div><div>Regards<br></div><div>Christian<br></div></div></div></div></div>