Hi list!
We have a problem upgrading hosted engine nodes (hypervisors) from 3.5.6 to 3.6.7 we get these errors:
from agent.log:
MainThread::ERROR::2017-01-24
15:17:58,981::upgrade::207::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there)
Unable to find HE conf volume
MainThread::
INFO::2017-01-24
15:17:58,981::upgrade::262::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_create_shared_conf_volume)
Creating hosted-engine configuration volume on the shared storage domain
MainThread::DEBUG::2017-01-24
15:17:59,079::heconflib::358::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image)
{'status': {'message': 'OK', 'code': 0}, 'uuid':
'df95e3fd-3715-4525-8955-abcc8cb24865'}
MainThread::DEBUG::2017-01-24
15:17:59,080::heconflib::372::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image)
Created configuration volume OK, request was:
- image: e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc
- volume: 9636163f-d634-4c28-b2bc-f84b5a90a17e
MainThread::DEBUG::2017-01-24
15:17:59,080::heconflib::283::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(task_wait)
Waiting for existing tasks to complete
MainThread::DEBUG::2017-01-24
15:18:00,137::heconflib::283::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(task_wait)
Waiting for existing tasks to complete
MainThread::DEBUG::2017-01-24
15:18:00,278::heconflib::379::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image)
configuration volume: prepareImage
MainThread::DEBUG::2017-01-24
15:18:00,350::heconflib::387::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(create_and_prepare_image)
{'status': {'message': "Volume does not exist:
('9636163f-d634-4c28-b2bc-f84b5a90a17e',)", 'code': 201}
from vdsm.log:
df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,187::task::752::Storage.TaskManager.Task::(_save)
Task=`df95e3fd-3715-4525-8955-abcc8cb24865`::_save: orig
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-
abcc8cb24865 temp
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-abcc8cb24865.temp
df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,201::task::752::Storage.TaskManager.Task::(_save)
Task=`df95e3fd-3715-4525-8955-abcc8cb24865`::_save: orig
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-
abcc8cb24865 temp
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/mastersd/master/tasks/df95e3fd-3715-4525-8955-abcc8cb24865.temp
df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,217::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-79 /usr/bin/sudo -n /usr/sbin/lvm lvcreate --config '
devices { preferred_names = ["^
/dev/mapper/"] ignore_suspended_devices=1
write_cac
he_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/2347b42194dacd3656c9ce900aea3dee2|'\'', '\''r|.*|'\''
] } global { locking_type=1 prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50
retain_days = 0 } ' --autobackup
n --contiguous n --size 1024m --addtag OVIRT_VOL_INITIALIZING --name
9636163f-d634-4c28-b2bc-f84b5a90a17e
8fa5a242-fcb4-454d-aaa1-23f8dd92373b (cwd None)
Thread-28556::DEBUG::2017-01-24
15:17:59,289::utils::671::root::(execCmd) /usr/bin/taskset --cpu-list
0-79 /usr/sbin/tc qdisc show (cwd None)
df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,301::lvm::290::Storage.Misc.excCmd::(cmd) FAILED: <err> = '
WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
WARNING: To avoid corruption, rescan devices to make changes visible
(pvscan --
cache).\n Volume group "8fa5a242-fcb4-454d-aaa1-23f8dd92373b" has
insufficient free space (4 extents): 8 required.\n'; <rc> = 5
df95e3fd-3715-4525-8955-abcc8cb24865::ERROR::2017-01-24
15:17:59,303::volume::485::Storage.Volume::(create) Failed to create
volume
/rhev/data-center/c202d58c-afc7-4848-8501-228d9bccb15d/8fa5a242-fcb4-454d-aaa1-23f8dd92373b/images/e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc/96361
63f-d634-4c28-b2bc-f84b5a90a17e: Cannot create Logical Volume:
('8fa5a242-fcb4-454d-aaa1-23f8dd92373b',
'9636163f-d634-4c28-b2bc-f84b5a90a17e')
df95e3fd-3715-4525-8955-abcc8cb24865::ERROR::2017-01-24
15:17:59,304::volume::521::Storage.Volume::(create) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/volume.py", line 482, in create
initialSize=initialSize)
File "/usr/share/vdsm/storage/blockVolume.py", line 133, in _create
initialTag=TAG_VOL_UNINIT)
File "/usr/share/vdsm/storage/lvm.py", line 1096, in createLV
raise se.CannotCreateLogicalVolume(vgName, lvName)
CannotCreateLogicalVolume: Cannot create Logical Volume:
('8fa5a242-fcb4-454d-aaa1-23f8dd92373b',
'9636163f-d634-4c28-b2bc-f84b5a90a17e')
df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,304::resourceManager::619::Storage.ResourceManager::(releaseResource)
Trying to release resource
'8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc'
df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,304::resourceManager::638::Storage.ResourceManager::(releaseResource)
Released resource
'8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc'
(0 active users)
df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,305::resourceManager::644::Storage.ResourceManager::(releaseResource)
Resource
'8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc'
is free, finding out if anyone is waiting for
it.
df95e3fd-3715-4525-8955-abcc8cb24865::DEBUG::2017-01-24
15:17:59,305::resourceManager::652::Storage.ResourceManager::(releaseResource)
No one is waiting for resource
'8fa5a242-fcb4-454d-aaa1-23f8dd92373b_imageNS.e4d09ccf-5d9b-4f90-9eda-1ebd4f55ecbc',
Clearing records.
df95e3fd-3715-4525-8955-abcc8cb24865::ERROR::2017-01-24
15:17:59,305::task::866::Storage.TaskManager.Task::(_setError)
Task=`df95e3fd-3715-4525-8955-abcc8cb24865`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 332, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 1934, in createVolume
initialSize=initialSize)
File "/usr/share/vdsm/storage/sd.py", line 488, in createVolume
initialSize=initialSize)
File "/usr/share/vdsm/storage/volume.py", line 482, in create
initialSize=initialSize)
File "/usr/share/vdsm/storage/blockVolume.py", line 133, in _create
initialTag=TAG_VOL_UNINIT)
File "/usr/share/vdsm/storage/lvm.py", line 1096, in createLV
raise se.CannotCreateLogicalVolume(vgName, lvName)
CannotCreateLogicalVolume: Cannot create Logical Volume:
('8fa5a242-fcb4-454d-aaa1-23f8dd92373b',
'9636163f-d634-4c28-b2bc-f84b5a90a17e')