5:05 a.m.
Hello
I have a problem with oVirt Hosted Engine Setup version: 4.0.5.5-1.el7.centos.
Setup is using FCP SAN for data and engine.
Cluster has worked fine for a while. It has two hosts with VMs running.
I extended storage with an additional LUN recently. This LUN seems to
be gone from data domain and one VM is paused which I assume has data
on that device.
Got these errors in events:
Apr 24, 2017 10:26:05 AM
Failed to activate Storage Domain SD (Data Center DC) by admin@internal-authz
Apr 10, 2017 3:38:08 PM
Status of host cl01 was set to Up.
Apr 10, 2017 3:38:03 PM
Host cl01 does not enforce SELinux. Current status: DISABLED
Apr 10, 2017 3:37:58 PM
Host cl01 is initializing. Message: Recovering from crash or Initializing
Apr 10, 2017 3:37:58 PM
VDSM cl01 command failed: Recovering from crash or Initializing
Apr 10, 2017 3:37:46 PM
Failed to Reconstruct Master Domain for Data Center DC.
Apr 10, 2017 3:37:46 PM
Host cl01 is not responding. Host cannot be fenced automatically
because power management for the host is disabled.
Apr 10, 2017 3:37:46 PM
VDSM cl01 command failed: Broken pipe
Apr 10, 2017 3:37:46 PM
VDSM cl01 command failed: Broken pipe
Apr 10, 2017 3:32:45 PM
Invalid status on Data Center DC. Setting Data Center status to Non
Responsive (On host cl01, Error: General Exception).
Apr 10, 2017 3:32:45 PM
VDSM cl01 command failed: [Errno 19] Could not find dm device named `[unknown]`
Apr 7, 2017 1:28:04 PM
VM HostedEngine is down with error. Exit message: resource busy:
Failed to acquire lock: error -243.
Apr 7, 2017 1:28:02 PM
Storage Pool Manager runs on Host cl01 (Address: cl01).
Apr 7, 2017 1:27:59 PM
Invalid status on Data Center DC. Setting status to Non Responsive.
Apr 7, 2017 1:27:53 PM
Host cl02 does not enforce SELinux. Current status: DISABLED
Apr 7, 2017 1:27:52 PM
Host cl01 does not enforce SELinux. Current status: DISABLED
Apr 7, 2017 1:27:49 PM
Affinity Rules Enforcement Manager started.
Apr 7, 2017 1:27:34 PM
ETL Service Started
Apr 7, 2017 1:26:01 PM
ETL Service Stopped
Apr 3, 2017 1:22:54 PM
Shutdown of VM HostedEngine failed.
Apr 3, 2017 1:22:52 PM
Storage Pool Manager runs on Host cl01 (Address: cl01).
Apr 3, 2017 1:22:49 PM
Invalid status on Data Center DC. Setting status to Non Responsive.
Master data domain is inactive.
vdsm.log:
jsonrpc.Executor/5::INFO::2017-04-20
07:01:26,796::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['ids']
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,796::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
'\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=
0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
bd616961-6da7-4eb0-939e-330b0a3fea6e/ids (cwd None)
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,880::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
WARNING: To avoid corruption, rescan devices to make changes
visible (pvscan --cache).\n Couldn't find device with uuid
jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
jsonrpc.Executor/5::INFO::2017-04-20
07:01:26,881::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['leases']
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,881::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
'\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=
0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
bd616961-6da7-4eb0-939e-330b0a3fea6e/leases (cwd None)
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,973::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
WARNING: To avoid corruption, rescan devices to make changes
visible (pvscan --cache).\n Couldn't find device with uuid
jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
jsonrpc.Executor/5::INFO::2017-04-20
07:01:26,973::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['metadata', 'leases',
'ids', 'inbox', 'outbox', 'master']
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,974::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
'\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=
0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
bd616961-6da7-4eb0-939e-330b0a3fea6e/metadata
bd616961-6da7-4eb0-939e-330b0a3fea6e/leases
bd616961-6da7-4eb0-939e-330b0a3fea6e/ids
bd616961-6da7-4eb0-939e-330b0a3fea6e/inbox b
d616961-6da7-4eb0-939e-330b0a3fea6e/outbox
bd616961-6da7-4eb0-939e-330b0a3fea6e/master (cwd None)
Reactor thread::INFO::2017-04-20
07:01:27,069::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept)
Accepting connection from ::1:44692
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,070::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
WARNING: To avoid corruption, rescan devices to make changes
visible (pvscan --cache).\n Couldn't find device with uuid
jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,070::sp::662::Storage.StoragePool::(_stopWatchingDomainsState)
Stop watching domains state
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,070::resourceManager::628::Storage.ResourceManager::(releaseResource)
Trying to release resource
'Storage.58493e81-01dc-01d8-0390-000000000032'
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::647::Storage.ResourceManager::(releaseResource)
Released resource 'Storage.58493e81-01dc-01d8-0390-000000000032' (0
active users)
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::653::Storage.ResourceManager::(releaseResource)
Resource 'Storage.58493e81-01dc-01d8-0390-000000000032' is free,
finding out if anyone is waiting for it.
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::661::Storage.ResourceManager::(releaseResource)
No one is waiting for resource
'Storage.58493e81-01dc-01d8-0390-000000000032', Clearing records.
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::628::Storage.ResourceManager::(releaseResource)
Trying to release resource 'Storage.HsmDomainMonitorLock'
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::647::Storage.ResourceManager::(releaseResource)
Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::653::Storage.ResourceManager::(releaseResource)
Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone
is waiting for it.
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::661::Storage.ResourceManager::(releaseResource)
No one is waiting for resource 'Storage.HsmDomainMonitorLock',
Clearing records.
jsonrpc.Executor/5::ERROR::2017-04-20
07:01:27,072::task::868::Storage.TaskManager.Task::(_setError)
Task=`15122a21-4fb7-45bf-9a9a-4b97f27bc1e1`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 875, in _run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 988, in connectStoragePool
spUUID, hostID, msdUUID, masterVersion, domainsMap)
File "/usr/share/vdsm/storage/hsm.py", line 1053, in _connectStoragePool
res = pool.connect(hostID, msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 646, in connect
self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1219, in __rebuild
self.setMasterDomain(msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1427, in setMasterDomain
domain = sdCache.produce(msdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 101, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 1441, in findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/blockSD.py", line 814, in __init__
lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize))
File "/usr/share/vdsm/storage/lvm.py", line 1056, in checkVGBlockSizes
_checkpvsblksize(pvs, vgBlkSize)
File "/usr/share/vdsm/storage/lvm.py", line 1033, in _checkpvsblksize
pvBlkSize = _getpvblksize(pv)
File "/usr/share/vdsm/storage/lvm.py", line 1027, in _getpvblksize
dev = devicemapper.getDmId(os.path.basename(pv))
File "/usr/share/vdsm/storage/devicemapper.py", line 40, in getDmId
deviceMultipathName)
OSError: [Errno 19] Could not find dm device named `[unknown]`
Any input how to diagnose or troubleshoot would be appreciated.
--
Best Regards
Jens Oechsler
7:17 a.m.
Hi,
Do you see the LUN in the host ?
Can you share pvs and lvs output ?
Thanks,
Fred
On Mon, Apr 24, 2017 at 1:05 PM, Jens Oechsler <joe(a)avaleo.net> wrote:
Hello
I have a problem with oVirt Hosted Engine Setup version:
4.0.5.5-1.el7.centos.
Setup is using FCP SAN for data and engine.
Cluster has worked fine for a while. It has two hosts with VMs running.
I extended storage with an additional LUN recently. This LUN seems to
be gone from data domain and one VM is paused which I assume has data
on that device.
Got these errors in events:
Apr 24, 2017 10:26:05 AM
Failed to activate Storage Domain SD (Data Center DC) by
admin@internal-authz
Apr 10, 2017 3:38:08 PM
Status of host cl01 was set to Up.
Apr 10, 2017 3:38:03 PM
Host cl01 does not enforce SELinux. Current status: DISABLED
Apr 10, 2017 3:37:58 PM
Host cl01 is initializing. Message: Recovering from crash or Initializing
Apr 10, 2017 3:37:58 PM
VDSM cl01 command failed: Recovering from crash or Initializing
Apr 10, 2017 3:37:46 PM
Failed to Reconstruct Master Domain for Data Center DC.
Apr 10, 2017 3:37:46 PM
Host cl01 is not responding. Host cannot be fenced automatically
because power management for the host is disabled.
Apr 10, 2017 3:37:46 PM
VDSM cl01 command failed: Broken pipe
Apr 10, 2017 3:37:46 PM
VDSM cl01 command failed: Broken pipe
Apr 10, 2017 3:32:45 PM
Invalid status on Data Center DC. Setting Data Center status to Non
Responsive (On host cl01, Error: General Exception).
Apr 10, 2017 3:32:45 PM
VDSM cl01 command failed: [Errno 19] Could not find dm device named
`[unknown]`
Apr 7, 2017 1:28:04 PM
VM HostedEngine is down with error. Exit message: resource busy:
Failed to acquire lock: error -243.
Apr 7, 2017 1:28:02 PM
Storage Pool Manager runs on Host cl01 (Address: cl01).
Apr 7, 2017 1:27:59 PM
Invalid status on Data Center DC. Setting status to Non Responsive.
Apr 7, 2017 1:27:53 PM
Host cl02 does not enforce SELinux. Current status: DISABLED
Apr 7, 2017 1:27:52 PM
Host cl01 does not enforce SELinux. Current status: DISABLED
Apr 7, 2017 1:27:49 PM
Affinity Rules Enforcement Manager started.
Apr 7, 2017 1:27:34 PM
ETL Service Started
Apr 7, 2017 1:26:01 PM
ETL Service Stopped
Apr 3, 2017 1:22:54 PM
Shutdown of VM HostedEngine failed.
Apr 3, 2017 1:22:52 PM
Storage Pool Manager runs on Host cl01 (Address: cl01).
Apr 3, 2017 1:22:49 PM
Invalid status on Data Center DC. Setting status to Non Responsive.
Master data domain is inactive.
vdsm.log:
jsonrpc.Executor/5::INFO::2017-04-20
07:01:26,796::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['ids']
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,796::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
'\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=
0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
bd616961-6da7-4eb0-939e-330b0a3fea6e/ids (cwd None)
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,880::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
WARNING: To avoid corruption, rescan devices to make changes
visible (pvscan --cache).\n Couldn't find device with uuid
jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
jsonrpc.Executor/5::INFO::2017-04-20
07:01:26,881::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['leases']
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,881::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
'\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=
0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
bd616961-6da7-4eb0-939e-330b0a3fea6e/leases (cwd None)
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,973::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
WARNING: To avoid corruption, rescan devices to make changes
visible (pvscan --cache).\n Couldn't find device with uuid
jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
jsonrpc.Executor/5::INFO::2017-04-20
07:01:26,973::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['metadata', 'leases',
'ids', 'inbox', 'outbox', 'master']
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:26,974::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
'\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=
0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
bd616961-6da7-4eb0-939e-330b0a3fea6e/metadata
bd616961-6da7-4eb0-939e-330b0a3fea6e/leases
bd616961-6da7-4eb0-939e-330b0a3fea6e/ids
bd616961-6da7-4eb0-939e-330b0a3fea6e/inbox b
d616961-6da7-4eb0-939e-330b0a3fea6e/outbox
bd616961-6da7-4eb0-939e-330b0a3fea6e/master (cwd None)
Reactor thread::INFO::2017-04-20
07:01:27,069::protocoldetector::72::ProtocolDetector.AcceptorImpl:
:(handle_accept)
Accepting connection from ::1:44692
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,070::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
WARNING: To avoid corruption, rescan devices to make changes
visible (pvscan --cache).\n Couldn't find device with uuid
jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,070::sp::662::Storage.StoragePool::(_stopWatchingDomainsState)
Stop watching domains state
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,070::resourceManager::628::Storage.ResourceManager:
:(releaseResource)
Trying to release resource
'Storage.58493e81-01dc-01d8-0390-000000000032'
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::647::Storage.ResourceManager:
:(releaseResource)
Released resource 'Storage.58493e81-01dc-01d8-0390-000000000032' (0
active users)
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::653::Storage.ResourceManager:
:(releaseResource)
Resource 'Storage.58493e81-01dc-01d8-0390-000000000032' is free,
finding out if anyone is waiting for it.
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::661::Storage.ResourceManager:
:(releaseResource)
No one is waiting for resource
'Storage.58493e81-01dc-01d8-0390-000000000032', Clearing records.
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::628::Storage.ResourceManager:
:(releaseResource)
Trying to release resource 'Storage.HsmDomainMonitorLock'
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::647::Storage.ResourceManager:
:(releaseResource)
Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::653::Storage.ResourceManager:
:(releaseResource)
Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone
is waiting for it.
jsonrpc.Executor/5::DEBUG::2017-04-20
07:01:27,071::resourceManager::661::Storage.ResourceManager:
:(releaseResource)
No one is waiting for resource 'Storage.HsmDomainMonitorLock',
Clearing records.
jsonrpc.Executor/5::ERROR::2017-04-20
07:01:27,072::task::868::Storage.TaskManager.Task::(_setError)
Task=`15122a21-4fb7-45bf-9a9a-4b97f27bc1e1`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 875, in _run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in
wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 988, in connectStoragePool
spUUID, hostID, msdUUID, masterVersion, domainsMap)
File "/usr/share/vdsm/storage/hsm.py", line 1053, in _connectStoragePool
res = pool.connect(hostID, msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 646, in connect
self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1219, in __rebuild
self.setMasterDomain(msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1427, in setMasterDomain
domain = sdCache.produce(msdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 101, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 1441, in findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/blockSD.py", line 814, in __init__
lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize))
File "/usr/share/vdsm/storage/lvm.py", line 1056, in checkVGBlockSizes
_checkpvsblksize(pvs, vgBlkSize)
File "/usr/share/vdsm/storage/lvm.py", line 1033, in _checkpvsblksize
pvBlkSize = _getpvblksize(pv)
File "/usr/share/vdsm/storage/lvm.py", line 1027, in _getpvblksize
dev = devicemapper.getDmId(os.path.basename(pv))
File "/usr/share/vdsm/storage/devicemapper.py", line 40, in getDmId
deviceMultipathName)
OSError: [Errno 19] Could not find dm device named `[unknown]`
Any input how to diagnose or troubleshoot would be appreciated.
--
Best Regards
Jens Oechsler
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9:11 a.m.
Hi,
LUN is not in pvs output, but I found it in lsblk output without any
partions on it apparently.
$ sudo pvs
PV VG
Fmt Attr PSize PFree
/dev/mapper/360050768018182b6c000000000000990 data
lvm2 a-- 200.00g 180.00g
/dev/mapper/360050768018182b6c000000000000998
9f10e00f-ae39-46a0-86da-8b157c6de7bc lvm2 a-- 499.62g 484.50g
/dev/sda2 system
lvm2 a-- 278.78g 208.41g
$ sudo lvs
LV VG
Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync
Convert
34a9328f-87fe-4190-96e9-a3580b0734fc
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 1.00g
506ff043-1058-448c-bbab-5c864adb2bfc
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 10.00g
65449c88-bc28-4275-bbbb-5fc75b692cbc
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
e2ee95ce-8105-4a20-8e1f-9f6dfa16bf59
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-ao---- 128.00m
ids
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-ao---- 128.00m
inbox
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
leases
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 2.00g
master
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 1.00g
metadata
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 512.00m
outbox
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
data data
-wi-ao---- 20.00g
home system
-wi-ao---- 1000.00m
prod system
-wi-ao---- 4.88g
root system
-wi-ao---- 7.81g
swap system
-wi-ao---- 4.00g
swap7 system
-wi-ao---- 20.00g
tmp system
-wi-ao---- 4.88g
var system
-wi-ao---- 27.81g
$ sudo lsblk
<output trimmed>
sdq
65:0 0 500G 0 disk
└─360050768018182b6c0000000000009d7
253:33 0 500G 0 mpath
Data domain was made with one 500 GB LUN and extended with 500 GB more.
On Tue, Apr 25, 2017 at 2:17 PM, Fred Rolland <frolland(a)redhat.com> wrote:
Hi,
Do you see the LUN in the host ?
Can you share pvs and lvs output ?
Thanks,
Fred
On Mon, Apr 24, 2017 at 1:05 PM, Jens Oechsler <joe(a)avaleo.net> wrote:
>
> Hello
> I have a problem with oVirt Hosted Engine Setup version:
> 4.0.5.5-1.el7.centos.
> Setup is using FCP SAN for data and engine.
> Cluster has worked fine for a while. It has two hosts with VMs running.
> I extended storage with an additional LUN recently. This LUN seems to
> be gone from data domain and one VM is paused which I assume has data
> on that device.
>
> Got these errors in events:
>
> Apr 24, 2017 10:26:05 AM
> Failed to activate Storage Domain SD (Data Center DC) by
> admin@internal-authz
> Apr 10, 2017 3:38:08 PM
> Status of host cl01 was set to Up.
> Apr 10, 2017 3:38:03 PM
> Host cl01 does not enforce SELinux. Current status: DISABLED
> Apr 10, 2017 3:37:58 PM
> Host cl01 is initializing. Message: Recovering from crash or Initializing
> Apr 10, 2017 3:37:58 PM
> VDSM cl01 command failed: Recovering from crash or Initializing
> Apr 10, 2017 3:37:46 PM
> Failed to Reconstruct Master Domain for Data Center DC.
> Apr 10, 2017 3:37:46 PM
> Host cl01 is not responding. Host cannot be fenced automatically
> because power management for the host is disabled.
> Apr 10, 2017 3:37:46 PM
> VDSM cl01 command failed: Broken pipe
> Apr 10, 2017 3:37:46 PM
> VDSM cl01 command failed: Broken pipe
> Apr 10, 2017 3:32:45 PM
> Invalid status on Data Center DC. Setting Data Center status to Non
> Responsive (On host cl01, Error: General Exception).
> Apr 10, 2017 3:32:45 PM
> VDSM cl01 command failed: [Errno 19] Could not find dm device named
> `[unknown]`
> Apr 7, 2017 1:28:04 PM
> VM HostedEngine is down with error. Exit message: resource busy:
> Failed to acquire lock: error -243.
> Apr 7, 2017 1:28:02 PM
> Storage Pool Manager runs on Host cl01 (Address: cl01).
> Apr 7, 2017 1:27:59 PM
> Invalid status on Data Center DC. Setting status to Non Responsive.
> Apr 7, 2017 1:27:53 PM
> Host cl02 does not enforce SELinux. Current status: DISABLED
> Apr 7, 2017 1:27:52 PM
> Host cl01 does not enforce SELinux. Current status: DISABLED
> Apr 7, 2017 1:27:49 PM
> Affinity Rules Enforcement Manager started.
> Apr 7, 2017 1:27:34 PM
> ETL Service Started
> Apr 7, 2017 1:26:01 PM
> ETL Service Stopped
> Apr 3, 2017 1:22:54 PM
> Shutdown of VM HostedEngine failed.
> Apr 3, 2017 1:22:52 PM
> Storage Pool Manager runs on Host cl01 (Address: cl01).
> Apr 3, 2017 1:22:49 PM
> Invalid status on Data Center DC. Setting status to Non Responsive.
>
>
> Master data domain is inactive.
>
>
> vdsm.log:
>
> jsonrpc.Executor/5::INFO::2017-04-20
> 07:01:26,796::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['ids']
> jsonrpc.Executor/5::DEBUG::2017-04-20
> 07:01:26,796::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
> wait_for_locks=1 use_lvmetad=
> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
> bd616961-6da7-4eb0-939e-330b0a3fea6e/ids (cwd None)
> jsonrpc.Executor/5::DEBUG::2017-04-20
> 07:01:26,880::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
> WARNING: To avoid corruption, rescan devices to make changes
> visible (pvscan --cache).\n Couldn't find device with uuid
> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
> jsonrpc.Executor/5::INFO::2017-04-20
> 07:01:26,881::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['leases']
> jsonrpc.Executor/5::DEBUG::2017-04-20
> 07:01:26,881::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
> wait_for_locks=1 use_lvmetad=
> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
> bd616961-6da7-4eb0-939e-330b0a3fea6e/leases (cwd None)
> jsonrpc.Executor/5::DEBUG::2017-04-20
> 07:01:26,973::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
> WARNING: To avoid corruption, rescan devices to make changes
> visible (pvscan --cache).\n Couldn't find device with uuid
> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
> jsonrpc.Executor/5::INFO::2017-04-20
> 07:01:26,973::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['metadata', 'leases',
> 'ids', 'inbox', 'outbox', 'master']
> jsonrpc.Executor/5::DEBUG::2017-04-20
> 07:01:26,974::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
> wait_for_locks=1 use_lvmetad=
> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
> bd616961-6da7-4eb0-939e-330b0a3fea6e/metadata
> bd616961-6da7-4eb0-939e-330b0a3fea6e/leases
> bd616961-6da7-4eb0-939e-330b0a3fea6e/ids
> bd616961-6da7-4eb0-939e-330b0a3fea6e/inbox b
> d616961-6da7-4eb0-939e-330b0a3fea6e/outbox
> bd616961-6da7-4eb0-939e-330b0a3fea6e/master (cwd None)
> Reactor thread::INFO::2017-04-20
>
> 07:01:27,069::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept)
> Accepting connection from ::1:44692
> jsonrpc.Executor/5::DEBUG::2017-04-20
> 07:01:27,070::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
> WARNING: To avoid corruption, rescan devices to make changes
> visible (pvscan --cache).\n Couldn't find device with uuid
> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
> jsonrpc.Executor/5::DEBUG::2017-04-20
> 07:01:27,070::sp::662::Storage.StoragePool::(_stopWatchingDomainsState)
> Stop watching domains state
> jsonrpc.Executor/5::DEBUG::2017-04-20
>
> 07:01:27,070::resourceManager::628::Storage.ResourceManager::(releaseResource)
> Trying to release resource
> 'Storage.58493e81-01dc-01d8-0390-000000000032'
> jsonrpc.Executor/5::DEBUG::2017-04-20
>
> 07:01:27,071::resourceManager::647::Storage.ResourceManager::(releaseResource)
> Released resource 'Storage.58493e81-01dc-01d8-0390-000000000032' (0
> active users)
> jsonrpc.Executor/5::DEBUG::2017-04-20
>
> 07:01:27,071::resourceManager::653::Storage.ResourceManager::(releaseResource)
> Resource 'Storage.58493e81-01dc-01d8-0390-000000000032' is free,
> finding out if anyone is waiting for it.
> jsonrpc.Executor/5::DEBUG::2017-04-20
>
> 07:01:27,071::resourceManager::661::Storage.ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.58493e81-01dc-01d8-0390-000000000032', Clearing records.
> jsonrpc.Executor/5::DEBUG::2017-04-20
>
> 07:01:27,071::resourceManager::628::Storage.ResourceManager::(releaseResource)
> Trying to release resource 'Storage.HsmDomainMonitorLock'
> jsonrpc.Executor/5::DEBUG::2017-04-20
>
> 07:01:27,071::resourceManager::647::Storage.ResourceManager::(releaseResource)
> Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
> jsonrpc.Executor/5::DEBUG::2017-04-20
>
> 07:01:27,071::resourceManager::653::Storage.ResourceManager::(releaseResource)
> Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone
> is waiting for it.
> jsonrpc.Executor/5::DEBUG::2017-04-20
>
> 07:01:27,071::resourceManager::661::Storage.ResourceManager::(releaseResource)
> No one is waiting for resource 'Storage.HsmDomainMonitorLock',
> Clearing records.
> jsonrpc.Executor/5::ERROR::2017-04-20
> 07:01:27,072::task::868::Storage.TaskManager.Task::(_setError)
> Task=`15122a21-4fb7-45bf-9a9a-4b97f27bc1e1`::Unexpected error
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/task.py", line 875, in _run
> return fn(*args, **kargs)
> File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in
> wrapper
> res = f(*args, **kwargs)
> File "/usr/share/vdsm/storage/hsm.py", line 988, in connectStoragePool
> spUUID, hostID, msdUUID, masterVersion, domainsMap)
> File "/usr/share/vdsm/storage/hsm.py", line 1053, in _connectStoragePool
> res = pool.connect(hostID, msdUUID, masterVersion)
> File "/usr/share/vdsm/storage/sp.py", line 646, in connect
> self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
> File "/usr/share/vdsm/storage/sp.py", line 1219, in __rebuild
> self.setMasterDomain(msdUUID, masterVersion)
> File "/usr/share/vdsm/storage/sp.py", line 1427, in setMasterDomain
> domain = sdCache.produce(msdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 101, in produce
> domain.getRealDomain()
> File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain
> return self._cache._realProduce(self._sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce
> domain = self._findDomain(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/blockSD.py", line 1441, in findDomain
> return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
> File "/usr/share/vdsm/storage/blockSD.py", line 814, in __init__
> lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize))
> File "/usr/share/vdsm/storage/lvm.py", line 1056, in checkVGBlockSizes
> _checkpvsblksize(pvs, vgBlkSize)
> File "/usr/share/vdsm/storage/lvm.py", line 1033, in _checkpvsblksize
> pvBlkSize = _getpvblksize(pv)
> File "/usr/share/vdsm/storage/lvm.py", line 1027, in _getpvblksize
> dev = devicemapper.getDmId(os.path.basename(pv))
> File "/usr/share/vdsm/storage/devicemapper.py", line 40, in getDmId
> deviceMultipathName)
> OSError: [Errno 19] Could not find dm device named `[unknown]`
>
>
> Any input how to diagnose or troubleshoot would be appreciated.
>
> --
> Best Regards
>
> Jens Oechsler
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--
Med Venlig Hilsen / Best Regards
Jens Oechsler
System administrator
KMD Nexus
+45 51 82 62 13
9:30 a.m.
Greetings,
Is there any way to get the oVirt Data Center described below active again?
On Tue, Apr 25, 2017 at 4:11 PM, Jens Oechsler <joe(a)avaleo.net> wrote:
Hi,
LUN is not in pvs output, but I found it in lsblk output without any
partions on it apparently.
$ sudo pvs
PV VG
Fmt Attr PSize PFree
/dev/mapper/360050768018182b6c000000000000990 data
lvm2 a-- 200.00g 180.00g
/dev/mapper/360050768018182b6c000000000000998
9f10e00f-ae39-46a0-86da-8b157c6de7bc lvm2 a-- 499.62g 484.50g
/dev/sda2 system
lvm2 a-- 278.78g 208.41g
$ sudo lvs
LV VG
Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync
Convert
34a9328f-87fe-4190-96e9-a3580b0734fc
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 1.00g
506ff043-1058-448c-bbab-5c864adb2bfc
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 10.00g
65449c88-bc28-4275-bbbb-5fc75b692cbc
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
e2ee95ce-8105-4a20-8e1f-9f6dfa16bf59
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-ao---- 128.00m
ids
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-ao---- 128.00m
inbox
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
leases
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 2.00g
master
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 1.00g
metadata
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 512.00m
outbox
9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
data data
-wi-ao---- 20.00g
home system
-wi-ao---- 1000.00m
prod system
-wi-ao---- 4.88g
root system
-wi-ao---- 7.81g
swap system
-wi-ao---- 4.00g
swap7 system
-wi-ao---- 20.00g
tmp system
-wi-ao---- 4.88g
var system
-wi-ao---- 27.81g
$ sudo lsblk
<output trimmed>
sdq
65:0 0 500G 0 disk
└─360050768018182b6c0000000000009d7
253:33 0 500G 0 mpath
Data domain was made with one 500 GB LUN and extended with 500 GB more.
On Tue, Apr 25, 2017 at 2:17 PM, Fred Rolland <frolland(a)redhat.com> wrote:
> Hi,
>
> Do you see the LUN in the host ?
> Can you share pvs and lvs output ?
>
> Thanks,
>
> Fred
>
> On Mon, Apr 24, 2017 at 1:05 PM, Jens Oechsler <joe(a)avaleo.net> wrote:
>>
>> Hello
>> I have a problem with oVirt Hosted Engine Setup version:
>> 4.0.5.5-1.el7.centos.
>> Setup is using FCP SAN for data and engine.
>> Cluster has worked fine for a while. It has two hosts with VMs running.
>> I extended storage with an additional LUN recently. This LUN seems to
>> be gone from data domain and one VM is paused which I assume has data
>> on that device.
>>
>> Got these errors in events:
>>
>> Apr 24, 2017 10:26:05 AM
>> Failed to activate Storage Domain SD (Data Center DC) by
>> admin@internal-authz
>> Apr 10, 2017 3:38:08 PM
>> Status of host cl01 was set to Up.
>> Apr 10, 2017 3:38:03 PM
>> Host cl01 does not enforce SELinux. Current status: DISABLED
>> Apr 10, 2017 3:37:58 PM
>> Host cl01 is initializing. Message: Recovering from crash or Initializing
>> Apr 10, 2017 3:37:58 PM
>> VDSM cl01 command failed: Recovering from crash or Initializing
>> Apr 10, 2017 3:37:46 PM
>> Failed to Reconstruct Master Domain for Data Center DC.
>> Apr 10, 2017 3:37:46 PM
>> Host cl01 is not responding. Host cannot be fenced automatically
>> because power management for the host is disabled.
>> Apr 10, 2017 3:37:46 PM
>> VDSM cl01 command failed: Broken pipe
>> Apr 10, 2017 3:37:46 PM
>> VDSM cl01 command failed: Broken pipe
>> Apr 10, 2017 3:32:45 PM
>> Invalid status on Data Center DC. Setting Data Center status to Non
>> Responsive (On host cl01, Error: General Exception).
>> Apr 10, 2017 3:32:45 PM
>> VDSM cl01 command failed: [Errno 19] Could not find dm device named
>> `[unknown]`
>> Apr 7, 2017 1:28:04 PM
>> VM HostedEngine is down with error. Exit message: resource busy:
>> Failed to acquire lock: error -243.
>> Apr 7, 2017 1:28:02 PM
>> Storage Pool Manager runs on Host cl01 (Address: cl01).
>> Apr 7, 2017 1:27:59 PM
>> Invalid status on Data Center DC. Setting status to Non Responsive.
>> Apr 7, 2017 1:27:53 PM
>> Host cl02 does not enforce SELinux. Current status: DISABLED
>> Apr 7, 2017 1:27:52 PM
>> Host cl01 does not enforce SELinux. Current status: DISABLED
>> Apr 7, 2017 1:27:49 PM
>> Affinity Rules Enforcement Manager started.
>> Apr 7, 2017 1:27:34 PM
>> ETL Service Started
>> Apr 7, 2017 1:26:01 PM
>> ETL Service Stopped
>> Apr 3, 2017 1:22:54 PM
>> Shutdown of VM HostedEngine failed.
>> Apr 3, 2017 1:22:52 PM
>> Storage Pool Manager runs on Host cl01 (Address: cl01).
>> Apr 3, 2017 1:22:49 PM
>> Invalid status on Data Center DC. Setting status to Non Responsive.
>>
>>
>> Master data domain is inactive.
>>
>>
>> vdsm.log:
>>
>> jsonrpc.Executor/5::INFO::2017-04-20
>> 07:01:26,796::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
>> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['ids']
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>> 07:01:26,796::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
>> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
>>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
>> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
>> wait_for_locks=1 use_lvmetad=
>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
>> bd616961-6da7-4eb0-939e-330b0a3fea6e/ids (cwd None)
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>> 07:01:26,880::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
>> WARNING: To avoid corruption, rescan devices to make changes
>> visible (pvscan --cache).\n Couldn't find device with uuid
>> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
>> jsonrpc.Executor/5::INFO::2017-04-20
>> 07:01:26,881::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
>> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['leases']
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>> 07:01:26,881::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
>> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
>>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
>> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
>> wait_for_locks=1 use_lvmetad=
>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
>> bd616961-6da7-4eb0-939e-330b0a3fea6e/leases (cwd None)
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>> 07:01:26,973::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
>> WARNING: To avoid corruption, rescan devices to make changes
>> visible (pvscan --cache).\n Couldn't find device with uuid
>> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
>> jsonrpc.Executor/5::INFO::2017-04-20
>> 07:01:26,973::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
>> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['metadata',
'leases',
>> 'ids', 'inbox', 'outbox', 'master']
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>> 07:01:26,974::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
>> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
>>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
>> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
>> wait_for_locks=1 use_lvmetad=
>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
>> bd616961-6da7-4eb0-939e-330b0a3fea6e/metadata
>> bd616961-6da7-4eb0-939e-330b0a3fea6e/leases
>> bd616961-6da7-4eb0-939e-330b0a3fea6e/ids
>> bd616961-6da7-4eb0-939e-330b0a3fea6e/inbox b
>> d616961-6da7-4eb0-939e-330b0a3fea6e/outbox
>> bd616961-6da7-4eb0-939e-330b0a3fea6e/master (cwd None)
>> Reactor thread::INFO::2017-04-20
>>
>>
07:01:27,069::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept)
>> Accepting connection from ::1:44692
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>> 07:01:27,070::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "
>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
>> WARNING: To avoid corruption, rescan devices to make changes
>> visible (pvscan --cache).\n Couldn't find device with uuid
>> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>> 07:01:27,070::sp::662::Storage.StoragePool::(_stopWatchingDomainsState)
>> Stop watching domains state
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>
>> 07:01:27,070::resourceManager::628::Storage.ResourceManager::(releaseResource)
>> Trying to release resource
>> 'Storage.58493e81-01dc-01d8-0390-000000000032'
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>
>> 07:01:27,071::resourceManager::647::Storage.ResourceManager::(releaseResource)
>> Released resource 'Storage.58493e81-01dc-01d8-0390-000000000032' (0
>> active users)
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>
>> 07:01:27,071::resourceManager::653::Storage.ResourceManager::(releaseResource)
>> Resource 'Storage.58493e81-01dc-01d8-0390-000000000032' is free,
>> finding out if anyone is waiting for it.
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>
>> 07:01:27,071::resourceManager::661::Storage.ResourceManager::(releaseResource)
>> No one is waiting for resource
>> 'Storage.58493e81-01dc-01d8-0390-000000000032', Clearing records.
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>
>> 07:01:27,071::resourceManager::628::Storage.ResourceManager::(releaseResource)
>> Trying to release resource 'Storage.HsmDomainMonitorLock'
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>
>> 07:01:27,071::resourceManager::647::Storage.ResourceManager::(releaseResource)
>> Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>
>> 07:01:27,071::resourceManager::653::Storage.ResourceManager::(releaseResource)
>> Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone
>> is waiting for it.
>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>
>> 07:01:27,071::resourceManager::661::Storage.ResourceManager::(releaseResource)
>> No one is waiting for resource 'Storage.HsmDomainMonitorLock',
>> Clearing records.
>> jsonrpc.Executor/5::ERROR::2017-04-20
>> 07:01:27,072::task::868::Storage.TaskManager.Task::(_setError)
>> Task=`15122a21-4fb7-45bf-9a9a-4b97f27bc1e1`::Unexpected error
>> Traceback (most recent call last):
>> File "/usr/share/vdsm/storage/task.py", line 875, in _run
>> return fn(*args, **kargs)
>> File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50,
in
>> wrapper
>> res = f(*args, **kwargs)
>> File "/usr/share/vdsm/storage/hsm.py", line 988, in
connectStoragePool
>> spUUID, hostID, msdUUID, masterVersion, domainsMap)
>> File "/usr/share/vdsm/storage/hsm.py", line 1053, in
_connectStoragePool
>> res = pool.connect(hostID, msdUUID, masterVersion)
>> File "/usr/share/vdsm/storage/sp.py", line 646, in connect
>> self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>> File "/usr/share/vdsm/storage/sp.py", line 1219, in __rebuild
>> self.setMasterDomain(msdUUID, masterVersion)
>> File "/usr/share/vdsm/storage/sp.py", line 1427, in setMasterDomain
>> domain = sdCache.produce(msdUUID)
>> File "/usr/share/vdsm/storage/sdc.py", line 101, in produce
>> domain.getRealDomain()
>> File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain
>> return self._cache._realProduce(self._sdUUID)
>> File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce
>> domain = self._findDomain(sdUUID)
>> File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain
>> dom = findMethod(sdUUID)
>> File "/usr/share/vdsm/storage/blockSD.py", line 1441, in findDomain
>> return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
>> File "/usr/share/vdsm/storage/blockSD.py", line 814, in __init__
>> lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize))
>> File "/usr/share/vdsm/storage/lvm.py", line 1056, in
checkVGBlockSizes
>> _checkpvsblksize(pvs, vgBlkSize)
>> File "/usr/share/vdsm/storage/lvm.py", line 1033, in _checkpvsblksize
>> pvBlkSize = _getpvblksize(pv)
>> File "/usr/share/vdsm/storage/lvm.py", line 1027, in _getpvblksize
>> dev = devicemapper.getDmId(os.path.basename(pv))
>> File "/usr/share/vdsm/storage/devicemapper.py", line 40, in getDmId
>> deviceMultipathName)
>> OSError: [Errno 19] Could not find dm device named `[unknown]`
>>
>>
>> Any input how to diagnose or troubleshoot would be appreciated.
>>
>> --
>> Best Regards
>>
>> Jens Oechsler
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
--
Med Venlig Hilsen / Best Regards
Jens Oechsler
System administrator
KMD Nexus
+45 51 82 62 13
--
Med Venlig Hilsen / Best Regards
Jens Oechsler
System administrator
KMD Nexus
+45 51 82 62 13
3:50 a.m.
Hi,
Can you provide the vdsm and engine logs ?
Thanks,
Fred
On Wed, Apr 26, 2017 at 5:30 PM, Jens Oechsler <joe(a)avaleo.net> wrote:
Greetings,
Is there any way to get the oVirt Data Center described below active again?
On Tue, Apr 25, 2017 at 4:11 PM, Jens Oechsler <joe(a)avaleo.net> wrote:
> Hi,
>
> LUN is not in pvs output, but I found it in lsblk output without any
> partions on it apparently.
>
> $ sudo pvs
> PV VG
> Fmt Attr PSize PFree
> /dev/mapper/360050768018182b6c000000000000990 data
> lvm2 a-- 200.00g 180.00g
> /dev/mapper/360050768018182b6c000000000000998
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc lvm2 a-- 499.62g 484.50g
> /dev/sda2 system
> lvm2 a-- 278.78g 208.41g
>
> $ sudo lvs
> LV VG
> Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync
> Convert
> 34a9328f-87fe-4190-96e9-a3580b0734fc
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 1.00g
> 506ff043-1058-448c-bbab-5c864adb2bfc
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 10.00g
> 65449c88-bc28-4275-bbbb-5fc75b692cbc
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
> e2ee95ce-8105-4a20-8e1f-9f6dfa16bf59
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-ao---- 128.00m
> ids
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-ao---- 128.00m
> inbox
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
> leases
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 2.00g
> master
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 1.00g
> metadata
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 512.00m
> outbox
> 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
> data data
> -wi-ao---- 20.00g
> home system
> -wi-ao---- 1000.00m
> prod system
> -wi-ao---- 4.88g
> root system
> -wi-ao---- 7.81g
> swap system
> -wi-ao---- 4.00g
> swap7 system
> -wi-ao---- 20.00g
> tmp system
> -wi-ao---- 4.88g
> var system
> -wi-ao---- 27.81g
>
> $ sudo lsblk
> <output trimmed>
> sdq
> 65:0 0 500G 0 disk
> └─360050768018182b6c0000000000009d7
> 253:33 0 500G 0 mpath
>
> Data domain was made with one 500 GB LUN and extended with 500 GB more.
>
> On Tue, Apr 25, 2017 at 2:17 PM, Fred Rolland <frolland(a)redhat.com>
wrote:
>> Hi,
>>
>> Do you see the LUN in the host ?
>> Can you share pvs and lvs output ?
>>
>> Thanks,
>>
>> Fred
>>
>> On Mon, Apr 24, 2017 at 1:05 PM, Jens Oechsler <joe(a)avaleo.net> wrote:
>>>
>>> Hello
>>> I have a problem with oVirt Hosted Engine Setup version:
>>> 4.0.5.5-1.el7.centos.
>>> Setup is using FCP SAN for data and engine.
>>> Cluster has worked fine for a while. It has two hosts with VMs running.
>>> I extended storage with an additional LUN recently. This LUN seems to
>>> be gone from data domain and one VM is paused which I assume has data
>>> on that device.
>>>
>>> Got these errors in events:
>>>
>>> Apr 24, 2017 10:26:05 AM
>>> Failed to activate Storage Domain SD (Data Center DC) by
>>> admin@internal-authz
>>> Apr 10, 2017 3:38:08 PM
>>> Status of host cl01 was set to Up.
>>> Apr 10, 2017 3:38:03 PM
>>> Host cl01 does not enforce SELinux. Current status: DISABLED
>>> Apr 10, 2017 3:37:58 PM
>>> Host cl01 is initializing. Message: Recovering from crash or
Initializing
>>> Apr 10, 2017 3:37:58 PM
>>> VDSM cl01 command failed: Recovering from crash or Initializing
>>> Apr 10, 2017 3:37:46 PM
>>> Failed to Reconstruct Master Domain for Data Center DC.
>>> Apr 10, 2017 3:37:46 PM
>>> Host cl01 is not responding. Host cannot be fenced automatically
>>> because power management for the host is disabled.
>>> Apr 10, 2017 3:37:46 PM
>>> VDSM cl01 command failed: Broken pipe
>>> Apr 10, 2017 3:37:46 PM
>>> VDSM cl01 command failed: Broken pipe
>>> Apr 10, 2017 3:32:45 PM
>>> Invalid status on Data Center DC. Setting Data Center status to Non
>>> Responsive (On host cl01, Error: General Exception).
>>> Apr 10, 2017 3:32:45 PM
>>> VDSM cl01 command failed: [Errno 19] Could not find dm device named
>>> `[unknown]`
>>> Apr 7, 2017 1:28:04 PM
>>> VM HostedEngine is down with error. Exit message: resource busy:
>>> Failed to acquire lock: error -243.
>>> Apr 7, 2017 1:28:02 PM
>>> Storage Pool Manager runs on Host cl01 (Address: cl01).
>>> Apr 7, 2017 1:27:59 PM
>>> Invalid status on Data Center DC. Setting status to Non Responsive.
>>> Apr 7, 2017 1:27:53 PM
>>> Host cl02 does not enforce SELinux. Current status: DISABLED
>>> Apr 7, 2017 1:27:52 PM
>>> Host cl01 does not enforce SELinux. Current status: DISABLED
>>> Apr 7, 2017 1:27:49 PM
>>> Affinity Rules Enforcement Manager started.
>>> Apr 7, 2017 1:27:34 PM
>>> ETL Service Started
>>> Apr 7, 2017 1:26:01 PM
>>> ETL Service Stopped
>>> Apr 3, 2017 1:22:54 PM
>>> Shutdown of VM HostedEngine failed.
>>> Apr 3, 2017 1:22:52 PM
>>> Storage Pool Manager runs on Host cl01 (Address: cl01).
>>> Apr 3, 2017 1:22:49 PM
>>> Invalid status on Data Center DC. Setting status to Non Responsive.
>>>
>>>
>>> Master data domain is inactive.
>>>
>>>
>>> vdsm.log:
>>>
>>> jsonrpc.Executor/5::INFO::2017-04-20
>>> 07:01:26,796::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
>>> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['ids']
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>> 07:01:26,796::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
>>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
>>> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
>>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
>>>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
>>> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
>>> wait_for_locks=1 use_lvmetad=
>>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
>>> bd616961-6da7-4eb0-939e-330b0a3fea6e/ids (cwd None)
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>> 07:01:26,880::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> =
"
>>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
>>> WARNING: To avoid corruption, rescan devices to make changes
>>> visible (pvscan --cache).\n Couldn't find device with uuid
>>> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
>>> jsonrpc.Executor/5::INFO::2017-04-20
>>> 07:01:26,881::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
>>> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['leases']
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>> 07:01:26,881::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
>>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
>>> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
>>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
>>>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
>>> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
>>> wait_for_locks=1 use_lvmetad=
>>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
>>> bd616961-6da7-4eb0-939e-330b0a3fea6e/leases (cwd None)
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>> 07:01:26,973::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> =
"
>>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
>>> WARNING: To avoid corruption, rescan devices to make changes
>>> visible (pvscan --cache).\n Couldn't find device with uuid
>>> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
>>> jsonrpc.Executor/5::INFO::2017-04-20
>>> 07:01:26,973::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
>>> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['metadata',
'leases',
>>> 'ids', 'inbox', 'outbox', 'master']
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>> 07:01:26,974::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
>>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
>>> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d
>>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
>>>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
>>> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
>>> wait_for_locks=1 use_lvmetad=
>>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
>>> bd616961-6da7-4eb0-939e-330b0a3fea6e/metadata
>>> bd616961-6da7-4eb0-939e-330b0a3fea6e/leases
>>> bd616961-6da7-4eb0-939e-330b0a3fea6e/ids
>>> bd616961-6da7-4eb0-939e-330b0a3fea6e/inbox b
>>> d616961-6da7-4eb0-939e-330b0a3fea6e/outbox
>>> bd616961-6da7-4eb0-939e-330b0a3fea6e/master (cwd None)
>>> Reactor thread::INFO::2017-04-20
>>>
>>> 07:01:27,069::protocoldetector::72::ProtocolDetector.AcceptorImpl:
:(handle_accept)
>>> Accepting connection from ::1:44692
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>> 07:01:27,070::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> =
"
>>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
>>> WARNING: To avoid corruption, rescan devices to make changes
>>> visible (pvscan --cache).\n Couldn't find device with uuid
>>> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>> 07:01:27,070::sp::662::Storage.StoragePool::(_
stopWatchingDomainsState)
>>> Stop watching domains state
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>>
>>> 07:01:27,070::resourceManager::628::Storage.ResourceManager:
:(releaseResource)
>>> Trying to release resource
>>> 'Storage.58493e81-01dc-01d8-0390-000000000032'
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>>
>>> 07:01:27,071::resourceManager::647::Storage.ResourceManager:
:(releaseResource)
>>> Released resource 'Storage.58493e81-01dc-01d8-0390-000000000032' (0
>>> active users)
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>>
>>> 07:01:27,071::resourceManager::653::Storage.ResourceManager:
:(releaseResource)
>>> Resource 'Storage.58493e81-01dc-01d8-0390-000000000032' is free,
>>> finding out if anyone is waiting for it.
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>>
>>> 07:01:27,071::resourceManager::661::Storage.ResourceManager:
:(releaseResource)
>>> No one is waiting for resource
>>> 'Storage.58493e81-01dc-01d8-0390-000000000032', Clearing records.
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>>
>>> 07:01:27,071::resourceManager::628::Storage.ResourceManager:
:(releaseResource)
>>> Trying to release resource 'Storage.HsmDomainMonitorLock'
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>>
>>> 07:01:27,071::resourceManager::647::Storage.ResourceManager:
:(releaseResource)
>>> Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>>
>>> 07:01:27,071::resourceManager::653::Storage.ResourceManager:
:(releaseResource)
>>> Resource 'Storage.HsmDomainMonitorLock' is free, finding out if
anyone
>>> is waiting for it.
>>> jsonrpc.Executor/5::DEBUG::2017-04-20
>>>
>>> 07:01:27,071::resourceManager::661::Storage.ResourceManager:
:(releaseResource)
>>> No one is waiting for resource 'Storage.HsmDomainMonitorLock',
>>> Clearing records.
>>> jsonrpc.Executor/5::ERROR::2017-04-20
>>> 07:01:27,072::task::868::Storage.TaskManager.Task::(_setError)
>>> Task=`15122a21-4fb7-45bf-9a9a-4b97f27bc1e1`::Unexpected error
>>> Traceback (most recent call last):
>>> File "/usr/share/vdsm/storage/task.py", line 875, in _run
>>> return fn(*args, **kargs)
>>> File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line
50,
in
>>> wrapper
>>> res = f(*args, **kwargs)
>>> File "/usr/share/vdsm/storage/hsm.py", line 988, in
connectStoragePool
>>> spUUID, hostID, msdUUID, masterVersion, domainsMap)
>>> File "/usr/share/vdsm/storage/hsm.py", line 1053, in
_connectStoragePool
>>> res = pool.connect(hostID, msdUUID, masterVersion)
>>> File "/usr/share/vdsm/storage/sp.py", line 646, in connect
>>> self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>>> File "/usr/share/vdsm/storage/sp.py", line 1219, in __rebuild
>>> self.setMasterDomain(msdUUID, masterVersion)
>>> File "/usr/share/vdsm/storage/sp.py", line 1427, in
setMasterDomain
>>> domain = sdCache.produce(msdUUID)
>>> File "/usr/share/vdsm/storage/sdc.py", line 101, in produce
>>> domain.getRealDomain()
>>> File "/usr/share/vdsm/storage/sdc.py", line 53, in
getRealDomain
>>> return self._cache._realProduce(self._sdUUID)
>>> File "/usr/share/vdsm/storage/sdc.py", line 125, in
_realProduce
>>> domain = self._findDomain(sdUUID)
>>> File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain
>>> dom = findMethod(sdUUID)
>>> File "/usr/share/vdsm/storage/blockSD.py", line 1441, in
findDomain
>>> return BlockStorageDomain(BlockStorageDomain.
findDomainPath(sdUUID))
>>> File "/usr/share/vdsm/storage/blockSD.py", line 814, in
__init__
>>> lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize))
>>> File "/usr/share/vdsm/storage/lvm.py", line 1056, in
checkVGBlockSizes
>>> _checkpvsblksize(pvs, vgBlkSize)
>>> File "/usr/share/vdsm/storage/lvm.py", line 1033, in
_checkpvsblksize
>>> pvBlkSize = _getpvblksize(pv)
>>> File "/usr/share/vdsm/storage/lvm.py", line 1027, in
_getpvblksize
>>> dev = devicemapper.getDmId(os.path.basename(pv))
>>> File "/usr/share/vdsm/storage/devicemapper.py", line 40, in
getDmId
>>> deviceMultipathName)
>>> OSError: [Errno 19] Could not find dm device named `[unknown]`
>>>
>>>
>>> Any input how to diagnose or troubleshoot would be appreciated.
>>>
>>> --
>>> Best Regards
>>>
>>> Jens Oechsler
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
>
>
> --
> Med Venlig Hilsen / Best Regards
>
> Jens Oechsler
> System administrator
> KMD Nexus
> +45 51 82 62 13
--
Med Venlig Hilsen / Best Regards
Jens Oechsler
System administrator
KMD Nexus
+45 51 82 62 13
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
6:07 a.m.
Hi,
Thanks for reply. Have tried to gather logs from hosts here on google
drive: https://drive.google.com/open?id=0B7R4U330JfWpbkNhb2pxZWhmUUk
On Sun, Apr 30, 2017 at 10:50 AM, Fred Rolland <frolland(a)redhat.com> wrote:
Hi,
Can you provide the vdsm and engine logs ?
Thanks,
Fred
On Wed, Apr 26, 2017 at 5:30 PM, Jens Oechsler <joe(a)avaleo.net> wrote:
>
> Greetings,
>
> Is there any way to get the oVirt Data Center described below active
> again?
>
> On Tue, Apr 25, 2017 at 4:11 PM, Jens Oechsler <joe(a)avaleo.net> wrote:
> > Hi,
> >
> > LUN is not in pvs output, but I found it in lsblk output without any
> > partions on it apparently.
> >
> > $ sudo pvs
> > PV VG
> > Fmt Attr PSize PFree
> > /dev/mapper/360050768018182b6c000000000000990 data
> > lvm2 a-- 200.00g 180.00g
> > /dev/mapper/360050768018182b6c000000000000998
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc lvm2 a-- 499.62g 484.50g
> > /dev/sda2 system
> > lvm2 a-- 278.78g 208.41g
> >
> > $ sudo lvs
> > LV VG
> > Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync
> > Convert
> > 34a9328f-87fe-4190-96e9-a3580b0734fc
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 1.00g
> > 506ff043-1058-448c-bbab-5c864adb2bfc
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 10.00g
> > 65449c88-bc28-4275-bbbb-5fc75b692cbc
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
> > e2ee95ce-8105-4a20-8e1f-9f6dfa16bf59
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-ao---- 128.00m
> > ids
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-ao---- 128.00m
> > inbox
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
> > leases
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 2.00g
> > master
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 1.00g
> > metadata
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 512.00m
> > outbox
> > 9f10e00f-ae39-46a0-86da-8b157c6de7bc -wi-a----- 128.00m
> > data data
> > -wi-ao---- 20.00g
> > home system
> > -wi-ao---- 1000.00m
> > prod system
> > -wi-ao---- 4.88g
> > root system
> > -wi-ao---- 7.81g
> > swap system
> > -wi-ao---- 4.00g
> > swap7 system
> > -wi-ao---- 20.00g
> > tmp system
> > -wi-ao---- 4.88g
> > var system
> > -wi-ao---- 27.81g
> >
> > $ sudo lsblk
> > <output trimmed>
> > sdq
> > 65:0 0 500G 0 disk
> > └─360050768018182b6c0000000000009d7
> > 253:33 0 500G 0 mpath
> >
> > Data domain was made with one 500 GB LUN and extended with 500 GB more.
> >
> > On Tue, Apr 25, 2017 at 2:17 PM, Fred Rolland <frolland(a)redhat.com>
> > wrote:
> >> Hi,
> >>
> >> Do you see the LUN in the host ?
> >> Can you share pvs and lvs output ?
> >>
> >> Thanks,
> >>
> >> Fred
> >>
> >> On Mon, Apr 24, 2017 at 1:05 PM, Jens Oechsler <joe(a)avaleo.net>
wrote:
> >>>
> >>> Hello
> >>> I have a problem with oVirt Hosted Engine Setup version:
> >>> 4.0.5.5-1.el7.centos.
> >>> Setup is using FCP SAN for data and engine.
> >>> Cluster has worked fine for a while. It has two hosts with VMs
> >>> running.
> >>> I extended storage with an additional LUN recently. This LUN seems to
> >>> be gone from data domain and one VM is paused which I assume has data
> >>> on that device.
> >>>
> >>> Got these errors in events:
> >>>
> >>> Apr 24, 2017 10:26:05 AM
> >>> Failed to activate Storage Domain SD (Data Center DC) by
> >>> admin@internal-authz
> >>> Apr 10, 2017 3:38:08 PM
> >>> Status of host cl01 was set to Up.
> >>> Apr 10, 2017 3:38:03 PM
> >>> Host cl01 does not enforce SELinux. Current status: DISABLED
> >>> Apr 10, 2017 3:37:58 PM
> >>> Host cl01 is initializing. Message: Recovering from crash or
> >>> Initializing
> >>> Apr 10, 2017 3:37:58 PM
> >>> VDSM cl01 command failed: Recovering from crash or Initializing
> >>> Apr 10, 2017 3:37:46 PM
> >>> Failed to Reconstruct Master Domain for Data Center DC.
> >>> Apr 10, 2017 3:37:46 PM
> >>> Host cl01 is not responding. Host cannot be fenced automatically
> >>> because power management for the host is disabled.
> >>> Apr 10, 2017 3:37:46 PM
> >>> VDSM cl01 command failed: Broken pipe
> >>> Apr 10, 2017 3:37:46 PM
> >>> VDSM cl01 command failed: Broken pipe
> >>> Apr 10, 2017 3:32:45 PM
> >>> Invalid status on Data Center DC. Setting Data Center status to Non
> >>> Responsive (On host cl01, Error: General Exception).
> >>> Apr 10, 2017 3:32:45 PM
> >>> VDSM cl01 command failed: [Errno 19] Could not find dm device named
> >>> `[unknown]`
> >>> Apr 7, 2017 1:28:04 PM
> >>> VM HostedEngine is down with error. Exit message: resource busy:
> >>> Failed to acquire lock: error -243.
> >>> Apr 7, 2017 1:28:02 PM
> >>> Storage Pool Manager runs on Host cl01 (Address: cl01).
> >>> Apr 7, 2017 1:27:59 PM
> >>> Invalid status on Data Center DC. Setting status to Non Responsive.
> >>> Apr 7, 2017 1:27:53 PM
> >>> Host cl02 does not enforce SELinux. Current status: DISABLED
> >>> Apr 7, 2017 1:27:52 PM
> >>> Host cl01 does not enforce SELinux. Current status: DISABLED
> >>> Apr 7, 2017 1:27:49 PM
> >>> Affinity Rules Enforcement Manager started.
> >>> Apr 7, 2017 1:27:34 PM
> >>> ETL Service Started
> >>> Apr 7, 2017 1:26:01 PM
> >>> ETL Service Stopped
> >>> Apr 3, 2017 1:22:54 PM
> >>> Shutdown of VM HostedEngine failed.
> >>> Apr 3, 2017 1:22:52 PM
> >>> Storage Pool Manager runs on Host cl01 (Address: cl01).
> >>> Apr 3, 2017 1:22:49 PM
> >>> Invalid status on Data Center DC. Setting status to Non Responsive.
> >>>
> >>>
> >>> Master data domain is inactive.
> >>>
> >>>
> >>> vdsm.log:
> >>>
> >>> jsonrpc.Executor/5::INFO::2017-04-20
> >>> 07:01:26,796::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
> >>> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['ids']
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>> 07:01:26,796::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> >>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
> >>> devices { preferred_names = ["^/dev/mapper/"]
ignore_suspended_d
> >>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
> >>>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
> >>> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
> >>> wait_for_locks=1 use_lvmetad=
> >>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
> >>> bd616961-6da7-4eb0-939e-330b0a3fea6e/ids (cwd None)
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>> 07:01:26,880::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err>
= "
> >>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
> >>> WARNING: To avoid corruption, rescan devices to make changes
> >>> visible (pvscan --cache).\n Couldn't find device with uuid
> >>> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
> >>> jsonrpc.Executor/5::INFO::2017-04-20
> >>> 07:01:26,881::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
> >>> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['leases']
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>> 07:01:26,881::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> >>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
> >>> devices { preferred_names = ["^/dev/mapper/"]
ignore_suspended_d
> >>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
> >>>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
> >>> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
> >>> wait_for_locks=1 use_lvmetad=
> >>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
> >>> bd616961-6da7-4eb0-939e-330b0a3fea6e/leases (cwd None)
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>> 07:01:26,973::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err>
= "
> >>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
> >>> WARNING: To avoid corruption, rescan devices to make changes
> >>> visible (pvscan --cache).\n Couldn't find device with uuid
> >>> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
> >>> jsonrpc.Executor/5::INFO::2017-04-20
> >>> 07:01:26,973::lvm::1226::Storage.LVM::(activateLVs) Refreshing lvs:
> >>> vg=bd616961-6da7-4eb0-939e-330b0a3fea6e lvs=['metadata',
'leases',
> >>> 'ids', 'inbox', 'outbox', 'master']
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>> 07:01:26,974::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> >>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '
> >>> devices { preferred_names = ["^/dev/mapper/"]
ignore_suspended_d
> >>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [
> >>>
'\''a|/dev/mapper/360050768018182b6c00000000000099e|[unknown]|'\'',
> >>> '\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
> >>> wait_for_locks=1 use_lvmetad=
> >>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh
> >>> bd616961-6da7-4eb0-939e-330b0a3fea6e/metadata
> >>> bd616961-6da7-4eb0-939e-330b0a3fea6e/leases
> >>> bd616961-6da7-4eb0-939e-330b0a3fea6e/ids
> >>> bd616961-6da7-4eb0-939e-330b0a3fea6e/inbox b
> >>> d616961-6da7-4eb0-939e-330b0a3fea6e/outbox
> >>> bd616961-6da7-4eb0-939e-330b0a3fea6e/master (cwd None)
> >>> Reactor thread::INFO::2017-04-20
> >>>
> >>>
> >>>
07:01:27,069::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept)
> >>> Accepting connection from ::1:44692
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>> 07:01:27,070::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err>
= "
> >>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n
> >>> WARNING: To avoid corruption, rescan devices to make changes
> >>> visible (pvscan --cache).\n Couldn't find device with uuid
> >>> jDB9VW-bNqY-UIKc-XxXp-xnyK-ZTlt-7Cpa1U.\n"; <rc> = 0
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>>
> >>> 07:01:27,070::sp::662::Storage.StoragePool::(_stopWatchingDomainsState)
> >>> Stop watching domains state
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>>
> >>>
> >>>
07:01:27,070::resourceManager::628::Storage.ResourceManager::(releaseResource)
> >>> Trying to release resource
> >>> 'Storage.58493e81-01dc-01d8-0390-000000000032'
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>>
> >>>
> >>>
07:01:27,071::resourceManager::647::Storage.ResourceManager::(releaseResource)
> >>> Released resource 'Storage.58493e81-01dc-01d8-0390-000000000032'
(0
> >>> active users)
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>>
> >>>
> >>>
07:01:27,071::resourceManager::653::Storage.ResourceManager::(releaseResource)
> >>> Resource 'Storage.58493e81-01dc-01d8-0390-000000000032' is
free,
> >>> finding out if anyone is waiting for it.
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>>
> >>>
> >>>
07:01:27,071::resourceManager::661::Storage.ResourceManager::(releaseResource)
> >>> No one is waiting for resource
> >>> 'Storage.58493e81-01dc-01d8-0390-000000000032', Clearing
records.
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>>
> >>>
> >>>
07:01:27,071::resourceManager::628::Storage.ResourceManager::(releaseResource)
> >>> Trying to release resource 'Storage.HsmDomainMonitorLock'
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>>
> >>>
> >>>
07:01:27,071::resourceManager::647::Storage.ResourceManager::(releaseResource)
> >>> Released resource 'Storage.HsmDomainMonitorLock' (0 active
users)
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>>
> >>>
> >>>
07:01:27,071::resourceManager::653::Storage.ResourceManager::(releaseResource)
> >>> Resource 'Storage.HsmDomainMonitorLock' is free, finding out if
anyone
> >>> is waiting for it.
> >>> jsonrpc.Executor/5::DEBUG::2017-04-20
> >>>
> >>>
> >>>
07:01:27,071::resourceManager::661::Storage.ResourceManager::(releaseResource)
> >>> No one is waiting for resource 'Storage.HsmDomainMonitorLock',
> >>> Clearing records.
> >>> jsonrpc.Executor/5::ERROR::2017-04-20
> >>> 07:01:27,072::task::868::Storage.TaskManager.Task::(_setError)
> >>> Task=`15122a21-4fb7-45bf-9a9a-4b97f27bc1e1`::Unexpected error
> >>> Traceback (most recent call last):
> >>> File "/usr/share/vdsm/storage/task.py", line 875, in _run
> >>> return fn(*args, **kargs)
> >>> File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py",
line 50,
> >>> in
> >>> wrapper
> >>> res = f(*args, **kwargs)
> >>> File "/usr/share/vdsm/storage/hsm.py", line 988, in
> >>> connectStoragePool
> >>> spUUID, hostID, msdUUID, masterVersion, domainsMap)
> >>> File "/usr/share/vdsm/storage/hsm.py", line 1053, in
> >>> _connectStoragePool
> >>> res = pool.connect(hostID, msdUUID, masterVersion)
> >>> File "/usr/share/vdsm/storage/sp.py", line 646, in connect
> >>> self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
> >>> File "/usr/share/vdsm/storage/sp.py", line 1219, in
__rebuild
> >>> self.setMasterDomain(msdUUID, masterVersion)
> >>> File "/usr/share/vdsm/storage/sp.py", line 1427, in
setMasterDomain
> >>> domain = sdCache.produce(msdUUID)
> >>> File "/usr/share/vdsm/storage/sdc.py", line 101, in produce
> >>> domain.getRealDomain()
> >>> File "/usr/share/vdsm/storage/sdc.py", line 53, in
getRealDomain
> >>> return self._cache._realProduce(self._sdUUID)
> >>> File "/usr/share/vdsm/storage/sdc.py", line 125, in
_realProduce
> >>> domain = self._findDomain(sdUUID)
> >>> File "/usr/share/vdsm/storage/sdc.py", line 144, in
_findDomain
> >>> dom = findMethod(sdUUID)
> >>> File "/usr/share/vdsm/storage/blockSD.py", line 1441, in
findDomain
> >>> return
> >>> BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
> >>> File "/usr/share/vdsm/storage/blockSD.py", line 814, in
__init__
> >>> lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize))
> >>> File "/usr/share/vdsm/storage/lvm.py", line 1056, in
> >>> checkVGBlockSizes
> >>> _checkpvsblksize(pvs, vgBlkSize)
> >>> File "/usr/share/vdsm/storage/lvm.py", line 1033, in
_checkpvsblksize
> >>> pvBlkSize = _getpvblksize(pv)
> >>> File "/usr/share/vdsm/storage/lvm.py", line 1027, in
_getpvblksize
> >>> dev = devicemapper.getDmId(os.path.basename(pv))
> >>> File "/usr/share/vdsm/storage/devicemapper.py", line 40, in
getDmId
> >>> deviceMultipathName)
> >>> OSError: [Errno 19] Could not find dm device named `[unknown]`
> >>>
> >>>
> >>> Any input how to diagnose or troubleshoot would be appreciated.
> >>>
> >>> --
> >>> Best Regards
> >>>
> >>> Jens Oechsler
> >>> _______________________________________________
> >>> Users mailing list
> >>> Users(a)ovirt.org
> >>> http://lists.ovirt.org/mailman/listinfo/users
> >>
> >>
> >
> >
> >
> > --
> > Med Venlig Hilsen / Best Regards
> >
> > Jens Oechsler
> > System administrator
> > KMD Nexus
> > +45 51 82 62 13
>
>
>
> --
> Med Venlig Hilsen / Best Regards
>
> Jens Oechsler
> System administrator
> KMD Nexus
> +45 51 82 62 13
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--
Med Venlig Hilsen / Best Regards
Jens Oechsler
System administrator
KMD Nexus
+45 51 82 62 13
2537
days inactive
2552
days old
5 comments
2 participants
participants (2)
-
Fred Rolland -
Jens Oechsler