<div dir="ltr"><div><div>Hi,<br><br>Can you provide the vdsm and engine logs ?<br></div><br>Thanks,<br></div>Fred<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 26, 2017 at 5:30 PM, Jens Oechsler <span dir="ltr"><<a href="mailto:joe@avaleo.net" target="_blank">joe@avaleo.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Greetings,<br>
<br>
Is there any way to get the oVirt Data Center described below active again?<br>
<div class="HOEnZb"><div class="h5"><br>
On Tue, Apr 25, 2017 at 4:11 PM, Jens Oechsler <<a href="mailto:joe@avaleo.net">joe@avaleo.net</a>> wrote:<br>
> Hi,<br>
><br>
> LUN is not in pvs output, but I found it in lsblk output without any<br>
> partions on it apparently.<br>
><br>
> $ sudo pvs<br>
> PV VG<br>
> Fmt Attr PSize PFree<br>
> /dev/mapper/<wbr>360050768018182b6c000000000000<wbr>990 data<br>
> lvm2 a-- 200.00g 180.00g<br>
> /dev/mapper/<wbr>360050768018182b6c000000000000<wbr>998<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc lvm2 a-- 499.62g 484.50g<br>
> /dev/sda2 system<br>
> lvm2 a-- 278.78g 208.41g<br>
><br>
> $ sudo lvs<br>
> LV VG<br>
> Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync<br>
> Convert<br>
> 34a9328f-87fe-4190-96e9-<wbr>a3580b0734fc<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-a----- 1.00g<br>
> 506ff043-1058-448c-bbab-<wbr>5c864adb2bfc<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-a----- 10.00g<br>
> 65449c88-bc28-4275-bbbb-<wbr>5fc75b692cbc<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-a----- 128.00m<br>
> e2ee95ce-8105-4a20-8e1f-<wbr>9f6dfa16bf59<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-ao---- 128.00m<br>
> ids<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-ao---- 128.00m<br>
> inbox<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-a----- 128.00m<br>
> leases<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-a----- 2.00g<br>
> master<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-a----- 1.00g<br>
> metadata<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-a----- 512.00m<br>
> outbox<br>
> 9f10e00f-ae39-46a0-86da-<wbr>8b157c6de7bc -wi-a----- 128.00m<br>
> data data<br>
> -wi-ao---- 20.00g<br>
> home system<br>
> -wi-ao---- 1000.00m<br>
> prod system<br>
> -wi-ao---- 4.88g<br>
> root system<br>
> -wi-ao---- 7.81g<br>
> swap system<br>
> -wi-ao---- 4.00g<br>
> swap7 system<br>
> -wi-ao---- 20.00g<br>
> tmp system<br>
> -wi-ao---- 4.88g<br>
> var system<br>
> -wi-ao---- 27.81g<br>
><br>
> $ sudo lsblk<br>
> <output trimmed><br>
> sdq<br>
> 65:0 0 500G 0 disk<br>
> └─<wbr>360050768018182b6c000000000000<wbr>9d7<br>
> 253:33 0 500G 0 mpath<br>
><br>
> Data domain was made with one 500 GB LUN and extended with 500 GB more.<br>
><br>
> On Tue, Apr 25, 2017 at 2:17 PM, Fred Rolland <<a href="mailto:frolland@redhat.com">frolland@redhat.com</a>> wrote:<br>
>> Hi,<br>
>><br>
>> Do you see the LUN in the host ?<br>
>> Can you share pvs and lvs output ?<br>
>><br>
>> Thanks,<br>
>><br>
>> Fred<br>
>><br>
>> On Mon, Apr 24, 2017 at 1:05 PM, Jens Oechsler <<a href="mailto:joe@avaleo.net">joe@avaleo.net</a>> wrote:<br>
>>><br>
>>> Hello<br>
>>> I have a problem with oVirt Hosted Engine Setup version:<br>
>>> 4.0.5.5-1.el7.centos.<br>
>>> Setup is using FCP SAN for data and engine.<br>
>>> Cluster has worked fine for a while. It has two hosts with VMs running.<br>
>>> I extended storage with an additional LUN recently. This LUN seems to<br>
>>> be gone from data domain and one VM is paused which I assume has data<br>
>>> on that device.<br>
>>><br>
>>> Got these errors in events:<br>
>>><br>
>>> Apr 24, 2017 10:26:05 AM<br>
>>> Failed to activate Storage Domain SD (Data Center DC) by<br>
>>> admin@internal-authz<br>
>>> Apr 10, 2017 3:38:08 PM<br>
>>> Status of host cl01 was set to Up.<br>
>>> Apr 10, 2017 3:38:03 PM<br>
>>> Host cl01 does not enforce SELinux. Current status: DISABLED<br>
>>> Apr 10, 2017 3:37:58 PM<br>
>>> Host cl01 is initializing. Message: Recovering from crash or Initializing<br>
>>> Apr 10, 2017 3:37:58 PM<br>
>>> VDSM cl01 command failed: Recovering from crash or Initializing<br>
>>> Apr 10, 2017 3:37:46 PM<br>
>>> Failed to Reconstruct Master Domain for Data Center DC.<br>
>>> Apr 10, 2017 3:37:46 PM<br>
>>> Host cl01 is not responding. Host cannot be fenced automatically<br>
>>> because power management for the host is disabled.<br>
>>> Apr 10, 2017 3:37:46 PM<br>
>>> VDSM cl01 command failed: Broken pipe<br>
>>> Apr 10, 2017 3:37:46 PM<br>
>>> VDSM cl01 command failed: Broken pipe<br>
>>> Apr 10, 2017 3:32:45 PM<br>
>>> Invalid status on Data Center DC. Setting Data Center status to Non<br>
>>> Responsive (On host cl01, Error: General Exception).<br>
>>> Apr 10, 2017 3:32:45 PM<br>
>>> VDSM cl01 command failed: [Errno 19] Could not find dm device named<br>
>>> `[unknown]`<br>
>>> Apr 7, 2017 1:28:04 PM<br>
>>> VM HostedEngine is down with error. Exit message: resource busy:<br>
>>> Failed to acquire lock: error -243.<br>
>>> Apr 7, 2017 1:28:02 PM<br>
>>> Storage Pool Manager runs on Host cl01 (Address: cl01).<br>
>>> Apr 7, 2017 1:27:59 PM<br>
>>> Invalid status on Data Center DC. Setting status to Non Responsive.<br>
>>> Apr 7, 2017 1:27:53 PM<br>
>>> Host cl02 does not enforce SELinux. Current status: DISABLED<br>
>>> Apr 7, 2017 1:27:52 PM<br>
>>> Host cl01 does not enforce SELinux. Current status: DISABLED<br>
>>> Apr 7, 2017 1:27:49 PM<br>
>>> Affinity Rules Enforcement Manager started.<br>
>>> Apr 7, 2017 1:27:34 PM<br>
>>> ETL Service Started<br>
>>> Apr 7, 2017 1:26:01 PM<br>
>>> ETL Service Stopped<br>
>>> Apr 3, 2017 1:22:54 PM<br>
>>> Shutdown of VM HostedEngine failed.<br>
>>> Apr 3, 2017 1:22:52 PM<br>
>>> Storage Pool Manager runs on Host cl01 (Address: cl01).<br>
>>> Apr 3, 2017 1:22:49 PM<br>
>>> Invalid status on Data Center DC. Setting status to Non Responsive.<br>
>>><br>
>>><br>
>>> Master data domain is inactive.<br>
>>><br>
>>><br>
>>> vdsm.log:<br>
>>><br>
>>> jsonrpc.Executor/5::INFO::<wbr>2017-04-20<br>
>>> 07:01:26,796::lvm::1226::<wbr>Storage.LVM::(activateLVs) Refreshing lvs:<br>
>>> vg=bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e lvs=['ids']<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>> 07:01:26,796::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) /usr/bin/taskset<br>
>>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '<br>
>>> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d<br>
>>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [<br>
>>> '\''a|/dev/mapper/<wbr>360050768018182b6c000000000000<wbr>99e|[unknown]|'\'',<br>
>>> '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1<br>
>>> wait_for_locks=1 use_lvmetad=<br>
>>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh<br>
>>> bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e/ids (cwd None)<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>> 07:01:26,880::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "<br>
>>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n<br>
>>> WARNING: To avoid corruption, rescan devices to make changes<br>
>>> visible (pvscan --cache).\n Couldn't find device with uuid<br>
>>> jDB9VW-bNqY-UIKc-XxXp-xnyK-<wbr>ZTlt-7Cpa1U.\n"; <rc> = 0<br>
>>> jsonrpc.Executor/5::INFO::<wbr>2017-04-20<br>
>>> 07:01:26,881::lvm::1226::<wbr>Storage.LVM::(activateLVs) Refreshing lvs:<br>
>>> vg=bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e lvs=['leases']<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>> 07:01:26,881::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) /usr/bin/taskset<br>
>>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '<br>
>>> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d<br>
>>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [<br>
>>> '\''a|/dev/mapper/<wbr>360050768018182b6c000000000000<wbr>99e|[unknown]|'\'',<br>
>>> '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1<br>
>>> wait_for_locks=1 use_lvmetad=<br>
>>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh<br>
>>> bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e/leases (cwd None)<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>> 07:01:26,973::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "<br>
>>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n<br>
>>> WARNING: To avoid corruption, rescan devices to make changes<br>
>>> visible (pvscan --cache).\n Couldn't find device with uuid<br>
>>> jDB9VW-bNqY-UIKc-XxXp-xnyK-<wbr>ZTlt-7Cpa1U.\n"; <rc> = 0<br>
>>> jsonrpc.Executor/5::INFO::<wbr>2017-04-20<br>
>>> 07:01:26,973::lvm::1226::<wbr>Storage.LVM::(activateLVs) Refreshing lvs:<br>
>>> vg=bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e lvs=['metadata', 'leases',<br>
>>> 'ids', 'inbox', 'outbox', 'master']<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>> 07:01:26,974::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) /usr/bin/taskset<br>
>>> --cpu-list 0-39 /usr/bin/sudo -n /usr/sbin/lvm lvchange --config '<br>
>>> devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_d<br>
>>> evices=1 write_cache_state=0 disable_after_error_count=3 filter = [<br>
>>> '\''a|/dev/mapper/<wbr>360050768018182b6c000000000000<wbr>99e|[unknown]|'\'',<br>
>>> '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1<br>
>>> wait_for_locks=1 use_lvmetad=<br>
>>> 0 } backup { retain_min = 50 retain_days = 0 } ' --refresh<br>
>>> bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e/metadata<br>
>>> bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e/leases<br>
>>> bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e/ids<br>
>>> bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e/inbox b<br>
>>> d616961-6da7-4eb0-939e-<wbr>330b0a3fea6e/outbox<br>
>>> bd616961-6da7-4eb0-939e-<wbr>330b0a3fea6e/master (cwd None)<br>
>>> Reactor thread::INFO::2017-04-20<br>
>>><br>
>>> 07:01:27,069::<wbr>protocoldetector::72::<wbr>ProtocolDetector.AcceptorImpl:<wbr>:(handle_accept)<br>
>>> Accepting connection from ::1:44692<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>> 07:01:27,070::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) SUCCESS: <err> = "<br>
>>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\n<br>
>>> WARNING: To avoid corruption, rescan devices to make changes<br>
>>> visible (pvscan --cache).\n Couldn't find device with uuid<br>
>>> jDB9VW-bNqY-UIKc-XxXp-xnyK-<wbr>ZTlt-7Cpa1U.\n"; <rc> = 0<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>> 07:01:27,070::sp::662::<wbr>Storage.StoragePool::(_<wbr>stopWatchingDomainsState)<br>
>>> Stop watching domains state<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>><br>
>>> 07:01:27,070::resourceManager:<wbr>:628::Storage.ResourceManager:<wbr>:(releaseResource)<br>
>>> Trying to release resource<br>
>>> 'Storage.58493e81-01dc-01d8-<wbr>0390-000000000032'<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>><br>
>>> 07:01:27,071::resourceManager:<wbr>:647::Storage.ResourceManager:<wbr>:(releaseResource)<br>
>>> Released resource 'Storage.58493e81-01dc-01d8-<wbr>0390-000000000032' (0<br>
>>> active users)<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>><br>
>>> 07:01:27,071::resourceManager:<wbr>:653::Storage.ResourceManager:<wbr>:(releaseResource)<br>
>>> Resource 'Storage.58493e81-01dc-01d8-<wbr>0390-000000000032' is free,<br>
>>> finding out if anyone is waiting for it.<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>><br>
>>> 07:01:27,071::resourceManager:<wbr>:661::Storage.ResourceManager:<wbr>:(releaseResource)<br>
>>> No one is waiting for resource<br>
>>> 'Storage.58493e81-01dc-01d8-<wbr>0390-000000000032', Clearing records.<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>><br>
>>> 07:01:27,071::resourceManager:<wbr>:628::Storage.ResourceManager:<wbr>:(releaseResource)<br>
>>> Trying to release resource 'Storage.HsmDomainMonitorLock'<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>><br>
>>> 07:01:27,071::resourceManager:<wbr>:647::Storage.ResourceManager:<wbr>:(releaseResource)<br>
>>> Released resource 'Storage.HsmDomainMonitorLock' (0 active users)<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>><br>
>>> 07:01:27,071::resourceManager:<wbr>:653::Storage.ResourceManager:<wbr>:(releaseResource)<br>
>>> Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone<br>
>>> is waiting for it.<br>
>>> jsonrpc.Executor/5::DEBUG::<wbr>2017-04-20<br>
>>><br>
>>> 07:01:27,071::resourceManager:<wbr>:661::Storage.ResourceManager:<wbr>:(releaseResource)<br>
>>> No one is waiting for resource 'Storage.HsmDomainMonitorLock'<wbr>,<br>
>>> Clearing records.<br>
>>> jsonrpc.Executor/5::ERROR::<wbr>2017-04-20<br>
>>> 07:01:27,072::task::868::<wbr>Storage.TaskManager.Task::(_<wbr>setError)<br>
>>> Task=`15122a21-4fb7-45bf-9a9a-<wbr>4b97f27bc1e1`::Unexpected error<br>
>>> Traceback (most recent call last):<br>
>>> File "/usr/share/vdsm/storage/task.<wbr>py", line 875, in _run<br>
>>> return fn(*args, **kargs)<br>
>>> File "/usr/lib/python2.7/site-<wbr>packages/vdsm/logUtils.py", line 50, in<br>
>>> wrapper<br>
>>> res = f(*args, **kwargs)<br>
>>> File "/usr/share/vdsm/storage/hsm.<wbr>py", line 988, in connectStoragePool<br>
>>> spUUID, hostID, msdUUID, masterVersion, domainsMap)<br>
>>> File "/usr/share/vdsm/storage/hsm.<wbr>py", line 1053, in _connectStoragePool<br>
>>> res = pool.connect(hostID, msdUUID, masterVersion)<br>
>>> File "/usr/share/vdsm/storage/sp.<wbr>py", line 646, in connect<br>
>>> self.__rebuild(msdUUID=<wbr>msdUUID, masterVersion=masterVersion)<br>
>>> File "/usr/share/vdsm/storage/sp.<wbr>py", line 1219, in __rebuild<br>
>>> self.setMasterDomain(msdUUID, masterVersion)<br>
>>> File "/usr/share/vdsm/storage/sp.<wbr>py", line 1427, in setMasterDomain<br>
>>> domain = sdCache.produce(msdUUID)<br>
>>> File "/usr/share/vdsm/storage/sdc.<wbr>py", line 101, in produce<br>
>>> domain.getRealDomain()<br>
>>> File "/usr/share/vdsm/storage/sdc.<wbr>py", line 53, in getRealDomain<br>
>>> return self._cache._realProduce(self.<wbr>_sdUUID)<br>
>>> File "/usr/share/vdsm/storage/sdc.<wbr>py", line 125, in _realProduce<br>
>>> domain = self._findDomain(sdUUID)<br>
>>> File "/usr/share/vdsm/storage/sdc.<wbr>py", line 144, in _findDomain<br>
>>> dom = findMethod(sdUUID)<br>
>>> File "/usr/share/vdsm/storage/<wbr>blockSD.py", line 1441, in findDomain<br>
>>> return BlockStorageDomain(<wbr>BlockStorageDomain.<wbr>findDomainPath(sdUUID))<br>
>>> File "/usr/share/vdsm/storage/<wbr>blockSD.py", line 814, in __init__<br>
>>> lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize))<br>
>>> File "/usr/share/vdsm/storage/lvm.<wbr>py", line 1056, in checkVGBlockSizes<br>
>>> _checkpvsblksize(pvs, vgBlkSize)<br>
>>> File "/usr/share/vdsm/storage/lvm.<wbr>py", line 1033, in _checkpvsblksize<br>
>>> pvBlkSize = _getpvblksize(pv)<br>
>>> File "/usr/share/vdsm/storage/lvm.<wbr>py", line 1027, in _getpvblksize<br>
>>> dev = devicemapper.getDmId(os.path.<wbr>basename(pv))<br>
>>> File "/usr/share/vdsm/storage/<wbr>devicemapper.py", line 40, in getDmId<br>
>>> deviceMultipathName)<br>
>>> OSError: [Errno 19] Could not find dm device named `[unknown]`<br>
>>><br>
>>><br>
>>> Any input how to diagnose or troubleshoot would be appreciated.<br>
>>><br>
>>> --<br>
>>> Best Regards<br>
>>><br>
>>> Jens Oechsler<br>
>>> ______________________________<wbr>_________________<br>
>>> Users mailing list<br>
>>> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>>> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
>><br>
>><br>
><br>
><br>
><br>
> --<br>
> Med Venlig Hilsen / Best Regards<br>
><br>
> Jens Oechsler<br>
> System administrator<br>
> KMD Nexus<br>
> <a href="tel:%2B45%2051%2082%2062%2013" value="+4551826213">+45 51 82 62 13</a><br>
<br>
<br>
<br>
--<br>
Med Venlig Hilsen / Best Regards<br>
<br>
Jens Oechsler<br>
System administrator<br>
KMD Nexus<br>
<a href="tel:%2B45%2051%2082%2062%2013" value="+4551826213">+45 51 82 62 13</a><br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>