11:19 p.m.
On Tue, Mar 10, 2020 at 8:14 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Tue, Mar 10, 2020 at 7:03 PM Amit Bawer <abawer(a)redhat.com> wrote:
>
> Seems like a reproduce of https://bugzilla.redhat.com/show_bug.cgi?id=1807050#c1
Agree, because...
> Snipped from
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/21146/arti...:
>
> 2020-03-10 05:59:18,549-0400 ERROR (jsonrpc/3) [storage.LVM] vg
cceb9d83-7b76-4840-a189-c82f3c18760e has pv_count 2 but pv_names
('/dev/mapper/3600140544bef7e411164e5f94e13b5d8',) (lvm:578)
> 2020-03-10 05:59:18,551-0400 INFO (jsonrpc/3) [storage.StorageDomain]
sdUUID=cceb9d83-7b76-4840-a189-c82f3c18760e (blockSD:1192)
> 2020-03-10 05:59:18,551-0400 DEBUG (jsonrpc/3) [common.commands] /usr/bin/taskset
--cpu-list 0-1 /usr/bin/sudo -n /sbin/lvm vgck --config 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/3600140544bef7e411164e5f94e13b5d8$|", "r|.*|"]
hints="none" } global { locking_type=1 prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }'
cceb9d83-7b76-4840-a189-c82f3c18760e (cwd None) (commands:153)
> 2020-03-10 05:59:18,634-0400 DEBUG (jsonrpc/3) [common.commands] FAILED: <err>
= b" WARNING: Couldn't find device with uuid
FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.\n WARNING: VG
cceb9d83-7b76-4840-a189-c82f3c18760e is missing PV
FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.\n The volume group is missing 1 physical
volumes.\n"; <rc> = 5 (commands:185)
> 2020-03-10 05:59:18,637-0400 INFO (jsonrpc/3) [vdsm.api] FINISH
getStorageDomainInfo error=Domain is either partially accessible or entirely inaccessible:
('cceb9d83-7b76-4840-a189-c82f3c18760e: [" WARNING: Couldn\'t find device
with uuid FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.", \' WARNING: VG
cceb9d83-7b76-4840-a189-c82f3c18760e is missing PV
FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.\', \' The volume group is missing 1
physical volumes.\']',) from=::ffff:192.168.201.4,47796,
flow_id=5f02a1ec-db37-470d-b329-41b22f23582b, task_id=9be86ca4-49ac-47ea-b0e2-8182e33924ff
(api:52)
This command was run only once. Usually when a command using specific filter
(e.g. filter=["a|^/dev/mapper/3600140544bef7e411164e5f94e13b5d8$|",
"r|.*|"])
fails, we rebuild the filter. If the new filter is different (e.g has
more devices) we
run the command again.
Since we ran the command only once we know that the filter is correct,
so we have
only /dev/mapper/3600140544bef7e411164e5f94e13b5d8 on the host. The other PV
is not available when this command was run.
We started the connection here:
2020-03-10 05:59:17,364-0400 DEBUG (jsonrpc/2) [common.commands]
/usr/bin/taskset --cpu-list 0-1 /usr/bin/sudo -n /sbin/iscsiadm -m
node -T iqn.2014-07.org.ovirt:storage -I default -p
192.168.200.4:3260,1 -l (cwd None) (commands:153)
2020-03-10 05:59:17,504-0400 DEBUG (jsonrpc/2) [common.commands]
SUCCESS: <err> = b''; <rc> = 0 (commands:98)
And finished here:
2020-03-10 05:59:17,610-0400 DEBUG (jsonrpc/2) [common.commands]
/usr/bin/taskset --cpu-list 0-1 /sbin/udevadm settle --timeout=5 (cwd
None) (commands:153)
2020-03-10 05:59:17,787-0400 DEBUG (jsonrpc/2) [common.commands]
SUCCESS: <err> = b''; <rc> = 0 (commands:98)
In /var/log/message we see the connection starting here:
Mar 10 05:59:17 lago-basic-suite-master-host-1 iscsid[21973]: iscsid:
Connection2:0 to [target: iqn.2014-07.org.ovirt:storage, portal:
192.168.200.4,3260] through [iface: default] is operational now
Adding devices:
Mar 10 05:59:17 lago-basic-suite-master-host-1 kernel: sd 3:0:0:0:
[sdf] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)
Mar 10 05:59:17 lago-basic-suite-master-host-1 kernel: sd 3:0:0:4:
[sdg] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)
Mar 10 05:59:17 lago-basic-suite-master-host-1 kernel: sd 3:0:0:3:
[sdh] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)
Mar 10 05:59:17 lago-basic-suite-master-host-1 kernel: sd 3:0:0:2:
[sdi] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)
Mar 10 05:59:17 lago-basic-suite-master-host-1 kernel: sd 3:0:0:1:
[sdj] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)
Multipath adding devices to maps:
Mar 10 05:59:18 lago-basic-suite-master-host-1 multipathd[21959]: sdb
[8:16]: path added to devmap 36001405b39cc4e33bd24f35a81c0c140
Mar 10 05:59:18 lago-basic-suite-master-host-1 multipathd[21959]: sdc
[8:32]: path added to devmap 36001405a70a062950224fc985825aa0d
Mar 10 05:59:18 lago-basic-suite-master-host-1 multipathd[21959]: sda
[8:0]: path added to devmap 3600140559c49ea12b0d4dc1994ba4ef0
Mar 10 05:59:18 lago-basic-suite-master-host-1 multipathd[21959]: sde
[8:64]: path added to devmap 36001405f277c71b13814669926ffbae4
Mar 10 05:59:18 lago-basic-suite-master-host-1 multipathd[21959]: sdi
[8:128]: path added to devmap 3600140544bef7e411164e5f94e13b5d8 <<<
This is probably the missing device
So we need to wait for a while, until multipath handles all the devices.
https://gerrit.ovirt.org/c/107206/ should avoid this issue.
Merged now. We should not see this issue now.
Benny, please try to run OST.
> 2020-03-10 05:59:18,637-0400 ERROR (jsonrpc/3) [storage.TaskManager.Task]
(Task='9be86ca4-49ac-47ea-b0e2-8182e33924ff') Unexpected error (task:880)
> Traceback (most recent call last):
> File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887,
in _run
> return fn(*args, **kargs)
> File "<decorator-gen-129>", line 2, in getStorageDomainInfo
> File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in
method
> ret = func(*args, **kwargs)
> File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2752,
in getStorageDomainInfo
> dom = self.validateSdUUID(sdUUID)
> File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 310,
in validateSdUUID
> sdDom.validate()
> File "/usr/lib/python3.6/site-packages/vdsm/storage/blockSD.py", line
1193, in validate
> lvm.chkVG(self.sdUUID)
> File "/usr/lib/python3.6/site-packages/vdsm/storage/lvm.py", line 1278,
in chkVG
> raise se.StorageDomainAccessError("%s: %s" % (vgName, err))
> vdsm.storage.exception.StorageDomainAccessError: Domain is either partially
accessible or entirely inaccessible: ('cceb9d83-7b76-4840-a189-c82f3c18760e: ["
WARNING: Couldn\'t find device with uuid
FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.", \' WARNING: VG
cceb9d83-7b76-4840-a189-c82f3c18760e is missing PV
FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.\', \' The volume group is missing 1
physical volumes.\']',)
> 2020-03-10 05:59:18,637-0400 INFO (jsonrpc/3) [storage.TaskManager.Task]
(Task='9be86ca4-49ac-47ea-b0e2-8182e33924ff') aborting: Task is aborted:
'value=Domain is either partially accessible or entirely inaccessible:
(\'cceb9d83-7b76-4840-a189-c82f3c18760e: [" WARNING: Couldn\\\'t find device
with uuid FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.", \\\' WARNING: VG
cceb9d83-7b76-4840-a189-c82f3c18760e is missing PV
FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.\\\', \\\' The volume group is missing 1
physical volumes.\\\']\',) abortedcode=379' (task:1190)
> 2020-03-10 05:59:18,638-0400 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH
getStorageDomainInfo error=Domain is either partially accessible or entirely inaccessible:
('cceb9d83-7b76-4840-a189-c82f3c18760e: [" WARNING: Couldn\'t find device
with uuid FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.", \' WARNING: VG
cceb9d83-7b76-4840-a189-c82f3c18760e is missing PV
FH6lfD-DZus-6Ndn-tkr8-5Hsy-lt2c-CDRPDU.\', \' The volume group is missing 1
physical volumes.\']',) (dispatcher:83)
>
>
> Suggest to try again once the BZ is fixed on master.
>
> On Tue, Mar 10, 2020 at 1:36 PM Yedidyah Bar David <didi(a)redhat.com> wrote:
> >
> > Hi all,
> >
> > Anyone looking at this?
> >
> > See e.g.:
> >
> > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/21146/
> >
> > Thanks,
> > --
> > Didi
> > _______________________________________________
> > Devel mailing list -- devel(a)ovirt.org
> > To unsubscribe send an email to devel-leave(a)ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ED57V5XW4B3...
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GFWZBWE3UT4...