[ovirt-users] VMs becoming non-responsive sporadically
nicolas at devels.es
nicolas at devels.es
Sun May 1 12:31:07 UTC 2016
El 2016-04-30 23:22, Nir Soffer escribió:
> On Sun, May 1, 2016 at 12:48 AM, <nicolas at devels.es> wrote:
>> El 2016-04-30 22:37, Nir Soffer escribió:
>>>
>>> On Sat, Apr 30, 2016 at 10:28 PM, Nir Soffer <nsoffer at redhat.com>
>>> wrote:
>>>>
>>>> On Sat, Apr 30, 2016 at 7:16 PM, <nicolas at devels.es> wrote:
>>>>>
>>>>> El 2016-04-30 16:55, Nir Soffer escribió:
>>>>>>
>>>>>>
>>>>>> On Sat, Apr 30, 2016 at 11:33 AM, Nicolás <nicolas at devels.es>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi Nir,
>>>>>>>
>>>>>>> El 29/04/16 a las 22:34, Nir Soffer escribió:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Apr 29, 2016 at 9:17 PM, <nicolas at devels.es> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> We're running oVirt 3.6.5.3-1 and lately we're experiencing
>>>>>>>>> some
>>>>>>>>> issues
>>>>>>>>> with
>>>>>>>>> some VMs being paused because they're marked as non-responsive.
>>>>>>>>> Mostly,
>>>>>>>>> after a few seconds they recover, but we want to debug
>>>>>>>>> precisely
>>>>>>>>> this
>>>>>>>>> problem so we can fix it consistently.
>>>>>>>>>
>>>>>>>>> Our scenario is the following:
>>>>>>>>>
>>>>>>>>> ~495 VMs, of which ~120 are constantly up
>>>>>>>>> 3 datastores, all of them iSCSI-based:
>>>>>>>>> * ds1: 2T, currently has 276 disks
>>>>>>>>> * ds2: 2T, currently has 179 disks
>>>>>>>>> * ds3: 500G, currently has 65 disks
>>>>>>>>> 7 hosts: All have mostly the same hardware. CPU and memory are
>>>>>>>>> currently
>>>>>>>>> very lowly used (< 10%).
>>>>>>>>>
>>>>>>>>> ds1 and ds2 are physically the same backend which exports
>>>>>>>>> two 2TB
>>>>>>>>> volumes.
>>>>>>>>> ds3 is a different storage backend where we're currently
>>>>>>>>> migrating
>>>>>>>>> some
>>>>>>>>> disks from ds1 and ds2.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> What the the storage backend behind ds1 and 2?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The storage backend for ds1 and ds2 is the iSCSI-based HP
>>>>>>> LeftHand
>>>>>>> P4000
>>>>>>> G2.
>>>>>>>
>>>>>>>>> Usually, when VMs become unresponsive, the whole host where
>>>>>>>>> they run
>>>>>>>>> gets
>>>>>>>>> unresponsive too, so that gives a hint about the problem, my
>>>>>>>>> bet is
>>>>>>>>> the
>>>>>>>>> culprit is somewhere on the host side and not on the VMs side.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Probably the vm became unresponsive because connection to the
>>>>>>>> host
>>>>>>>> was
>>>>>>>> lost.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I forgot to mention that less commonly we have situations where
>>>>>>> the
>>>>>>> host
>>>>>>> doesn't get unresponsive but the VMs on it do and they don't
>>>>>>> become
>>>>>>> responsive ever again, so we have to forcibly power them off and
>>>>>>> start
>>>>>>> them
>>>>>>> on a different host. But in this case the connection with the
>>>>>>> host
>>>>>>> doesn't
>>>>>>> ever get lost (so basically the host is Up, but any VM run on
>>>>>>> them is
>>>>>>> unresponsive).
>>>>>>>
>>>>>>>
>>>>>>>>> When that
>>>>>>>>> happens, the host itself gets non-responsive and only
>>>>>>>>> recoverable
>>>>>>>>> after
>>>>>>>>> reboot, since it's unable to reconnect.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Piotr, can you check engine log and explain why host is not
>>>>>>>> reconnected?
>>>>>>>>
>>>>>>>>> I must say this is not specific to
>>>>>>>>> this oVirt version, when we were using v.3.6.4 the same
>>>>>>>>> happened,
>>>>>>>>> and
>>>>>>>>> it's
>>>>>>>>> also worthy mentioning we've not done any configuration changes
>>>>>>>>> and
>>>>>>>>> everything had been working quite well for a long time.
>>>>>>>>>
>>>>>>>>> We were monitoring our ds1 and ds2 physical backend to see
>>>>>>>>> performance
>>>>>>>>> and
>>>>>>>>> we suspect we've run out of IOPS since we're reaching the
>>>>>>>>> maximum
>>>>>>>>> specified
>>>>>>>>> by the manufacturer, probably at certain times the host cannot
>>>>>>>>> perform
>>>>>>>>> a
>>>>>>>>> storage operation within some time limit and it marks VMs as
>>>>>>>>> unresponsive.
>>>>>>>>> That's why we've set up ds3 and we're migrating ds1 and ds2 to
>>>>>>>>> ds3.
>>>>>>>>> When
>>>>>>>>> we
>>>>>>>>> run out of space on ds3 we'll create more smaller volumes to
>>>>>>>>> keep
>>>>>>>>> migrating.
>>>>>>>>>
>>>>>>>>> On the host side, when this happens, we've run repoplot on the
>>>>>>>>> vdsm
>>>>>>>>> log
>>>>>>>>> and
>>>>>>>>> I'm attaching the result. Clearly there's a *huge* LVM response
>>>>>>>>> time
>>>>>>>>> (~30
>>>>>>>>> secs.).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Indeed the log show very slow vgck and vgs commands - these are
>>>>>>>> called
>>>>>>>> every
>>>>>>>> 5 minutes for checking the vg health and refreshing vdsm lvm
>>>>>>>> cache.
>>>>>>>>
>>>>>>>> 1. starting vgck
>>>>>>>>
>>>>>>>> Thread-96::DEBUG::2016-04-29
>>>>>>>> 13:17:48,682::lvm::290::Storage.Misc.excCmd::(cmd)
>>>>>>>> /usr/bin/taskset
>>>>>>>> --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgck --config '
>>>>>>>> devices
>>>>>>>> { pre
>>>>>>>> ferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
>>>>>>>> write_cache_state=0 disable_after_error_count=3 filter = [
>>>>>>>> '\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|'\
>>>>>>>> '', '\''r|.*|'\'' ] } global { locking_type=1
>>>>>>>> prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 }
>>>>>>>> backup {
>>>>>>>> retain_min = 50 retain_days = 0 } ' 5de4a000-a9c4-48
>>>>>>>> 9c-8eee-10368647c413 (cwd None)
>>>>>>>>
>>>>>>>> 2. vgck ends after 55 seconds
>>>>>>>>
>>>>>>>> Thread-96::DEBUG::2016-04-29
>>>>>>>> 13:18:43,173::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS:
>>>>>>>> <err> = '
>>>>>>>> WARNING: lvmetad is running but disabled. Restart lvmetad before
>>>>>>>> enabling it!\n'; <rc> = 0
>>>>>>>>
>>>>>>>> 3. starting vgs
>>>>>>>>
>>>>>>>> Thread-96::DEBUG::2016-04-29
>>>>>>>> 13:17:11,963::lvm::290::Storage.Misc.excCmd::(cmd)
>>>>>>>> /usr/bin/taskset
>>>>>>>> --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config '
>>>>>>>> devices
>>>>>>>> { pref
>>>>>>>> erred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
>>>>>>>> write_cache_state=0 disable_after_error_count=3 filter = [
>>>>>>>> '\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|/de
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> v/mapper/36000eb3a4f1acbc200000000000000b9|/dev/mapper/360014056f0dc8930d744f83af8ddc709|/dev/mapper/WDC_WD5003ABYZ-011FA0_WD-WMAYP0J73DU6|'\'',
>>>>>>>> '\''r|.*|'\'' ] } global {
>>>>>>>> locking_type=1 prioritise_write_locks=1 wait_for_locks=1
>>>>>>>> use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } '
>>>>>>>> --noheadings --units b --nosuffix --separator '|
>>>>>>>> ' --ignoreskippedcluster -o
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
>>>>>>>> 5de4a000-a9c4-489c-8eee-10368
>>>>>>>> 647c413 (cwd None)
>>>>>>>>
>>>>>>>> 4. vgs finished after 37 seconds
>>>>>>>>
>>>>>>>> Thread-96::DEBUG::2016-04-29
>>>>>>>> 13:17:48,680::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS:
>>>>>>>> <err> = '
>>>>>>>> WARNING: lvmetad is running but disabled. Restart lvmetad before
>>>>>>>> enabling it!\n'; <rc> = 0
>>>>>>>>
>>>>>>>> Zdenek, how do you suggest to debug this slow lvm commands?
>>>>>>>>
>>>>>>>> Can you run the following commands on a host in trouble, and on
>>>>>>>> some
>>>>>>>> other
>>>>>>>> hosts in the same timeframe?
>>>>>>>>
>>>>>>>> time vgck -vvvv --config ' devices { filter =
>>>>>>>> ['\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|'\'',
>>>>>>>> '\''r|.*|'\'' ] } global { locking_type=1
>>>>>>>> prioritise_write_locks=1
>>>>>>>> wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50
>>>>>>>> retain_days = 0 } ' 5de4a000-a9c4-489c-8eee-10368647c413
>>>>>>>>
>>>>>>>> time vgs -vvvv --config ' global { locking_type=1
>>>>>>>> prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 }
>>>>>>>> backup {
>>>>>>>> retain_min = 50 retain_days = 0 } '
>>>>>>>> 5de4a000-a9c4-489c-8eee-10368647c413
>>>>>>>>
>>>>>>>> Note that I added -vvvv to both commands, this will generate
>>>>>>>> huge
>>>>>>>> amount
>>>>>>>> of debugging info. Please share the output of these commands.
>>>>>>>>
>>>>>>>> You may need to fix the commands. You can always copy and paste
>>>>>>>> directly
>>>>>>>> from vdsm log into the shell and add the -vvvv flag.
>>>>>>>>
>>>>>>>
>>>>>>> Indeed, there seems to be a big difference on hosts 5 and 6. I'm
>>>>>>> attaching
>>>>>>> the results of the execution of both commands on all hosts. Both
>>>>>>> commands
>>>>>>> show a pretty bigger output on hosts 5 and 6, and also a much
>>>>>>> bigger
>>>>>>> execution time. Times are also attached in a file called TIMES.
>>>>>>>
>>>>>>>>> Our host storage network is correctly configured and on a 1G
>>>>>>>>> interface, no errors on the host itself, switches, etc.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 1G serving about 20-70 vms per host? (495 vms, 120 always up, 7
>>>>>>>> hosts)?
>>>>>>>>
>>>>>>>> Do you have separate network for management and storage, or both
>>>>>>>> use this 1G interface?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Yes, we have separate networks for management, storage and
>>>>>>> motion.
>>>>>>> Storage
>>>>>>> and motion have 1G each (plus, for storage we use a bond of 2
>>>>>>> interfaces
>>>>>>> in
>>>>>>> ALB-mode (6)). Currently no host has more than 30 VMs at a time.
>>>>>>>
>>>>>>>>> We've also limited storage in QoS to use 10MB/s and 40 IOPS,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> How did you configure this limit?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> In the Data Center tab, I chose our DC and a QoS sub-tab appears,
>>>>>>> just
>>>>>>> like
>>>>>>> described here:
>>>>>>> http://www.ovirt.org/documentation/sla/network-qos/
>>>>>>>
>>>>>>>>
>>>>>>>>> but this issue
>>>>>>>>> still happens, which leads me to be concerned whether this is
>>>>>>>>> not
>>>>>>>>> just
>>>>>>>>> an
>>>>>>>>> IOPS issue; each host handles about cca. 600 LVs. Could this be
>>>>>>>>> an
>>>>>>>>> issue
>>>>>>>>> too? I remark the LVM response times are low in normal
>>>>>>>>> conditions
>>>>>>>>> (~1-2
>>>>>>>>> seconds).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> We recommend to limit number lvs per vg to 350. If you have 276
>>>>>>>> disks
>>>>>>>> on
>>>>>>>> ds1, and the disks are using snapshots, you may have too many
>>>>>>>> lvs,
>>>>>>>> which
>>>>>>>> can cause slowdowns in lvm operations.
>>>>>>>>
>>>>>>>> Can you share the output of:
>>>>>>>>
>>>>>>>> vdsm-tool dump-volume-chains
>>>>>>>> 5de4a000-a9c4-489c-8eee-10368647c413
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ok, right now no VG has more than 350 VGs so I guess this is not
>>>>>>> currently
>>>>>>> the problem. Unfortunately, I run the vdsm-tool command but it
>>>>>>> didn't
>>>>>>> end
>>>>>>> nor provide any output in cca. 1 hour, so I guess it was hanged
>>>>>>> and I
>>>>>>> stopped it. If you confirm this is a normal execution time I can
>>>>>>> leave
>>>>>>> it
>>>>>>> running whatever time it takes.
>>>>>>>
>>>>>>> host5:~# vgs
>>>>>>> VG #PV #LV #SN Attr VSize
>>>>>>> VFree
>>>>>>> 0927c050-6fb6-463c-bb37-8b8da641dcd3 1 63 0 wz--n- 499,62g
>>>>>>> 206,50g
>>>>>>> 5de4a000-a9c4-489c-8eee-10368647c413 1 335 0 wz--n- 2,00t
>>>>>>> 518,62g
>>>>>>> b13b9eac-1f2e-4a7e-bcd9-49f5f855c3d8 1 215 0 wz--n- 2,00t
>>>>>>> 495,25g
>>>>>>> sys_vg 1 6 0 wz--n- 136,44g
>>>>>>> 122,94g
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm attaching the vdsm.log, engine.log and repoplot PDF;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This is very useful, thanks. Can you send also the vdsm logs and
>>>>>>>> repoplots
>>>>>>>> from other hosts for the same timeframe?
>>>>>>>>
>>>>>>>>> if someone could
>>>>>>>>> give a hint on some additional problems in them and shed some
>>>>>>>>> light
>>>>>>>>> on
>>>>>>>>> the
>>>>>>>>> above thoughts I'd be very grateful.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Do you have sar configured on the host? having sar logs can
>>>>>>>> reveal
>>>>>>>> more
>>>>>>>> info about this timeframe.
>>>>>>>>
>>>>>>>> Do you have information about amount of io from vms during this
>>>>>>>> timeframe?
>>>>>>>> amount of io on the storage backend during this timeframe?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Not currently, but I'll be alert for the time this happens again
>>>>>>> and
>>>>>>> I'll
>>>>>>> check some sar commands related to I/O and I'll provide feedback.
>>>>>>> Nevertheless, by the time I run the commands above no machine was
>>>>>>> unresponsive and I'm still getting such huge execution times. I
>>>>>>> tried
>>>>>>> running iotop now and I see there are 2 vgck and vgs processes
>>>>>>> with a
>>>>>>> rate
>>>>>>> of ~500Kb/s each for reading.
>>>>>>>
>>>>>>> root 24446 1.3 0.0 57008 6824 ? D< 09:15 0:00
>>>>>>> /usr/sbin/lvm vgs --config devices { preferred_names =
>>>>>>> ["^/dev/mapper/"]
>>>>>>> ignore_suspended_devices=1 write_cache_state=0
>>>>>>> disable_after_error_count=3
>>>>>>> filter = [
>>>>>>>
>>>>>>>
>>>>>>> 'a|/dev/mapper/36000eb3a4f1acbc20000000000000043|/dev/mapper/36000eb3a4f1acbc200000000000000b9|/dev/mapper/360014056f0dc8930d744f83af8ddc709|',
>>>>>>> 'r|.*|' ] } global { locking_type=1 prioritise_write_locks=1
>>>>>>> wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50
>>>>>>> retain_days
>>>>>>> = 0
>>>>>>> } --noheadings --units b --nosuffix --separator |
>>>>>>> --ignoreskippedcluster
>>>>>>> -o
>>>>>>>
>>>>>>>
>>>>>>> uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
>>>>>>> b13b9eac-1f2e-4a7e-bcd9-49f5f855c3d
>>>>>>>
>>>>>>> I'm also attaching the iostat output for host 5 in a file called
>>>>>>> iostat-host5.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Interesting, all hosts are checking the same storage, but on host5
>>>>>> and
>>>>>> host6
>>>>>> the output of vgs and vgck is 10 times bigger - this explain why
>>>>>> they
>>>>>> take
>>>>>> about
>>>>>> 10 times longer to run.
>>>>>>
>>>>>> $ ls -lh vgs-*
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 221K Apr 30 08:55 vgs-host1
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 239K Apr 30 08:55 vgs-host2
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 228K Apr 30 08:55 vgs-host3
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 288K Apr 30 08:55 vgs-host4
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 2.2M Apr 30 08:55 vgs-host5
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 2.2M Apr 30 08:55 vgs-host6
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 232K Apr 30 08:55 vgs-host7
>>>>>> $ ls -lh vgck-*
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 238K Apr 30 08:55 vgck-host1
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 254K Apr 30 08:55 vgck-host2
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 244K Apr 30 08:55 vgck-host3
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 298K Apr 30 08:55 vgck-host4
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 2.0M Apr 30 08:55 vgck-host5
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 2.0M Apr 30 08:55 vgck-host6
>>>>>> -rw-r--r--. 1 nsoffer nsoffer 248K Apr 30 08:55 vgck-host7
>>>>>>
>>>>>> Can you do collect the output of "dmsetup table -v" on host 1, 5,
>>>>>> and
>>>>>> 6?
>>>>>>
>>>>>> Nir
>>>>>
>>>>>
>>>>>
>>>>> Sure, please find attached results of the command. Once again, for
>>>>> hosts
>>>>> 5
>>>>> and 6 the size is 10 times bigger than for host 1.
>>>>
>>>>
>>>> I think the issue causing slow vgs and vgck commands is stale
>>>> lvs on host 5 and 6. This may be the root caused for paused vms, but
>>>> I
>>>> cannot
>>>> explain how it is related yet.
>>>>
>>>> Comparing vgck on host1 and host5 - we can see that vgck opens
>>>> 213 /dev/mapper/* devices, actually 53 uniq devices, most of them
>>>> are opened 4 times. On host5, vgck opens 2489 devices (622 uniq).
>>>> This explains why the operation takes about 10 times longer.
>>>>
>>>> Checking dmsteup table output, we can see that host1 has 53 devices,
>>>> and host5 622 devices.
>>>>
>>>> Checking the device open count, host1 has 15 stale devices
>>>> (Open count: 0), but host5 has 597 stale devices.
>>>>
>>>> Leaving stale devices is a known issue, but we never had evidence
>>>> that it
>>>> cause
>>>> trouble except warnings in lvm commands.
>>>>
>>>> Please open an ovirt bug about this issue, and include all the files
>>>> you
>>>> sent
>>>> so far in this thread, and all the vdsm logs on host5.
>>>>
>>>> To remove the stale devices, you can do:
>>>>
>>>> for name in `dmsetup info -c -o open,name | awk '/ 0 / {print $2}'`;
>>>> do
>>>> echo "removing stale device: $name"
>>>> dmsetup remove $name
>>>> done
>>>
>>>
>>> Note that this cleanup is safe only in maintenance mode, since it
>>> will
>>> also remove
>>> the mapping for vdsm special lvs (ids, leases, inbox, outbox, master,
>>> metatada)
>>> that are accessed by vdsm regularly, and are not stale.
>>>
>>> This is also not safe if you use hosted engine, hosted engine agent
>>> should maintain
>>> its vg.
>>>
>>> The safest way to remove stale devices is to restart vdsm - it
>>> deactivates all unused
>>> lvs (except special lvs) during startup.
>>>
>>
>> Hm, I run it without even putting the host on maintenance, but I will
>> next
>> time. Fortunately it seems it didn't do any harm and everything is
>> working.
>> I'll probably do a script that will put host on maintenance, cleanup
>> or
>> restart vdsm and start again (this is not a hosted engine) if the
>> number of
>> stale devices is bigger than a threshold.
>
> I would not do anything automatic at this point except monitoring this
> issue.
>
> For cleanup - you have 2 options:
>
> - host is up - restart vdsm
> - host in maintenance - remove stale devices
>
> Based on local testing, I think the issue is this:
>
> - When connecting to storage, all lvs becomes active - this may be a
> change
> in lvm during el 7 development that we did not notice.
>
> - Each time vdsm is restarted, all active lvs that should not be active
> are
> deactivated during vdsm startup
>
> - When deactivating storage domain, vdsm does not deactivate the lvs
> leaving stale devices.
>
> Can you confirm that works like this on your system?
>
> To test this:
>
I did these tests on host5. Before switching the host to maintenance, I
run the vgck command and execution time was: 0m1.347s. Current stale
devices at that time were:
5de4a000--a9c4--489c--8eee--10368647c413-master
5de4a000--a9c4--489c--8eee--10368647c413-outbox
0927c050--6fb6--463c--bb37--8b8da641dcd3-metadata
5de4a000--a9c4--489c--8eee--10368647c413-ids
0927c050--6fb6--463c--bb37--8b8da641dcd3-leases
5de4a000--a9c4--489c--8eee--10368647c413-inbox
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-leases
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-metadata
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-ids
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-inbox
5de4a000--a9c4--489c--8eee--10368647c413-leases
0927c050--6fb6--463c--bb37--8b8da641dcd3-master
5de4a000--a9c4--489c--8eee--10368647c413-metadata
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-master
0927c050--6fb6--463c--bb37--8b8da641dcd3-outbox
0927c050--6fb6--463c--bb37--8b8da641dcd3-inbox
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-outbox
0927c050--6fb6--463c--bb37--8b8da641dcd3-ids
(seems there were not any new stale devices since yesterday after
running the loop).
> 1. Put one host to maintenance
>
> You should see some stale devices - the special lvs
> (ids, leases, master, metadata, inbox, outbox)
>
> And probably 2 OVF_STORE lvs per vg (128m lvs used to store
> vm ovf)
>
Still had the same devices as above.
> 2. Cleanup the stale devices
>
After cleaning the stale devices, those remained:
36000eb3a4f1acbc20000000000000043
360014056f0dc8930d744f83af8ddc709
36000eb3a4f1acbc200000000000000b9
> 3. Activate host
>
> You should see again about 600 stale devices, and lvm commands
> will probably run much slower as you reported.
>
Current stale devices were these:
5de4a000--a9c4--489c--8eee--10368647c413-master
5de4a000--a9c4--489c--8eee--10368647c413-outbox
0927c050--6fb6--463c--bb37--8b8da641dcd3-metadata
0927c050--6fb6--463c--bb37--8b8da641dcd3-leases
5de4a000--a9c4--489c--8eee--10368647c413-inbox
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-leases
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-metadata
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-inbox
5de4a000--a9c4--489c--8eee--10368647c413-leases
0927c050--6fb6--463c--bb37--8b8da641dcd3-master
5de4a000--a9c4--489c--8eee--10368647c413-metadata
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-master
0927c050--6fb6--463c--bb37--8b8da641dcd3-outbox
b13b9eac--1f2e--4a7e--bcd9--49f5f855c3d8-outbox
0927c050--6fb6--463c--bb37--8b8da641dcd3-inbox
> 4. Restart vdsm
>
> All the regular lvs should be inactive now, no stale devices
> should exists, except the special lvs (expected)
>
Same result as above. The strange thing is that now I don't see any VGs
running the 'vgs' command but the system's:
host5:~# vgs
VG #PV #LV #SN Attr VSize VFree
sys_vg 1 6 0 wz--n- 136,44g 122,94g
However, machines running on that host have no issues and storage is
working well. vgscan doesn't do the trick either. I tested this on host6
too and the same happens: After reactivating host I cannot see VGs with
the vgs command - however, everything seems to work quite well.
This is CentOS Linux release 7.2.1511 FWIW.
> Nir
More information about the Users
mailing list