9:06 a.m.
Hello,
I'm doing some tests related to storage latency or problems manually
created to debug and manage reactions of hosts and VMs.
What is the subsystem/process/daemon responsible to pause a VM when
problems arise on storage for the host where the VM is running?
How is determined the timeout to use to put the VM in pause mode?
Sometimes I see after clearing the problems that the VM is automatically
un-paused, sometimes no: how is this managed? Are there any counters so
that if VM has been paused and and problems are not solved in a certain
timeframe the unpause can be done only manually by the sysadmin?
Thanks in advance,
Gianluca
7:09 a.m.
On Tue, Oct 8, 2019 at 4:06 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
Hello,
I'm doing some tests related to storage latency or problems manually
created to debug and manage reactions of hosts and VMs.
What is the subsystem/process/daemon responsible to pause a VM when
problems arise on storage for the host where the VM is running?
How is determined the timeout to use to put the VM in pause mode?
Sometimes I see after clearing the problems that the VM is automatically
un-paused, sometimes no: how is this managed? Are there any counters so
that if VM has been paused and and problems are not solved in a certain
timeframe the unpause can be done only manually by the sysadmin?
Thanks in advance,
Gianluca
I have noticed that when virtual disk is virtio, the VM is not able to be
unpaused in storage unreachable for many seconds, while if I have
virtio-scsi and set high virtual disk timeout (like vSphere does on VMs
when vmware tools have been installed), then VM is able to be resumed.
The udev rule I have put into a CentOS 7 VM
inside /etc/udev/rules.d/99-ovirt.rules is this one
# Set timeout of virtio-SCSI disks to 180 secons like vSphere vmware tools
#
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="QEMU*",
ATTRS{model}=="QEMU HARDDISK*", ENV{DEVTYPE}=="disk",
RUN+="/bin/sh -c
'echo 180 > /sys$DEVPATH/device/timeout'"
What I have not understood is if it is possible to prevent at all vdsm (is
it the responsible?) to suddenly put the VM in paused state.
Eg for experiment I have iSCSI based storage domains and put this in
multipath.conf
devices {
device {
all_devs yes
# Set timeout of queuing of 5*28 = 140 seconds
# similar to vSphere APD timeout
# no_path_retry fail
no_path_retry 28
polling_interval 5
}
Then I create an iptables rule that for 100 seconds prevents host to reach
storage and a dd task that writes on disk inside VM
The effect is that vm is paused and after about 100 seconds
VM mydbsrv has recovered from paused back to up. 10/9/19 1:59:02 PM
VM mydbsrv has been paused due to storage I/O problem. 10/9/19 1:57:32 PM
VM mydbsrv has been paused. 10/9/19 1:57:32 PM
Any hint on how to prevent action of pausing the VM?
Thanks,
Gianluca
7:46 a.m.
On 10/8/19 4:06 PM, Gianluca Cecchi wrote:
Hi Gianluca
Hello,
I'm doing some tests related to storage latency or problems manually
created to debug and manage reactions of hosts and VMs.
What is the subsystem/process/daemon responsible to pause a VM when
problems arise on storage for the host where the VM is running?
It's Vdsm itself.
How is determined the timeout to use to put the VM in pause mode?
The VM is paused immediately as soon as libvirt, through QEMU, reports
IOError, to avoid data corruption. Now, when libvirt reports this error
depends laregly on the timeout set for the storage configuration, which
is done at host level, using system tools (e.g. it is not a Vdsm tunable)
Sometimes I see after clearing the problems that the VM is
automatically un-paused, sometimes no: how is this managed?
It depends on the error condition that happens. Vdsm tries to recovery
automatically when it is safe to do so. When in doubt, Vdsm always plays
it safe wrt user data
Are there any counters so that if VM has been paused and and
problems are not solved in a certain timeframe the unpause can be done
only manually by the sysadmin?
AFAIR no, because if Vdsm can't be sure, the only real option is to let
the sysadmin check and decide.
Bests,
--
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh
2:07 a.m.
On Wed, Oct 9, 2019 at 2:55 PM Francesco Romani <fromani(a)redhat.com> wrote:
On 10/8/19 4:06 PM, Gianluca Cecchi wrote:
Hi Gianluca
> Hello,
> I'm doing some tests related to storage latency or problems manually
> created to debug and manage reactions of hosts and VMs.
> What is the subsystem/process/daemon responsible to pause a VM when
> problems arise on storage for the host where the VM is running?
It's Vdsm itself.
ok
> How is determined the timeout to use to put the VM in pause mode?
The VM is paused immediately as soon as libvirt, through QEMU, reports
IOError, to avoid data corruption. Now, when libvirt reports this error
depends laregly on the timeout set for the storage configuration, which
is done at host level, using system tools (e.g. it is not a Vdsm tunable)
For test I have set this in multipath.conf of host:
devices {
device {
all_devs yes
# Set timeout of queuing of 5*28 = 140 seconds
# similar to vSphere APD timeout
# no_path_retry fail
no_path_retry 28
polling_interval 5
}
So it should wait at least 140 seconds before passing error to upper layer
correct?
> Sometimes I see after clearing the problems that the VM is
> automatically un-paused, sometimes no: how is this managed?
I noticed that if I set disk as virtio-scsi (it seems virtio has no timeout
definable and passes suddenly the error to upper layer) and disk timeout of
vm disk (through udev rule) to 180 seconds, I can block access to the
storage for example for 100 seconds and the host is able to reinstate paths
and then vm is always unpaused.
But I would like to prevent VM from pausing at all
What else to tweak?
Thanks,
Gianluca
2:55 a.m.
On 10/10/19 9:07 AM, Gianluca Cecchi wrote:
> How is determined the timeout to use to put the VM in pause mode?
The VM is paused immediately as soon as libvirt, through QEMU,
reports
IOError, to avoid data corruption. Now, when libvirt reports this
error
depends laregly on the timeout set for the storage configuration,
which
is done at host level, using system tools (e.g. it is not a Vdsm
tunable)
For test I have set this in multipath.conf of host:
devices {
device {
all_devs yes
# Set timeout of queuing of 5*28 = 140 seconds
# similar to vSphere APD timeout
# no_path_retry fail
no_path_retry 28
polling_interval 5
}
So it should wait at least 140 seconds before passing error to upper
layer correct?
AFAICT yes
> Sometimes I see after clearing the problems that the VM is
> automatically un-paused, sometimes no: how is this managed?
I noticed that if I set disk as virtio-scsi (it seems virtio has no
timeout definable and passes suddenly the error to upper layer) and
disk timeout of vm disk (through udev rule) to 180 seconds, I can
block access to the storage for example for 100 seconds and the host
is able to reinstate paths and then vm is always unpaused.
But I would like to prevent VM from pausing at all
What else to tweak?
The only way Vdsm will not pause the VM is if libvirt+qemu never reports
any ioerror, which is something I'm not sure is possible and that I'd
never recommend anyway.
Vdsm always tries hard to be super-careful with respect possible data
corruption.
Bests,
--
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh
3:44 a.m.
On Thu, Oct 10, 2019 at 9:56 AM Francesco Romani <fromani(a)redhat.com> wrote:
The only way Vdsm will not pause the VM is if libvirt+qemu never reports
any ioerror, which is something I'm not sure is possible and that I'd never
recommend anyway.
Vdsm always tries hard to be super-careful with respect possible data
corruption.
OK.
In case of storage not accessible for a bunch of seconds is more a matter
of I/O blocked than data corruption.
If no other host powers on the VM I think there is no risk of data
corruption itself, or at least no more than when you have a physical server
and for some reason the I/O operations to its physical disks (local or on a
SAN) are blocked for some tens of seconds.
The host could ever do a poweroff of the VM itself, instead of leaving
control to the underlying libvirt+qemu
I see that by default the qemu-kvm process in my oVirt 4.3.6 is spawned for
every disk with the options:
...,werror=stop,rerror=stop,...
Only for the ide channel of the CD device I have:
...,werror=report,rerror=report,readonly=on
and the manual page for qemu-kvm tells:
werror=action,rerror=action
Specify which action to take on write and read errors. Valid
actions are: "ignore"
(ignore the error and try to continue), "stop" (pause QEMU),
"report" (report the
error to the guest), "enospc" (pause QEMU only if the host
disk is full; report
the error to the guest otherwise). The default setting is
werror=enospc and
rerror=report.
So I think that if I want in any way to modify behavior I have to change
the options so that I keep "report" for both write and read errors on
virtual disks.
I'm only experimenting to see possible different options to manage
"temporary" problems at storage level, that often resolve without manual
actions in tens of seconds, sometimes due to uncorrect operations at levels
managed by other teams (network, storage, ecc).
In these circumstances experience told me it is better to "do nothing and
wait", instead of trying to taking any action that anyway will fail until
the "external" problem has been solved (automatically, thanks to logic
outside oVirt control, or manually).
It would be nice to "mimic" the behavior of vSphere in this sense and I'm
investigating possible actions to reach it...
Hope I clarified a bit the origin of my actions...
Thanks,
Gianluca
6:10 a.m.
On 10/10/19 10:44 AM, Gianluca Cecchi wrote:
On Thu, Oct 10, 2019 at 9:56 AM Francesco Romani
<fromani(a)redhat.com
<mailto:fromani@redhat.com>> wrote:
The only way Vdsm will not pause the VM is if libvirt+qemu never
reports any ioerror, which is something I'm not sure is possible
and that I'd never recommend anyway.
Vdsm always tries hard to be super-careful with respect possible
data corruption.
OK.
In case of storage not accessible for a bunch of seconds is more a
matter of I/O blocked than data corruption.
True, but we can know only ex-poste that the storage was just
temporarily unavailable, don't we?
If no other host powers on the VM I think there is no risk of data
corruption itself, or at least no more than when you have a physical
server and for some reason the I/O operations to its physical disks
(local or on a SAN) are blocked for some tens of seconds.
IMO, a storage unresponsive for tens of seconds is something which
should be uncommon and very alarming in every circumstances, especially
for physical servers.
What i'm trying to say is that yes, there probabily are ways to sidestep
this behaviour, but I think this is the wrong direction and adds
fragility rather than convenience to the system.
The host could ever do a poweroff of the VM itself, instead of
leaving
control to the underlying libvirt+qemu
I see that by default the qemu-kvm process in my oVirt 4.3.6 is
spawned for every disk with the options:
...,werror=stop,rerror=stop,...
Only for the ide channel of the CD device I have:
...,werror=report,rerror=report,readonly=on
and the manual page for qemu-kvm tells:
werror=action,rerror=action
Specify which action to take on write and read errors.
Valid actions are: "ignore"
(ignore the error and try to continue), "stop" (pause
QEMU), "report" (report the
error to the guest), "enospc" (pause QEMU only if the
host disk is full; report
the error to the guest otherwise). The default setting
is werror=enospc and
rerror=report.
So I think that if I want in any way to modify behavior I have to
change the options so that I keep "report" for both write and read
errors on virtual disks.
Yep. I don't remember what Engine allows. Worst case you can use an
hook, but once again this is making things a bit more fragile.
I'm only experimenting to see possible different options to
manage
"temporary" problems at storage level, that often resolve without
manual actions in tens of seconds, sometimes due to uncorrect
operations at levels managed by other teams (network, storage, ecc).
I think the best option is improve the current behaviour: learn why Vdsm
fails to unpause the VM and improve here.
--
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh
8:20 a.m.
On Thu, Oct 10, 2019 at 1:10 PM Francesco Romani <fromani(a)redhat.com> wrote:
On 10/10/19 10:44 AM, Gianluca Cecchi wrote:
On Thu, Oct 10, 2019 at 9:56 AM Francesco Romani <fromani(a)redhat.com>
wrote:
>
> The only way Vdsm will not pause the VM is if libvirt+qemu never reports
> any ioerror, which is something I'm not sure is possible and that I'd never
> recommend anyway.
>
> Vdsm always tries hard to be super-careful with respect possible data
> corruption.
>
>
> OK.
In case of storage not accessible for a bunch of seconds is more a matter
of I/O blocked than data corruption.
True, but we can know only ex-poste that the storage was just temporarily
unavailable, don't we?
yes but I would like to have an option to say: don't do anything for X
seconds,
both at host level and guest level.
X could be 5 seconds, or 10 seconds or 20 seconds.... according to several
needs.
If no other host powers on the VM I think there is no risk of data
corruption itself, or at least no more than when you have a physical server
and for some reason the I/O operations to its physical disks (local or on a
SAN) are blocked for some tens of seconds.
IMO, a storage unresponsive for tens of seconds is something which should
be uncommon and very alarming in every circumstances, especially for
physical servers.
What i'm trying to say is that yes, there probabily are ways to sidestep
this behaviour, but I think this is the wrong direction and adds fragility
rather than convenience to the system.
In general I agree with you on this
So I think that if I want in any way to modify behavior I have to change
the options so that I keep "report" for both write and read errors on
virtual disks.
Yep. I don't remember what Engine allows. Worst case you can use an hook,
but once again this is making things a bit more fragile.
I'm only experimenting to see possible different options to manage
"temporary" problems at storage level, that often resolve without manual
actions in tens of seconds, sometimes due to uncorrect operations at levels
managed by other teams (network, storage, ecc).
I think the best option is improve the current behaviour: learn why Vdsm
fails to unpause the VM and improve here.
yes, I'm just experimenting on possible options and their pros & cons
I see that on my 4.3.6 environment with plain CentOS 7.7 hosts the qemu-kvm
process is spawned with "werror=stop,rerror=stop" for all virtual disks
I didn't find any related option in VM edit page
In my Fedora 30 when I start a VM (with virt-manager or "virsh start") I
see that the options are not present in command line and based on qemu-kvm
manual page:
"
The default setting is werror=enospc and rerror=report
"
In the mean time I created a wrapper script for qemu-kvm that changes
command line
1)
from werror=stop to werror=report
and
from rerror=stop to rerror=report
This seems worse, in the sense that the VM is not paused at all, as
expected, but strange behavior inside it
From host point of view:
[root@ov300 ~]# virsh -r list
Id Name State
----------------------------------------------------
7 mydbsrv running
I suddenly get in VM /var/log/messsages something like
Oct 10 12:42:55 mydbsrv kernel: sd 2:0:0:1: [sdc] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 10 12:42:55 mydbsrv kernel: sd 2:0:0:1: [sdc] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 10 12:42:55 mydbsrv kernel: sd 2:0:0:1: [sdc] Sense Key : Aborted
Command [current]
Oct 10 12:42:55 mydbsrv kernel: sd 2:0:0:1: [sdc] Add. Sense: I/O process
terminated
Oct 10 12:42:55 mydbsrv kernel: sd 2:0:0:1: [sdc] CDB: Write(10) 2a 00 03
07 a8 78 00 00 08 00
Oct 10 12:42:55 mydbsrv kernel: blk_update_request: I/O error, dev sdc,
sector 50833528
Oct 10 12:42:55 mydbsrv kernel: EXT4-fs warning (device dm-3):
ext4_end_bio:322: I/O error -5 writing to inode 1573304 (offset 0 size 0
starting block 6353935)
Oct 10 12:42:55 mydbsrv kernel: Buffer I/O error on device dm-3, logical
block 6353935
Oct 10 12:42:55 mydbsrv kernel: sd 2:0:0:1: [sdc] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 10 12:42:55 mydbsrv kernel: sd 2:0:0:1: [sdc] Sense Key : Aborted
Command [current]
Oct 10 12:42:55 mydbsrv kernel: sd 2:0:0:1: [sdc] Add. Sense: I/O process
terminated
Oct 10 12:42:55 mydbsrv kernel: sd 2:0:0:1: [sdc] CDB: Write(10) 2a 00 03
07 a8 98 00 00 08 00
Oct 10 12:42:55 mydbsrv kernel: blk_update_request: I/O error, dev sdc,
sector 50833560
Oct 10 12:42:55 mydbsrv kernel: EXT4-fs warning (device dm-3):
ext4_end_bio:322: I/O error -5 writing to inode 1573308 (offset 0 size 0
starting block 6353939)
...
and only shell builtin commands apparently working inside VM making
necessary anyway a power off (from engine) and power on
[root@mydbsrv ~]# uptime
-bash: uptime: command not found
[root@mydbsrv ~]# df -h
-bash: df: command not found
[root@mydbsrv ~]# id
-bash: id: command not found
[root@mydbsrv ~]# ll
-bash: ls: command not found
[root@mydbsrv ~]#
[root@mydbsrv ~]# jobs
[root@mydbsrv ~]# ps
-bash: ps: command not found
[root@mydbsrv ~]# sync
-bash: sync: command not found
[root@mydbsrv ~]# pwd
/root
[root@mydbsrv ~]# ls
-bash: ls: command not found
[root@mydbsrv ~]# /bin/ls
-bash: /bin/ls: Input/output error
[root@mydbsrv ~]# type mount
-bash: type: mount: not found
[root@mydbsrv ~]# /bin/mount -o remount,rw /myfs
-bash: /bin/mount: Input/output error
[root@mydbsrv ~]# tail /var/log/messages
-bash: tail: command not found
[root@mydbsrv ~]# cat /var/log/messages
-bash: cat: command not found
[root@mydbsrv ~]# echo some_word
some_word
[root@mydbsrv ~]#
even after storage accessible again the wrong behavior continues.
2)
from werror=stop to werror=ignore
and
from rerror=stop to rerror=ignore
This is the more non-intrusive approach in respect of VM in my opinion,
letting it discover itself there is a problem.
In this case I get inside /var/log/messages of VM
Oct 10 12:54:00 mydbsrv chronyd[4133]: Selected source 131.175.12.3
Oct 10 12:54:00 mydbsrv chronyd[4133]: System clock wrong by 1.065994
seconds, adjustment started
Oct 10 12:54:00 mydbsrv systemd: Time has been changed
Oct 10 12:54:00 mydbsrv chronyd[4133]: System clock was stepped by 1.065994
seconds
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 289, block bitmap and bg descriptor
inconsistent: 30748 vs 28302 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 290, block bitmap and bg descriptor
inconsistent: 31999 vs 32768 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 291, block bitmap and bg descriptor
inconsistent: 28880 vs 32768 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 292, block bitmap and bg descriptor
inconsistent: 32478 vs 32768 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 293, block bitmap and bg descriptor
inconsistent: 32698 vs 32768 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 294, block bitmap and bg descriptor
inconsistent: 32151 vs 32768 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 295, block bitmap and bg descriptor
inconsistent: 31925 vs 32768 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 296, block bitmap and bg descriptor
inconsistent: 32468 vs 32768 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 0, block bitmap and bg descriptor
inconsistent: 32768 vs 1496 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 1, block bitmap and bg descriptor
inconsistent: 32768 vs 279 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_mb_generate_buddy:757: group 112, block bitmap and bg descriptor
inconsistent: 32767 vs 23733 free clusters
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_m000_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_m002_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:54:23 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_m005_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:54:27 mydbsrv kernel: JBD2: Spotted dirty metadata buffer (dev =
dm-3, blocknr = 0). There's a risk of filesystem corruption in case of
system crash.
Oct 10 12:54:28 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_cjq0_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:54:31 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_pmon_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:54:49 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_reco_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:54:49 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_tmon_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:54:56 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_dia0_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:54:58 mydbsrv kernel: EXT4-fs error: 49 callbacks suppressed
Oct 10 12:54:58 mydbsrv kernel: EXT4-fs error (device dm-3):
ext4_mb_generate_buddy:757: group 192, block bitmap and bg descriptor
inconsistent: 32379 vs 24527 free clusters
Oct 10 12:54:58 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_tt00_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:07 mydbsrv chronyd[4133]: Selected source 131.175.12.6
Oct 10 12:55:21 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_gen1_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:22 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_mman_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:22 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_psp0_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:22 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_dbrm_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:22 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_pxmn_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:22 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_smco_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:22 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_mmnl_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:26 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_lgwr_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:26 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_gen0_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
Oct 10 12:55:26 mydbsrv kernel: EXT4-fs error (device dm-2):
ext4_find_dest_de:1829: inode #928680: block 3679007: comm ora_mmon_test:
bad entry in directory: rec_len is smaller than minimal - offset=0(0),
inode=0, rec_len=0, name_len=0
and then when the 100 seconds of artificial storage access inhibition ends,
the VM is able again to be accessible.
[root@mydbsrv ~]# id
uid=0(root) gid=0(root) groups=0(root)
[root@mydbsrv ~]# uptime
12:56:36 up 3 min, 1 user, load average: 0.28, 0.38, 0.17
[root@mydbsrv ~]#
[root@mydbsrv log]# time dd if=/dev/zero bs=1024k count=10240
of=/myfs/testfile
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 60.785 s, 177 MB/s
real 1m1.771s
user 0m0.016s
sys 0m10.459s
[root@mydbsrv log]#
Obviously, as mentioned in messages, this behavior could potentially lead
to fs/journal corruption...
Gianluca
1916
days inactive
1918
days old
7 comments
2 participants
participants (2)
-
Francesco Romani -
Gianluca Cecchi