3:15 p.m.
Hi Guys,
I had a situation 2 times that due to unexpected power outage something went wrong and VMs
on glusterfs where not recoverable.
Gluster heal did not help and I could not start the VMs any more.
Is there a way to make such setup bulletproof?
Does it matter which volume type I choose - raw or qcow2? Or thin provision versus
reallocated?
Any other advise?
3:29 p.m.
IMO this is best handled at hardware level with UPS and battery/flash
backed controllers. Can you share more details about your oVirt setup? How
many servers are you working with andare you using replica 3 or replica 3
arbiter?
On Thu, Oct 8, 2020 at 9:15 AM Jarosław Prokopowski <jprokopowski(a)gmail.com>
wrote:
Hi Guys,
I had a situation 2 times that due to unexpected power outage something
went wrong and VMs on glusterfs where not recoverable.
Gluster heal did not help and I could not start the VMs any more.
Is there a way to make such setup bulletproof?
Does it matter which volume type I choose - raw or qcow2? Or thin
provision versus reallocated?
Any other advise?
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MRM6H2YENBP...
4:11 p.m.
Hi Jayme, there is UPS but anyway the outages happened. We have also Raritan KVM but it is
not supported by oVirt.
The setup is 6 hosts - Tow pairs of 3 hosts each using one replica 3 volume.
BTW what would be the best gluster volume solution for 6+ hosts?
9:30 p.m.
are you using JBOD bricks or do you have some sort of RAID for each of
the bricks?
Are you using sharding?
-wk
On 10/8/2020 6:11 AM, Jarosław Prokopowski wrote:
> Hi Jayme, there is UPS but anyway the outages happened. We have also Raritan KVM but
it is not supported by oVirt.
> The setup is 6 hosts - Tow pairs of 3 hosts each using one replica 3 volume.
> BTW what would be the best gluster volume solution for 6+ hosts?
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AUPSDDU3665...
4:04 p.m.
Hmm, I'm not sure. I just created glusterfs volumes on LVM volumes, changed ownership
to vdsm.kvm and applied virt group. Then I added it to oVirt as storage for VMs
9:52 p.m.
Based on the logs you shared, it looks like a network issue - but it could always be
something else.
If you ever experience something like that situation, please share the logs immediately
and add the gluster mailing list - in order to get assistance with the root cause.
Best Regards,
Strahil Nikolov
В петък, 9 октомври 2020 г., 16:26:14 Гринуич+3, Jarosław Prokopowski
<jprokopowski(a)gmail.com> написа:
Hmm, I'm not sure. I just created glusterfs volumes on LVM volumes, changed ownership
to vdsm.kvm and applied virt group. Then I added it to oVirt as storage for VMs
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/DJEOW53SSPB...
6:17 p.m.
Thanks Strahil
The data center is remote so I will definitely ask the lab guys to ensure the switch is
connected to battery supported power socket.
So the gluster's weak point is actually the switch in the network? Can it have
difficulty finding out which version of data is correct after the switch was off for some
time?
10:53 a.m.
Hi Jaroslaw,
That point was from someone else. I don't think that gluster has a such weak point.
The only weak point I have seen is the infrastructure it relies ontop and of course the
built-in limitations it has.
You need to verify the following:
- mount options are important . Using 'nobarrier' but without radi-controller
protection is devastating. Also I use the following option when using gluster + selinux in
enforcing mode:
context=system_u:object_r:glusterd_brick_t:s0 - it tells the kernel what is the selinux
context on all files/dirs in the gluster brick and this reduces I/O requests to the
bricks
My mount options are:
noatime,nodiratime,inode64,nouuid,context="system_u:object_r:glusterd_brick_t:s0"
- Next is your FS - if you use HW raid controller , you need to specify the sunit= and
swidth= for the 'mkfs.xfs' (and don't forget the '-i size=512')
This tells the XFS about the hardware beneath
- If you use thin LVM , you need to be sure that your '_tmeta' LV of the Thinpool
LV is not over a VDO device as it doesn't dedupe quite good
I'm using VDO in 'emulate512' as my 'PHY-SEC' is 4096 and oVirt
doesn't like it :) . You can check yours via 'lsblk -t'.
- Configure and tune your VDO. I think that 1 VDO = 1 Fast disk (NVMe/SSD) as I'm not
very good in tuning VDO. If you need dedupe - check RedHat's documentation about the
indexing as the defaults are not optimal.
- Next is the disk scheduler. In case you use NVMe - the linux kernel is taking care of it
, but for SSDs and large HW arrays - you can enable the multiqueue and switch to
'none' via UDEV rules.Of course , testing is needed for every prod environment :)
Also consider using noop/none I/O scheduler in the VMs as you don't want to reorder
I/O requests on VM level , just to do it on Host level.
- You can set your CPU to avoid switching to lower C states -> that adds extra latency
for the host/VM processes
- Transparent Huge Pages can be a real problem , especially with large VMs. oVirt 4.4.x
now should support native Huge and Gumbo pages which will reduce the stress over the OS.
- vm.swappiness, vm.dirty_background**** , vm.dirty_*** settings. You can check what RH
gluster storage is using the ones in the redhat-storage-server rpms:
in ftp://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/
They control the behaviour of the system when to start flushing memory to disk and when to
block any process until all memory is flushed.
Best Regards,
Strahil Nikolov
В събота, 10 октомври 2020 г., 18:18:55 Гринуич+3, Jarosław Prokopowski
<jprokopowski(a)gmail.com> написа:
Thanks Strahil
The data center is remote so I will definitely ask the lab guys to ensure the switch is
connected to battery supported power socket.
So the gluster's weak point is actually the switch in the network? Can it have
difficulty finding out which version of data is correct after the switch was off for some
time?
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/VFP2FX2YRAP...
4:33 p.m.
Hi Nikolov,
Thanks for the very interesting answer :-)
I do not use any raid controller. I was hoping glusterfs would take care of fault
tolerance but apparently it failed.
I have one Samsung 1TB SSD drives in each server for VM storage. I see it is of type
"multipath". There is XFS filesystem over standard LVM (not thin).
Mount options are: inode64,noatime,nodiratime
SELinux was in permissive mode.
I must read more about the things you described as have never dived into it.
Please let me know if you have any suggestions :-)
Thanks a lot!
Jarek
7:17 p.m.
One recommendation is to get rid of the multipath for your SSD.
Replica 3 volumes are quite resilient and I'm really surprised it happened to you.
For the multipath stuff , you can create something like this:
[root@ovirt1 ~]# cat /etc/multipath/conf.d/blacklist.conf
blacklist {
wwid Crucial_CT256MX100SSD1_14390D52DCF5
}
As you are running multipath already , just run the following to get the wwid of your ssd
:
multipath -v4 | grep 'got wwid of'
What were the gluster vol options you were running with ? oVirt is running the volume with
'performance.strict-o-direct' and Direct I/O , so you should not loose any data.
Best Regards,
Strahil Nikolov
В вторник, 13 октомври 2020 г., 16:35:26 Гринуич+3, Jarosław Prokopowski
<jprokopowski(a)gmail.com> написа:
Hi Nikolov,
Thanks for the very interesting answer :-)
I do not use any raid controller. I was hoping glusterfs would take care of fault
tolerance but apparently it failed.
I have one Samsung 1TB SSD drives in each server for VM storage. I see it is of type
"multipath". There is XFS filesystem over standard LVM (not thin).
Mount options are: inode64,noatime,nodiratime
SELinux was in permissive mode.
I must read more about the things you described as have never dived into it.
Please let me know if you have any suggestions :-)
Thanks a lot!
Jarek
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/RBIDHY6P3KK...
12:28 p.m.
Thanks. I will get rid of multipath.
I did not set performance.strict-o-direct specifically, only changed permissions of the
volume to vdsm.kvm and applied the virt gourp.
Now is see performance.strict-o-direct was off. Could it be the reason of the data loss?
Direct I/O is enabled in oVirt by gluster mount option "-o
direct-io-mode=enable" right?
Below is full list of the volume options.
Option Value
------ -----
cluster.lookup-unhashed on
cluster.lookup-optimize on
cluster.min-free-disk 10%
cluster.min-free-inodes 5%
cluster.rebalance-stats off
cluster.subvols-per-directory (null)
cluster.readdir-optimize off
cluster.rsync-hash-regex (null)
cluster.extra-hash-regex (null)
cluster.dht-xattr-name trusted.glusterfs.dht
cluster.randomize-hash-range-by-gfid off
cluster.rebal-throttle normal
cluster.lock-migration off
cluster.force-migration off
cluster.local-volume-name (null)
cluster.weighted-rebalance on
cluster.switch-pattern (null)
cluster.entry-change-log on
cluster.read-subvolume (null)
cluster.read-subvolume-index -1
cluster.read-hash-mode 1
cluster.background-self-heal-count 8
cluster.metadata-self-heal off
cluster.data-self-heal off
cluster.entry-self-heal off
cluster.self-heal-daemon on
cluster.heal-timeout 600
cluster.self-heal-window-size 1
cluster.data-change-log on
cluster.metadata-change-log on
cluster.data-self-heal-algorithm full
cluster.eager-lock enable
disperse.eager-lock on
disperse.other-eager-lock on
disperse.eager-lock-timeout 1
disperse.other-eager-lock-timeout 1
cluster.quorum-type auto
cluster.quorum-count (null)
cluster.choose-local off
cluster.self-heal-readdir-size 1KB
cluster.post-op-delay-secs 1
cluster.ensure-durability on
cluster.consistent-metadata no
cluster.heal-wait-queue-length 128
cluster.favorite-child-policy none
cluster.full-lock yes
diagnostics.latency-measurement off
diagnostics.dump-fd-stats off
diagnostics.count-fop-hits off
diagnostics.brick-log-level INFO
diagnostics.client-log-level INFO
diagnostics.brick-sys-log-level CRITICAL
diagnostics.client-sys-log-level CRITICAL
diagnostics.brick-logger (null)
diagnostics.client-logger (null)
diagnostics.brick-log-format (null)
diagnostics.client-log-format (null)
diagnostics.brick-log-buf-size 5
diagnostics.client-log-buf-size 5
diagnostics.brick-log-flush-timeout 120
diagnostics.client-log-flush-timeout 120
diagnostics.stats-dump-interval 0
diagnostics.fop-sample-interval 0
diagnostics.stats-dump-format json
diagnostics.fop-sample-buf-size 65535
diagnostics.stats-dnscache-ttl-sec 86400
performance.cache-max-file-size 0
performance.cache-min-file-size 0
performance.cache-refresh-timeout 1
performance.cache-priority
performance.cache-size 32MB
performance.io-thread-count 16
performance.high-prio-threads 16
performance.normal-prio-threads 16
performance.low-prio-threads 32
performance.least-prio-threads 1
performance.enable-least-priority on
performance.iot-watchdog-secs (null)
performance.iot-cleanup-disconnected-reqsoff
performance.iot-pass-through false
performance.io-cache-pass-through false
performance.cache-size 128MB
performance.qr-cache-timeout 1
performance.cache-invalidation false
performance.ctime-invalidation false
performance.flush-behind on
performance.nfs.flush-behind on
performance.write-behind-window-size 1MB
performance.resync-failed-syncs-after-fsyncoff
performance.nfs.write-behind-window-size1MB
performance.strict-o-direct off
performance.nfs.strict-o-direct off
performance.strict-write-ordering off
performance.nfs.strict-write-ordering off
performance.write-behind-trickling-writeson
performance.aggregate-size 128KB
performance.nfs.write-behind-trickling-writeson
performance.lazy-open yes
performance.read-after-open yes
performance.open-behind-pass-through false
performance.read-ahead-page-count 4
performance.read-ahead-pass-through false
performance.readdir-ahead-pass-through false
performance.md-cache-pass-through false
performance.md-cache-timeout 1
performance.cache-swift-metadata true
performance.cache-samba-metadata false
performance.cache-capability-xattrs true
performance.cache-ima-xattrs true
performance.md-cache-statfs off
performance.xattr-cache-list
performance.nl-cache-pass-through false
features.encryption off
network.frame-timeout 1800
network.ping-timeout 42
network.tcp-window-size (null)
client.ssl off
network.remote-dio enable
client.event-threads 4
client.tcp-user-timeout 0
client.keepalive-time 20
client.keepalive-interval 2
client.keepalive-count 9
network.tcp-window-size (null)
network.inode-lru-limit 16384
auth.allow *
auth.reject (null)
transport.keepalive 1
server.allow-insecure on
server.root-squash off
server.all-squash off
server.anonuid 65534
server.anongid 65534
server.statedump-path /var/run/gluster
server.outstanding-rpc-limit 64
server.ssl off
auth.ssl-allow *
server.manage-gids off
server.dynamic-auth on
client.send-gids on
server.gid-timeout 300
server.own-thread (null)
server.event-threads 4
server.tcp-user-timeout 42
server.keepalive-time 20
server.keepalive-interval 2
server.keepalive-count 9
transport.listen-backlog 1024
transport.address-family inet
performance.write-behind on
performance.read-ahead off
performance.readdir-ahead on
performance.io-cache off
performance.open-behind on
performance.quick-read off
performance.nl-cache off
performance.stat-prefetch on
performance.client-io-threads on
performance.nfs.write-behind on
performance.nfs.read-ahead off
performance.nfs.io-cache off
performance.nfs.quick-read off
performance.nfs.stat-prefetch off
performance.nfs.io-threads off
performance.force-readdirp true
performance.cache-invalidation false
performance.global-cache-invalidation true
features.uss off
features.snapshot-directory .snaps
features.show-snapshot-directory off
features.tag-namespaces off
network.compression off
network.compression.window-size -15
network.compression.mem-level 8
network.compression.min-size 0
network.compression.compression-level -1
network.compression.debug false
features.default-soft-limit 80%
features.soft-timeout 60
features.hard-timeout 5
features.alert-time 86400
features.quota-deem-statfs off
geo-replication.indexing off
geo-replication.indexing off
geo-replication.ignore-pid-check off
geo-replication.ignore-pid-check off
features.quota off
features.inode-quota off
features.bitrot disable
debug.trace off
debug.log-history no
debug.log-file no
debug.exclude-ops (null)
debug.include-ops (null)
debug.error-gen off
debug.error-failure (null)
debug.error-number (null)
debug.random-failure off
debug.error-fops (null)
nfs.disable on
features.read-only off
features.worm off
features.worm-file-level off
features.worm-files-deletable on
features.default-retention-period 120
features.retention-mode relax
features.auto-commit-period 180
storage.linux-aio off
storage.batch-fsync-mode reverse-fsync
storage.batch-fsync-delay-usec 0
storage.owner-uid 36
storage.owner-gid 36
storage.node-uuid-pathinfo off
storage.health-check-interval 30
storage.build-pgfid off
storage.gfid2path on
storage.gfid2path-separator :
storage.reserve 1
storage.health-check-timeout 10
storage.fips-mode-rchecksum off
storage.force-create-mode 0000
storage.force-directory-mode 0000
storage.create-mask 0777
storage.create-directory-mask 0777
storage.max-hardlinks 100
features.ctime on
config.gfproxyd off
cluster.server-quorum-type server
cluster.server-quorum-ratio 0
changelog.changelog off
changelog.changelog-dir {{ brick.path }}/.glusterfs/changelogs
changelog.encoding ascii
changelog.rollover-time 15
changelog.fsync-interval 5
changelog.changelog-barrier-timeout 120
changelog.capture-del-path off
features.barrier disable
features.barrier-timeout 120
features.trash off
features.trash-dir .trashcan
features.trash-eliminate-path (null)
features.trash-max-filesize 5MB
features.trash-internal-op off
cluster.enable-shared-storage disable
locks.trace off
locks.mandatory-locking off
cluster.disperse-self-heal-daemon enable
cluster.quorum-reads no
client.bind-insecure (null)
features.shard on
features.shard-block-size 64MB
features.shard-lru-limit 16384
features.shard-deletion-rate 100
features.scrub-throttle lazy
features.scrub-freq biweekly
features.scrub false
features.expiry-time 120
features.cache-invalidation off
features.cache-invalidation-timeout 60
features.leases off
features.lease-lock-recall-timeout 60
disperse.background-heals 8
disperse.heal-wait-qlength 128
cluster.heal-timeout 600
dht.force-readdirp on
disperse.read-policy gfid-hash
cluster.shd-max-threads 8
cluster.shd-wait-qlength 10000
cluster.locking-scheme granular
cluster.granular-entry-heal no
features.locks-revocation-secs 0
features.locks-revocation-clear-all false
features.locks-revocation-max-blocked 0
features.locks-monkey-unlocking false
features.locks-notify-contention no
features.locks-notify-contention-delay 5
disperse.shd-max-threads 1
disperse.shd-wait-qlength 1024
disperse.cpu-extensions auto
disperse.self-heal-window-size 1
cluster.use-compound-fops off
performance.parallel-readdir off
performance.rda-request-size 131072
performance.rda-low-wmark 4096
performance.rda-high-wmark 128KB
performance.rda-cache-limit 10MB
performance.nl-cache-positive-entry false
performance.nl-cache-limit 10MB
performance.nl-cache-timeout 60
cluster.brick-multiplex off
cluster.max-bricks-per-process 250
disperse.optimistic-change-log on
disperse.stripe-cache 4
cluster.halo-enabled False
cluster.halo-shd-max-latency 99999
cluster.halo-nfsd-max-latency 5
cluster.halo-max-latency 5
cluster.halo-max-replicas 99999
cluster.halo-min-replicas 2
features.selinux on
cluster.daemon-log-level INFO
debug.delay-gen off
delay-gen.delay-percentage 10%
delay-gen.delay-duration 100000
delay-gen.enable
disperse.parallel-writes on
features.sdfs off
features.cloudsync off
features.ctime on
ctime.noatime on
feature.cloudsync-storetype (null)
features.enforce-mandatory-lock off
1:15 p.m.
strict-o-direct just allows the app to define if direct I/O is needed and yes, that could
be a reason for your data loss.
The good thing is that the feature is part of the virt group and there is a "Optimize
for Virt" button somewhere in the UI . Yet, I prefer the manual approach of building
gluster volumes ,as UI's primary focus is oVirt (quite natural , right).
Best Regards,
Strahil Nikolov
В сряда, 14 октомври 2020 г., 12:30:42 Гринуич+3, Jarosław Prokopowski
<jprokopowski(a)gmail.com> написа:
Thanks. I will get rid of multipath.
I did not set performance.strict-o-direct specifically, only changed permissions of the
volume to vdsm.kvm and applied the virt gourp.
Now is see performance.strict-o-direct was off. Could it be the reason of the data loss?
Direct I/O is enabled in oVirt by gluster mount option "-o
direct-io-mode=enable" right?
Below is full list of the volume options.
Option Value
------ -----
cluster.lookup-unhashed on
cluster.lookup-optimize on
cluster.min-free-disk 10%
cluster.min-free-inodes 5%
cluster.rebalance-stats off
cluster.subvols-per-directory (null)
cluster.readdir-optimize off
cluster.rsync-hash-regex (null)
cluster.extra-hash-regex (null)
cluster.dht-xattr-name trusted.glusterfs.dht
cluster.randomize-hash-range-by-gfid off
cluster.rebal-throttle normal
cluster.lock-migration off
cluster.force-migration off
cluster.local-volume-name (null)
cluster.weighted-rebalance on
cluster.switch-pattern (null)
cluster.entry-change-log on
cluster.read-subvolume (null)
cluster.read-subvolume-index -1
cluster.read-hash-mode 1
cluster.background-self-heal-count 8
cluster.metadata-self-heal off
cluster.data-self-heal off
cluster.entry-self-heal off
cluster.self-heal-daemon on
cluster.heal-timeout 600
cluster.self-heal-window-size 1
cluster.data-change-log on
cluster.metadata-change-log on
cluster.data-self-heal-algorithm full
cluster.eager-lock enable
disperse.eager-lock on
disperse.other-eager-lock on
disperse.eager-lock-timeout 1
disperse.other-eager-lock-timeout 1
cluster.quorum-type auto
cluster.quorum-count (null)
cluster.choose-local off
cluster.self-heal-readdir-size 1KB
cluster.post-op-delay-secs 1
cluster.ensure-durability on
cluster.consistent-metadata no
cluster.heal-wait-queue-length 128
cluster.favorite-child-policy none
cluster.full-lock yes
diagnostics.latency-measurement off
diagnostics.dump-fd-stats off
diagnostics.count-fop-hits off
diagnostics.brick-log-level INFO
diagnostics.client-log-level INFO
diagnostics.brick-sys-log-level CRITICAL
diagnostics.client-sys-log-level CRITICAL
diagnostics.brick-logger (null)
diagnostics.client-logger (null)
diagnostics.brick-log-format (null)
diagnostics.client-log-format (null)
diagnostics.brick-log-buf-size 5
diagnostics.client-log-buf-size 5
diagnostics.brick-log-flush-timeout 120
diagnostics.client-log-flush-timeout 120
diagnostics.stats-dump-interval 0
diagnostics.fop-sample-interval 0
diagnostics.stats-dump-format json
diagnostics.fop-sample-buf-size 65535
diagnostics.stats-dnscache-ttl-sec 86400
performance.cache-max-file-size 0
performance.cache-min-file-size 0
performance.cache-refresh-timeout 1
performance.cache-priority
performance.cache-size 32MB
performance.io-thread-count 16
performance.high-prio-threads 16
performance.normal-prio-threads 16
performance.low-prio-threads 32
performance.least-prio-threads 1
performance.enable-least-priority on
performance.iot-watchdog-secs (null)
performance.iot-cleanup-disconnected-reqsoff
performance.iot-pass-through false
performance.io-cache-pass-through false
performance.cache-size 128MB
performance.qr-cache-timeout 1
performance.cache-invalidation false
performance.ctime-invalidation false
performance.flush-behind on
performance.nfs.flush-behind on
performance.write-behind-window-size 1MB
performance.resync-failed-syncs-after-fsyncoff
performance.nfs.write-behind-window-size1MB
performance.strict-o-direct off
performance.nfs.strict-o-direct off
performance.strict-write-ordering off
performance.nfs.strict-write-ordering off
performance.write-behind-trickling-writeson
performance.aggregate-size 128KB
performance.nfs.write-behind-trickling-writeson
performance.lazy-open yes
performance.read-after-open yes
performance.open-behind-pass-through false
performance.read-ahead-page-count 4
performance.read-ahead-pass-through false
performance.readdir-ahead-pass-through false
performance.md-cache-pass-through false
performance.md-cache-timeout 1
performance.cache-swift-metadata true
performance.cache-samba-metadata false
performance.cache-capability-xattrs true
performance.cache-ima-xattrs true
performance.md-cache-statfs off
performance.xattr-cache-list
performance.nl-cache-pass-through false
features.encryption off
network.frame-timeout 1800
network.ping-timeout 42
network.tcp-window-size (null)
client.ssl off
network.remote-dio enable
client.event-threads 4
client.tcp-user-timeout 0
client.keepalive-time 20
client.keepalive-interval 2
client.keepalive-count 9
network.tcp-window-size (null)
network.inode-lru-limit 16384
auth.allow *
auth.reject (null)
transport.keepalive 1
server.allow-insecure on
server.root-squash off
server.all-squash off
server.anonuid 65534
server.anongid 65534
server.statedump-path /var/run/gluster
server.outstanding-rpc-limit 64
server.ssl off
auth.ssl-allow *
server.manage-gids off
server.dynamic-auth on
client.send-gids on
server.gid-timeout 300
server.own-thread (null)
server.event-threads 4
server.tcp-user-timeout 42
server.keepalive-time 20
server.keepalive-interval 2
server.keepalive-count 9
transport.listen-backlog 1024
transport.address-family inet
performance.write-behind on
performance.read-ahead off
performance.readdir-ahead on
performance.io-cache off
performance.open-behind on
performance.quick-read off
performance.nl-cache off
performance.stat-prefetch on
performance.client-io-threads on
performance.nfs.write-behind on
performance.nfs.read-ahead off
performance.nfs.io-cache off
performance.nfs.quick-read off
performance.nfs.stat-prefetch off
performance.nfs.io-threads off
performance.force-readdirp true
performance.cache-invalidation false
performance.global-cache-invalidation true
features.uss off
features.snapshot-directory .snaps
features.show-snapshot-directory off
features.tag-namespaces off
network.compression off
network.compression.window-size -15
network.compression.mem-level 8
network.compression.min-size 0
network.compression.compression-level -1
network.compression.debug false
features.default-soft-limit 80%
features.soft-timeout 60
features.hard-timeout 5
features.alert-time 86400
features.quota-deem-statfs off
geo-replication.indexing off
geo-replication.indexing off
geo-replication.ignore-pid-check off
geo-replication.ignore-pid-check off
features.quota off
features.inode-quota off
features.bitrot disable
debug.trace off
debug.log-history no
debug.log-file no
debug.exclude-ops (null)
debug.include-ops (null)
debug.error-gen off
debug.error-failure (null)
debug.error-number (null)
debug.random-failure off
debug.error-fops (null)
nfs.disable on
features.read-only off
features.worm off
features.worm-file-level off
features.worm-files-deletable on
features.default-retention-period 120
features.retention-mode relax
features.auto-commit-period 180
storage.linux-aio off
storage.batch-fsync-mode reverse-fsync
storage.batch-fsync-delay-usec 0
storage.owner-uid 36
storage.owner-gid 36
storage.node-uuid-pathinfo off
storage.health-check-interval 30
storage.build-pgfid off
storage.gfid2path on
storage.gfid2path-separator :
storage.reserve 1
storage.health-check-timeout 10
storage.fips-mode-rchecksum off
storage.force-create-mode 0000
storage.force-directory-mode 0000
storage.create-mask 0777
storage.create-directory-mask 0777
storage.max-hardlinks 100
features.ctime on
config.gfproxyd off
cluster.server-quorum-type server
cluster.server-quorum-ratio 0
changelog.changelog off
changelog.changelog-dir {{ brick.path }}/.glusterfs/changelogs
changelog.encoding ascii
changelog.rollover-time 15
changelog.fsync-interval 5
changelog.changelog-barrier-timeout 120
changelog.capture-del-path off
features.barrier disable
features.barrier-timeout 120
features.trash off
features.trash-dir .trashcan
features.trash-eliminate-path (null)
features.trash-max-filesize 5MB
features.trash-internal-op off
cluster.enable-shared-storage disable
locks.trace off
locks.mandatory-locking off
cluster.disperse-self-heal-daemon enable
cluster.quorum-reads no
client.bind-insecure (null)
features.shard on
features.shard-block-size 64MB
features.shard-lru-limit 16384
features.shard-deletion-rate 100
features.scrub-throttle lazy
features.scrub-freq biweekly
features.scrub false
features.expiry-time 120
features.cache-invalidation off
features.cache-invalidation-timeout 60
features.leases off
features.lease-lock-recall-timeout 60
disperse.background-heals 8
disperse.heal-wait-qlength 128
cluster.heal-timeout 600
dht.force-readdirp on
disperse.read-policy gfid-hash
cluster.shd-max-threads 8
cluster.shd-wait-qlength 10000
cluster.locking-scheme granular
cluster.granular-entry-heal no
features.locks-revocation-secs 0
features.locks-revocation-clear-all false
features.locks-revocation-max-blocked 0
features.locks-monkey-unlocking false
features.locks-notify-contention no
features.locks-notify-contention-delay 5
disperse.shd-max-threads 1
disperse.shd-wait-qlength 1024
disperse.cpu-extensions auto
disperse.self-heal-window-size 1
cluster.use-compound-fops off
performance.parallel-readdir off
performance.rda-request-size 131072
performance.rda-low-wmark 4096
performance.rda-high-wmark 128KB
performance.rda-cache-limit 10MB
performance.nl-cache-positive-entry false
performance.nl-cache-limit 10MB
performance.nl-cache-timeout 60
cluster.brick-multiplex off
cluster.max-bricks-per-process 250
disperse.optimistic-change-log on
disperse.stripe-cache 4
cluster.halo-enabled False
cluster.halo-shd-max-latency 99999
cluster.halo-nfsd-max-latency 5
cluster.halo-max-latency 5
cluster.halo-max-replicas 99999
cluster.halo-min-replicas 2
features.selinux on
cluster.daemon-log-level INFO
debug.delay-gen off
delay-gen.delay-percentage 10%
delay-gen.delay-duration 100000
delay-gen.enable
disperse.parallel-writes on
features.sdfs off
features.cloudsync off
features.ctime on
ctime.noatime on
feature.cloudsync-storetype (null)
features.enforce-mandatory-lock off
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/C5YTREAG33N...
10:43 p.m.
Hi Jaroslaw,
it's more important to find the root cause of the data loss , as this is definately
not supposed to happen (I got myself several power outages without issues).
Do you keep the logs ?
For now , check if your gluster settings (gluster volume info VOL) matches the settings in
the virt group (/var/lib/glusterd/group/virt - or somethinhg like that).
Best Regards,
Strahil Nikolov
В четвъртък, 8 октомври 2020 г., 15:16:10 Гринуич+3, Jarosław Prokopowski
<jprokopowski(a)gmail.com> написа:
Hi Guys,
I had a situation 2 times that due to unexpected power outage something went wrong and VMs
on glusterfs where not recoverable.
Gluster heal did not help and I could not start the VMs any more.
Is there a way to make such setup bulletproof?
Does it matter which volume type I choose - raw or qcow2? Or thin provision versus
reallocated?
Any other advise?
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MRM6H2YENBP...
4:03 p.m.
Hi Strahil,
I remember during after creating the volume I applied the virt group to it.
Volume info:
----------------
Volume Name: data
Type: Replicate
Volume ID: 05842cd6-7f16-4329-9ffd-64a0b4366fbe
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: host1storage:/gluster_bricks/data/data
Brick2: host2storage:/gluster_bricks/data/data
Brick3: host3storage:/gluster_bricks/data/data
Options Reconfigured:
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
storage.owner-gid: 36
storage.owner-uid: 36
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
cluster.choose-local: off
client.event-threads: 4
server.event-threads: 4
I do not have full logs but I have some saved.
/var/log/messages:
-------------------------
Sep 14 08:36:20 host1 vdsm[4301]: ERROR Unhandled exception in <Task discardable
<UpdateVolumes vm=fd5123cb-9364-448d-b41c-8a48fb1826c5 at 0x7f1244099050>
timeout=30.0, duration=0.01 at 0x7f1244099210>#012Traceback (most recent call
last):#012 File "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315,
in _execute_task#012 task()#012 File
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012
self._callable()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 315, in
__call__#012 self._execute()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 357, in
_execute#012 self._vm.updateDriveVolume(drive)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4189, in
updateDriveVolume#012 vmDrive.volumeID)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6101, in
_getVolumeSize#012 (domainID, volumeID))#012StorageUnavailableError: Unable to get
volume size for doma
in 88f5972f-58bd-469f-bc77-5bf3b1802291 volume cdf313d7-bed3-4fae-a803-1297cdf8c82f
Sep 14 08:37:20 host1 vdsm[4301]: ERROR Unhandled exception in <Task discardable
<UpdateVolumes vm=fd5123cb-9364-448d-b41c-8a48fb1826c5 at 0x7f12442efad0>
timeout=30.0, duration=0.00 at 0x7f1244078490>#012Traceback (most recent call
last):#012 File "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315,
in _execute_task#012 task()#012 File
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012
self._callable()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 315, in
__call__#012 self._execute()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 357, in
_execute#012 self._vm.updateDriveVolume(drive)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4189, in
updateDriveVolume#012 vmDrive.volumeID)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6101, in
_getVolumeSize#012 (domainID, volumeID))#012StorageUnavailableError: Unable to get
volume size for doma
in 88f5972f-58bd-469f-bc77-5bf3b1802291 volume cdf313d7-bed3-4fae-a803-1297cdf8c82f
Sep 14 08:38:20 host1 vdsm[4301]: ERROR Unhandled exception in <Task discardable
<UpdateVolumes vm=fd5123cb-9364-448d-b41c-8a48fb1826c5 at 0x7f12045aa550>
timeout=30.0, duration=0.00 at 0x7f12045aaa90>#012Traceback (most recent call
last):#012 File "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315,
in _execute_task#012 task()#012 File
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012
self._callable()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 315, in
__call__#012 self._execute()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 357, in
_execute#012 self._vm.updateDriveVolume(drive)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4189, in
updateDriveVolume#012 vmDrive.volumeID)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6101, in
_getVolumeSize#012 (domainID, volumeID))#012StorageUnavailableError: Unable to get
volume size for doma
in 88f5972f-58bd-469f-bc77-5bf3b1802291 volume cdf313d7-bed3-4fae-a803-1297cdf8c82f
Sep 14 08:39:20 host1 vdsm[4301]: ERROR Unhandled exception in <Task discardable
<UpdateVolumes vm=fd5123cb-9364-448d-b41c-8a48fb1826c5 at 0x7f12441d6f50>
timeout=30.0, duration=0.01 at 0x7f1287f189d0>#012Traceback (most recent call
last):#012 File "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315,
in _execute_task#012 task()#012 File
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012
self._callable()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 315, in
__call__#012 self._execute()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 357, in
_execute#012 self._vm.updateDriveVolume(drive)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4189, in
updateDriveVolume#012 vmDrive.volumeID)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6101, in
_getVolumeSize#012 (domainID, volumeID))#012StorageUnavailableError: Unable to get
volume size for doma
in 88f5972f-58bd-469f-bc77-5bf3b1802291 volume cdf313d7-bed3-4fae-a803-1297cdf8c82f
Sep 14 08:40:01 host1 systemd: Created slice User Slice of root.
Sep 14 08:40:01 host1 systemd: Started Session 52513 of user root.
Sep 14 08:40:01 host1 systemd: Removed slice User Slice of root.
Sep 14 08:40:20 host1 vdsm[4301]: ERROR Unhandled exception in <Task discardable
<UpdateVolumes vm=fd5123cb-9364-448d-b41c-8a48fb1826c5 at 0x7f12442ef290>
timeout=30.0, duration=0.00 at 0x7f11e73d5b50>#012Traceback (most recent call
last):#012 File "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315,
in _execute_task#012 task()#012 File
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012
self._callable()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 315, in
__call__#012 self._execute()#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 357, in
_execute#012 self._vm.updateDriveVolume(drive)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4189, in
updateDriveVolume#012 vmDrive.volumeID)#012 File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6101, in
_getVolumeSize#012 (domainID, volumeID))#012StorageUnavailableError: Unable to get
volume size for doma
in 88f5972f-58bd-469f-bc77-5bf3b1802291 volume cdf313d7-bed3-4fae-a803-1297cdf8c82f
Sep 14 08:40:42 host1 systemd: Created slice User Slice of root.
Sep 14 08:40:42 host1 systemd: Started Session c69119 of user root.
Sep 14 08:40:42 host1 systemd: Removed slice User Slice of root.
rhev-data-center-mnt-glusterSD-host1storage::_data.log
----------------------------------------------------------------
2020-09-14 08:40:39.648159] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-data-client-2: disconnected from data-client-2. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:40:39.648183] W [MSGID: 108001] [afr-common.c:5613:afr_notify]
0-data-replicate-0: Client-quorum is not met
[2020-09-14 08:40:50.243632] E [MSGID: 114058]
[client-handshake.c:1449:client_query_portmap_cbk] 0-data-client-2: failed to get the port
number for remote subvolume. Please run 'gluster volume status' on server to see
if brick process is running.
[2020-09-14 08:40:50.243686] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-data-client-2: disconnected from data-client-2. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:41:05.147093] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-data-client-1: disconnected from data-client-1. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:41:05.147139] E [MSGID: 108006]
[afr-common.c:5323:__afr_handle_child_down_event] 0-data-replicate-0: All subvolumes are
down. Going offline until at least one of them comes back up.
[2020-09-14 08:41:05.156727] I [MSGID: 108006] [afr-common.c:5669:afr_local_init]
0-data-replicate-0: no subvolumes up
The message "I [MSGID: 108006] [afr-common.c:5669:afr_local_init] 0-data-replicate-0:
no subvolumes up" repeated 195 times between [2020-09-14 08:41:05.156727] and
[2020-09-14 08:41:15.222500]
[2020-09-14 08:41:15.288560] E [MSGID: 114058]
[client-handshake.c:1449:client_query_portmap_cbk] 0-data-client-1: failed to get the port
number for remote subvolume. Please run 'gluster volume status' on server to see
if brick process is running.
[2020-09-14 08:41:15.288608] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-data-client-1: disconnected from data-client-1. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:41:15.722793] I [MSGID: 108006] [afr-common.c:5669:afr_local_init]
0-data-replicate-0: no subvolumes up
The message "I [MSGID: 108006] [afr-common.c:5669:afr_local_init] 0-data-replicate-0:
no subvolumes up" repeated 170 times between [2020-09-14 08:41:15.722793] and
[2020-09-14 08:41:24.230828]
[2020-09-14 08:41:24.731222] I [MSGID: 108006] [afr-common.c:5669:afr_local_init]
0-data-replicate-0: no subvolumes up
[2020-09-14 08:41:41.352791] I [socket.c:811:__socket_shutdown] 0-data-client-2:
intentional socket shutdown(6)
[2020-09-14 08:41:44.363054] I [socket.c:811:__socket_shutdown] 0-data-client-0:
intentional socket shutdown(6)
[2020-09-14 08:42:42.512364] I [socket.c:811:__socket_shutdown] 0-data-client-1:
intentional socket shutdown(6)
The message "I [MSGID: 108006] [afr-common.c:5669:afr_local_init] 0-data-replicate-0:
no subvolumes up" repeated 708 times between [2020-09-14 08:41:24.731222] and
[2020-09-14 08:43:23.874840]
[2020-09-14 08:43:26.411190] I [MSGID: 108006] [afr-common.c:5669:afr_local_init]
0-data-replicate-0: no subvolumes up
...
...
...
[2020-09-14 12:19:59.517980] I [MSGID: 108006] [afr-common.c:5669:afr_local_init]
0-data-replicate-0: no subvolumes up
[2020-09-14 12:20:29.093669] W [socket.c:721:__socket_rwv] 0-glusterfs: readv on
192.168.0.101:24007 failed (No data available)
[2020-09-14 12:20:29.093695] I [glusterfsd-mgmt.c:2443:mgmt_rpc_notify] 0-glusterfsd-mgmt:
disconnected from remote-host: host1storage.mydomain.com
[2020-09-14 12:20:29.093707] I [glusterfsd-mgmt.c:2483:mgmt_rpc_notify] 0-glusterfsd-mgmt:
connecting to next volfile server host2.mydomain.com
[2020-09-14 12:20:39.630645] I [socket.c:811:__socket_shutdown] 0-data-client-2:
intentional socket shutdown(7)
[2020-09-14 12:20:40.635471] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-data-client-1:
changing port to 49152 (from 0)
[2020-09-14 12:20:40.639281] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-data-client-1: Connected to
data-client-1, attached to remote volume '/gluster_bricks/data/data'.
[2020-09-14 12:20:40.639299] I [MSGID: 114042]
[client-handshake.c:930:client_post_handshake] 0-data-client-1: 5 fds open - Delaying
child_up until they are re-opened
[2020-09-14 12:20:40.639945] I [MSGID: 114041]
[client-handshake.c:318:client_child_up_reopen_done] 0-data-client-1: last fd
open'd/lock-self-heal'd - notifying CHILD-UP
[2020-09-14 12:20:40.639967] I [MSGID: 108005]
[afr-common.c:5245:__afr_handle_child_up_event] 0-data-replicate-0: Subvolume
'data-client-1' came back up; going online.
[2020-09-14 12:20:42.640257] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-data-client-0:
changing port to 49152 (from 0)
[2020-09-14 12:20:42.640288] I [socket.c:811:__socket_shutdown] 0-data-client-0:
intentional socket shutdown(10)
[2020-09-14 12:20:42.643664] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-data-client-0: Connected to
data-client-0, attached to remote volume '/gluster_bricks/data/data'.
[2020-09-14 12:20:42.643683] I [MSGID: 114042]
[client-handshake.c:930:client_post_handshake] 0-data-client-0: 5 fds open - Delaying
child_up until they are re-opened
[2020-09-14 12:20:42.644327] I [MSGID: 114041]
[client-handshake.c:318:client_child_up_reopen_done] 0-data-client-0: last fd
open'd/lock-self-heal'd - notifying CHILD-UP
[2020-09-14 12:20:42.644347] I [MSGID: 108002] [afr-common.c:5607:afr_notify]
0-data-replicate-0: Client-quorum is met
[2020-09-14 12:20:42.843176] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-data-client-2:
changing port to 49152 (from 0)
[2020-09-14 12:20:42.846562] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-data-client-2: Connected to
data-client-2, attached to remote volume '/gluster_bricks/data/data'.
[2020-09-14 12:20:42.846598] I [MSGID: 114042]
[client-handshake.c:930:client_post_handshake] 0-data-client-2: 5 fds open - Delaying
child_up until they are re-opened
[2020-09-14 12:20:42.847429] I [MSGID: 114041]
[client-handshake.c:318:client_child_up_reopen_done] 0-data-client-2: last fd
open'd/lock-self-heal'd - notifying CHILD-UP
[2020-09-14 12:20:49.640811] I [MSGID: 133022] [shard.c:3674:shard_delete_shards]
0-data-shard: Deleted shards of gfid=18250f19-3820-4a98-9c49-37ba23c08dfd from backend
[2020-09-14 12:20:50.244771] E [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_opendir_cbk] 0-data-client-1: remote operation
failed. Path: /88f5972f-58bd-469f-bc77-5bf3b1802291/images
(5fdbb512-e924-4945-a633-10820133e5ff) [Permission denied]
[2020-09-14 12:20:50.244829] E [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_opendir_cbk] 0-data-client-2: remote operation
failed. Path: /88f5972f-58bd-469f-bc77-5bf3b1802291/images
(5fdbb512-e924-4945-a633-10820133e5ff) [Permission denied]
[2020-09-14 12:20:50.244880] W [MSGID: 114061]
[client-common.c:3325:client_pre_readdirp_v2] 0-data-client-1:
(5fdbb512-e924-4945-a633-10820133e5ff) remote_fd is -1. EBADFD [File descriptor in bad
state]
[2020-09-14 12:21:20.364907] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data-client-1: remote operation failed.
Path: /88f5972f-58bd-469f-bc77-5bf3b1802291/images/4e79467c-4707-4e7d-8a2a-909b208d4b97
(b0f8bb67-09e5-431b-acea-c03f0280fb34) [Permission denied]
[2020-09-14 12:21:20.365181] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data-client-2: remote operation failed.
Path: /88f5972f-58bd-469f-bc77-5bf3b1802291/images/4e79467c-4707-4e7d-8a2a-909b208d4b97
(b0f8bb67-09e5-431b-acea-c03f0280fb34) [Permission denied]
[2020-09-14 12:21:20.368381] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data-client-2: remote operation failed.
Path: (null) (00000000-0000-0000-0000-000000000000) [Permission denied]
[2020-09-14 12:21:20.368401] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data-client-1: remote operation failed.
Path: (null) (00000000-0000-0000-0000-000000000000) [Permission denied]
[2020-09-14 12:21:20.368462] W [MSGID: 108027]
[afr-common.c:2274:afr_attempt_readsubvol_set] 0-data-replicate-0: no read subvols for
/88f5972f-58bd-469f-bc77-5bf3b1802291/images/4e79467c-4707-4e7d-8a2a-909b208d4b97
[2020-09-14 12:21:20.369001] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data-client-1: remote operation failed.
Path: /88f5972f-58bd-469f-bc77-5bf3b1802291/images/4e79467c-4707-4e7d-8a2a-909b208d4b97
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2020-09-14 12:21:20.369002] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data-client-2: remote operation failed.
Path: /88f5972f-58bd-469f-bc77-5bf3b1802291/images/4e79467c-4707-4e7d-8a2a-909b208d4b97
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2020-09-14 12:21:20.370979] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data-client-2: remote operation failed.
Path: (null) (00000000-0000-0000-0000-000000000000) [Permission denied]
[2020-09-14 12:21:20.371009] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data-client-1: remote operation failed.
Path: (null) (00000000-0000-0000-0000-000000000000) [Permission denied]
[2020-09-14 12:21:20.371075] W [MSGID: 108027]
[afr-common.c:2274:afr_attempt_readsubvol_set] 0-data-replicate-0: no read subvols for
/88f5972f-58bd-469f-bc77-5bf3b1802291/images/4e79467c-4707-4e7d-8a2a-909b208d4b97
The message "I [MSGID: 108006] [afr-common.c:5669:afr_local_init] 0-data-replicate-0:
no subvolumes up" repeated 102 times between [2020-09-14 12:19:59.517980] and
[2020-09-14 12:20:39.539124]
[2020-09-14 12:22:11.510751] I [MSGID: 133022] [shard.c:3674:shard_delete_shards]
0-data-shard: Deleted shards of gfid=40104663-41d6-4210-b9aa-065e0ba48c1f from backend
[2020-09-14 12:22:11.853348] E [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_opendir_cbk] 0-data-client-2: remote operation
failed. Path: /88f5972f-58bd-469f-bc77-5bf3b1802291/images
(5fdbb512-e924-4945-a633-10820133e5ff) [Permission denied]
glusterd.log
----------------
[2020-09-14 08:40:39.750074] E [socket.c:2282:__socket_read_frag] 0-rpc: wrong MSG-TYPE
(9) received from 10.0.0.116:30052
[2020-09-14 08:42:36.920865] C [rpcsvc.c:1029:rpcsvc_notify] 0-rpcsvc: got MAP_XID event,
which should have not come
[2020-09-14 08:42:56.426832] E [socket.c:2282:__socket_read_frag] 0-rpc: wrong MSG-TYPE
(3866624) received from 10.0.0.116:53501
[2020-09-14 08:48:50.776839] E [socket.c:2282:__socket_read_frag] 0-rpc: wrong MSG-TYPE
(-66911352) received from 10.0.0.116:46153
[2020-09-14 09:00:51.692183] E [socket.c:2282:__socket_read_frag] 0-rpc: wrong MSG-TYPE
(7602176) received from 10.0.0.116:64053
[2020-09-14 12:17:50.501625] W [MSGID: 101095] [xlator.c:210:xlator_volopt_dynload]
0-xlator: /usr/lib64/glusterfs/6.6/xlator/encryption/crypt.so: cannot open shared object
file: No such file or directory
[2020-09-14 12:17:50.508662] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload]
0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so:
undefined symbol: xlator_api
[2020-09-14 12:17:50.510694] W [MSGID: 101095] [xlator.c:210:xlator_volopt_dynload]
0-xlator: /usr/lib64/glusterfs/6.6/xlator/nfs/server.so: cannot open shared object file:
No such file or directory
[2020-09-14 12:17:50.518341] W [MSGID: 101095] [xlator.c:210:xlator_volopt_dynload]
0-xlator: /usr/lib64/glusterfs/6.6/xlator/storage/bd.so: cannot open shared object file:
No such file or directory
[2020-09-14 12:18:08.102606] I [MSGID: 106499]
[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status
volume req for volume data
[2020-09-14 12:18:08.118107] I [MSGID: 106499]
[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status
volume req for volume engine
[2020-09-14 12:18:37.847581] E [MSGID: 106537]
[glusterd-volume-ops.c:1763:glusterd_op_stage_start_volume] 0-management: Volume engine
already started
[2020-09-14 12:18:37.847606] W [MSGID: 106121]
[glusterd-mgmt.c:178:gd_mgmt_v3_pre_validate_fn] 0-management: Volume start prevalidation
failed.
[2020-09-14 12:18:37.847618] E [MSGID: 106121]
[glusterd-mgmt.c:1079:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed
for operation Start on local node
[2020-09-14 12:18:37.847625] E [MSGID: 106121]
[glusterd-mgmt.c:2457:glusterd_mgmt_v3_initiate_all_phases] 0-management: Pre Validation
Failed
The message "W [MSGID: 101095] [xlator.c:210:xlator_volopt_dynload] 0-xlator:
/usr/lib64/glusterfs/6.6/xlator/encryption/crypt.so: cannot open shared object file: No
such file or directory" repeated 2 times between [2020-09-14 12:17:50.501625] and
[2020-09-14 12:17:50.501663]
The message "E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator:
dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined
symbol: xlator_api" repeated 7 times between [2020-09-14 12:17:50.508662] and
[2020-09-14 12:17:50.508709]
The message "W [MSGID: 101095] [xlator.c:210:xlator_volopt_dynload] 0-xlator:
/usr/lib64/glusterfs/6.6/xlator/nfs/server.so: cannot open shared object file: No such
file or directory" repeated 30 times between [2020-09-14 12:17:50.510694] and
[2020-09-14 12:17:50.510909]
[2020-09-14 12:19:41.946661] I [MSGID: 106487]
[glusterd-handler.c:1516:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list
req
[2020-09-14 12:19:49.871903] I [MSGID: 106533]
[glusterd-volume-ops.c:982:__glusterd_handle_cli_heal_volume] 0-management: Received heal
vol req for volume engine
[2020-09-14 12:20:29.091554] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7e65) [0x7fabf0a81e65]
-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x5643e96971f5]
-->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x5643e969705b] ) 0-: received signum
(15), shutting down
[2020-09-14 12:20:29.123582] I [MSGID: 100030] [glusterfsd.c:2847:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 6.6 (args:
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2020-09-14 12:20:29.124349] I [glusterfsd.c:2556:daemonize] 0-glusterfs: Pid of current
running process is 33290
[2020-09-14 12:20:29.129801] I [MSGID: 106478] [glusterd.c:1422:init] 0-management:
Maximum allowed open file descriptors set to 65536
[2020-09-14 12:20:29.129827] I [MSGID: 106479] [glusterd.c:1478:init] 0-management: Using
/var/lib/glusterd as working directory
[2020-09-14 12:20:29.129834] I [MSGID: 106479] [glusterd.c:1484:init] 0-management: Using
/var/run/gluster as pid file working directory
[2020-09-14 12:20:29.133500] I [socket.c:961:__socket_server_bind] 0-socket.management:
process started listening on port (24007)
[2020-09-14 12:20:29.136005] W [MSGID: 103071] [rdma.c:4472:__gf_rdma_ctx_create]
0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2020-09-14 12:20:29.136024] W [MSGID: 103055] [rdma.c:4782:init] 0-rdma.management:
Failed to initialize IB Device
[2020-09-14 12:20:29.136032] W [rpc-transport.c:363:rpc_transport_load] 0-rpc-transport:
'rdma' initialization failed
[2020-09-14 12:20:29.136102] W [rpcsvc.c:1985:rpcsvc_create_listener] 0-rpc-service:
cannot create listener, initing the transport failed
[2020-09-14 12:20:29.136112] E [MSGID: 106244] [glusterd.c:1785:init] 0-management:
creation of 1 listeners failed, continuing with succeeded transport
[2020-09-14 12:20:29.137180] I [socket.c:904:__socket_server_bind] 0-socket.management:
closing (AF_UNIX) reuse check socket 12
[2020-09-14 12:20:29.137478] I [MSGID: 106059] [glusterd.c:1865:init] 0-management:
max-port override: 60999
[2020-09-14 12:20:30.969620] I [MSGID: 106513]
[glusterd-store.c:2394:glusterd_restore_op_version] 0-glusterd: retrieved op-version:
60000
[2020-09-14 12:20:31.061282] I [MSGID: 106544] [glusterd.c:152:glusterd_uuid_init]
0-management: retrieved UUID: 6fe3d6e3-ab45-4004-af59-93c2fc3afc93
[2020-09-14 12:20:31.086867] I [MSGID: 106498]
[glusterd-handler.c:3687:glusterd_friend_add_from_peerinfo] 0-management: connect returned
0
[2020-09-14 12:20:31.087760] I [MSGID: 106498]
[glusterd-handler.c:3687:glusterd_friend_add_from_peerinfo] 0-management: connect returned
0
[2020-09-14 12:20:31.087814] W [MSGID: 106061]
[glusterd-handler.c:3490:glusterd_transport_inet_options_build] 0-glusterd: Failed to get
tcp-user-timeout
[2020-09-14 12:20:31.087844] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600
[2020-09-14 12:20:31.091059] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600
[2020-09-14 12:20:31.091052] W [MSGID: 106061]
[glusterd-handler.c:3490:glusterd_transport_inet_options_build] 0-glusterd: Failed to get
tcp-user-timeout
[2020-09-14 12:20:31.095394] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-09-14 12:20:40.073223] I [MSGID: 106493]
[glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid:
1c2f1776-4830-4b6f-8e9e-c29d0133a0e9, host: myhost3.mydomain.com, port: 0
[2020-09-14 12:20:40.076519] C [MSGID: 106003]
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: Server
quorum regained for volume data. Starting local bricks.
[2020-09-14 12:20:40.077376] I [glusterd-utils.c:6312:glusterd_brick_start] 0-management:
starting a fresh brick process for brick /gluster_bricks/data/data
[2020-09-14 12:20:40.080459] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600
[2020-09-14 12:20:40.093965] C [MSGID: 106003]
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: Server
quorum regained for volume engine. Starting local bricks.
[2020-09-14 12:20:40.094129] I [glusterd-utils.c:6312:glusterd_brick_start] 0-management:
starting a fresh brick process for brick /gluster_bricks/engine/engine
[2020-09-14 12:20:40.095873] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600
[2020-09-14 12:20:40.112487] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600
[2020-09-14 12:20:40.125181] I [MSGID: 106493]
[glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid:
b5a0980f-015b-4720-9b5b-7792ac022b54, host: myhost2.mydomain.com, port: 0
[2020-09-14 12:20:40.163361] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-nfs: setting
frame-timeout to 600
[2020-09-14 12:20:40.163464] I [MSGID: 106131]
[glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped
[2020-09-14 12:20:40.163482] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: nfs service is stopped
[2020-09-14 12:20:40.163493] I [MSGID: 106599]
[glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not
installed
[2020-09-14 12:20:40.163520] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-glustershd:
setting frame-timeout to 600
[2020-09-14 12:20:40.165762] I [MSGID: 106568]
[glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon
running in pid: 28027
[2020-09-14 12:20:40.170812] I [MSGID: 106492]
[glusterd-handler.c:2796:__glusterd_handle_friend_update] 0-glusterd: Received friend
update from uuid: 1c2f1776-4830-4b6f-8e9e-c29d0133a0e9
[2020-09-14 12:20:40.170841] I [MSGID: 106502]
[glusterd-handler.c:2837:__glusterd_handle_friend_update] 0-management: Received my uuid
as Friend
[2020-09-14 12:20:40.175230] I [MSGID: 106493]
[glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from
uuid: b5a0980f-015b-4720-9b5b-7792ac022b54
[2020-09-14 12:20:40.175329] I [MSGID: 106492]
[glusterd-handler.c:2796:__glusterd_handle_friend_update] 0-glusterd: Received friend
update from uuid: b5a0980f-015b-4720-9b5b-7792ac022b54
[2020-09-14 12:20:40.175353] I [MSGID: 106502]
[glusterd-handler.c:2837:__glusterd_handle_friend_update] 0-management: Received my uuid
as Friend
[2020-09-14 12:20:40.183400] I [MSGID: 106163]
[glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the
op-version 60000
[2020-09-14 12:20:40.188715] I [MSGID: 106490]
[glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe
from uuid: b5a0980f-015b-4720-9b5b-7792ac022b54
[2020-09-14 12:20:40.193866] I [MSGID: 106493]
[glusterd-handler.c:3883:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
myhost2.mydomain.com (0), ret: 0, op_ret: 0
[2020-09-14 12:20:40.198118] I [MSGID: 106492]
[glusterd-handler.c:2796:__glusterd_handle_friend_update] 0-glusterd: Received friend
update from uuid: b5a0980f-015b-4720-9b5b-7792ac022b54
[2020-09-14 12:20:40.198144] I [MSGID: 106502]
[glusterd-handler.c:2837:__glusterd_handle_friend_update] 0-management: Received my uuid
as Friend
[2020-09-14 12:20:40.200072] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind]
0-pmap: adding brick /gluster_bricks/data/data on port 49152
[2020-09-14 12:20:40.200198] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind]
0-pmap: adding brick /gluster_bricks/engine/engine on port 49153
[2020-09-14 12:20:40.200245] I [MSGID: 106493]
[glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from
uuid: b5a0980f-015b-4720-9b5b-7792ac022b54
[2020-09-14 12:20:40.875962] I [MSGID: 106493]
[glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from
uuid: 1c2f1776-4830-4b6f-8e9e-c29d0133a0e9
[2020-09-14 12:20:40.888913] I [MSGID: 106163]
[glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the
op-version 60000
[2020-09-14 12:20:40.892888] I [MSGID: 106490]
[glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe
from uuid: 1c2f1776-4830-4b6f-8e9e-c29d0133a0e9
[2020-09-14 12:20:40.896510] I [MSGID: 106493]
[glusterd-handler.c:3883:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
myhost3.mydomain.com (0), ret: 0, op_ret: 0
[2020-09-14 12:20:40.928796] I [MSGID: 106492]
[glusterd-handler.c:2796:__glusterd_handle_friend_update] 0-glusterd: Received friend
update from uuid: 1c2f1776-4830-4b6f-8e9e-c29d0133a0e9
[2020-09-14 12:20:40.928820] I [MSGID: 106502]
[glusterd-handler.c:2837:__glusterd_handle_friend_update] 0-management: Received my uuid
as Friend
[2020-09-14 12:20:40.997183] I [MSGID: 106493]
[glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from
uuid: 1c2f1776-4830-4b6f-8e9e-c29d0133a0e9
[2020-09-14 12:20:41.165913] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: glustershd service is stopped
[2020-09-14 12:20:41.165965] I [MSGID: 106567]
[glusterd-svc-mgmt.c:220:glusterd_svc_start] 0-management: Starting glustershd service
[2020-09-14 12:20:42.168619] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-quotad:
setting frame-timeout to 600
[2020-09-14 12:20:42.168901] I [MSGID: 106131]
[glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already stopped
[2020-09-14 12:20:42.168917] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: quotad service is stopped
[2020-09-14 12:20:42.168942] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-bitd: setting
frame-timeout to 600
[2020-09-14 12:20:42.169151] I [MSGID: 106131]
[glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
[2020-09-14 12:20:42.169164] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: bitd service is stopped
[2020-09-14 12:20:42.169193] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-scrub: setting
frame-timeout to 600
[2020-09-14 12:20:42.169377] I [MSGID: 106131]
[glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already stopped
[2020-09-14 12:20:42.169388] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: scrub service is stopped
[2020-09-14 12:20:42.169420] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-snapd: setting
frame-timeout to 600
[2020-09-14 12:20:42.169506] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-snapd: setting
frame-timeout to 600
[2020-09-14 12:20:42.169586] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-snapd: setting
frame-timeout to 600
[2020-09-14 12:20:42.169670] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-gfproxyd:
setting frame-timeout to 600
[2020-09-14 12:20:42.169797] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-gfproxyd:
setting frame-timeout to 600
[2020-09-14 12:20:42.169928] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-gfproxyd:
setting frame-timeout to 600
[2020-09-14 12:20:42.170136] I [glusterd-utils.c:6225:glusterd_brick_start] 0-management:
discovered already-running brick /gluster_bricks/data/data
[2020-09-14 12:20:42.170151] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind]
0-pmap: adding brick /gluster_bricks/data/data on port 49152
[2020-09-14 12:20:42.182826] I [glusterd-utils.c:6225:glusterd_brick_start] 0-management:
discovered already-running brick /gluster_bricks/engine/engine
[2020-09-14 12:20:42.182844] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind]
0-pmap: adding brick /gluster_bricks/engine/engine on port 49153
[2020-09-14 12:20:42.207292] I [MSGID: 106131]
[glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped
[2020-09-14 12:20:42.207314] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: nfs service is stopped
[2020-09-14 12:20:42.207324] I [MSGID: 106599]
[glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not
installed
[2020-09-14 12:20:42.209056] I [MSGID: 106568]
[glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon
running in pid: 33647
[2020-09-14 12:20:42.210310] I [MSGID: 106006]
[glusterd-svc-mgmt.c:356:glusterd_svc_common_rpc_notify] 0-management: glustershd has
disconnected from glusterd.
[2020-09-14 12:20:43.209219] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: glustershd service is stopped
[2020-09-14 12:20:43.209621] I [MSGID: 106567]
[glusterd-svc-mgmt.c:220:glusterd_svc_start] 0-management: Starting glustershd service
[2020-09-14 12:20:44.212440] I [MSGID: 106131]
[glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already stopped
[2020-09-14 12:20:44.212490] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: quotad service is stopped
[2020-09-14 12:20:44.212691] I [MSGID: 106131]
[glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
[2020-09-14 12:20:44.212712] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: bitd service is stopped
[2020-09-14 12:20:44.212897] I [MSGID: 106131]
[glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already stopped
[2020-09-14 12:20:44.212911] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop]
0-management: scrub service is stopped
[2020-09-14 12:20:44.213829] I [MSGID: 106499]
[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management: Received status
volume req for volume engine
[2020-09-14 12:20:46.779903] I [MSGID: 106488]
[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req
The message "I [MSGID: 106488]
[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req" repeated 5 times between [2020-09-14 12:20:46.779903] and [2020-09-14
12:20:48.081053]
[2020-09-14 12:20:54.612834] I [MSGID: 106533]
[glusterd-volume-ops.c:982:__glusterd_handle_cli_heal_volume] 0-management: Received heal
vol req for volume engine
[2020-09-14 12:21:01.461754] I [MSGID: 106533]
[glusterd-volume-ops.c:982:__glusterd_handle_cli_heal_volume] 0-management: Received heal
vol req for volume data
[2020-09-14 12:21:37.658372] I [MSGID: 106488]
[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req
The message "I [MSGID: 106488]
[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req" repeated 8 times between [2020-09-14 12:21:37.658372] and [2020-09-14
12:22:28.492880]
[2020-09-14 12:22:49.552197] I [MSGID: 106488]
[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req
[2020-09-14 12:23:39.208008] I [MSGID: 106533]
[glusterd-volume-ops.c:982:__glusterd_handle_cli_heal_volume] 0-management: Received heal
vol req for volume data
[2020-09-14 13:23:42.678379] I [MSGID: 106488]
[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req
[2020-09-14 13:23:43.038221] I [MSGID: 106488]
[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req
glustersh.log
-----------------
[2020-09-14 08:40:09.990714] I [socket.c:811:__socket_shutdown] 0-data-client-0:
intentional socket shutdown(5)
[2020-09-14 08:40:15.005498] I [socket.c:811:__socket_shutdown] 0-engine-client-0:
intentional socket shutdown(5)
[2020-09-14 08:40:27.068884] W [socket.c:721:__socket_rwv] 0-engine-client-2: readv on
192.168.0.103:49153 failed (No data available)
[2020-09-14 08:40:27.068922] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-engine-client-2: disconnected from engine-client-2. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:40:27.068954] W [MSGID: 108001] [afr-common.c:5613:afr_notify]
0-engine-replicate-0: Client-quorum is not met
[2020-09-14 08:40:37.082276] E [MSGID: 114058]
[client-handshake.c:1449:client_query_portmap_cbk] 0-engine-client-2: failed to get the
port number for remote subvolume. Please run 'gluster volume status' on server to
see if brick process is running.
[2020-09-14 08:40:39.648157] W [socket.c:721:__socket_rwv] 0-data-client-2: readv on
192.168.0.103:49152 failed (No data available)
[2020-09-14 08:40:39.648190] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-data-client-2: disconnected from data-client-2. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:40:39.648217] W [MSGID: 108001] [afr-common.c:5613:afr_notify]
0-data-replicate-0: Client-quorum is not met
[2020-09-14 08:40:50.133996] E [MSGID: 114058]
[client-handshake.c:1449:client_query_portmap_cbk] 0-data-client-2: failed to get the port
number for remote subvolume. Please run 'gluster volume status' on server to see
if brick process is running.
[2020-09-14 08:40:50.134055] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-data-client-2: disconnected from data-client-2. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:41:05.100136] W [socket.c:721:__socket_rwv] 0-engine-client-1: readv on
192.168.0.102:49153 failed (No data available)
[2020-09-14 08:41:05.100176] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-engine-client-1: disconnected from engine-client-1. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:41:05.100193] E [MSGID: 108006]
[afr-common.c:5323:__afr_handle_child_down_event] 0-engine-replicate-0: All subvolumes are
down. Going offline until at least one of them comes back up.
[2020-09-14 08:41:05.147135] W [socket.c:721:__socket_rwv] 0-data-client-1: readv on
192.168.0.102:49152 failed (No data available)
[2020-09-14 08:41:05.147165] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-data-client-1: disconnected from data-client-1. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:41:05.147179] E [MSGID: 108006]
[afr-common.c:5323:__afr_handle_child_down_event] 0-data-replicate-0: All subvolumes are
down. Going offline until at least one of them comes back up.
[2020-09-14 08:41:15.279570] E [MSGID: 114058]
[client-handshake.c:1449:client_query_portmap_cbk] 0-engine-client-1: failed to get the
port number for remote subvolume. Please run 'gluster volume status' on server to
see if brick process is running.
[2020-09-14 08:41:15.279610] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-engine-client-1: disconnected from engine-client-1. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:41:15.281851] E [MSGID: 114058]
[client-handshake.c:1449:client_query_portmap_cbk] 0-data-client-1: failed to get the port
number for remote subvolume. Please run 'gluster volume status' on server to see
if brick process is running.
[2020-09-14 08:41:15.281891] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-data-client-1: disconnected from data-client-1. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:40:37.082339] I [MSGID: 114018] [client.c:2331:client_rpc_notify]
0-engine-client-2: disconnected from engine-client-2. Client process will keep trying to
connect to glusterd until brick's port is available
[2020-09-14 08:42:16.707442] I [socket.c:811:__socket_shutdown] 0-data-client-0:
intentional socket shutdown(5)
[2020-09-14 08:42:21.745058] I [socket.c:811:__socket_shutdown] 0-engine-client-0:
intentional socket shutdown(5)
[2020-09-14 08:42:40.886787] I [socket.c:811:__socket_shutdown] 0-engine-client-2:
intentional socket shutdown(5)
...
...
[2020-09-14 12:20:29.094083] W [socket.c:721:__socket_rwv] 0-glusterfs: readv on
127.0.0.1:24007 failed (No data available)
[2020-09-14 12:20:29.094148] I [glusterfsd-mgmt.c:2443:mgmt_rpc_notify] 0-glusterfsd-mgmt:
disconnected from remote-host: localhost
[2020-09-14 12:20:34.927760] I [socket.c:811:__socket_shutdown] 0-engine-client-2:
intentional socket shutdown(5)
[2020-09-14 12:20:39.959928] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-14 12:20:40.166029] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7e65) [0x7f5f21f62e65]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x557934e2e1f5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x557934e2e05b] ) 0-: received signum
(15), shutting down
[2020-09-14 12:20:41.186725] I [MSGID: 100030] [glusterfsd.c:2847:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 6.6 (args:
/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
/var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S
/var/run/gluster/9713eea6455553ca.socket --xlator-option
*replicate*.node-uuid=6fe3d6e3-ab45-4004-af59-93c2fc3afc93 --process-name glustershd
--client-pid=-6)
[2020-09-14 12:20:41.187235] I [glusterfsd.c:2556:daemonize] 0-glusterfs: Pid of current
running process is 33647
[2020-09-14 12:20:41.191605] I [socket.c:904:__socket_server_bind] 0-socket.glusterfsd:
closing (AF_UNIX) reuse check socket 9
[2020-09-14 12:20:41.196867] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-09-14 12:20:41.196870] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-09-14 12:20:42.209282] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7e65) [0x7f995868ee65]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55963354e1f5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55963354e05b] ) 0-: received signum
(15), shutting down
[2020-09-14 12:20:43.236232] I [MSGID: 100030] [glusterfsd.c:2847:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 6.6 (args:
/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
/var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S
/var/run/gluster/9713eea6455553ca.socket --xlator-option
*replicate*.node-uuid=6fe3d6e3-ab45-4004-af59-93c2fc3afc93 --process-name glustershd
--client-pid=-6)
[2020-09-14 12:20:43.236667] I [glusterfsd.c:2556:daemonize] 0-glusterfs: Pid of current
running process is 33705
[2020-09-14 12:20:43.240743] I [socket.c:904:__socket_server_bind] 0-socket.glusterfsd:
closing (AF_UNIX) reuse check socket 9
[2020-09-14 12:20:43.246262] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-09-14 12:20:43.246283] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-09-14 12:20:44.217875] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2020-09-14 12:20:44.217962] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2020-09-14 12:20:44.221374] I [MSGID: 114020] [client.c:2401:notify] 0-data-client-0:
parent translators are ready, attempting connect on transport
[2020-09-14 12:20:44.224361] I [MSGID: 114020] [client.c:2401:notify] 0-data-client-1:
parent translators are ready, attempting connect on transport
[2020-09-14 12:20:44.224637] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-data-client-0:
changing port to 49152 (from 0)
[2020-09-14 12:20:44.224673] I [socket.c:811:__socket_shutdown] 0-data-client-0:
intentional socket shutdown(12)
[2020-09-14 12:20:44.227134] I [MSGID: 114020] [client.c:2401:notify] 0-data-client-2:
parent translators are ready, attempting connect on transport
[2020-09-14 12:20:44.227728] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-data-client-1:
changing port to 49152 (from 0)
[2020-09-14 12:20:44.227758] I [socket.c:811:__socket_shutdown] 0-data-client-1:
intentional socket shutdown(13)
[2020-09-14 12:20:44.230217] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-data-client-0: Connected to
data-client-0, attached to remote volume '/gluster_bricks/data/data'.
[2020-09-14 12:20:44.230242] I [MSGID: 108005]
[afr-common.c:5245:__afr_handle_child_up_event] 0-data-replicate-0: Subvolume
'data-client-0' came back up; going online.
[2020-09-14 12:20:44.231537] I [MSGID: 114020] [client.c:2401:notify] 0-engine-client-0:
parent translators are ready, attempting connect on transport
[2020-09-14 12:20:44.232072] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-data-client-2:
changing port to 49152 (from 0)
[2020-09-14 12:20:44.232112] I [socket.c:811:__socket_shutdown] 0-data-client-2:
intentional socket shutdown(15)
[2020-09-14 12:20:44.234981] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-data-client-1: Connected to
data-client-1, attached to remote volume '/gluster_bricks/data/data'.
[2020-09-14 12:20:44.235005] I [MSGID: 108002] [afr-common.c:5607:afr_notify]
0-data-replicate-0: Client-quorum is met
[2020-09-14 12:20:44.235988] I [MSGID: 114020] [client.c:2401:notify] 0-engine-client-1:
parent translators are ready, attempting connect on transport
[2020-09-14 12:20:44.236176] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-engine-client-0:
changing port to 49153 (from 0)
[2020-09-14 12:20:44.236217] I [socket.c:811:__socket_shutdown] 0-engine-client-0:
intentional socket shutdown(12)
[2020-09-14 12:20:44.239425] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-data-client-2: Connected to
data-client-2, attached to remote volume '/gluster_bricks/data/data'.
[2020-09-14 12:20:44.240456] I [MSGID: 114020] [client.c:2401:notify] 0-engine-client-2:
parent translators are ready, attempting connect on transport
[2020-09-14 12:20:44.240960] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-engine-client-1:
changing port to 49153 (from 0)
[2020-09-14 12:20:44.240987] I [socket.c:811:__socket_shutdown] 0-engine-client-1:
intentional socket shutdown(17)
[2020-09-14 12:20:44.243360] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-engine-client-0: Connected to
engine-client-0, attached to remote volume '/gluster_bricks/engine/engine'.
[2020-09-14 12:20:44.243385] I [MSGID: 108005]
[afr-common.c:5245:__afr_handle_child_up_event] 0-engine-replicate-0: Subvolume
'engine-client-0' came back up; going online.
[2020-09-14 12:20:44.245314] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-engine-client-2:
changing port to 49153 (from 0)
[2020-09-14 12:20:44.245344] I [socket.c:811:__socket_shutdown] 0-engine-client-2:
intentional socket shutdown(15)
[2020-09-14 12:20:44.248543] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-engine-client-1: Connected to
engine-client-1, attached to remote volume '/gluster_bricks/engine/engine'.
[2020-09-14 12:20:44.248564] I [MSGID: 108002] [afr-common.c:5607:afr_notify]
0-engine-replicate-0: Client-quorum is met
[2020-09-14 12:20:44.253661] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-engine-client-2: Connected to
engine-client-2, attached to remote volume '/gluster_bricks/engine/engine'.
[2020-09-14 12:20:44.486491] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on b72e302c-478d-4f90-bdd7-5d542928bcc1. sources=[1] 2 sinks=0
[2020-09-14 12:20:47.661708] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on b0ba8fbc-750e-4a2d-a218-d49500d95d26. sources=[1] 2 sinks=0
[2020-09-14 12:20:53.451995] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on bbfa9185-5383-44f8-a32d-9b1aaa6e43a7. sources=[1] 2 sinks=0
[2020-09-14 12:20:54.148249] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 1f2ee964-e4bc-432e-be43-5e95783cace1. sources=[1] 2 sinks=0
[2020-09-14 12:20:54.148386] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 937a7d82-5b99-487b-9614-3aad7ef2dc97. sources=[1] 2 sinks=0
[2020-09-14 12:20:54.148420] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on b9a47d47-111b-403b-9d05-b741c2878ff6. sources=[1] 2 sinks=0
[2020-09-14 12:20:54.443785] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on abe59023-a60b-495e-a1a3-2698dbf747b5. sources=[1] 2 sinks=0
[2020-09-14 12:20:54.647683] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 7d1de36f-d6ff-4a34-91f3-30d69e28cadb. sources=[1] 2 sinks=0
[2020-09-14 12:20:54.796212] I [MSGID: 108026]
[afr-self-heal-entry.c:898:afr_selfheal_entry_do] 0-engine-replicate-0: performing entry
selfheal on 65fc9ad6-69bb-4d7a-87de-8f286450ea5b
[2020-09-14 12:20:54.805989] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-engine-replicate-0: Completed entry
selfheal on 65fc9ad6-69bb-4d7a-87de-8f286450ea5b. sources=[0] 1 sinks=2
[2020-09-14 12:20:55.295470] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 7bae68bc-7629-4fa3-8fe4-b13768a98fcc. sources=[1] 2 sinks=0
[2020-09-14 12:21:03.753629] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 7432e014-3990-44fe-9409-9eba4090f547. sources=[1] 2 sinks=0
[2020-09-14 12:21:03.773390] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on cf3abe71-ba89-490a-bb33-5e900cb78967. sources=[1] 2 sinks=0
[2020-09-14 12:21:04.421092] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on f623d6ec-1541-4db2-bc21-35a450d142de. sources=[1] 2 sinks=0
[2020-09-14 12:21:04.503141] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 06874587-6885-4ab3-9cc4-fabb2db99d25. sources=[1] 2 sinks=0
[2020-09-14 12:21:04.505516] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 46c3a7b2-7d5a-432e-857a-dbf332908c89. sources=[1] 2 sinks=0
[2020-09-14 12:21:04.506978] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on f93818dc-0279-4280-856c-151c4eb00f8f. sources=[1] 2 sinks=0
[2020-09-14 12:21:04.507327] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 2fcf8b6a-79a2-4328-bde8-e5e8021c0256. sources=[1] 2 sinks=0
[2020-09-14 12:21:04.507510] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 7f20e494-afc6-4d60-822e-33aa30fe52f7. sources=[1] 2 sinks=0
[2020-09-14 12:21:11.025834] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 4719908b-2613-4976-bdc9-ed10847ea24e. sources=[1] 2 sinks=0
[2020-09-14 12:21:11.692657] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on db070d51-7c16-44f5-8b58-cb923edfc72a. sources=[1] 2 sinks=0
[2020-09-14 12:21:19.500323] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 1f87aefd-684f-4dd3-ad77-a3e47006ef14. sources=[1] 2 sinks=0
[2020-09-14 12:21:19.556305] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 22639cf7-62f5-4198-912f-2b446062ee9b. sources=[1] 2 sinks=0
[2020-09-14 12:21:20.147475] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 0a109124-0e71-4912-9475-47be55805c7f. sources=[1] 2 sinks=0
[2020-09-14 12:21:20.369201] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 028b1032-61ab-49e8-a427-b982d22ede87. sources=[1] 2 sinks=0
[2020-09-14 12:21:20.498106] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 7d99c05f-45b7-4c5e-a7a0-29a6eb6d5353. sources=[1] 2 sinks=0
[2020-09-14 12:21:20.627626] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 731016ec-16be-456f-9f94-d545e83fb730. sources=[1] 2 sinks=0
[2020-09-14 12:21:20.932841] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 54a4dd15-f2ed-4cee-b52c-d76c3a14ee1d. sources=[1] 2 sinks=0
[2020-09-14 12:21:22.950551] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on d9f8db65-bbf6-43d2-a58d-02062e68b094. sources=[1] 2 sinks=0
[2020-09-14 12:21:27.591944] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 2a369725-e692-4985-b1e6-c2ca919977f7. sources=[1] 2 sinks=0
[2020-09-14 12:21:27.811709] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 220f5b3f-7f1f-4afa-8e02-7039dfc46c71. sources=[1] 2 sinks=0
[2020-09-14 12:21:27.847105] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on cf06956a-4726-4a62-9800-310b860ad8c0. sources=[1] 2 sinks=0
[2020-09-14 12:21:27.850830] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on c25c8901-a0f6-40fe-92a2-c616e9bad32e. sources=[1] 2 sinks=0
[2020-09-14 12:21:28.051319] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on ad151cc0-f703-4ba8-8d88-9f21b9b808af. sources=[1] 2 sinks=0
[2020-09-14 12:21:28.254506] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 24a52f25-a440-431a-9ed2-9b7bfa640507. sources=[1] 2 sinks=0
[2020-09-14 12:21:28.436180] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 7f56b5c9-632c-4112-9c9e-bf62b8af09ff. sources=[1] 2 sinks=0
[2020-09-14 12:21:30.127830] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 0f96020d-b860-45ab-9e1d-4ca1839e19f6. sources=[1] 2 sinks=0
[2020-09-14 12:21:31.986013] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on b3b7dc3b-8429-4cb6-8f80-4fe7c23715ac. sources=[1] 2 sinks=0
[2020-09-14 12:21:32.928726] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on d0c4461a-0fa2-439c-ab81-8b97c16a9a8b. sources=[1] 2 sinks=0
[2020-09-14 12:21:33.096508] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 5dff0bbe-cfc6-458f-871a-47cfebfa018a. sources=[1] 2 sinks=0
[2020-09-14 12:21:33.394394] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on f9705bf7-4213-45ca-a17a-f1ebbf11f9ce. sources=[1] 2 sinks=0
[2020-09-14 12:21:38.089910] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on ebe1cce3-3abd-42d3-9504-9e7893721e4b. sources=[1] 2 sinks=0
[2020-09-14 12:21:43.452533] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on ee204b66-5890-469b-a796-7d877b4bf1ab. sources=[1] 2 sinks=0
[2020-09-14 12:21:43.454253] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on ea047187-b01d-49e2-8d02-c38dbf9e77b5. sources=[1] 2 sinks=0
[2020-09-14 12:21:43.710631] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on ce6556b5-a644-4634-91fa-bc70765dbc5a. sources=[1] 2 sinks=0
[2020-09-14 12:21:44.641002] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 96691acb-b98f-4913-a261-c06e15ca18d1. sources=[1] 2 sinks=0
[2020-09-14 12:21:45.332554] I [MSGID: 108026]
[afr-self-heal-common.c:1742:afr_log_selfheal] 0-data-replicate-0: Completed data selfheal
on 8a84ab8b-3890-4bf6-bd49-ebd8d9b96ce2. sources=[1] 2 sinks=0
11:22 p.m.
A few things to consider,
what is your RAID situation per host. If you're using mdadm based soft
raid, you need to make sure your drives support power loss data
protection. This is mostly only a feature on enterprise drives.
Essenstially it ensures the drives reserve enough energy to flush the
write cache to disk on power loss. Most modern drives have a non-trivial
amount of built in write cache and losing that data on power loss will
gladly corrupt files, especially on soft raid setups.
If you're using hardware raid, make sure you have disabled drive based
write cache, and that you have a battery / capacitor connected for the
raid cards cache module.
If you're using ZFS, which isn't really supported, you need a good UPS
and to have it set up to shut systems down cleanly. ZFS will not take
power outages well. Power loss data protection is really important too,
but it's not a fixall for ZFS as it also caches writes in systems RAM
quite a bit. A dedicated cache device with power loss data protection
can help mitigate that, but really the power issues are a more pressing
concern in this situation.
As far as gluster is concerned, there is not much that can easily
corrupt data on power loss. My only thought would be if your switches
are not also battery backed, this would be an issue.
On 2020-10-08 08:15, Jarosław Prokopowski wrote:
> Hi Guys,
>
> I had a situation 2 times that due to unexpected power outage
> something went wrong and VMs on glusterfs where not recoverable.
> Gluster heal did not help and I could not start the VMs any more.
> Is there a way to make such setup bulletproof?
> Does it matter which volume type I choose - raw or qcow2? Or thin
> provision versus reallocated?
> Any other advise?
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MRM6H2YENBP...
6:18 p.m.
Thanks Alex. I actually think that the issue was caused by power loss on the switch
socket.
1499
days inactive
1507
days old
16 comments
5 participants
participants (5)
-
Alex McWhirter -
Jarosław Prokopowski -
Jayme -
Strahil Nikolov -
WK