Multiple Data Storage Domains
--Apple-Mail=_6EA3CA6C-F1CA-43B1-A0ED-5EC9B2DD6690 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I am having an issue in a Hosted Engine GlusterFS setup. I have 4 = hosts in a cluster, with the Engined being hosted on the Cluster. This = follows the pattern shown in the docs for a glusterized setup, except = that I have 4 hosts. I have engine, data, iso and export storage = domains all as glusterfs on a replica 3 glusterfs on the first 3 hosts. = These gluster volumes are running on an SSD Hardware Raid 6, which is = identical on all the hosts. All the hosts have a second Raid 6 Array = with Physical Hard Drives and I have created a second data storage = domain as a glusterfs across all 4 hosts as a stripe 2 replica 2 and = have added it to the Data Center. However if I use this second Storage = Domain as the boot disk for a VM, or as second disk for a VM that is = already running, the VM will become non-responsive as soon as the VM = starts using this disk. Happens during the OS install if the VM is = using this storage domain for its boot disk, or if I try copying = anything large to it when it is a second disk for a VM that has its boot = drive on the Master Data Storage Domain. =20 If I mount the gluster volume that is this second storage domain on one = of the hosts directly or any other machine on my local network, the = gluster volume works fine. It is only when it is used as a storage = domain (second data domain) on VMs in the cluster. Once the vm becomes non-responsive it cannot be stopped, removed or = destroyed without restarting the host machine that the VM is currently = running on. The 4 hosts are connected via 10gig ethernet, so should = not be a network issue. Any ideas? Gary ------------------------------------------------------------------------ Gary Pedretty gary@ravnalaska.net = <mailto:gary@eraalaska.net> Systems Manager www.flyravn.com = <http://www.flyravn.com/> Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Really loving the record green up date! Summmer!! yourself=E2=80=9D = Matt 22:39 ------------------------------------------------------------------------ --Apple-Mail=_6EA3CA6C-F1CA-43B1-A0ED-5EC9B2DD6690 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">I am having an issue in a Hosted Engine GlusterFS setup. = I have 4 hosts in a cluster, with the Engined being hosted on the = Cluster. This follows the pattern shown in the docs for a = glusterized setup, except that I have 4 hosts. I have engine, = data, iso and export storage domains all as glusterfs on a replica 3 = glusterfs on the first 3 hosts. These gluster volumes are running = on an SSD Hardware Raid 6, which is identical on all the hosts. = All the hosts have a second Raid 6 Array with Physical Hard Drives = and I have created a second data storage domain as a glusterfs across = all 4 hosts as a stripe 2 replica 2 and have added it to the Data = Center. However if I use this second Storage Domain as the boot = disk for a VM, or as second disk for a VM that is already running, the = VM will become non-responsive as soon as the VM starts using this disk. = Happens during the OS install if the VM is using this storage = domain for its boot disk, or if I try copying anything large to it when = it is a second disk for a VM that has its boot drive on the Master Data = Storage Domain. <div class=3D""><br class=3D""></div><div = class=3D"">If I mount the gluster volume that is this second storage = domain on one of the hosts directly or any other machine on my local = network, the gluster volume works fine. It is only when it is used = as a storage domain (second data domain) on VMs in the = cluster.</div><div class=3D""><br class=3D""></div><div class=3D"">Once = the vm becomes non-responsive it cannot be stopped, removed or destroyed = without restarting the host machine that the VM is currently running on. = The 4 hosts are connected via 10gig ethernet, so should not be a = network issue.</div><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""></div><div class=3D"">Any ideas?</div><div = class=3D""><br class=3D""></div><div class=3D"">Gary</div><div = class=3D""><br class=3D""></div><div class=3D""><div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; orphans: = auto; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); font-size: 12px; letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div><div class=3D"">Gary Pedretty = = <a = href=3D"mailto:gary@eraalaska.net" = class=3D"">gary@ravnalaska.net</a></div><div class=3D"">Systems Manager = = = <a href=3D"http://www.flyravn.com" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska = = /\ = 907-450-7251</div><div class=3D"">5245 Airport Industrial = Road / \/\ = 907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska = 99709 /\ / \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span style=3D"font-size: 12px;" class=3D"">Serving All of = Alaska / \/ /\ \ \/\ = =E2=80=9CLove your neighbor as</span></font><br = style=3D"font-family: Monaco;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-size: 12px;" class=3D"">Really = loving the record green up date! Summmer!! yourself=E2=80=9D M= att 22:39</span></font><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-family: Menlo;" = class=3D""></div><font face=3D"Menlo" style=3D"font-size: 12px;" = class=3D""></font><span style=3D"color: rgb(0, 0, 0); letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; font-size: 12px;" class=3D""><font = face=3D"Menlo" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div></font></span><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><font = face=3D"Menlo" style=3D"font-size: 12px;" class=3D""><br = class=3D""></font></div></div><span style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 12px;" = class=3D""><br class=3D"Apple-interchange-newline"></span></div><span = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px; font-size: 12px;" class=3D""><br = class=3D"Apple-interchange-newline"></span></div><span style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: = 12px;" class=3D""><br class=3D"Apple-interchange-newline"></span></div><br= class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"><br = class=3D"Apple-interchange-newline"> </div> <br class=3D""></div></body></html>= --Apple-Mail=_6EA3CA6C-F1CA-43B1-A0ED-5EC9B2DD6690--
Hi Gary, Do you have other disks on this storage domain? Have you tried to use other VMs with disks on this storage domain? Is this disk is preallocated? If not can you try to create a pre-allocate disk and re-try Regards, Maor On Sat, Nov 5, 2016 at 2:28 AM, Gary Pedretty <gary@ravnalaska.net> wrote:
I am having an issue in a Hosted Engine GlusterFS setup. I have 4 hosts in a cluster, with the Engined being hosted on the Cluster. This follows the pattern shown in the docs for a glusterized setup, except that I have 4 hosts. I have engine, data, iso and export storage domains all as glusterfs on a replica 3 glusterfs on the first 3 hosts. These gluster volumes are running on an SSD Hardware Raid 6, which is identical on all the hosts. All the hosts have a second Raid 6 Array with Physical Hard Drives and I have created a second data storage domain as a glusterfs across all 4 hosts as a stripe 2 replica 2 and have added it to the Data Center. However if I use this second Storage Domain as the boot disk for a VM, or as second disk for a VM that is already running, the VM will become non-responsive as soon as the VM starts using this disk. Happens during the OS install if the VM is using this storage domain for its boot disk, or if I try copying anything large to it when it is a second disk for a VM that has its boot drive on the Master Data Storage Domain.
If I mount the gluster volume that is this second storage domain on one of the hosts directly or any other machine on my local network, the gluster volume works fine. It is only when it is used as a storage domain (second data domain) on VMs in the cluster.
Once the vm becomes non-responsive it cannot be stopped, removed or destroyed without restarting the host machine that the VM is currently running on. The 4 hosts are connected via 10gig ethernet, so should not be a network issue.
Any ideas?
Gary
------------------------------------------------------------------------ Gary Pedretty gary@ravnalaska.net <gary@eraalaska.net> Systems Manager www.flyravn.com Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ “Love your neighbor as Really loving the record green up date! Summmer!! yourself” Matt 22:39 ------------------------------------------------------------------------
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Nov 6, 2016, at 6:28 AM, Maor Lipchuk <mlipchuk@redhat.com> wrote: =20 Hi Gary, =20 Do you have other disks on this storage domain? Have you tried to use other VMs with disks on this storage domain? Is this disk is preallocated? If not can you try to create a =
=20 Regards, Maor =20 =20 =20 On Sat, Nov 5, 2016 at 2:28 AM, Gary Pedretty <gary@ravnalaska.net = <mailto:gary@ravnalaska.net>> wrote: I am having an issue in a Hosted Engine GlusterFS setup. I have 4 = hosts in a cluster, with the Engined being hosted on the Cluster. This = follows the pattern shown in the docs for a glusterized setup, except =
--Apple-Mail=_06BC44F7-FB91-448D-94C1-8C5321567393 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 As a storage domain, this gluster volume will not work whether it is = preallocated or thin provision. It will work as a straight gluster = volume mounted directly to any VM on the ovirt Cluster, or any physical = machine, just not as a data storage domain in the Data Center. Are there restrictions to having more than one data storage domain that = has it gluster volumes on the same hosts that are also part of the Data = Center and Cluster? Gary ------------------------------------------------------------------------ Gary Pedretty gary@ravnalaska.net = <mailto:gary@eraalaska.net> Systems Manager www.flyravn.com = <http://www.flyravn.com/> Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Really loving the record green up date! Summmer!! yourself=E2=80=9D = Matt 22:39 ------------------------------------------------------------------------ pre-allocate disk and re-try that I have 4 hosts. I have engine, data, iso and export storage = domains all as glusterfs on a replica 3 glusterfs on the first 3 hosts. = These gluster volumes are running on an SSD Hardware Raid 6, which is = identical on all the hosts. All the hosts have a second Raid 6 Array = with Physical Hard Drives and I have created a second data storage = domain as a glusterfs across all 4 hosts as a stripe 2 replica 2 and = have added it to the Data Center. However if I use this second Storage = Domain as the boot disk for a VM, or as second disk for a VM that is = already running, the VM will become non-responsive as soon as the VM = starts using this disk. Happens during the OS install if the VM is = using this storage domain for its boot disk, or if I try copying = anything large to it when it is a second disk for a VM that has its boot = drive on the Master Data Storage Domain.=20
=20 If I mount the gluster volume that is this second storage domain on = one of the hosts directly or any other machine on my local network, the = gluster volume works fine. It is only when it is used as a storage = domain (second data domain) on VMs in the cluster. =20 Once the vm becomes non-responsive it cannot be stopped, removed or = destroyed without restarting the host machine that the VM is currently = running on. The 4 hosts are connected via 10gig ethernet, so should = not be a network issue. =20 =20 Any ideas? =20 Gary
--Apple-Mail=_06BC44F7-FB91-448D-94C1-8C5321567393 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">As a storage domain, this gluster volume will not work = whether it is preallocated or thin provision. It will work as a = straight gluster volume mounted directly to any VM on the ovirt Cluster, = or any physical machine, just not as a data storage domain in the Data = Center.<div class=3D""><br class=3D""></div><div class=3D"">Are there = restrictions to having more than one data storage domain that has it = gluster volumes on the same hosts that are also part of the Data Center = and Cluster?<br class=3D""><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""></div><div class=3D"">Gary</div><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""><div = class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; orphans: = auto; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); font-size: 12px; letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div><div class=3D"">Gary Pedretty = = <a = href=3D"mailto:gary@eraalaska.net" = class=3D"">gary@ravnalaska.net</a></div><div class=3D"">Systems Manager = = = <a href=3D"http://www.flyravn.com" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska = = /\ = 907-450-7251</div><div class=3D"">5245 Airport Industrial = Road / \/\ = 907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska = 99709 /\ / \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span style=3D"font-size: 12px;" class=3D"">Serving All of = Alaska / \/ /\ \ \/\ = =E2=80=9CLove your neighbor as</span></font><br = style=3D"font-family: Monaco;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-size: 12px;" class=3D"">Really = loving the record green up date! Summmer!! yourself=E2=80=9D M= att 22:39</span></font><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-family: Menlo;" = class=3D""></div><font face=3D"Menlo" style=3D"font-size: 12px;" = class=3D""></font><span style=3D"color: rgb(0, 0, 0); letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; font-size: 12px;" class=3D""><font = face=3D"Menlo" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div></font></span><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><font = face=3D"Menlo" style=3D"font-size: 12px;" class=3D""><br = class=3D""></font></div></div><span style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 12px;" = class=3D""><br class=3D"Apple-interchange-newline"></span></div><span = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px; font-size: 12px;" class=3D""><br = class=3D"Apple-interchange-newline"></span></div><span style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: = 12px;" class=3D""><br class=3D"Apple-interchange-newline"></span></div><br= class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"><br = class=3D"Apple-interchange-newline"> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Nov 6, 2016, at 6:28 AM, Maor Lipchuk <<a = href=3D"mailto:mlipchuk@redhat.com" class=3D"">mlipchuk@redhat.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" style=3D"font-family: LucidaGrande; font-size: 13px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D"">Hi = Gary,<div class=3D""><br class=3D""></div><div class=3D"">Do you have = other disks on this storage domain?</div><div class=3D"">Have you tried = to use other VMs with disks on this storage domain?</div><div = class=3D"">Is this disk is preallocated? If not can you try to create a = pre-allocate disk and re-try</div><div class=3D""><br = class=3D""></div><div class=3D"">Regards,</div><div = class=3D"">Maor</div><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""></div></div><div class=3D"gmail_extra" = style=3D"font-family: LucidaGrande; font-size: 13px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;"><br class=3D""><div = class=3D"gmail_quote">On Sat, Nov 5, 2016 at 2:28 AM, Gary Pedretty<span = class=3D"Apple-converted-space"> </span><span dir=3D"ltr" = class=3D""><<a href=3D"mailto:gary@ravnalaska.net" target=3D"_blank" = class=3D"">gary@ravnalaska.net</a>></span><span = class=3D"Apple-converted-space"> </span>wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px = 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, = 204); border-left-style: solid; padding-left: 1ex;"><div = style=3D"word-wrap: break-word;" class=3D"">I am having an issue in a = Hosted Engine GlusterFS setup. I have 4 hosts in a cluster, with = the Engined being hosted on the Cluster. This follows the pattern = shown in the docs for a glusterized setup, except that I have 4 hosts. = I have engine, data, iso and export storage domains all as = glusterfs on a replica 3 glusterfs on the first 3 hosts. These = gluster volumes are running on an SSD Hardware Raid 6, which is = identical on all the hosts. All the hosts have a second Raid 6 = Array with Physical Hard Drives and I have created a second data storage = domain as a glusterfs across all 4 hosts as a stripe 2 replica 2 = and have added it to the Data Center. However if I use this second = Storage Domain as the boot disk for a VM, or as second disk for a VM = that is already running, the VM will become non-responsive as soon as = the VM starts using this disk. Happens during the OS install if = the VM is using this storage domain for its boot disk, or if I try = copying anything large to it when it is a second disk for a VM that has = its boot drive on the Master Data Storage Domain. <div = class=3D""><br class=3D""></div><div class=3D"">If I mount the gluster = volume that is this second storage domain on one of the hosts directly = or any other machine on my local network, the gluster volume works = fine. It is only when it is used as a storage domain (second data = domain) on VMs in the cluster.</div><div class=3D""><br = class=3D""></div><div class=3D"">Once the vm becomes non-responsive it = cannot be stopped, removed or destroyed without restarting the host = machine that the VM is currently running on. The 4 hosts are = connected via 10gig ethernet, so should not be a network = issue.</div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D"">Any ideas?</div><div class=3D""><br = class=3D""></div><div = class=3D"">Gary</div></div></blockquote></div></div></div></blockquote></d= iv><br class=3D""></div></div></body></html>= --Apple-Mail=_06BC44F7-FB91-448D-94C1-8C5321567393--
On Mon, Nov 7, 2016 at 11:20 AM, Gary Pedretty <gary@ravnalaska.net> wrote:
As a storage domain, this gluster volume will not work whether it is preallocated or thin provision. It will work as a straight gluster volume mounted directly to any VM on the ovirt Cluster, or any physical machine, just not as a data storage domain in the Data Center.
Are there restrictions to having more than one data storage domain that has it gluster volumes on the same hosts that are also part of the Data Center and Cluster?
There are no such restrictions. However your volume configuration seems suspect -"stripe 2 replica 2". Can you provide gluster volume info of your second storage domain gluster volume? The mount logs of the volume (under /var/log/glusterfs/rhev-datacenter..<volname>.log) from the host where the volume is being mounted will also help.
Gary
------------------------------------------------------------------------ Gary Pedretty gary@ravnalaska.net <gary@eraalaska.net> Systems Manager www.flyravn.com Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ “Love your neighbor as Really loving the record green up date! Summmer!! yourself” Matt 22:39 ------------------------------------------------------------------------
On Nov 6, 2016, at 6:28 AM, Maor Lipchuk <mlipchuk@redhat.com> wrote:
Hi Gary,
Do you have other disks on this storage domain? Have you tried to use other VMs with disks on this storage domain? Is this disk is preallocated? If not can you try to create a pre-allocate disk and re-try
Regards, Maor
On Sat, Nov 5, 2016 at 2:28 AM, Gary Pedretty <gary@ravnalaska.net> wrote:
I am having an issue in a Hosted Engine GlusterFS setup. I have 4 hosts in a cluster, with the Engined being hosted on the Cluster. This follows the pattern shown in the docs for a glusterized setup, except that I have 4 hosts. I have engine, data, iso and export storage domains all as glusterfs on a replica 3 glusterfs on the first 3 hosts. These gluster volumes are running on an SSD Hardware Raid 6, which is identical on all the hosts. All the hosts have a second Raid 6 Array with Physical Hard Drives and I have created a second data storage domain as a glusterfs across all 4 hosts as a stripe 2 replica 2 and have added it to the Data Center. However if I use this second Storage Domain as the boot disk for a VM, or as second disk for a VM that is already running, the VM will become non-responsive as soon as the VM starts using this disk. Happens during the OS install if the VM is using this storage domain for its boot disk, or if I try copying anything large to it when it is a second disk for a VM that has its boot drive on the Master Data Storage Domain.
If I mount the gluster volume that is this second storage domain on one of the hosts directly or any other machine on my local network, the gluster volume works fine. It is only when it is used as a storage domain (second data domain) on VMs in the cluster.
Once the vm becomes non-responsive it cannot be stopped, removed or destroyed without restarting the host machine that the VM is currently running on. The 4 hosts are connected via 10gig ethernet, so should not be a network issue.
Any ideas?
Gary
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Nov 6, 2016, at 9:50 PM, Sahina Bose <sabose@redhat.com> wrote: =20 However your volume configuration seems suspect -"stripe 2 replica 2". = Can you provide gluster volume info of your second storage domain = gluster volume? The mount logs of the volume (under = /var/log/glusterfs/rhev-datacenter..<volname>.log) from the host where =
--Apple-Mail=_74966C91-4D7B-4631-992A-7104BF8281C2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 [root@fai-kvm-1-gfs admin]# gluster volume status data2 Status of volume: data2 Gluster process TCP Port RDMA Port Online = Pid = --------------------------------------------------------------------------= ---- Brick fai-kvm-1-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49156 0 Y = 3484 Brick fai-kvm-2-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49156 0 Y = 34791 Brick fai-kvm-3-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49156 0 Y = 177340 Brick fai-kvm-4-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49152 0 Y = 146038 NFS Server on localhost 2049 0 Y = 40844 Self-heal Daemon on localhost N/A N/A Y = 40865 NFS Server on fai-kvm-2-gfs.ravnalaska.net 2049 0 Y = 99905 Self-heal Daemon on fai-kvm-2-gfs.ravnalask a.net N/A N/A Y = 99915 NFS Server on fai-kvm-4-gfs.ravnalaska.net 2049 0 Y = 176305 Self-heal Daemon on fai-kvm-4-gfs.ravnalask a.net N/A N/A Y = 176326 NFS Server on fai-kvm-3-gfs.ravnalaska.net 2049 0 Y = 226271 Self-heal Daemon on fai-kvm-3-gfs.ravnalask a.net N/A N/A Y = 226287 Task Status of Volume data2 = --------------------------------------------------------------------------= ---- There are no active volume tasks [root@fai-kvm-1-gfs admin]# gluster volume info data2 Volume Name: data2 Type: Striped-Replicate Volume ID: 20f85c9a-541b-4df4-9dba-44c5179bbfb0 Status: Started Number of Bricks: 1 x 2 x 2 =3D 4 Transport-type: tcp Bricks: Brick1: fai-kvm-1-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Brick2: fai-kvm-2-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Brick3: fai-kvm-3-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Brick4: fai-kvm-4-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 See attached file for the mount log. Gary ------------------------------------------------------------------------ Gary Pedretty gary@ravnalaska.net = <mailto:gary@eraalaska.net> Systems Manager www.flyravn.com = <http://www.flyravn.com/> Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Really loving the record green up date! Summmer!! yourself=E2=80=9D = Matt 22:39 ------------------------------------------------------------------------ the volume is being mounted will also help.
=20
<div class=3D"">Brick2: <a href=3D"http://fai-kvm-2-vmn.ravnalaska.net" = class=3D"">fai-kvm-2-vmn.ravnalaska.net</a>:/kvm2/gluster/data2/brick</div= <div class=3D"">Brick3: <a href=3D"http://fai-kvm-3-vmn.ravnalaska.net" = class=3D"">fai-kvm-3-vmn.ravnalaska.net</a>:/kvm2/gluster/data2/brick</div= <div class=3D"">Brick4: <a href=3D"http://fai-kvm-4-vmn.ravnalaska.net" = class=3D"">fai-kvm-4-vmn.ravnalaska.net</a>:/kvm2/gluster/data2/brick</div= <div class=3D"">Options Reconfigured:</div><div = class=3D"">performance.readdir-ahead: on</div><div = class=3D"">performance.quick-read: off</div><div = class=3D"">performance.read-ahead: off</div><div class=3D""><a =
--Apple-Mail=_74966C91-4D7B-4631-992A-7104BF8281C2 Content-Type: multipart/mixed; boundary="Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062" --Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D"">[root@fai-kvm-1-gfs admin]# gluster volume = status data2</div><div class=3D"">Status of volume: data2</div><div = class=3D"">Gluster process = TCP Port = RDMA Port Online Pid</div><div = class=3D"">---------------------------------------------------------------= ---------------</div><div class=3D"">Brick <a = href=3D"http://fai-kvm-1-vmn.ravnalaska.net" = class=3D"">fai-kvm-1-vmn.ravnalaska.net</a>:/kvm2/gl</div><div = class=3D"">uster/data2/brick = 49156 0 = Y 3484</div><div = class=3D"">Brick <a href=3D"http://fai-kvm-2-vmn.ravnalaska.net" = class=3D"">fai-kvm-2-vmn.ravnalaska.net</a>:/kvm2/gl</div><div = class=3D"">uster/data2/brick = 49156 0 = Y 34791</div><div = class=3D"">Brick <a href=3D"http://fai-kvm-3-vmn.ravnalaska.net" = class=3D"">fai-kvm-3-vmn.ravnalaska.net</a>:/kvm2/gl</div><div = class=3D"">uster/data2/brick = 49156 0 = Y = 177340</div><div class=3D"">Brick <a = href=3D"http://fai-kvm-4-vmn.ravnalaska.net" = class=3D"">fai-kvm-4-vmn.ravnalaska.net</a>:/kvm2/gl</div><div = class=3D"">uster/data2/brick = 49152 0 = Y = 146038</div><div class=3D"">NFS Server on localhost = 2049 = 0 Y = 40844</div><div class=3D"">Self-heal Daemon on localhost = N/A N/A = Y 40865</div><div class=3D"">NFS = Server on <a href=3D"http://fai-kvm-2-gfs.ravnalaska.net" = class=3D"">fai-kvm-2-gfs.ravnalaska.net</a> 2049 = 0 Y = 99905</div><div class=3D"">Self-heal Daemon on = fai-kvm-2-gfs.ravnalask</div><div class=3D""><a href=3D"http://a.net" = class=3D"">a.net</a> = = N/A N/A Y = 99915</div><div class=3D"">NFS Server on <a = href=3D"http://fai-kvm-4-gfs.ravnalaska.net" = class=3D"">fai-kvm-4-gfs.ravnalaska.net</a> 2049 = 0 Y = 176305</div><div class=3D"">Self-heal Daemon on = fai-kvm-4-gfs.ravnalask</div><div class=3D""><a href=3D"http://a.net" = class=3D"">a.net</a> = = N/A N/A Y = 176326</div><div class=3D"">NFS Server on <a = href=3D"http://fai-kvm-3-gfs.ravnalaska.net" = class=3D"">fai-kvm-3-gfs.ravnalaska.net</a> 2049 = 0 Y = 226271</div><div class=3D"">Self-heal Daemon on = fai-kvm-3-gfs.ravnalask</div><div class=3D""><a href=3D"http://a.net" = class=3D"">a.net</a> = = N/A N/A Y = 226287</div><div class=3D""><br class=3D""></div><div= class=3D"">Task Status of Volume data2</div><div = class=3D"">---------------------------------------------------------------= ---------------</div><div class=3D"">There are no active volume = tasks</div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D""><div class=3D"">[root@fai-kvm-1-gfs = admin]# gluster volume info data2</div><div class=3D""><br = class=3D""></div><div class=3D"">Volume Name: data2</div><div = class=3D"">Type: Striped-Replicate</div><div class=3D"">Volume ID: = 20f85c9a-541b-4df4-9dba-44c5179bbfb0</div><div class=3D"">Status: = Started</div><div class=3D"">Number of Bricks: 1 x 2 x 2 =3D 4</div><div = class=3D"">Transport-type: tcp</div><div class=3D"">Bricks:</div><div = class=3D"">Brick1: <a href=3D"http://fai-kvm-1-vmn.ravnalaska.net" = class=3D"">fai-kvm-1-vmn.ravnalaska.net</a>:/kvm2/gluster/data2/brick</div= href=3D"http://performance.io" class=3D"">performance.io</a>-cache: = off</div><div class=3D"">performance.stat-prefetch: off</div><div = class=3D"">cluster.eager-lock: enable</div><div = class=3D"">network.remote-dio: enable</div><div = class=3D"">cluster.quorum-type: auto</div><div = class=3D"">cluster.server-quorum-type: server</div><div = class=3D"">storage.owner-uid: 36</div><div class=3D"">storage.owner-gid: = 36</div></div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D"">See attached file for the mount = log.</div><div class=3D""><br class=3D""></div><div = class=3D"">Gary</div><div class=3D""><br class=3D""></div><div = class=3D""></div></body></html>= --Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062 Content-Disposition: attachment; filename=glusterlog.txt Content-Type: text/plain; x-unix-mode=0644; name="glusterlog.txt" Content-Transfer-Encoding: quoted-printable more rhev-data-center-mnt-glusterSD-glustermount2:data2.log-20161106 [2016-11-04 22:54:32.118587] I [MSGID: 100030] [glusterfsd.c:2338:main] = 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version = 3.7.16 (args: /usr/sbin/glusterfs --volfile-server=3Dglustermou nt2 --volfile-server=3Dfai-kvm-1-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-2-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-3-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-4-vmn.ravnalaska.net --volfile-id =3Ddata2 /rhev/data-center/mnt/glusterSD/glustermount2:data2) [2016-11-04 22:54:32.128807] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 1 [2016-11-04 22:54:32.138959] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 2 [2016-11-04 22:54:32.139878] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-0: parent translators are ready, attempting connect on = transport [2016-11-04 22:54:32.142057] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-1: parent translators are ready, attempting connect on = transport [2016-11-04 22:54:32.142331] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-0: changing port to 49156 (from 0) [2016-11-04 22:54:32.143961] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-2: parent translators are ready, attempting connect on = transport [2016-11-04 22:54:32.146123] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-04 22:54:32.146308] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-1: changing port to 49156 (from 0) [2016-11-04 22:54:32.147618] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-3: parent translators are ready, attempting connect on = transport [2016-11-04 22:54:32.149874] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-0: = Connected to data2-client-0, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-04 22:54:32.149891] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-0: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-04 22:54:32.149941] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-0: Subvolume = 'data2-client-0' came back up; going online. [2016-11-04 22:54:32.150000] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-0: = Server lk version =3D 1 [2016-11-04 22:54:32.150080] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-04 22:54:32.150274] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-2: changing port to 49156 (from 0) Final graph: = +-------------------------------------------------------------------------= -----+ 1: volume data2-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key data2 6: option client-version 3.7.16 7: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-105264-2016/11/04-22:54:32:114962-data2-clien= t-0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host fai-kvm-1-vmn.ravnalaska.net 11: option remote-subvolume /kvm2/gluster/data2/brick 12: option transport-type socket 13: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 14: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume data2-client-1 20: type protocol/client 21: option clnt-lk-version 1 22: option volfile-checksum 0 23: option volfile-key data2 24: option client-version 3.7.16 25: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-105264-2016/11/04-22:54:32:114962-data2-clien= t-1-0-0 26: option fops-version 1298437 27: option ping-timeout 42 28: option remote-host fai-kvm-2-vmn.ravnalaska.net 29: option remote-subvolume /kvm2/gluster/data2/brick 30: option transport-type socket 31: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 32: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 33: option filter-O_DIRECT enable 34: option send-gids true 35: end-volume 36: 37: volume data2-replicate-0 38: type cluster/replicate 39: option eager-lock enable 40: option quorum-type auto 41: subvolumes data2-client-0 data2-client-1 42: end-volume 43: 44: volume data2-client-2 45: type protocol/client 46: option ping-timeout 42 47: option remote-host fai-kvm-3-vmn.ravnalaska.net 48: option remote-subvolume /kvm2/gluster/data2/brick 49: option transport-type socket 50: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 51: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 52: option filter-O_DIRECT enable 53: option send-gids true 54: end-volume 55: 56: volume data2-client-3 57: type protocol/client 58: option ping-timeout 42 59: option remote-host fai-kvm-4-vmn.ravnalaska.net 60: option remote-subvolume /kvm2/gluster/data2/brick 61: option transport-type socket 62: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 63: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 64: option filter-O_DIRECT enable 65: option send-gids true 66: end-volume 67: 68: volume data2-replicate-1 69: type cluster/replicate 70: option eager-lock enable 71: option quorum-type auto 72: subvolumes data2-client-2 data2-client-3 73: end-volume 74: 75: volume data2-stripe-0 76: type cluster/stripe 77: subvolumes data2-replicate-0 data2-replicate-1 78: end-volume 79: 80: volume data2-dht 81: type cluster/distribute 82: subvolumes data2-stripe-0 83: end-volume 84: 85: volume data2-write-behind 86: type performance/write-behind 87: subvolumes data2-dht 88: end-volume 89: 90: volume data2-readdir-ahead 91: type performance/readdir-ahead 92: subvolumes data2-write-behind 93: end-volume 94: 95: volume data2-open-behind 96: type performance/open-behind 97: subvolumes data2-readdir-ahead 98: end-volume 99: 100: volume data2 101: type debug/io-stats 102: option log-level INFO 103: option latency-measurement off 104: option count-fop-hits off 105: subvolumes data2-open-behind 106: end-volume 107: 108: volume meta-autoload 109: type meta 110: subvolumes data2 111: end-volume 112: = +-------------------------------------------------------------------------= -----+ [2016-11-04 22:54:32.151927] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-1: = Connected to data2-client-1, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-04 22:54:32.151939] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-1: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-04 22:54:32.152083] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-1: = Server lk version =3D 1 [2016-11-04 22:54:32.152341] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-3: changing port to 49152 (from 0) [2016-11-04 22:54:32.154296] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-04 22:54:32.154774] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-2: = Connected to data2-client-2, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-04 22:54:32.154789] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-2: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-04 22:54:32.154821] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-1: Subvolume = 'data2-client-2' came back up; going online. [2016-11-04 22:54:32.154916] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-2: = Server lk version =3D 1 [2016-11-04 22:54:32.156348] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-3: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-04 22:54:32.156860] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-3: = Connected to data2-client-3, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-04 22:54:32.156874] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-3: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-04 22:54:32.160382] I [fuse-bridge.c:5171:fuse_graph_setup] = 0-fuse: switched to graph 0 [2016-11-04 22:54:32.160451] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-3: = Server lk version =3D 1 [2016-11-04 22:54:32.160524] I [fuse-bridge.c:4083:fuse_init] = 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 = kernel 7.22 [2016-11-04 22:54:32.161171] I [MSGID: 108031] = [afr-common.c:2071:afr_local_discovery_cbk] 0-data2-replicate-0: = selecting local read_child data2-client-0 [2016-11-05 00:05:37.796730] W [glusterfsd.c:1251:cleanup_and_exit] = (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fe90527adc5] = -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fe9068f5915] = -->/usr/sbin/ glusterfs(cleanup_and_exit+0x6b) [0x7fe9068f578b] ) 0-: received signum = (15), shutting down [2016-11-05 00:05:37.796770] I [fuse-bridge.c:5719:fini] 0-fuse: = Unmounting '/rhev/data-center/mnt/glusterSD/glustermount2:data2'. [2016-11-05 00:13:04.440431] I [MSGID: 100030] [glusterfsd.c:2338:main] = 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version = 3.7.16 (args: /usr/sbin/glusterfs --volfile-server=3Dglustermou nt2 --volfile-server=3Dfai-kvm-1-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-2-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-3-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-4-vmn.ravnalaska.net --volfile-id =3Ddata2 /rhev/data-center/mnt/glusterSD/glustermount2:data2) [2016-11-05 00:13:04.452151] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 1 [2016-11-05 00:13:04.463593] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 2 [2016-11-05 00:13:04.464394] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-0: parent translators are ready, attempting connect on = transport [2016-11-05 00:13:04.466698] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-1: parent translators are ready, attempting connect on = transport [2016-11-05 00:13:04.466885] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-0: changing port to 49156 (from 0) [2016-11-05 00:13:04.468672] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-2: parent translators are ready, attempting connect on = transport [2016-11-05 00:13:04.470686] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:13:04.470849] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-1: changing port to 49156 (from 0) [2016-11-05 00:13:04.472375] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-3: parent translators are ready, attempting connect on = transport [2016-11-05 00:13:04.475429] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-0: = Connected to data2-client-0, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:13:04.475448] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-0: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:13:04.475503] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-0: Subvolume = 'data2-client-0' came back up; going online. [2016-11-05 00:13:04.475615] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-0: = Server lk version =3D 1 [2016-11-05 00:13:04.475687] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:13:04.475857] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-2: changing port to 49156 (from 0) Final graph: = +-------------------------------------------------------------------------= -----+ 1: volume data2-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key data2 6: option client-version 3.7.16 7: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-13785-2016/11/05-00:13:04:437247-data2-client= -0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host fai-kvm-1-vmn.ravnalaska.net 11: option remote-subvolume /kvm2/gluster/data2/brick 12: option transport-type socket 13: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 14: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume data2-client-1 20: type protocol/client 21: option clnt-lk-version 1 22: option volfile-checksum 0 23: option volfile-key data2 24: option client-version 3.7.16 25: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-13785-2016/11/05-00:13:04:437247-data2-client= -1-0-0 26: option fops-version 1298437 27: option ping-timeout 42 28: option remote-host fai-kvm-2-vmn.ravnalaska.net 29: option remote-subvolume /kvm2/gluster/data2/brick 30: option transport-type socket 31: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 32: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 33: option filter-O_DIRECT enable 34: option send-gids true 35: end-volume 36: 37: volume data2-replicate-0 38: type cluster/replicate 39: option eager-lock enable 40: option quorum-type auto 41: subvolumes data2-client-0 data2-client-1 42: end-volume 43: 44: volume data2-client-2 45: type protocol/client 46: option ping-timeout 42 47: option remote-host fai-kvm-3-vmn.ravnalaska.net 48: option remote-subvolume /kvm2/gluster/data2/brick 49: option transport-type socket 50: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 51: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 52: option filter-O_DIRECT enable 53: option send-gids true 54: end-volume 55: 56: volume data2-client-3 57: type protocol/client 58: option ping-timeout 42 59: option remote-host fai-kvm-4-vmn.ravnalaska.net 60: option remote-subvolume /kvm2/gluster/data2/brick 61: option transport-type socket 62: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 63: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 64: option filter-O_DIRECT enable 65: option send-gids true 66: end-volume 67: 68: volume data2-replicate-1 69: type cluster/replicate 70: option eager-lock enable 71: option quorum-type auto 72: subvolumes data2-client-2 data2-client-3 73: end-volume 74: 75: volume data2-stripe-0 76: type cluster/stripe 77: subvolumes data2-replicate-0 data2-replicate-1 78: end-volume 79: 80: volume data2-dht 81: type cluster/distribute 82: subvolumes data2-stripe-0 83: end-volume 84: 85: volume data2-write-behind 86: type performance/write-behind 87: subvolumes data2-dht 88: end-volume 89: 90: volume data2-readdir-ahead 91: type performance/readdir-ahead 92: subvolumes data2-write-behind 93: end-volume 94: 95: volume data2-open-behind 96: type performance/open-behind 97: subvolumes data2-readdir-ahead 98: end-volume 99: 100: volume data2 101: type debug/io-stats 102: option log-level INFO 103: option latency-measurement off 104: option count-fop-hits off 105: subvolumes data2-open-behind 106: end-volume 107: 108: volume meta-autoload 109: type meta 110: subvolumes data2 111: end-volume 112: = +-------------------------------------------------------------------------= -----+ [2016-11-05 00:13:04.477682] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-1: = Connected to data2-client-1, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:13:04.477702] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-1: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:13:04.477834] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-1: = Server lk version =3D 1 [2016-11-05 00:13:04.478162] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-3: changing port to 49152 (from 0) [2016-11-05 00:13:04.480593] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:13:04.480961] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-2: = Connected to data2-client-2, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:13:04.480982] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-2: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:13:04.481020] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-1: Subvolume = 'data2-client-2' came back up; going online. [2016-11-05 00:13:04.481120] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-2: = Server lk version =3D 1 [2016-11-05 00:13:04.483500] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-3: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:13:04.483956] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-3: = Connected to data2-client-3, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:13:04.483973] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-3: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:13:04.488332] I [fuse-bridge.c:5171:fuse_graph_setup] = 0-fuse: switched to graph 0 [2016-11-05 00:13:04.488889] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-3: = Server lk version =3D 1 [2016-11-05 00:13:04.488933] I [fuse-bridge.c:4083:fuse_init] = 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 = kernel 7.22 [2016-11-05 00:13:04.489627] I [MSGID: 108031] = [afr-common.c:2071:afr_local_discovery_cbk] 0-data2-replicate-0: = selecting local read_child data2-client-0 [2016-11-05 00:52:07.549997] I [fuse-bridge.c:5012:fuse_thread_proc] = 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/glustermount2:data2 [2016-11-05 00:52:07.550544] W [glusterfsd.c:1251:cleanup_and_exit] = (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f40b1c24dc5] = -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f40b329f915] = -->/usr/sbin/ glusterfs(cleanup_and_exit+0x6b) [0x7f40b329f78b] ) 0-: received signum = (15), shutting down [2016-11-05 00:52:07.550583] I [fuse-bridge.c:5719:fini] 0-fuse: = Unmounting '/rhev/data-center/mnt/glusterSD/glustermount2:data2'. [2016-11-05 00:52:39.557750] I [MSGID: 100030] [glusterfsd.c:2338:main] = 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version = 3.7.16 (args: /usr/sbin/glusterfs --volfile-server=3Dglustermou nt2 --volfile-server=3Dfai-kvm-1-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-2-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-3-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-4-vmn.ravnalaska.net --volfile-id =3Ddata2 /rhev/data-center/mnt/glusterSD/glustermount2:data2) [2016-11-05 00:52:39.567614] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 1 [2016-11-05 00:52:39.578148] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 2 [2016-11-05 00:52:39.579039] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-0: parent translators are ready, attempting connect on = transport [2016-11-05 00:52:39.581159] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-1: parent translators are ready, attempting connect on = transport [2016-11-05 00:52:39.581483] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-0: changing port to 49156 (from 0) [2016-11-05 00:52:39.583109] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-2: parent translators are ready, attempting connect on = transport [2016-11-05 00:52:39.585673] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:52:39.585849] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-1: changing port to 49156 (from 0) [2016-11-05 00:52:39.587044] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-3: parent translators are ready, attempting connect on = transport [2016-11-05 00:52:39.589467] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-0: = Connected to data2-client-0, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:52:39.589499] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-0: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:52:39.589571] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-0: Subvolume = 'data2-client-0' came back up; going online. [2016-11-05 00:52:39.589654] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-0: = Server lk version =3D 1 [2016-11-05 00:52:39.589753] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:52:39.590026] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-2: changing port to 49156 (from 0) [2016-11-05 00:52:39.590074] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-1: = Connected to data2-client-1, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:52:39.590088] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-1: Server = and Client lk-version numbers are not same, reopening the fds Final graph: = +-------------------------------------------------------------------------= -----+ 1: volume data2-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key data2 6: option client-version 3.7.16 7: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-30520-2016/11/05-00:52:39:553646-data2-client= -0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host fai-kvm-1-vmn.ravnalaska.net 11: option remote-subvolume /kvm2/gluster/data2/brick 12: option transport-type socket 13: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 14: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume data2-client-1 20: type protocol/client 21: option clnt-lk-version 1 22: option volfile-checksum 0 23: option volfile-key data2 24: option client-version 3.7.16 25: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-30520-2016/11/05-00:52:39:553646-data2-client= -1-0-0 26: option fops-version 1298437 27: option ping-timeout 42 28: option remote-host fai-kvm-2-vmn.ravnalaska.net 29: option remote-subvolume /kvm2/gluster/data2/brick 30: option transport-type socket 31: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 32: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 33: option filter-O_DIRECT enable 34: option send-gids true 35: end-volume 36: 37: volume data2-replicate-0 38: type cluster/replicate 39: option eager-lock enable 40: option quorum-type auto 41: subvolumes data2-client-0 data2-client-1 42: end-volume 43: 44: volume data2-client-2 45: type protocol/client 46: option ping-timeout 42 47: option remote-host fai-kvm-3-vmn.ravnalaska.net 48: option remote-subvolume /kvm2/gluster/data2/brick 49: option transport-type socket 50: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 51: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 52: option filter-O_DIRECT enable 53: option send-gids true 54: end-volume 55: 56: volume data2-client-3 57: type protocol/client 58: option ping-timeout 42 59: option remote-host fai-kvm-4-vmn.ravnalaska.net 60: option remote-subvolume /kvm2/gluster/data2/brick 61: option transport-type socket 62: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 63: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 64: option filter-O_DIRECT enable 65: option send-gids true 66: end-volume 67: 68: volume data2-replicate-1 69: type cluster/replicate 70: option eager-lock enable 71: option quorum-type auto 72: subvolumes data2-client-2 data2-client-3 73: end-volume 74: 75: volume data2-stripe-0 76: type cluster/stripe 77: subvolumes data2-replicate-0 data2-replicate-1 78: end-volume 79: 80: volume data2-dht 81: type cluster/distribute 82: subvolumes data2-stripe-0 83: end-volume 84: 85: volume data2-write-behind 86: type performance/write-behind 87: subvolumes data2-dht 88: end-volume 89: 90: volume data2-readdir-ahead 91: type performance/readdir-ahead 92: subvolumes data2-write-behind 93: end-volume 94: 95: volume data2-open-behind 96: type performance/open-behind 97: subvolumes data2-readdir-ahead 98: end-volume 99: 100: volume data2 101: type debug/io-stats 102: option log-level INFO 103: option latency-measurement off 104: option count-fop-hits off 105: subvolumes data2-open-behind 106: end-volume 107: 108: volume meta-autoload 109: type meta 110: subvolumes data2 111: end-volume 112: = +-------------------------------------------------------------------------= -----+ [2016-11-05 00:52:39.591502] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-1: = Server lk version =3D 1 [2016-11-05 00:52:39.591817] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-3: changing port to 49152 (from 0) [2016-11-05 00:52:39.593714] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:52:39.594285] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-2: = Connected to data2-client-2, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:52:39.594308] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-2: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:52:39.594365] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-1: Subvolume = 'data2-client-2' came back up; going online. [2016-11-05 00:52:39.594552] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-2: = Server lk version =3D 1 [2016-11-05 00:52:39.595616] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-3: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:52:39.596115] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-3: = Connected to data2-client-3, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:52:39.596130] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-3: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:52:39.600510] I [fuse-bridge.c:5171:fuse_graph_setup] = 0-fuse: switched to graph 0 [2016-11-05 00:52:39.600578] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-3: = Server lk version =3D 1 [2016-11-05 00:52:39.600654] I [fuse-bridge.c:4083:fuse_init] = 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 = kernel 7.22 [2016-11-05 00:52:39.601452] I [MSGID: 108031] = [afr-common.c:2071:afr_local_discovery_cbk] 0-data2-replicate-0: = selecting local read_child data2-client-0 [2016-11-05 00:52:41.619852] I [fuse-bridge.c:5012:fuse_thread_proc] = 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/glustermount2:data2 [2016-11-05 00:52:41.620095] W [glusterfsd.c:1251:cleanup_and_exit] = (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f3444227dc5] = -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f34458a2915] = -->/usr/sbin/ glusterfs(cleanup_and_exit+0x6b) [0x7f34458a278b] ) 0-: received signum = (15), shutting down [2016-11-05 00:52:41.620110] I [fuse-bridge.c:5719:fini] 0-fuse: = Unmounting '/rhev/data-center/mnt/glusterSD/glustermount2:data2'. [root@fai-kvm-1-gfs glusterfs]#= --Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D""></div><div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; orphans: = auto; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); font-size: 12px; letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div><div class=3D"">Gary Pedretty = = <a = href=3D"mailto:gary@eraalaska.net" = class=3D"">gary@ravnalaska.net</a></div><div class=3D"">Systems Manager = = = <a href=3D"http://www.flyravn.com" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska = = /\ = 907-450-7251</div><div class=3D"">5245 Airport Industrial = Road / \/\ = 907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska = 99709 /\ / \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span style=3D"font-size: 12px;" class=3D"">Serving All of = Alaska / \/ /\ \ \/\ = =E2=80=9CLove your neighbor as</span></font><br = style=3D"font-family: Monaco;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-size: 12px;" class=3D"">Really = loving the record green up date! Summmer!! yourself=E2=80=9D M= att 22:39</span></font><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-family: Menlo;" = class=3D""></div><font face=3D"Menlo" style=3D"font-size: 12px;" = class=3D""></font><span style=3D"color: rgb(0, 0, 0); letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; font-size: 12px;" class=3D""><font = face=3D"Menlo" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div></font></span><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><font = face=3D"Menlo" style=3D"font-size: 12px;" class=3D""><br = class=3D""></font></div></div><span style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 12px;" = class=3D""><br class=3D"Apple-interchange-newline"></span></div><span = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px; font-size: 12px;" class=3D""><br = class=3D"Apple-interchange-newline"></span></div><span style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: = 12px;" class=3D""><br class=3D"Apple-interchange-newline"></span></div><br= class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"><br = class=3D"Apple-interchange-newline"> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Nov 6, 2016, at 9:50 PM, Sahina Bose <<a = href=3D"mailto:sabose@redhat.com" class=3D"">sabose@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><span = style=3D"font-family: LucidaGrande; font-size: 13px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; float: none; display: inline = !important;" class=3D"">However your volume configuration seems suspect = -"stripe 2 replica 2". Can you provide gluster volume info of your = second storage domain gluster volume? The mount logs of the volume = (under /var/log/glusterfs/rhev-datacenter..<volname>.log) from the = host where the volume is being mounted will also help.</span><br = style=3D"font-family: LucidaGrande; font-size: 13px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><span style=3D"font-family: = LucidaGrande; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = float: none; display: inline !important;" = class=3D""> </span></div></blockquote></div><br = class=3D""></body></html>= --Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062-- --Apple-Mail=_74966C91-4D7B-4631-992A-7104BF8281C2--
On Mon, Nov 7, 2016 at 3:27 PM, Gary Pedretty <gary@ravnalaska.net> wrote:
[root@fai-kvm-1-gfs admin]# gluster volume status data2 Status of volume: data2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------ ------------------ Brick fai-kvm-1-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49156 0 Y 3484 Brick fai-kvm-2-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49156 0 Y 34791 Brick fai-kvm-3-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49156 0 Y 177340 Brick fai-kvm-4-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49152 0 Y 146038 NFS Server on localhost 2049 0 Y 40844 Self-heal Daemon on localhost N/A N/A Y 40865 NFS Server on fai-kvm-2-gfs.ravnalaska.net 2049 0 Y 99905 Self-heal Daemon on fai-kvm-2-gfs.ravnalask a.net N/A N/A Y 99915 NFS Server on fai-kvm-4-gfs.ravnalaska.net 2049 0 Y 176305 Self-heal Daemon on fai-kvm-4-gfs.ravnalask a.net N/A N/A Y 176326 NFS Server on fai-kvm-3-gfs.ravnalaska.net 2049 0 Y 226271 Self-heal Daemon on fai-kvm-3-gfs.ravnalask a.net N/A N/A Y 226287
Task Status of Volume data2 ------------------------------------------------------------ ------------------ There are no active volume tasks
[root@fai-kvm-1-gfs admin]# gluster volume info data2
Volume Name: data2 Type: Striped-Replicate Volume ID: 20f85c9a-541b-4df4-9dba-44c5179bbfb0 Status: Started Number of Bricks: 1 x 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: fai-kvm-1-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Brick2: fai-kvm-2-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Brick3: fai-kvm-3-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Brick4: fai-kvm-4-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36
See attached file for the mount log.
Striped-Replicate is no longer supported in GlusterFS upstream. Instead, you should be using a Distribute-Replicate with sharding enabled. Also when using a gluster volume as storage domain, it is recommended to use replica 3.
From the mount logs, there is no indication as to why the volume is unmounted frequently. Could you try again with a replica 3 volume that has sharding enabled?
Gary
------------------------------------------------------------------------ Gary Pedretty gary@ravnalaska.net <gary@eraalaska.net> Systems Manager www.flyravn.com Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ “Love your neighbor as Really loving the record green up date! Summmer!! yourself” Matt 22:39 ------------------------------------------------------------------------
On Nov 6, 2016, at 9:50 PM, Sahina Bose <sabose@redhat.com> wrote:
However your volume configuration seems suspect -"stripe 2 replica 2". Can you provide gluster volume info of your second storage domain gluster volume? The mount logs of the volume (under /var/log/glusterfs/rhev-datacenter..<volname>.log) from the host where the volume is being mounted will also help.
--Apple-Mail=_5640DDB5-E478-43B7-AB26-61A2B8B4688F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Solved: Changing the second storage domain to a glusterfs Distributed Replicate = with sharding turned on, works great. Thanks for the solution. Gary ------------------------------------------------------------------------ Gary Pedretty gary@ravnalaska.net = <mailto:gary@eraalaska.net> Systems Manager www.flyravn.com = <http://www.flyravn.com/> Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Really loving the record green up date! Summmer!! yourself=E2=80=9D = Matt 22:39 ------------------------------------------------------------------------
On Nov 7, 2016, at 2:10 AM, Sahina Bose <sabose@redhat.com> wrote: =20 =20 =20 On Mon, Nov 7, 2016 at 3:27 PM, Gary Pedretty <gary@ravnalaska.net = <mailto:gary@ravnalaska.net>> wrote: [root@fai-kvm-1-gfs admin]# gluster volume status data2 Status of volume: data2 Gluster process TCP Port RDMA Port = Online Pid = --------------------------------------------------------------------------=
Brick fai-kvm-1-vmn.ravnalaska.net = <http://fai-kvm-1-vmn.ravnalaska.net/>:/kvm2/gl uster/data2/brick 49156 0 Y = 3484 Brick fai-kvm-2-vmn.ravnalaska.net = <http://fai-kvm-2-vmn.ravnalaska.net/>:/kvm2/gl uster/data2/brick 49156 0 Y = 34791 Brick fai-kvm-3-vmn.ravnalaska.net = <http://fai-kvm-3-vmn.ravnalaska.net/>:/kvm2/gl uster/data2/brick 49156 0 Y = 177340 Brick fai-kvm-4-vmn.ravnalaska.net = <http://fai-kvm-4-vmn.ravnalaska.net/>:/kvm2/gl uster/data2/brick 49152 0 Y = 146038 NFS Server on localhost 2049 0 Y = 40844 Self-heal Daemon on localhost N/A N/A Y = 40865 NFS Server on fai-kvm-2-gfs.ravnalaska.net = <http://fai-kvm-2-gfs.ravnalaska.net/> 2049 0 Y = 99905 Self-heal Daemon on fai-kvm-2-gfs.ravnalask a.net <http://a.net/> N/A = N/A Y 99915 NFS Server on fai-kvm-4-gfs.ravnalaska.net = <http://fai-kvm-4-gfs.ravnalaska.net/> 2049 0 Y = 176305 Self-heal Daemon on fai-kvm-4-gfs.ravnalask a.net <http://a.net/> N/A = N/A Y 176326 NFS Server on fai-kvm-3-gfs.ravnalaska.net = <http://fai-kvm-3-gfs.ravnalaska.net/> 2049 0 Y = 226271 Self-heal Daemon on fai-kvm-3-gfs.ravnalask a.net <http://a.net/> N/A = N/A Y 226287 =20 Task Status of Volume data2 = --------------------------------------------------------------------------=
There are no active volume tasks =20 =20 [root@fai-kvm-1-gfs admin]# gluster volume info data2 =20 Volume Name: data2 Type: Striped-Replicate Volume ID: 20f85c9a-541b-4df4-9dba-44c5179bbfb0 Status: Started Number of Bricks: 1 x 2 x 2 =3D 4 Transport-type: tcp Bricks: Brick1: fai-kvm-1-vmn.ravnalaska.net = <http://fai-kvm-1-vmn.ravnalaska.net/>:/kvm2/gluster/data2/brick Brick2: fai-kvm-2-vmn.ravnalaska.net = <http://fai-kvm-2-vmn.ravnalaska.net/>:/kvm2/gluster/data2/brick Brick3: fai-kvm-3-vmn.ravnalaska.net = <http://fai-kvm-3-vmn.ravnalaska.net/>:/kvm2/gluster/data2/brick Brick4: fai-kvm-4-vmn.ravnalaska.net = <http://fai-kvm-4-vmn.ravnalaska.net/>:/kvm2/gluster/data2/brick Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io <http://performance.io/>-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 =20 =20 See attached file for the mount log. =20 =20 Striped-Replicate is no longer supported in GlusterFS upstream. = Instead, you should be using a Distribute-Replicate with sharding = enabled. Also when using a gluster volume as storage domain, it is = recommended to use replica 3. =20 =46rom the mount logs, there is no indication as to why the volume is = unmounted frequently. Could you try again with a replica 3 volume that = has sharding enabled? =20 =20 Gary =20 =20 =
Gary Pedretty = gary@ravnalaska.net <mailto:gary@eraalaska.net> Systems Manager = www.flyravn.com <http://www.flyravn.com/> Ravn Alaska /\ = 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 = fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest = commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Really loving the record green up date! Summmer!! yourself=E2=80=9D = Matt 22:39 =
On Nov 6, 2016, at 9:50 PM, Sahina Bose <sabose@redhat.com = <mailto:sabose@redhat.com>> wrote: =20 However your volume configuration seems suspect -"stripe 2 replica = 2". Can you provide gluster volume info of your second storage domain = gluster volume? The mount logs of the volume (under = /var/log/glusterfs/rhev-datacenter..<volname>.log) from the host where =
=20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 the volume is being mounted will also help.
=20 =20 =20 =20
--Apple-Mail=_5640DDB5-E478-43B7-AB26-61A2B8B4688F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">Solved:<div class=3D""><br class=3D""></div><div = class=3D"">Changing the second storage domain to a glusterfs Distributed = Replicate with sharding turned on, works great. Thanks for the = solution.</div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D"">Gary</div><div class=3D""><br = class=3D""></div><div class=3D""><br class=3D""></div><div class=3D""><div= class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; orphans: = auto; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); font-size: 12px; letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div><div class=3D"">Gary Pedretty = = <a = href=3D"mailto:gary@eraalaska.net" = class=3D"">gary@ravnalaska.net</a></div><div class=3D"">Systems Manager = = = <a href=3D"http://www.flyravn.com" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska = = /\ = 907-450-7251</div><div class=3D"">5245 Airport Industrial = Road / \/\ = 907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska = 99709 /\ / \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span style=3D"font-size: 12px;" class=3D"">Serving All of = Alaska / \/ /\ \ \/\ = =E2=80=9CLove your neighbor as</span></font><br = style=3D"font-family: Monaco;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-size: 12px;" class=3D"">Really = loving the record green up date! Summmer!! yourself=E2=80=9D M= att 22:39</span></font><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-family: Menlo;" = class=3D""></div><font face=3D"Menlo" style=3D"font-size: 12px;" = class=3D""></font><span style=3D"color: rgb(0, 0, 0); letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; font-size: 12px;" class=3D""><font = face=3D"Menlo" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div></font></span><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><font = face=3D"Menlo" style=3D"font-size: 12px;" class=3D""><br = class=3D""></font></div></div><span style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 12px;" = class=3D""><br class=3D"Apple-interchange-newline"></span></div><span = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px; font-size: 12px;" class=3D""><br = class=3D"Apple-interchange-newline"></span></div><span style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: = 12px;" class=3D""><br class=3D"Apple-interchange-newline"></span></div><br= class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"><br = class=3D"Apple-interchange-newline"> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Nov 7, 2016, at 2:10 AM, Sahina Bose <<a = href=3D"mailto:sabose@redhat.com" class=3D"">sabose@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><br class=3D""><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Mon, Nov 7, 2016 at 3:27 PM, = Gary Pedretty <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:gary@ravnalaska.net" target=3D"_blank" = class=3D"">gary@ravnalaska.net</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = style=3D"word-wrap:break-word" class=3D""><div = class=3D"">[root@fai-kvm-1-gfs admin]# gluster volume status = data2</div><div class=3D"">Status of volume: data2</div><div = class=3D"">Gluster process = TCP Port = RDMA Port Online Pid</div><div = class=3D"">------------------------------<wbr = class=3D"">------------------------------<wbr = class=3D"">------------------</div><div class=3D"">Brick <a = href=3D"http://fai-kvm-1-vmn.ravnalaska.net/" target=3D"_blank" = class=3D"">fai-kvm-1-vmn.ravnalaska.net</a>:/<wbr = class=3D"">kvm2/gl</div><div class=3D"">uster/data2/brick = = 49156 0 Y = 3484</div><div class=3D"">Brick <a = href=3D"http://fai-kvm-2-vmn.ravnalaska.net/" target=3D"_blank" = class=3D"">fai-kvm-2-vmn.ravnalaska.net</a>:/<wbr = class=3D"">kvm2/gl</div><div class=3D"">uster/data2/brick = = 49156 0 Y = 34791</div><div class=3D"">Brick <a = href=3D"http://fai-kvm-3-vmn.ravnalaska.net/" target=3D"_blank" = class=3D"">fai-kvm-3-vmn.ravnalaska.net</a>:/<wbr = class=3D"">kvm2/gl</div><div class=3D"">uster/data2/brick = = 49156 0 Y = 177340</div><div class=3D"">Brick <a = href=3D"http://fai-kvm-4-vmn.ravnalaska.net/" target=3D"_blank" = class=3D"">fai-kvm-4-vmn.ravnalaska.net</a>:/<wbr = class=3D"">kvm2/gl</div><div class=3D"">uster/data2/brick = = 49152 0 Y = 146038</div><div class=3D"">NFS Server on localhost = 2049 = 0 Y = 40844</div><div class=3D"">Self-heal Daemon on localhost = N/A N/A = Y 40865</div><div = class=3D"">NFS Server on <a href=3D"http://fai-kvm-2-gfs.ravnalaska.net/" = target=3D"_blank" class=3D"">fai-kvm-2-gfs.ravnalaska.net</a> 2049 = 0 Y = 99905</div><div class=3D"">Self-heal Daemon on = fai-kvm-2-gfs.ravnalask</div><div class=3D""><a href=3D"http://a.net/" = target=3D"_blank" class=3D"">a.net</a> = = N/A N/A = Y 99915</div><div class=3D"">NFS = Server on <a href=3D"http://fai-kvm-4-gfs.ravnalaska.net/" = target=3D"_blank" class=3D"">fai-kvm-4-gfs.ravnalaska.net</a> 2049 = 0 Y = 176305</div><div class=3D"">Self-heal Daemon on = fai-kvm-4-gfs.ravnalask</div><div class=3D""><a href=3D"http://a.net/" = target=3D"_blank" class=3D"">a.net</a> = = N/A N/A = Y 176326</div><div class=3D"">NFS= Server on <a href=3D"http://fai-kvm-3-gfs.ravnalaska.net/" = target=3D"_blank" class=3D"">fai-kvm-3-gfs.ravnalaska.net</a> 2049 = 0 Y = 226271</div><div class=3D"">Self-heal Daemon on = fai-kvm-3-gfs.ravnalask</div><div class=3D""><a href=3D"http://a.net/" = target=3D"_blank" class=3D"">a.net</a> = = N/A N/A = Y 226287</div><div class=3D""><br= class=3D""></div><div class=3D"">Task Status of Volume data2</div><div = class=3D"">------------------------------<wbr = class=3D"">------------------------------<wbr = class=3D"">------------------</div><div class=3D"">There are no active = volume tasks</div><div class=3D""><br class=3D""></div><div class=3D""><br= class=3D""></div><div class=3D""><div class=3D"">[root@fai-kvm-1-gfs = admin]# gluster volume info data2</div><div class=3D""><br = class=3D""></div><div class=3D"">Volume Name: data2</div><div = class=3D"">Type: Striped-Replicate</div><div class=3D"">Volume ID: = 20f85c9a-541b-4df4-9dba-<wbr class=3D"">44c5179bbfb0</div><div = class=3D"">Status: Started</div><div class=3D"">Number of Bricks: 1 x 2 = x 2 =3D 4</div><div class=3D"">Transport-type: tcp</div><div = class=3D"">Bricks:</div><div class=3D"">Brick1: <a = href=3D"http://fai-kvm-1-vmn.ravnalaska.net/" target=3D"_blank" = class=3D"">fai-kvm-1-vmn.ravnalaska.net</a>:/<wbr = class=3D"">kvm2/gluster/data2/brick</div><div class=3D"">Brick2: <a = href=3D"http://fai-kvm-2-vmn.ravnalaska.net/" target=3D"_blank" = class=3D"">fai-kvm-2-vmn.ravnalaska.net</a>:/<wbr = class=3D"">kvm2/gluster/data2/brick</div><div class=3D"">Brick3: <a = href=3D"http://fai-kvm-3-vmn.ravnalaska.net/" target=3D"_blank" = class=3D"">fai-kvm-3-vmn.ravnalaska.net</a>:/<wbr = class=3D"">kvm2/gluster/data2/brick</div><div class=3D"">Brick4: <a = href=3D"http://fai-kvm-4-vmn.ravnalaska.net/" target=3D"_blank" = class=3D"">fai-kvm-4-vmn.ravnalaska.net</a>:/<wbr = class=3D"">kvm2/gluster/data2/brick</div><div class=3D"">Options = Reconfigured:</div><div class=3D"">performance.readdir-ahead: = on</div><div class=3D"">performance.quick-read: off</div><div = class=3D"">performance.read-ahead: off</div><div class=3D""><a = href=3D"http://performance.io/" target=3D"_blank" = class=3D"">performance.io</a>-cache: off</div><div = class=3D"">performance.stat-prefetch: off</div><div = class=3D"">cluster.eager-lock: enable</div><div = class=3D"">network.remote-dio: enable</div><div = class=3D"">cluster.quorum-type: auto</div><div = class=3D"">cluster.server-quorum-type: server</div><div = class=3D"">storage.owner-uid: 36</div><div class=3D"">storage.owner-gid: = 36</div></div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D"">See attached file for the mount = log.</div></div></blockquote><div class=3D""><br class=3D""><br = class=3D""></div><div class=3D"">Striped-Replicate is no longer = supported in GlusterFS upstream. Instead, you should be using a = Distribute-Replicate with sharding enabled. Also when using a gluster = volume as storage domain, it is recommended to use replica 3.<br = class=3D""><br class=3D""></div><div class=3D"">=46rom the mount logs, = there is no indication as to why the volume is unmounted frequently. = Could you try again with a replica 3 volume that has sharding = enabled?<br class=3D""> <br class=3D""></div><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"><div = style=3D"word-wrap:break-word" class=3D""><span class=3D"HOEnZb"><font = color=3D"#888888" class=3D""><div class=3D""><br class=3D""></div><div = class=3D"">Gary</div><div class=3D""><br class=3D""></div><div = class=3D""></div></font></span></div><br class=3D""><div = style=3D"word-wrap:break-word" class=3D""><div class=3D""></div><div = class=3D""> <div style=3D"letter-spacing: normal; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; word-spacing: 0px; = word-wrap: break-word;" class=3D""><div style=3D"letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; word-wrap: break-word;" class=3D""><div = style=3D"text-align:start;text-indent:0px;word-wrap:break-word" = class=3D""><div = style=3D"text-align:start;text-indent:0px;word-wrap:break-word" = class=3D""><div = style=3D"text-align:start;text-indent:0px;word-wrap:break-word" = class=3D""><div = style=3D"text-align:start;text-indent:0px;word-wrap:break-word" = class=3D""><div = style=3D"text-align:start;text-indent:0px;word-wrap:break-word" = class=3D""><div = style=3D"text-align:start;text-indent:0px;word-wrap:break-word" = class=3D""><div = style=3D"text-align:start;text-indent:0px;word-wrap:break-word" = class=3D""><font style=3D"font-size: 12px; letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px;" = face=3D"Menlo" class=3D""><div = class=3D"">------------------------------<wbr = class=3D"">------------------------------<wbr = class=3D"">------------</div><div class=3D"">Gary Pedretty = = <a = href=3D"mailto:gary@eraalaska.net" target=3D"_blank" = class=3D"">gary@ravnalaska.net</a></div><div class=3D"">Systems Manager = = = <a href=3D"http://www.flyravn.com/" target=3D"_blank" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska = = /\ = 907-450-7251</div><div class=3D"">5245 Airport Industrial = Road / \/\ = 907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska = 99709 /\ / \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span style=3D"font-size:12px" class=3D"">Serving All of = Alaska / \/ /\ \ \/\ = =E2=80=9CLove your neighbor as</span></font><br = style=3D"font-family:Monaco" class=3D""><font style=3D"letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px;" = face=3D"Menlo" class=3D""><span style=3D"font-size:12px" class=3D"">Really= loving the record green up date! Summmer!! yourself=E2=80=9D = Matt 22:39</span></font><div style=3D"letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; = font-family: Menlo;" class=3D""></div><font style=3D"font-size:12px" = face=3D"Menlo" class=3D""></font><span style=3D"letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; font-size: = 12px;" class=3D""><font face=3D"Menlo" class=3D""><div = class=3D"">------------------------------<wbr = class=3D"">------------------------------<wbr = class=3D"">------------</div></font></span><div style=3D"letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px;" = class=3D""><font style=3D"font-size:12px" face=3D"Menlo" class=3D""><br = class=3D""></font></div></div><span style=3D"letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; font-size: = 12px;" class=3D""><br = class=3D"m_8434586956379529661Apple-interchange-newline"></span></div><spa= n style=3D"letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; font-size: 12px;" class=3D""><br = class=3D"m_8434586956379529661Apple-interchange-newline"></span></div><spa= n style=3D"letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; font-size: 12px;" class=3D""><br = class=3D"m_8434586956379529661Apple-interchange-newline"></span></div><br = class=3D"m_8434586956379529661Apple-interchange-newline"></div><br = class=3D"m_8434586956379529661Apple-interchange-newline"></div><br = class=3D"m_8434586956379529661Apple-interchange-newline"></div><br = class=3D"m_8434586956379529661Apple-interchange-newline"></div><br = class=3D"m_8434586956379529661Apple-interchange-newline"></div><br = class=3D"m_8434586956379529661Apple-interchange-newline"><br = class=3D"m_8434586956379529661Apple-interchange-newline"> </div> <br class=3D""><div class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Nov 6, 2016, at 9:50 PM, Sahina Bose <<a = href=3D"mailto:sabose@redhat.com" target=3D"_blank" = class=3D"">sabose@redhat.com</a>> wrote:</div><br = class=3D"m_8434586956379529661Apple-interchange-newline"><div = class=3D""><span = style=3D"font-family:LucidaGrande;font-size:13px;font-style:normal;font-va= riant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:star= t;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;= float:none;display:inline!important" class=3D"">However your volume = configuration seems suspect -"stripe 2 replica 2". Can you provide = gluster volume info of your second storage domain gluster volume? The = mount logs of the volume (under /var/log/glusterfs/rhev-<wbr = class=3D"">datacenter..<volname>.log) from the host where the = volume is being mounted will also help.</span><br = style=3D"font-family:LucidaGrande;font-size:13px;font-style:normal;font-va= riant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:star= t;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"= class=3D""><span = style=3D"font-family:LucidaGrande;font-size:13px;font-style:normal;font-va= riant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:star= t;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;= float:none;display:inline!important" = class=3D""> </span></div></blockquote></div><br class=3D""></div><br = class=3D""></blockquote></div><br class=3D""></div></div> </div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_5640DDB5-E478-43B7-AB26-61A2B8B4688F--
participants (3)
-
Gary Pedretty -
Maor Lipchuk -
Sahina Bose