<div dir="ltr"><div>Hi thanks for the reply. Here is an extract from a grep I ran on the vdsm log grepping for the volume name vm-store. It seems to indicate the bricks are ONLINE.</div><div><br></div><div>I am uncertain how to extract meaningful information from the engine.log can you provide some guidance?</div>
<div><br></div><div>Thanks, </div><div><br></div><div>Alastair</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Thread-100::DEBUG::2014-05-27 15:01:06,335::BindingXMLRPC::1067::vds::(wrapper) client [129.174.94.239]::call volumeStatus with ('vm-store', '', '') {}<br>Thread-100::DEBUG::2014-05-27 15:01:06,356::BindingXMLRPC::1074::vds::(wrapper) return volumeStatus with {'volumeStatus': {'bricks': [{'status': 'ONLINE', 'brick': 'gluster0:/export/brick0', 'pid': '2675', 'port': '49158', 'hostuuid': 'bcff5245-ea86-4384-a1bf-9219c8be8001'}, {'status': 'ONLINE', 'brick': 'gluster1:/export/brick4/vm-store', 'pid': '2309', 'port': '49158', 'hostuuid': '54d39ae4-91ae-410b-828c-67031f3d8a68'}], 'nfs': [{'status': 'ONLINE', 'hostname': '129.174.126.56', 'pid': '27012', 'port': '2049', 'hostuuid': '54d39ae4-91ae-410b-828c-67031f3d8a68'}, {'status': 'ONLINE', 'hostname': 'gluster0', 'pid': '12875', 'port': '2049', 'hostuuid': 'bcff5245-ea86-4384-a1bf-9219c8be8001'}], 'shd': [{'status': 'ONLINE', 'hostname': '129.174.126.56', 'pid': '27019', 'hostuuid': '54d39ae4-91ae-410b-828c-67031f3d8a68'}, {'status': 'ONLINE', 'hostname': 'gluster0', 'pid': '12882', 'hostuuid': 'bcff5245-ea86-4384-a1bf-9219c8be8001'}], 'name': 'vm-store'}, 'status': {'message': 'Done', 'code': 0}}<br>
Thread-16::DEBUG::2014-05-27 15:01:15,339::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:01:25,381::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:01:35,423::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:01:45,465::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:01:55,507::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:02:05,549::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:02:15,590::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:02:25,657::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:02:35,698::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:02:45,740::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:02:55,784::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:03:05,827::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:03:15,869::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:03:25,910::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:03:35,953::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:03:45,996::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:03:56,037::fileSD::225::Storage.Misc.excCmd::(getR7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>Thread-16::DEBUG::2014-05-27 15:04:06,078::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:04:16,107::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8<br>Thread-16::DEBUG::2014-05-27 15:04:16,126::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=Gluster-VM-Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=VS-VM', 'POOL_DOMAINS=6d637c7f-a4ab-4510-a0d9-63a04c55d6d8:Active,6d1e2f10-e6ec-42ce-93d5-ee93e8eeeb10:Active', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=7', 'POOL_UUID=9a0b5f4a-4a0f-432c-b70c-53fd5643cbb7', 'REMOTE_PATH=gluster0:vm-store', 'ROLE=Master', 'SDUUID=6d637c7f-a4ab-4510-a0d9-63a04c55d6d8', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=8e747f0ebf360f1db6801210c574405dd71fe731']<br>
Thread-16::DEBUG::2014-05-27 15:04:16,153::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:04:26,196::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd None)<br>
Thread-16::DEBUG::2014-05-27 15:04:36,238::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata bs=4096 count=1' (cwd eadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637cNone)</blockquote>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 21 May 2014 23:51, Kanagaraj <span dir="ltr"><<a href="mailto:kmayilsa@redhat.com" target="_blank">kmayilsa@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
engine.log and vdsm.log?<br>
<br>
This can mostly happen due to following reasons<br>
- "gluster volume status vm-store" is not consistently returning the
right output<br>
- ovirt-engine is not able to identify the bricks properly<br>
<br>
Anyway, engine.log will give better clarity.<div><div class="h5"><br>
<br>
<br>
<div>On 05/22/2014 02:24 AM, Alastair Neil
wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="h5">
<div dir="ltr">I just did a rolling upgrade of my gluster storage
cluster to the latest 3.5 bits. This all seems to have gone
smoothly and all the volumes are on line. All volumes are
replicated 1x2
<div><br>
</div>
<div>The ovirt console now insists that two of my volumes ,
including the vm-store volume with my vm's happily running
have no bricks up.</div>
<div><br>
</div>
<div>It reports "Up but all bricks are down"</div>
<div><br>
</div>
<div>This would seem to be impossible. Gluster on the nodes
itself reports no issues</div>
<div><br>
</div>
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[root@gluster1
~]# gluster volume status vm-store<br>
Status of volume: vm-store<br>
Gluster process<span style="white-space:pre-wrap"> </span>Port<span style="white-space:pre-wrap"> </span>Online<span style="white-space:pre-wrap"> </span>Pid<br>
------------------------------------------------------------------------------<br>
Brick gluster0:/export/brick0/vm-store<span style="white-space:pre-wrap"> </span>49158<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>2675<br>
Brick gluster1:/export/brick4/vm-store<span style="white-space:pre-wrap"> </span>49158<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>2309<br>
NFS Server on localhost<span style="white-space:pre-wrap"> </span>2049<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>27012<br>
Self-heal Daemon on localhost<span style="white-space:pre-wrap"> </span>N/A<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>27019<br>
NFS Server on gluster0<span style="white-space:pre-wrap">
</span>2049<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>12875<br>
Self-heal Daemon on gluster0<span style="white-space:pre-wrap"> </span>N/A<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>12882<br>
<br>
Task Status of Volume vm-store<br>
------------------------------------------------------------------------------<br>
There are no active volume tasks</blockquote>
</div>
<div><br>
</div>
<div><br>
</div>
<div>As I mentioned the vms are running happily</div>
<div>
initially the ISOs volume had the same issue. I did a volume
start and stop on the volume as it was not being activly used
and that cleared up the issue in the console. However, as I
have VMs running I can't so this for the vm-store volume. </div>
<div><br>
</div>
<div><br>
</div>
<div>Any suggestions, Alastair</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><pre>_______________________________________________
Users mailing list
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
</blockquote></div><br></div>