Re: [ovirt-users] post glusterfs 3.4 -> 3.5 upgrade issue in ovirt (3.4.0-1.fc19): bricks unavailable

Wednesday, 28 May 2014

This is a multi-part message in MIME format.
--------------010008050006020009010501
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Alastair,

This could be a mismatch in the hostname identified in ovirt and gluster.

You could check for any exceptions from GlusterSyncJob in engine.log.

Also, what version of ovirt are you using. And the compatibility version 
of your cluster?

On 05/28/2014 12:40 AM, Alastair Neil wrote:
...
 Hi thanks for the reply. Here is an extract from a grep I ran on the

 vdsm log grepping for the volume name vm-store.  It seems to indicate 
 the bricks are ONLINE.

 I am uncertain how to extract meaningful information from the 
 engine.log can you provide some guidance?

 Thanks,

 Alastair

     Thread-100::DEBUG::2014-05-27
     15:01:06,335::BindingXMLRPC::1067::vds::(wrapper) client
     [129.174.94.239]::call volumeStatus with ('vm-store', '', '')
{}
     Thread-100::DEBUG::2014-05-27
     15:01:06,356::BindingXMLRPC::1074::vds::(wrapper) return
     volumeStatus with {'volumeStatus': {'bricks': [{'status':
     'ONLINE', 'brick': 'gluster0:/export/brick0', 'pid':
'2675',
     'port': '49158', 'hostuuid':
     'bcff5245-ea86-4384-a1bf-9219c8be8001'}, {'status':
'ONLINE',
     'brick': 'gluster1:/export/brick4/vm-store', 'pid':
'2309',
     'port': '49158', 'hostuuid':
     '54d39ae4-91ae-410b-828c-67031f3d8a68'}], 'nfs': [{'status':
     'ONLINE', 'hostname': '129.174.126.56', 'pid':
'27012', 'port':
     '2049', 'hostuuid': '54d39ae4-91ae-410b-828c-67031f3d8a68'},
     {'status': 'ONLINE', 'hostname': 'gluster0',
'pid': '12875',
     'port': '2049', 'hostuuid':
     'bcff5245-ea86-4384-a1bf-9219c8be8001'}], 'shd': [{'status':
     'ONLINE', 'hostname': '129.174.126.56', 'pid':
'27019',
     'hostuuid': '54d39ae4-91ae-410b-828c-67031f3d8a68'},
{'status':
     'ONLINE', 'hostname': 'gluster0', 'pid':
'12882', 'hostuuid':
     'bcff5245-ea86-4384-a1bf-9219c8be8001'}], 'name':
'vm-store'},
     'status': {'message': 'Done', 'code': 0}}
     Thread-16::DEBUG::2014-05-27
     15:01:15,339::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:01:25,381::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:01:35,423::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:01:45,465::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:01:55,507::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:02:05,549::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:02:15,590::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:02:25,657::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:02:35,698::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:02:45,740::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:02:55,784::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:03:05,827::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:03:15,869::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:03:25,910::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:03:35,953::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:03:45,996::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27

15:03:56,037::fileSD::225::Storage.Misc.excCmd::(getR7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:04:06,078::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:04:16,107::fileSD::140::Storage.StorageDomain::(__init__)
     Reading domain in path

/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8
     Thread-16::DEBUG::2014-05-27
     15:04:16,126::persistentDict::234::Storage.PersistentDict::(refresh)
     read lines (FileMetadataRW)=['CLASS=Data',
     'DESCRIPTION=Gluster-VM-Store', 'IOOPTIMEOUTSEC=10',
     'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
     'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1',
     'POOL_DESCRIPTION=VS-VM',

'POOL_DOMAINS=6d637c7f-a4ab-4510-a0d9-63a04c55d6d8:Active,6d1e2f10-e6ec-42ce-93d5-ee93e8eeeb10:Active',
     'POOL_SPM_ID=3', 'POOL_SPM_LVER=7',
     'POOL_UUID=9a0b5f4a-4a0f-432c-b70c-53fd5643cbb7',
     'REMOTE_PATH=gluster0:vm-store', 'ROLE=Master',
     'SDUUID=6d637c7f-a4ab-4510-a0d9-63a04c55d6d8', 'TYPE=GLUSTERFS',
     'VERSION=3', '_SHA_CKSUM=8e747f0ebf360f1db6801210c574405dd71fe731']
     Thread-16::DEBUG::2014-05-27
     15:04:16,153::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:04:26,196::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd None)
     Thread-16::DEBUG::2014-05-27
     15:04:36,238::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
     '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
     bs=4096 count=1' (cwd eadDelay) '/bin/dd iflag=direct
     if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637cNone)

 On 21 May 2014 23:51, Kanagaraj <kmayilsa(a)redhat.com 
 <mailto:kmayilsa@redhat.com>> wrote:

     engine.log and vdsm.log?

     This can mostly happen due to following reasons
     - "gluster volume status vm-store" is not consistently returning
     the right output
     - ovirt-engine is not able to identify the bricks properly

     Anyway, engine.log will give better clarity.

     On 05/22/2014 02:24 AM, Alastair Neil wrote:
>     I just did a rolling upgrade of my gluster storage cluster to the
>     latest 3.5 bits.  This all seems to have gone smoothly and all
>     the volumes are on line.  All volumes are replicated 1x2
>
>     The ovirt console now insists that two of my volumes , including
>     the vm-store  volume with my vm's happily running have no bricks up.
>
>     It reports "Up but all bricks are down"
>
>     This would seem to be impossible.  Gluster  on the nodes itself
>     reports no issues
>
>         [root@gluster1 ~]# gluster volume status vm-store
>         Status of volume: vm-store
>         Gluster processPortOnlinePid
>        
------------------------------------------------------------------------------
>         Brick gluster0:/export/brick0/vm-store49158Y2675
>         Brick gluster1:/export/brick4/vm-store49158Y2309
>         NFS Server on localhost2049Y27012
>         Self-heal Daemon on localhostN/AY27019
>         NFS Server on gluster02049Y12875
>         Self-heal Daemon on gluster0N/AY12882
>
>         Task Status of Volume vm-store
>        
------------------------------------------------------------------------------
>         There are no active volume tasks
>
>
>
>     As I mentioned the vms are running happily
>     initially the ISOs volume had the same issue.  I did a volume
>     start and stop on the volume as it was not being activly used and
>     that cleared up the issue in the console.  However, as I have VMs
>     running  I can't so this for the vm-store volume.
>
>
>     Any suggestions, Alastair
>
>
>
>     _______________________________________________
>     Users mailing list
>     Users(a)ovirt.org  <mailto:Users@ovirt.org>
>     http://lists.ovirt.org/mailman/listinfo/users

 _______________________________________________
 Users mailing list
 Users(a)ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users 

--------------010008050006020009010501
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Hi Alastair,<br>
    <br>
    This could be a mismatch in the hostname identified in ovirt and
    gluster.<br>
    <br>
    You could check for any exceptions from GlusterSyncJob in
    engine.log.<br>
    <br>
    Also, what version of ovirt are you using. And the compatibility
    version of your cluster?<br>
    <br>
    <div class="moz-cite-prefix">On 05/28/2014 12:40 AM, Alastair Neil
      wrote:<br>
    </div>
    <blockquote
cite="mid:CA+SarwqGVfY_CNo0M9FGxCL==bzVdqsV-EdXeqrhaYkRPAqMgg@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>Hi thanks for the reply. Here is an extract from a grep I
          ran on the vdsm log grepping for the volume name vm-store. &nbsp;It
          seems to indicate the bricks are ONLINE.</div>
        <div><br>
        </div>
        <div>I am uncertain how to extract meaningful information from
          the engine.log can you provide some guidance?</div>
        <div><br>
        </div>
        <div>Thanks,&nbsp;</div>
        <div><br>
        </div>
        <div>Alastair</div>
        <div><br>
        </div>
        <div>&nbsp;</div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Thread-100::DEBUG::2014-05-27
          15:01:06,335::BindingXMLRPC::1067::vds::(wrapper) client
          [129.174.94.239]::call volumeStatus with ('vm-store', '',
'')
          {}<br>
          Thread-100::DEBUG::2014-05-27
          15:01:06,356::BindingXMLRPC::1074::vds::(wrapper) return
          volumeStatus with {'volumeStatus': {'bricks':
[{'status':
          'ONLINE', 'brick': 'gluster0:/export/brick0',
'pid': '2675',
          'port': '49158', 'hostuuid':
          'bcff5245-ea86-4384-a1bf-9219c8be8001'}, {'status':
'ONLINE',
          'brick': 'gluster1:/export/brick4/vm-store', 'pid':
'2309',
          'port': '49158', 'hostuuid':
          '54d39ae4-91ae-410b-828c-67031f3d8a68'}], 'nfs':
[{'status':
          'ONLINE', 'hostname': '129.174.126.56', 'pid':
'27012',
          'port': '2049', 'hostuuid':
          '54d39ae4-91ae-410b-828c-67031f3d8a68'}, {'status':
'ONLINE',
          'hostname': 'gluster0', 'pid': '12875',
'port': '2049',
          'hostuuid': 'bcff5245-ea86-4384-a1bf-9219c8be8001'}],
'shd':
          [{'status': 'ONLINE', 'hostname':
'129.174.126.56', 'pid':
          '27019', 'hostuuid':
'54d39ae4-91ae-410b-828c-67031f3d8a68'},
          {'status': 'ONLINE', 'hostname': 'gluster0',
'pid': '12882',
          'hostuuid': 'bcff5245-ea86-4384-a1bf-9219c8be8001'}],
'name':
          'vm-store'}, 'status': {'message': 'Done',
'code': 0}}<br>
          Thread-16::DEBUG::2014-05-27
          15:01:15,339::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:01:25,381::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:01:35,423::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:01:45,465::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:01:55,507::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:02:05,549::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:02:15,590::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:02:25,657::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:02:35,698::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:02:45,740::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:02:55,784::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:03:05,827::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:03:15,869::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:03:25,910::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:03:35,953::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:03:45,996::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27

15:03:56,037::fileSD::225::Storage.Misc.excCmd::(getR7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:04:06,078::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:04:16,107::fileSD::140::Storage.StorageDomain::(__init__)
          Reading domain in path
/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8<br>
          Thread-16::DEBUG::2014-05-27
          15:04:16,126::persistentDict::234::Storage.PersistentDict::(refresh)
          read lines (FileMetadataRW)=['CLASS=Data',
          'DESCRIPTION=Gluster-VM-Store', 'IOOPTIMEOUTSEC=10',
          'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
          'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1',
          'POOL_DESCRIPTION=VS-VM',

'POOL_DOMAINS=6d637c7f-a4ab-4510-a0d9-63a04c55d6d8:Active,6d1e2f10-e6ec-42ce-93d5-ee93e8eeeb10:Active',
          'POOL_SPM_ID=3', 'POOL_SPM_LVER=7',
          'POOL_UUID=9a0b5f4a-4a0f-432c-b70c-53fd5643cbb7',
          'REMOTE_PATH=gluster0:vm-store', 'ROLE=Master',
          'SDUUID=6d637c7f-a4ab-4510-a0d9-63a04c55d6d8',
          'TYPE=GLUSTERFS', 'VERSION=3',
          '_SHA_CKSUM=8e747f0ebf360f1db6801210c574405dd71fe731']<br>
          Thread-16::DEBUG::2014-05-27
          15:04:16,153::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:04:26,196::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd None)<br>
          Thread-16::DEBUG::2014-05-27
          15:04:36,238::fileSD::225::Storage.Misc.excCmd::(getReadDelay)
          '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637c7f-a4ab-4510-a0d9-63a04c55d6d8/dom_md/metadata
          bs=4096 count=1' (cwd&nbsp;eadDelay) '/bin/dd iflag=direct

if=/rhev/data-center/mnt/glusterSD/gluster0:vm-store/6d637cNone)</blockquote>
        <div><br>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <br>
        <div class="gmail_quote">On 21 May 2014 23:51, Kanagaraj <span
            dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:kmayilsa@redhat.com"
target=&quot;_blank&quot;&gt;kmayilsa(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000"> engine.log
and
              vdsm.log?<br>
              <br>
              This can mostly happen due to following reasons<br>
              - "gluster volume status vm-store" is not consistently
              returning the right output<br>
              - ovirt-engine is not able to identify the bricks properly<br>
              <br>
              Anyway, engine.log will give better clarity.
              <div>
                <div class="h5"><br>
                  <br>
                  <br>
                  <div>On 05/22/2014 02:24 AM, Alastair Neil wrote:<br>
                  </div>
                </div>
              </div>
              <blockquote type="cite">
                <div>
                  <div class="h5">
                    <div dir="ltr">I just did a rolling upgrade of my
                      gluster storage cluster to the latest 3.5 bits.
                      &nbsp;This all seems to have gone smoothly and all the
                      volumes are on line. &nbsp;All volumes are replicated
                      1x2
                      <div><br>
                      </div>
                      <div>The ovirt console now insists that two of my
                        volumes , including the vm-store &nbsp;volume with my
                        vm's happily running have no bricks up.</div>
                      <div><br>
                      </div>
                      <div>It reports "Up but all bricks are
down"</div>
                      <div><br>
                      </div>
                      <div>This would seem to be impossible. &nbsp;Gluster
                        &nbsp;on the nodes itself reports no issues</div>
                      <div><br>
                      </div>
                      <div>
                        <blockquote class="gmail_quote"
                          style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[root@gluster1

                          ~]# gluster volume status vm-store<br>
                          Status of volume: vm-store<br>
                          Gluster process<span
                            style="white-space:pre-wrap">
</span>Port<span
                            style="white-space:pre-wrap">
</span>Online<span
                            style="white-space:pre-wrap">
</span>Pid<br>
------------------------------------------------------------------------------<br>
                          Brick gluster0:/export/brick0/vm-store<span
                            style="white-space:pre-wrap">
</span>49158<span
                            style="white-space:pre-wrap">
</span>Y<span
                            style="white-space:pre-wrap">
</span>2675<br>
                          Brick gluster1:/export/brick4/vm-store<span
                            style="white-space:pre-wrap">
</span>49158<span
                            style="white-space:pre-wrap">
</span>Y<span
                            style="white-space:pre-wrap">
</span>2309<br>
                          NFS Server on localhost<span
                            style="white-space:pre-wrap">
</span>2049<span
                            style="white-space:pre-wrap">
</span>Y<span
                            style="white-space:pre-wrap">
</span>27012<br>
                          Self-heal Daemon on localhost<span
                            style="white-space:pre-wrap">
</span>N/A<span
                            style="white-space:pre-wrap">
</span>Y<span
                            style="white-space:pre-wrap">
</span>27019<br>
                          NFS Server on gluster0<span
                            style="white-space:pre-wrap">
</span>2049<span
                            style="white-space:pre-wrap">
</span>Y<span
                            style="white-space:pre-wrap">
</span>12875<br>
                          Self-heal Daemon on gluster0<span
                            style="white-space:pre-wrap">
</span>N/A<span
                            style="white-space:pre-wrap">
</span>Y<span
                            style="white-space:pre-wrap">
</span>12882<br>
                          &nbsp;<br>
                          Task Status of Volume vm-store<br>
------------------------------------------------------------------------------<br>
                          There are no active volume tasks</blockquote>
                      </div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                      <div>As I mentioned the vms are running happily</div>
                      <div> initially the ISOs volume had the same
                        issue. &nbsp;I did a volume start and stop on the
                        volume as it was not being activly used and that
                        cleared up the issue in the console. &nbsp;However,
                        as I have VMs running &nbsp;I can't so this for the
                        vm-store volume. &nbsp;</div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                      <div>Any suggestions, Alastair</div>
                      <div><br>
                      </div>
                    </div>
                    <br>
                    <fieldset></fieldset>
                    <br>
                  </div>
                </div>
                <pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org"
target=&quot;_blank&quot;&gt;Users(a)ovirt.org&lt;/a&gt;
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
</pre>
              </blockquote>
              <br>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
    </blockquote>
    <br>
  </body>
</html>

--------------010008050006020009010501--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] post glusterfs 3.4 -> 3.5 upgrade issue in ovirt (3.4.0-1.fc19): bricks unavailable