[ovirt-devel] [HC] Weird issue while deploying hosted-engine

Thu Feb 12 16:20:42 UTC 2015

----- Original Message -----
> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> To: "Nir Soffer" <nsoffer at redhat.com>
> Cc: "David Teigland" <teigland at redhat.com>, "Yeela Kaplan" <ykaplan at redhat.com>, "Alon Bar-Lev" <alonbl at redhat.com>,
> "Allon Mureinik" <amureini at redhat.com>, "Yaniv Dary" <ydary at redhat.com>, "Federico Simoncelli"
> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>, "Yedidyah Bar David" <didi at redhat.com>, "Simone
> Tiraboschi" <stirabos at redhat.com>, devel at ovirt.org, "Michal Skrivanek" <mskrivan at redhat.com>
> Sent: Thursday, February 12, 2015 6:11:16 PM
> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
> 
> Il 12/02/2015 17:03, Nir Soffer ha scritto:
> > 
> > 
> > ----- Original Message -----
> >> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >> To: "Nir Soffer" <nsoffer at redhat.com>
> >> Cc: "David Teigland" <teigland at redhat.com>, "Yeela Kaplan"
> >> <ykaplan at redhat.com>, "Alon Bar-Lev" <alonbl at redhat.com>,
> >> "Allon Mureinik" <amureini at redhat.com>, "Yaniv Dary" <ydary at redhat.com>,
> >> "Federico Simoncelli"
> >> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>, "Yedidyah Bar
> >> David" <didi at redhat.com>, "Simone
> >> Tiraboschi" <stirabos at redhat.com>, devel at ovirt.org, "Michal Skrivanek"
> >> <mskrivan at redhat.com>
> >> Sent: Thursday, February 12, 2015 5:43:28 PM
> >> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
> >>
> >> Il 12/02/2015 16:26, Nir Soffer ha scritto:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >>>> To: "Nir Soffer" <nsoffer at redhat.com>, "David Teigland"
> >>>> <teigland at redhat.com>
> >>>> Cc: "Yeela Kaplan" <ykaplan at redhat.com>, "Alon Bar-Lev"
> >>>> <alonbl at redhat.com>, "Allon Mureinik" <amureini at redhat.com>,
> >>>> "Yaniv Dary" <ydary at redhat.com>, "Federico Simoncelli"
> >>>> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>,
> >>>> "Yedidyah Bar David" <didi at redhat.com>, "Simone Tiraboschi"
> >>>> <stirabos at redhat.com>, devel at ovirt.org, "Michal
> >>>> Skrivanek" <mskrivan at redhat.com>
> >>>> Sent: Thursday, February 12, 2015 5:22:48 PM
> >>>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying
> >>>> hosted-engine
> >>>>
> >>>> Il 12/02/2015 16:20, Nir Soffer ha scritto:
> >>>>> ----- Original Message -----
> >>>>>> From: "David Teigland" <teigland at redhat.com>
> >>>>>> To: "Nir Soffer" <nsoffer at redhat.com>
> >>>>>> Cc: "Sandro Bonazzola" <sbonazzo at redhat.com>, "Yeela Kaplan"
> >>>>>> <ykaplan at redhat.com>, "Alon Bar-Lev"
> >>>>>> <alonbl at redhat.com>, "Allon Mureinik" <amureini at redhat.com>, "Yaniv
> >>>>>> Dary"
> >>>>>> <ydary at redhat.com>, "Federico Simoncelli"
> >>>>>> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>, "Yedidyah
> >>>>>> Bar
> >>>>>> David" <didi at redhat.com>, "Simone
> >>>>>> Tiraboschi" <stirabos at redhat.com>, devel at ovirt.org, "Michal Skrivanek"
> >>>>>> <mskrivan at redhat.com>
> >>>>>> Sent: Thursday, February 12, 2015 5:13:49 PM
> >>>>>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying
> >>>>>> hosted-engine
> >>>>>>
> >>>>>>>> 2015-02-12 13:08:25+0100 7254 [683]: open error -1
> >>>>>>>> /rhev/data-center/mnt/glusterSD/minidell.home:_hosted__engine__glusterfs/a7dba6e3-09ac-430a-8d6e-eea17cf18b8f/images/51156c24-5ea1-42df-bf33-daebf2c4780c/3719b0
> >>>>>>>> 06-68a2-4742-964c-413a3f77c0b4.lease
> >>>>>>
> >>>>>> open(2) error was EPERM
> >>>>>
> >>>>> selinux acting again?
> >>>>>
> >>>>> Sandro, can you attach also /var/log/audit/audit.log?
> >>>>
> >>>> Sure, but other than what I already reported in
> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1191989
> >>>> I don't think you'll find much more.
> >>>
> >>> Can you reproduce this in permissive mode?
> >>
> >> Yes, it's fully reproducible on a host in permissive mode.
> >>
> >> Also writing to the storage works:
> >>
> >> while true; do echo starting dd `date`; dd if=/dev/zero
> >> of=/tmp/test/testfile
> >> bs=4k count=1k oflag=direct  ; echo dd finished `date`;  rm -f
> >> /tmp/test/testfile ; sleep 10; done
> > 
> > What does "df -T /tmp" show?
> > 
> 
> 
> # df -T /tmp /tmp/test/
> File system                            Tipo           1K-blocchi   Usati
> Disponib. Uso% Montato su
> /dev/sda6                              ext4             40185112 5347916
> 32772812  15% /
> 192.168.1.107:/hosted_engine_glusterfs fuse.glusterfs   40185088 5347968
> 32772736  15% /tmp/test
> 
> 
> > /tmp is typically tempfs, so your dd test never touched any storage.
> > 
> > Also, copying zeros can be ignored by smart storage, so better copy
> > "real" file data:
> > 
> >     dd if=/dev/urandom of=testfile bs=1M count=1
> > 
> > Now use testfile for writing to storage:
> > 
> >     dd if=testfile of=/path/to/glusterfs/mount/testfile bs=1M count=1
> >     oflag=direct
> > 
> > You should do something like:
> > 
> > 1. mount your gluster volume - same as vdsm does
> > 2. start the write test, writing to the gluster storage
> > 3. start your hosted engine flow
> > 4. when it fails, keep your write test running until for a while
> 
> If you prefer I can also do a double test, one on /tmp/test and one on
> /rhev/data-center/mnt/glusterSD/mididell.home\:_hosted__engine__glusterfs/
> I'll try to do the test tomorrow.

Writing to /tmp/test seems ok. It will be nice to see a test using random
data instead of zeros.

I think David should continue to investigate this to understand why
sanlock cannot access storage.

> >> didn't show any issue while sanlock is failing.
> >>
> >> Attached the tar.gz of the ovirt-hosted-engine-setup I'm running and the
> >> full
> >> /var/log directory content.
> >>
> >> Note that both glusterd and vdsmd are running on the same host for the
> >> hyper-converged setup.
> >>
> >>>
> >>>>>>
> >>>>>>>> 2015-02-12 13:08:25+0100 7254 [683]: r4 release_token open error -2
> >>>>>>
> >>>>>> open(2) error was ENOENT
> >>>>>>
> >>>>>>>> 2015-02-12 13:08:31+0100 7260 [16910]: a7dba6e3 aio collect 0
> >>>>>>>> 0x7f1d640008c0:0x7f1d640008d0:0x7f1d7ec52000 result -107:0 match res
> >>>>>>
> >>>>>> aio result on open file was ENOTCONN
> >>>>>>
> >>>>>> Maybe gluster logged some errors about what happened here?
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sandro Bonazzola
> >>>> Better technology. Faster innovation. Powered by community
> >>>> collaboration.
> >>>> See how it works at redhat.com
> >>>>
> >>
> >>
> >> --
> >> Sandro Bonazzola
> >> Better technology. Faster innovation. Powered by community collaboration.
> >> See how it works at redhat.com
> >>
> 
> 
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
> 