[ovirt-devel] [HC] Weird issue while deploying hosted-engine

[Nir Soffer]

----- Original Message -----
> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> To: "Nir Soffer" <nsoffer at redhat.com>
> Cc: "David Teigland" <teigland at redhat.com>, "Yeela Kaplan" <ykaplan at redhat.com>, "Alon Bar-Lev" <alonbl at redhat.com>,
> "Allon Mureinik" <amureini at redhat.com>, "Yaniv Dary" <ydary at redhat.com>, "Federico Simoncelli"
> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>, "Yedidyah Bar David" <didi at redhat.com>, "Simone
> Tiraboschi" <stirabos at redhat.com>, devel at ovirt.org, "Michal Skrivanek" <mskrivan at redhat.com>
> Sent: Thursday, February 12, 2015 5:43:28 PM
> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
> 
> Il 12/02/2015 16:26, Nir Soffer ha scritto:
> > 
> > 
> > ----- Original Message -----
> >> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
> >> To: "Nir Soffer" <nsoffer at redhat.com>, "David Teigland"
> >> <teigland at redhat.com>
> >> Cc: "Yeela Kaplan" <ykaplan at redhat.com>, "Alon Bar-Lev"
> >> <alonbl at redhat.com>, "Allon Mureinik" <amureini at redhat.com>,
> >> "Yaniv Dary" <ydary at redhat.com>, "Federico Simoncelli"
> >> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>,
> >> "Yedidyah Bar David" <didi at redhat.com>, "Simone Tiraboschi"
> >> <stirabos at redhat.com>, devel at ovirt.org, "Michal
> >> Skrivanek" <mskrivan at redhat.com>
> >> Sent: Thursday, February 12, 2015 5:22:48 PM
> >> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
> >>
> >> Il 12/02/2015 16:20, Nir Soffer ha scritto:
> >>> ----- Original Message -----
> >>>> From: "David Teigland" <teigland at redhat.com>
> >>>> To: "Nir Soffer" <nsoffer at redhat.com>
> >>>> Cc: "Sandro Bonazzola" <sbonazzo at redhat.com>, "Yeela Kaplan"
> >>>> <ykaplan at redhat.com>, "Alon Bar-Lev"
> >>>> <alonbl at redhat.com>, "Allon Mureinik" <amureini at redhat.com>, "Yaniv
> >>>> Dary"
> >>>> <ydary at redhat.com>, "Federico Simoncelli"
> >>>> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>, "Yedidyah Bar
> >>>> David" <didi at redhat.com>, "Simone
> >>>> Tiraboschi" <stirabos at redhat.com>, devel at ovirt.org, "Michal Skrivanek"
> >>>> <mskrivan at redhat.com>
> >>>> Sent: Thursday, February 12, 2015 5:13:49 PM
> >>>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying
> >>>> hosted-engine
> >>>>
> >>>>>> 2015-02-12 13:08:25+0100 7254 [683]: open error -1
> >>>>>> /rhev/data-center/mnt/glusterSD/minidell.home:_hosted__engine__glusterfs/a7dba6e3-09ac-430a-8d6e-eea17cf18b8f/images/51156c24-5ea1-42df-bf33-daebf2c4780c/3719b0
> >>>>>> 06-68a2-4742-964c-413a3f77c0b4.lease
> >>>>
> >>>> open(2) error was EPERM
> >>>
> >>> selinux acting again?
> >>>
> >>> Sandro, can you attach also /var/log/audit/audit.log?
> >>
> >> Sure, but other than what I already reported in
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1191989
> >> I don't think you'll find much more.
> > 
> > Can you reproduce this in permissive mode?
> 
> Yes, it's fully reproducible on a host in permissive mode.
> 
> Also writing to the storage works:
> 
> while true; do echo starting dd `date`; dd if=/dev/zero of=/tmp/test/testfile
> bs=4k count=1k oflag=direct  ; echo dd finished `date`;  rm -f
> /tmp/test/testfile ; sleep 10; done

What does "df -T /tmp" show?

/tmp is typically tempfs, so your dd test never touched any storage.

Also, copying zeros can be ignored by smart storage, so better copy
"real" file data:

    dd if=/dev/urandom of=testfile bs=1M count=1

Now use testfile for writing to storage:

    dd if=testfile of=/path/to/glusterfs/mount/testfile bs=1M count=1 oflag=direct

You should do something like:

1. mount your gluster volume - same as vdsm does
2. start the write test, writing to the gluster storage
3. start your hosted engine flow
4. when it fails, keep your write test running until for a while

> 
> 
> didn't show any issue while sanlock is failing.
> 
> Attached the tar.gz of the ovirt-hosted-engine-setup I'm running and the full
> /var/log directory content.
> 
> Note that both glusterd and vdsmd are running on the same host for the
> hyper-converged setup.
> 
> > 
> >>>>
> >>>>>> 2015-02-12 13:08:25+0100 7254 [683]: r4 release_token open error -2
> >>>>
> >>>> open(2) error was ENOENT
> >>>>
> >>>>>> 2015-02-12 13:08:31+0100 7260 [16910]: a7dba6e3 aio collect 0
> >>>>>> 0x7f1d640008c0:0x7f1d640008d0:0x7f1d7ec52000 result -107:0 match res
> >>>>
> >>>> aio result on open file was ENOTCONN
> >>>>
> >>>> Maybe gluster logged some errors about what happened here?
> >>>>
> >>>>
> >>
> >>
> >> --
> >> Sandro Bonazzola
> >> Better technology. Faster innovation. Powered by community collaboration.
> >> See how it works at redhat.com
> >>
> 
> 
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
>