[ovirt-devel] [HC] Weird issue while deploying hosted-engine

Thu Feb 12 16:11:16 UTC 2015

Il 12/02/2015 17:03, Nir Soffer ha scritto:
> 
> 
> ----- Original Message -----
>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>> To: "Nir Soffer" <nsoffer at redhat.com>
>> Cc: "David Teigland" <teigland at redhat.com>, "Yeela Kaplan" <ykaplan at redhat.com>, "Alon Bar-Lev" <alonbl at redhat.com>,
>> "Allon Mureinik" <amureini at redhat.com>, "Yaniv Dary" <ydary at redhat.com>, "Federico Simoncelli"
>> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>, "Yedidyah Bar David" <didi at redhat.com>, "Simone
>> Tiraboschi" <stirabos at redhat.com>, devel at ovirt.org, "Michal Skrivanek" <mskrivan at redhat.com>
>> Sent: Thursday, February 12, 2015 5:43:28 PM
>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
>>
>> Il 12/02/2015 16:26, Nir Soffer ha scritto:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>> To: "Nir Soffer" <nsoffer at redhat.com>, "David Teigland"
>>>> <teigland at redhat.com>
>>>> Cc: "Yeela Kaplan" <ykaplan at redhat.com>, "Alon Bar-Lev"
>>>> <alonbl at redhat.com>, "Allon Mureinik" <amureini at redhat.com>,
>>>> "Yaniv Dary" <ydary at redhat.com>, "Federico Simoncelli"
>>>> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>,
>>>> "Yedidyah Bar David" <didi at redhat.com>, "Simone Tiraboschi"
>>>> <stirabos at redhat.com>, devel at ovirt.org, "Michal
>>>> Skrivanek" <mskrivan at redhat.com>
>>>> Sent: Thursday, February 12, 2015 5:22:48 PM
>>>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
>>>>
>>>> Il 12/02/2015 16:20, Nir Soffer ha scritto:
>>>>> ----- Original Message -----
>>>>>> From: "David Teigland" <teigland at redhat.com>
>>>>>> To: "Nir Soffer" <nsoffer at redhat.com>
>>>>>> Cc: "Sandro Bonazzola" <sbonazzo at redhat.com>, "Yeela Kaplan"
>>>>>> <ykaplan at redhat.com>, "Alon Bar-Lev"
>>>>>> <alonbl at redhat.com>, "Allon Mureinik" <amureini at redhat.com>, "Yaniv
>>>>>> Dary"
>>>>>> <ydary at redhat.com>, "Federico Simoncelli"
>>>>>> <fsimonce at redhat.com>, "Sahina Bose" <sabose at redhat.com>, "Yedidyah Bar
>>>>>> David" <didi at redhat.com>, "Simone
>>>>>> Tiraboschi" <stirabos at redhat.com>, devel at ovirt.org, "Michal Skrivanek"
>>>>>> <mskrivan at redhat.com>
>>>>>> Sent: Thursday, February 12, 2015 5:13:49 PM
>>>>>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying
>>>>>> hosted-engine
>>>>>>
>>>>>>>> 2015-02-12 13:08:25+0100 7254 [683]: open error -1
>>>>>>>> /rhev/data-center/mnt/glusterSD/minidell.home:_hosted__engine__glusterfs/a7dba6e3-09ac-430a-8d6e-eea17cf18b8f/images/51156c24-5ea1-42df-bf33-daebf2c4780c/3719b0
>>>>>>>> 06-68a2-4742-964c-413a3f77c0b4.lease
>>>>>>
>>>>>> open(2) error was EPERM
>>>>>
>>>>> selinux acting again?
>>>>>
>>>>> Sandro, can you attach also /var/log/audit/audit.log?
>>>>
>>>> Sure, but other than what I already reported in
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1191989
>>>> I don't think you'll find much more.
>>>
>>> Can you reproduce this in permissive mode?
>>
>> Yes, it's fully reproducible on a host in permissive mode.
>>
>> Also writing to the storage works:
>>
>> while true; do echo starting dd `date`; dd if=/dev/zero of=/tmp/test/testfile
>> bs=4k count=1k oflag=direct  ; echo dd finished `date`;  rm -f
>> /tmp/test/testfile ; sleep 10; done
> 
> What does "df -T /tmp" show?
> 

# df -T /tmp /tmp/test/
File system                            Tipo           1K-blocchi   Usati Disponib. Uso% Montato su
/dev/sda6                              ext4             40185112 5347916  32772812  15% /
192.168.1.107:/hosted_engine_glusterfs fuse.glusterfs   40185088 5347968  32772736  15% /tmp/test

> /tmp is typically tempfs, so your dd test never touched any storage.
> 
> Also, copying zeros can be ignored by smart storage, so better copy
> "real" file data:
> 
>     dd if=/dev/urandom of=testfile bs=1M count=1
> 
> Now use testfile for writing to storage:
> 
>     dd if=testfile of=/path/to/glusterfs/mount/testfile bs=1M count=1 oflag=direct
> 
> You should do something like:
> 
> 1. mount your gluster volume - same as vdsm does
> 2. start the write test, writing to the gluster storage
> 3. start your hosted engine flow
> 4. when it fails, keep your write test running until for a while

If you prefer I can also do a double test, one on /tmp/test and one on /rhev/data-center/mnt/glusterSD/mididell.home\:_hosted__engine__glusterfs/
I'll try to do the test tomorrow.

> 
>>
>>
>> didn't show any issue while sanlock is failing.
>>
>> Attached the tar.gz of the ovirt-hosted-engine-setup I'm running and the full
>> /var/log directory content.
>>
>> Note that both glusterd and vdsmd are running on the same host for the
>> hyper-converged setup.
>>
>>>
>>>>>>
>>>>>>>> 2015-02-12 13:08:25+0100 7254 [683]: r4 release_token open error -2
>>>>>>
>>>>>> open(2) error was ENOENT
>>>>>>
>>>>>>>> 2015-02-12 13:08:31+0100 7260 [16910]: a7dba6e3 aio collect 0
>>>>>>>> 0x7f1d640008c0:0x7f1d640008d0:0x7f1d7ec52000 result -107:0 match res
>>>>>>
>>>>>> aio result on open file was ENOTCONN
>>>>>>
>>>>>> Maybe gluster logged some errors about what happened here?
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Sandro Bonazzola
>>>> Better technology. Faster innovation. Powered by community collaboration.
>>>> See how it works at redhat.com
>>>>
>>
>>
>> --
>> Sandro Bonazzola
>> Better technology. Faster innovation. Powered by community collaboration.
>> See how it works at redhat.com
>>

-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com