11:20 a.m.
----- Original Message -----
From: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
To: "Nir Soffer" <nsoffer(a)redhat.com>
Cc: "David Teigland" <teigland(a)redhat.com>, "Yeela Kaplan"
<ykaplan(a)redhat.com>, "Alon Bar-Lev" <alonbl(a)redhat.com>,
"Allon Mureinik" <amureini(a)redhat.com>, "Yaniv Dary"
<ydary(a)redhat.com>, "Federico Simoncelli"
<fsimonce(a)redhat.com>, "Sahina Bose" <sabose(a)redhat.com>,
"Yedidyah Bar David" <didi(a)redhat.com>, "Simone
Tiraboschi" <stirabos(a)redhat.com>, devel(a)ovirt.org, "Michal
Skrivanek" <mskrivan(a)redhat.com>
Sent: Thursday, February 12, 2015 6:11:16 PM
Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
Il 12/02/2015 17:03, Nir Soffer ha scritto:
>
>
> ----- Original Message -----
>> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
>> To: "Nir Soffer" <nsoffer(a)redhat.com>
>> Cc: "David Teigland" <teigland(a)redhat.com>, "Yeela
Kaplan"
>> <ykaplan(a)redhat.com>, "Alon Bar-Lev" <alonbl(a)redhat.com>,
>> "Allon Mureinik" <amureini(a)redhat.com>, "Yaniv Dary"
<ydary(a)redhat.com>,
>> "Federico Simoncelli"
>> <fsimonce(a)redhat.com>, "Sahina Bose" <sabose(a)redhat.com>,
"Yedidyah Bar
>> David" <didi(a)redhat.com>, "Simone
>> Tiraboschi" <stirabos(a)redhat.com>, devel(a)ovirt.org, "Michal
Skrivanek"
>> <mskrivan(a)redhat.com>
>> Sent: Thursday, February 12, 2015 5:43:28 PM
>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
>>
>> Il 12/02/2015 16:26, Nir Soffer ha scritto:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
>>>> To: "Nir Soffer" <nsoffer(a)redhat.com>, "David
Teigland"
>>>> <teigland(a)redhat.com>
>>>> Cc: "Yeela Kaplan" <ykaplan(a)redhat.com>, "Alon
Bar-Lev"
>>>> <alonbl(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>,
>>>> "Yaniv Dary" <ydary(a)redhat.com>, "Federico
Simoncelli"
>>>> <fsimonce(a)redhat.com>, "Sahina Bose"
<sabose(a)redhat.com>,
>>>> "Yedidyah Bar David" <didi(a)redhat.com>, "Simone
Tiraboschi"
>>>> <stirabos(a)redhat.com>, devel(a)ovirt.org, "Michal
>>>> Skrivanek" <mskrivan(a)redhat.com>
>>>> Sent: Thursday, February 12, 2015 5:22:48 PM
>>>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying
>>>> hosted-engine
>>>>
>>>> Il 12/02/2015 16:20, Nir Soffer ha scritto:
>>>>> ----- Original Message -----
>>>>>> From: "David Teigland" <teigland(a)redhat.com>
>>>>>> To: "Nir Soffer" <nsoffer(a)redhat.com>
>>>>>> Cc: "Sandro Bonazzola" <sbonazzo(a)redhat.com>,
"Yeela Kaplan"
>>>>>> <ykaplan(a)redhat.com>, "Alon Bar-Lev"
>>>>>> <alonbl(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Yaniv
>>>>>> Dary"
>>>>>> <ydary(a)redhat.com>, "Federico Simoncelli"
>>>>>> <fsimonce(a)redhat.com>, "Sahina Bose"
<sabose(a)redhat.com>, "Yedidyah
>>>>>> Bar
>>>>>> David" <didi(a)redhat.com>, "Simone
>>>>>> Tiraboschi" <stirabos(a)redhat.com>, devel(a)ovirt.org,
"Michal Skrivanek"
>>>>>> <mskrivan(a)redhat.com>
>>>>>> Sent: Thursday, February 12, 2015 5:13:49 PM
>>>>>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying
>>>>>> hosted-engine
>>>>>>
>>>>>>>> 2015-02-12 13:08:25+0100 7254 [683]: open error -1
>>>>>>>>
/rhev/data-center/mnt/glusterSD/minidell.home:_hosted__engine__glusterfs/a7dba6e3-09ac-430a-8d6e-eea17cf18b8f/images/51156c24-5ea1-42df-bf33-daebf2c4780c/3719b0
>>>>>>>> 06-68a2-4742-964c-413a3f77c0b4.lease
>>>>>>
>>>>>> open(2) error was EPERM
>>>>>
>>>>> selinux acting again?
>>>>>
>>>>> Sandro, can you attach also /var/log/audit/audit.log?
>>>>
>>>> Sure, but other than what I already reported in
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1191989
>>>> I don't think you'll find much more.
>>>
>>> Can you reproduce this in permissive mode?
>>
>> Yes, it's fully reproducible on a host in permissive mode.
>>
>> Also writing to the storage works:
>>
>> while true; do echo starting dd `date`; dd if=/dev/zero
>> of=/tmp/test/testfile
>> bs=4k count=1k oflag=direct ; echo dd finished `date`; rm -f
>> /tmp/test/testfile ; sleep 10; done
>
> What does "df -T /tmp" show?
>
# df -T /tmp /tmp/test/
File system Tipo 1K-blocchi Usati
Disponib. Uso% Montato su
/dev/sda6 ext4 40185112 5347916
32772812 15% /
192.168.1.107:/hosted_engine_glusterfs fuse.glusterfs 40185088 5347968
32772736 15% /tmp/test
> /tmp is typically tempfs, so your dd test never touched any storage.
>
> Also, copying zeros can be ignored by smart storage, so better copy
> "real" file data:
>
> dd if=/dev/urandom of=testfile bs=1M count=1
>
> Now use testfile for writing to storage:
>
> dd if=testfile of=/path/to/glusterfs/mount/testfile bs=1M count=1
> oflag=direct
>
> You should do something like:
>
> 1. mount your gluster volume - same as vdsm does
> 2. start the write test, writing to the gluster storage
> 3. start your hosted engine flow
> 4. when it fails, keep your write test running until for a while
If you prefer I can also do a double test, one on /tmp/test and one on
/rhev/data-center/mnt/glusterSD/mididell.home\:_hosted__engine__glusterfs/
I'll try to do the test tomorrow.
Writing to /tmp/test seems ok. It will be nice to see a test using random
data instead of zeros.
I think David should continue to investigate this to understand why
sanlock cannot access storage.
>> didn't show any issue while sanlock is failing.
>>
>> Attached the tar.gz of the ovirt-hosted-engine-setup I'm running and the
>> full
>> /var/log directory content.
>>
>> Note that both glusterd and vdsmd are running on the same host for the
>> hyper-converged setup.
>>
>>>
>>>>>>
>>>>>>>> 2015-02-12 13:08:25+0100 7254 [683]: r4 release_token
open error -2
>>>>>>
>>>>>> open(2) error was ENOENT
>>>>>>
>>>>>>>> 2015-02-12 13:08:31+0100 7260 [16910]: a7dba6e3 aio
collect 0
>>>>>>>> 0x7f1d640008c0:0x7f1d640008d0:0x7f1d7ec52000 result
-107:0 match res
>>>>>>
>>>>>> aio result on open file was ENOTCONN
>>>>>>
>>>>>> Maybe gluster logged some errors about what happened here?
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Sandro Bonazzola
>>>> Better technology. Faster innovation. Powered by community
>>>> collaboration.
>>>> See how it works at redhat.com
>>>>
>>
>>
>> --
>> Sandro Bonazzola
>> Better technology. Faster innovation. Powered by community collaboration.
>> See how it works at redhat.com
>>
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com