10:11 a.m.
Il 12/02/2015 17:03, Nir Soffer ha scritto:
----- Original Message -----
> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
> To: "Nir Soffer" <nsoffer(a)redhat.com>
> Cc: "David Teigland" <teigland(a)redhat.com>, "Yeela Kaplan"
<ykaplan(a)redhat.com>, "Alon Bar-Lev" <alonbl(a)redhat.com>,
> "Allon Mureinik" <amureini(a)redhat.com>, "Yaniv Dary"
<ydary(a)redhat.com>, "Federico Simoncelli"
> <fsimonce(a)redhat.com>, "Sahina Bose" <sabose(a)redhat.com>,
"Yedidyah Bar David" <didi(a)redhat.com>, "Simone
> Tiraboschi" <stirabos(a)redhat.com>, devel(a)ovirt.org, "Michal
Skrivanek" <mskrivan(a)redhat.com>
> Sent: Thursday, February 12, 2015 5:43:28 PM
> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
>
> Il 12/02/2015 16:26, Nir Soffer ha scritto:
>>
>>
>> ----- Original Message -----
>>> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
>>> To: "Nir Soffer" <nsoffer(a)redhat.com>, "David
Teigland"
>>> <teigland(a)redhat.com>
>>> Cc: "Yeela Kaplan" <ykaplan(a)redhat.com>, "Alon
Bar-Lev"
>>> <alonbl(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>,
>>> "Yaniv Dary" <ydary(a)redhat.com>, "Federico
Simoncelli"
>>> <fsimonce(a)redhat.com>, "Sahina Bose"
<sabose(a)redhat.com>,
>>> "Yedidyah Bar David" <didi(a)redhat.com>, "Simone
Tiraboschi"
>>> <stirabos(a)redhat.com>, devel(a)ovirt.org, "Michal
>>> Skrivanek" <mskrivan(a)redhat.com>
>>> Sent: Thursday, February 12, 2015 5:22:48 PM
>>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying hosted-engine
>>>
>>> Il 12/02/2015 16:20, Nir Soffer ha scritto:
>>>> ----- Original Message -----
>>>>> From: "David Teigland" <teigland(a)redhat.com>
>>>>> To: "Nir Soffer" <nsoffer(a)redhat.com>
>>>>> Cc: "Sandro Bonazzola" <sbonazzo(a)redhat.com>,
"Yeela Kaplan"
>>>>> <ykaplan(a)redhat.com>, "Alon Bar-Lev"
>>>>> <alonbl(a)redhat.com>, "Allon Mureinik"
<amureini(a)redhat.com>, "Yaniv
>>>>> Dary"
>>>>> <ydary(a)redhat.com>, "Federico Simoncelli"
>>>>> <fsimonce(a)redhat.com>, "Sahina Bose"
<sabose(a)redhat.com>, "Yedidyah Bar
>>>>> David" <didi(a)redhat.com>, "Simone
>>>>> Tiraboschi" <stirabos(a)redhat.com>, devel(a)ovirt.org,
"Michal Skrivanek"
>>>>> <mskrivan(a)redhat.com>
>>>>> Sent: Thursday, February 12, 2015 5:13:49 PM
>>>>> Subject: Re: [ovirt-devel] [HC] Weird issue while deploying
>>>>> hosted-engine
>>>>>
>>>>>>> 2015-02-12 13:08:25+0100 7254 [683]: open error -1
>>>>>>>
/rhev/data-center/mnt/glusterSD/minidell.home:_hosted__engine__glusterfs/a7dba6e3-09ac-430a-8d6e-eea17cf18b8f/images/51156c24-5ea1-42df-bf33-daebf2c4780c/3719b0
>>>>>>> 06-68a2-4742-964c-413a3f77c0b4.lease
>>>>>
>>>>> open(2) error was EPERM
>>>>
>>>> selinux acting again?
>>>>
>>>> Sandro, can you attach also /var/log/audit/audit.log?
>>>
>>> Sure, but other than what I already reported in
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1191989
>>> I don't think you'll find much more.
>>
>> Can you reproduce this in permissive mode?
>
> Yes, it's fully reproducible on a host in permissive mode.
>
> Also writing to the storage works:
>
> while true; do echo starting dd `date`; dd if=/dev/zero of=/tmp/test/testfile
> bs=4k count=1k oflag=direct ; echo dd finished `date`; rm -f
> /tmp/test/testfile ; sleep 10; done
What does "df -T /tmp" show?
# df -T /tmp /tmp/test/
File system Tipo 1K-blocchi Usati Disponib. Uso%
Montato su
/dev/sda6 ext4 40185112 5347916 32772812 15% /
192.168.1.107:/hosted_engine_glusterfs fuse.glusterfs 40185088 5347968 32772736 15%
/tmp/test
/tmp is typically tempfs, so your dd test never touched any storage.
Also, copying zeros can be ignored by smart storage, so better copy
"real" file data:
dd if=/dev/urandom of=testfile bs=1M count=1
Now use testfile for writing to storage:
dd if=testfile of=/path/to/glusterfs/mount/testfile bs=1M count=1 oflag=direct
You should do something like:
1. mount your gluster volume - same as vdsm does
2. start the write test, writing to the gluster storage
3. start your hosted engine flow
4. when it fails, keep your write test running until for a while
If you prefer I can also do a double test, one on /tmp/test and one on
/rhev/data-center/mnt/glusterSD/mididell.home\:_hosted__engine__glusterfs/
I'll try to do the test tomorrow.
>
>
> didn't show any issue while sanlock is failing.
>
> Attached the tar.gz of the ovirt-hosted-engine-setup I'm running and the full
> /var/log directory content.
>
> Note that both glusterd and vdsmd are running on the same host for the
> hyper-converged setup.
>
>>
>>>>>
>>>>>>> 2015-02-12 13:08:25+0100 7254 [683]: r4 release_token open
error -2
>>>>>
>>>>> open(2) error was ENOENT
>>>>>
>>>>>>> 2015-02-12 13:08:31+0100 7260 [16910]: a7dba6e3 aio collect
0
>>>>>>> 0x7f1d640008c0:0x7f1d640008d0:0x7f1d7ec52000 result -107:0
match res
>>>>>
>>>>> aio result on open file was ENOTCONN
>>>>>
>>>>> Maybe gluster logged some errors about what happened here?
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Sandro Bonazzola
>>> Better technology. Faster innovation. Powered by community collaboration.
>>> See how it works at redhat.com
>>>
>
>
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
>
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com