5:37 p.m.
Hello all,
I'm setting up a fairly (?) complex oVirt over Gluster setup built around 3
Xeon servers-turned-into-workstations, each doubling as oVirt node + one
primary Fedora VM w/ a dedicated passthrough GPU (+audio and a couple of
USB root devices).
One of the servers seems to have some weird issue w/ the passthrough nVidia
GPU that seems to require me to edit the VM iommu (1) and passthrough
device (2) command line.
I tried using the qemu-cmdline addon to add the missing parameters, but it
seems that qemu treats the added parameters as an additional device / iommu
instead of editing the existing parameters.
So:
1. How can I view the VM qemu command line?
2. Can I somehow manually edit the qemu command line, either directly or by
somehow adding parameters in the HE XML file?
- Gilboa.
[1] iommu: VM XXX is down with error. Exit message: internal error: qemu
unexpectedly closed the monitor: 2021-11-05T14:59:44.499366Z qemu-kvm: We
need to set caching-mode=on for intel-iommu to enable device assignment
with IOMMU protection.
[2] GPU: May need to add x-vga=off,
5:47 p.m.
On Mon, Nov 8, 2021 at 5:37 PM Gilboa Davara <gilboad(a)gmail.com> wrote:
Hello all,
I'm setting up a fairly (?) complex oVirt over Gluster setup built around
3 Xeon servers-turned-into-workstations, each doubling as oVirt node + one
primary Fedora VM w/ a dedicated passthrough GPU (+audio and a couple of
USB root devices).
One of the servers seems to have some weird issue w/ the passthrough
nVidia GPU that seems to require me to edit the VM iommu (1) and
passthrough device (2) command line.
I tried using the qemu-cmdline addon to add the missing parameters, but it
seems that qemu treats the added parameters as an additional device / iommu
instead of editing the existing parameters.
So:
1. How can I view the VM qemu command line?
2. Can I somehow manually edit the qemu command line, either directly or
by somehow adding parameters in the HE XML file?
- Gilboa.
[1] iommu: VM XXX is down with error. Exit message: internal error: qemu
unexpectedly closed the monitor: 2021-11-05T14:59:44.499366Z qemu-kvm: We
need to set caching-mode=on for intel-iommu to enable device assignment
with IOMMU protection.
[2] GPU: May need to add x-vga=off,
BTW, FWIW, by isolated I mean device attached during boot to vfio_pci, and
passed to the VM with all of its iommu group members (e.g. nVidia GPU +
nVidia HDMI audio).
- Gilboa
8:52 p.m.
New subject: Viewing and hopefully, modifying the VM's qemu command line
On Mon, Nov 8, 2021 at 5:47 PM Gilboa Davara <gilboad(a)gmail.com> wrote:
Hello all,
I'm setting up a fairly (?) complex oVirt over Gluster setup built around 3 Xeon
servers-turned-into-workstations, each doubling as oVirt node + one primary Fedora VM w/ a
dedicated passthrough GPU (+audio and a couple of USB root devices).
One of the servers seems to have some weird issue w/ the passthrough nVidia GPU that
seems to require me to edit the VM iommu (1) and passthrough device (2) command line.
I tried using the qemu-cmdline addon to add the missing parameters, but it seems that
qemu treats the added parameters as an additional device / iommu instead of editing the
existing parameters.
So:
1. How can I view the VM qemu command line?
less /var/log/libvirt/qemu/vm-name.log
2. Can I somehow manually edit the qemu command line, either directly
or by somehow adding parameters in the HE XML file?
I think this should be possible via vdsm hook, but hooks are bad.
Can you explain what do you want to change?
Nir
11:52 a.m.
New subject: Viewing and hopefully, modifying the VM's qemu command line
Hello,
Many thanks for the prompt reply.
On Mon, Nov 8, 2021 at 8:52 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Mon, Nov 8, 2021 at 5:47 PM Gilboa Davara
<gilboad(a)gmail.com> wrote:
>
> Hello all,
>
> I'm setting up a fairly (?) complex oVirt over Gluster setup built
around 3 Xeon servers-turned-into-workstations, each doubling as oVirt node
+ one primary Fedora VM w/ a dedicated passthrough GPU (+audio and a couple
of USB root devices).
> One of the servers seems to have some weird issue w/ the passthrough
nVidia GPU that seems to require me to edit the VM iommu (1) and
passthrough device (2) command line.
> I tried using the qemu-cmdline addon to add the missing parameters, but
it seems that qemu treats the added parameters as an additional device /
iommu instead of editing the existing parameters.
>
> So:
> 1. How can I view the VM qemu command line?
less /var/log/libvirt/qemu/vm-name.log
Found it, thanks!
> 2. Can I somehow manually edit the qemu command line, either
directly or
by somehow adding parameters in the HE XML file?
I think this should be possible via vdsm hook, but hooks are bad.
Can you explain what do you want to change?
Nir
1. IOMMU:
-device intel-iommu,intremap=on,eim=on
To (Add: caching-mode=on, per qemu log message)
-device intel-iommu,intremap=on,eim=on,caching-mode=on
(2021-11-05T18:42:36.651191Z qemu-kvm: We need to set caching-mode=on for
intel-iommu to enable device assignment with IOMMU protection.)
2. Device:
-device
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.10,addr=0x0
-device
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8,addr=0x0
To (Attach hdmi sound as function of GPU, possibly add x-vga=on):
-device
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.8.0,x-vga=on,addr=0x0
-device
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8.1,addr=0x0
3. I'm facing some odd memory allocation issues (only if the VM is Q35, all
works well if the VM is configured as i440fx), which may req
11:54 a.m.
New subject: Viewing and hopefully, modifying the VM's qemu command line
(Sorry: Pressed send too fast...)
3. I'm facing some odd memory allocation issues (only if the VM is Q35, all
works well if the VM is configured as i440fx), which may require me to
change additional VM parameters.
BTW, ironically, the two other hosts in the same cluster, with older MBs,
have zero issues with GPU/Audio/USB passthrough...)
- Gilboa
On Tue, Nov 9, 2021 at 11:52 AM Gilboa Davara <gilboad(a)gmail.com> wrote:
Hello,
Many thanks for the prompt reply.
On Mon, Nov 8, 2021 at 8:52 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
> On Mon, Nov 8, 2021 at 5:47 PM Gilboa Davara <gilboad(a)gmail.com> wrote:
> >
> > Hello all,
> >
> > I'm setting up a fairly (?) complex oVirt over Gluster setup built
> around 3 Xeon servers-turned-into-workstations, each doubling as oVirt node
> + one primary Fedora VM w/ a dedicated passthrough GPU (+audio and a couple
> of USB root devices).
> > One of the servers seems to have some weird issue w/ the passthrough
> nVidia GPU that seems to require me to edit the VM iommu (1) and
> passthrough device (2) command line.
> > I tried using the qemu-cmdline addon to add the missing parameters, but
> it seems that qemu treats the added parameters as an additional device /
> iommu instead of editing the existing parameters.
> >
> > So:
> > 1. How can I view the VM qemu command line?
>
> less /var/log/libvirt/qemu/vm-name.log
>
Found it, thanks!
> > 2. Can I somehow manually edit the qemu command line, either directly
> or by somehow adding parameters in the HE XML file?
>
> I think this should be possible via vdsm hook, but hooks are bad.
> Can you explain what do you want to change?
>
> Nir
>
1. IOMMU:
-device intel-iommu,intremap=on,eim=on
To (Add: caching-mode=on, per qemu log message)
-device intel-iommu,intremap=on,eim=on,caching-mode=on
(2021-11-05T18:42:36.651191Z qemu-kvm: We need to set caching-mode=on for
intel-iommu to enable device assignment with IOMMU protection.)
2. Device:
-device
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.10,addr=0x0
-device
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8,addr=0x0
To (Attach hdmi sound as function of GPU, possibly add x-vga=on):
-device
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.8.0,x-vga=on,addr=0x0
-device
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8.1,addr=0x0
3. I'm facing some odd memory allocation issues (only if the VM is Q35,
all works well if the VM is configured as i440fx), which may req
3:28 p.m.
New subject: Viewing and hopefully, modifying the VM's qemu command line
On Tue, Nov 9, 2021 at 11:57 AM Gilboa Davara <gilboad(a)gmail.com> wrote:
(Sorry: Pressed send too fast...)
3. I'm facing some odd memory allocation issues (only if the VM is Q35,
all works well if the VM is configured as i440fx), which may require me to
change additional VM parameters.
BTW, ironically, the two other hosts in the same cluster, with older MBs,
have zero issues with GPU/Audio/USB passthrough...)
- Gilboa
On Tue, Nov 9, 2021 at 11:52 AM Gilboa Davara <gilboad(a)gmail.com> wrote:
> Hello,
>
> Many thanks for the prompt reply.
>
> On Mon, Nov 8, 2021 at 8:52 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
>
>> On Mon, Nov 8, 2021 at 5:47 PM Gilboa Davara <gilboad(a)gmail.com> wrote:
>> >
>> > Hello all,
>> >
>> > I'm setting up a fairly (?) complex oVirt over Gluster setup built
>> around 3 Xeon servers-turned-into-workstations, each doubling as oVirt node
>> + one primary Fedora VM w/ a dedicated passthrough GPU (+audio and a couple
>> of USB root devices).
>> > One of the servers seems to have some weird issue w/ the passthrough
>> nVidia GPU that seems to require me to edit the VM iommu (1) and
>> passthrough device (2) command line.
>> > I tried using the qemu-cmdline addon to add the missing parameters,
>> but it seems that qemu treats the added parameters as an additional device
>> / iommu instead of editing the existing parameters.
>> >
>> > So:
>> > 1. How can I view the VM qemu command line?
>>
>> less /var/log/libvirt/qemu/vm-name.log
>>
>
> Found it, thanks!
>
>
>> > 2. Can I somehow manually edit the qemu command line, either directly
>> or by somehow adding parameters in the HE XML file?
>>
>> I think this should be possible via vdsm hook, but hooks are bad.
>> Can you explain what do you want to change?
>>
>> Nir
>>
>
> 1. IOMMU:
> -device intel-iommu,intremap=on,eim=on
> To (Add: caching-mode=on, per qemu log message)
> -device intel-iommu,intremap=on,eim=on,caching-mode=on
> (2021-11-05T18:42:36.651191Z qemu-kvm: We need to set caching-mode=on for
> intel-iommu to enable device assignment with IOMMU protection.)
>
Note that this one should be fixed in the latest ovirt-engine
(ovirt-engine-4.4.9.4) -
https://bugzilla.redhat.com/show_bug.cgi?id=2013752
> 2. Device:
> -device
>
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.10,addr=0x0
>
> -device
>
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8,addr=0x0
> To (Attach hdmi sound as function of GPU, possibly add x-vga=on):
> -device
>
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.8.0,x-vga=on,addr=0x0
>
> -device
>
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8.1,addr=0x0
>
> 3. I'm facing some odd memory allocation issues (only if the VM is Q35,
> all works well if the VM is configured as i440fx), which may req
>
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XKZLE6ZEU7Z...
5:48 p.m.
New subject: Viewing and hopefully, modifying the VM's qemu command line
Many thanks!
Somehow my Google-fu skills (and DDG-Fu skills) missed this bug report.
I wonder if 4.4.9 will fix the sporadic -ENOMEM (1) that I'm getting under
Q35...
Thanks,
- Gilboa
(1)
2021-11-05T13:13:06.290874Z qemu-kvm: -device
vfio-pci,host=0000:07:00.0,id=ua-a3048786-da87-425f-83e6-51afbb906423,bus=pci.10,addr=0x0:
VFIO_MAP_DMA failed: Cannot allocate memory
2021-11-05T13:13:06.362210Z qemu-kvm: -device
vfio-pci,host=0000:07:00.0,id=ua-a3048786-da87-425f-83e6-51afbb906423,bus=pci.10,addr=0x0:
VFIO_MAP_DMA failed: Cannot allocate memory
2021-11-05T13:13:06.362609Z qemu-kvm: -device
vfio-pci,host=0000:07:00.0,id=ua-a3048786-da87-425f-83e6-51afbb906423,bus=pci.10,addr=0x0:
vfio 0000:07:00.0: failed to setup container for group 33: memory listener
initialization failed: Region ram-node0: vfio_dma_map(0x55fe05d647f0,
0xc0000, 0x7ff40000, 0x7f821fec0000) = -12 (Cannot allocate memory)
2021-11-05 13:13:06.580+0000: shutting down, reason=failed
On Tue, Nov 9, 2021 at 3:29 PM Arik Hadas <ahadas(a)redhat.com> wrote:
On Tue, Nov 9, 2021 at 11:57 AM Gilboa Davara <gilboad(a)gmail.com> wrote:
> (Sorry: Pressed send too fast...)
> 3. I'm facing some odd memory allocation issues (only if the VM is Q35,
> all works well if the VM is configured as i440fx), which may require me to
> change additional VM parameters.
>
> BTW, ironically, the two other hosts in the same cluster, with older MBs,
> have zero issues with GPU/Audio/USB passthrough...)
>
> - Gilboa
>
> On Tue, Nov 9, 2021 at 11:52 AM Gilboa Davara <gilboad(a)gmail.com> wrote:
>
>> Hello,
>>
>> Many thanks for the prompt reply.
>>
>> On Mon, Nov 8, 2021 at 8:52 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
>>
>>> On Mon, Nov 8, 2021 at 5:47 PM Gilboa Davara <gilboad(a)gmail.com>
wrote:
>>> >
>>> > Hello all,
>>> >
>>> > I'm setting up a fairly (?) complex oVirt over Gluster setup built
>>> around 3 Xeon servers-turned-into-workstations, each doubling as oVirt node
>>> + one primary Fedora VM w/ a dedicated passthrough GPU (+audio and a couple
>>> of USB root devices).
>>> > One of the servers seems to have some weird issue w/ the passthrough
>>> nVidia GPU that seems to require me to edit the VM iommu (1) and
>>> passthrough device (2) command line.
>>> > I tried using the qemu-cmdline addon to add the missing parameters,
>>> but it seems that qemu treats the added parameters as an additional device
>>> / iommu instead of editing the existing parameters.
>>> >
>>> > So:
>>> > 1. How can I view the VM qemu command line?
>>>
>>> less /var/log/libvirt/qemu/vm-name.log
>>>
>>
>> Found it, thanks!
>>
>>
>>> > 2. Can I somehow manually edit the qemu command line, either directly
>>> or by somehow adding parameters in the HE XML file?
>>>
>>> I think this should be possible via vdsm hook, but hooks are bad.
>>> Can you explain what do you want to change?
>>>
>>> Nir
>>>
>>
>> 1. IOMMU:
>> -device intel-iommu,intremap=on,eim=on
>> To (Add: caching-mode=on, per qemu log message)
>> -device intel-iommu,intremap=on,eim=on,caching-mode=on
>> (2021-11-05T18:42:36.651191Z qemu-kvm: We need to set caching-mode=on
>> for intel-iommu to enable device assignment with IOMMU protection.)
>>
>
Note that this one should be fixed in the latest ovirt-engine
(ovirt-engine-4.4.9.4) -
https://bugzilla.redhat.com/show_bug.cgi?id=2013752
>
>> 2. Device:
>> -device
>>
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.10,addr=0x0
>>
>> -device
>>
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8,addr=0x0
>> To (Attach hdmi sound as function of GPU, possibly add x-vga=on):
>> -device
>>
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.8.0,x-vga=on,addr=0x0
>>
>> -device
>>
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8.1,addr=0x0
>>
>> 3. I'm facing some odd memory allocation issues (only if the VM is Q35,
>> all works well if the VM is configured as i440fx), which may req
>>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XKZLE6ZEU7Z...
>
6 p.m.
New subject: Viewing and hopefully, modifying the VM's qemu command line
Strike that, said cluster is already using 4.40.90.4-1. Adding myself to
the BZ.
- Gilboa
On Tue, Nov 9, 2021 at 5:48 PM Gilboa Davara <gilboad(a)gmail.com> wrote:
Many thanks!
Somehow my Google-fu skills (and DDG-Fu skills) missed this bug report.
I wonder if 4.4.9 will fix the sporadic -ENOMEM (1) that I'm getting under
Q35...
Thanks,
- Gilboa
(1)
2021-11-05T13:13:06.290874Z qemu-kvm: -device
vfio-pci,host=0000:07:00.0,id=ua-a3048786-da87-425f-83e6-51afbb906423,bus=pci.10,addr=0x0:
VFIO_MAP_DMA failed: Cannot allocate memory
2021-11-05T13:13:06.362210Z qemu-kvm: -device
vfio-pci,host=0000:07:00.0,id=ua-a3048786-da87-425f-83e6-51afbb906423,bus=pci.10,addr=0x0:
VFIO_MAP_DMA failed: Cannot allocate memory
2021-11-05T13:13:06.362609Z qemu-kvm: -device
vfio-pci,host=0000:07:00.0,id=ua-a3048786-da87-425f-83e6-51afbb906423,bus=pci.10,addr=0x0:
vfio 0000:07:00.0: failed to setup container for group 33: memory listener
initialization failed: Region ram-node0: vfio_dma_map(0x55fe05d647f0,
0xc0000, 0x7ff40000, 0x7f821fec0000) = -12 (Cannot allocate memory)
2021-11-05 13:13:06.580+0000: shutting down, reason=failed
On Tue, Nov 9, 2021 at 3:29 PM Arik Hadas <ahadas(a)redhat.com> wrote:
>
>
> On Tue, Nov 9, 2021 at 11:57 AM Gilboa Davara <gilboad(a)gmail.com> wrote:
>
>> (Sorry: Pressed send too fast...)
>> 3. I'm facing some odd memory allocation issues (only if the VM is Q35,
>> all works well if the VM is configured as i440fx), which may require me to
>> change additional VM parameters.
>>
>> BTW, ironically, the two other hosts in the same cluster, with older
>> MBs, have zero issues with GPU/Audio/USB passthrough...)
>>
>> - Gilboa
>>
>> On Tue, Nov 9, 2021 at 11:52 AM Gilboa Davara <gilboad(a)gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Many thanks for the prompt reply.
>>>
>>> On Mon, Nov 8, 2021 at 8:52 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
>>>
>>>> On Mon, Nov 8, 2021 at 5:47 PM Gilboa Davara <gilboad(a)gmail.com>
>>>> wrote:
>>>> >
>>>> > Hello all,
>>>> >
>>>> > I'm setting up a fairly (?) complex oVirt over Gluster setup
built
>>>> around 3 Xeon servers-turned-into-workstations, each doubling as oVirt
node
>>>> + one primary Fedora VM w/ a dedicated passthrough GPU (+audio and a
couple
>>>> of USB root devices).
>>>> > One of the servers seems to have some weird issue w/ the
passthrough
>>>> nVidia GPU that seems to require me to edit the VM iommu (1) and
>>>> passthrough device (2) command line.
>>>> > I tried using the qemu-cmdline addon to add the missing parameters,
>>>> but it seems that qemu treats the added parameters as an additional
device
>>>> / iommu instead of editing the existing parameters.
>>>> >
>>>> > So:
>>>> > 1. How can I view the VM qemu command line?
>>>>
>>>> less /var/log/libvirt/qemu/vm-name.log
>>>>
>>>
>>> Found it, thanks!
>>>
>>>
>>>> > 2. Can I somehow manually edit the qemu command line, either
>>>> directly or by somehow adding parameters in the HE XML file?
>>>>
>>>> I think this should be possible via vdsm hook, but hooks are bad.
>>>> Can you explain what do you want to change?
>>>>
>>>> Nir
>>>>
>>>
>>> 1. IOMMU:
>>> -device intel-iommu,intremap=on,eim=on
>>> To (Add: caching-mode=on, per qemu log message)
>>> -device intel-iommu,intremap=on,eim=on,caching-mode=on
>>> (2021-11-05T18:42:36.651191Z qemu-kvm: We need to set caching-mode=on
>>> for intel-iommu to enable device assignment with IOMMU protection.)
>>>
>>
> Note that this one should be fixed in the latest ovirt-engine
> (ovirt-engine-4.4.9.4) -
> https://bugzilla.redhat.com/show_bug.cgi?id=2013752
>
>
>>
>>> 2. Device:
>>> -device
>>>
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.10,addr=0x0
>>>
>>> -device
>>>
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8,addr=0x0
>>> To (Attach hdmi sound as function of GPU, possibly add x-vga=on):
>>> -device
>>>
vfio-pci,host=0000:84:00.0,id=ua-c774115b-c0ee-43a5-97c8-12ac73cb6f3a,bus=pci.8.0,x-vga=on,addr=0x0
>>>
>>> -device
>>>
vfio-pci,host=0000:84:00.1,id=ua-3c7d3b66-57aa-470b-afc8-78383ac9a025,bus=pci.8.1,addr=0x0
>>>
>>> 3. I'm facing some odd memory allocation issues (only if the VM is Q35,
>>> all works well if the VM is configured as i440fx), which may req
>>>
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XKZLE6ZEU7Z...
>>
>
1103
days inactive
1104
days old
7 comments
3 participants
participants (3)
-
Arik Hadas -
Gilboa Davara -
Nir Soffer