3:45 p.m.
This is a multi-part message in MIME format.
--------------C1E2D94265E0210FD5646D84
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Kai,
The symptoms after removal of vdsm-hook-ovs look just like the ovirtmgmt
bridged network was not created. The log to look in would be
supervdsm.log, but it looks like yours has been rotated after supervdsmd
was restarted (the warning is benign). Can you dig out "supervdsm.log.1"?
OVS started to be supported by the hook initially (this was the first
attempt at supporting OVS). Currently, proper OVS support is in
development and should appear in 4.0 as a tech-preview.
Thanks,
Ondra (osvoboda(a)redhat.com)
On 25.5.2016 11:01, Kai Kang wrote:
On Tue, May 24, 2016 at 8:43 PM, Ondřej Svoboda <ondrej(a)svobodasoft.cz
<mailto:ondrej@svobodasoft.cz>> wrote:
Hi again, (correcting myself a bit)
are using the Open vSwitch hook intentionally? If not, you can
|yum remove vdsm-hook-ovs| for a while. What is your exact VDSM
version?
My VDSM version is 4.17.24 and ovirt-engine is 3.6.4. When after
remove ovs hooks, it still fails with:
"Host node3 does not comply with the cluster Default networks, the
following networks are missing on host: 'ovirtmgmt'".
I checked vdsm.log it seems no erros there. And in supervdsm.log, it
shows warnings:
sourceRoute::INFO::2016-05-25
07:57:41,779::sourceroutethread::60::root::(process_IN_CLOSE_WRITE_filePath)
interface ovirtmgmt is not a libvirt interface
sourceRoute::WARNING::2016-05-25
07:57:41,780::utils::140::root::(rmFile) File:
/var/run/vdsm/trackedInterfaces/ovirtmgmt already removed
What should I check next? Thanks.
I uploaded vdsm.log to:
https://gist.github.com/parr0tr1ver/1e38171b5d12cf77321101530276d368
and supervdsm.log:
https://gist.github.com/parr0tr1ver/97805698a485f1cd49ded2b095297531
Also, can you give a little more context from your vdsm.log, and
also from supervdsm.log? I think the vdsm.log failure is only
related to stats reporting, and is not the root problem.
If you don't have any confidential information in the logs (or you
can remove it), I suggest that you open a bug on product=vdsm at
https://bugzilla.redhat.com/ Have no fear about naming the bug, it
can be renamed if necessary.
I find there is a bug on
https://bugzilla.redhat.com/show_bug.cgi?id=1234867. The "Target
Milestone" to "support ovs via vdsm hook" is ovirt-3.6.7. Does that
mean ovs does not work actually in previous versions?
Thanks a lot.
--Kai
Thanks, Ondra
> On 24.5.2016 11:47, Kai Kang wrote:
>> Hi,
>>
>> I checked vdsm.log, it show error:
>>
>> jsonrpc.Executor/0::DEBUG::2016-05-24
>> 09:46:04,056::utils::671::root::(execCmd) /usr/bin/taskset
>> --cpu-list 0-7 /usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs
>> (cwd None)
>> jsonrpc.Executor/0::DEBUG::2016-05-24
>> 09:46:04,136::utils::689::root::(execCmd) FAILED: <err> =
>> 'Traceback (most recent call last):\n File
>> "/usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs", line 72, in
>> <module>\n main()\n File
>> "/usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs", line 66, in
>> main\n
>>
stats[\'network\'].update(ovs_networks_stats(stats[\'network\']))\nKeyError:
>> \'network\'\n\n'; <rc> = 2
>> jsonrpc.Executor/0::INFO::2016-05-24
>> 09:46:04,136::hooks::98::root::(_runHooksDir) Traceback (most
>> recent call last):
>> File "/usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs", line
>> 72, in <module>
>> main()
>> File "/usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs", line
>> 66, in main
>>
stats['network'].update(ovs_networks_stats(stats['network']))
>> KeyError: 'network'
>>
>>
>> I'm checking what wrong with it.
>>
>> Thanks,
>> Kai
>>
>>
>>
>> On Tue, May 24, 2016 at 3:57 PM, Kai Kang <kai.7.kang(a)gmail.com
>> <mailto:kai.7.kang@gmail.com>> wrote:
>>
>> And network configurations on node:
>>
>> [root at ovirt-node] # brctl show
>> bridge name bridge id STP enabled interfaces
>> docker0 8000.0242ae1de711 no
>> ovirtmgmt 8000.001320ff73aa no eth0
>>
>> [root at ovirt-node] # ifconfig -a
>> docker0 Link encap:Ethernet HWaddr 02:42:ae:1d:e7:11
>> inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
>> UP BROADCAST MULTICAST MTU:1500 Metric:1
>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
>>
>> eth0 Link encap:Ethernet HWaddr 00:13:20:ff:73:aa
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:6331 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:2866 errors:0 dropped:0 overruns:0
>> carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:762182 (744.3 KiB) TX bytes:210611
>> (205.6 KiB)
>> Interrupt:20 Memory:d1100000-d1120000
>>
>> lo Link encap:Local Loopback
>> inet addr:127.0.0.1 Mask:255.0.0.0
>> inet6 addr: ::1/128 Scope:Host
>> UP LOOPBACK RUNNING MTU:65536 Metric:1
>> RX packets:36 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:2478 (2.4 KiB) TX bytes:2478 (2.4 KiB)
>>
>> ovirtmgmt Link encap:Ethernet HWaddr 00:13:20:ff:73:aa
>> inet addr:128.224.165.170 Bcast:128.224.165.255
>> Mask:255.255.255.0
>> inet6 addr: fe80::213:20ff:feff:73aa/64 Scope:Link
>> inet6 addr:
>> 11:2233:4455:6677:213:20ff:feff:73aa/64 Scope:Global
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:6295 errors:0 dropped:6 overruns:0 frame:0
>> TX packets:2831 errors:0 dropped:0 overruns:0
>> carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:644421 (629.3 KiB) TX bytes:177616
>> (173.4 KiB)
>>
>> sit0 Link encap:UNSPEC HWaddr
>> 00-00-00-00-32-33-33-00-00-00-00-00-00-00-00-00
>> NOARP MTU:1480 Metric:1
>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:0
>> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
>>
>> wlan0 Link encap:Ethernet HWaddr 80:86:f2:8b:1d:cf
>> BROADCAST MULTICAST MTU:1500 Metric:1
>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
>>
>>
>> Thanks,
>> Kai
>>
>>
>>
>> On Tue, May 24, 2016 at 3:36 PM, Kai Kang
>> <kai.7.kang(a)gmail.com <mailto:kai.7.kang@gmail.com>> wrote:
>>
>> Hi,
>>
>> When I install a host, it fails with:
>>
>> 2016-05-24 07:00:01,749 ERROR
>> [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
>> (DefaultQuartzScheduler_Worker-4) [1bf36cd4] Host
>> 'node3' is set to Non-Operational, it is missing the
>> following networks: 'ovirtmgmt'
>> 2016-05-24 07:00:01,781 WARN
>>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler_Worker-4) [1bf36cd4] Correlation
>> ID: 1bf36cd4, Job ID:
>> db281e8f-67cc-441a-b44c-90b135e509bd, Call Stack: null,
>> Custom Event ID: -1, Message: Host node3 does not comply
>> with the cluster Default networks, the following
>> networks are missing on host: 'ovirtmgmt'
>>
>> After I click the "Hosts" -> subtab "Network
Interfaces"
>> -> "Setup Host Networks" on the web ui and drag
>> ovirtmgmt to "Assigned Logical" to link with eth0, then
>> activate host "node3" successfully.
>>
>> My question is how to make such manual operation
>> automatically? Then I can run some automatic tests.
>>
>> Thanks a lot.
>>
>> --Kai
>>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users
>
--------------C1E2D94265E0210FD5646D84
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Kai,<br>
<br>
The symptoms after removal of vdsm-hook-ovs look just like the
ovirtmgmt bridged network was not created. The log to look in
would be supervdsm.log, but it looks like yours has been rotated
after supervdsmd was restarted (the warning is benign). Can you
dig out "supervdsm.log.1"?<br>
<br>
OVS started to be supported by the hook initially (this was the
first attempt at supporting OVS). Currently, proper OVS support is
in development and should appear in 4.0 as a tech-preview.<br>
<br>
Thanks,<br>
Ondra (<a class="moz-txt-link-abbreviated"
href="mailto:osvoboda@redhat.com">osvoboda@redhat.com</a>)<br>
<br>
</p>
<div class="moz-cite-prefix">On 25.5.2016 11:01, Kai Kang
wrote:<br>
</div>
<blockquote
cite="mid:CADC9KmpRk=ZhEt==Q5nWaKGdS-NGbibCJmSp01CbK1Cx=utP0w@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, May 24, 2016 at 8:43 PM,
Ondřej Svoboda <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ondrej@svobodasoft.cz"
target="_blank"><a class="moz-txt-link-abbreviated"
href="mailto:ondrej@svobodasoft.cz">ondrej@svobodasoft.cz</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hi again,
(correcting myself a bit)<br>
<br>
are using the Open vSwitch hook intentionally? If not,
you can <code>yum remove vdsm-hook-ovs</code> for a
while. What is your exact VDSM version?<br>
</div>
</blockquote>
<div><br>
</div>
<div>My VDSM version is 4.17.24 and ovirt-engine is 3.6.4.
When after remove ovs hooks, it still fails with:</div>
<div><br>
</div>
<div> "Host node3 does not comply with the cluster Default
networks, the following networks are missing on host:
'ovirtmgmt'".</div>
<div><br>
</div>
<div>I checked vdsm.log it seems no erros there. And in
supervdsm.log, it shows warnings:</div>
<div><br>
</div>
<div>
<div>sourceRoute::INFO::2016-05-25
07:57:41,779::sourceroutethread::60::root::(process_IN_CLOSE_WRITE_filePath)
interface ovirtmgmt is not a libvirt interface</div>
<div>sourceRoute::WARNING::2016-05-25
07:57:41,780::utils::140::root::(rmFile) File:
/var/run/vdsm/trackedInterfaces/ovirtmgmt already
removed</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>What should I check next? Thanks.</div>
<div><br>
</div>
<div>I uploaded vdsm.log to:</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="https://gist.github.com/parr0tr1ver/1e38171b5d12cf773211015302...
</div>
<div><br>
</div>
<div>and supervdsm.log:</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="https://gist.github.com/parr0tr1ver/97805698a485f1cd49ded2b095...
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Also, can you give a little more context from your
vdsm.log, and also from supervdsm.log? I think the
vdsm.log failure is only related to stats reporting, and
is not the root problem.<br>
<br>
If you don't have any confidential information in the
logs (or you can remove it), I suggest that you open a
bug on product=vdsm at <a moz-do-not-send="true"
href="https://bugzilla.redhat.com/"
target="_blank">https://bugzilla.redhat.com/</a>
Have no fear about naming the bug, it can be renamed if
necessary.</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div>I find there is a bug on <a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1234867">h...;.
The "Target Milestone" to <span class=""
style="white-space:pre">"support ovs via vdsm hook" is
</span>ovirt-3.6.7.
Does that mean ovs does not work actually in previous
versions?</div>
<div><br>
</div>
<div>Thanks a lot.</div>
<div><br>
</div>
<div>--Kai</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div class="h5"><br>
<br>
Thanks, Ondra<br>
<blockquote type="cite"> </blockquote>
<br>
<blockquote type="cite">
<div>On 24.5.2016 11:47, Kai Kang wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I checked vdsm.log, it show error:</div>
<div><br>
</div>
<div>
<div>jsonrpc.Executor/0::DEBUG::2016-05-24
09:46:04,056::utils::671::root::(execCmd)
/usr/bin/taskset --cpu-list 0-7
/usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs
(cwd None)</div>
<div>jsonrpc.Executor/0::DEBUG::2016-05-24
09:46:04,136::utils::689::root::(execCmd)
FAILED: <err> = 'Traceback (most
recent call last):\n File
"/usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs",
line 72, in <module>\n main()\n
File
"/usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs",
line 66, in main\n
stats[\'network\'].update(ovs_networks_stats(stats[\'network\']))\nKeyError:
\'network\'\n\n'; <rc> =
2</div>
<div>jsonrpc.Executor/0::INFO::2016-05-24
09:46:04,136::hooks::98::root::(_runHooksDir)
Traceback (most recent call last):</div>
<div> File
"/usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs",
line 72, in <module></div>
<div> main()</div>
<div> File
"/usr/lib64/vdsm/vdsm/hooks/after_get_stats/50_ovs",
line 66, in main</div>
<div>
stats['network'].update(ovs_networks_stats(stats['network']))</div>
<div>KeyError: 'network'</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>I'm checking what wrong with it.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Kai</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, May 24, 2016
at 3:57 PM, Kai Kang <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:kai.7.kang@gmail.com"
target="_blank"><a
class="moz-txt-link-abbreviated"
href="mailto:kai.7.kang@gmail.com">kai.7.kang@gmail.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">And network configurations
on node:
<div><br>
</div>
<div>[root at ovirt-node] # brctl show<br>
</div>
<div>bridge name bridge id
STP enabled interfaces</div>
<div>docker0 8000.0242ae1de711
no</div>
<div>ovirtmgmt
8000.001320ff73aa no
eth0</div>
<div><br>
</div>
<div>[root at ovirt-node] # ifconfig -a<br>
</div>
<div>
<div>docker0 Link encap:Ethernet
HWaddr 02:42:ae:1d:e7:11</div>
<div> inet addr:172.17.0.1
Bcast:0.0.0.0 Mask:255.255.0.0</div>
<div> UP BROADCAST MULTICAST
MTU:1500 Metric:1</div>
<div> RX packets:0 errors:0
dropped:0 overruns:0 frame:0</div>
<div> TX packets:0 errors:0
dropped:0 overruns:0 carrier:0</div>
<div> collisions:0
txqueuelen:0</div>
<div> RX bytes:0 (0.0 B) TX
bytes:0 (0.0 B)</div>
<div><br>
</div>
<div>eth0 Link encap:Ethernet
HWaddr 00:13:20:ff:73:aa</div>
<div> UP BROADCAST RUNNING
MULTICAST MTU:1500 Metric:1</div>
<div> RX packets:6331
errors:0 dropped:0 overruns:0
frame:0</div>
<div> TX packets:2866
errors:0 dropped:0 overruns:0
carrier:0</div>
<div> collisions:0
txqueuelen:1000</div>
<div> RX bytes:762182 (744.3
KiB) TX bytes:210611 (205.6 KiB)</div>
<div> Interrupt:20
Memory:d1100000-d1120000</div>
<div><br>
</div>
<div>lo Link encap:Local
Loopback</div>
<div> inet addr:127.0.0.1
Mask:255.0.0.0</div>
<div> inet6 addr: ::1/128
Scope:Host</div>
<div> UP LOOPBACK RUNNING
MTU:65536 Metric:1</div>
<div> RX packets:36 errors:0
dropped:0 overruns:0 frame:0</div>
<div> TX packets:36 errors:0
dropped:0 overruns:0 carrier:0</div>
<div> collisions:0
txqueuelen:0</div>
<div> RX bytes:2478 (2.4 KiB)
TX bytes:2478 (2.4 KiB)</div>
<div><br>
</div>
<div>ovirtmgmt Link encap:Ethernet
HWaddr 00:13:20:ff:73:aa</div>
<div> inet
addr:128.224.165.170
Bcast:128.224.165.255
Mask:255.255.255.0</div>
<div> inet6 addr:
fe80::213:20ff:feff:73aa/64
Scope:Link</div>
<div> inet6 addr:
11:2233:4455:6677:213:20ff:feff:73aa/64
Scope:Global</div>
<div> UP BROADCAST RUNNING
MULTICAST MTU:1500 Metric:1</div>
<div> RX packets:6295
errors:0 dropped:6 overruns:0
frame:0</div>
<div> TX packets:2831
errors:0 dropped:0 overruns:0
carrier:0</div>
<div> collisions:0
txqueuelen:0</div>
<div> RX bytes:644421 (629.3
KiB) TX bytes:177616 (173.4 KiB)</div>
<div><br>
</div>
<div>sit0 Link encap:UNSPEC
HWaddr
00-00-00-00-32-33-33-00-00-00-00-00-00-00-00-00</div>
<div> NOARP MTU:1480
Metric:1</div>
<div> RX packets:0 errors:0
dropped:0 overruns:0 frame:0</div>
<div> TX packets:0 errors:0
dropped:0 overruns:0 carrier:0</div>
<div> collisions:0
txqueuelen:0</div>
<div> RX bytes:0 (0.0 B) TX
bytes:0 (0.0 B)</div>
<div><br>
</div>
<div>wlan0 Link encap:Ethernet
HWaddr 80:86:f2:8b:1d:cf</div>
<div> BROADCAST MULTICAST
MTU:1500 Metric:1</div>
<div> RX packets:0 errors:0
dropped:0 overruns:0 frame:0</div>
<div> TX packets:0 errors:0
dropped:0 overruns:0 carrier:0</div>
</div>
<div>
<div> collisions:0
txqueuelen:1000</div>
<div> RX bytes:0 (0.0 B) TX
bytes:0 (0.0 B)</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Kai</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, May
24, 2016 at 3:36 PM, Kai Kang <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:kai.7.kang@gmail.com"
target="_blank"><a
class="moz-txt-link-abbreviated"
href="mailto:kai.7.kang@gmail.com">kai.7.kang@gmail.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>When I install a host, it
fails with:</div>
<div><br>
</div>
<div>
<div>2016-05-24 07:00:01,749
ERROR
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(DefaultQuartzScheduler_Worker-4) [1bf36cd4] Host 'node3' is set to
Non-Operational, it is
missing the following
networks: 'ovirtmgmt'</div>
<div>2016-05-24 07:00:01,781
WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-4) [1bf36cd4] Correlation ID: 1bf36cd4,
Job ID:
db281e8f-67cc-441a-b44c-90b135e509bd,
Call Stack: null, Custom
Event ID: -1, Message:
Host node3 does not comply
with the cluster Default
networks, the following
networks are missing on
host: 'ovirtmgmt'</div>
</div>
<div><br>
</div>
<div>After I click the "Hosts"
-> subtab "Network
Interfaces" -> "Setup
Host Networks" on the web ui
and drag ovirtmgmt to
"Assigned Logical" to link
with eth0, then activate
host "node3"
successfully.</div>
<div><br>
</div>
<div>My question is how to
make such manual operation
automatically? Then I can
run some automatic tests.</div>
<div><br>
</div>
<div>Thanks a lot.</div>
<span><font
color="#888888">
<div><br>
</div>
<div>--Kai</div>
</font></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
</pre>
</blockquote>
<br>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------C1E2D94265E0210FD5646D84--
3103
days inactive
3103
days old
0 comments
1 participants
participants (1)
-
Ondřej Svoboda