[Users] Questions on ovirt 3.3 browser based spice/novnc working

Dead Horse deadhorseconsulting at gmail.com
Fri Aug 16 13:58:18 UTC 2013


I was just more curious about exactly what files/options  database
options/configurations in the engine had to be changed to disable SSL for
this and just allow for http. I am not quite 100% on what the engine option
"SSLEnabled" exactly disables SSL wise (EG: HTTP/VDSM?) or what effect the
SSL_ONLY option in the websocket configuration has (by default it is set to
false but only SSL works?).

Thus I am just curious on the underpinnings and how things are tied
together and cause/effect ;-)

- DHC


On Fri, Aug 16, 2013 at 2:42 AM, Frantisek Kobzik <fkobzik at redhat.com>wrote:

> I'll try to resolve that soon.
>
> Thanks,
> F.
>
> ----- Original Message -----
> From: "Alon Bar-Lev" <alonbl at redhat.com>
> To: "Frantisek Kobzik" <fkobzik at redhat.com>
> Cc: "Dead Horse" <deadhorseconsulting at gmail.com>, "users" <users at ovirt.org
> >
> Sent: Friday, August 16, 2013 9:04:09 AM
> Subject: Re: [Users] Questions on ovirt 3.3 browser based spice/novnc
> working
>
>
>
> ----- Original Message -----
> > From: "Frantisek Kobzik" <fkobzik at redhat.com>
> > To: "Alon Bar-Lev" <alonbl at redhat.com>
> > Cc: "Dead Horse" <deadhorseconsulting at gmail.com>, "users" <
> users at ovirt.org>
> > Sent: Friday, August 16, 2013 9:58:27 AM
> > Subject: Re: [Users] Questions on ovirt 3.3 browser based spice/novnc
> working
> >
> > Hi,
> >
> > exactly - the fact about the vdc option is true.
> >
> > (and I think we also have to allow serving novnc/spice-html5 pages using
> > plain http. afaik now apache or jboss forces you to https).
>
> No... just a setting for the proxy.
> As the html files them-selves comes from same location of where user is on.
> Can you please handle that?
>
> >
> > Regards,
> > F.
> >
> > ----- Original Message -----
> > From: "Alon Bar-Lev" <alonbl at redhat.com>
> > To: "Dead Horse" <deadhorseconsulting at gmail.com>
> > Cc: "users" <users at ovirt.org>, "Frantisek Kobzik" <fkobzik at redhat.com>
> > Sent: Friday, August 16, 2013 8:45:05 AM
> > Subject: Re: [Users] Questions on ovirt 3.3 browser based spice/novnc
> working
> >
> >
> >
> > ----- Original Message -----
> > > From: "Dead Horse" <deadhorseconsulting at gmail.com>
> > > To: "Alon Bar-Lev" <alonbl at redhat.com>
> > > Cc: "users" <users at ovirt.org>, "Frantisek Kobzik" <fkobzik at redhat.com>
> > > Sent: Friday, August 16, 2013 3:55:28 AM
> > > Subject: Re: [Users] Questions on ovirt 3.3 browser based spice/novnc
> > > working
> > >
> > > Curiously if one wanted the disable the need to download the Server CA
> > > certificate what are the changes needed to do so? (Realizing the
> security
> > > implications)
> >
> > I do not understand, what alternative do you propose?
> >
> > You can disable ssl.... but Frantisek, we need a vdc option for that so
> url
> > will contain http or https.
> >
> > >
> > >
> > > On Fri, Aug 2, 2013 at 2:49 PM, Alon Bar-Lev <alonbl at redhat.com>
> wrote:
> > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Dead Horse" <deadhorseconsulting at gmail.com>
> > > > > To: "Alon Bar-Lev" <alonbl at redhat.com>
> > > > > Cc: "users" <users at ovirt.org>
> > > > > Sent: Friday, August 2, 2013 10:39:48 PM
> > > > > Subject: Re: [Users] Questions on ovirt 3.3 browser based
> spice/novnc
> > > > working
> > > > >
> > > > > Thanks Alon,
> > > > > That did the trick. Is there any way to get the engine to push this
> > > > > cert
> > > > to
> > > > > a first time visitor by default?
> > > > > - DHC
> > > >
> > > > Well, it is actually depend on browser behavior... Internet Explorer
> does
> > > > allow you to trust the root.
> > > >
> > > > I could not find such option in firefox.
> > > >
> > > > Frantisek:
> > > >
> > > > Maybe we can have the link for the ca certificate so people can
> press it
> > > > to establish trust.
> > > >
> > > > Have you tried to perform XMLHttpRequest and see if you get some
> error we
> > > > can use to warn user?
> > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 2, 2013 at 1:18 AM, Alon Bar-Lev <alonbl at redhat.com>
> wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > ----- Original Message -----
> > > > > > > From: "Dead Horse" <deadhorseconsulting at gmail.com>
> > > > > > > To: "Alon Bar-Lev" <alonbl at redhat.com>
> > > > > > > Cc: "users" <users at ovirt.org>
> > > > > > > Sent: Thursday, August 1, 2013 11:06:11 PM
> > > > > > > Subject: Re: [Users] Questions on ovirt 3.3 browser based
> > > > > > > spice/novnc
> > > > > > working
> > > > > > >
> > > > > > > Attached Firefox and Chrome screenshots of Certificates.
> > > > > > > errors thrown by websockify
> > > > > > > Firefox: 1: handler exception: [Errno 1] _ssl.c:1359:
> > > > error:14094418:SSL
> > > > > > > routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
> > > > > > > Chrome: 11: handler exception: WSRequestHandler instance has no
> > > > attribute
> > > > > > > 'last_code'
> > > > > > >
> > > > > > > For Firefox it looks like firefox needs a bit of proding to
> get it
> > > > > > > to
> > > > > > > accept the Websocket CA Cert:
> > > > > > > https://github.com/kanaka/websockify/issues/34
> > > > > > >
> > > > > > > The error generated by chrome seems to be a websockify issue:
> > > > > > > https://github.com/kanaka/noVNC/issues/86
> > > > > > >
> https://github.com/kanaka/websockify/issues/22#issuecomment-3263065
> > > > > > > https://github.com/kanaka/noVNC/issues/177
> > > > > > >
> > > > > > > In any event I got both Chrome and Firefox working by manually
> > > > browsing
> > > > > > to:
> > > > > > > https://ENGINEFQDN:6100 and accepting the self signed cert
> > > > > >
> > > > > > This is because your browser does not support the CA.
> > > > > > Please go to:
> > > > > >
> > > > > > http://engine/ca.crt
> > > > > >
> > > > > > And install that certificate as trusted, remove the explicit
> trust
> > > > > > you
> > > > > > have added, and try again.
> > > > > >
> > > > > > >
> > > > > > > Not pretty but it worked.
> > > > > > >
> > > > > > > - DHC
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Aug 1, 2013 at 2:08 PM, Alon Bar-Lev <
> alonbl at redhat.com>
> > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ----- Original Message -----
> > > > > > > > > From: "Dead Horse" <deadhorseconsulting at gmail.com>
> > > > > > > > > To: "Alon Bar-Lev" <alonbl at redhat.com>
> > > > > > > > > Cc: "users" <users at ovirt.org>
> > > > > > > > > Sent: Thursday, August 1, 2013 9:59:14 PM
> > > > > > > > > Subject: Re: [Users] Questions on ovirt 3.3 browser based
> > > > spice/novnc
> > > > > > > > working
> > > > > > > > >
> > > > > > > > > That did the trick for getting the websocket proxy
> configured (
> > > > > > > > > i
> > > > > > backed
> > > > > > > > > out all my changes prior to running engine-setup). I do
> notice
> > > > that
> > > > > > it
> > > > > > > > > still seems to leave the ovirt-websocket-proxy.conf in it's
> > > > default
> > > > > > state
> > > > > > > > > and makes no dedications to it. Instead it generated
> > > > > > > > >
> /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
> > > > > > > > >
> > > > > > > > > I also noted engine setup generated:
> > > > > > > > > /etc/pki/ovirt-engine/certs/websocket-proxy.cer
> > > > > > > > > /etc/pki/ovirt-engine/keys/websocket-proxy.p12
> > > > > > > > > /etc/pki/ovirt-engine/keys/websocket-proxy.key.nopass
> > > > > > > > > /etc/pki/ovirt-engine/requests/websocket-proxy.req
> > > > > > > > >
> > > > > > > > > None the less still neither spice nor novnc will connect. I
> > > > > > > > > tried
> > > > > > > > changing
> > > > > > > > > Engine:6100 to EngineIP:6100 so that IP would be used
> instead.
> > > > > > However
> > > > > > > > > using either the FQDN or IP still yielded the same results.
> > > > > > > >
> > > > > > > > You should not touch anything... all should be configured...
> > > > > > > > Make sure your browser trust the *CA* of the engine and not
> the
> > > > engine
> > > > > > > > certificate directly.
> > > > > > > > And try to open vnc console via webadmin.
> > > > > > > >
> > > > > > > > > There was nothing interesting in the logs either. I do
> notice
> > > > that
> > > > > > whilst
> > > > > > > > > the websocket-proxy service is running I never see an
> > > > > > > > > websockify
> > > > > > > > processes
> > > > > > > > > but instead in /var/log/messages I see:
> > > > > > > > > Aug  1 13:44:10 ovirtfoo ovirt-websocket-proxy.py[435]: 11:
> > > > handler
> > > > > > > > > exception: [Errno 1] _ssl.c:1359: error:14094418:SSL
> > > > > > > > > routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
> > > > > > > > >
> > > > > > > > > Thus I changed SSL_ONLY=True to SSL_ONLY=False in
> > > > > > > > >
> /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
> > > > > > > > > and
> > > > > > > > restarted
> > > > > > > > > engine and websocket-proxy
> > > > > > > > > No dice it still generated the same error as above during
> an
> > > > > > attempted
> > > > > > > > > connection to /var/log/messages
> > > > > > > > >
> > > > > > > > > I also not the following error message at VM power off
> (albeit
> > > > > > > > > I
> > > > am
> > > > > > > > > guessing it has nothing to do with this issue):
> > > > > > > > > 2013-08-01 13:41:03,742 ERROR
> > > > > > > > > [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand]
> > > > > > (pool-6-thread-50)
> > > > > > > > > [304efb3e] VDS::destroy Failed destroying vm
> > > > > > > > > fec3260c-871a-4fbe-a006-9eee4fbfbbcc in vds =
> > > > > > > > > 5713e5c8-6252-4bce-a3f6-bbd8e1e6eb57 : ovirtnodefoo, error
> =
> > > > > > > > >
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
> > > > > > > > > VDSGenericException: VDSErrorException: Failed to
> DestroyVDS,
> > > > error =
> > > > > > > > > Unexpected exception
> > > > > > > > >
> > > > > > > > > - DHC
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Aug 1, 2013 at 1:07 PM, Alon Bar-Lev
> > > > > > > > > <alonbl at redhat.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > If you install the proxy on the engine machine you just
> need:
> > > > > > > > > >
> > > > > > > > > > # yum install ovirt-engine-websocket-proxy
> > > > > > > > > > # engine-setup
> > > > > > > > > >
> > > > > > > > > > then answer yes when prompt if you like to configure
> > > > > > > > > > websocket
> > > > > > proxy.
> > > > > > > > > >
> > > > > > > > > > you can execute engine-setup again even if you already
> > > > installed.
> > > > > > > > > >
> > > > > > > > > > ----- Original Message -----
> > > > > > > > > > > From: "Dead Horse" <deadhorseconsulting at gmail.com>
> > > > > > > > > > > To: "<users at ovirt.org>" <users at ovirt.org>
> > > > > > > > > > > Sent: Thursday, August 1, 2013 9:01:47 PM
> > > > > > > > > > > Subject: [Users] Questions on ovirt 3.3 browser based
> > > > spice/novnc
> > > > > > > > working
> > > > > > > > > > >
> > > > > > > > > > > After Referencing:
> > > > > > > > > > > http://www.ovirt.org/Features/noVNC_console
> > > > > > > > > > > http://www.ovirt.org/Features/SpiceHTML5
> > > > > > > > > > >
> > > > > > > > > > > and looking at some of the related engine code.
> > > > > > > > > > >
> > > > > > > > > > > I am still attempting to get the spice/novnc browser
> based
> > > > > > consoles
> > > > > > > > to
> > > > > > > > > > work.
> > > > > > > > > > >
> > > > > > > > > > > I am working from a build from master yesterday I used
> to
> > > > upgrade
> > > > > > > > over a
> > > > > > > > > > > previous 3.3 master build from about a month back.
> > > > > > > > > > >
> > > > > > > > > > > VDSM version on host is 4.12.0 built minutes ago.
> > > > > > > > > > >
> > > > > > > > > > > I have installed and configured the websocket proxy
> like
> > > > > > > > > > > so:
> > > > > > > > > > >
> > > > > > > > > > > Set WebSocketProxy to engine ENGINEIP port 6100
> > > > > > > > > > > engine-config -s WebSocketProxy=ENGINEIP:6100
> > > > > > > > > > >
> > > > > > > > > > > /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh
> > > > > > > > --name=websocket-proxy
> > > > > > > > > > > --password=install
> --subject="/C=US/O=DHC/CN=ENGINEFQDN"
> > > > > > > > > > >
> > > > > > > > > > > This generates:
> > > > > > > > > > > /etc/pki/ovirt-engine/keys/websocket-proxy.p12
> > > > > > > > > > > /etc/pki/ovirt-engine/certs/websocket-proxy.cer
> > > > > > > > > > > /etc/pki/ovirt-engine/requests/websocket-proxy.req
> > > > > > > > > > >
> > > > > > > > > > > However it does not generate the key that websockify
> wants
> > > > so we
> > > > > > do:
> > > > > > > > > > > openssl pkcs12 -in websocket-proxy.p12 -nocerts -nodes
> -out
> > > > > > > > > > > /etc/pki/ovirt-engine/keys/websocket-proxy.key
> > > > > > > > > > >
> > > > > > > > > > > The configuration of ovirt-websocket-proxy:
> > > > > > > > > > > PROXY_HOST=*
> > > > > > > > > > > PROXY_PORT=6100
> > > > > > > > > > > SOURCE_IS_IPV6=False
> > > > > > > > > > >
> > > > SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/websocket-proxy.cer
> > > > > > > > > > > SSL_KEY=/etc/pki/ovirt-engine/keys/websocket-proxy.key
> > > > > > > > > > > FORCE_DATA_VERIFICATION=False
> > > > > > > > > > >
> > > > CERT_FOR_DATA_VERIFICATION=/etc/pki/ovirt-engine/certs/engine.cer
> > > > > > > > > > > SSL_ONLY=True
> > > > > > > > > > > TRACE_ENABLE=False
> > > > > > > > > > > TRACE_FILE=
> > > > > > > > > > > ENGINE_USR="/usr/share/ovirt-engine"
> > > > > > > > > > >
> > > > > > > > > > > Install spice-html5
> > > > > > > > > > > git clone
> > > > > > http://anongit.freedesktop.org/git/spice/spice-html5.git
> > > > > > > > > > > mv spice-html5 /usr/share
> > > > > > > > > > >
> > > > > > > > > > > Test spice:
> > > > > > > > > > > In Webadmin UI we set create a VM, set display as
> spice,
> > > > start it
> > > > > > > > and set
> > > > > > > > > > > it's console to spice-html5.
> > > > > > > > > > > Result spice-html client opens in a new tab but does
> not
> > > > connect.
> > > > > > > > > > >
> > > > > > > > > > > From engine.log:
> > > > > > > > > > > 2013-08-01 12:49:52,352 INFO
> > > > > > > > > > [org.ovirt.engine.core.bll.SetVmTicketCommand]
> > > > > > > > > > > (ajp--127.0.0.1-8702-9) Running command:
> SetVmTicketCommand
> > > > > > internal:
> > > > > > > > > > false.
> > > > > > > > > > > Entities affected : ID:
> > > > > > > > > > > fec3260c-871a-4fbe-a006-9eee4fbfbbcc
> > > > > > Type: VM
> > > > > > > > > > > 2013-08-01 12:49:52,371 INFO
> > > > > > > > > > >
> > > > [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand]
> > > > > > > > > > > (ajp--127.0.0.1-8702-9) START,
> > > > SetVmTicketVDSCommand(HostName =
> > > > > > > > > > > ovirtnodefoo, HostId =
> > > > > > > > > > > 5713e5c8-6252-4bce-a3f6-bbd8e1e6eb57,
> > > > > > > > > > > vmId=fec3260c-871a-4fbe-a006-9eee4fbfbbcc,
> > > > ticket=TKfzUQJLLrUI,
> > > > > > > > > > > validTime=120,m userName=admin at internal,
> > > > > > > > > > > userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id:
> > > > 5d258049
> > > > > > > > > > > 2013-08-01 12:49:52,445 INFO
> > > > > > > > > > >
> > > > [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand]
> > > > > > > > > > > (ajp--127.0.0.1-8702-9) FINISH, SetVmTicketVDSCommand,
> log
> > > > id:
> > > > > > > > 5d258049
> > > > > > > > > > >
> > > > > > > > > > > Test novnc:
> > > > > > > > > > > In Webadmin UI we set create a VM, set display as VNC,
> > > > > > > > > > > start
> > > > it
> > > > > > and
> > > > > > > > set
> > > > > > > > > > it's
> > > > > > > > > > > console to novnc.
> > > > > > > > > > > Result novnc client opens in a new tab but does not
> > > > > > > > > > > connect,
> > > > but
> > > > > > does
> > > > > > > > > > display
> > > > > > > > > > > error: "Server disconnected (code: 1006)
> > > > > > > > > > >
> > > > > > > > > > > From engine.log:
> > > > > > > > > > > 2013-08-01 12:50:44,800 INFO
> > > > > > > > > > [org.ovirt.engine.core.bll.SetVmTicketCommand]
> > > > > > > > > > > (ajp--127.0.0.1-8702-9) Running command:
> SetVmTicketCommand
> > > > > > internal:
> > > > > > > > > > false.
> > > > > > > > > > > Entities affected : ID:
> > > > > > > > > > > fec3260c-871a-4fbe-a006-9eee4fbfbbcc
> > > > > > Type: VM
> > > > > > > > > > > 2013-08-01 12:50:44,833 INFO
> > > > > > > > > > >
> > > > [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand]
> > > > > > > > > > > (ajp--127.0.0.1-8702-9) START,
> > > > SetVmTicketVDSCommand(HostName =
> > > > > > > > > > > ovirtnodefoo, HostId =
> > > > > > > > > > > 5713e5c8-6252-4bce-a3f6-bbd8e1e6eb57,
> > > > > > > > > > > vmId=fec3260c-871a-4fbe-a006-9eee4fbfbbcc,
> > > > ticket=IPWOWh6U9erd,
> > > > > > > > > > > validTime=120,m userName=admin at internal,
> > > > > > > > > > > userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id:
> > > > > > > > > > > bff6161
> > > > > > > > > > > 2013-08-01 12:50:44,917 INFO
> > > > > > > > > > >
> > > > [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand]
> > > > > > > > > > > (ajp--127.0.0.1-8702-9) FINISH, SetVmTicketVDSCommand,
> log
> > > > id:
> > > > > > > > bff6161
> > > > > > > > > > >
> > > > > > > > > > > I verified connection of both the spice/vnc console
> > > > > > > > > > > directly
> > > > at
> > > > > > the
> > > > > > > > host
> > > > > > > > > > > level with a quick connect via virt-viewer.
> > > > > > > > > > >
> > > > > > > > > > > A quick scan with nmap of engine and host to verify
> sockets
> > > > are
> > > > > > open:
> > > > > > > > > > >
> > > > > > > > > > > Nmap scan report for engine
> > > > > > > > > > > Host is up (0.0042s latency).
> > > > > > > > > > > Not shown: 995 closed ports
> > > > > > > > > > > PORT STATE SERVICE
> > > > > > > > > > > 22/tcp open ssh
> > > > > > > > > > > 80/tcp open http
> > > > > > > > > > > 111/tcp open rpcbind
> > > > > > > > > > > 443/tcp open https
> > > > > > > > > > > 6100/tcp open synchronet-db
> > > > > > > > > > >
> > > > > > > > > > > Nmap scan report for host
> > > > > > > > > > > Host is up (0.0045s latency).
> > > > > > > > > > > Not shown: 997 closed ports
> > > > > > > > > > > PORT STATE SERVICE
> > > > > > > > > > > 22/tcp open ssh
> > > > > > > > > > > 111/tcp open rpcbind
> > > > > > > > > > > 5900/tcp open vnc
> > > > > > > > > > >
> > > > > > > > > > > For grins I stopped the websocket proxy and manually
> > > > > > > > > > > started
> > > > a
> > > > > > > > websockify
> > > > > > > > > > > like so:
> > > > > > > > > > > websockify 3.57.111.11:6100 3.57.111.12:5900
> > > > > > > > > > > --cert=/etc/pki/ovirt-engine/certs/websocket-proxy.cer
> > > > > > > > > > > --key=/etc/pki/ovirt-engine/keys/websocket-proxy.key
> > > > > > > > > > >
> > > > > > > > > > > WARNING: no 'numpy' module, HyBi protocol is slower or
> > > > disabled
> > > > > > > > > > > WebSocket server settings:
> > > > > > > > > > > - Listen on ENGINEIP:6100
> > > > > > > > > > > - Flash security policy server
> > > > > > > > > > > - SSL/TLS support
> > > > > > > > > > > - proxying from ENGINEIP:6100 to HOSTIP:5900
> > > > > > > > > > >
> > > > > > > > > > > Attempting another connection via
> > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> https://ENGINEFQDN//ovirt-engine-novnc-main.html?host=ENGINEIP&port=6100
> > > > > > > > > > > results in:
> > > > > > > > > > >
> > > > > > > > > > > 1: handler exception: [Errno 1] _ssl.c:1359:
> > > > error:14094418:SSL
> > > > > > > > > > > routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I should also note in case it matters that the
> > > > SSLEnabled=false,
> > > > > > and
> > > > > > > > > > > EnableSpiceRootCertificateValidation are both set as
> false
> > > > are
> > > > > > set
> > > > > > > > in my
> > > > > > > > > > > engine options.
> > > > > > > > > > >
> > > > > > > > > > > Am I doing something wrong here, I don't see any reason
> > > > > > > > > > > this
> > > > > > should
> > > > > > > > not
> > > > > > > > > > work?
> > > > > > > > > > >
> > > > > > > > > > > - DHC
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > Users mailing list
> > > > > > > > > > > Users at ovirt.org
> > > > > > > > > > > http://lists.ovirt.org/mailman/listinfo/users
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20130816/59e8e43d/attachment-0001.html>


More information about the Users mailing list