[ovirt-devel] Can't add DC with API v4 - client issue

Juan Hernández jhernand at redhat.com
Thu Nov 10 08:47:10 UTC 2016


On 11/10/2016 09:33 AM, Yaniv Kaul wrote:
> 
> 
> On Wed, Nov 9, 2016 at 8:05 PM, Juan Hernández <jhernand at redhat.com
> <mailto:jhernand at redhat.com>> wrote:
> 
>     On 11/09/2016 11:12 AM, Yaniv Kaul wrote:
>     >
>     >
>     > On Sat, Oct 15, 2016 at 1:04 AM, Ravi Nori <rnori at redhat.com <mailto:rnori at redhat.com>
>     > <mailto:rnori at redhat.com <mailto:rnori at redhat.com>>> wrote:
>     >
>     >     Also can you please try following command to directly obtain token
>     >     from SSO. Can replace engine with FQDN and IP to see if both work
>     >
>     >     curl -v -k -H "Accept: application/json"
>     >     'https://<engine>:443/ovirt-engine/sso/oauth/token?grant_type=password&username=admin at internal&password=123&scope=ovirt-app-api'
>     >
>     >     You should see output similar to the one below
>     >
>     >     {"access_token":"K0sBa0D3rLtmNTdMJ-Q4FzOgCtGGY2cSFSCwbLkG94te9nDdmEzHSizsFaOeNMdwOziIv3l2-Uqm8bxWkMpwMA","scope":"ovirt-app-api
>     >     ovirt-ext=token-info:authz-search
>     >     ovirt-ext=token-info:public-authz-search
>     >     ovirt-ext=token-info:validate","exp":-381399824,"token_type":"bearer"}
>     >
>     >
>     > Sorry it took me so long to get back to it, but here it is:
>     > {"access_token":"eA8w0DaapkKAQ8tfHakzA-R0l-mjD_CsTlAqBaH4iVVjXxQN33poXzt9UhPJLxMU8YOvVNX6LICcxL1EeAiAlw","scope":"ovirt-app-api
>     > ovirt-ext=token-info:authz-search
>     > ovirt-ext=token-info:public-authz-search
>     > ovirt-ext=token-info:validate","exp":["java.lang.Long",1479290132000],"token_type":"bearer"}
>     >
> 
>     That "java.lang.Long" there is an error, but not related to this
>     problem, as the SDK doesn't use the "exp" attribute. I guess it is a
>     side effect of the recent change to use "long" instead of "int", looks
>     like the JSON library used in the engine doesn't like longs.
> 
>     > And here's the difference between the SDK and the manual curl command in
>     > ssl_access log:
>     > 192.168.201.1 - - [09/Nov/2016:04:52:19 -0500] "POST
>     > /ovirt-engine/sso/oauth/token HTTP/1.1" 404 74
>     > 192.168.201.1 - - [09/Nov/2016:04:55:32 -0500] "GET
>     > /ovirt-engine/sso/oauth/token?grant_type=password&username=admin at internal&password=123&scope=ovirt-app-api
>     > HTTP/1.1" 200 295
>     >
> 
>     That difference is by design. The SDK uses POST to avoid sending the
>     credentials (specially the password) as a query parameter, as that is
>     most probably logged and archived.
> 
>     We discovered recently an issue with the Python SDK, due to a bug in the
>     "pycurl" library:
> 
>       Debug mode raises UnicodeDecodeError: 'utf8' codec can't decode byte
>     0x8d in position 7: invalid start byte
>       https://bugzilla.redhat.com/1392878
>     <https://bugzilla.redhat.com/1392878>
> 
>     It isn't exactly the same problem, but as the cause of that bug is a
>     pointer that is used after releasing, it can cause all kinds of strange
>     effects.
> 
>     Please try the latest build of the SDK.
> 
> 
> I thought I was:
> python-ovirt-engine-sdk4-4.1.0-0.1.a0.20161108gitaad5627.fc24.x86_64
>  

Yes, that build contains the fix, so we can discard it as the source of
your problem.

> 
> 
>     >
>     >
>     >     Thanks
>     >
>     >     Ravi
>     >
>     >     On Fri, Oct 14, 2016 at 4:00 PM, Yaniv Kaul <ykaul at redhat.com <mailto:ykaul at redhat.com>
>     >     <mailto:ykaul at redhat.com <mailto:ykaul at redhat.com>>> wrote:
>     >
>     >         On Oct 14, 2016 7:13 PM, "Ravi Nori" <rnori at redhat.com <mailto:rnori at redhat.com>
>     >         <mailto:rnori at redhat.com <mailto:rnori at redhat.com>>> wrote:
>     >         >
>     >         > SSO configuration looks good.
>     >         >
>     >         > Can you please share any additional httpd configuration in /etc/httpd/conf.d. Anything to do with LocationMatch for ovirt-engine urls.
>     >
>     >         This is a standard ovirt-system-tests on Lago installation,
>     >         nothing out of the ordinary,  but I'll check.
>     >         Y.
>     >
>     >         >
>     >         > On Fri, Oct 14, 2016 at 12:52 PM, Yaniv Kaul <ykaul at redhat.com <mailto:ykaul at redhat.com>
>     >         <mailto:ykaul at redhat.com <mailto:ykaul at redhat.com>>> wrote:
>     >         >>
>     >         >>
>     >         >>
>     >         >> On Fri, Oct 14, 2016 at 3:50 PM, Ravi Nori <rnori at redhat.com <mailto:rnori at redhat.com>
>     >         <mailto:rnori at redhat.com <mailto:rnori at redhat.com>>> wrote:
>     >         >>>
>     >         >>> Hi Yaniv,
>     >         >>>
>     >         >>> Can you check the output of
>     >         https:://<engine>/ovirt-engine/sso/status in your browser and
>     >         see if the SSO service is active.
>     >         >>>
>     >         >>> If SSO is deployed, you should see an output similar
>     to the
>     >         one below. Also are you able to login to webadmin using the
>     >         browser?
>     >         >>
>     >         >>
>     >         >> I am able to login using the webui.
>     >         >>
>     >         >>>
>     >         >>>
>     >         >>> {"status_description":"SSO Webapp
>     >         Deployed","version":"0","status":"active"}
>     >         >>
>     >         >>
>     >         >> Indeed:
>     >         >> {"status_description":"SSO Webapp
>     >         Deployed","version":"0","status":"active"}
>     >         >>
>     >         >> (not sure what 'version 0' means?)
>     >         >>
>     >         >>>
>     >         >>>
>     >         >>> Please share the content of
>     >         /etc/ovirt-engine/engine.conf.d/11-setup-sso.conf
>     >         >>
>     >         >>
>     >         >> [root at lago-basic-suite-master-engine ~]# cat
>     >         /etc/ovirt-engine/engine.conf.d/11-setup-sso.conf
>     >         >> ENGINE_SSO_CLIENT_ID="ovirt-engine-core"
>     >         >> ENGINE_SSO_CLIENT_SECRET="bsOabtD7gE2McwLe80P109UV800XLx4O"
>     >         >>
>     ENGINE_SSO_AUTH_URL="https://${ENGINE_FQDN}:443/ovirt-engine/sso"
>     >         >>
>     >       
>      ENGINE_SSO_SERVICE_URL="https://localhost:443/ovirt-engine/sso
>     <https://localhost:443/ovirt-engine/sso>
>     >         <https://localhost:443/ovirt-engine/sso
>     <https://localhost:443/ovirt-engine/sso>>"
>     >         >> ENGINE_SSO_SERVICE_SSL_VERIFY_HOST=false
>     >         >> ENGINE_SSO_SERVICE_SSL_VERIFY_CHAIN=true
>     >         >> SSO_ALTERNATE_ENGINE_FQDNS=""
>     >         >> SSO_ENGINE_URL="https://${ENGINE_FQDN}:443/ovirt-engine/"
>     >         >>
>     >         >>
>     >         >> Thanks,
>     >         >> Y.
>     >         >>
>     >         >>
>     >         >>>
>     >         >>>
>     >         >>> Thanks
>     >         >>>
>     >         >>> Ravi
>     >         >>>
>     >         >>>
>     >         >>>
>     >         >>>
>     >         >>>
>     >         >>> On Fri, Oct 14, 2016 at 7:57 AM, Juan Hernández
>     >         <jhernand at redhat.com <mailto:jhernand at redhat.com>
>     <mailto:jhernand at redhat.com <mailto:jhernand at redhat.com>>> wrote:
>     >         >>>>
>     >         >>>> On 10/14/2016 01:45 PM, Yaniv Kaul wrote:
>     >         >>>> >
>     >         >>>> >
>     >         >>>> > On Thu, Oct 13, 2016 at 11:13 AM, Juan Hernández
>     >         <jhernand at redhat.com <mailto:jhernand at redhat.com>
>     <mailto:jhernand at redhat.com <mailto:jhernand at redhat.com>>
>     >         >>>> > <mailto:jhernand at redhat.com
>     <mailto:jhernand at redhat.com>
>     >         <mailto:jhernand at redhat.com
>     <mailto:jhernand at redhat.com>>>> wrote:
>     >         >>>> >
>     >         >>>> >     On 10/13/2016 12:04 AM, Yaniv Kaul wrote:
>     >         >>>> >     > On Fri, Oct 7, 2016 at 10:44 PM, Yaniv Kaul
>     >         <ykaul at redhat.com <mailto:ykaul at redhat.com>
>     <mailto:ykaul at redhat.com <mailto:ykaul at redhat.com>>
>     >         <mailto:ykaul at redhat.com <mailto:ykaul at redhat.com>
>     <mailto:ykaul at redhat.com <mailto:ykaul at redhat.com>>>
>     >         >>>> >     > <mailto:ykaul at redhat.com
>     <mailto:ykaul at redhat.com> <mailto:ykaul at redhat.com
>     <mailto:ykaul at redhat.com>>
>     >         <mailto:ykaul at redhat.com <mailto:ykaul at redhat.com>
>     <mailto:ykaul at redhat.com <mailto:ykaul at redhat.com>>>>> wrote:
>     >         >>>> >     >
>     >         >>>> >     >     I'm trying on FC24, using
>     >         >>>> >     >
>     >         >>>> >
>     >       
>      python-ovirt-engine-sdk4-4.1.0-0.0.20161003git056315d.fc24.x86_64 to
>     >         >>>> >     >     add a DC, and failing - against master. The
>     >         client is unhappy:
>     >         >>>> >     >     File
>     >         >>>> >     >
>     >         >>>> >
>     >       
>      "/home/ykaul/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py",
>     >         >>>> >     >     line 98, in add_dc4
>     >         >>>> >     >
>     >         
>     version=sdk4.types.Version(major=DC_VER_MAJ,minor=DC_VER_MIN),
>     >         >>>> >     >       File
>     >         "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py",
>     >         >>>> >     >     line 4347, in add
>     >         >>>> >     >         response = self._connection.send(request)
>     >         >>>> >     >       File
>     >         "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py",
>     >         >>>> >     >     line 276, in send
>     >         >>>> >     >         return self.__send(request)
>     >         >>>> >     >       File
>     >         "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py",
>     >         >>>> >     >     line 298, in __send
>     >         >>>> >     >         self._sso_token =
>     self._get_access_token()
>     >         >>>> >     >       File
>     >         "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py",
>     >         >>>> >     >     line 460, in _get_access_token
>     >         >>>> >     >         sso_response =
>     >         self._get_sso_response(self._sso_url,
>     >         >>>> >     post_data)
>     >         >>>> >     >       File
>     >         "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py",
>     >         >>>> >     >     line 498, in _get_sso_response
>     >         >>>> >     >         return
>     >         json.loads(body_buf.getvalue().decode('utf-8'))
>     >         >>>> >     >       File
>     "/usr/lib64/python2.7/json/__init__.py",
>     >         line 339, in loads
>     >         >>>> >     >         return _default_decoder.decode(s)
>     >         >>>> >     >       File
>     "/usr/lib64/python2.7/json/decoder.py",
>     >         line 364, in decode
>     >         >>>> >     >         obj, end = self.raw_decode(s, idx=_w(s,
>     >         0).end())
>     >         >>>> >     >       File
>     "/usr/lib64/python2.7/json/decoder.py",
>     >         line 382, in
>     >         >>>> >     raw_decode
>     >         >>>> >     >         raise ValueError("No JSON object could be
>     >         decoded")
>     >         >>>> >     >     ValueError: No JSON object could be decoded
>     >         >>>> >     >
>     >         >>>> >     >
>     >         >>>> >     >     Surprisingly, I now can't find that RPM
>     of this
>     >         SDK in
>     >         >>>> >     >     resources.ovirt.org
>     <http://resources.ovirt.org>
>     >         <http://resources.ovirt.org> <http://resources.ovirt.org>
>     >         >>>> >     <http://resources.ovirt.org> now.
>     >         >>>> >     >
>     >         >>>> >     >     I've tried
>     >         >>>> >     >     with
>     >         >>>> >
>     >         
>     http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm>
>     >       
>      <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm>>
>     >         >>>> >
>     >         
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm>
>     >       
>      <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm>>>
>     >         >>>> >     >
>     >         >>>> >
>     >       
>      <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm>
>     >       
>      <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm>>
>     >         >>>> >
>     >         
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm>
>     >       
>      <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm
>     <http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004gitf94eeb5.fc24.x86_64.rpm>>>>
>     >         >>>> >     >
>     >         >>>> >     >     - same result.
>     >         >>>> >     >
>     >         >>>> >     >     Did not see anything obvious on server or
>     >         engine logs.
>     >         >>>> >     >     The code:
>     >         >>>> >     >     def add_dc4(api):
>     >         >>>> >     >         nt.assert_true(api != None)
>     >         >>>> >     >         dcs_service =
>     >         api.system_service().data_centers_service()
>     >         >>>> >     >         nt.assert_true(
>     >         >>>> >     >             dc = dcs_service.add(
>     >         >>>> >     >                 sdk4.types.DataCenter(
>     >         >>>> >     >                     name=DC_NAME4,
>     >         >>>> >     >                     description='APIv4 DC',
>     >         >>>> >     >                     local=False,
>     >         >>>> >     >
>     >         >>>> >     >
>     >         
>     version=sdk4.types.Version(major=DC_VER_MAJ,minor=DC_VER_MIN),
>     >         >>>> >     >                 ),
>     >         >>>> >     >             )
>     >         >>>> >     >         )
>     >         >>>> >     >
>     >         >>>> >     >
>     >         >>>> >     >     And the api object is from:
>     >         >>>> >     >                 return sdk4.Connection(
>     >         >>>> >     >                     url=url,
>     >         >>>> >     >                   
>      username=constants.ENGINE_USER,
>     >         >>>> >     >
>     >         >>>> >     
>     password=str(self.metadata['ovirt-engine-password']),
>     >         >>>> >     >                     insecure=True,
>     >         >>>> >     >                     debug=True,
>     >         >>>> >     >                 )
>     >         >>>> >     >
>     >         >>>> >     >
>     >         >>>> >     > The clue is actually on the HTTPd logs:
>     >         >>>> >     > 192.168.203.1 - - [12/Oct/2016:17:56:27
>     -0400] "POST
>     >         >>>> >     > /ovirt-engine/sso/oauth/token HTTP/1.1" 404 74
>     >         >>>> >     >
>     >         >>>> >     > And indeed, from the deubg log:
>     >         >>>> >     > begin captured logging << --------------------\n
>     >         >>>> >     > root: DEBUG: Trying 192.168.203.3...\n
>     >         >>>> >     > root: DEBUG: Connected to 192.168.203.3
>     >         (192.168.203.3) port 443
>     >         >>>> >     (#0)\n
>     >         >>>> >     > root: DEBUG: Initializing NSS with certpath:
>     >         sql:/etc/pki/nssdb\n
>     >         >>>> >     > root: DEBUG: skipping SSL peer certificate
>     >         verification\n
>     >         >>>> >     > root: DEBUG: ALPN/NPN, server did not agree to a
>     >         protocol\n
>     >         >>>> >     > root: DEBUG: SSL connection using
>     >         >>>> >     TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\n
>     >         >>>> >     > root: DEBUG: Server certificate:\n
>     >         >>>> >     > root: DEBUG: subject: CN=engine,O=Test,C=US\n
>     >         >>>> >     > root: DEBUG: start date: Oct 11 21:55:29 2016
>     GMT\n
>     >         >>>> >     > root: DEBUG: expire date: Sep 16 21:55:29
>     2021 GMT\n
>     >         >>>> >     > root: DEBUG: common name: engine\nroot:
>     DEBUG: issuer:
>     >         >>>> >     > CN=engine.38998,O=Test,C=US\n
>     >         >>>> >     > *root: DEBUG: POST /ovirt-engine/sso/oauth/token
>     >         HTTP/1.1\n*
>     >         >>>> >     > *root: DEBUG: Host: 192.168.203.3\n*
>     >         >>>> >     > *root: DEBUG: User-Agent: PythonSDK/4.1.0a0\n*
>     >         >>>> >     > *root: DEBUG: Accept: application/json\n*
>     >         >>>> >     > *root: DEBUG: Content-Length: 78\n*
>     >         >>>> >     > *root: DEBUG: Content-Type:
>     >         application/x-www-form-urlencoded\nroot:
>     >         >>>> >     > DEBUG:
>     >         >>>> >     >
>     >         >>>> >
>     >         
>     username=admin%40internal&scope=ovirt-app-api&password=123&grant_type=password\n*
>     >         >>>> >     > *root: DEBUG: upload completely sent off: 78
>     out of
>     >         78 bytes\n*
>     >         >>>> >     > *root: DEBUG: HTTP/1.1 404 Not Found\n*
>     >         >>>> >     > *root: DEBUG: Date: Wed, 12 Oct 2016 21:56:27
>     GMT\n*
>     >         >>>> >     > *root: DEBUG: Server: Apache/2.4.6 (CentOS)
>     >         OpenSSL/1.0.1e-fips\n*
>     >         >>>> >     > *root: DEBUG: Content-Length: 74\n*
>     >         >>>> >     > *root: DEBUG: Content-Type: text/html;
>     charset=UTF-8\n*
>     >         >>>> >     > *root: DEBUG: \n*
>     >         >>>> >     > *root: DEBUG:
>     >         <html><head><title>Error</title></head><body>404 - Not
>     >         >>>> >     > Found</body></html>\n*
>     >         >>>> >     > root: DEBUG: Connection #0 to host 192.168.203.3
>     >         left intact\n
>     >         >>>> >     > --------------------- >> end captured logging
>     >         >>>> >     >
>     >         >>>> >
>     >         >>>> >     That definitively looks like version 3 of the
>     engine.
>     >         Either that or
>     >         >>>> >     version 4 of the engine with web server
>     configuration
>     >         modified so that
>     >         >>>> >     the SSO doesn't work as expected.
>     >         >>>> >
>     >         >>>> >     What do you get if you run this against that
>     server?
>     >         >>>> >
>     >         >>>> >
>     >         >>>> > Attached.
>     >         >>>> > Y.
>     >         >>>> >
>     >         >>>>
>     >         >>>> OK, that is version 4.1 of the engine, so next
>     question is
>     >         why the SSO
>     >         >>>> service is not responding. Do you see any message in
>     >         >>>> /var/log/ovirt-engine/server.log about
>     "enginesso.war" not
>     >         being
>     >         >>>> deployed? Did you do any modification to the
>     >         >>>> /etc/httpd/conf.d/z-ovirt-engine.conf file?
>     >         >>>>
>     >         >>>> Ravi, Martin, any idea of why the SSO service may not be
>     >         working?
>     >         >>>>
>     >         >>>> >
>     >         >>>> >
>     >         >>>> >       curl \
>     >         >>>> >       --verbose \
>     >         >>>> >       --insecure \
>     >         >>>> >       --request GET \
>     >         >>>> >       --user "admin at internal:yourpassword" \
>     >         >>>> >       --header "Version: 4" \
>     >         >>>> >       --header "Accept: application/xml" \
>     >         >>>> >       "https://thatserver/ovirt-engine/api
>     <https://thatserver/ovirt-engine/api>
>     >         <https://thatserver/ovirt-engine/api
>     <https://thatserver/ovirt-engine/api>>
>     >         >>>> >     <https://thatserver/ovirt-engine/api
>     <https://thatserver/ovirt-engine/api>
>     >         <https://thatserver/ovirt-engine/api
>     <https://thatserver/ovirt-engine/api>>>"
>     >         >>>> >
>     >         >>>>
>     >         >>>>
>     >         >>>> --
>     >         >>>> Dirección Comercial: C/Jose Bardasano Baos, 9, Edif.
>     Gorbea
>     >         3, planta
>     >         >>>> 3ºD, 28016 Madrid, Spain
>     >         >>>> Inscrita en el Reg. Mercantil de Madrid – C.I.F.
>     B82657941
>     >         - Red Hat S.L.
>     >         >>>
>     >         >>>
>     >         >>
>     >         >
>     >
>     >
>     >
> 
> 
>     --
>     Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
>     3ºD, 28016 Madrid, Spain
>     Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat
>     S.L.
> 
> 


-- 
Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3ºD, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.



More information about the Devel mailing list