can't use vmconsole anymore

15 Apr 2021

Thanks for your answer, I checked but I am still stuck :

I confirm that the servlet can be reached, according to your recommended
test (Method Not Allowed.):

[root@vs-inf-prd-ovt-fr-501 ~]# wget
https://localhost:443/ovirt-engine/services/vmconsole-proxy
--no-check-certificate
--2022-06-13 10:30:11--
https://localhost/ovirt-engine/services/vmconsole-proxy
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:443... connected.
The certificate's owner does not match hostname 'localhost'
HTTP request sent, awaiting response... 405 Method Not Allowed
2022-06-13 10:30:11 ERROR 405: Method Not Allowed.

I retried ovirt-vmconsole-list.py with "--debug", and looked at the logs :

[root@vs-inf-prd-ovt-fr-501 ~]#
/usr/libexec/ovirt-vmconsole-proxy-helper/ovirt-vmconsole-list.py --debug
--version "1" keys
[root@vs-inf-prd-ovt-fr-501 ~]#
[root@vs-inf-prd-ovt-fr-501 ~]# grep vmconsole /var/log/messages
Jun 13 10:35:41 vs-inf-prd-ovt-fr-501 journal[3112274]: 2022-06-13
10:35:41,222+0200 ovirt-vmconsole-list: ERROR main:265 Error: HTTP Error
403: Forbidden

To be noted,
We did change the engine's CA certificate at some point by following this
procedure
https://ovirt.org/documentation/administration_guide/index.html#Replacing_the_Manager_CA_Certificate
We also renewed the certificates during a standard engine --setup upgrade
to 4.4.10

Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group

On Mon, Jun 13, 2022 at 4:47 PM Radoslaw Szwajkowski <rszwajko@redhat.com>
wrote:

> Hi,
> the first thing to check is the firewall:  check with wget if the
> servlet can be reached (method not allow error means you have
> connected)
>
> wget localhost:8080/ovirt-engine/services/vmconsole-proxy
> HTTP request sent, awaiting response... 405 Method Not Allowed
> 2022-06-13 09:36:48 ERROR 405: Method Not Allowed.
>
> When using the ovirt-vmconsole-list you can also check system log[1]
> i.e. if the server cannot be reached you should see sth like this
>
> grep vmconsole  /var/log/messages
> Jun 13 08:58:02 developer journal[2972]: 2022-06-13 08:58:02,992+0200
> ovirt-vmconsole-list: ERROR main:265 Error: <urlopen error [Errno 111]
> Connection refused>
>
> Note also that you can increase the log level by passing "--debug"
> param or just look inside the script.
>
> best regards,
> radek
>
> [1] https://github.com/oVirt/ovirt-vmconsole#problem-determination
>
> On Mon, Jun 13, 2022 at 8:22 AM Guillaume Pavese
> <guillaume.pavese@interactiv-group.com> wrote:
> >
> > Hello everyone,
> >
> > We have the same problem on our oVirt 4.4.10 Production server.
> > ssh connection to vmconsole@engine was previously working in 4.4.6. but
> it stopped working at some point, maybe since upgraded to 4.4.10
> >
> > contrary to a working test environment that was directly installed on
> 4.4.10,
> > And as for Nathanaël,
> > the following returns nothing : ovirt-vmconsole-list.py --version "1"
> keys
> >
> > [root@vs-inf-prd-ovt-fr-501 ~]#
> /usr/libexec/ovirt-vmconsole-proxy-helper/ovirt-vmconsole-list.py --version
> "1" keys
> > [root@vs-inf-prd-ovt-fr-501 ~]#
> >
> > I have verified that the keys stills appear on users' Option -> "User's
> Public Key" in the engine's UI
> >
> > What can I try to fix this?
> >
> >
> > Guillaume Pavese
> > Ingénieur Système et Réseau
> > Interactiv-Group
> >
> >
> > On Mon, May 10, 2021 at 9:47 PM Nathanaël Blanchet <blanchet@abes.fr>
> wrote:
> >>
> >> Hi,
> >>
> >> I can't still connect to my vms with vmconsole proxy on my production
> engine (other test and dev engine are OK).
> >>
> >> the ssh key for the wanted user is available in the the API:
> >>
> >> <ssh_public_keys>
> >> <ssh_public_key
> href="/ovirt-engine/api/users/64b7f3bf-9d43-4508-af93-63ad77652be3/sshpublickeys/aaace8d4-08d3-4452-ac91-df4b491bd899"
> id="aaace8d4-08d3-4452-ac91-df4b491bd899">
> >> <content>
> >> ssh-rsa
> AAAAB3NzaC1yc2EAAAABIwAAAQEAyfrDI84RWtSvFOUvpb9DkbnIuEfZEQAt4ZCXDHNXcmRwa9iXfPbj69gkOJyj7Jhj9RinJn9at4NgJtrO/rRRgT+SzYUWpdO2KWHgRM5v1rpYcw820ZDdAZk+yxCjQsy6kd49q/q6B+Uzg8Kpth+CAV1ubRrBYqFiuT/qQe9y+0N1TkNdASWL38oZH9K0rzbDb4WlU2Er2BCXzoLF2NBk7iyaS3+Y65DqWPPHHdh89nilC6k5N7SCUkSOayrjh7NnErkBAKZ6PPaarZqZhZPrCbHZnu0oqA0XQXKLcYpwuhNwcK8e4ZWsDwMmArnNcmS6JFxnPIrGYxxmv01K6VXVvw==
> >> </content>
> >> <user
> href="/ovirt-engine/api/users/64b7f3bf-9d43-4508-af93-63ad77652be3"
> id="64b7f3bf-9d43-4508-af93-63ad77652be3"/>
> >> </ssh_public_key>
> >> </ssh_public_keys>
> >>
> >> But /usr/libexec/ovirt-vmconsole-proxy-helper/ovirt-vmconsole-list.py
> --version "1" keys still returns nothing.
> >>
> >> On the engine:
> >>
> >> [root@air ~]# systemctl status ovirt-vmconsole-proxy-sshd.service
> >> ● ovirt-vmconsole-proxy-sshd.service - oVirt VM Console SSH server
> daemon
> >>    Loaded: loaded
> (/usr/lib/systemd/system/ovirt-vmconsole-proxy-sshd.service; enabled;
> vendor preset: disabled)
> >>    Active: active (running) since Mon 2021-05-10 14:16:55 CEST; 22min
> ago
> >>  Main PID: 3649210 (sshd)
> >>     Tasks: 1 (limit: 204594)
> >>    Memory: 2.7M
> >>    CGroup: /system.slice/ovirt-vmconsole-proxy-sshd.service
> >>            └─3649210 /usr/sbin/sshd -f
> /usr/share/ovirt-vmconsole/ovirt-vmconsole-proxy/ovirt-vmconsole-proxy-sshd/sshd_config
> -D
> >>
> >> mai 10 14:16:55 air.v100.abes.fr systemd[1]: Started oVirt VM Console
> SSH server daemon.
> >> mai 10 14:16:55 air.v100.abes.fr sshd[3649210]: Server listening on
> 0.0.0.0 port 2222.
> >> mai 10 14:16:55 air.v100.abes.fr sshd[3649210]: Server listening on ::
> port 2222.
> >> mai 10 14:17:01 air.v100.abes.fr ovirt-vmconsole-proxy-keys[3649214]:
> ERROR '"keys"'
> >> mai 10 14:17:01 air.v100.abes.fr sshd[3649212]: AuthorizedKeysCommand
> /usr/libexec/ovirt-vmconsole-proxy-keys ovirt-vmconsole failed, status 1
> >> mai 10 14:17:02 air.v100.abes.fr ovirt-vmconsole-proxy-keys[3649218]:
> ERROR '"keys"'
> >> mai 10 14:17:02 air.v100.abes.fr sshd[3649212]: AuthorizedKeysCommand
> /usr/libexec/ovirt-vmconsole-proxy-keys ovirt-vmconsole failed, status 1
> >> mai 10 14:17:02 air.v100.abes.fr sshd[3649212]: Connection closed by
> authenticating user ovirt-vmconsole 10.34.100.131 port 46874 [preauth]
> >>
> >> I tried to execute /usr/libexec/ovirt-vmconsole-proxy-keys
> ovirt-vmconsole but it gives an internal ERROR (as on the other working
> engine, so it may be not relevant)
> >>
> >> What can I test more?
> >>
> >> Le 18/04/2021 à 15:59, Sharon Gratch a écrit :
> >>
> >> Hi,
> >>
> >> Please follow the instructions mentioned here:
> >>
> https://www.ovirt.org/documentation/virtual_machine_management_guide/#Logging_in_to_a_virtual_machine_using_SPICE
> - > " Opening a Serial Console to a Virtual Machine".
> >>
> >> It seems that something is wrong with the user permissions/keys.
> >> Is the 4.4.5 oVirt installation an upgraded or a new installation?
> >> You mentioned that it's working with your other engines? Do they all
> use the 4.4.5 version?
> >>
> >> Thanks,
> >> Sharon
> >>
> >>
> >> On Fri, Apr 16, 2021 at 1:31 PM Nathanaël Blanchet <blanchet@abes.fr>
> wrote:
> >>>
> >>> I removed the user and created an other time. Now, I have this
> >>>
> >>> The key seems to be present in the DB
> >>>
> >>> engine=# SELECT users.username, user_profiles.property_content::text
> >>> FROM user_profiles
> >>> JOIN users ON users.user_id = user_profiles.user_id
> >>> WHERE user_profiles.property_type= 'SSH_PUBLIC_KEY';
> >>>           username |
> >>> property_content
> >>>
> >>>
> --------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >>>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >>> -------------------------------
> >>>   sblanchet@levant.abes.fr | "ssh-rsa
> >>>
> AAAAB3NzaC1yc2EAAAABIwAAAQEAyfrDI84RWtSvFOUvpb9DkbnIuEfZEQAt4ZCXDHNXcmRwa9iXfPbj69gkOJyj7Jhj9RinJn9at4NgJtrO/rRRgT+SzYUWpdO2KWHgRM5v1rpYcw820ZDdAZk+yxCjQ
> >>>
> sy6kd49q/q6B+Uzg8Kpth+CAV1ubRrBYqFiuT/qQe9y+0N1TkNdASWL38oZH9K0rzbDb4WlU2Er2BCXzoLF2NBk7iyaS3+Y65DqWPPHHdh89nilC6k5N7SCUkSOayrjh7NnErkBAKZ6PPaarZqZhZPrCbHZnu0oqA0XQXKLcYpwuhNwcK8e4ZWsDwMmArn
> >>> NcmS6JFxnPIrGYxxmv01K6VXVvw=="
> >>> (1 row)
> >>>
> >>> and now in the api
> >>>
> >>> <ssh_public_keys>
> >>> <ssh_public_key
> >>>
> href="/ovirt-engine/api/users/64b7f3bf-9d43-4508-af93-63ad77652be3/sshpublickeys/70850a0e-1b20-4dd5-9fcd-4f64303509d1"
> >>> id="70850a0e-1b20-4dd5-9fcd-4f64303509d1">
> >>> <content>
> >>> ssh-rsa
> >>>
> AAAAB3NzaC1yc2EAAAABIwAAAQEAyfrDI84RWtSvFOUvpb9DkbnIuEfZEQAt4ZCXDHNXcmRwa9iXfPbj69gkOJyj7Jhj9RinJn9at4NgJtrO/rRRgT+SzYUWpdO2KWHgRM5v1rpYcw820ZDdAZk+yxCjQsy6kd49q/q6B+Uzg8Kpth+CAV1ubRrBYqFiuT/qQe9y+0N1TkNdASWL38oZH9K0rzbDb4WlU2Er2BCXzoLF2NBk7iyaS3+Y65DqWPPHHdh89nilC6k5N7SCUkSOayrjh7NnErkBAKZ6PPaarZqZhZPrCbHZnu0oqA0XQXKLcYpwuhNwcK8e4ZWsDwMmArnNcmS6JFxnPIrGYxxmv01K6VXVvw==
> >>> </content>
> >>> <user
> >>> href="/ovirt-engine/api/users/64b7f3bf-9d43-4508-af93-63ad77652be3"
> >>> id="64b7f3bf-9d43-4508-af93-63ad77652be3"/>
> >>> </ssh_public_key>
> >>> </ssh_public_keys>
> >>>
> >>> but I still can't connect
> >>>
> >>> $ ssh -t -p 2222  ovirt-vmconsole@air.v100.abes.fr connect
> >>> ovirt-vmconsole@air.v100.abes.fr: Permission denied (publickey).
> >>>
> >>> and
> >>>
> >>> [root@air ~]#
> >>> /usr/libexec/ovirt-vmconsole-proxy-helper/ovirt-vmconsole-list.py
> >>> --version "1" keys
> >>>
> >>> still returns empty string...
> >>>
> >>>
> >>> Le 16/04/2021 à 11:07, Nathanaël Blanchet a écrit :
> >>> >
> >>> > Le 16/04/2021 à 10:31, Radoslaw Szwajkowski a écrit :
> >>> >>> [root@air-dev ~]#
> >>> >>> /usr/libexec/ovirt-vmconsole-proxy-helper/ovirt-vmconsole-list.py
> >>> >>> --version "1" keys
> >>> >>> {"keys": [{"entityid": "d5e69fa0-96a0-4aae-952d-18fe36940248",
> >>> >>> "entity":
> >>> >>> "sblanchet@levant.abes.fr@abes.fr-authz", "key": "ssh-rsa
> >>> >>>
> AAAAB3NzaC1yc2EAAAABIwAAAQEAyfrDI84RWtSvFOUvpb9DkbnIuEfZEQAt4ZCXDHNXcmRwa9iXfPbj69gkOJyj7Jhj9RinJn9at4NgJtrO/rRRgT+SzYUWpdO2KWHgRM5v1rpYcw820ZDdAZk+yxCjQsy6kd49q/q6B+Uzg8Kpth+CAV1ubRrBYqFiuT/qQe9y+0N1TkNdASWL38oZH9K0rzbDb4WlU2Er2BCXzoLF2NBk7iyaS3+Y65DqWPPHHdh89nilC6k5N7SCUkSOayrjh7NnErkBAKZ6PPaarZqZhZPrCbHZnu0oqA0XQXKLcYpwuhNwcK8e4ZWsDwMmArnNcmS6JFxnPIrGYxxmv01K6VXVvw=="}],
> >>> >>>
> >>> >>> "version": 1, "content": "key_list"}
> >>> >>>
> >>> >>> but the same command on the main  engine returns empty
> >>> >>>
> >>> >>> [root@air ~]#
> >>> >>> /usr/libexec/ovirt-vmconsole-proxy-helper/ovirt-vmconsole-list.py
> >>> >>> --version "1" keys
> >>> >>>
> >>> >> Empty list (no keys) should look similar to: {"keys": [], "version":
> >>> >> 1, "content": "key_list"}
> >>> >> In your case it seems that VMConsoleProxyServlet is not responding
> >>> >> i.e. on my dev env I get a similar result (empty output,error code
> 1)
> >>> >> when server is down.
> >>> >
> >>> > it is up
> >>> >
> >>> >
> >>> > ● ovirt-vmconsole-proxy-sshd.service - oVirt VM Console SSH server
> daemon
> >>> >    Loaded: loaded
> >>> > (/usr/lib/systemd/system/ovirt-vmconsole-proxy-sshd.service; enabled;
> >>> > vendor preset: disabled)
> >>> >    Active: active (running) since Fri 2021-04-16 10:50:41 CEST; 1min
> >>> > 27s ago
> >>> >  Main PID: 1914370 (sshd)
> >>> >     Tasks: 1 (limit: 204594)
> >>> >    Memory: 3.5M
> >>> >    CGroup: /system.slice/ovirt-vmconsole-proxy-sshd.service
> >>> >            └─1914370 /usr/sbin/sshd -f
> >>> >
> /usr/share/ovirt-vmconsole/ovirt-vmconsole-proxy/ovirt-vmconsole-proxy-sshd/sshd_config
> >>> > -D
> >>> >
> >>> > avril 16 10:50:41 air.v100.abes.fr systemd[1]: Started oVirt VM
> >>> > Console SSH server daemon.
> >>> > avril 16 10:50:41 air.v100.abes.fr sshd[1914370]: Server listening
> on
> >>> > 0.0.0.0 port 2222.
> >>> > avril 16 10:50:41 air.v100.abes.fr sshd[1914370]: Server listening
> on
> >>> > :: port 2222.
> >>> > avril 16 10:52:02 air.v100.abes.fr ovirt-vmconsole[1914540]:
> >>> > 2021-04-16 10:52:02,241+0200 ovirt-vmconsole-list: ERROR main:265
> >>> > Error: HTTP Error 403: Forbidden
> >>> > avril 16 10:52:02 air.v100.abes.fr
> >>> > ovirt-vmconsole-proxy-keys[1914536]: ERROR Key list execution failed
> rc=1
> >>> > avril 16 10:52:02 air.v100.abes.fr sshd[1914534]:
> >>> > AuthorizedKeysCommand /usr/libexec/ovirt-vmconsole-proxy-keys
> >>> > ovirt-vmconsole failed, status 1
> >>> > avril 16 10:52:02 air.v100.abes.fr ovirt-vmconsole[1914547]:
> >>> > 2021-04-16 10:52:02,806+0200 ovirt-vmconsole-list: ERROR main:265
> >>> > Error: HTTP Error 403: Forbidden
> >>> > avril 16 10:52:02 air.v100.abes.fr
> >>> > ovirt-vmconsole-proxy-keys[1914543]: ERROR Key list execution failed
> rc=1
> >>> > avril 16 10:52:02 air.v100.abes.fr sshd[1914534]:
> >>> > AuthorizedKeysCommand /usr/libexec/ovirt-vmconsole-proxy-keys
> >>> > ovirt-vmconsole failed, status 1
> >>> > avril 16 10:52:03 air.v100.abes.fr sshd[1914534]: Connection closed
> by
> >>> > authenticating user ovirt-vmconsole 10.34.100.131 port 53674
> [preauth]
> >>> >
> >>> >>
> >>> >> However you can check if DB contains the right data (key is encoded
> as
> >>> >> JSON string - enclosed in double quotes):
> >>> >> SELECT users.username, user_profiles.property_content::text
> >>> >> FROM user_profiles
> >>> >> JOIN users ON users.user_id = user_profiles.user_id
> >>> >> WHERE user_profiles.property_type= 'SSH_PUBLIC_KEY';
> >>> >
> >>> >
> https://air.v100.abes.fr//ovirt-engine/api/users/1bb90486-d431-4554-a6a1-37631d8c16d4/sshpublickeys
> >>> >
> >>> >
> >>> > <ssh_public_keys/>
> >>> >
> >>> > is empty
> >>> >
> >>> > while
> >>> >
> >>> >
> https://air-dev.v100.abes.fr/ovirt-engine/api/users/d5e69fa0-96a0-4aae-952d-18fe36940248/sshpublickeys
> >>> >
> >>> >
> >>> > returns
> >>> >
> >>> > <ssh_public_keys>
> >>> > <ssh_public_key
> >>> >
> href="/ovirt-engine/api/users/d5e69fa0-96a0-4aae-952d-18fe36940248/sshpublickeys/1fa3fcaf-7475-4c72-9565-b32425d3c8fd"
> >>> > id="1fa3fcaf-7475-4c72-9565-b32425d3c8fd">
> >>> > <content>
> >>> > ssh-rsa
> >>> >
> AAAAB3NzaC1yc2EAAAABIwAAAQEAyfrDI84RWtSvFOUvpb9DkbnIuEfZEQAt4ZCXDHNXcmRwa9iXfPbj69gkOJyj7Jhj9RinJn9at4NgJtrO/rRRgT+SzYUWpdO2KWHgRM5v1rpYcw820ZDdAZk+yxCjQsy6kd49q/q6B+Uzg8Kpth+CAV1ubRrBYqFiuT/qQe9y+0N1TkNdASWL38oZH9K0rzbDb4WlU2Er2BCXzoLF2NBk7iyaS3+Y65DqWPPHHdh89nilC6k5N7SCUkSOayrjh7NnErkBAKZ6PPaarZqZhZPrCbHZnu0oqA0XQXKLcYpwuhNwcK8e4ZWsDwMmArnNcmS6JFxnPIrGYxxmv01K6VXVvw==
> >>> > </content>
> >>> > <user
> >>> > href="/ovirt-engine/api/users/d5e69fa0-96a0-4aae-952d-18fe36940248"
> >>> > id="d5e69fa0-96a0-4aae-952d-18fe36940248"/>
> >>> > </ssh_public_key>
> >>> > </ssh_public_keys>
> >>> >
> >>> >>
> >>> >> best regards,
> >>> >> Radek
> >>> >>
> >>> --
> >>> Nathanaël Blanchet
> >>>
> >>> Supervision réseau
> >>> SIRE
> >>> 227 avenue Professeur-Jean-Louis-Viala
> >>> 34193 MONTPELLIER CEDEX 5
> >>> Tél. 33 (0)4 67 54 84 55
> >>> Fax  33 (0)4 67 54 84 14
> >>> blanchet@abes.fr
> >>> _______________________________________________
> >>> Users mailing list -- users@ovirt.org
> >>> To unsubscribe send an email to users-leave@ovirt.org
> >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> >>> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> >>> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/TUHJA7C32NPJ5K5ITX4YGXEKNOZCXVHF/
> >>
> >> --
> >> Nathanaël Blanchet
> >>
> >> Supervision réseau
> >> SIRE
> >> 227 avenue Professeur-Jean-Louis-Viala
> >> 34193 MONTPELLIER CEDEX 5
> >> Tél. 33 (0)4 67 54 84 55
> >> Fax  33 (0)4 67 54 84 14
> >> blanchet@abes.fr
> >>
> >> _______________________________________________
> >> Users mailing list -- users@ovirt.org
> >> To unsubscribe send an email to users-leave@ovirt.org
> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> >> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> >> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/OZNT3FINAXWFKQZ3TSZWYWGQVIWJ7KPI/
> >
> >
> > Ce message et toutes les pièces jointes (ci-après le “message”) sont
> établis à l’intention exclusive de ses destinataires et sont confidentiels.
> Si vous recevez ce message par erreur, merci de le détruire et d’en avertir
> immédiatement l’expéditeur. Toute utilisation de ce message non conforme a
> sa destination, toute diffusion ou toute publication, totale ou partielle,
> est interdite, sauf autorisation expresse. L’internet ne permettant pas
> d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales)
> décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse
> ou il aurait été modifié. IT, ES, UK.
>
>

-- 

Ce message et toutes les pièces jointes (ci-après le “message”) sont 
établis à l’intention exclusive de ses destinataires et sont confidentiels. 
Si vous recevez ce message par erreur, merci de le détruire et d’en avertir 
immédiatement l’expéditeur. Toute utilisation de ce message non conforme a 
sa destination, toute diffusion ou toute publication, totale ou partielle, 
est interdite, sauf autorisation expresse. L’internet ne permettant pas 
d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) 
décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse 
ou il aurait été modifié. IT, ES, UK.  
<https://interactiv-group.com/disclaimer.html>

Nathanaël Blanchet

Nathanaël Blanchet

Radoslaw Szwajkowski

Nathanaël Blanchet

Radoslaw Szwajkowski

Nathanaël Blanchet

Radoslaw Szwajkowski

Nathanaël Blanchet

Sharon Gratch

Nathanaël Blanchet

Nathanaël Blanchet

Guillaume Pavese

Radoslaw Szwajkowski

Guillaume Pavese

Guillaume Pavese

Michal Skrivanek

Michal Skrivanek

tags

participants (5)