Re: [ovirt-users] Engine Down Score is 0
by Sandro Bonazzola
The hosted engine fails to start due to:
Traceback (most recent call last):
File "/usr/share/vdsm/virt/vm.py", line 714, in _startUnderlyingVm
self._run()
File "/usr/share/vdsm/virt/vm.py", line 2026, in _run
self._connection.createXML(domxml, flags),
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line
123, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 917, in
wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in
createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed',
conn=self)
libvirtError: Cannot get interface MTU on 'None': No such device
The Hosted Engine VM has been configured with:
<interface address="None" type="bridge">
<mac address="00:1a:4a:16:01:75"/>
<model type="virtio"/>
<source bridge="storage"/>
<link state="up"/>
</interface>
Not sure how you got there.
Can you share steps you did before having this issue?
On Wed, May 31, 2017 at 4:04 PM, Rizwan Qureshi <rqureshi(a)connexin.co.uk>
wrote:
> Hi Sandro,
>
> PFA the sos report and vdsm logs.
>
>
>
> *From:* Sandro Bonazzola [mailto:sbonazzo@redhat.com]
> *Sent:* Wednesday, May 31, 2017 11:04 AM
> *To:* Rizwan Qureshi <rqureshi(a)connexin.co.uk>; Martin Sivak <
> msivak(a)redhat.com>
> *Cc:* Michal Skrivanek <mskrivan(a)redhat.com>; users(a)ovirt.org
>
> *Subject:* Re: [ovirt-users] Engine Down Score is 0
>
>
>
>
>
>
>
> On Wed, May 31, 2017 at 11:10 AM, Rizwan Qureshi <rqureshi(a)connexin.co.uk>
> wrote:
>
> Hi Sandro,
>
> Thanks for your response.
>
>
>
> We have tried that already but the VM shuts down after the –vm-start
> executes.
>
>
>
> Also, apologies for wrong information bfore. We have 3 dedicated hosts not
> 1.
>
>
>
> Please see below the output of –vm-status command:
>
>
>
>
>
> Can you please share sos report or at least vdsm logs from the hosts?
>
> Need to understand what happened to the vm.
>
>
>
>
>
>
>
> --== Host 1 status ==--
>
>
>
> Status up-to-date : True
>
> Hostname : vmhost1.le1.uk.cxn-network.net
>
> Host ID : 1
>
> Engine status : {"reason": "bad vm status", "health":
> "bad", "vm": "down", "detail": "down"}
>
> Score : 0
>
> stopped : False
>
> Local maintenance : False
>
> crc32 : 37a4ce89
>
> Host timestamp : 76387
>
> Extra metadata (valid at timestamp):
>
> metadata_parse_version=1
>
> metadata_feature_version=1
>
> timestamp=76387 (Wed May 31 10:04:47 2017)
>
> host-id=1
>
> score=0
>
> maintenance=False
>
> state=EngineUnexpectedlyDown
>
> stopped=False
>
> timeout=Thu Jan 1 22:17:00 1970
>
>
>
>
>
> --== Host 2 status ==--
>
>
>
> Status up-to-date : True
>
> Hostname : vmhost2.le1.uk.cxn-network.net
>
> Host ID : 2
>
> Engine status : {"reason": "bad vm status", "health":
> "bad", "vm": "down", "detail": "down"}
>
> Score : 0
>
> stopped : False
>
> Local maintenance : False
>
> crc32 : 937b5542
>
> Host timestamp : 76069
>
> Extra metadata (valid at timestamp):
>
> metadata_parse_version=1
>
> metadata_feature_version=1
>
> timestamp=76069 (Wed May 31 10:04:51 2017)
>
> host-id=2
>
> score=0
>
> maintenance=False
>
> state=EngineUnexpectedlyDown
>
> stopped=False
>
> timeout=Thu Jan 1 22:17:49 1970
>
>
>
>
>
> --== Host 3 status ==--
>
>
>
> Status up-to-date : True
>
> Hostname : vmhost3.le1.uk.cxn-network.net
>
> Host ID : 3
>
> Engine status : {"reason": "bad vm status", "health":
> "bad", "vm": "down", "detail": "down"}
>
> Score : 0
>
> stopped : False
>
> Local maintenance : False
>
> crc32 : 8ffac898
>
> Host timestamp : 76212
>
> Extra metadata (valid at timestamp):
>
> metadata_parse_version=1
>
> metadata_feature_version=1
>
> timestamp=76212 (Wed May 31 10:04:55 2017)
>
> host-id=3
>
> score=0
>
> maintenance=False
>
> state=EngineUnexpectedlyDown
>
> stopped=False
>
> timeout=Thu Jan 1 22:16:58 1970
>
>
>
> *From:* Sandro Bonazzola [mailto:sbonazzo@redhat.com]
> *Sent:* Wednesday, May 31, 2017 7:52 AM
> *To:* Rizwan Qureshi <rqureshi(a)connexin.co.uk>; Michal Skrivanek <
> mskrivan(a)redhat.com>
> *Cc:* users(a)ovirt.org
> *Subject:* Re: [ovirt-users] Engine Down Score is 0
>
>
>
>
>
>
>
> On Tue, May 30, 2017 at 6:32 PM, Rizwan Qureshi <rqureshi(a)connexin.co.uk>
> wrote:
>
> Hello Ovirt Users,
>
> I am new to ovirt.
>
>
>
> Hi, welcome ot oVirt community!
>
>
>
>
>
> Just trying to fix the issue with the engine which seems to be down and
> hence all our VMs which we are very much dependent upon are not working.
>
>
>
> Other VMs shouldn't be affected by an unavailability of the engine, they
> should keep going if already started.
>
>
>
>
>
> Tried googling the log snippet but to no avail. Hoping to get some help
> from you guys.
>
>
>
> I am completely blank and don’t know whats wrong with it.
>
>
>
> We have 3 Dell servers. One for Engine and the other two for nodes. Please
> see the log snippet from the engine server agent.log. Please let me know if
> more information is needed to debug the issue.
>
>
>
> So you have only 1 host dedicated to running hosted engine?
>
>
>
>
>
>
>
> MainThread::INFO::2017-05-30 17:21:51,649::hosted_engine::
> 612::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_vdsm) Initializing VDSM
>
> MainThread::INFO::2017-05-30 17:21:53,820::hosted_engine::
> 639::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Connecting the storage
>
> MainThread::INFO::2017-05-30 17:21:53,821::storage_server::
> 219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
>
> MainThread::INFO::2017-05-30 17:21:58,134::storage_server::
> 226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
>
> MainThread::INFO::2017-05-30 17:21:58,142::storage_server::
> 233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
>
> MainThread::INFO::2017-05-30 17:21:58,258::hosted_engine::
> 666::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Preparing images
>
> MainThread::INFO::2017-05-30 17:21:58,258::image::126::
> ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
>
> MainThread::INFO::2017-05-30 17:22:00,637::hosted_engine::
> 669::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Reloading vm.conf from the
> shared storage domain
>
> MainThread::INFO::2017-05-30 17:22:00,638::config::206::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(refresh_local_conf_file) Trying to get a fresher copy of vm
> configuration from the OVF_STORE
>
> MainThread::INFO::2017-05-30 17:22:02,838::ovf_store::103::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found
> OVF_STORE: imgUUID:f4d55192-a2c8-4c2b-8e6c-46a49c30967c,
> volUUID:0ae2c43f-5883-4354-a33f-e68e21ae3733
>
> MainThread::INFO::2017-05-30 17:22:02,977::ovf_store::103::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found
> OVF_STORE: imgUUID:4ef3bd37-e48d-44a8-b906-5353de1a32cc,
> volUUID:d63b9f36-c5f1-4ddc-9d4a-e01f22023e73
>
> MainThread::INFO::2017-05-30 17:22:03,032::ovf_store::112::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
>
> MainThread::INFO::2017-05-30 17:22:03,033::ovf_store::119::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path: /rhev/data-center/mnt/nas1.le1.uk.cxn-network.net:
> _oVirt/4f9e46d7-594e-473c-b0c5-1770c5773a2e/images/4ef3bd37-
> e48d-44a8-b906-5353de1a32cc/d63b9f36-c5f1-4ddc-9d4a-e01f22023e73
>
> MainThread::INFO::2017-05-30 17:22:03,044::config::226::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(refresh_local_conf_file) Found an OVF for HE VM, trying to
> convert
>
> MainThread::INFO::2017-05-30 17:22:03,046::config::231::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(refresh_local_conf_file) Got vm.conf from OVF_STORE
>
> MainThread::INFO::2017-05-30 17:22:03,082::states::672::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is
> 0 due to unexpected vm shutdown at Tue May 30 17:18:38 2017
>
> MainThread::INFO::2017-05-30 17:22:03,082::hosted_engine::
> 461::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUnexpectedlyDown (score: 0)
>
> MainThread::INFO::2017-05-30 17:22:03,082::hosted_engine::
> 466::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Best remote host vmhost2.le1.uk.cxn-network.net (id: 2, score: 0)
>
> MainThread::INFO::2017-05-30 17:22:13,135::hosted_engine::
> 612::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_vdsm) Initializing VDSM
>
> MainThread::INFO::2017-05-30 17:22:15,305::hosted_engine::
> 639::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Connecting the storage
>
> MainThread::INFO::2017-05-30 17:22:15,306::storage_server::
> 219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
>
> MainThread::INFO::2017-05-30 17:22:19,618::storage_server::
> 226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
>
> MainThread::INFO::2017-05-30 17:22:19,626::storage_server::
> 233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
>
> MainThread::INFO::2017-05-30 17:22:19,742::hosted_engine::
> 666::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Preparing images
>
> MainThread::INFO::2017-05-30 17:22:19,742::image::126::
> ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
>
> MainThread::INFO::2017-05-30 17:22:22,140::hosted_engine::
> 669::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Reloading vm.conf from the
> shared storage domain
>
> MainThread::INFO::2017-05-30 17:22:22,140::config::206::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(refresh_local_conf_file) Trying to get a fresher copy of vm
> configuration from the OVF_STORE
>
> MainThread::INFO::2017-05-30 17:22:24,333::ovf_store::103::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found
> OVF_STORE: imgUUID:f4d55192-a2c8-4c2b-8e6c-46a49c30967c,
> volUUID:0ae2c43f-5883-4354-a33f-e68e21ae3733
>
> MainThread::INFO::2017-05-30 17:22:24,472::ovf_store::103::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found
> OVF_STORE: imgUUID:4ef3bd37-e48d-44a8-b906-5353de1a32cc,
> volUUID:d63b9f36-c5f1-4ddc-9d4a-e01f22023e73
>
> MainThread::INFO::2017-05-30 17:22:24,519::ovf_store::112::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
>
> MainThread::INFO::2017-05-30 17:22:24,520::ovf_store::119::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path: /rhev/data-center/mnt/nas1.le1.uk.cxn-network.net:
> _oVirt/4f9e46d7-594e-473c-b0c5-1770c5773a2e/images/4ef3bd37-
> e48d-44a8-b906-5353de1a32cc/d63b9f36-c5f1-4ddc-9d4a-e01f22023e73
>
> MainThread::INFO::2017-05-30 17:22:24,531::config::226::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(refresh_local_conf_file) Found an OVF for HE VM, trying to
> convert
>
> MainThread::INFO::2017-05-30 17:22:24,533::config::231::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(refresh_local_conf_file) Got vm.conf from OVF_STORE
>
> MainThread::INFO::2017-05-30 17:22:24,568::states::672::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is
> 0 due to unexpected vm shutdown at Tue May 30 17:18:39 2017
>
> MainThread::INFO::2017-05-30 17:22:24,568::hosted_engine::
> 461::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUnexpectedlyDown (score: 0)
>
> MainThread::INFO::2017-05-30 17:22:24,568::hosted_engine::
> 466::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Best remote host vmhost2.le1.uk.cxn-network.net (id: 2, score: 0)
>
>
>
>
>
> Looks like hosted engine has been turned off at Tue May 30 17:18:38 2017
> without moving to maintenance before doing it.
>
> Above logs are timed 2017-05-30 17:22:24, so 4 minutes after the shutdown.
>
> The agent should have already restarted it in a few minutes after above
> logs.
>
> Next time, if you need to urgently bring it up again you can use:
>
> hosted-engine ---vm-start
>
> as described in http://www.ovirt.org/documentation/self-hosted/
> chap-Troubleshooting/
>
>
>
>
>
> --
>
> Best Regards,
>
> Rizwan Qureshi
>
> VoIP Admin
>
> Ph: 01482xxxxxxx
>
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
>
>
> --
>
> *SANDRO BONAZZOLA*
>
> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
>
> Red Hat EMEA <https://www.redhat.com/>
>
> <https://red.ht/sig>
>
> *TRIED. TESTED. TRUSTED.* <https://red.ht/sig>
>
> <https://red.ht/sig>
>
>
>
> <https://red.ht/sig>
>
> <https://red.ht/sig>
>
> -- <https://red.ht/sig>
>
> *SANDRO BONAZZOLA <https://red.ht/sig>*
>
> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
> <https://red.ht/sig>
>
> *Red Hat EMEA* <https://red.ht/sig>
>
> <https://red.ht/sig>
>
> *TRIED. TESTED. TRUSTED.* <https://red.ht/sig>
>
> <https://red.ht/sig>
>
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
3 years, 11 months
- 2
- 3
Importing VMs from kvm to oVirt. "VM zarafa is down with error. Exit message: Failed to find the libvirt domain."
by Marc Pfaendler
Hello
I'm trying to import an existing kvm VM into oVirt 4.1. It a new server
with a fresh installation of oVirt.
I'm working with the oVirt gui to achieve this.
After moving the VM from the old system into the export folder, the gui
allows me to import it.
The import seems to succeed. But after the VM is imported it fails to start.
Giving this error:
"VM zarafa is down with error. Exit message: Failed to find the
libvirt domain."
Cheers
Marc
3 years, 11 months
- 3
- 2
Re: [ovirt-users] Ovirt Hosted-Engine VM iptables
by Andrew Dent
I can add a rule into ipdates such as this
iptables -I INPUT -s 192.168.0.10 -p tcp -m tcp --dport 5666 -j ACCEPT
I can see the addition has succeeded with this
iptables-save > /etc/sysconfig/iptables
But a reboot of the Engine VM (not the Host) doesn't keep the new rule,
and I was expecting that during bootup CentOS would read from
/etc/sysconfig/iptables.
Alas it isn't.
Found a solution.
After reading this
https://stackoverflow.com/questions/24756240/how-can-i-use-iptables-on-ce...
I installed iptables-services
But once installed I found that iptables -L showed no rules.
thankfully I still had the default hosted-engine rules in
/etc/sysconfig/iptables
iptables-restore < /etc/sysconfig/iptables
Then.....
service iptables save
restored the default hosted-engine rules including my rule for 5666.
Rebooting the hosted-engine VM and my rule 5666 for NRPE is still there.
Success!!
To answer your other questions
> Did you ask to configure the firewall during engine-setup?
Yes.
Looks like it setup firewalld for me.
> Alternatively, it's recommended to use firewalld.
For the moment I have disabled firewalld and are using iptables....Is
there a reason why firewalld is preferred over iptables?
Kind regards
Andrew
------ Original Message ------
From: "Yedidyah Bar David" <didi(a)redhat.com>
To: "Andrew Dent" <adent(a)ctcroydon.com.au>
Cc: "users" <users(a)ovirt.org>
Sent: 29/05/2017 9:26:23 PM
Subject: Re: [ovirt-users] Ovirt Hosted-Engine VM iptables
>On Mon, May 29, 2017 at 1:14 PM, Andrew Dent <adent(a)ctcroydon.com.au>
>wrote:
>> Hi
>>
>> I would like to add rules into the iptables of the Hosted Engine VM
>>in
>> Ovirt.
>> I am wanting to monitor the Ovirt Engine using Nagios -> NRPE and I
>>would
>> like to open port 5666
>>
>> the version is oVirt Engine Version: 4.1.1.8-1.el7.centos
>> I have tried using the normal process for iptables (iptables-save
>>etc), but
>> it seems that the file
>> /etc/sysconfig/iptables
>> is ignored when the Ovirt Engine VM starts.
>
>What do you mean in "ignored"?
>
>What's the output of 'iptables-save'?
>
>Did you ask to configure the firewall during engine-setup?
>
>>
>> How can I add permanent iptables rules into the Engine VM?
>
>On the engine vm (unlike hosts), the only thing that touches iptables
>is engine-setup. Before doing that it asks you if you want to configure
>the firewall. There aren't currently means to add your custom rules -
>either you manage it all by yourself or you let engine-setup do that.
>
>Alternatively, it's recommended to use firewalld. engine-setup can
>add to firewalld the stuff it wants, and you still can add your own
>stuff.
>
>If I got you wrong and you refer to the hosts (not engine), see also:
>
>https://www.ovirt.org/blog/2016/12/extension-iptables-rules-oVirt-hosts/
>
>Best,
>
>>
>> Kind regards
>>
>>
>> Andrew
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
>
>--
>Didi
3 years, 11 months
- 2
- 1