Re: Strange Issue with imageio
by Nir Soffer
On Fri, Apr 16, 2021 at 7:22 PM Nur Imam Febrianto <nur_imam(a)outlook.com> wrote:
> After upgrading to 4.4.5 from 4.4.4, I’m having a strange issue. Whenever I try to upload something (image or iso) to Storage Domain without clicking Test Connection, upload process wont start. It will shows the file area already uploaded but no upload process. When I try to open the log, seems if I don’t click test connection, it wont open any session in imageio. Upload process will going normally, if I click Test connection first and proceed to upload.
Please file ovirt-engine bug, testing the connection should not be required
for upload.
> Any idea why this is happening ? Don’t have this issue before in 4.4.4. Rebooting the engine doesn’t help either.
This is most likely a bug introduced in engine 4.4.5.
Nir
3 years, 8 months
Odd question: changing network MTU
by Chris Adams
I have an oVirt 4.3 cluster, running in one location. I have to move it
to another location. I've got a couple of 1G links between the sites,
and that's enough bandwidth for this (at least temporarily), but... I
have my iSCSI networks defined with a MTU of 9000, and it turns out the
site-to-site links only allow 1500 (and these links are going away after
this is done, so I don't think either carrier would be interested in
changing things to support larger).
Because of that, the storage won't connect up. I tried going "under the
hood" and setting a firewalld rule to force the MSS to a smaller value,
but that didn't seem to get it.
What happens if I change the MTU of an active iSCSI network in oVirt? I
could just go manually change it on each node's iSCSI interfaces, but
I'm not sure if oVirt might change it back. Also, I'm not sure what
would happen to open iSCSI TCP connections (would they reduce
gracefully).
Any other suggestions/tips/etc.? I'd like to make this as transparent
as possible, so was hoping to live-migrate VMs and storage.
--
Chris Adams <cma(a)cmadams.net>
3 years, 8 months
Strange Issue with imageio
by Nur Imam Febrianto
Hi,
After upgrading to 4.4.5 from 4.4.4, I’m having a strange issue. Whenever I try to upload something (image or iso) to Storage Domain without clicking Test Connection, upload process wont start. It will shows the file area already uploaded but no upload process. When I try to open the log, seems if I don’t click test connection, it wont open any session in imageio. Upload process will going normally, if I click Test connection first and proceed to upload.
Any idea why this is happening ? Don’t have this issue before in 4.4.4. Rebooting the engine doesn’t help either.
Regards,
Nur Imam Febrianto
3 years, 8 months
Re: Expand gluster volumes
by David White
Sorry, I meant to reply-all, to Strahil's most recent message.
So I'm doing so now.
In addition to my comments below, this StackOverflow thread summarizes what I have in mind: https://stackoverflow.com/questions/43756405/extend-glusterfs-on-top-of-lvm
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, April 16, 2021 5:33 AM, David White <dmwhite823(a)protonmail.com> wrote:
> I don't think I would have the capability to take on the expense of adding two more servers to my environment, on top of the 4th server I already have, right now.
>
> I realize the following isn't recommended, but would it be possible to, instead, intentionally degrade the gluster cluster by shutting 1 of the hosts down (put it in maintenance mode), add storage, and bring it back online?
>
> Perhaps that would be a better approach, rather than try to shift things around with a 4th server.
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, April 15, 2021 7:36 PM, Strahil Nikolov hunter86_bg(a)yahoo.com wrote:
>
> > Gluster allows converting from replicted to distributed-replicated (so called expansion) of the volume.
> > Of course you need to add the same amount of bricks as the replica count (which should be either 'replica 3' or 'replica 3 arbiter 1').
> > The Gluster documentation on the topic is quite extensive , but it's worth mentioning that you need to 'rebalance' your cluster after the expansion or you risk filling the old bricks - while having free space on the new bricks.
> > That is expected ,as each file/dir's name is hashed and each subvolume (new vs old bricks) will have it's own range of hashes. When you expand the volume, some of the files/dirs on the old bricks have hashes that match to the new "triplet".
> > Also , if you do not rebalance - that will have some performance impact as gluster will search the new bricks (for the files' whose hash matches the new subvolume) before searching the old bricks.
> > P.S: I think that the Engine's web interface fully supports that operation, although I'm used to the cli.
> > Best Regards,
> > Strahil Nikolov
> > В четвъртък, 15 април 2021 г., 20:05:50 ч. Гринуич+3, David White via Users users(a)ovirt.org написа:
> > Is it possible to expand an existing gluster volume?
> > I have a hyperconverged environment, and have enough space right now, but I'm going to have to significantly over-provision my environment. The vast majority of our customers are using a small fraction of the amount of space that they are technically allocated, but we still need to make that space available to them.
> > As a result, I'd like to plan ahead, and go ahead and have a plan to add storage later down the road.
> > I'd like to plan to shut down each server in the cluster (individually, not at once), add storage, and then bring them back online. Once all 3 are back online with the additional storage, I'd like to expand the gluster volume to use the additional space. Is that possible? How?
> > Sent with ProtonMail Secure Email.
> > Users mailing list -- users(a)ovirt.org
> > To unsubscribe send an email to users-leave(a)ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/A7C3X2PW4D4...
3 years, 8 months
glance repo in 4.3
by Christoph Köhler
I use ovirt 4.3.10 and need to get access to a glance repo for vm
images/templates. The buld-in external provider 'Public Glance
repository for oVirt' (glance.ovirt.org) does not work, it seems not to
be maintained.
Does someone knows what to do?
Greetings from
Christoph
3 years, 8 months
ovirt-hosted-engine-cleanup
by Marko Vrgotic
Dear oVirt,
I have three HE Hosts. Upon upgrade from 4.3.5 to 4.3.10 the Hosted Engine did not start anymore on Host2, but working just fine on Host1 and Host3.
I tried to Undeploy – Deploy on Host2 but I noticed that hosted-engine.conf contains only ca path and host id. It looks like either Undeploy properly or Deploy is failing due to reason I am unable to find in the logs.
When checking the hosted-engine –vm-status, even though undeployed, the output was still containing Host2 in the list. Than I cleaned metadata on the Host1 and 3 for Host2 and at that point it was forgotten.
After that I tried adding the Host2 back to HE pool, first installed the Host2 with fresh OS and than added it to Hosts pool with HostedEngine DEPLOY option. Result was the same.
The VDSM on the Host2 was showing ERROR: “2021-04-16 06:47:30,422+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196)”
The broker is showing the error: “MainThread::WARNING::2021-04-16 06:47:27,514::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None'
Atm, I am thinking of executing the /usr/sbin/ovirt-hosted-engine-cleanup but its effect is not clear to me. Is it going to cleanup any HE deploy remains from the Host its executed on, or its going to cleanup the other two Hosts, rendering my platfrom offline? I have been reading through https://www.ovirt.org/images/Hosted-Engine-4.3-deep-dive.pdf but its effect is not clear.
I would have tested it already but this issue is on our production platform, so I need some experts input before taking further action.
Please advise.
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
m: +31 (65) 5734174
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<http://www.activevideo.com>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
3 years, 8 months
Need to change switch... ovirt kvm concerns?
by jpedrolima＠gmail.com
Hi
I need to migrate the switch (equipment change and VLAN implementation) so I ask you to tell me what to do about the ovirt KVM cluster.
What procedures and concerns I must have ?
best regards,
PL
3 years, 8 months
Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Marko Vrgotic
Adding screenshot representing the state:
[Graphical user interface, text Description automatically generated]
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
m: +31 (65) 5734174
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<http://www.activevideo.com>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
From: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
Date: Thursday, 15 April 2021 at 23:00
To: users(a)ovirt.org <users(a)ovirt.org>
Cc: Yedidyah Bar David <didi(a)redhat.com>
Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
Looking further onto storage part, checking the host which I am unable to re-add to HE Host pool:
[root@ovirt-sj-02 10.210.13.64:_hosted__engine]# ls -la
total 8
drwxr-xr-x. 3 nobody nobody 4096 Apr 15 20:47 .
drwxr-xr-x. 3 vdsm kvm 42 Apr 15 14:30 ..
drwxr-xr-x. 6 nobody nobody 4096 Aug 20 2019 054c43fc-1924-4106-9f80-0f2ac62b9886
-rwxr-xr-x. 1 nobody nobody 0 Feb 18 2020 __DIRECT_IO_TEST__
[root@ovirt-sj-02 10.210.13.64:_hosted__engine]# cd 054c43fc-1924-4106-9f80-0f2ac62b9886/
[root@ovirt-sj-02 054c43fc-1924-4106-9f80-0f2ac62b9886]# ls
dom_md ha_agent images master
[root@ovirt-sj-02 054c43fc-1924-4106-9f80-0f2ac62b9886]# cd ha_agent/
[root@ovirt-sj-02 ha_agent]# ls
hosted-engine.lockspace hosted-engine.metadata
[root@ovirt-sj-02 ha_agent]# cat hosted-engine.lockspace
cat: hosted-engine.lockspace: No such file or directory
[root@ovirt-sj-02 ha_agent]# ls -la
total 16
drwxr-xr-x. 2 nobody nobody 4096 Mar 31 10:30 .
drwxr-xr-x. 6 nobody nobody 4096 Aug 20 2019 ..
lrwxrwxrwx. 1 nobody nobody 132 Mar 31 10:30 hosted-engine.lockspace -> /var/run/vdsm/storage/054c43fc-1924-4106-9f80-0f2ac62b9886/e08188be-f733-4d5c-9222-a4b4e2228955/081f81c5-b2b2-46d5-9f82-9d9041ccc108
lrwxrwxrwx. 1 nobody nobody 132 Mar 31 10:30 hosted-engine.metadata -> /var/run/vdsm/storage/054c43fc-1924-4106-9f80-0f2ac62b9886/16b3e5ac-e70b-46e3-bf81-322954fe0b44/b6326e48-a7d2-4cba-af91-441db9f353c2
[root@ovirt-sj-02 ha_agent]# cat
^C
[root@ovirt-sj-02 ha_agent]# cat /var/run/vdsm/storage/054c43fc-1924-4106-9f80-0f2ac62b9886/e08188be-f733-4d5c-9222-a4b4e2228955/081f81c5-b2b2-46d5-9f82-9d9041ccc108
cat: /var/run/vdsm/storage/054c43fc-1924-4106-9f80-0f2ac62b9886/e08188be-f733-4d5c-9222-a4b4e2228955/081f81c5-b2b2-46d5-9f82-9d9041ccc108: No such file or directory
It looks like there is still lockspace and metalinks which point to location thatno longer exists – the links are marked with red.
The broker.log is showing the following:
Main Thread WARNING storage_broker ovirt_hosted_engine_ha.broker.storage_broker.Storage Broker Can't connect vdsm storage: 'metadata_image_UUID can't be 'None'
I am starting to think that I need to run the lockspace reinitialization:
1. on each HE host
systemctl stop ovirt-ha-agent ovirt-ha-brokersanlock client shutdown -f 1 # carefully, it could trigger the watchdog and reboot
2. on a single hosthosted-engine --reinitialize-lockspace
3. on each HE hostsystemctl start ovirt-ha-agent ovirt-ha-broker
Is action 2 required to be executed only on Host with an issue or the action itself is gonna reinitialize lockspace on all HE Hosts?
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
m: +31 (65) 5734174
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<http://www.activevideo.com>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
From: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
Date: Thursday, 15 April 2021 at 16:57
To: Yedidyah Bar David <didi(a)redhat.com>
Cc: users(a)ovirt.org <users(a)ovirt.org>
Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
Hi Didi,
I compared the hosted-engine.conf on all three machines and indeed, host 1 and 3 have identical ones , except hosted.
Hosted-engine.conf on host2 that I am trying to add back contains only hostid and ca path:
ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem
host_id=2
Can someone help me how to check if there is DB or Storage corruption?
Would it be dectructive or risky to try to populate the hosted-engine.conf of host 2 with missing values?
Any advices?
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
m: +31 (65) 5734174
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<http://www.activevideo.com>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
From: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
Date: Wednesday, 14 April 2021 at 16:16
To: Yedidyah Bar David <didi(a)redhat.com>
Cc: users(a)ovirt.org <users(a)ovirt.org>
Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
Hi Didi,
It looks like the issue was with Hosted-engine Undeploy, being incomplete – the other HE Hosts still had the entries of the Host I was trying to remove, so any following HE Deploy on that Host was failing.
I was able to get the other hosts to forget about this one, by running hosted-engine –clean-metadate –host-id=2
Now I would like to try to add the host back to HE pool, but I have a question: “Is there a time I should wait, between cleaning metadata and re-adding the host?”
Kindly awaiting your reply.
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
m: +31 (65) 5734174
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<http://www.activevideo.com>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
From: Yedidyah Bar David <didi(a)redhat.com>
Date: Thursday, 18 March 2021 at 15:09
To: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
Cc: users(a)ovirt.org <users(a)ovirt.org>
Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!***
Hi,
On Mon, Mar 8, 2021 at 4:55 PM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote:
>
> The broker log, these lines are pretty much repeating:
>
>
>
> MainThread::WARNING::2021-03-03 09:19:12,086::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None'
Please compare the content of
/etc/ovirt-hosted-engine/hosted-engine.conf between all your hosts.
host id should be unique per host, but otherwise they should be
identical. If they are not, most likely there is some corruption
somewhere - in the engine db or shared storage.
You might want to skim this for a general rather-low-level overview:
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovi...
Do you see no errors on your other hosts? In -ha logs?
Please also note that 4.3 is EOL. The deploy process was completely
rewritten in 4.4 (in ansible, previous was python), although should in
principle behave similarly - so if your data is corrupted, upgrade to
4.4 probably won't fix it.
Good luck and best regards,
>
> MainThread::INFO::2021-03-03 09:19:12,829::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.6 started
>
> MainThread::INFO::2021-03-03 09:19:12,829::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/sub
>
> monitors
>
> MainThread::INFO::2021-03-03 09:19:12,829::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load
>
> MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine
>
> MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health
>
> MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free
>
> MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge
>
> MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor network
>
> MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain
>
> MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load
>
> MainThread::INFO::2021-03-03 09:19:12,834::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine
>
> MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health
>
> MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free
>
> MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge
>
> MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor network
>
> MainThread::INFO::2021-03-03 09:19:12,836::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain
>
> MainThread::INFO::2021-03-03 09:19:12,836::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors
>
> MainThread::WARNING::2021-03-03 09:19:12,836::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None'
>
> MainThread::INFO::2021-03-03 09:19:13,574::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.6 started
>
> MainThread::INFO::2021-03-03 09:19:13,575::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors
>
> MainThread::INFO::2021-03-03 09:19:13,575::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load
>
> MainThread::INFO::2021-03-03 09:19:13,577::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine
>
> MainThread::INFO::2021-03-03 09:19:13,578::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health
>
>
>
>
>
>
>
> -----
>
> kind regards/met vriendelijke groeten
>
>
>
> Marko Vrgotic
> Sr. System Engineer @ System Administration
>
>
> ActiveVideo
>
> o: +31 (35) 6774131
>
> m: +31 (65) 5734174
>
> e: m.vrgotic(a)activevideo.com
> w: https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.acti...
>
>
>
> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
>
>
>
>
>
>
>
> From: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
> Date: Monday, 8 March 2021 at 15:34
> To: Yedidyah Bar David <didi(a)redhat.com>
> Cc: users(a)ovirt.org <users(a)ovirt.org>
> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
>
> Hi Didi,
>
>
>
> Please find the attached logs from Host and Engine.
>
>
>
> Host ovirt-sj-02 HE Undeploy 2021-03-08 14:15:52 till 2021-03-08 14:18:24
>
>
>
>
>
> Host ovirt-sj-02 HE Deploy 2021-03-08 14:20:51 till 2021-03-08 14:23:22
>
>
>
> I do see errors in the agent and broker and vdsm, but I do not see why it happened.
>
>
>
> Thank you for helping, let me know if any additional files are needed.
>
>
>
>
>
> -----
>
> kind regards/met vriendelijke groeten
>
>
>
> Marko Vrgotic
> Sr. System Engineer @ System Administration
>
>
> ActiveVideo
>
> o: +31 (35) 6774131
>
> m: +31 (65) 5734174
>
> e: m.vrgotic(a)activevideo.com
> w: https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.acti...
>
>
>
> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
>
>
>
>
>
>
>
> From: Yedidyah Bar David <didi(a)redhat.com>
> Date: Monday, 8 March 2021 at 09:25
> To: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
> Cc: users(a)ovirt.org <users(a)ovirt.org>
> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
>
> ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!***
>
> Hi,
>
> On Mon, Mar 8, 2021 at 10:13 AM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote:
> >
> > I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before.
> >
> > No errors, found int the deployment, but it seems half done, based on messages I sent in previous email.
>
> Please check/share all relevant logs. Thanks. Can be all of /var/log
> from engine and hosts, and at least:
>
> /var/log/ovirt-engine/engine.log
>
> /var/log/vdsm/*
>
> /var/log/ovirt-hosted-engine-ha/*
>
> Best regards,
> --
> Didi
--
Didi
3 years, 8 months
Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Marko Vrgotic
Looking further onto storage part, checking the host which I am unable to re-add to HE Host pool:
[root@ovirt-sj-02 10.210.13.64:_hosted__engine]# ls -la
total 8
drwxr-xr-x. 3 nobody nobody 4096 Apr 15 20:47 .
drwxr-xr-x. 3 vdsm kvm 42 Apr 15 14:30 ..
drwxr-xr-x. 6 nobody nobody 4096 Aug 20 2019 054c43fc-1924-4106-9f80-0f2ac62b9886
-rwxr-xr-x. 1 nobody nobody 0 Feb 18 2020 __DIRECT_IO_TEST__
[root@ovirt-sj-02 10.210.13.64:_hosted__engine]# cd 054c43fc-1924-4106-9f80-0f2ac62b9886/
[root@ovirt-sj-02 054c43fc-1924-4106-9f80-0f2ac62b9886]# ls
dom_md ha_agent images master
[root@ovirt-sj-02 054c43fc-1924-4106-9f80-0f2ac62b9886]# cd ha_agent/
[root@ovirt-sj-02 ha_agent]# ls
hosted-engine.lockspace hosted-engine.metadata
[root@ovirt-sj-02 ha_agent]# cat hosted-engine.lockspace
cat: hosted-engine.lockspace: No such file or directory
[root@ovirt-sj-02 ha_agent]# ls -la
total 16
drwxr-xr-x. 2 nobody nobody 4096 Mar 31 10:30 .
drwxr-xr-x. 6 nobody nobody 4096 Aug 20 2019 ..
lrwxrwxrwx. 1 nobody nobody 132 Mar 31 10:30 hosted-engine.lockspace -> /var/run/vdsm/storage/054c43fc-1924-4106-9f80-0f2ac62b9886/e08188be-f733-4d5c-9222-a4b4e2228955/081f81c5-b2b2-46d5-9f82-9d9041ccc108
lrwxrwxrwx. 1 nobody nobody 132 Mar 31 10:30 hosted-engine.metadata -> /var/run/vdsm/storage/054c43fc-1924-4106-9f80-0f2ac62b9886/16b3e5ac-e70b-46e3-bf81-322954fe0b44/b6326e48-a7d2-4cba-af91-441db9f353c2
[root@ovirt-sj-02 ha_agent]# cat
^C
[root@ovirt-sj-02 ha_agent]# cat /var/run/vdsm/storage/054c43fc-1924-4106-9f80-0f2ac62b9886/e08188be-f733-4d5c-9222-a4b4e2228955/081f81c5-b2b2-46d5-9f82-9d9041ccc108
cat: /var/run/vdsm/storage/054c43fc-1924-4106-9f80-0f2ac62b9886/e08188be-f733-4d5c-9222-a4b4e2228955/081f81c5-b2b2-46d5-9f82-9d9041ccc108: No such file or directory
It looks like there is still lockspace and metalinks which point to location thatno longer exists – the links are marked with red.
The broker.log is showing the following:
Main Thread WARNING storage_broker ovirt_hosted_engine_ha.broker.storage_broker.Storage Broker Can't connect vdsm storage: 'metadata_image_UUID can't be 'None'
I am starting to think that I need to run the lockspace reinitialization:
1. on each HE host
systemctl stop ovirt-ha-agent ovirt-ha-brokersanlock client shutdown -f 1 # carefully, it could trigger the watchdog and reboot
2. on a single hosthosted-engine --reinitialize-lockspace
3. on each HE hostsystemctl start ovirt-ha-agent ovirt-ha-broker
Is action 2 required to be executed only on Host with an issue or the action itself is gonna reinitialize lockspace on all HE Hosts?
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
m: +31 (65) 5734174
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<http://www.activevideo.com>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
From: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
Date: Thursday, 15 April 2021 at 16:57
To: Yedidyah Bar David <didi(a)redhat.com>
Cc: users(a)ovirt.org <users(a)ovirt.org>
Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
Hi Didi,
I compared the hosted-engine.conf on all three machines and indeed, host 1 and 3 have identical ones , except hosted.
Hosted-engine.conf on host2 that I am trying to add back contains only hostid and ca path:
ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem
host_id=2
Can someone help me how to check if there is DB or Storage corruption?
Would it be dectructive or risky to try to populate the hosted-engine.conf of host 2 with missing values?
Any advices?
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
m: +31 (65) 5734174
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<http://www.activevideo.com>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
From: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
Date: Wednesday, 14 April 2021 at 16:16
To: Yedidyah Bar David <didi(a)redhat.com>
Cc: users(a)ovirt.org <users(a)ovirt.org>
Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
Hi Didi,
It looks like the issue was with Hosted-engine Undeploy, being incomplete – the other HE Hosts still had the entries of the Host I was trying to remove, so any following HE Deploy on that Host was failing.
I was able to get the other hosts to forget about this one, by running hosted-engine –clean-metadate –host-id=2
Now I would like to try to add the host back to HE pool, but I have a question: “Is there a time I should wait, between cleaning metadata and re-adding the host?”
Kindly awaiting your reply.
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
m: +31 (65) 5734174
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<http://www.activevideo.com>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
From: Yedidyah Bar David <didi(a)redhat.com>
Date: Thursday, 18 March 2021 at 15:09
To: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
Cc: users(a)ovirt.org <users(a)ovirt.org>
Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!***
Hi,
On Mon, Mar 8, 2021 at 4:55 PM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote:
>
> The broker log, these lines are pretty much repeating:
>
>
>
> MainThread::WARNING::2021-03-03 09:19:12,086::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None'
Please compare the content of
/etc/ovirt-hosted-engine/hosted-engine.conf between all your hosts.
host id should be unique per host, but otherwise they should be
identical. If they are not, most likely there is some corruption
somewhere - in the engine db or shared storage.
You might want to skim this for a general rather-low-level overview:
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovi...
Do you see no errors on your other hosts? In -ha logs?
Please also note that 4.3 is EOL. The deploy process was completely
rewritten in 4.4 (in ansible, previous was python), although should in
principle behave similarly - so if your data is corrupted, upgrade to
4.4 probably won't fix it.
Good luck and best regards,
>
> MainThread::INFO::2021-03-03 09:19:12,829::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.6 started
>
> MainThread::INFO::2021-03-03 09:19:12,829::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/sub
>
> monitors
>
> MainThread::INFO::2021-03-03 09:19:12,829::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load
>
> MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine
>
> MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health
>
> MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free
>
> MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge
>
> MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor network
>
> MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain
>
> MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load
>
> MainThread::INFO::2021-03-03 09:19:12,834::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine
>
> MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health
>
> MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free
>
> MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge
>
> MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor network
>
> MainThread::INFO::2021-03-03 09:19:12,836::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain
>
> MainThread::INFO::2021-03-03 09:19:12,836::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors
>
> MainThread::WARNING::2021-03-03 09:19:12,836::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None'
>
> MainThread::INFO::2021-03-03 09:19:13,574::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.6 started
>
> MainThread::INFO::2021-03-03 09:19:13,575::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors
>
> MainThread::INFO::2021-03-03 09:19:13,575::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load
>
> MainThread::INFO::2021-03-03 09:19:13,577::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine
>
> MainThread::INFO::2021-03-03 09:19:13,578::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health
>
>
>
>
>
>
>
> -----
>
> kind regards/met vriendelijke groeten
>
>
>
> Marko Vrgotic
> Sr. System Engineer @ System Administration
>
>
> ActiveVideo
>
> o: +31 (35) 6774131
>
> m: +31 (65) 5734174
>
> e: m.vrgotic(a)activevideo.com
> w: https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.acti...
>
>
>
> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
>
>
>
>
>
>
>
> From: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
> Date: Monday, 8 March 2021 at 15:34
> To: Yedidyah Bar David <didi(a)redhat.com>
> Cc: users(a)ovirt.org <users(a)ovirt.org>
> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
>
> Hi Didi,
>
>
>
> Please find the attached logs from Host and Engine.
>
>
>
> Host ovirt-sj-02 HE Undeploy 2021-03-08 14:15:52 till 2021-03-08 14:18:24
>
>
>
>
>
> Host ovirt-sj-02 HE Deploy 2021-03-08 14:20:51 till 2021-03-08 14:23:22
>
>
>
> I do see errors in the agent and broker and vdsm, but I do not see why it happened.
>
>
>
> Thank you for helping, let me know if any additional files are needed.
>
>
>
>
>
> -----
>
> kind regards/met vriendelijke groeten
>
>
>
> Marko Vrgotic
> Sr. System Engineer @ System Administration
>
>
> ActiveVideo
>
> o: +31 (35) 6774131
>
> m: +31 (65) 5734174
>
> e: m.vrgotic(a)activevideo.com
> w: https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.acti...
>
>
>
> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
>
>
>
>
>
>
>
> From: Yedidyah Bar David <didi(a)redhat.com>
> Date: Monday, 8 March 2021 at 09:25
> To: Marko Vrgotic <M.Vrgotic(a)activevideo.com>
> Cc: users(a)ovirt.org <users(a)ovirt.org>
> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
>
> ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!***
>
> Hi,
>
> On Mon, Mar 8, 2021 at 10:13 AM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote:
> >
> > I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before.
> >
> > No errors, found int the deployment, but it seems half done, based on messages I sent in previous email.
>
> Please check/share all relevant logs. Thanks. Can be all of /var/log
> from engine and hosts, and at least:
>
> /var/log/ovirt-engine/engine.log
>
> /var/log/vdsm/*
>
> /var/log/ovirt-hosted-engine-ha/*
>
> Best regards,
> --
> Didi
--
Didi
3 years, 8 months
