How to clean stale engine from "hosted-engine unknow stale-data"
Hi All, How to clean stale hosted engine from the database. I want to remove host id=3 Ovirt - 4.4.9 [root@iondelsvr14 vdsm]# hosted-engine --vm-status --== Host iondelsvr10.iontrading.com (id: 1) status ==-- Host ID : 1 Host timestamp : 2632783 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : iondelsvr10.iontrading.com Local maintenance : False stopped : False crc32 : 882bb390 conf_on_shared_storage : True local_conf_timestamp : 2632783 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2632783 (Thu Feb 3 09:00:14 2022) host-id=1 score=3400 vm_conf_refresh_time=2632783 (Thu Feb 3 09:00:14 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host iondelsvr11.iontrading.com (id: 2) status ==-- Host ID : 2 Host timestamp : 6215454 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : iondelsvr11.iontrading.com Local maintenance : False stopped : False crc32 : 30153e90 conf_on_shared_storage : True local_conf_timestamp : 6215454 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=6215454 (Thu Feb 3 09:00:16 2022) host-id=2 score=3400 vm_conf_refresh_time=6215454 (Thu Feb 3 09:00:16 2022) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False --== Host iondelsvr12.iontrading.com (id: 3) status ==-- Host ID : 3 Host timestamp : 734 Score : 1800 Engine status : unknown stale-data Hostname : iondelsvr12.iontrading.com Local maintenance : False stopped : False crc32 : 062e92e9 conf_on_shared_storage : True local_conf_timestamp : 734 Status up-to-date : False Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=734 (Fri Jan 28 17:35:18 2022) host-id=3 score=1800 vm_conf_refresh_time=734 (Fri Jan 28 17:35:18 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False
On Thu, Feb 3, 2022 at 5:53 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi All,
How to clean stale hosted engine from the database. I want to remove host id=3
Perhaps by using the options '--clean_metadata' and '--host-id' of the command 'hosted-engine'. You can check documentation, list archives, etc. for details. Good luck and best regards, -- Didi
Hi Yedidyah, I am thinking to do below - # hosted-engine --set-maintenance --mode=global # hosted-engine --clean-metadata --host-id=<old_host_id> --force-clean" I hope it is safe and tested ? Thanks On Thu, Feb 3, 2022 at 12:17 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Feb 3, 2022 at 5:53 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi All,
How to clean stale hosted engine from the database. I want to remove
host id=3
Perhaps by using the options '--clean_metadata' and '--host-id' of the command 'hosted-engine'. You can check documentation, list archives, etc. for details.
Good luck and best regards, -- Didi
On Thu, Feb 3, 2022 at 9:11 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
I am thinking to do below -
# hosted-engine --set-maintenance --mode=global # hosted-engine --clean-metadata --host-id=<old_host_id> --force-clean"
I hope it is safe and tested ?
It is tested, and it's "safe" as in "Does what it's designed to do". It's definitely not "safe" as in "riskless" - if you make a mistake, you risk damage. I didn't use it myself recently, and do not remember details anymore. It's definitely not something that you are supposed to use/need casually. If unsure, and if your environment is important to you (not just some test/dev/learn thing), I warmly suggest to try this first on a test env (can be done with nested-kvm VMs, no need for physical machines for this). That said, others on this list (and elsewhere) did use it successfully, and I recommend again to search for examples/reports/experiences. Good luck and best regards,
Thanks
On Thu, Feb 3, 2022 at 12:17 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Feb 3, 2022 at 5:53 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi All,
How to clean stale hosted engine from the database. I want to remove host id=3
Perhaps by using the options '--clean_metadata' and '--host-id' of the command 'hosted-engine'. You can check documentation, list archives, etc. for details.
Good luck and best regards, -- Didi
-- Didi
Hi Yedidyah, Thanks, After cleaning it from one host i am getting below error messages on /var/log/messages Feb 3 18:01:02 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:17 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:18 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes On Thu, Feb 3, 2022 at 1:04 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Feb 3, 2022 at 9:11 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
I am thinking to do below -
# hosted-engine --set-maintenance --mode=global # hosted-engine --clean-metadata --host-id=<old_host_id> --force-clean"
I hope it is safe and tested ?
It is tested, and it's "safe" as in "Does what it's designed to do".
It's definitely not "safe" as in "riskless" - if you make a mistake, you risk damage. I didn't use it myself recently, and do not remember details anymore. It's definitely not something that you are supposed to use/need casually.
If unsure, and if your environment is important to you (not just some test/dev/learn thing), I warmly suggest to try this first on a test env (can be done with nested-kvm VMs, no need for physical machines for this).
That said, others on this list (and elsewhere) did use it successfully, and I recommend again to search for examples/reports/experiences.
Good luck and best regards,
Thanks
On Thu, Feb 3, 2022 at 12:17 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Thu, Feb 3, 2022 at 5:53 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi All,
How to clean stale hosted engine from the database. I want to remove
host id=3
Perhaps by using the options '--clean_metadata' and '--host-id' of the command 'hosted-engine'. You can check documentation, list archives, etc. for details.
Good luck and best regards, -- Didi
-- Didi
Do I need to clean it from every host which has the hosted engine ? Sinc the other host still can see it. On Thu, Feb 3, 2022 at 6:02 PM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
Thanks, After cleaning it from one host i am getting below error messages on /var/log/messages
Feb 3 18:01:02 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:17 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:18 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes
On Thu, Feb 3, 2022 at 1:04 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Feb 3, 2022 at 9:11 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
I am thinking to do below -
# hosted-engine --set-maintenance --mode=global # hosted-engine --clean-metadata --host-id=<old_host_id> --force-clean"
I hope it is safe and tested ?
It is tested, and it's "safe" as in "Does what it's designed to do".
It's definitely not "safe" as in "riskless" - if you make a mistake, you risk damage. I didn't use it myself recently, and do not remember details anymore. It's definitely not something that you are supposed to use/need casually.
If unsure, and if your environment is important to you (not just some test/dev/learn thing), I warmly suggest to try this first on a test env (can be done with nested-kvm VMs, no need for physical machines for this).
That said, others on this list (and elsewhere) did use it successfully, and I recommend again to search for examples/reports/experiences.
Good luck and best regards,
Thanks
On Thu, Feb 3, 2022 at 12:17 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Thu, Feb 3, 2022 at 5:53 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi All,
How to clean stale hosted engine from the database. I want to remove
host id=3
Perhaps by using the options '--clean_metadata' and '--host-id' of the command 'hosted-engine'. You can check documentation, list archives, etc. for details.
Good luck and best regards, -- Didi
-- Didi
On Thu, Feb 3, 2022 at 2:48 PM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Do I need to clean it from every host which has the hosted engine ? Sinc the other host still can see it.
Are you sure it's reading the same space? Perhaps there is some misconfiguration or something? Caching?
On Thu, Feb 3, 2022 at 6:02 PM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
Thanks, After cleaning it from one host i am getting below error messages on /var/log/messages
Feb 3 18:01:02 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:17 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:18 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes
Not sure. Perhaps try to restart the ha broker, agent or both. If you do this in global maintenance, I think it should be safe - wait until everything looks ok before exiting global maint. Check also vdsm and ha logs. Good luck and best regards, -- Didi
On Thu, Feb 3, 2022 at 1:50 PM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Do I need to clean it from every host which has the hosted engine ? Sinc the other host still can see it.
No, the metadata "whiteboard" is shared. Just one copy that you removed. Once node 3 start reporting again, it will simply put new content in there. Not sure about vdsm, maybe some component (vdsm, engine) is still trying to get status of node 3. I have been away for too long, so Didi will have to pitch in. Martin Sivak
On Thu, Feb 3, 2022 at 6:02 PM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
Thanks, After cleaning it from one host i am getting below error messages on /var/log/messages
Feb 3 18:01:02 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:17 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:18 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes
On Thu, Feb 3, 2022 at 1:04 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Feb 3, 2022 at 9:11 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
I am thinking to do below -
# hosted-engine --set-maintenance --mode=global # hosted-engine --clean-metadata --host-id=<old_host_id> --force-clean"
I hope it is safe and tested ?
It is tested, and it's "safe" as in "Does what it's designed to do".
It's definitely not "safe" as in "riskless" - if you make a mistake, you risk damage. I didn't use it myself recently, and do not remember details anymore. It's definitely not something that you are supposed to use/need casually.
It should be safe in global maintenance, the only risk is when you remove the report of a node that currently runs the engine. As other nodes might decide that the engine needs restarting.
If unsure, and if your environment is important to you (not just some
test/dev/learn thing), I warmly suggest to try this first on a test env (can be done with nested-kvm VMs, no need for physical machines for this).
That said, others on this list (and elsewhere) did use it successfully, and I recommend again to search for examples/reports/experiences.
Good luck and best regards,
Thanks
On Thu, Feb 3, 2022 at 12:17 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Thu, Feb 3, 2022 at 5:53 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi All,
How to clean stale hosted engine from the database. I want to
remove host id=3
Perhaps by using the options '--clean_metadata' and '--host-id' of the command 'hosted-engine'. You can check documentation, list archives, etc. for details.
Good luck and best regards, -- Didi
-- Didi
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IR3SMGNMRRGEKX...
VDSM Logs - ual': True}} from=internal, task_id=b017c5d2-a941-4ccd-af02-3fac0a767df5 (api:54) 2022-02-03 18:53:53,368+0530 ERROR (periodic/3) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 3: received 0 of 512 expected bytes (client:137) 2022-02-03 18:53:54,529+0530 INFO (jsonrpc/5) [api.host] START getStats() from=::1,38990 (api:48) 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,38990, task_id=3864f22e-98cc-426f-a4d5-2e2254f34aae (api:48) 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'8ab8b572-39a3-4603-b0e7-2b6fe71c4d73': {'code': 0, 'lastCheck': '6.3', 'delay': '0.000359743', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'f3114b74-8b12-4493-9faf-0f514afa3a16': {'code': 0, 'lastCheck': '2.1', 'delay': '0.000447464', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '26bfbe5d-e427-47a9-9111-cca8a628fea8': {'code': 0, 'lastCheck': '2.0', 'delay': '0.00034851', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23540fa8-4de2-4715-97e1-4ce9fe4f66cc': {'code': 0, 'lastCheck': '2.1', 'delay': '0.000352501', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8fa0a28e-480f-444c-9cf2-bef536d3c52f': {'code': 0, 'lastCheck': '2.0', 'delay': '0.000320087', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'd1dba7f5-cb30-4571-8fe6-5fb5005bcdb2': {'code': 0, 'lastCheck': '2.6', 'delay': '0.000452836', 'valid': True, 'version': 0, 'acquired': True, 'actual': True}, '88b94c9a-6855-4199-9d37-1f88d21f211c': {'code': 0, 'lastCheck': '2.0', 'delay': '0.000299592', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '2e0ec671-5972-4276-aa49-aefc306f5da4': {'code': 0, 'lastCheck': '2.0', 'delay': '0.000316513', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,38990, task_id=3864f22e-98cc-426f-a4d5-2e2254f34aae (api:54) 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,38990, task_id=7ecaad0c-aa5d-45c0-96a5-dffc78572248 (api:48) 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,38990, task_id=7ecaad0c-aa5d-45c0-96a5-dffc78572248 (api:54) 2022-02-03 18:53:54,548+0530 ERROR (jsonrpc/5) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 3: received 0 of 512 expected bytes (client:137) 2022-02-03 18:53:54,549+0530 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,38990 (api:54) 2022-02-03 18:53:56,038+0530 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['8ab8b572-39a3-4603-b0e7-2b6fe71c4d73']) from=::1,38990, task_id=8855f314-9334-4b06-88e3-7a1a3f577049 (api:48) 2022-02-03 18:53:56,038+0530 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'8ab8b572-39a3-4603-b0e7-2b6fe71c4d73': {'code': 0, 'lastCheck': '1.3', 'delay': '0.000349788', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,38990, task_id=8855f314-9334-4b06-88e3-7a1a3f577049 (api:54) On Thu, Feb 3, 2022 at 6:42 PM Martin Sivak <msivak@redhat.com> wrote:
On Thu, Feb 3, 2022 at 1:50 PM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Do I need to clean it from every host which has the hosted engine ? Sinc the other host still can see it.
No, the metadata "whiteboard" is shared. Just one copy that you removed. Once node 3 start reporting again, it will simply put new content in there.
Not sure about vdsm, maybe some component (vdsm, engine) is still trying to get status of node 3. I have been away for too long, so Didi will have to pitch in.
Martin Sivak
On Thu, Feb 3, 2022 at 6:02 PM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
Thanks, After cleaning it from one host i am getting below error messages on /var/log/messages
Feb 3 18:01:02 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:17 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:18 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes
On Thu, Feb 3, 2022 at 1:04 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Feb 3, 2022 at 9:11 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
I am thinking to do below -
# hosted-engine --set-maintenance --mode=global # hosted-engine --clean-metadata --host-id=<old_host_id>
--force-clean"
I hope it is safe and tested ?
It is tested, and it's "safe" as in "Does what it's designed to do".
It's definitely not "safe" as in "riskless" - if you make a mistake, you risk damage. I didn't use it myself recently, and do not remember details anymore. It's definitely not something that you are supposed to use/need casually.
It should be safe in global maintenance, the only risk is when you remove the report of a node that currently runs the engine. As other nodes might decide that the engine needs restarting.
If unsure, and if your environment is important to you (not just some
test/dev/learn thing), I warmly suggest to try this first on a test env (can be done with nested-kvm VMs, no need for physical machines for this).
That said, others on this list (and elsewhere) did use it successfully, and I recommend again to search for examples/reports/experiences.
Good luck and best regards,
Thanks
On Thu, Feb 3, 2022 at 12:17 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Thu, Feb 3, 2022 at 5:53 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote: > > Hi All, > > How to clean stale hosted engine from the database. I want to
remove host id=3
Perhaps by using the options '--clean_metadata' and '--host-id' of the command 'hosted-engine'. You can check documentation, list archives, etc. for details.
Good luck and best regards, -- Didi
-- Didi
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IR3SMGNMRRGEKX...
I restarted the host and it seems like it's running fine...However do it need to reboot other hosts or restart some services on other hosts as well clean this up. Because they still can see the Host ID=3 [root@iondelsvr10 vdsm]# hosted-engine --vm-status --== Host iondelsvr10.iontrading.com (id: 1) status ==-- Host ID : 1 Host timestamp : 2672943 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : iondelsvr10.iontrading.com Local maintenance : False stopped : False crc32 : b737573e conf_on_shared_storage : True local_conf_timestamp : 2672943 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2672943 (Thu Feb 3 20:09:34 2022) host-id=1 score=3400 vm_conf_refresh_time=2672943 (Thu Feb 3 20:09:34 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host iondelsvr11.iontrading.com (id: 2) status ==-- Host ID : 2 Host timestamp : 6255602 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : iondelsvr11.iontrading.com Local maintenance : False stopped : False crc32 : 5ff71c1a conf_on_shared_storage : True local_conf_timestamp : 6255602 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=6255602 (Thu Feb 3 20:09:24 2022) host-id=2 score=3400 vm_conf_refresh_time=6255602 (Thu Feb 3 20:09:24 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host iondelsvr12.iontrading.com (id: 3) status ==-- Host ID : 3 Host timestamp : 734 Score : 1800 Engine status : unknown stale-data Hostname : iondelsvr12.iontrading.com Local maintenance : False stopped : False crc32 : 062e92e9 conf_on_shared_storage : True local_conf_timestamp : 734 Status up-to-date : False Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=734 (Fri Jan 28 17:35:18 2022) host-id=3 score=1800 vm_conf_refresh_time=734 (Fri Jan 28 17:35:18 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host iondelsvr14.iontrading.com (id: 4) status ==-- Host ID : 4 Host timestamp : 2670 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : iondelsvr14.iontrading.com Local maintenance : False stopped : False crc32 : fdf46b17 conf_on_shared_storage : True local_conf_timestamp : 2670 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2670 (Thu Feb 3 20:09:30 2022) host-id=4 score=3400 vm_conf_refresh_time=2670 (Thu Feb 3 20:09:30 2022) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False [root@iondelsvr10 vdsm]# On Thu, Feb 3, 2022 at 6:54 PM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
VDSM Logs -
ual': True}} from=internal, task_id=b017c5d2-a941-4ccd-af02-3fac0a767df5 (api:54) 2022-02-03 18:53:53,368+0530 ERROR (periodic/3) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 3: received 0 of 512 expected bytes (client:137) 2022-02-03 18:53:54,529+0530 INFO (jsonrpc/5) [api.host] START getStats() from=::1,38990 (api:48) 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,38990, task_id=3864f22e-98cc-426f-a4d5-2e2254f34aae (api:48) 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'8ab8b572-39a3-4603-b0e7-2b6fe71c4d73': {'code': 0, 'lastCheck': '6.3', 'delay': '0.000359743', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'f3114b74-8b12-4493-9faf-0f514afa3a16': {'code': 0, 'lastCheck': '2.1', 'delay': '0.000447464', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '26bfbe5d-e427-47a9-9111-cca8a628fea8': {'code': 0, 'lastCheck': '2.0', 'delay': '0.00034851', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23540fa8-4de2-4715-97e1-4ce9fe4f66cc': {'code': 0, 'lastCheck': '2.1', 'delay': '0.000352501', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8fa0a28e-480f-444c-9cf2-bef536d3c52f': {'code': 0, 'lastCheck': '2.0', 'delay': '0.000320087', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'd1dba7f5-cb30-4571-8fe6-5fb5005bcdb2': {'code': 0, 'lastCheck': '2.6', 'delay': '0.000452836', 'valid': True, 'version': 0, 'acquired': True, 'actual': True}, '88b94c9a-6855-4199-9d37-1f88d21f211c': {'code': 0, 'lastCheck': '2.0', 'delay': '0.000299592', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '2e0ec671-5972-4276-aa49-aefc306f5da4': {'code': 0, 'lastCheck': '2.0', 'delay': '0.000316513', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,38990, task_id=3864f22e-98cc-426f-a4d5-2e2254f34aae (api:54) 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,38990, task_id=7ecaad0c-aa5d-45c0-96a5-dffc78572248 (api:48) 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,38990, task_id=7ecaad0c-aa5d-45c0-96a5-dffc78572248 (api:54) 2022-02-03 18:53:54,548+0530 ERROR (jsonrpc/5) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 3: received 0 of 512 expected bytes (client:137) 2022-02-03 18:53:54,549+0530 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,38990 (api:54) 2022-02-03 18:53:56,038+0530 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['8ab8b572-39a3-4603-b0e7-2b6fe71c4d73']) from=::1,38990, task_id=8855f314-9334-4b06-88e3-7a1a3f577049 (api:48) 2022-02-03 18:53:56,038+0530 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'8ab8b572-39a3-4603-b0e7-2b6fe71c4d73': {'code': 0, 'lastCheck': '1.3', 'delay': '0.000349788', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,38990, task_id=8855f314-9334-4b06-88e3-7a1a3f577049 (api:54)
On Thu, Feb 3, 2022 at 6:42 PM Martin Sivak <msivak@redhat.com> wrote:
On Thu, Feb 3, 2022 at 1:50 PM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Do I need to clean it from every host which has the hosted engine ? Sinc the other host still can see it.
No, the metadata "whiteboard" is shared. Just one copy that you removed. Once node 3 start reporting again, it will simply put new content in there.
Not sure about vdsm, maybe some component (vdsm, engine) is still trying to get status of node 3. I have been away for too long, so Didi will have to pitch in.
Martin Sivak
On Thu, Feb 3, 2022 at 6:02 PM Ayansh Rocks < shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
Thanks, After cleaning it from one host i am getting below error messages on /var/log/messages
Feb 3 18:01:02 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:17 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:18 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for host 3: received 0 of 512 expected bytes
On Thu, Feb 3, 2022 at 1:04 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Thu, Feb 3, 2022 at 9:11 AM Ayansh Rocks <shashank123rastogi@gmail.com> wrote:
Hi Yedidyah,
I am thinking to do below -
# hosted-engine --set-maintenance --mode=global # hosted-engine --clean-metadata --host-id=<old_host_id>
--force-clean"
I hope it is safe and tested ?
It is tested, and it's "safe" as in "Does what it's designed to do".
It's definitely not "safe" as in "riskless" - if you make a mistake, you risk damage. I didn't use it myself recently, and do not remember details anymore. It's definitely not something that you are supposed to use/need casually.
It should be safe in global maintenance, the only risk is when you remove the report of a node that currently runs the engine. As other nodes might decide that the engine needs restarting.
If unsure, and if your environment is important to you (not just some
test/dev/learn thing), I warmly suggest to try this first on a test env (can be done with nested-kvm VMs, no need for physical machines for this).
That said, others on this list (and elsewhere) did use it successfully, and I recommend again to search for examples/reports/experiences.
Good luck and best regards,
Thanks
On Thu, Feb 3, 2022 at 12:17 PM Yedidyah Bar David <didi@redhat.com>
wrote:
> > On Thu, Feb 3, 2022 at 5:53 AM Ayansh Rocks > <shashank123rastogi@gmail.com> wrote: > > > > Hi All, > > > > How to clean stale hosted engine from the database. I want to remove host id=3 > > Perhaps by using the options '--clean_metadata' and '--host-id' > of the command 'hosted-engine'. > You can check documentation, list archives, etc. for details. > > Good luck and best regards, > -- > Didi >
-- Didi
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IR3SMGNMRRGEKX...
participants (3)
-
Ayansh Rocks -
Martin Sivak -
Yedidyah Bar David