On Thu, Aug 18, 2016 at 10:06 AM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: > Hello, > > I'm having issues after the last 4.0.1 package just before 4.0.2 came > out. My engine was running great, hosts started the vm also and the > hosted-engine command was working fine. > > Now I get rpcxml 3.6 deprecated warning which I never got before. See > the attachment for this output. That one it's just a warning and we are going to fix the root cause for the next version, it's absolutely harmless. > I have checked is IPv6 was disabled, but it isn't as that is what is > needed for this. > > Any clue ? If you have any issue, it's somewhere else. What are the symptoms? > Thanks! > > Matt > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >

OK, it seems to be an issue related to something like a sanlock which I cannot solve: vdsm "Failed to acquire lock: No space left on device" is what happens when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is newly created on a vdsmd restart.

As I'm not able to change the maintenance mode on the hosts I need to figure out something else. 2016-08-18 10:40 GMT+02:00 Matt . <yamakasi.014(a)gmail.com&gt;: > OK nice to know, I never saw that message before. > > I cannot do anything anymore with hosted engine. It doesn't change the > maintenance state, it doesn't start the vm and this happens on all > hosts. Nothing else changed then just the ovirt* and vdsm* package > upgrades. > > The HE storage is also mounted and the agent.log shows it's > information as normal. > > Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. > > > > > 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi <stirabos(a)redhat.com&gt;: >> On Thu, Aug 18, 2016 at 10:06 AM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >>> Hello, >>> >>> I'm having issues after the last 4.0.1 package just before 4.0.2 came >>> out. My engine was running great, hosts started the vm also and the >>> hosted-engine command was working fine. >>> >>> Now I get rpcxml 3.6 deprecated warning which I never got before. See >>> the attachment for this output. >> >> That one it's just a warning and we are going to fix the root cause >> for the next version, it's absolutely harmless. >> >>> I have checked is IPv6 was disabled, but it isn't as that is what is >>> needed for this. >>> >>> Any clue ? >> >> If you have any issue, it's somewhere else. >> What are the symptoms? >> >>> Thanks! >>> >>> Matt >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

There is not storage issue, there is plenty of space, around 900G and this is shown in the past as the same issue. What is strange is that I cannot change the maintenance status of the hosts, when I set it to whatever status it's not changed and the status are various it seems from Global to Local. So I'm stuck.

2016-08-18 16:41 GMT+02:00 Nir Soffer <nsoffer(a)redhat.com&gt;: > On Thu, Aug 18, 2016 at 5:33 PM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >> OK, it seems to be an issue related to something like a sanlock which >> I cannot solve: >> >> vdsm "Failed to acquire lock: No space left on device" is what happens >> when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is >> newly created on a vdsmd restart. > > __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O > to this storage, there is no need to remove it. > > "No space left on device" seems to be your problem, you should make some > space if you want to use this storage domain. > >> >> As I'm not able to change the maintenance mode on the hosts I need to >> figure out something else. >> >> >> >> >> >> 2016-08-18 10:40 GMT+02:00 Matt . <yamakasi.014(a)gmail.com&gt;: >>> OK nice to know, I never saw that message before. >>> >>> I cannot do anything anymore with hosted engine. It doesn't change the >>> maintenance state, it doesn't start the vm and this happens on all >>> hosts. Nothing else changed then just the ovirt* and vdsm* package >>> upgrades. >>> >>> The HE storage is also mounted and the agent.log shows it's >>> information as normal. >>> >>> Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. >>> >>> >>> >>> >>> 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi <stirabos(a)redhat.com&gt;: >>>> On Thu, Aug 18, 2016 at 10:06 AM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >>>>> Hello, >>>>> >>>>> I'm having issues after the last 4.0.1 package just before 4.0.2 came >>>>> out. My engine was running great, hosts started the vm also and the >>>>> hosted-engine command was working fine. >>>>> >>>>> Now I get rpcxml 3.6 deprecated warning which I never got before. See >>>>> the attachment for this output. >>>> >>>> That one it's just a warning and we are going to fix the root cause >>>> for the next version, it's absolutely harmless. >>>> >>>>> I have checked is IPv6 was disabled, but it isn't as that is what is >>>>> needed for this. >>>>> >>>>> Any clue ? >>>> >>>> If you have any issue, it's somewhere else. >>>> What are the symptoms? >>>> >>>>> Thanks! >>>>> >>>>> Matt >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users

Sure, It's in the attachment as screenshot.

2016-08-18 17:13 GMT+02:00 Simone Tiraboschi <stirabos(a)redhat.com&gt;: > On Thu, Aug 18, 2016 at 5:05 PM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >> There is not storage issue, there is plenty of space, around 900G and >> this is shown in the past as the same issue. >> >> What is strange is that I cannot change the maintenance status of the >> hosts, when I set it to whatever status it's not changed and the >> status are various it seems from Global to Local. >> >> So I'm stuck. > > can you please post the output of > sanlock client status > ? > >> 2016-08-18 16:41 GMT+02:00 Nir Soffer <nsoffer(a)redhat.com&gt;: >>> On Thu, Aug 18, 2016 at 5:33 PM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >>>> OK, it seems to be an issue related to something like a sanlock which >>>> I cannot solve: >>>> >>>> vdsm "Failed to acquire lock: No space left on device" is what happens >>>> when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is >>>> newly created on a vdsmd restart. >>> >>> __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O >>> to this storage, there is no need to remove it. >>> >>> "No space left on device" seems to be your problem, you should make some >>> space if you want to use this storage domain. >>> >>>> >>>> As I'm not able to change the maintenance mode on the hosts I need to >>>> figure out something else. >>>> >>>> >>>> >>>> >>>> >>>> 2016-08-18 10:40 GMT+02:00 Matt . <yamakasi.014(a)gmail.com&gt;: >>>>> OK nice to know, I never saw that message before. >>>>> >>>>> I cannot do anything anymore with hosted engine. It doesn't change the >>>>> maintenance state, it doesn't start the vm and this happens on all >>>>> hosts. Nothing else changed then just the ovirt* and vdsm* package >>>>> upgrades. >>>>> >>>>> The HE storage is also mounted and the agent.log shows it's >>>>> information as normal. >>>>> >>>>> Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. >>>>> >>>>> >>>>> >>>>> >>>>> 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi <stirabos(a)redhat.com&gt;: >>>>>> On Thu, Aug 18, 2016 at 10:06 AM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I'm having issues after the last 4.0.1 package just before 4.0.2 came >>>>>>> out. My engine was running great, hosts started the vm also and the >>>>>>> hosted-engine command was working fine. >>>>>>> >>>>>>> Now I get rpcxml 3.6 deprecated warning which I never got before. See >>>>>>> the attachment for this output. >>>>>> >>>>>> That one it's just a warning and we are going to fix the root cause >>>>>> for the next version, it's absolutely harmless. >>>>>> >>>>>>> I have checked is IPv6 was disabled, but it isn't as that is what is >>>>>>> needed for this. >>>>>>> >>>>>>> Any clue ? >>>>>> >>>>>> If you have any issue, it's somewhere else. >>>>>> What are the symptoms? >>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users

It takes a while and this is the status output of systemctl after it, see attachment. 2016-08-18 17:40 GMT+02:00 Simone Tiraboschi <stirabos(a)redhat.com&gt;: > On Thu, Aug 18, 2016 at 5:28 PM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >> Sure, It's in the attachment as screenshot. > > can you please try with > systemctl restart sanlock --force > ? > >> 2016-08-18 17:13 GMT+02:00 Simone Tiraboschi <stirabos(a)redhat.com&gt;: >>> On Thu, Aug 18, 2016 at 5:05 PM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >>>> There is not storage issue, there is plenty of space, around 900G and >>>> this is shown in the past as the same issue. >>>> >>>> What is strange is that I cannot change the maintenance status of the >>>> hosts, when I set it to whatever status it's not changed and the >>>> status are various it seems from Global to Local. >>>> >>>> So I'm stuck. >>> >>> can you please post the output of >>> sanlock client status >>> ? >>> >>>> 2016-08-18 16:41 GMT+02:00 Nir Soffer <nsoffer(a)redhat.com&gt;: >>>>> On Thu, Aug 18, 2016 at 5:33 PM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >>>>>> OK, it seems to be an issue related to something like a sanlock which >>>>>> I cannot solve: >>>>>> >>>>>> vdsm "Failed to acquire lock: No space left on device" is what happens >>>>>> when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is >>>>>> newly created on a vdsmd restart. >>>>> >>>>> __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O >>>>> to this storage, there is no need to remove it. >>>>> >>>>> "No space left on device" seems to be your problem, you should make some >>>>> space if you want to use this storage domain. >>>>> >>>>>> >>>>>> As I'm not able to change the maintenance mode on the hosts I need to >>>>>> figure out something else. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> 2016-08-18 10:40 GMT+02:00 Matt . <yamakasi.014(a)gmail.com&gt;: >>>>>>> OK nice to know, I never saw that message before. >>>>>>> >>>>>>> I cannot do anything anymore with hosted engine. It doesn't change the >>>>>>> maintenance state, it doesn't start the vm and this happens on all >>>>>>> hosts. Nothing else changed then just the ovirt* and vdsm* package >>>>>>> upgrades. >>>>>>> >>>>>>> The HE storage is also mounted and the agent.log shows it's >>>>>>> information as normal. >>>>>>> >>>>>>> Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi <stirabos(a)redhat.com&gt;: >>>>>>>> On Thu, Aug 18, 2016 at 10:06 AM, Matt . <yamakasi.014(a)gmail.com&gt; wrote: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I'm having issues after the last 4.0.1 package just before 4.0.2 came >>>>>>>>> out. My engine was running great, hosts started the vm also and the >>>>>>>>> hosted-engine command was working fine. >>>>>>>>> >>>>>>>>> Now I get rpcxml 3.6 deprecated warning which I never got before. See >>>>>>>>> the attachment for this output. >>>>>>>> >>>>>>>> That one it's just a warning and we are going to fix the root cause >>>>>>>> for the next version, it's absolutely harmless. >>>>>>>> >>>>>>>>> I have checked is IPv6 was disabled, but it isn't as that is what is >>>>>>>>> needed for this. >>>>>>>>> >>>>>>>>> Any clue ? >>>>>>>> >>>>>>>> If you have any issue, it's somewhere else. >>>>>>>> What are the symptoms? >>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users(a)ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users