7:04 a.m.
Not sure if all the same issue, but seems to failing around the same time:
  ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
min  integ-tests
  ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr -
#1045 51 min  integ-tests
  ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec 
integ-tests
  ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min 
integ-tests
  ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10
hr - #873 1 hr 36 min  integ-tests
  ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr -
#1149 1 hr 26 min  integ-tests
  ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr -
#729 1 hr 54 min  integ-tests
  ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr - #929 45
min  integ-tests
  ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10
hr - #190 45 min  integ-tests
Links to jobs can be found here: https://jenkins.ovirt.org/
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
2:23 a.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
At least for the network suite these failures are due to the fact that
DisconnectStoragePoolVDSCommand
is not finished and a change_cluster request is issued by the test but
revoked by engine which fails the test setup.
Ahmed,
Can you please have a look at the below engine log [2]- currently
change_cluster requests from the test are being revoked by engine. I
believe this is because the locking group of DisconnectStoragePoolVDSCommand
was moved to VDS in patch [1]
Is this a desired outcome and should the tests be changed to reflect it
(wait for which status notification from engine?)?
Thanks
P.S. please note that due to this patch
LockingGroup.VDS_POOL_AND_STORAGE_CONNECTIONS
has only one consumer, which means it does not lock against anything. Is
this a desired outcome of the patch?
[1] https://gerrit.ovirt.org/#/c/98405
[2]
Network-suite-master.tests.test_mac_pools.test_mac_pools_in_different_clusters_dont_overlap
Error Message
test setup failure
Stacktrace
default_data_center = <ovirtlib.datacenterlib.DataCenter object at
0x7fbe88aa8790>
host_1_up = <ovirtlib.hostlib.Host object at 0x7fbe8404a810>
cluster_0 = <ovirtlib.clusterlib.Cluster object at 0x7fbe77f5a050>
@pytest.fixture(scope='module')
def host_1_in_cluster_0(default_data_center, host_1_up, cluster_0):
current_cluster = host_1_up.get_cluster()
host_1_up.change_cluster(cluster_0)
network-suite-master/tests/test_mac_pools.py:49
2019-03-17 07:16:40,493-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-65) [] START,
DisconnectStoragePoolVDSCommand(HostName =
lago-network-suite-master-host-1,
DisconnectStoragePoolVDSCommandParameters:{hostId='2ab16b7a-fd38-41c5-98be-c8db1fdece42',
storagePoolId='239d1be2-48a4-11e9-bec6-5452c0a8c902', vds_spm_id='2'}),
log
id: 44ad4438
2019-03-17 07:16:43,635-04 INFO
[org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2)
[eded9adc-b780-4640-ba5e-ec989f620d57] Failed to Acquire Lock to object
'EngineLock:{exclusiveLocks='[2ab16b7a-fd38-41c5-98be-c8db1fdece42=VDS]',
sharedLocks=''}'
2019-03-17 07:16:43,635-04 WARN
[org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2)
[eded9adc-b780-4640-ba5e-ec989f620d57] Validation of action
'ChangeVDSCluster' failed for user admin@internal-authz. Reasons:
VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
2019-03-17 07:16:43,636-04 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] method: runAction,
params: [ChangeVDSCluster,
ChangeVDSClusterParameters:{commandId='00522716-6557-4da0-8d02-802fbf402c24',
user='null', commandType='Unknown'}], timeElapsed: 18ms
2019-03-17 07:16:43,637-04 ERROR
[org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
task-2) [] Operation Failed: [Cannot edit Host. Related operation is
currently in progress. Please try again later.]
2019-03-17 07:16:44,575-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-65) [] FINISH,
DisconnectStoragePoolVDSCommand, return: , log id: 44ad4438
On Sun, Mar 17, 2019 at 4:05 PM Eyal Edri <eedri(a)redhat.com> wrote:
Not sure if all the same issue, but seems to failing around the same
time:
  ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
min  integ-tests
  ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr -
#1045 51 min  integ-tests
  ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec 
integ-tests
  ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min 
integ-tests
  ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10
hr - #873 1 hr 36 min  integ-tests
  ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr -
#1149 1 hr 26 min  integ-tests
  ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr
- #729 1 hr 54 min  integ-tests
  ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr -
#929 45 min  integ-tests
  ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10
hr - #190 45 min  integ-tests
Links to jobs can be found here: https://jenkins.ovirt.org/
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________
Devel mailing list -- devel(a)ovirt.org
To unsubscribe send an email to devel-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/I75ZI4LYD3I...
4:58 a.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
We should probably lock both groups in
VdsEventListener#processStorageOnVdsInactive or in RemoveVdsCommand
Ahmad, please evaluate and adjust
On Wed, Mar 20, 2019 at 11:25 AM Eitan Raviv <eraviv(a)redhat.com> wrote:
At least for the network suite these failures are due to the fact
that DisconnectStoragePoolVDSCommand
is not finished and a change_cluster request is issued by the test but
revoked by engine which fails the test setup.
Ahmed,
Can you please have a look at the below engine log [2]- currently
change_cluster requests from the test are being revoked by engine. I
believe this is because the locking group of DisconnectStoragePoolVDSCommand
was moved to VDS in patch [1]
Is this a desired outcome and should the tests be changed to reflect it
(wait for which status notification from engine?)?
Thanks
P.S. please note that due to this patch LockingGroup.VDS_POOL_AND_STORAGE_CONNECTIONS
has only one consumer, which means it does not lock against anything. Is
this a desired outcome of the patch?
[1] https://gerrit.ovirt.org/#/c/98405
[2]
Network-suite-master.tests.test_mac_pools.test_mac_pools_in_different_clusters_dont_overlap
Error Message
test setup failure
Stacktrace
default_data_center = <ovirtlib.datacenterlib.DataCenter object at
0x7fbe88aa8790>
host_1_up = <ovirtlib.hostlib.Host object at 0x7fbe8404a810>
cluster_0 = <ovirtlib.clusterlib.Cluster object at 0x7fbe77f5a050>
@pytest.fixture(scope='module')
def host_1_in_cluster_0(default_data_center, host_1_up, cluster_0):
current_cluster = host_1_up.get_cluster()
> host_1_up.change_cluster(cluster_0)
network-suite-master/tests/test_mac_pools.py:49
2019-03-17 07:16:40,493-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-65) [] START,
DisconnectStoragePoolVDSCommand(HostName =
lago-network-suite-master-host-1,
DisconnectStoragePoolVDSCommandParameters:{hostId='2ab16b7a-fd38-41c5-98be-c8db1fdece42',
storagePoolId='239d1be2-48a4-11e9-bec6-5452c0a8c902', vds_spm_id='2'}),
log
id: 44ad4438
2019-03-17 07:16:43,635-04 INFO
[org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2)
[eded9adc-b780-4640-ba5e-ec989f620d57] Failed to Acquire Lock to object
'EngineLock:{exclusiveLocks='[2ab16b7a-fd38-41c5-98be-c8db1fdece42=VDS]',
sharedLocks=''}'
2019-03-17 07:16:43,635-04 WARN
[org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2)
[eded9adc-b780-4640-ba5e-ec989f620d57] Validation of action
'ChangeVDSCluster' failed for user admin@internal-authz. Reasons:
VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
2019-03-17 07:16:43,636-04 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] method: runAction,
params: [ChangeVDSCluster,
ChangeVDSClusterParameters:{commandId='00522716-6557-4da0-8d02-802fbf402c24',
user='null', commandType='Unknown'}], timeElapsed: 18ms
2019-03-17 07:16:43,637-04 ERROR
[org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
task-2) [] Operation Failed: [Cannot edit Host. Related operation is
currently in progress. Please try again later.]
2019-03-17 07:16:44,575-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-65) [] FINISH,
DisconnectStoragePoolVDSCommand, return: , log id: 44ad4438
On Sun, Mar 17, 2019 at 4:05 PM Eyal Edri <eedri(a)redhat.com> wrote:
> Not sure if all the same issue, but seems to failing around the same time:
>
>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
> min  integ-tests
>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr
> - #1045 51 min  integ-tests
>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec
>  integ-tests
>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min
>  integ-tests
>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10
> hr - #873 1 hr 36 min  integ-tests
>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr
> - #1149 1 hr 26 min  integ-tests
>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr
> - #729 1 hr 54 min  integ-tests
>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr -
> #929 45 min  integ-tests
>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10
> hr - #190 45 min  integ-tests
>
> Links to jobs can be found here: https://jenkins.ovirt.org/
>
> --
>
> Eyal edri
>
>
> MANAGER
>
> RHV/CNV DevOps
>
> EMEA VIRTUALIZATION R&D
>
>
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
<https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/I75ZI4LYD3I...
>
_______________________________________________
Devel mailing list -- devel(a)ovirt.org
To unsubscribe send an email to devel-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UXLFIW2KK4J...
5:05 a.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri <eedri(a)redhat.com> wrote:
Not sure if all the same issue, but seems to failing around the same
time:
  ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
min  integ-tests
  ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr -
#1045 51 min  integ-tests
  ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec 
integ-tests
  ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min 
integ-tests
  ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10
hr - #873 1 hr 36 min  integ-tests
  ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr -
#1149 1 hr 26 min  integ-tests
  ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr
- #729 1 hr 54 min  integ-tests
It's a NullPointerException on engine side:
I opened a bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which
is not on POST
  ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr
-
#929 45 min  integ-tests
  ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10
hr - #190 45 min  integ-tests
Links to jobs can be found here: https://jenkins.ovirt.org/
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
5:37 a.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
I am not sure that locking both groups would be sufficient, because there
is still a chance that the removeNetowrks request will start and acquire
the lock before the DisconnectStorage operation starts.
So probably the correct and full proof solution is to not move the host to
maintenance until all related storage ops terminate.
On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <stirabos(a)redhat.com>
wrote:
On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri <eedri(a)redhat.com> wrote:
> Not sure if all the same issue, but seems to failing around the same time:
>
>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
> min  integ-tests
>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr
> - #1045 51 min  integ-tests
>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec
>  integ-tests
>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min
>  integ-tests
>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10
> hr - #873 1 hr 36 min  integ-tests
>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr
> - #1149 1 hr 26 min  integ-tests
>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr
> - #729 1 hr 54 min  integ-tests
>
It's a NullPointerException on engine side:
I opened a bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which
is not on POST
>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr -
> #929 45 min  integ-tests
>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10
> hr - #190 45 min  integ-tests
>
> Links to jobs can be found here: https://jenkins.ovirt.org/
>
> --
>
> Eyal edri
>
>
> MANAGER
>
> RHV/CNV DevOps
>
> EMEA VIRTUALIZATION R&D
>
>
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
<https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
_______________________________________________
Devel mailing list -- devel(a)ovirt.org
To unsubscribe send an email to devel-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
11:48 a.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
I was told that intervening in the host state machine is delicate, but
I think that this is the only correct approach.
Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
a third week with this constant failure.
On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv <eraviv(a)redhat.com> wrote:
>
> I am not sure that locking both groups would be sufficient, because there is still a
chance that the removeNetowrks request will start and acquire the lock before the
DisconnectStorage operation starts.
> So probably the correct and full proof solution is to not move the host to
maintenance until all related storage ops terminate.
>
>
> On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <stirabos(a)redhat.com> wrote:
>>
>>
>>
>> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri <eedri(a)redhat.com> wrote:
>>>
>>> Not sure if all the same issue, but seems to failing around the same time:
>>>
>>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
min  integ-tests
>>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr -
#1045 51 min  integ-tests
>>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec 
integ-tests
>>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min 
integ-tests
>>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10 hr
- #873 1 hr 36 min  integ-tests
>>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr -
#1149 1 hr 26 min  integ-tests
>>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr -
#729 1 hr 54 min  integ-tests
>>
>>
>> It's a NullPointerException on engine side:
>> I opened a bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is
not on POST
>>
>>
>>>
>>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr - #929
45 min  integ-tests
>>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10 hr
- #190 45 min  integ-tests
>>>
>>> Links to jobs can be found here: https://jenkins.ovirt.org/
>>>
>>> --
>>>
>>> Eyal edri
>>>
>>>
>>> MANAGER
>>>
>>> RHV/CNV DevOps
>>>
>>> EMEA VIRTUALIZATION R&D
>>>
>>>
>>> Red Hat EMEA
>>>
>>> TRIED. TESTED. TRUSTED.
>>> phone: +972-9-7692018
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>> _______________________________________________
>> Devel mailing list -- devel(a)ovirt.org
>> To unsubscribe send an email to devel-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
>
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54...
12:27 p.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
The patch was reverted on Thursday
On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg <danken(a)redhat.com> wrote:
I was told that intervening in the host state machine is delicate,
but
I think that this is the only correct approach.
Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
a third week with this constant failure.
On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv <eraviv(a)redhat.com> wrote:
>
> I am not sure that locking both groups would be sufficient, because
there is still a chance that the removeNetowrks request will start and
acquire the lock before the DisconnectStorage operation starts.
> So probably the correct and full proof solution is to not move the host
to maintenance until all related storage ops terminate.
>
>
> On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <stirabos(a)redhat.com>
wrote:
>>
>>
>>
>> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri <eedri(a)redhat.com> wrote:
>>>
>>> Not sure if all the same issue, but seems to failing around the same
time:
>>>
>>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr -
#825 57 min  integ-tests
>>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12
hr - #1045 51 min  integ-tests
>>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24
sec  integ-tests
>>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10
min  integ-tests
>>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871
10 hr - #873 1 hr 36 min  integ-tests
>>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11
hr - #1149 1 hr 26 min  integ-tests
>>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12
hr - #729 1 hr 54 min  integ-tests
>>
>>
>> It's a NullPointerException on engine side:
>> I opened a bug here:
https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on POST
>>
>>
>>>
>>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr
- #929 45 min  integ-tests
>>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr -
#187 10 hr - #190 45 min  integ-tests
>>>
>>> Links to jobs can be found here: https://jenkins.ovirt.org/
>>>
>>> --
>>>
>>> Eyal edri
>>>
>>>
>>> MANAGER
>>>
>>> RHV/CNV DevOps
>>>
>>> EMEA VIRTUALIZATION R&D
>>>
>>>
>>> Red Hat EMEA
>>>
>>> TRIED. TESTED. TRUSTED.
>>> phone: +972-9-7692018
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>> _______________________________________________
>> Devel mailing list -- devel(a)ovirt.org
>> To unsubscribe send an email to devel-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
>
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54...
2:36 p.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
Unfortunately, the network suite is still failing on
Cannot edit Host. Related operation is currently in progress. Please try
again later.
Can you check if that's the same issue? Did you revert from 4.3 too?
http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
On Sat, 23 Mar 2019, 21:27 Benny Zlotnik, <bzlotnik(a)redhat.com> wrote:
The patch was reverted on Thursday
On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg <danken(a)redhat.com> wrote:
> I was told that intervening in the host state machine is delicate, but
> I think that this is the only correct approach.
>
> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
> a third week with this constant failure.
>
>
> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv <eraviv(a)redhat.com> wrote:
> >
> > I am not sure that locking both groups would be sufficient, because
> there is still a chance that the removeNetowrks request will start and
> acquire the lock before the DisconnectStorage operation starts.
> > So probably the correct and full proof solution is to not move the host
> to maintenance until all related storage ops terminate.
> >
> >
> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <stirabos(a)redhat.com>
> wrote:
> >>
> >>
> >>
> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri <eedri(a)redhat.com> wrote:
> >>>
> >>> Not sure if all the same issue, but seems to failing around the same
> time:
> >>>
> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr -
> #825 57 min  integ-tests
> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12
> hr - #1045 51 min  integ-tests
> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24
> sec  integ-tests
> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10
> min  integ-tests
> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr -
> #871 10 hr - #873 1 hr 36 min  integ-tests
> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11
> hr - #1149 1 hr 26 min  integ-tests
> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12
> hr - #729 1 hr 54 min  integ-tests
> >>
> >>
> >> It's a NullPointerException on engine side:
> >> I opened a bug here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on POST
> >>
> >>
> >>>
> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr
> - #929 45 min  integ-tests
> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr -
> #187 10 hr - #190 45 min  integ-tests
> >>>
> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
> >>>
> >>> --
> >>>
> >>> Eyal edri
> >>>
> >>>
> >>> MANAGER
> >>>
> >>> RHV/CNV DevOps
> >>>
> >>> EMEA VIRTUALIZATION R&D
> >>>
> >>>
> >>> Red Hat EMEA
> >>>
> >>> TRIED. TESTED. TRUSTED.
> >>> phone: +972-9-7692018
> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >>
> >> _______________________________________________
> >> Devel mailing list -- devel(a)ovirt.org
> >> To unsubscribe send an email to devel-leave(a)ovirt.org
> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> >> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> >> List Archives:
>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
> >
> > _______________________________________________
> > Devel mailing list -- devel(a)ovirt.org
> > To unsubscribe send an email to devel-leave(a)ovirt.org
> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> > oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54...
>
12:06 a.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
After some off line discussions it seems that the change that should be
implemented in order to solve the original problem (remove host fails due
to disconnect storage in progress) is to leave the host in 'preparing for
maintenance' until all relevant storage operations are completed.
On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg <danken(a)redhat.com> wrote:
Unfortunately, the network suite is still failing on
Cannot edit Host. Related operation is currently in progress. Please try
again later.
Can you check if that's the same issue? Did you revert from 4.3 too?
http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
On Sat, 23 Mar 2019, 21:27 Benny Zlotnik, <bzlotnik(a)redhat.com> wrote:
> The patch was reverted on Thursday
>
> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg <danken(a)redhat.com> wrote:
>
>> I was told that intervening in the host state machine is delicate, but
>> I think that this is the only correct approach.
>>
>> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
>> a third week with this constant failure.
>>
>>
>> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv <eraviv(a)redhat.com> wrote:
>> >
>> > I am not sure that locking both groups would be sufficient, because
>> there is still a chance that the removeNetowrks request will start and
>> acquire the lock before the DisconnectStorage operation starts.
>> > So probably the correct and full proof solution is to not move the
>> host to maintenance until all related storage ops terminate.
>> >
>> >
>> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi
<stirabos(a)redhat.com>
>> wrote:
>> >>
>> >>
>> >>
>> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri <eedri(a)redhat.com>
wrote:
>> >>>
>> >>> Not sure if all the same issue, but seems to failing around the
same
>> time:
>> >>>
>> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr
-
>> #825 57 min  integ-tests
>> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043
12
>> hr - #1045 51 min  integ-tests
>> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11
24
>> sec  integ-tests
>> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11
10
>> min  integ-tests
>> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr -
>> #871 10 hr - #873 1 hr 36 min  integ-tests
>> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147
11
>> hr - #1149 1 hr 26 min  integ-tests
>> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727
>> 12 hr - #729 1 hr 54 min  integ-tests
>> >>
>> >>
>> >> It's a NullPointerException on engine side:
>> >> I opened a bug here:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on POST
>> >>
>> >>
>> >>>
>> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12
>> hr - #929 45 min  integ-tests
>> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr -
>> #187 10 hr - #190 45 min  integ-tests
>> >>>
>> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
>> >>>
>> >>> --
>> >>>
>> >>> Eyal edri
>> >>>
>> >>>
>> >>> MANAGER
>> >>>
>> >>> RHV/CNV DevOps
>> >>>
>> >>> EMEA VIRTUALIZATION R&D
>> >>>
>> >>>
>> >>> Red Hat EMEA
>> >>>
>> >>> TRIED. TESTED. TRUSTED.
>> >>> phone: +972-9-7692018
>> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >>
>> >> _______________________________________________
>> >> Devel mailing list -- devel(a)ovirt.org
>> >> To unsubscribe send an email to devel-leave(a)ovirt.org
>> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> >> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> >> List Archives:
>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
>> >
>> > _______________________________________________
>> > Devel mailing list -- devel(a)ovirt.org
>> > To unsubscribe send an email to devel-leave(a)ovirt.org
>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> > oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> > List Archives:
>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54...
>>
>
1:13 a.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
patched 4.3!
On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv <eraviv(a)redhat.com> wrote:
After some off line discussions it seems that the change that should
be
implemented in order to solve the original problem (remove host fails due
to disconnect storage in progress) is to leave the host in 'preparing for
maintenance' until all relevant storage operations are completed.
On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg <danken(a)redhat.com> wrote:
> Unfortunately, the network suite is still failing on
>
> Cannot edit Host. Related operation is currently in progress. Please try
> again later.
>
> Can you check if that's the same issue? Did you revert from 4.3 too?
>
> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
>
>
>
> On Sat, 23 Mar 2019, 21:27 Benny Zlotnik, <bzlotnik(a)redhat.com> wrote:
>
>> The patch was reverted on Thursday
>>
>> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg <danken(a)redhat.com>
>> wrote:
>>
>>> I was told that intervening in the host state machine is delicate, but
>>> I think that this is the only correct approach.
>>>
>>> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
>>> a third week with this constant failure.
>>>
>>>
>>> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv <eraviv(a)redhat.com> wrote:
>>> >
>>> > I am not sure that locking both groups would be sufficient, because
>>> there is still a chance that the removeNetowrks request will start and
>>> acquire the lock before the DisconnectStorage operation starts.
>>> > So probably the correct and full proof solution is to not move the
>>> host to maintenance until all related storage ops terminate.
>>> >
>>> >
>>> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <
>>> stirabos(a)redhat.com> wrote:
>>> >>
>>> >>
>>> >>
>>> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri <eedri(a)redhat.com>
wrote:
>>> >>>
>>> >>> Not sure if all the same issue, but seems to failing around the
>>> same time:
>>> >>>
>>> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12
hr
>>> - #825 57 min  integ-tests
>>> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr -
#1043
>>> 12 hr - #1045 51 min  integ-tests
>>> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr -
#11
>>> 24 sec  integ-tests
>>> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr -
#11
>>> 10 min  integ-tests
>>> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr
-
>>> #871 10 hr - #873 1 hr 36 min  integ-tests
>>> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr -
#1147
>>> 11 hr - #1149 1 hr 26 min  integ-tests
>>> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr -
#727
>>> 12 hr - #729 1 hr 54 min  integ-tests
>>> >>
>>> >>
>>> >> It's a NullPointerException on engine side:
>>> >> I opened a bug here:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on
>>> POST
>>> >>
>>> >>
>>> >>>
>>> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926
12
>>> hr - #929 45 min  integ-tests
>>> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr
-
>>> #187 10 hr - #190 45 min  integ-tests
>>> >>>
>>> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
>>> >>>
>>> >>> --
>>> >>>
>>> >>> Eyal edri
>>> >>>
>>> >>>
>>> >>> MANAGER
>>> >>>
>>> >>> RHV/CNV DevOps
>>> >>>
>>> >>> EMEA VIRTUALIZATION R&D
>>> >>>
>>> >>>
>>> >>> Red Hat EMEA
>>> >>>
>>> >>> TRIED. TESTED. TRUSTED.
>>> >>> phone: +972-9-7692018
>>> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>> >>
>>> >> _______________________________________________
>>> >> Devel mailing list -- devel(a)ovirt.org
>>> >> To unsubscribe send an email to devel-leave(a)ovirt.org
>>> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> >> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> >> List Archives:
>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
>>> >
>>> > _______________________________________________
>>> > Devel mailing list -- devel(a)ovirt.org
>>> > To unsubscribe send an email to devel-leave(a)ovirt.org
>>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> > oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> > List Archives:
>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54...
>>>
>>
11:41 p.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
but http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
is still failing
Was your patch merged?
On Sun, Mar 24, 2019 at 10:14 AM Ahmad Khiet <akhiet(a)redhat.com> wrote:
patched 4.3!
On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv <eraviv(a)redhat.com> wrote:
> After some off line discussions it seems that the change that should be
> implemented in order to solve the original problem (remove host fails due
> to disconnect storage in progress) is to leave the host in 'preparing for
> maintenance' until all relevant storage operations are completed.
>
> On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg <danken(a)redhat.com>
> wrote:
>
>> Unfortunately, the network suite is still failing on
>>
>> Cannot edit Host. Related operation is currently in progress. Please try
>> again later.
>>
>> Can you check if that's the same issue? Did you revert from 4.3 too?
>>
>> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
>>
>>
>>
>> On Sat, 23 Mar 2019, 21:27 Benny Zlotnik, <bzlotnik(a)redhat.com> wrote:
>>
>>> The patch was reverted on Thursday
>>>
>>> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg <danken(a)redhat.com>
>>> wrote:
>>>
>>>> I was told that intervening in the host state machine is delicate, but
>>>> I think that this is the only correct approach.
>>>>
>>>> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
>>>> a third week with this constant failure.
>>>>
>>>>
>>>> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv <eraviv(a)redhat.com>
wrote:
>>>> >
>>>> > I am not sure that locking both groups would be sufficient, because
>>>> there is still a chance that the removeNetowrks request will start and
>>>> acquire the lock before the DisconnectStorage operation starts.
>>>> > So probably the correct and full proof solution is to not move the
>>>> host to maintenance until all related storage ops terminate.
>>>> >
>>>> >
>>>> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <
>>>> stirabos(a)redhat.com> wrote:
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri
<eedri(a)redhat.com> wrote:
>>>> >>>
>>>> >>> Not sure if all the same issue, but seems to failing around
the
>>>> same time:
>>>> >>>
>>>> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824
12 hr
>>>> - #825 57 min  integ-tests
>>>> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr -
#1043
>>>> 12 hr - #1045 51 min  integ-tests
>>>> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr
- #11
>>>> 24 sec  integ-tests
>>>> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr
- #11
>>>> 10 min  integ-tests
>>>> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10
hr -
>>>> #871 10 hr - #873 1 hr 36 min  integ-tests
>>>> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr -
#1147
>>>> 11 hr - #1149 1 hr 26 min  integ-tests
>>>> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr
- #727
>>>> 12 hr - #729 1 hr 54 min  integ-tests
>>>> >>
>>>> >>
>>>> >> It's a NullPointerException on engine side:
>>>> >> I opened a bug here:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on
>>>> POST
>>>> >>
>>>> >>
>>>> >>>
>>>> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr -
#926 12
>>>> hr - #929 45 min  integ-tests
>>>> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days
10 hr -
>>>> #187 10 hr - #190 45 min  integ-tests
>>>> >>>
>>>> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
>>>> >>>
>>>> >>> --
>>>> >>>
>>>> >>> Eyal edri
>>>> >>>
>>>> >>>
>>>> >>> MANAGER
>>>> >>>
>>>> >>> RHV/CNV DevOps
>>>> >>>
>>>> >>> EMEA VIRTUALIZATION R&D
>>>> >>>
>>>> >>>
>>>> >>> Red Hat EMEA
>>>> >>>
>>>> >>> TRIED. TESTED. TRUSTED.
>>>> >>> phone: +972-9-7692018
>>>> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>> >>
>>>> >> _______________________________________________
>>>> >> Devel mailing list -- devel(a)ovirt.org
>>>> >> To unsubscribe send an email to devel-leave(a)ovirt.org
>>>> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>> >> oVirt Code of Conduct:
>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>> >> List Archives:
>>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
>>>> >
>>>> > _______________________________________________
>>>> > Devel mailing list -- devel(a)ovirt.org
>>>> > To unsubscribe send an email to devel-leave(a)ovirt.org
>>>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>> > oVirt Code of Conduct:
>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>> > List Archives:
>>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54...
>>>>
>>>
11:58 p.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
the patch was merged on 4.3 but test run [1] has build [2] which is one
patch before Ahmed's merge...
[1] http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
[2] ovirt-engine-4.3.2.2-0.0.master.20190324105929.git8b0969c.el7.noarch.rpm
On Mon, Mar 25, 2019 at 8:42 AM Dan Kenigsberg <danken(a)redhat.com> wrote:
but
http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
is still failing
Was your patch merged?
On Sun, Mar 24, 2019 at 10:14 AM Ahmad Khiet <akhiet(a)redhat.com> wrote:
> patched 4.3!
>
> On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv <eraviv(a)redhat.com> wrote:
>
>> After some off line discussions it seems that the change that should be
>> implemented in order to solve the original problem (remove host fails due
>> to disconnect storage in progress) is to leave the host in 'preparing for
>> maintenance' until all relevant storage operations are completed.
>>
>> On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg <danken(a)redhat.com>
>> wrote:
>>
>>> Unfortunately, the network suite is still failing on
>>>
>>> Cannot edit Host. Related operation is currently in progress. Please
>>> try again later.
>>>
>>> Can you check if that's the same issue? Did you revert from 4.3 too?
>>>
>>> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
>>>
>>>
>>>
>>> On Sat, 23 Mar 2019, 21:27 Benny Zlotnik, <bzlotnik(a)redhat.com> wrote:
>>>
>>>> The patch was reverted on Thursday
>>>>
>>>> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg <danken(a)redhat.com>
>>>> wrote:
>>>>
>>>>> I was told that intervening in the host state machine is delicate,
but
>>>>> I think that this is the only correct approach.
>>>>>
>>>>> Benny, Ahmad, Tal: do you have a plan to resolve this? We are
entering
>>>>> a third week with this constant failure.
>>>>>
>>>>>
>>>>> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv
<eraviv(a)redhat.com>
>>>>> wrote:
>>>>> >
>>>>> > I am not sure that locking both groups would be sufficient,
because
>>>>> there is still a chance that the removeNetowrks request will start
and
>>>>> acquire the lock before the DisconnectStorage operation starts.
>>>>> > So probably the correct and full proof solution is to not move
the
>>>>> host to maintenance until all related storage ops terminate.
>>>>> >
>>>>> >
>>>>> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <
>>>>> stirabos(a)redhat.com> wrote:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri
<eedri(a)redhat.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Not sure if all the same issue, but seems to failing
around the
>>>>> same time:
>>>>> >>>
>>>>> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr -
#824 12
>>>>> hr - #825 57 min  integ-tests
>>>>> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12
hr - #1043
>>>>> 12 hr - #1045 51 min  integ-tests
>>>>> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11
hr - #11
>>>>> 24 sec  integ-tests
>>>>> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12
hr - #11
>>>>> 10 min  integ-tests
>>>>> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2
days 10 hr -
>>>>> #871 10 hr - #873 1 hr 36 min  integ-tests
>>>>> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11
hr - #1147
>>>>> 11 hr - #1149 1 hr 26 min  integ-tests
>>>>> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12
hr -
>>>>> #727 12 hr - #729 1 hr 54 min  integ-tests
>>>>> >>
>>>>> >>
>>>>> >> It's a NullPointerException on engine side:
>>>>> >> I opened a bug here:
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on
>>>>> POST
>>>>> >>
>>>>> >>
>>>>> >>>
>>>>> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr
- #926
>>>>> 12 hr - #929 45 min  integ-tests
>>>>> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3
days 10 hr
>>>>> - #187 10 hr - #190 45 min  integ-tests
>>>>> >>>
>>>>> >>> Links to jobs can be found here:
https://jenkins.ovirt.org/
>>>>> >>>
>>>>> >>> --
>>>>> >>>
>>>>> >>> Eyal edri
>>>>> >>>
>>>>> >>>
>>>>> >>> MANAGER
>>>>> >>>
>>>>> >>> RHV/CNV DevOps
>>>>> >>>
>>>>> >>> EMEA VIRTUALIZATION R&D
>>>>> >>>
>>>>> >>>
>>>>> >>> Red Hat EMEA
>>>>> >>>
>>>>> >>> TRIED. TESTED. TRUSTED.
>>>>> >>> phone: +972-9-7692018
>>>>> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> Devel mailing list -- devel(a)ovirt.org
>>>>> >> To unsubscribe send an email to devel-leave(a)ovirt.org
>>>>> >> Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
>>>>> >> oVirt Code of Conduct:
>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>> >> List Archives:
>>>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
>>>>> >
>>>>> > _______________________________________________
>>>>> > Devel mailing list -- devel(a)ovirt.org
>>>>> > To unsubscribe send an email to devel-leave(a)ovirt.org
>>>>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>>> > oVirt Code of Conduct:
>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>> > List Archives:
>>>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54...
>>>>>
>>>>
12:38 p.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
This issue seems to be solved now, please let me know if not.
The build
https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4408/
with ovirt-engine d0a215d862eb819f0bbdd51fed012f9b972c1bdf
which includes Ahmed's commit a236c90d54652503d43d8315582effb74050d22e
succeeded.
Also the build
https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/4407/
with default repos succeeded.
On Mon, 25 Mar 2019 08:58:25 +0200
Eitan Raviv <eraviv(a)redhat.com> wrote:
the patch was merged on 4.3 but test run [1] has build [2] which is
one
patch before Ahmed's merge...
[1] http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
[2] ovirt-engine-4.3.2.2-0.0.master.20190324105929.git8b0969c.el7.noarch.rpm
On Mon, Mar 25, 2019 at 8:42 AM Dan Kenigsberg <danken(a)redhat.com> wrote:
> but http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
> is still failing
> Was your patch merged?
>
> On Sun, Mar 24, 2019 at 10:14 AM Ahmad Khiet <akhiet(a)redhat.com> wrote:
>
>> patched 4.3!
>>
>> On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv <eraviv(a)redhat.com> wrote:
>>
>>> After some off line discussions it seems that the change that should be
>>> implemented in order to solve the original problem (remove host fails due
>>> to disconnect storage in progress) is to leave the host in 'preparing
for
>>> maintenance' until all relevant storage operations are completed.
>>>
>>> On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg <danken(a)redhat.com>
>>> wrote:
>>>
>>>> Unfortunately, the network suite is still failing on
>>>>
>>>> Cannot edit Host. Related operation is currently in progress. Please
>>>> try again later.
>>>>
>>>> Can you check if that's the same issue? Did you revert from 4.3
too?
>>>>
>>>> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
>>>>
>>>>
>>>>
>>>> On Sat, 23 Mar 2019, 21:27 Benny Zlotnik, <bzlotnik(a)redhat.com>
wrote:
>>>>
>>>>> The patch was reverted on Thursday
>>>>>
>>>>> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg
<danken(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>> I was told that intervening in the host state machine is
delicate, but
>>>>>> I think that this is the only correct approach.
>>>>>>
>>>>>> Benny, Ahmad, Tal: do you have a plan to resolve this? We are
entering
>>>>>> a third week with this constant failure.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv
<eraviv(a)redhat.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > I am not sure that locking both groups would be sufficient,
because
>>>>>> there is still a chance that the removeNetowrks request will
start and
>>>>>> acquire the lock before the DisconnectStorage operation starts.
>>>>>> > So probably the correct and full proof solution is to not
move the
>>>>>> host to maintenance until all related storage ops terminate.
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <
>>>>>> stirabos(a)redhat.com> wrote:
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri
<eedri(a)redhat.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Not sure if all the same issue, but seems to
failing around the
>>>>>> same time:
>>>>>> >>>
>>>>>> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12
hr - #824 12
>>>>>> hr - #825 57 min  integ-tests
>>>>>> >>>   ovirt-system-tests_hc-basic-suite-master 2 days
12 hr - #1043
>>>>>> 12 hr - #1045 51 min  integ-tests
>>>>>> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3
N/A 11 hr - #11
>>>>>> 24 sec  integ-tests
>>>>>> >>>   ovirt-system-tests_he-basic-ipv6-suite-master
N/A 12 hr - #11
>>>>>> 10 min  integ-tests
>>>>>> >>>   ovirt-system-tests_he-basic-iscsi-suite-master
2 days 10 hr -
>>>>>> #871 10 hr - #873 1 hr 36 min  integ-tests
>>>>>> >>>   ovirt-system-tests_he-basic-suite-master 2 days
11 hr - #1147
>>>>>> 11 hr - #1149 1 hr 26 min  integ-tests
>>>>>> >>>   ovirt-system-tests_he-node-ng-suite-master 2
days 12 hr -
>>>>>> #727 12 hr - #729 1 hr 54 min  integ-tests
>>>>>> >>
>>>>>> >>
>>>>>> >> It's a NullPointerException on engine side:
>>>>>> >> I opened a bug here:
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not
on
>>>>>> POST
>>>>>> >>
>>>>>> >>
>>>>>> >>>
>>>>>> >>>   ovirt-system-tests_network-suite-master 3 days
12 hr - #926
>>>>>> 12 hr - #929 45 min  integ-tests
>>>>>> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2
3 days 10 hr
>>>>>> - #187 10 hr - #190 45 min  integ-tests
>>>>>> >>>
>>>>>> >>> Links to jobs can be found here:
https://jenkins.ovirt.org/
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>>
>>>>>> >>> Eyal edri
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> MANAGER
>>>>>> >>>
>>>>>> >>> RHV/CNV DevOps
>>>>>> >>>
>>>>>> >>> EMEA VIRTUALIZATION R&D
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> Red Hat EMEA
>>>>>> >>>
>>>>>> >>> TRIED. TESTED. TRUSTED.
>>>>>> >>> phone: +972-9-7692018
>>>>>> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>> >>
>>>>>> >> _______________________________________________
>>>>>> >> Devel mailing list -- devel(a)ovirt.org
>>>>>> >> To unsubscribe send an email to devel-leave(a)ovirt.org
>>>>>> >> Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
>>>>>> >> oVirt Code of Conduct:
>>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>>> >> List Archives:
>>>>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Devel mailing list -- devel(a)ovirt.org
>>>>>> > To unsubscribe send an email to devel-leave(a)ovirt.org
>>>>>> > Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
>>>>>> > oVirt Code of Conduct:
>>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>>> > List Archives:
>>>>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54...
>>>>>>
>>>>>
8:18 p.m.
New subject: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago
ovirt-engine change queue and ovit-system-tests check_patch also look good
with ovirt-engine rpm including Ahmed's revert.
Thanks
On Mon, Mar 25, 2019 at 9:38 PM Dominik Holler <dholler(a)redhat.com> wrote:
This issue seems to be solved now, please let me know if not.
The build
https://jenkins.ovirt.org/view/oVirt system
tests/job/ovirt-system-tests_manual/4408/
with ovirt-engine d0a215d862eb819f0bbdd51fed012f9b972c1bdf
which includes Ahmed's commit a236c90d54652503d43d8315582effb74050d22e
succeeded.
Also the build
https://jenkins.ovirt.org/view/oVirt system
tests/job/ovirt-system-tests_manual/4407/
with default repos succeeded.
On Mon, 25 Mar 2019 08:58:25 +0200
Eitan Raviv <eraviv(a)redhat.com> wrote:
> the patch was merged on 4.3 but test run [1] has build [2] which is one
> patch before Ahmed's merge...
>
>
> [1]
http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
> [2]
ovirt-engine-4.3.2.2-0.0.master.20190324105929.git8b0969c.el7.noarch.rpm
>
>
> On Mon, Mar 25, 2019 at 8:42 AM Dan Kenigsberg <danken(a)redhat.com>
wrote:
>
> > but
http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
> > is still failing
> > Was your patch merged?
> >
> > On Sun, Mar 24, 2019 at 10:14 AM Ahmad Khiet <akhiet(a)redhat.com>
wrote:
> >
> >> patched 4.3!
> >>
> >> On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv <eraviv(a)redhat.com>
wrote:
> >>
> >>> After some off line discussions it seems that the change that should
be
> >>> implemented in order to solve the original problem (remove host
fails due
> >>> to disconnect storage in progress) is to leave the host in
'preparing for
> >>> maintenance' until all relevant storage operations are completed.
> >>>
> >>> On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg
<danken(a)redhat.com>
> >>> wrote:
> >>>
> >>>> Unfortunately, the network suite is still failing on
> >>>>
> >>>> Cannot edit Host. Related operation is currently in progress.
Please
> >>>> try again later.
> >>>>
> >>>> Can you check if that's the same issue? Did you revert from 4.3
too?
> >>>>
> >>>>
http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
> >>>>
> >>>>
> >>>>
> >>>> On Sat, 23 Mar 2019, 21:27 Benny Zlotnik,
<bzlotnik(a)redhat.com>
wrote:
> >>>>
> >>>>> The patch was reverted on Thursday
> >>>>>
> >>>>> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg
<danken(a)redhat.com>
> >>>>> wrote:
> >>>>>
> >>>>>> I was told that intervening in the host state machine is
delicate, but
> >>>>>> I think that this is the only correct approach.
> >>>>>>
> >>>>>> Benny, Ahmad, Tal: do you have a plan to resolve this? We
are
entering
> >>>>>> a third week with this constant failure.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv
<eraviv(a)redhat.com>
> >>>>>> wrote:
> >>>>>> >
> >>>>>> > I am not sure that locking both groups would be
sufficient,
because
> >>>>>> there is still a chance that the removeNetowrks request
will
start and
> >>>>>> acquire the lock before the DisconnectStorage operation
starts.
> >>>>>> > So probably the correct and full proof solution is to
not move
the
> >>>>>> host to maintenance until all related storage ops
terminate.
> >>>>>> >
> >>>>>> >
> >>>>>> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi
<
> >>>>>> stirabos(a)redhat.com> wrote:
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri
<eedri(a)redhat.com>
> >>>>>> wrote:
> >>>>>> >>>
> >>>>>> >>> Not sure if all the same issue, but seems to
failing around
the
> >>>>>> same time:
> >>>>>> >>>
> >>>>>> >>>   ovirt-system-tests_hc-basic-suite-4.2 1
day 12 hr - #824
12
> >>>>>> hr - #825 57 min  integ-tests
> >>>>>> >>>   ovirt-system-tests_hc-basic-suite-master 2
days 12 hr -
#1043
> >>>>>> 12 hr - #1045 51 min  integ-tests
> >>>>>> >>>  
ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr -
#11
> >>>>>> 24 sec  integ-tests
> >>>>>> >>>  
ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr -
#11
> >>>>>> 10 min  integ-tests
> >>>>>> >>>  
ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10
hr -
> >>>>>> #871 10 hr - #873 1 hr 36 min  integ-tests
> >>>>>> >>>   ovirt-system-tests_he-basic-suite-master 2
days 11 hr -
#1147
> >>>>>> 11 hr - #1149 1 hr 26 min  integ-tests
> >>>>>> >>>   ovirt-system-tests_he-node-ng-suite-master
2 days 12 hr -
> >>>>>> #727 12 hr - #729 1 hr 54 min  integ-tests
> >>>>>> >>
> >>>>>> >>
> >>>>>> >> It's a NullPointerException on engine side:
> >>>>>> >> I opened a bug here:
> >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which
is not
on
> >>>>>> POST
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>>
> >>>>>> >>>   ovirt-system-tests_network-suite-master 3
days 12 hr -
#926
> >>>>>> 12 hr - #929 45 min  integ-tests
> >>>>>> >>>  
ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10
hr
> >>>>>> - #187 10 hr - #190 45 min  integ-tests
> >>>>>> >>>
> >>>>>> >>> Links to jobs can be found here:
https://jenkins.ovirt.org/
> >>>>>> >>>
> >>>>>> >>> --
> >>>>>> >>>
> >>>>>> >>> Eyal edri
> >>>>>> >>>
> >>>>>> >>>
> >>>>>> >>> MANAGER
> >>>>>> >>>
> >>>>>> >>> RHV/CNV DevOps
> >>>>>> >>>
> >>>>>> >>> EMEA VIRTUALIZATION R&D
> >>>>>> >>>
> >>>>>> >>>
> >>>>>> >>> Red Hat EMEA
> >>>>>> >>>
> >>>>>> >>> TRIED. TESTED. TRUSTED.
> >>>>>> >>> phone: +972-9-7692018
> >>>>>> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >>>>>> >>
> >>>>>> >> _______________________________________________
> >>>>>> >> Devel mailing list -- devel(a)ovirt.org
> >>>>>> >> To unsubscribe send an email to
devel-leave(a)ovirt.org
> >>>>>> >> Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
> >>>>>> >> oVirt Code of Conduct:
> >>>>>>
https://www.ovirt.org/community/about/community-guidelines/
> >>>>>> >> List Archives:
> >>>>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53...
> >>>>>> >
> >>>>>> > _______________________________________________
> >>>>>> > Devel mailing list -- devel(a)ovirt.org
> >>>>>> > To unsubscribe send an email to devel-leave(a)ovirt.org
> >>>>>> > Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
> >>>>>> > oVirt Code of Conduct:
> >>>>>>
https://www.ovirt.org/community/about/community-guidelines/
> >>>>>> > List Archives:
> >>>>>>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54...
> >>>>>>
> >>>>>
1858
days inactive
1867
days old
13 comments
7 participants
participants (7)
-
Ahmad Khiet -
Benny Zlotnik -
Dan Kenigsberg -
Dominik Holler -
Eitan Raviv -
Eyal Edri -
Simone Tiraboschi