- Devel - oVirt List Archives

[ OST Failure Report ] [ oVirt Master (ovirt-host) ] [ 22-02-2018 ] [002_bootstrap.add_hosts ]
by Dafna Ron 22 Feb '18

22 Feb '18

hi, we have a failed test for ovirt-host in upgrade suite. *Link and headline of suspected patches: * *Require collectd-virt plugin - https://gerrit.ovirt.org/#/c/87311/ <https://gerrit.ovirt.org/#/c/87311/>Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5827/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5827/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5827/art… <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5827/artifacts>(Relevant) error snippet from the log: <error>* 2018-02-22 05:38:47,587-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [546df98d] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host0. Setting kernel arguments. 2018-02-22 05:38:47,858-05 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] Swallowing exception as preferring stderr 2018-02-22 05:38:47,859-05 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] SSH error running command root@lago-upgrade-from-release-suite-master-host0:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True': RuntimeException: Unexpected error during execution: bash: line 1: 1395 Segmentation fault "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True 2018-02-22 05:38:47,859-05 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [546df98d] Error during deploy dialog 2018-02-22 05:38:47,860-05 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] Error during host lago-upgrade-from-release-suite-master-host0 install 2018-02-22 05:38:47,864-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host0: Unexpected error during execution: bash: line 1: 1395 Segmentation fault "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True . 2018-02-22 05:38:47,864-05 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] Error during host lago-upgrade-from-release-suite-master-host0 install, preferring first exception: Unexpected connection termination 2018-02-22 05:38:47,864-05 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] Host installation failed for host '65c048e5-a97a-422d-88b8-abe1fd925602', 'lago-upgrade-from-release-suite-master-host0': Unexpected connection termination 2018-02-22 05:38:47,869-05 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host0, SetVdsStatusVDSCommandParameters:{hostId='65c048e5-a97a-422d-88b8-abe1fd925602', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1973b09c 2018-02-22 05:38:47,898-05 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] FINISH, SetVdsStatusVDSCommand, log id: 1973b09c 2018-02-22 05:38:47,911-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host0 installation failed. Unexpected connection termination. 2018-02-22 05:38:47,920-05 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [546df98d] Lock freed to object 'EngineLock:{exclusiveLocks='[65c048e5-a97a-422d-88b8-abe1fd925602=VDS]', sharedLocks=''}' *</error>*

1 0

ovirt-engine-api-model jenkins jobs
by Sandro Bonazzola 22 Feb '18

22 Feb '18

Hi, I'm investigating on a failure[2 <http://jenkins.ovirt.org/job/ovirt-engine-api-model_master_build-artifacts-…>] on jenkins jobs for ovirt-engine-api-model[1 <http://jenkins.ovirt.org/search/?q=ovirt-engine-api-model_master_build-arti…>] on el7. Before I further invest time on it, I'd like to understand the purpose of the job. As far as I can tell the project is building some documentation in different formats and a tarball with the sources: asciidoctor.css 28.92 KB model.adoc 1.24 MB model.csv 264.77 KB model.html 3.37 MB model.json 7.39 MB model.xml 6.09 MB ovirt-engine-api-model-4.3.8.20180213gita1f6d9a.tar.gz 246.48 KB I think that some way the output may be used by http://ovirt.github.io/ovirt-engine-api-model/ but I don't feel like this is really done. I also see that artifacts are published on https://search.maven.org/#artifactdetails%7Corg.ovirt.engine.api%7Cmodel%7C… <https://search.maven.org/#artifactdetails%7Corg.ovirt.engine.api%7Cmodel%7C…> but this is also not done by the jenkins jobs. I see this artifacts is required in ovirt-engine pom.xml: <model.version>4.3.7</model.version> So I would expect the job to prepare a rpm including this jar and pom to be required in ovirt-engine spec file, at least if we follow Fedora packaging guidelines. As is, this jobs seams meaningless. What should we do here? Drop the job? Fix it for building rpms? [1] http://jenkins.ovirt.org/search/?q=ovirt-engine-api-model_master_build-arti… [2] http://jenkins.ovirt.org/job/ovirt-engine-api-model_master_build-artifacts-… -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

2 2

CentOS Virt SIG - oVirt 4.3 repositories
by Sandro Bonazzola 22 Feb '18

22 Feb '18

Hi, tags have been created for oVirt 4.3 development. I'm going to cross tag dependencies from oVirt 4.2 to oVirt 4.3 in the next weeks. Once done, I'll push patches to oVirt System Test master and oVirt Release master switching from oVirt 4.2 to oVirt 4.3. Repositories that will contain 4.3 related stuff while I'm working on it: https://cbs.centos.org/repos/virt7-ovirt-43-candidate https://cbs.centos.org/repos/virt7-ovirt-common-candidate Once I'll be sure master works with these, I'll tag for testing switching to: https://cbs.centos.org/repos/virt7-ovirt-43-testing https://cbs.centos.org/repos/virt7-ovirt-common-testing Once repoclosure passes, I'll ask to start signing and publishing rpms for testing getting the rpms into final testing destination: https://buildlogs.centos.org/centos/7/virt/$basearch/ovirt-4.3/ This will be the repo that will be added to OST and ovirt-release. If new dependencies needs to be added to 4.3 repo during this process, please let me know. Thanks -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

1 0

[ OST Failure Report ] [ oVirt $VER (ovirt-log-collector) ] [ 21-02-2018 ] [ 098_ovirt_provider_ovn.use_ovn_provider ]
by Dafna Ron 21 Feb '18

21 Feb '18

Hi, We failed test on duplicate network name. I don't think that the change is related. I can see that we are importing network_1 and then trying to create a network by the same name. I don't think there was a change made to ovn before this failure that could have cause this to fail but adding Dan in case he is aware of any changes. *Link and headline of suspected patches: failed change: * inventory: handle to postgresql dumps from sosreport - https://gerrit.ovirt.org/#/c/88010/ *Change reported as root cause of failure: *collector: fix list function - https://gerrit.ovirt.org/#/c/87798/ *Link to Job:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5805 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5805>Link to all logs:* http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5805/artifacts *(Relevant) error snippet from the log: <error>* 2018-02-20 23:51:03,179-05 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] Compiled stored procedure. Call string is [{call getdcidbyexternalnetworkid(?)}] 2018-02-20 23:51:03,179-05 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] SqlCall for procedure [GetDcIdByExternalNetworkId] compiled 2018-02-20 23:51:03,180-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] method: runQuery, params: [GetAllExternalNetworksOnProvider, IdQueryParameters:{refresh='false', filtered='false'}], timeElapsed: 266ms 2018-02-20 23:51:03,187-05 INFO [org.ovirt.engine.core.bll.network.dc.AddNetworkCommand] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] Lock Acquired to object 'EngineLock:{exclusiveLocks='[network_1=NETWORK, 94fc81b9-fbf2-4262-a877-32a066a573c2=PROVIDER]', sharedLocks=''}' 2018-02-20 23:51:03,190-05 WARN [org.ovirt.engine.core.bll.network.dc.AddNetworkCommand] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] Validation of action 'AddNetwork' failed for user admin@internal-authz. Reasons: VAR__TYPE__NETWORK,VAR__ACTION__ADD,ACTION_TYPE_FAILED_NETWORK_NAME_IN_USE,$NetworkName network_1 2018-02-20 23:51:03,190-05 INFO [org.ovirt.engine.core.bll.network.dc.AddNetworkCommand] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] Lock freed to object 'EngineLock:{exclusiveLocks='[network_1=NETWORK, 94fc81b9-fbf2-4262-a877-32a066a573c2=PROVIDER]', sharedLocks=''}' 2018-02-20 23:51:03,191-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-19) [4a0a588a-e606-4151-936a-f66a8c580a78] method: runAction, params: [AddNetwork, AddNetworkStoragePoolParameters:{commandId='41c44658-04e5-4aeb-9f65-cc2ad76e0c42', user='null', commandType='Unknown'}], timeElapsed: 10ms 2018-02-20 23:51:03,196-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-19) [] Operation Failed: [Cannot add Network. The name of the logical network 'network_1' is already used by an existing logical network in the same data-center. -Please choose a different name.] 2018-02-20 23:51:03,346-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] method: get, params: [da0bebdb-cb83-415f-bcba-8920126e31a4], timeElapsed: 6ms 2018-02-20 23:51:03,346-05 DEBUG [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] vdsManager::refreshVdsStats entered, host='lago-basic-suite-master-host-1'(da0bebdb-cb83-415f-bcba-8920126e31a4) 2018-02-20 23:51:03,365-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] START, GetStatsAsyncVDSCommand(HostName = lago-basic-suite-master-host-1, VdsIdAndVdsVDSCommandParametersBase:{hostId='da0bebdb-cb83-415f-bcba-8920126e31a4', vds='Host[lago-basic-suite-master-host-1,da0bebdb-cb83-415f-bcba-8920126e31a4]'}), log id: 279de334 2018-02-20 23:51:03,365-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] SEND destination:jms.topic.vdsm_requests reply-to:jms.topic.vdsm_responses content-length:98 *</error>*

1 0

Documenting HA VMs on Wiki
by Milan Zamazal 21 Feb '18

21 Feb '18

Hi, to document and clarify some facts about HA VMs to users, especially regarding I/O errors, I work on a HOWTO document to be added to oVirt Wiki. If you know how HA VMs or related mechanisms (such as VM handling on I/O errors) work and would like to help review the document, please look at the pull request: https://github.com/oVirt/ovirt-site/pull/1530 Thanks, Milan

1 0

[vdsm][network] false test failiures in CI
by Francesco Romani 20 Feb '18

20 Feb '18

Hi there, recently some patches of mine failed on CI with this error which is totally unrelated and which seems bogus. Could some network developer please have a look and improve the current state? :) https://gerrit.ovirt.org/#/c/87881/ 19:43:33 ====================================================================== 19:43:33 ERROR: test_list_ipv4_ipv6 (network.ip_address_test.IPAddressTest) 19:43:33 ---------------------------------------------------------------------- 19:43:33 Traceback (most recent call last): 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/tests/testValidation.py", line 330, in wrapper 19:43:33 return f(*args, **kwargs) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/tests/network/ip_address_test.py", line 289, in test_list_ipv4_ipv6 19:43:33 ipv6_addresses=[IPV6_B_WITH_PREFIXLEN] 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/tests/network/ip_address_test.py", line 297, in _test_list 19:43:33 address.IPAddressData(addr, device=nic)) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ip/address/iproute2.py", line 40, in add 19:43:33 addr_data.address, addr_data.prefixlen, addr_data.family 19:43:33 File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ 19:43:33 self.gen.throw(type, value, traceback) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ip/address/iproute2.py", line 91, in _translate_iproute2_exception 19:43:33 new_exception, new_exception(str(address_data), error_message), tb) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ip/address/iproute2.py", line 86, in _translate_iproute2_exception 19:43:33 yield 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ip/address/iproute2.py", line 40, in add 19:43:33 addr_data.address, addr_data.prefixlen, addr_data.family 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 561, in addrAdd 19:43:33 _exec_cmd(command) 19:43:33 File "/home/jenkins/workspace/vdsm_4.2_check-patch-fc27-x86_64/vdsm/lib/vdsm/network/ipwrapper.py", line 482, in _exec_cmd 19:43:33 raise exc(returnCode, error.splitlines()) 19:43:33 IPAddressAddError: ("IPAddressData(device='dummy_ZVN3L' address=IPv6Interface(u'2002:99::1/64') scope=None flags=None)", 'RTNETLINK answers: Permission denied') I had other failures, but they look like known issues (mkimage, no port on protocol detector) -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh

2 1

[ OST Failure Report ] [ oVirt 4.2 (ovirt-engine-metrics) ] [ 20-02-2018 ] [ 003_00_metrics_bootstrap.metrics_and_log_collector ]
by Dafna Ron 20 Feb '18

20 Feb '18

Hi, We have a failed test for project ovirt-engine-metrics. It seems that CQ is reporting that the root cause of the failure was a patch from 6 days ago. Shirly, can you please check the changes? *Link and headline of suspected patches: change reported as failed: * replace vdsm stats with collectd virt plugin - https://gerrit.ovirt.org/#/c/87310/ *change reported as root cause: * ansible: add role to copy required files - https://gerrit.ovirt.org/#/c/87430/ TASK [oVirt.ovirt-initial-validations/validate-config-yml : Validate viaq_metrics_store parameter] *** fatal: [localhost]: FAILED! => {"msg": "The conditional check 'viaq_metrics_store != true' failed. The error was: error while evaluating conditional (viaq_metrics_store != true): 'viaq_metrics_store' is undefined\n\nThe error appears to have been in '/usr/share/ansible/roles/oVirt.metrics/roles/oVirt.ovirt-initial-validations/validate-config-yml/tasks/main.yml': line 49, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Validate viaq_metrics_store parameter\"\n ^ here\n"} NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* localhost : ok=4 changed=0 unreachable=0 failed=1 lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/003_00_metrics_bootstrap.py", line 54, in configure_metrics ' Exit code is %s' % result.code File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, in eq_ raise AssertionError(msg or "%r != %r" % (a, b)) AssertionError: Configuring ovirt machines for metrics failed. Exit code is 2 lago.ssh: DEBUG: Command 59b597f6 on lago-basic-suite-4-2-engine returned with 0 lago.ssh: DEBUG: Command 59b597f6 on lago-basic-suite-4-2-engine output: This command will collect system configuration and diagnostic information from this system. The generated archive may contain data considered sensitive and its content should be reviewed by the originating organization before being passed to any third party. No changes will be made to system configuration. INFO: /dev/shm/log does not exist. It will be created. DEBUG: API Vendor(None) API Version(4.2.0) WARNING: This ovirt-log-collector call will collect logs from all available hosts. This may take long time, depending on the size of your deployment INFO: Gathering oVirt Engine information... DEBUG: loading config '/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf' DEBUG: calling(['sosreport', '--list-plugins']) DEBUG: returncode(0) DEBUG: STDOUT( sosreport (version 3.4) *Link to Job:http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/864/ <http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/864/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/864/artifac… <http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/864/artifacts>(Relevant) error snippet from the log: <error></error>*

3 4

CI outage notice
by Evgheni Dereveanchin 20 Feb '18

20 Feb '18

Hi everyone, After the Jenkins update tonight it started behaving unstably and there were several restarts recorded throughout the night. The root cause is now found and mitigated [1], Jenkins is up and stable again. There may still be slaves around that were running jobs when the master restarted which caused leftover files in the workspace. If you had any CI jobs fail with errors unrelated to code between midnight and 8:15 GMT today - please re-trigger those. If you still get failures - please inform me or other members of the infra team to offline such slaves and perform a cleanup. -- Regards, Evgheni Dereveanchin [1] https://ovirt-jira.atlassian.net/browse/OVIRT-1904

1 0

planned Jenkins restart
by Evgheni Dereveanchin 20 Feb '18

20 Feb '18

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is over. Regards, Evgheni Dereveanchin

1 1

OST Failure - Weekly update [10/02/2018-16/02/2018]
by Dafna Ron 18 Feb '18

18 Feb '18

Hello, I would like to update on this week's failure and OST current status. At the end of the week I am happy to say that we are no longer failing on any issues and the testers are completely green. We had a great week of many people rallying together to fix OST and resolve all outstanding issues. During the week we saw and resolved the following failures: *Master: * 002_bootstrap.get_host_devices - this was related to timing issues with the test and was fixed by: https://gerrit.ovirt.org/87526 001_initialize_engine.initialize_engine - dwh service failed to start - this seemed liked a packaging issue and was resolved. *4.2:* 002_bootstrap.add_cluster - cluster.cpu.type error - this was resolved a week ago and was resolved by patch: https://gerrit.ovirt.org/#/c/87126/ 003_00_metrics_bootstrap.metrics_and_log_collector - was resolved by Shirly. 002_bootstrap.get_host_devices - this was related to timing issues with the test and was fixed by: https://gerrit.ovirt.org/87526 *4.1* 001_initialize_engine.initialize_engine - dwh service failed to start - this seemed liked a packaging issue and was resolved. Build-artifacts jobs failing has a high percentage but* many of the build-artifacts failures were caused because of fcraw issues *and I am hoping this would be resolved over time. *below you can see the chart of OST resolved issues based on cause and failures: * * Based on feedback, I made some changes to the definitions Code= regression of working components/functionalities Configurations - package related issues Other = failed build artifacts Infra = infrastructure/OST/Lago related issues [image: Inline image 2] [image: Inline image 1] *Below is a chart of resolved failures based on Ovirt Version: * [image: Inline image 4] [image: Inline image 3] *Below is a chart showing failures by suite type: * * Suite type None means that it was a failure that did not result in a failed test such artifacts related or packaging related. [image: Inline image 8] [image: Inline image 7] Thanks, Dafna

3 2