- Users - oVirt List Archives

oVirt Weekly Sync Meeting: Jan. 7, 2015
by Brian Proffitt 07 Jan '15

07 Jan '15

========================= #ovirt: oVirt Weekly Sync ========================= Meeting started by bkp at 15:03:23 UTC. Minutes: http://ovirt.org/meetings/ovirt/2015/ovirt.2015-01-07-15.03.html Minutes (text): http://ovirt.org/meetings/ovirt/2015/ovirt.2015-01-07-15.03.txt Log: http://ovirt.org/meetings/ovirt/2015/ovirt.2015-01-07-15.03.log.html Meeting summary --------------- * Agenda and Roll Call (bkp, 15:03:35) * infra update (bkp, 15:03:35) * 3.5.z updates (bkp, 15:03:35) * 3.6.0 status (bkp, 15:03:35) * conferences and workshops (bkp, 15:03:35) * other topics (bkp, 15:03:38) * infra update (bkp, 15:05:39) * infra update The ILO issue was fixed on the PHX lab, and we're waiting now for dcaro to return to start phase II (bkp, 15:13:45) * infra update Phase II includes assining more IPs to the lab, migrating linode services, etc. Expected start: newxt week. (bkp, 15:13:48) * infra update A new unstable view was added to show productions jobs to better focus on fixing critical jobs first, and leave jobs that are work-in-progress for another view (bkp, 15:13:51) * 3.5.z updates (bkp, 15:14:29) * 3.5.z updates 3.5.1 RC is postponed due to two known blockers: 1160846 (which SLA is still working on) and 1177290 (ovirt-engine not installable on CentOS 6: broken dependency novnc) (bkp, 15:34:35) * 3.5.z updates Status of 1177290 fix: Not sure of status, but may be able to have a workaround by the end of this week, according to sbonazzo (bkp, 15:34:39) * 3.5.z updates According to didi, there is already a "workaround" mentioned in the 1177290 - we took downstream rhel sources and built on upstream jenkins, eedri signed them and they are available in the 3.5 repo. So not sure it's a blocker for 3.5.1, although we do need a longer-term solution (bkp, 15:34:42) * 3.5.z updates 1177290 dropped as a blocker, retargeted to 3.6 for proper handling (bkp, 15:34:47) * 3.5.z updates Status of 1160846 fix: Hopefully Monday, according to gchaplik (bkp, 15:34:50) * 3.5.z updates danken reported another potential blocker with vdsm. No bug assigned/tracked yet. sbonazzo asked to assist. (bkp, 15:34:53) * 3.5.z updates There are still 57 bugs targeted to 3.5.1. Excluding node and documentation bugs we still have 37 bugs targeted to 3.5.1 (bkp, 15:34:56) * 3.5.z updates 3.5.1 RC tentatively scheduled for Jan. 14, 2015. (bkp, 15:34:59) * 3.6 status (bkp, 15:35:07) * ACTION: 3.6.0 status Feature proposed for 3.6.0 must now be collected in the 3.6 Google doc (http://goo.gl/9X3G49) and reviewed by maintainers (bkp, 15:47:28) * 3.6.0 status Finished the review process, the remaining key milestones for this release will be scheduled. (bkp, 15:47:31) * 3.6.0 status Will now need to think about what to do with novnc in this release. (bkp, 15:47:34) * 3.6.0 status Jenkins build for 3.6 currently failing, but once back to nomral, 3.6 should be able to run on F21 (thanks emesika) (bkp, 15:47:37) * 3.6.0 status There are 466 bugs targeted to 3.6.0. Excluding node and documentation bugs we have 440 bugs targeted to 3.6.0. (bkp, 15:47:40) * 3.6.0 status Host Network QoS is all merged, at long last. (bkp, 15:47:43) * 3.6.0 status emesika's patch to introduce 3.6 clusters into oVirt was merged today, now this feature (and any other that's already merged) will be available for users to use. (bkp, 15:47:45) * 3.6.0 status Networking is also making decent progress on a bunch of other features, updating on the spreadsheet. More should be merged in the coming weeks. (bkp, 15:47:48) * 3.6.0 status UX has completed a feature for 3.6 that a lot of people have apparently wanted returned since 3.0. When switching main tabs the search now remembers the query you entered. (bkp, 15:47:51) * conferences and workshops (bkp, 15:47:58) * conferences and workshops FOSDEM is fast approaching. If you plan to attend this event on Jan. 31-Feb. 1, stop by the oVirt booth as well as the Virt and IaaS devrooms to hear more about oVirt! (bkp, 15:48:22) * conferences and workshops Stay tuned for more info on a FOSDEM social gathering for oVirt during the event, too! (bkp, 15:48:26) * conferences and workshops We are currently planning a one-day oVirt Workshop to coincide with FOSSAsia in Singapore, Feb. 13-15. Asia-Pac, Australia, and West Asia users are invited to attend. Details and registration to follow soon! (bkp, 15:48:29) * conferences and workshops jbrooks will be speaking on the new Smart VM Scheduler at SCALE 13X, as well as bkp, who will be discussing data centers vs. cloud VM management. A shared booth will be present as well. (bkp, 15:48:33) * conferences and workshops bkp will be speaking at the Linux Collab Summit on VM and container management (bkp, 15:48:37) * conferences and workshops amureini will be giving a talk at DevConfCZ in Feb, after FOSDEM (bkp, 15:51:27) * other topics (bkp, 15:51:53) * other topics bkp has completed a case study on CloudSpin, and will be posting it today on ovirt.org. (bkp, 15:52:25) * other topics One of the 2015 goals is updating oVirt documentation. To that end, we will compile a list of known use cases/tasks that users and admins do with oVirt in the real world. Once this task list is compiled, we can start building an action-oriented set of user documentation. bkp will be setting this up and sending out a notice soon. (bkp, 15:52:36) * other topics The Dec. 2014 newsletter was just posted (which is why this meeting was late), and can be found at http://lists.ovirt.org/pipermail/users/2015-January/030436.html (bkp, 15:52:50) Meeting ended at 15:55:53 UTC. Action Items ------------ * 3.6.0 status Feature proposed for 3.6.0 must now be collected in the 3.6 Google doc (http://goo.gl/9X3G49) and reviewed by maintainers Action Items, by person ----------------------- * **UNASSIGNED** * 3.6.0 status Feature proposed for 3.6.0 must now be collected in the 3.6 Google doc (http://goo.gl/9X3G49) and reviewed by maintainers -- Brian Proffitt Community Liaison oVirt Open Source and Standards, Red Hat - http://community.redhat.com Phone: +1 574 383 9BKP IRC: bkp @ OFTC

1 0

oVirt Monthly Newsletter: December 2014
by Brian Proffitt 07 Jan '15

07 Jan '15

The close of the old year was a little more of a downbeat, but the news and views about oVirt were still happening! News and Community oVirt is now officially part of the CentOS Virt SIG! (http://lists.centos.org/pipermail/centos-virt/2014-December/004187.html) Meanwhile, CentOS 7 gets reviewed using oVirt (http://www.networkworld.com/article/2861458/linux/centos-7-the-perfect-gift…) CloudSpin is a great new *free* (as in everything) machine-hosting service that uses oVirt. Learn more at https://cloudspin.me/ Better performance than VMware? Here's how to set up oVirt and Gluster to achieve better benchmarks [Spanish] http://blog.sotoca.es/2014/08/agilizando-la-infraestructura-iii-ovirt.html Find out how the University of Sevilla used oVirt and tools from UDS Enterprise to build a better virtual desktop solution for their campus: http://www.slideshare.net/UDSenterprise/virtual-desktops-in-educational-env… (http://www.slideshare.net/UDSenterpriseSpain/escritorios-virtuales-en-entor… [Spanish]) Allon Murenik opines on The Helpful Stranger and Meaning of Open Source (http://opensource.com/life/14/12/the-meaning-of-open-source) Open Source Is Just Another Way of Doing Good Business, and everyone knows it: http://community.redhat.com/blog/2014/12/open-source-is-just-another-way-of… ManageIQ partner VMTurbo asks: Does a Monitoring Tool Provide Automated Decisions? Especially using technologies like oVirt and RHEV? http://vmturbo.com/blog/monitoring-tool-automated-decisions/ Deploying and Managing Gluster using oVirt Webadmin Portal http://blogs-ramesh.blogspot.in/2014/12/deploying-gluster-using-ovirt.html oVirt's China community just posted the oVirt 3.5 release notes (http://ovirt-china.org/mediawiki/index.php/OVirt_3.5_Release_Notes) Discover more about open source virtualization options (http://computerworld.cz/software/virtualizace-ve-viru-open-source-51667) [Czech] Software Check out rbovirt, a Ruby client for the oVirt REST API (http://rubygems.org/gems/rbovirt) Learn more about the new UI plugin for oVirt, iso-uploader-plugin: https://github.com/ovirt-china/iso-uploader-plugin/wiki/Specifications Videos oVirt is featured software on the popular Hak5 video podcast http://youtu.be/6w8F5k41_9E Learn more about another successful VMware to oVirt migrtation: http://tv.rediris.es/es/jt2014/400.html [Spanish] Introduction to oVirt as CISL: http://youtu.be/EuxRrlsGO3k [Portuguese] Manage oVirt Open Source Project Infrastructure with Jenkins: http://youtu.be/WIRxw3noMmA [Hebrew] Take a video tour of oVirt http://youtu.be/cgH20bYt2z8 [Russian] -- Brian Proffitt Community Liaison oVirt Open Source and Standards, Red Hat - http://community.redhat.com Phone: +1 574 383 9BKP IRC: bkp @ OFTC

1 0

hosted-engine --deploy fails
by Andreas Mather 07 Jan '15

07 Jan '15

Hi All! Just tried to install the hosted engine on a fresh CentOS 6.6: beside setting up a gluster cluster, I just added the repo and installed hosted-engine-setup package. Otherwise it's just a very minimal installation. hosted-engine deploy failed immediately. The issue seems to be the same as described here http://lists.ovirt.org/pipermail/users/2014-October/028461.html but that conversation didn't continue after the reporting user was asked for additional details. Failed installation: [root@vhost1 ~]# hosted-engine --deploy [ INFO ] Stage: Initializing Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards. Are you sure you want to continue? (Yes, No)[Yes]: [ INFO ] Generating a temporary VNC password. [ INFO ] Stage: Environment setup Configuration files: [] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20141223224236-y9ttk4.log Version: otopi-1.3.0 (otopi-1.3.0-1.el6) [ INFO ] Hardware supports virtualization [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ ERROR ] Failed to execute stage 'Environment setup': Command '/sbin/service' failed to execute [ INFO ] Stage: Clean up [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf' [ INFO ] Answer file '/etc/ovirt-hosted-engine/answers.conf' has been updated [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination Packages: [root@vhost1 ~]# rpm -qa|grep hosted-engine ovirt-hosted-engine-setup-1.2.1-1.el6.noarch ovirt-hosted-engine-ha-1.2.4-1.el6.noarch [root@vhost1 ~]# rpm -qa|grep vdsm vdsm-yajsonrpc-4.16.7-1.gitdb83943.el6.noarch vdsm-python-zombiereaper-4.16.7-1.gitdb83943.el6.noarch vdsm-python-4.16.7-1.gitdb83943.el6.noarch vdsm-xmlrpc-4.16.7-1.gitdb83943.el6.noarch vdsm-4.16.7-1.gitdb83943.el6.x86_64 vdsm-cli-4.16.7-1.gitdb83943.el6.noarch vdsm-jsonrpc-4.16.7-1.gitdb83943.el6.noarch So here's the output that was requested in the other thread. Hope someone can help me here. Thanks! [root@vhost1 ~]# find /var/lib/vdsm/persistence /var/lib/vdsm/persistence [root@vhost1 ~]# find /var/run/vdsm/netconf find: `/var/run/vdsm/netconf': No such file or directory [root@vhost1 ~]# ip l 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:25:90:d8:0a:b0 brd ff:ff:ff:ff:ff:ff 4: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN link/ether 62:e5:28:13:9d:ba brd ff:ff:ff:ff:ff:ff 5: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff virsh -r net [root@vhost1 ~]# virsh -r net error: unknown command: 'net' [root@vhost1 ~]# virsh -r net-list Name State Autostart Persistent -------------------------------------------------- ;vdsmdummy; active no no [root@vhost1 ~]# vdsm-tool restore-nets Traceback (most recent call last): File "/usr/share/vdsm/vdsm-restore-net-config", line 137, in <module> restore() File "/usr/share/vdsm/vdsm-restore-net-config", line 123, in restore unified_restoration() File "/usr/share/vdsm/vdsm-restore-net-config", line 57, in unified_restoration _inRollback=True) File "/usr/share/vdsm/network/api.py", line 616, in setupNetworks netinfo._libvirtNets2vdsm(libvirt_nets))) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 822, in get d['nics'][dev.name] = _nicinfo(dev, paddr, ipaddrs) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 653, in _nicinfo info = _devinfo(link, ipaddrs) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 681, in _devinfo ipv4addr, ipv4netmask, ipv4addrs, ipv6addrs = getIpInfo(link.name, ipaddrs) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 370, in getIpInfo ipv4addr, prefix = addr['address'].split('/') ValueError: need more than 1 value to unpack Traceback (most recent call last): File "/usr/bin/vdsm-tool", line 209, in main return tool_command[cmd]["command"](*args) File "/usr/lib/python2.6/site-packages/vdsm/tool/restore_nets.py", line 36, in restore_command restore() File "/usr/lib/python2.6/site-packages/vdsm/tool/restore_nets.py", line 45, in restore raise EnvironmentError('Failed to restore the persisted networks') EnvironmentError: Failed to restore the persisted networks following is mentioned in the original thread, but should help other googlers: ovirt-hosted-engine-setup.log (first attempt): 2014-12-23 22:04:38 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:861 execute-output: ('/sbin/service', 'vdsmd', 'start') stdout: Starting multipathd daemon: [ OK ] Starting rpcbind: [ OK ] Starting ntpd: [ OK ] Loading the softdog kernel module: [ OK ] Starting wdmd: [ OK ] Starting sanlock: [ OK ] supervdsm start[ OK ] Starting iscsid: [ OK ] [ OK ] vdsm: Running mkdirs vdsm: Running configure_coredump vdsm: Running configure_vdsm_logs vdsm: Running run_init_hooks vdsm: Running check_is_configured libvirt is already configured for vdsm vdsm: Running validate_configuration SUCCESS: ssl configured to true. No conflicts vdsm: Running prepare_transient_repository vdsm: Running syslog_available vdsm: Running nwfilter vdsm: Running dummybr vdsm: Running load_needed_modules vdsm: Running tune_system vdsm: Running test_space vdsm: Running test_lo vdsm: Running unified_network_persistence_upgrade vdsm: stopped during execute unified_network_persistence_upgrade task (task returned with error code 1). vdsm start[FAILED] 2014-12-23 22:04:38 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:866 execute-output: ('/sbin/service', 'vdsmd', 'start') stderr: initctl: Job is already running: libvirtd libvirt: Network Filter Driver error : Network filter not found: no nwfilter with matching name 'vdsm-no-mac-spoofing' 2014-12-23 22:04:38 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/system/vdsmenv.py", line 155, in _late_setup state=True File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 188, in state 'start' if state else 'stop' File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 96, in _executeServiceCommand raiseOnError=raiseOnError File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 871, in execute command=args[0], RuntimeError: Command '/sbin/service' failed to execute 2014-12-23 22:04:38 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment setup': Command '/sbin/service' failed to execute ovirt-hosted-engine-setup.log (further attempts): 2014-12-23 22:42:40 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:861 execute-output: ('/sbin/service', 'vdsmd', 'start') stdout: vdsm: Running mkdirs vdsm: Running configure_coredump vdsm: Running configure_vdsm_logs vdsm: Running run_init_hooks vdsm: Running check_is_configured libvirt is already configured for vdsm vdsm: Running validate_configuration SUCCESS: ssl configured to true. No conflicts vdsm: Running prepare_transient_repository vdsm: Running syslog_available vdsm: Running nwfilter vdsm: Running dummybr vdsm: Running load_needed_modules vdsm: Running tune_system vdsm: Running test_space vdsm: Running test_lo vdsm: Running unified_network_persistence_upgrade vdsm: stopped during execute unified_network_persistence_upgrade task (task returned with error code 1). vdsm start[FAILED] 2014-12-23 22:42:40 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:866 execute-output: ('/sbin/service', 'vdsmd', 'start') stderr: initctl: Job is already running: libvirtd 2014-12-23 22:42:40 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/system/vdsmenv.py", line 155, in _late_setup state=True File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 188, in state 'start' if state else 'stop' File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 96, in _executeServiceCommand raiseOnError=raiseOnError File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 871, in execute command=args[0], RuntimeError: Command '/sbin/service' failed to execute 2014-12-23 22:42:40 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment setup': Command '/sbin/service' failed to execute ---- Andreas

4 6

[QE][ACTION REQUIRED] oVirt 3.6.0 status
by Sandro Bonazzola 07 Jan '15

07 Jan '15

Hi, I haven't many news for 3.6 this week: ACTION: Feature proposed for 3.6.0 must now be collected in the 3.6 Google doc [1] and reviewed by maintainers. Finished the review process, the remaining key milestones for this release will be scheduled. For reference, external project schedules we're tracking are: Fedora 21: 2014-12-09 (RELEASED) Fedora 22: no earlier than 2015-05-19 Foreman 1.8.0: 2015-03-01 GlusterFS 3.7: 2015-04-29 OpenStack Kilo: 2015-04-30 QEMU 2.1.3: 2014-12-29 (DELAYED) QEMU 2.2.0: 2014-12-09 (RELEASED) The tracker bug for 3.6.0 [2] currently shows no blockers. There are 466 bugs [3] targeted to 3.6.0. Excluding node and documentation bugs we have 440 bugs [4] targeted to 3.6.0. [1] http://goo.gl/9X3G49 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1155425 [3] http://goo.gl/zwkF3r [4] http://goo.gl/ZbUiMc -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

MooseFS cluster as storage domain
by Andrzej Suś 07 Jan '15

07 Jan '15

Hi everyone, I would like to create storage domain based on MooseFS cluster. The first idea was to do it by mount Posix compliant FS, but it doesn't like to mount (or i don't know how to do it). The next idea was to use NFS, but even that won't work. NFS causing Kernel crash and rebooting a Host when trying to create storage domain. I've checked everything I found on Google and all my NFS settings seems to be OK but it's still crashing. I know that MFS's issue have been taken here and it may be supported in future, but does anybody know how to get it working in oVirt 3.4 or 3.5? Or maybe somebody known problem with NFS and kernel panics on storage domain creating?

1 0

Re: [ovirt-users] LDAP Certificate Location?
by Alon Bar-Lev 07 Jan '15

07 Jan '15

Please use the formal documentation and if needed help improve them. References for ssl: http://gerrit.ovirt.org/gitweb?p=ovirt-engine-extension-aaa-ldap.git;a=blob… http://gerrit.ovirt.org/gitweb?p=ovirt-engine-extension-aaa-ldap.git;a=blob… http://gerrit.ovirt.org/gitweb?p=ovirt-engine-extension-aaa-ldap.git;a=blob… ----- Original Message ----- > From: "Donny Davis" <donny(a)cloudspin.me> > To: "Sandvik Agustin" <agustinsandvik(a)gmail.com>, users(a)ovirt.org > Sent: Wednesday, January 7, 2015 12:12:23 AM > Subject: Re: [ovirt-users] LDAP Certificate Location? > > > > In the article you referenced you didn't setup tls > On Jan 6, 2015 2:04 PM, Sandvik Agustin <agustinsandvik(a)gmail.com> wrote: > > > > Hi Donny, > > > Sorry to bother you at this time, I installed the 389ds by following this > http://www.unixmen.com/setup-directory-serverldap-in-centos-6-4-rhel-6-4/ > and now I'm following your documentation at > https://cloudspin.me/ovirt-simple-ldap-aaa/ I'm wondering if where can I > find this CA or pem thing you mention on your website " > /etc/pki/tls/cacerts/ldapCA.pem". > > Thanks in Advance, > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >

1 0

ERROR 'no free file handlers in pool' while creating VM from template
by Tiemen Ruiten 07 Jan '15

07 Jan '15

Hello, I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58. The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs. I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :) Tiemen

3 12

Re: [ovirt-users] LDAP Certificate Location?
by Donny Davis 06 Jan '15

06 Jan '15

1 0

Re: [ovirt-users] Can't remove a storage domain related to a broken hardware
by Olivier Navas 06 Jan '15

06 Jan '15

Parts of engine.log follow. - Part 1 repeats every 5 minutes, all the time - Part 2 is when I try to detach storage domain from the GUI (mixed with some logs as in 1) plus logs related to a VM start initiated by one of my colleagues). You will notice that engine reports some VM as "not responding". It happens during the process of removing the defective storage domain, but in fact VMs are fine and their state reverts back to normal when the task of removing the storage domain finally fails. Thank you for your help. ===== part 1 ===== 2015-01-06 03:50:00,016 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) [5ca5ae35] Autorecovering 1 storage domains 2015-01-06 03:50:00,017 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) [5ca5ae35] Autorecovering storage domains id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 2015-01-06 03:50:00,020 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-81) [789ace0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: Storage 2015-01-06 03:50:00,022 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-81) [789ace0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:1/6/15 3:50 AM 2015-01-06 03:50:00,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [789ace0] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 65ce62da 2015-01-06 03:50:00,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [789ace0] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50bc0a6d 2015-01-06 03:50:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-29) [789ace0] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: e40ca91 2015-01-06 03:50:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-32) [789ace0] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5ddb16e 2015-01-06 03:53:00,216 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [789ace0] Command ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 03:53:00,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [789ace0] FINISH, ConnectStorageServerVDSCommand, log id: 65ce62da 2015-01-06 03:53:00,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [789ace0] Command ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 03:53:00,224 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-12) [789ace0] Failed to connect host n2orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 03:53:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [789ace0] FINISH, ConnectStorageServerVDSCommand, log id: 50bc0a6d 2015-01-06 03:53:00,245 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-37) [789ace0] Failed to connect host n4orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 03:53:00,249 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-29) [789ace0] Command ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 03:53:00,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-29) [789ace0] FINISH, ConnectStorageServerVDSCommand, log id: e40ca91 2015-01-06 03:53:00,265 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-32) [789ace0] Command ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 03:53:00,266 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-29) [789ace0] Failed to connect host n3orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 03:53:00,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-32) [789ace0] FINISH, ConnectStorageServerVDSCommand, log id: 5ddb16e 2015-01-06 03:53:00,287 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-32) [789ace0] Failed to connect host n1orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 03:53:00,301 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-81) [789ace0] ConnectDomainToStorage. After Connect all hosts to pool. Time:1/6/15 3:53 AM ===== part 2 ===== 2015-01-06 09:55:00,009 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-9) [71cf84f9] Autorecovering 1 storage domains 2015-01-06 09:55:00,011 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-9) [71cf84f9] Autorecovering storage domains id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 2015-01-06 09:55:00,013 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-9) [6ce74644] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: Storage 2015-01-06 09:55:00,016 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-9) [6ce74644] ConnectDomainToStorage. Before Connect all hosts to pool. Time:1/6/15 9:55 AM 2015-01-06 09:55:00,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [6ce74644] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2bb163a4 2015-01-06 09:55:00,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [6ce74644] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 49562818 2015-01-06 09:55:00,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6ce74644] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4cb88aac 2015-01-06 09:55:00,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [6ce74644] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 23eae7f9 2015-01-06 09:55:57,598 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp--127.0.0.1-8702-6) Invalid locale found in configuration: 2015-01-06 09:55:57,600 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp--127.0.0.1-8702-6) Invalid locale found in configuration: 2015-01-06 09:56:00,437 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp--127.0.0.1-8702-11) Invalid locale found in configuration: 2015-01-06 09:56:00,438 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp--127.0.0.1-8702-11) Invalid locale found in configuration: 2015-01-06 09:56:10,268 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginAdminUserCommand internal: false. 2015-01-06 09:56:10,276 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. 2015-01-06 09:56:10,388 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. 2015-01-06 09:56:24,013 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-13) [c2fb019] Lock Acquired to object EngineLock [exclusiveLocks= key: e2abb477-f807-470f-ae20-a7205e690638 value: VM , sharedLocks= ] 2015-01-06 09:56:24,134 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-13) [c2fb019] START, IsVmDuringInitiatingVDSCommand( vmId = e2abb477-f807-470f-ae20-a7205e690638), log id: 5c2ceac9 2015-01-06 09:56:24,136 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-13) [c2fb019] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5c2ceac9 2015-01-06 09:56:24,357 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] Running command: RunVmCommand internal: false. Entities affected : ID: e2abb477-f807-470f-ae20-a7205e690638 Type: VMAction group VM_BASIC_OPERATIONS with role type USER 2015-01-06 09:56:24,500 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] (org.ovirt.thread.pool-8-thread-9) [c2fb019] Started HA reservation scoring method 2015-01-06 09:56:24,561 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] START, UpdateVmDynamicDataVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@57214812), log id: 63eaa18e 2015-01-06 09:56:24,573 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] FINISH, UpdateVmDynamicDataVDSCommand, log id: 63eaa18e 2015-01-06 09:56:24,612 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] START, CreateVmVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vmId=e2abb477-f807-470f-ae20-a7205e690638, vm=VM [ciril-recette]), log id: 5c6b51df 2015-01-06 09:56:24,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] START, CreateVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vmId=e2abb477-f807-470f-ae20-a7205e690638, vm=VM [ciril-recette]), log id: 6bcd4ecc 2015-01-06 09:56:24,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,pitReinjection=false,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=e2abb477-f807-470f-ae20-a7205e690638,acpiEnable=true,custom={device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388device_41d6489a-07ab-481d-b35e-a14e007df6cd=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=41d6489a-07ab-481d-b35e-a14e007df6cd, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388device_41d6489a-07ab-481d-b35e-a14e007df6cddevice_e3fa7b29-5b4f-462d-bcf4-298916996aaf=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=e3fa7b29-5b4f-462d-bcf4-298916996aaf, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=5cc74d49-61c4-4887-8cf3-41135fbdee99, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=ee0562ba-7750-4857-8151-f422de37a388, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}},spiceSslCipherSuite=DEFAULT,memSize=6000,smp=4,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1500,display=qxl,smartcardEnable=false,bootMenuEnable=false,numaTune={mode=interleave, nodeset=0,1},spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,maxVCpus=16,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={ram=65536, vram=32768, heads=1}, device=qxl, type=video, deviceId=061ed0c0-22f5-4099-8bee-44a3dfa19956}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=dd759f21-7cca-4d56-8f0f-348d61492714}, {shared=false, index=0, volumeID=82528148-a4db-4892-87e6-b41c7b2f3ab8, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=05d4e406-de6d-47dc-8414-bdc7381b6d4a, imageID=f09ba312-5078-4070-801c-4a8c7e28d0aa, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-00000000000f, readonly=false, deviceId=f09ba312-5078-4070-801c-4a8c7e28d0aa}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:70:41:a3, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=Cluster_VM, deviceId=b2e197f0-00f4-44f4-bd5c-32824d872c95}, {address={bus=0x00, domain=0x0000, slot=0x07, type=pci, function=0x0}, specParams={model=virtio}, device=memballoon, type=balloon, deviceId=e0a803e7-eb97-4285-a49f-6e99a5df974c}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=bd1cf13f-212b-4ceb-96c7-ee4950f7a7d3}, {address={bus=0x00, domain=0x0000, slot=0x05, type=pci, function=0x0}, specParams={}, device=virtio-serial, type=controller, deviceId=a2df4f4f-2920-4243-ba9a-26206057de6e}],vmName=ciril-recette,cpuType=Westmere,fileTransferEnable=true 2015-01-06 09:56:24,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] FINISH, CreateVDSCommand, log id: 6bcd4ecc 2015-01-06 09:56:24,852 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5c6b51df 2015-01-06 09:56:24,854 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] Lock freed to object EngineLock [exclusiveLocks= key: e2abb477-f807-470f-ae20-a7205e690638 value: VM , sharedLocks= ] 2015-01-06 09:56:24,873 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [c2fb019] Correlation ID: c2fb019, Job ID: 42d86800-fc18-42d6-98a6-0cf205a85a2c, Call Stack: null, Custom Event ID: -1, Message: VM ciril-recette was started by admin (Host: n4orna). 2015-01-06 09:56:28,296 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) [128f72d7] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from WaitForLaunch --> PoweringUp 2015-01-06 09:56:28,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-73) [128f72d7] START, FullListVdsCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vds=Host[n4orna,920bf64c-62f5-4b12-a69e-eef9936576c5], vmIds=[e2abb477-f807-470f-ae20-a7205e690638]), log id: 79d27c1c 2015-01-06 09:56:28,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-73) [128f72d7] FINISH, FullListVdsCommand, return: [{displaySecurePort=5909, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.99.23.4, pauseCode=NOERR, pitReinjection=false, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=e2abb477-f807-470f-ae20-a7205e690638, acpiEnable=true, custom={device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388device_41d6489a-07ab-481d-b35e-a14e007df6cd=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=41d6489a-07ab-481d-b35e-a14e007df6cd, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388device_41d6489a-07ab-481d-b35e-a14e007df6cddevice_e3fa7b29-5b4f-462d-bcf4-298916996aaf=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=e3fa7b29-5b4f-462d-bcf4-298916996aaf, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=5cc74d49-61c4-4887-8cf3-41135fbdee99, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=ee0562ba-7750-4857-8151-f422de37a388, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}}, spiceSslCipherSuite=DEFAULT, memSize=6000, smp=4, displayPort=-1, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1500, display=qxl, pid=23762, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={nodeset=0,1, mode=interleave}, smpCoresPerSocket=1, maxVCpus=16, clientIp=, devices=[Ljava.lang.Object;@69d250bf, vmName=ciril-recette, fileTransferEnable=true, cpuType=Westmere}], log id: 79d27c1c 2015-01-06 09:56:28,328 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) [128f72d7] Received a spice Device without an address when processing VM e2abb477-f807-470f-ae20-a7205e690638 devices, skipping device: {specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.99.23.4}, device=spice, tlsPort=5909, type=graphics} 2015-01-06 09:56:46,936 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [27e4f6a6] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from PoweringUp --> Up 2015-01-06 09:56:47,065 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [27e4f6a6] Correlation ID: c2fb019, Job ID: 42d86800-fc18-42d6-98a6-0cf205a85a2c, Call Stack: null, Custom Event ID: -1, Message: VM ciril-recette started on Host n4orna 2015-01-06 09:57:57,112 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (ajp--127.0.0.1-8702-6) [631b7fb5] Lock Acquired to object EngineLock [exclusiveLocks= key: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 value: STORAGE , sharedLocks= ] 2015-01-06 09:57:57,517 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2015-01-06 09:57:57,520 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Start detach storage domain 2015-01-06 09:57:57,530 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Detach storage domain: before connect 2015-01-06 09:57:57,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [631b7fb5] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 13e1b92c 2015-01-06 09:57:57,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [631b7fb5] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b9a83cf 2015-01-06 09:57:57,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [631b7fb5] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2a3e2925 2015-01-06 09:57:57,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [631b7fb5] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6386df71 2015-01-06 09:58:00,209 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [6ce74644] Command ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 09:58:00,214 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [6ce74644] Command ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 09:58:00,220 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6ce74644] Command ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 09:58:00,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [6ce74644] FINISH, ConnectStorageServerVDSCommand, log id: 2bb163a4 2015-01-06 09:58:00,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6ce74644] FINISH, ConnectStorageServerVDSCommand, log id: 4cb88aac 2015-01-06 09:58:00,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [6ce74644] Command ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 09:58:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [6ce74644] FINISH, ConnectStorageServerVDSCommand, log id: 49562818 2015-01-06 09:58:00,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [6ce74644] FINISH, ConnectStorageServerVDSCommand, log id: 23eae7f9 2015-01-06 09:58:00,230 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-39) [6ce74644] Failed to connect host n3orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 09:58:00,228 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-49) [6ce74644] Failed to connect host n2orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 09:58:00,239 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-7) [6ce74644] Failed to connect host n4orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 09:58:00,237 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-18) [6ce74644] Failed to connect host n1orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 09:58:00,294 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-9) [6ce74644] ConnectDomainToStorage. After Connect all hosts to pool. Time:1/6/15 9:58 AM 2015-01-06 09:58:03,032 INFO [org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Running command: SyncLunsInfoForBlockStorageDomainCommand internal: true. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: Storage 2015-01-06 09:58:03,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] START, GetVGInfoVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, VGID=Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom), log id: a46928a 2015-01-06 09:58:03,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Failed in GetVGInfoVDS method 2015-01-06 09:58:03,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand return value OneVGReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=506, mMessage=Volume Group does not exist: ('vg_uuid: Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom',)]] 2015-01-06 09:58:03,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] HostName = n2orna 2015-01-06 09:58:03,231 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Command GetVGInfoVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, VGID=Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetVGInfoVDS, error = Volume Group does not exist: ('vg_uuid: Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom',), code = 506 2015-01-06 09:58:03,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] FINISH, GetVGInfoVDSCommand, log id: a46928a 2015-01-06 09:58:03,236 ERROR [org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Command org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetVGInfoVDS, error = Volume Group does not exist: ('vg_uuid: Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom',), code = 506 (Failed with error VolumeGroupDoesNotExist and code 506) 2015-01-06 09:58:06,526 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 34c95a44-db7f-4d0f-ba13-5f06a7feefe7:fujitsu_backup_rsync in problem. vds: n1orna 2015-01-06 09:58:06,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Host 60d8c75b-38f1-4cd0-b162-729285eadefd has reported new storage access problem to the following domains 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 marking it for storage connections and pool metadata refresh (report id: 6a081ad8-201f-48e4-ba56-5a581fac4475) 2015-01-06 09:58:06,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Host 38893029-0eb8-4d19-a28f-07680d8d6868 has reported new storage access problem to the following domains 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 marking it for storage connections and pool metadata refresh (report id: 922d1075-c975-4d9f-87d1-db8d5aa4059d) 2015-01-06 09:58:06,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [6e3156c6] Running storage connections refresh for hosts [38893029-0eb8-4d19-a28f-07680d8d6868, 60d8c75b-38f1-4cd0-b162-729285eadefd] 2015-01-06 09:58:06,648 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 09:58:06,648 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 09:58:06,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) Host 920bf64c-62f5-4b12-a69e-eef9936576c5 has reported new storage access problem to the following domains 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 marking it for storage connections and pool metadata refresh (report id: 12dfe3fe-966a-486e-8eda-89575648545f) 2015-01-06 09:58:06,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70a8393b 2015-01-06 09:58:06,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 396e347 2015-01-06 09:58:15,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) Host ab744426-2294-4c0b-aaf5-08ebb162f542 has reported new storage access problem to the following domains 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 marking it for storage connections and pool metadata refresh (report id: 18fffe35-403c-413e-acb1-8d173aa447da) 2015-01-06 09:58:52,862 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [7face9e1] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from Up --> RebootInProgress 2015-01-06 09:59:23,829 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [4c461177] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from RebootInProgress --> Up 2015-01-06 09:59:48,453 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) domain 05d4e406-de6d-47dc-8414-bdc7381b6d4a:MSA1-O2-vd01 in problem. vds: n3orna 2015-01-06 10:00:34,503 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [289a4e02] VM ldap-recette ad20808d-b62c-4087-a41a-31ed7672e364 moved from Up --> NotResponding 2015-01-06 10:00:34,550 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [289a4e02] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ldap-recette is not responding. 2015-01-06 10:00:34,572 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [289a4e02] VM ciril 627fbbe4-4812-4218-a512-cc1ed26124fd moved from Up --> NotResponding 2015-01-06 10:00:34,618 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [289a4e02] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ciril is not responding. 2015-01-06 10:00:50,138 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [50e93ebc] VM antivirus2 1437b20a-d3cb-4537-ace5-2dee650f561f moved from Up --> NotResponding 2015-01-06 10:00:50,183 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [50e93ebc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM antivirus2 is not responding. 2015-01-06 10:00:50,188 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [50e93ebc] VM rsst-formation 064f2b1f-f245-4f34-9747-27c28ce388df moved from Up --> NotResponding 2015-01-06 10:00:50,257 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [50e93ebc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM rsst-formation is not responding. 2015-01-06 10:00:50,262 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [50e93ebc] VM forum c94e5f1f-1f10-47e5-b4bb-87d1ef31caf9 moved from Up --> NotResponding 2015-01-06 10:00:50,303 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [50e93ebc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM forum is not responding. 2015-01-06 10:00:57,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [631b7fb5] Command ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:00:57,737 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [631b7fb5] Command ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:00:57,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [631b7fb5] Command ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:00:57,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [631b7fb5] FINISH, ConnectStorageServerVDSCommand, log id: 13e1b92c 2015-01-06 10:00:57,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [631b7fb5] FINISH, ConnectStorageServerVDSCommand, log id: 2a3e2925 2015-01-06 10:00:57,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [631b7fb5] FINISH, ConnectStorageServerVDSCommand, log id: 7b9a83cf 2015-01-06 10:00:57,764 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-37) [631b7fb5] Failed to connect host n4orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 10:00:57,762 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-50) [631b7fb5] Failed to connect host n2orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 10:00:57,767 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-21) [631b7fb5] Failed to connect host n1orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 10:00:57,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [631b7fb5] Command ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:00:57,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [631b7fb5] FINISH, ConnectStorageServerVDSCommand, log id: 6386df71 2015-01-06 10:00:57,871 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-16) [631b7fb5] Failed to connect host n3orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 10:00:57,902 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Detach storage domain: after connect 2015-01-06 10:00:57,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] START, DetachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-00000000000f, ignoreFailoverLimit = false, storageDomainId = 34c95a44-db7f-4d0f-ba13-5f06a7feefe7, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 7e14d5fa 2015-01-06 10:01:05,877 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [3c054f3d] VM voip-admin 34d7033b-72bd-4c61-9a24-a73c5541955c moved from Up --> NotResponding 2015-01-06 10:01:05,925 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [3c054f3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM voip-admin is not responding. 2015-01-06 10:01:06,770 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] Command ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:01:06,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] Command ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:01:06,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] FINISH, ConnectStorageServerVDSCommand, log id: 70a8393b 2015-01-06 10:01:06,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] FINISH, ConnectStorageServerVDSCommand, log id: 396e347 2015-01-06 10:01:06,799 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) 2015-01-06 10:01:06,801 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) 2015-01-06 10:01:06,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [6e3156c6] Submitting to the event queue pool refresh for hosts [38893029-0eb8-4d19-a28f-07680d8d6868, 60d8c75b-38f1-4cd0-b162-729285eadefd] 2015-01-06 10:01:06,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) Running storage pool metadata refresh for hosts {1} 2015-01-06 10:01:06,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-47) START, ConnectStoragePoolVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, vdsId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, masterVersion = 1), log id: 377e2518 2015-01-06 10:01:06,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, ConnectStoragePoolVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, vdsId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, masterVersion = 1), log id: 2d074200 2015-01-06 10:01:15,895 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Failed in DetachStorageDomainVDS method 2015-01-06 10:01:15,898 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] IrsBroker::Failed::DetachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to DetachStorageDomainVDS, error = Storage domain does not exist: ('34c95a44-db7f-4d0f-ba13-5f06a7feefe7',), code = 358 2015-01-06 10:01:21,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, ConnectStoragePoolVDSCommand, log id: 2d074200 2015-01-06 10:01:22,578 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [2ba9b406] VM packetfence adef4f71-d322-4307-b1d1-c0988bd4efcb moved from Up --> NotResponding 2015-01-06 10:01:22,624 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [2ba9b406] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM packetfence is not responding. 2015-01-06 10:01:22,629 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [2ba9b406] VM afs1 2256fed4-ff9e-4520-9e89-fe0bca30ec1b moved from Up --> NotResponding 2015-01-06 10:01:22,672 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [2ba9b406] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM afs1 is not responding. 2015-01-06 10:01:22,676 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [2ba9b406] VM piros e22277e0-723f-40b6-a46d-ca8f821807e7 moved from Up --> NotResponding 2015-01-06 10:01:22,719 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [2ba9b406] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM piros is not responding. 2015-01-06 10:03:21,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] START, SpmStopVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f), log id: 15cd9c64 2015-01-06 10:03:22,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] SpmStopVDSCommand::Stopping SPM on vds n2orna, pool id 00000002-0002-0002-0002-00000000000f 2015-01-06 10:03:22,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 377e2518 2015-01-06 10:03:22,037 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97:MSA2-O2 in problem. vds: n2orna 2015-01-06 10:03:22,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 34c95a44-db7f-4d0f-ba13-5f06a7feefe7:fujitsu_backup_rsync 2015-01-06 10:03:22,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] FINISH, SpmStopVDSCommand, log id: 15cd9c64 2015-01-06 10:03:22,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Irs placed on server 38893029-0eb8-4d19-a28f-07680d8d6868 failed. Proceed Failover 2015-01-06 10:03:22,219 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 34c95a44-db7f-4d0f-ba13-5f06a7feefe7:fujitsu_backup_rsync was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2015-01-06 10:03:22,251 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [74a0283] Failed to Acquire Lock to object EngineLock [exclusiveLocks= key: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 value: STORAGE , sharedLocks= key: 00000002-0002-0002-0002-00000000000f value: POOL ] 2015-01-06 10:03:22,258 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [74a0283] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_OBJECT_LOCKED 2015-01-06 10:03:22,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [74a0283] Removing vds [] from the domain in maintenance cache 2015-01-06 10:03:22,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [74a0283] Removing host(s) [] from hosts unseen domain report cache 2015-01-06 10:03:22,300 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 10:03:22,306 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (org.ovirt.thread.pool-8-thread-17) [422d495e] Storage Pool 00000002-0002-0002-0002-00000000000f - Updating Storage Domain 05d4e406-de6d-47dc-8414-bdc7381b6d4a status from Active to Unknown, reason : null 2015-01-06 10:03:22,311 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (org.ovirt.thread.pool-8-thread-17) [422d495e] Storage Pool 00000002-0002-0002-0002-00000000000f - Updating Storage Domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97 status from Active to Unknown, reason : null 2015-01-06 10:03:22,315 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (org.ovirt.thread.pool-8-thread-17) [422d495e] Storage Pool 00000002-0002-0002-0002-00000000000f - Updating Storage Domain 5a8e48d4-25f0-46f4-9682-e92785e9057a status from Active to Unknown, reason : null 2015-01-06 10:03:22,319 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (org.ovirt.thread.pool-8-thread-17) [422d495e] Storage Pool 00000002-0002-0002-0002-00000000000f - Updating Storage Domain 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 status from Active to Unknown, reason : null 2015-01-06 10:03:22,363 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [422d495e] Correlation ID: 422d495e, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. 2015-01-06 10:03:22,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [422d495e] hostFromVds::selectedVds - n3orna, spmStatus Free, storage pool Default 2015-01-06 10:03:22,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [422d495e] starting spm on vds n3orna, storage pool Default, prevId -1, LVER -1 2015-01-06 10:03:22,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] START, SpmStartVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 51aadbb5 2015-01-06 10:03:22,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] spmStart polling started: taskId = 58d4fff4-d230-4c6b-aae3-62b7ff427098 2015-01-06 10:03:26,697 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM ldap-recette ad20808d-b62c-4087-a41a-31ed7672e364 moved from NotResponding --> Up 2015-01-06 10:03:26,699 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM voip-admin 34d7033b-72bd-4c61-9a24-a73c5541955c moved from NotResponding --> Up 2015-01-06 10:03:26,702 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM packetfence adef4f71-d322-4307-b1d1-c0988bd4efcb moved from NotResponding --> Up 2015-01-06 10:03:26,704 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM afs1 2256fed4-ff9e-4520-9e89-fe0bca30ec1b moved from NotResponding --> Up 2015-01-06 10:03:26,707 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM piros e22277e0-723f-40b6-a46d-ca8f821807e7 moved from NotResponding --> Up 2015-01-06 10:03:26,709 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM ciril 627fbbe4-4812-4218-a512-cc1ed26124fd moved from NotResponding --> Up 2015-01-06 10:03:26,712 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM antivirus2 1437b20a-d3cb-4537-ace5-2dee650f561f moved from NotResponding --> Up 2015-01-06 10:03:26,714 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM rsst-formation 064f2b1f-f245-4f34-9747-27c28ce388df moved from NotResponding --> Up 2015-01-06 10:03:26,716 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM forum c94e5f1f-1f10-47e5-b4bb-87d1ef31caf9 moved from NotResponding --> Up 2015-01-06 10:03:28,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] spmStart polling ended: taskId = 58d4fff4-d230-4c6b-aae3-62b7ff427098 task status = finished 2015-01-06 10:03:28,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] spmStart polling ended, spm status: SPM 2015-01-06 10:03:28,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] START, HSMClearTaskVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, taskId=58d4fff4-d230-4c6b-aae3-62b7ff427098), log id: 3c3c7472 2015-01-06 10:03:28,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] FINISH, HSMClearTaskVDSCommand, log id: 3c3c7472 2015-01-06 10:03:28,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@146b8db3, log id: 51aadbb5 2015-01-06 10:03:28,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [422d495e] Initialize Irs proxy from vds: 10.99.23.3 2015-01-06 10:03:28,248 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [422d495e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host n3orna (Address: 10.99.23.3). 2015-01-06 10:03:28,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-00000000000f, ignoreFailoverLimit = false), log id: 5be8ceb1 2015-01-06 10:03:28,530 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] Failed in DetachStorageDomainVDS method 2015-01-06 10:03:28,533 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] IrsBroker::Failed::DetachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to DetachStorageDomainVDS, error = Storage domain does not exist: ('34c95a44-db7f-4d0f-ba13-5f06a7feefe7',), code = 358 2015-01-06 10:03:28,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] START, SpmStopVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f), log id: 430bdc38 2015-01-06 10:03:28,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] SpmStopVDSCommand::Stopping SPM on vds n3orna, pool id 00000002-0002-0002-0002-00000000000f 2015-01-06 10:03:28,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] FINISH, SpmStopVDSCommand, log id: 430bdc38 2015-01-06 10:03:28,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [422d495e] Irs placed on server ab744426-2294-4c0b-aaf5-08ebb162f542 failed. Proceed Failover 2015-01-06 10:03:29,052 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 10:03:29,099 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [18526185] Correlation ID: 18526185, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. 2015-01-06 10:03:29,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [18526185] hostFromVds::selectedVds - n4orna, spmStatus Free, storage pool Default 2015-01-06 10:03:29,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [18526185] starting spm on vds n4orna, storage pool Default, prevId -1, LVER -1 2015-01-06 10:03:29,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] START, SpmStartVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3992a29c 2015-01-06 10:03:29,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] spmStart polling started: taskId = d2c1864d-c71c-4378-a4a5-97d0afafd775 2015-01-06 10:03:31,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] spmStart polling ended: taskId = d2c1864d-c71c-4378-a4a5-97d0afafd775 task status = finished 2015-01-06 10:03:31,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] spmStart polling ended, spm status: SPM 2015-01-06 10:03:31,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] START, HSMClearTaskVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, taskId=d2c1864d-c71c-4378-a4a5-97d0afafd775), log id: 29266dd2 2015-01-06 10:03:31,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] FINISH, HSMClearTaskVDSCommand, log id: 29266dd2 2015-01-06 10:03:31,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59cd7a77, log id: 3992a29c 2015-01-06 10:03:31,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [18526185] Initialize Irs proxy from vds: 10.99.23.4 2015-01-06 10:03:31,575 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [18526185] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host n4orna (Address: 10.99.23.4). 2015-01-06 10:03:31,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-00000000000f, ignoreFailoverLimit = false), log id: 5e2b03ab 2015-01-06 10:03:31,711 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Failed in DetachStorageDomainVDS method 2015-01-06 10:03:31,714 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] IrsBroker::Failed::DetachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to DetachStorageDomainVDS, error = Storage domain does not exist: ('34c95a44-db7f-4d0f-ba13-5f06a7feefe7',), code = 358 2015-01-06 10:03:31,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] FINISH, DetachStorageDomainVDSCommand, log id: 7e14d5fa 2015-01-06 10:03:31,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-00000000000f 2015-01-06 10:03:31,721 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Command org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to DetachStorageDomainVDS, error = Storage domain does not exist: ('34c95a44-db7f-4d0f-ba13-5f06a7feefe7',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) 2015-01-06 10:03:31,741 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Command [id=3b4688b2-b626-470c-9ac9-4a65b0ef0864]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 00000002-0002-0002-0002-00000000000f, storageId = 34c95a44-db7f-4d0f-ba13-5f06a7feefe7, status=Inactive]. 2015-01-06 10:03:31,761 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [18526185] Correlation ID: 631b7fb5, Job ID: 780fb013-3a4c-4cc7-9bfd-a755989a6eeb, Call Stack: null, Custom Event ID: -1, Message: Failed to detach Storage Domain fujitsu_backup_rsync to Data Center Default. (User: admin) 2015-01-06 10:03:31,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] START, HSMGetAllTasksInfoVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5), log id: 32a82db7 2015-01-06 10:03:31,773 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Lock freed to object EngineLock [exclusiveLocks= key: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 value: STORAGE , sharedLocks= ] 2015-01-06 10:03:31,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32a82db7 2015-01-06 10:03:31,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e2b03ab 2015-01-06 10:03:31,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-00000000000f 2015-01-06 10:03:31,785 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [18526185] Discovered no tasks on Storage Pool Default 2015-01-06 10:03:31,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] START, HSMGetAllTasksInfoVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5), log id: 4eb41b3a 2015-01-06 10:03:31,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4eb41b3a 2015-01-06 10:03:31,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5be8ceb1 2015-01-06 10:03:31,831 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-26) [422d495e] Discovered no tasks on Storage Pool Default 2015-01-06 10:03:32,008 INFO [org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-8-thread-32) [afbbe0a] Running command: SyncLunsInfoForBlockStorageDomainCommand internal: true. Entities affected : ID: 05d4e406-de6d-47dc-8414-bdc7381b6d4a Type: Storage 2015-01-06 10:03:32,023 INFO [org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-8-thread-7) [73793c2b] Running command: SyncLunsInfoForBlockStorageDomainCommand internal: true. Entities affected : ID: 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97 Type: Storage 2015-01-06 10:03:32,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [afbbe0a] START, GetVGInfoVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, VGID=CtELsP-UYZb-zgRH-bVz7-lijD-Elfy-frzEJp), log id: 19fe6181 2015-01-06 10:03:32,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-7) [73793c2b] START, GetVGInfoVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, VGID=dEB6XY-cNBu-hsoz-avXf-m6Vp-4jOq-eDMCBK), log id: 14fc74a1 2015-01-06 10:03:32,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) [27e4f6a6] Running storage connections refresh for hosts [ab744426-2294-4c0b-aaf5-08ebb162f542, 920bf64c-62f5-4b12-a69e-eef9936576c5] 2015-01-06 10:03:32,107 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-27) [4bd57497] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 10:03:32,107 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-15) [2ad98c66] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 10:03:32,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-27) [4bd57497] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = NFS, connectionList = [{ id: 79bae336-2d76-4986-bf1e-50e350cde9f7, connection: cluster.33sdis.fr:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1acd4935 2015-01-06 10:03:32,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2ad98c66] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = NFS, connectionList = [{ id: 79bae336-2d76-4986-bf1e-50e350cde9f7, connection: cluster.33sdis.fr:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 10087d1 2015-01-06 10:03:32,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2ad98c66] FINISH, ConnectStorageServerVDSCommand, return: {79bae336-2d76-4986-bf1e-50e350cde9f7=0}, log id: 10087d1 2015-01-06 10:03:32,199 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-15) [2ad98c66] Host n3orna storage connection was succeeded 2015-01-06 10:03:32,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [afbbe0a] FINISH, GetVGInfoVDSCommand, return: [LUNs [id=3600c0ff000149e9d22c0105401000000, physicalVolumeId=OMa7pu-EyxG-mdNM-9EP5-eP2f-2GPV-tWiexe, volumeGroupId=CtELsP-UYZb-zgRH-bVz7-lijD-Elfy-frzEJp, serial=SHP_P2000_G3_FC_00c0ff149e9d000022c0105401000000, lunMapping=0, vendorId=HP, productId=P2000 G3 FC, _lunConnections=[], deviceSize=5583, vendorName=HP, pathsDictionary={sdb=true, sdd=true}, lunType=UNKNOWN, status=null, diskId=null, diskAlias=null, storageDomainId=05d4e406-de6d-47dc-8414-bdc7381b6d4a, storageDomainName=null]], log id: 19fe6181 2015-01-06 10:03:32,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-7) [73793c2b] FINISH, GetVGInfoVDSCommand, return: [LUNs [id=3600c0ff00015986a35d98a5401000000, physicalVolumeId=oWeQjL-dOc0-G9dH-wu1g-8pCd-jGjG-rzL0ZM, volumeGroupId=dEB6XY-cNBu-hsoz-avXf-m6Vp-4jOq-eDMCBK, serial=SHP_P2000_G3_FC_00c0ff15986a000035d98a5401000000, lunMapping=0, vendorId=HP, productId=P2000 G3 FC, _lunConnections=[], deviceSize=5583, vendorName=HP, pathsDictionary={sdc=true, sde=true}, lunType=UNKNOWN, status=null, diskId=null, diskAlias=null, storageDomainId=75a3e25a-2ea6-4b95-9d9f-9fe791e38e97, storageDomainName=null]], log id: 14fc74a1 2015-01-06 10:03:32,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-27) [4bd57497] FINISH, ConnectStorageServerVDSCommand, return: {79bae336-2d76-4986-bf1e-50e350cde9f7=0}, log id: 1acd4935 2015-01-06 10:03:32,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-27) [4bd57497] Host n4orna storage connection was succeeded 2015-01-06 10:03:32,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) [27e4f6a6] Submitting to the event queue pool refresh for hosts [ab744426-2294-4c0b-aaf5-08ebb162f542, 920bf64c-62f5-4b12-a69e-eef9936576c5] 2015-01-06 10:03:32,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Running storage pool metadata refresh for hosts {1} 2015-01-06 10:03:32,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, ConnectStoragePoolVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vdsId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, masterVersion = 1), log id: 66dc8feb 2015-01-06 10:03:32,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, vdsId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, masterVersion = 1), log id: 1a6cc20f 2015-01-06 10:03:34,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, ConnectStoragePoolVDSCommand, log id: 66dc8feb 2015-01-06 10:03:34,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 1a6cc20f 2015-01-06 10:03:37,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Host 60d8c75b-38f1-4cd0-b162-729285eadefd no longer storage access problem to any relevant domain clearing its report (report id: 6a081ad8-201f-48e4-ba56-5a581fac4475) 2015-01-06 10:03:37,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97:MSA2-O2 recovered from problem. vds: n1orna 2015-01-06 10:03:37,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Host 38893029-0eb8-4d19-a28f-07680d8d6868 no longer storage access problem to any relevant domain clearing its report (report id: 922d1075-c975-4d9f-87d1-db8d5aa4059d) 2015-01-06 10:03:37,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97:MSA2-O2 recovered from problem. vds: n2orna 2015-01-06 10:03:37,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97:MSA2-O2 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. 2015-01-06 10:03:37,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Host 920bf64c-62f5-4b12-a69e-eef9936576c5 no longer storage access problem to any relevant domain clearing its report (report id: 12dfe3fe-966a-486e-8eda-89575648545f) 2015-01-06 10:03:39,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Host ab744426-2294-4c0b-aaf5-08ebb162f542 no longer storage access problem to any relevant domain clearing its report (report id: 18fffe35-403c-413e-acb1-8d173aa447da) 2015-01-06 10:03:39,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 05d4e406-de6d-47dc-8414-bdc7381b6d4a:MSA1-O2-vd01 recovered from problem. vds: n3orna 2015-01-06 10:03:39,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 05d4e406-de6d-47dc-8414-bdc7381b6d4a:MSA1-O2-vd01 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. 2015-01-06 10:03:44,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [5770eda1] No hosts has reported storage access problem to domains, clearing the handled hosts reports map 2015-01-06 10:05:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) [1cbf3978] Autorecovering 1 storage domains 2015-01-06 10:05:00,009 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) [1cbf3978] Autorecovering storage domains id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 2015-01-06 10:05:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [344e83b5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: Storage 2015-01-06 10:05:00,019 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [344e83b5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:1/6/15 10:05 AM 2015-01-06 10:05:00,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [344e83b5] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1d6d563c 2015-01-06 10:05:00,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-28) [344e83b5] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 12a471c7 2015-01-06 10:05:00,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [344e83b5] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70547eab 2015-01-06 10:05:00,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-19) [344e83b5] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6438f79b 2015-01-06 10:06:11,746 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [453f31bd] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from Up --> RebootInProgress 2015-01-06 10:06:39,611 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) [537321db] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from RebootInProgress --> Up -- Ce courriel et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à la personne à laquelle ils sont adressés. Si vous avez reçu ce courriel par erreur, merci de le retourner à son expéditeur et de le détruire. Il est rappelé que tout message électronique est susceptible d'alteration au cours de son acheminement sur internet. Seuls les documents officiels du SDIS sont de nature à engager sa responsabilité. Les idées ou opinions présentées dans ce courriel sont celles de son auteur et ne représentent pas nécessairement celles du SDIS de la Gironde.

1 0

Re: [ovirt-users] 3.5 live merge findings and mysteries [was Re: Simple way to activate live merge in FC20 cluster]
by Gianluca Cecchi 06 Jan '15

06 Jan '15

On Fri, Dec 12, 2014 at 4:32 PM, Itamar Heim <iheim(a)redhat.com> wrote: > > On 11/21/2014 09:53 AM, Gianluca Cecchi wrote: > >> So the official statement is this one at: >> http://www.ovirt.org/OVirt_3.5_Release_Notes >> >> Live Merge >> If an image has one or more snapshots, oVirt 3.5's merge command will >> combine the data of one volume into another. Live merges can be >> performed with data is pulled from one snapshot into another snapshot. >> The engine can merge multiple disks at the same time and each merge can >> independently fail or succeed in each operation. >> >> I think we should remove the part above, or at least have some of the >> developers to better clarify it. >> The feature in my opinion is very important and crucial for oVirt/RHEV >> because it is able to almost-fill the gap with VMware, especially in >> development environments, where having flexibility on snapshots >> management is very important and could be a starting point to have >> greater user base familiarize with the product and adopt it. >> >> So these are my findings and all combinations and none of them was able >> to provide live merge.... >> Could anyone tell me where I'm failing? Or correct release notes? >> >> 1) Environment with All-In-One F20 >> >> installing oVirt AIO on F20 automatically gives the virt-preview repo >> through the ovirt-3.5-dependencies.repo file, but only for libvirt* >> packages: >> >> the same server is the engine and the hypervisor >> >> [root@tekkaman qemu]# rpm -q libvirt >> libvirt-1.2.9.1-1.fc20.x86_64 >> >> [root@tekkaman qemu]# vdsClient -s 0 getVdsCaps | grep -i merge >> liveMerge = 'true' >> >> but >> [root@tekkaman qemu]# rpm -q qemu >> qemu-1.6.2-10.fc20.x86_64 >> >> So that tryng live merge it initially start but you get this in vdsm.log: >> libvirtError: unsupported configuration: active commit not supported >> with this QEMU binary >> >> >> 2) Another seaparate environment with a dedicated 3.5 engine f20 engine >> and 4 test cases tried >> >> a) ovirt node installed and put in a dedicated cluster >> latest available seems ovirt-node-iso-3.5.0.ovirt35.20140912.el6.iso >> from 3.5 rc test days >> >> At the end of oVirt Node install and activation in engine: >> [root@ovnode01 ~]# vdsClient -s 0 getVdsCaps | grep -i merge >> liveMerge = 'false' >> >> [root@ovnode01 ~]# rpm -qa libvirt* qemu* >> libvirt-0.10.2-29.el6_5.12.x86_64 >> libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64 >> libvirt-python-0.10.2-29.el6_5.12.x86_64 >> qemu-kvm-tools-0.12.1.2-2.415.el6_5.14.x86_64 >> qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >> libvirt-client-0.10.2-29.el6_5.12.x86_64 >> qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >> >> >> >> b) f20 + latest updates host installed as OS and then installed from >> webadmin in another cluster >> virt-preview on host it is not enabled so that libvirt/qemu are not ready >> >> >> [root@ovnode02 network-scripts]# vdsClient -s 0 getVdsCaps | grep -i >> merge >> liveMerge = 'false' >> >> [root@ovnode02 network-scripts]# rpm -qa libvirt* qemu* >> libvirt-daemon-1.1.3.6-2.fc20.x86_64 >> libvirt-python-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-config-nwfilter-1.1.3.6-2.fc20.x86_64 >> qemu-kvm-1.6.2-10.fc20.x86_64 >> qemu-common-1.6.2-10.fc20.x86_64 >> libvirt-client-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-network-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-nwfilter-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-interface-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-nodedev-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-secret-1.1.3.6-2.fc20.x86_64 >> qemu-system-x86-1.6.2-10.fc20.x86_64 >> libvirt-daemon-kvm-1.1.3.6-2.fc20.x86_64 >> qemu-kvm-tools-1.6.2-10.fc20.x86_64 >> qemu-img-1.6.2-10.fc20.x86_64 >> libvirt-daemon-driver-qemu-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-storage-1.1.3.6-2.fc20.x86_64 >> libvirt-lock-sanlock-1.1.3.6-2.fc20.x86_64 >> >> >> >> c) CentOS 6.6 host + latest updates installed as OS and then installed >> from webadmin in another cluster >> >> [root@ovnode03 ~]# vdsClient -s 0 getVdsCaps | grep -i merge >> liveMerge = 'false' >> >> [root@ovnode03 ~]# rpm -qa libvirt* qemu* >> libvirt-python-0.10.2-46.el6_6.2.x86_64 >> qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.14.x86_64 >> qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >> libvirt-client-0.10.2-46.el6_6.2.x86_64 >> libvirt-0.10.2-46.el6_6.2.x86_64 >> qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >> libvirt-lock-sanlock-0.10.2-46.el6_6.2.x86_64 >> >> >> >> d) CentOS 7.0 host + latest updates installed as OS and then installed >> from webadmin in another cluster >> >> [root@ovnode04 ~]# vdsClient -s 0 getVdsCaps | grep -i merge >> liveMerge = 'false' >> >> [root@ovnode04 ~]# rpm -qa qemu* libvirt* >> qemu-img-rhev-1.5.3-60.el7_0.2.x86_64 >> libvirt-daemon-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-storage-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-nodedev-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-kvm-1.1.1-29.el7_0.3.x86_64 >> qemu-kvm-tools-rhev-1.5.3-60.el7_0.2.x86_64 >> qemu-kvm-common-rhev-1.5.3-60.el7_0.2.x86_64 >> libvirt-client-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-nwfilter-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-interface-1.1.1-29.el7_0.3.x86_64 >> libvirt-lock-sanlock-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-config-nwfilter-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-network-1.1.1-29.el7_0.3.x86_64 >> qemu-kvm-rhev-1.5.3-60.el7_0.2.x86_64 >> libvirt-python-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-secret-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-qemu-1.1.1-29.el7_0.3.x86_64 >> >> >> Thanks in advance, >> Gianluca >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > was this resolved? > In my opinion it was not resolved and what written into release notes doesn't correspond to features set. See my test cases and eventually describe other test cases where live merge is usable See also other findings about live merge on active layer here: http://lists.ovirt.org/pipermail/users/2014-November/029450.html Gianluca

4 7