March 2017 - Users - oVirt List Archives

Maintenance GUI vs CLI
by Marcin Kruk 14 Mar '17

14 Mar '17

What is the difference between: host CLI command: hosted-engine --vm-maintenance --mode=local and RHVEM gui click action: Hosts -> <hostname> -> Maintenance?

2 3

Serious Trouble - Lost Domains
by JC Clark 14 Mar '17

14 Mar '17

Dear Fellows and Fellettes, I am having serious disaster problems. After a power transformer outside the building literally exploded, surged and fried the motherboard to the main SPM domain computer in my Ovirt 4.0 system. After getting the computer working again with a new mother board. I managed to basically loose integrity of the old engine. I have rebuilt the engine. I have backups of the old engine and 4 data domains an ISO domain and an export domain which appear to not be damaged. They are all accessible from the CL. I had to create a new host and SPM. How do I get the floating domains into the new Engine? I have 1 storage container (85GB) I must get back. MUST GET BACK!! I really appreciate you reading my sob story. Hope you can help.. Thank You much

2 1

idempiere
by Logan Kuhn 14 Mar '17

14 Mar '17

We have 4 idempiere-* VMs in an older cluster and we can search for VMs based on all sorts of strings, but if it contains idempiere it shows: 2017-03-13 11:27:33,268 INFO [org.ovirt.engine.core.bll.SearchQuery] (default task-19) [] ResourceManager::searchBusinessObjects - erroneous search text - ''Vms: idempiere'' Can anyone else duplicate this problem in their environment? Logan

3 2

Lab enviroment with vmware workstation---- Host rhev01 installation failed. Command returned failure code 1 during SSH session 'root@192.168.2.191'.
by 单延明 13 Mar '17

13 Mar '17

This is a multipart message in MIME format. ------=_NextPart_000_000F_01D29BE7.475B4480 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: quoted-printable Hi everyone, =20 I made two virtual machine with vmware workstation. The virtual machine as flollows. =20 oVirt Engine is rhevm.example.com. oVirt Node is rhev01.example.com. Host rhev01 installation failed. Command returned failure code 1 during = SSH session 'root(a)192.168.2.191'. =20 =20 shan yanming =B5=A5=D1=D3=C3=F7 =20 ################################### =D6=D0=B9=FA =BA=DA=C1=FA=BD=AD=A3=AC=B4=F3=C7=EC=CA=D0 =B4=F3=C7=EC=D3=CD=CC=EF=D3=D0=CF=DE=D4=F0=C8=CE=B9=AB=CB=BE=BF=B1=CC=BD=BF= =AA=B7=A2=D1=D0=BE=BF=D4=BA =D3=A6=D3=C3=C8=ED=BC=FE=D1=D0=BE=BF=CA=D2 =20 =D3=CA=B1=E0=A3=BA163712 =CA=D6=BB=FA=A3=BA13945919499 =B0=EC=B9=AB=A3=BA04595093871 ################################### =20 =20 ------=_NextPart_000_000F_01D29BE7.475B4480 Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; charset=3Dgb2312"><meta = name=3DGenerator content=3D"Microsoft Word 14 (filtered = medium)"><style></style></head><body lang=3DZH-CN link=3Dblue = vlink=3Dpurple style=3D'text-justify-trim:punctuation'><div = class=3DWordSection1><p class=3DMsoNormal><span lang=3DEN-US>Hi = everyone,<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US><o:p> </o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US>I made two virtual machine with vmware = workstation.<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US>The virtual machine as flollows.<o:p></o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p><p = class=3DMsoNormal align=3Dleft = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-align:le= ft'><span lang=3DEN-US>oVirt Engine is = rhevm.example.com.<i><o:p></o:p></i></span></p><p class=3DMsoNormal = align=3Dleft = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-align:le= ft'><i><span lang=3DEN-US>oVirt Node is = rhev01.example.com.<o:p></o:p></span></i></p><p class=3DMsoNormal><span = lang=3DEN-US style=3D'color:red'>Host rhev01 installation failed. = Command returned failure code 1 during SSH session = 'root(a)192.168.2.191'.<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US style=3D'color:red'><o:p> </o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '>shan yanming<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:14.0pt;font-family:=CB=CE=CC=E5;color:#1F497D'>=B5=A5=D1= =D3=C3=F7</span><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '><o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '><o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '>###################################<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:14.0pt;font-family:=CB=CE=CC=E5;color:#1F497D'>=D6=D0=B9= =FA</span><span = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '> </span><span = style=3D'font-size:14.0pt;font-family:=CB=CE=CC=E5;color:#1F497D'>=BA=DA=C1= =FA=BD=AD=A3=AC=B4=F3=C7=EC=CA=D0</span><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '><o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:14.0pt;font-family:=CB=CE=CC=E5;color:#1F497D'>=B4=F3=C7= =EC=D3=CD=CC=EF=D3=D0=CF=DE=D4=F0=C8=CE=B9=AB=CB=BE=BF=B1=CC=BD=BF=AA=B7=A2= =D1=D0=BE=BF=D4=BA</span><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '><o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:14.0pt;font-family:=CB=CE=CC=E5;color:#1F497D'>=D3=A6=D3= =C3=C8=ED=BC=FE=D1=D0=BE=BF=CA=D2</span><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '><o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '><o:p> </o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:14.0pt;font-family:=CB=CE=CC=E5;color:#1F497D'>=D3=CA=B1= =E0=A3=BA</span><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '>163712<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:14.0pt;font-family:=CB=CE=CC=E5;color:#1F497D'>=CA=D6=BB= =FA=A3=BA</span><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '>13945919499<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:14.0pt;font-family:=CB=CE=CC=E5;color:#1F497D'>=B0=EC=B9= =AB=A3=BA</span><span lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '>04595093871<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'font-size:14.0pt;font-family:"Tahoma","sans-serif";color:#1F497D= '>###################################<o:p></o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p><p = class=3DMsoNormal><span = lang=3DEN-US><o:p> </o:p></span></p></div></body></html> ------=_NextPart_000_000F_01D29BE7.475B4480--

2 1

Info on network interface threshold alert
by Gianluca Cecchi 13 Mar '17

13 Mar '17

Hello, this morning I get these two messages regarding bond1 and its active interface on one host (ovmsrv07): Host ovmsrv07 has network interface which exceeded the defined threshold [95%] (bond1: transmit rate[96%], receive rate [0%]) Host ovmsrv07 has network interface which exceeded the defined threshold [95%] (enp7s4f0: transmit rate[96%], receive rate [0%]) Actually there is no particular network activity during this latest night... There have been some different resore jobs during past days... How is determined this message? Is it perhaps cumulative? On the host there are 3 VMs and only one of them actively using the network. In its "Network Interfaces subtab I see Total RX (bytes) 542,696,952,529 Inside VM OS: $ uptime 10:26:50 up 14 days, 1:35, 3 users, load average: 0.00, 0.04, 0.05 $ ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:1A:4A:17:01:5B inet addr:10.4.4.66 Bcast:10.4.4.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:40280066 errors:0 dropped:612718 overruns:0 frame:0 TX packets:21110417 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:193066319488 (179.8 GiB) TX bytes:20983189756 (19.5 GiB) Almost confirmaed on host where I have these counter indeed, but I think they are cumulative for the VM, and not an alert in se.... ON ovmsrv07: $ ifconfig bond1 bond1: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 9000 ether 00:1c:c4:ab:3a:dd txqueuelen 1000 (Ethernet) RX packets 140610596 bytes 200368822262 (186.6 GiB) RX errors 0 dropped 126874 overruns 0 frame 0 TX packets 33886066 bytes 21972025081 (20.4 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 $ cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: enp7s4f0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: enp7s4f0 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:1c:c4:ab:3a:dd Slave queue ID: 0 Slave Interface: enp7s4f1 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:1c:c4:ab:3a:de Slave queue ID: 0 Could anyone clarify? Thanks, Gianluca

2 1

Re: [ovirt-users] Extend disk in engine appliance in ovirt 4.1.0?
by Simone Tiraboschi 13 Mar '17

13 Mar '17

On Mon, Mar 13, 2017 at 11:55 AM, gflwqs gflwqs <gflwqs(a)gmail.com> wrote: > [root@engine ~]# rpm -q cloud-utils-growpart > package cloud-utils-growpart is not installed > I opened a bug to track it: https://bugzilla.redhat.com/show_bug.cgi?id=1431624 thanks for pointing it out. now in your case the quick option to get if fixed is: - choose the host where the engine VM is running - set global maintenance mode with hosted-engine --set-maintenance --mode=global - connect to the engine VM and run fdisk /dev/vda - press d to and delete the root partition - press n to add a new one, ensure it starts on the same sector where the original partition was starting (2048) - press w to write it - shut down the engine VM - come back to the host and restart the engine VM with hosted-engine --vm-start - connect again to the engine VM and run resize2fs /dev/vda1 to resize the FS - come back to the host and exit the global maintenance mode with hosted-engine --set-maintenance --mode=none Before this, please use engine-backup on the engine VM to create a backup of your engine. > From the engine-setup log: > 2017-02-14 17:08:17 INFO otopi.plugins.gr_he_common.vm.boot_disk > boot_disk._setup:509 Detecting available oVirt engine appliances > 2017-02-14 17:08:17 DEBUG otopi.plugins.gr_he_common.vm.boot_disk > boot_disk._detect_appliances:253 parsing: /etc/ovirt-hosted-engine/10- > appliance.conf > 2017-02-14 17:08:17 DEBUG otopi.plugins.gr_he_common.vm.boot_disk > boot_disk._detect_appliances:269 available appliances: [{'path': > u'/usr/share/ovirt-engine-appliance/ovirt-engine- > appliance-4.1-20170201.1.el7.centos.ova', 'version': > u'4.1-20170201.1.el7.centos', 'index': '1', 'description': u'The oVirt > Engine Appliance image (OVA)', 'sha1sum': u'87aef3093c93896ab1c40adf112286 > af385247df'}] > > > 2017-03-13 11:43 GMT+01:00 Simone Tiraboschi <stirabos(a)redhat.com>: > >> >> >> On Mon, Mar 13, 2017 at 11:31 AM, gflwqs gflwqs <gflwqs(a)gmail.com> wrote: >> >>> Hi Simone, >>> Since there is a lot of stuff in the setup log that i don't want to send >>> to the internet, are there anything specific that i could pick out from the >>> log? >>> >>> Here is the lsbl and fdisk output from the engine: >>> >>> [root@engine ~]# lsblk >>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT >>> sr0 11:0 1 1024M 0 rom >>> vda 253:0 0 90G 0 disk >>> └─vda1 253:1 0 10G 0 part / >>> >>> >> OK, so the disk is 90 GB but the partition is just 10 GB. >> >> Could you please check if you have cloud-utils-growpart package inside >> your engine VM? >> If not, could you please report the version of ovirt-engine-appliance you >> used at setup time? >> >> >>> [root@engine ~]# fdisk -l /dev/vda >>> >>> Disk /dev/vda: 96.6 GB, 96636764160 bytes, 188743680 sectors >>> Units = sectors of 1 * 512 = 512 bytes >>> Sector size (logical/physical): 512 bytes / 512 bytes >>> I/O size (minimum/optimal): 512 bytes / 512 bytes >>> Disk label type: dos >>> Disk identifier: 0x0009e1bc >>> >>> Device Boot Start End Blocks Id System >>> /dev/vda1 * 2048 20971519 10484736 83 Linux >>> >>> 2017-03-13 10:44 GMT+01:00 Simone Tiraboschi <stirabos(a)redhat.com>: >>> >>>> >>>> >>>> On Mon, Mar 13, 2017 at 8:26 AM, gflwqs gflwqs <gflwqs(a)gmail.com> >>>> wrote: >>>> >>>>> Hi, I have recently installed a new ovirt environment from scratch. >>>>> The version is 4.1.0. >>>>> I made a 90GB disk for the hosted engine. >>>>> When i had installed it and log into the engine i see that it has only >>>>> configured 10GB of those 90GB. >>>>> So there is 80GB of unconfigured disk unused. >>>>> >>>> >>>> Can you please attach your hosted-engine-setup log file? >>>> >>>> Could you please check if the disk is 90GB but the main disk partition >>>> is just 10 GB? >>>> >>>> cloud-utils-growpart should grow the partition on the first boot, if >>>> the issue is there we need to understand why it didn't triggered in your >>>> case. >>>> >>>> >>>>> That is fine as long as it had used lvm but it has not meaning i have >>>>> to take the engine down to extend the disk? >>>>> >>>>> My questions is: >>>>> 1. Why does it not configure all disk? >>>>> 2. Why is it not using lvm? >>>>> 3. What should i do to extend the disk? >>>>> >>>> >>>> It depends on where the issue is (is on the disk size or just on the >>>> partition size?). >>>> >>>> >>>>> >>>>> Regards >>>>> Christian >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>> >>> >> >

1 0

Removing all the Users Permission on Data Center
by Manuel Luis Aznar 13 Mar '17

13 Mar '17

Hello there, I need to remove the specific Users Permission a user have in a DataCenter. I manage to do it in this way: username = "..." dc = "..." system_service = connection.system_service() users_service = system_service.users_service() data_centers_service = system_service.data_centers_service() # Getting the User object for user in users_service.list(): if username in user.user_name: break # Getting the DataCenter object for data_center in data_centers_service.list(): if dc in data_center.name: break # Getting DataCenter service and its Permission Service data_center_service = data_centers_service.data_center_service(id= data_center.id) data_center_permissions_service = data_center_service.permissions_service() # Getting the Permission for the User in the DataCenter for data_center_permission in data_center_permissions_service.list(): data_center_permission_user = data_center_permission.user if data_center_permission_user: if data_center_permission_user.id == user.id: data_center_permission_user.remove() As you can see I can do it, but, I am using the permissions of the DataCenter and this list could be very long. In the old ovirtsdk (version 3) this was done by the following: u = kvm.users.get(id=userid) for perm in u.permissions.list(): udc = perm.get_data_center() if udc: globaldc = kvm.datacenters.get(id=udc.get_id()) if globaldc.get_name() == dc: perm.delete() that last piece of code iterate by the user permission list and delete the specific Data Center permission. I have been trying this doing the following: user_service = users_service.user_service(id=user.id) user_permissions_service = user_service.permissions_service() list = user_permissions_service.list() Is that last variable, list: the permissions list for the specified user, I ask that because if I print the size of the list for an specific user, the number I get is not correct... Thanks for all in advance to all Manuel

2 3

Re: [ovirt-users] Contributing on ovirt
by shubham dubey 13 Mar '17

13 Mar '17

Hello, I hope you got my irc message that I got my exams earlier(which are now over) because of which I was not able to spent much time on project related things. since I am free now,I have read about ovirt architecture and documentation to get more familiar with ovirt. Now the purpose of the mail is that I am having some issue while installing ovirt from source.( http://www.ovirt.org/develop/developer-guide/engine/engine-development-envi… ) when I am trying to do $ make install-dev PREFIX="$HOME/ovirt-engine" .. .. .. [ERROR] Failed to execute goal org.codehaus.mojo:gwt-maven-plugin:2.6.1:compile (gwtcompile) on project webadmin: Command [[ [ERROR] /bin/sh -c /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-8.b14.fc25.x86_64/jre/bin/java -javaagent:/home/0day/.m2/repository/org/aspectj/aspectjweaver/1.8.2/aspectjweaver-1.8.2.jar -Dgwt.jjs.permutationWorkerFactory=com.google.gwt.dev.ThreadedPermutationWorkerFactory \ [ERROR] -Dgwt.jjs.maxThreads=4 \ [ERROR] -Djava.io.tmpdir="/home/0day/git/ovirt-engine/frontend/webadmin/modules/webadmin/target/tmp" \ [ERROR] -Djava.util.prefs.systemRoot="/home/0day/git/ovirt-engine/frontend/webadmin/modules/webadmin/target/tmp" \ [ERROR] -Djava.util.prefs.userRoot="/home/0day/git/ovirt-engine/frontend/webadmin/modules/webadmin/target/tmp" \ [ERROR] -Djava.util.logging.config.class=org.ovirt.engine.ui.gwtextension.JavaLoggingConfig \ [ERROR] -Xms1024M -Xmx8192M '-Dgwt.dontPrune=org\.ovirt\.engine\.core\.(common|compat)\..*' -classpath /home/0day/git/ovirt-engine/frontend/webadmin/modules/webadmin/target/webadmin-4.2.0-SNAPSHOT/WEB-INF/classes:/home/0day/git/ovirt-engine/frontend/webadmin/modules/webadmin/src/main/java:/home/0day/git/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/annotations:/home/0day/git/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt:/home/0day/.m2/repository/com/gwtplatform/gwtp-mvp-client/1.3.1/gwtp-mvp-client-1.3.1.jar:/home/0day/.m2/repository/com/gwtplatform/gwtp-clients-common/1.3.1/gwtp-clients-common-1.3.1.jar:/home/0day/.m2/repository/com/gwtplatform/gwtp-mvp-shared/1.3.1/gwtp-mvp-shared-1.3.1.jar:/home/0day/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar:/home/0day/.m2/repository/com/gwtplatform/gwtp-processors/1.3.1/gwtp-processors-1.3.1.jar:/home/0day/.m2/repository/com/google/inject/guice/3.0/guice-3.0.jar:/home/0day/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/home/0day/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/home/0day/.m2/repository/com/google/inject/extensions/guice-assistedinject/3.0/guice-assistedinject-3.0.jar:/home/0day/.m2/repository/com/google/gwt/inject/gin/2.1.2/gin-2.1.2.jar:/home/0day/.m2/repository/com/google/gwt/gwt-user/2.6.1/gwt-user-2.6.1.jar:/home/0day/.m2/repository/javax/validation/validation-api/1.0.0.GA/validation-api-1.0.0.GA.jar:/home/0day/.m2/repository/javax/validation/validation-api/1.0.0.GA/validation-api-1.0.0.GA-sources.jar:/home/0day/.m2/repository/org/json/json/20090211/json-20090211.jar:/home/0day/.m2/repository/com/google/gwt/gwt-dev/2.6.1/gwt-dev-2.6.1.jar:/home/0day/.m2/repository/org/gwtbootstrap3/gwtbootstrap3/0.9.3/gwtbootstrap3-0.9.3.jar:/home/0day/.m2/repository/org/gwtbootstrap3/gwtbootstrap3-extras/0.9.3/gwtbootstrap3-extras-0.9.3.jar:/home/0day/git/ovirt-engine/frontend/webadmin/modules/uicommonweb/target/uicommonweb-4.2.0-SNAPSHOT.jar:/home/0day/git/ovirt-engine/backend/manager/modules/searchbackend/target/searchbackend-4.2.0-SNAPSHOT.jar:/home/0day/git/ovirt-engine/frontend/webadmin/modules/uicompat/target/uicompat-4.2.0-SNAPSHOT.jar:/home/0day/git/ovirt-engine/frontend/webadmin/modules/gwt-common/target/gwt-common-4.2.0-SNAPSHOT.jar:/home/0day/git/ovirt-engine/backend/manager/modules/common/target/common-4.2.0-SNAPSHOT.jar:/home/0day/.m2/repository/org/hibernate/hibernate-validator/4.2.0.Final/hibernate-validator-4.2.0.Final.jar:/home/0day/.m2/repository/org/jboss/modules/jboss-modules/1.1.1.GA/jboss-modules-1.1.1.GA.jar:/home/0day/git/ovirt-engine/backend/manager/modules/common/target/common-4.2.0-SNAPSHOT-sources.jar:/home/0day/git/ovirt-engine/backend/manager/modules/compat/target/compat-4.2.0-SNAPSHOT-sources.jar:/home/0day/git/ovirt-engine/backend/manager/modules/searchbackend/target/searchbackend-4.2.0-SNAPSHOT-sources.jar:/home/0day/git/ovirt-engine/frontend/webadmin/modules/frontend/target/frontend-4.2.0-SNAPSHOT.jar:/home/0day/git/ovirt-engine/frontend/webadmin/modules/gwt-extension/target/ovirt-engine-gwt-extension-4.2.0-SNAPSHOT.jar:/home/0day/.m2/repository/javax/xml/bind/jaxb-api/2.1/jaxb-api-2.1.jar:/home/0day/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/home/0day/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/home/0day/.m2/repository/org/jboss/spec/javax/servlet/jboss-servlet-api_3.1_spec/1.0.0.Final/jboss-servlet-api_3.1_spec-1.0.0.Final.jar:/home/0day/.m2/repository/org/jboss/spec/javax/servlet/jstl/jboss-jstl-api_1.2_spec/1.0.3.Final/jboss-jstl-api_1.2_spec-1.0.3.Final.jar:/home/0day/.m2/repository/org/jboss/spec/javax/el/jboss-el-api_2.2_spec/1.0.1.Final/jboss-el-api_2.2_spec-1.0.1.Final.jar:/home/0day/.m2/repository/org/jboss/spec/javax/servlet/jboss-servlet-api_3.0_spec/1.0.1.Final/jboss-servlet-api_3.0_spec-1.0.1.Final.jar:/home/0day/.m2/repository/org/jboss/spec/javax/servlet/jsp/jboss-jsp-api_2.2_spec/1.0.1.Final/jboss-jsp-api_2.2_spec-1.0.1.Final.jar:/home/0day/.m2/repository/xalan/xalan/2.7.1.jbossorg-2/xalan-2.7.1.jbossorg-2.jar:/home/0day/.m2/repository/xalan/serializer/2.7.1.jbossorg-2/serializer-2.7.1.jbossorg-2.jar:/home/0day/.m2/repository/org/jboss/spec/javax/ejb/jboss-ejb-api_3.1_spec/1.0.2.Final/jboss-ejb-api_3.1_spec-1.0.2.Final.jar:/home/0day/.m2/repository/org/jboss/spec/javax/enterprise/concurrent/jboss-concurrency-api_1.0_spec/1.0.0.Final/jboss-concurrency-api_1.0_spec-1.0.0.Final.jar:/home/0day/.m2/repository/org/aspectj/aspectjweaver/1.8.2/aspectjweaver-1.8.2.jar:/home/0day/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar:/home/0day/git/ovirt-engine/backend/manager/modules/utils/target/utils-4.2.0-SNAPSHOT.jar:/home/0day/git/ovirt-engine/backend/manager/modules/extensions-api-root/extensions-api/target/ovirt-engine-extensions-api-0.0.0-SNAPSHOT.jar:/home/0day/git/ovirt-engine/backend/manager/modules/extensions-manager/target/extensions-manager-4.2.0-SNAPSHOT.jar:/home/0day/.m2/repository/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar:/home/0day/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/home/0day/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/home/0day/git/ovirt-engine/backend/manager/modules/compat/target/compat-4.2.0-SNAPSHOT.jar:/home/0day/.m2/repository/org/ovirt/otopi/otopi/1.5.2/otopi-1.5.2.jar:/home/0day/.m2/repository/org/ovirt/ovirt-host-deploy/ovirt-host-deploy/1.5.0/ovirt-host-deploy-1.5.0.jar:/home/0day/.m2/repository/org/apache/commons/commons-compress/1.5/commons-compress-1.5.jar:/home/0day/.m2/repository/org/tukaani/xz/1.2/xz-1.2.jar:/home/0day/git/ovirt-engine/backend/manager/modules/uutils/target/uutils-4.2.0-SNAPSHOT.jar:/home/0day/.m2/repository/org/apache/sshd/sshd-core/0.12.0/sshd-core-0.12.0.jar:/home/0day/.m2/repository/org/apache/mina/mina-core/2.0.7/mina-core-2.0.7.jar:/home/0day/.m2/repository/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar:/home/0day/.m2/repository/org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar:/home/0day/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/0day/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar:/home/0day/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/0day/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.9/jackson-core-asl-1.9.9.jar:/home/0day/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.9/jackson-mapper-asl-1.9.9.jar:/home/0day/.m2/repository/org/springframework/spring-core/4.2.4.RELEASE/spring-core-4.2.4.RELEASE.jar:/home/0day/.m2/repository/org/infinispan/infinispan-core/5.2.5.Final/infinispan-core-5.2.5.Final.jar:/home/0day/.m2/repository/org/jgroups/jgroups/3.2.7.Final/jgroups-3.2.7.Final.jar:/home/0day/.m2/repository/org/jboss/spec/javax/transaction/jboss-transaction-api_1.1_spec/1.0.0.Final/jboss-transaction-api_1.1_spec-1.0.0.Final.jar:/home/0day/.m2/repository/org/jboss/marshalling/jboss-marshalling-river/1.3.15.GA/jboss-marshalling-river-1.3.15.GA.jar:/home/0day/.m2/repository/org/jboss/marshalling/jboss-marshalling/1.3.15.GA/jboss-marshalling-1.3.15.GA.jar:/home/0day/.m2/repository/org/jboss/logging/jboss-logging/3.1.1.GA/jboss-logging-3.1.1.GA.jar:/home/0day/.m2/repository/org/jboss/staxmapper/1.1.0.Final/staxmapper-1.1.0.Final.jar:/home/0day/.m2/repository/com/woorea/keystone-client/3.1.2/keystone-client-3.1.2.jar:/home/0day/.m2/repository/com/woorea/openstack-client/3.1.2/openstack-client-3.1.2.jar:/home/0day/.m2/repository/com/woorea/keystone-model/3.1.2/keystone-model-3.1.2.jar:/home/0day/.m2/repository/com/woorea/glance-client/3.1.2/glance-client-3.1.2.jar:/home/0day/.m2/repository/com/woorea/glance-model/3.1.2/glance-model-3.1.2.jar:/home/0day/.m2/repository/com/woorea/cinder-client/3.1.2/cinder-client-3.1.2.jar:/home/0day/.m2/repository/com/woorea/cinder-model/3.1.2/cinder-model-3.1.2.jar:/home/0day/.m2/repository/org/ebaysf/web/cors-filter/1.0.1/cors-filter-1.0.1.jar:/home/0day/git/ovirt-engine/backend/manager/modules/branding/target/branding-4.2.0-SNAPSHOT.jar:/home/0day/.m2/repository/com/google/gwt/gwt-user/2.6.1/gwt-user-2.6.1.jar:/home/0day/.m2/repository/com/google/gwt/gwt-dev/2.6.1/gwt-dev-2.6.1.jar:/home/0day/git/ovirt-engine/frontend/webadmin/modules/gwt-extension/target/ovirt-engine-gwt-extension-4.2.0-SNAPSHOT-sources.jar:/home/0day/git/ovirt-engine/frontend/webadmin/modules/uicommonweb/target/uicommonweb-4.2.0-SNAPSHOT-sources.jar com.google.gwt.dev.Compiler -logLevel INFO -style OBF -war /home/0day/git/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-gwt -localWorkers 4 -failOnError -XfragmentCount -1 -sourceLevel auto -gen /home/0day/git/ovirt-engine/frontend/webadmin/modules/webadmin/gen org.ovirt.engine.ui.webadmin.WebAdmin [ERROR] ]] failed with status 1 [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :webadmin Makefile:257: recipe for target 'maven' failed make[2]: *** [maven] Error 1 make[2]: Leaving directory '/home/0day/git/ovirt-engine' Makefile:264: recipe for target 'tmp.built' failed make[1]: *** [tmp.built] Error 2 make[1]: Leaving directory '/home/0day/git/ovirt-engine' Makefile:501: recipe for target 'all-dev' failed make: *** [all-dev] Error 2 then I try instruction given at this link to setup maven http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException It successfully run without any error but again make install fail. I have even tried $mvn initilize -X -e that also run successfully but didn't help in removing the error. So,any solution. Thanks, Shubham On Mon, Mar 6, 2017 at 12:32 PM, Maor Lipchuk <mlipchuk(a)redhat.com> wrote: > Hi shubham, > > You should not worry about the java language, the feature is focused > mainly on logic and the development should be pretty simple. > Also, your patches will be reviewed through gerrit, so I will help and > it should be very convenient to develop. > and of course, you will have the change to get to know a new language, > which should be fun :) > > Regards, > Maor > > On Mon, Mar 6, 2017 at 2:06 AM, shubham dubey <sdubey504(a)gmail.com> wrote: > > I was expecting it to be in python since major parts of ovirt are in > > python:). > > I am little less familiar with java then I am with python.So,did even > > moderate control on java language will work for project or did I try for > > some other ideas? > > Although I have enough experience in Storage technology. > > > > Thanks, > > Shubham > > > > On Mon, Mar 6, 2017 at 5:24 AM, Maor Lipchuk <mlipchuk(a)redhat.com> > wrote: > >> > >> Hi Shubham, > >> > >> I just saw your message in the oVirt channel, sorry for the late respond > >> The new backup storage should be developed mainly in Java, it will be > >> part of the ovirt-engine > >> > >> On Sun, Mar 5, 2017 at 10:47 AM, Maor Lipchuk <mlipchuk(a)redhat.com> > wrote: > >> > Hi shubham, > >> > > >> > Thank you for your interest in the Configuring the backup storage as > >> > part of the GOSC project. > >> > The idea is to have an alternative for backing up entities like VMs, > >> > Templates and disks from a storage domain. > >> > > >> > oVirt has many methods of backup entities such as DB backup, import > >> > storage domain and use of export storage domain. > >> > With this new capability to define a storage domain as a backup it > >> > might help oVirt to achieve an easier alternative solution for the > >> > users to backup entities > >> > > >> > I suggest that for start, try to build a working oVirt development > >> > setup with hosts and a few storage domains. > >> > > >> > You can follow this wiki: > >> > > >> > http://www.ovirt.org/develop/developer-guide/engine/engine- > development-environment/ > >> > > >> > Once you have a working oVirt env we can go forward to the next step. > >> > Please don't hesitate to ask if you have any questions. > >> > You can use the #ovirt channel also. > >> > > >> > Regards, > >> > Maor > >> > > >> > On Fri, Mar 3, 2017 at 11:27 AM, shubham dubey <sdubey504(a)gmail.com> > >> > wrote: > >> >> Hello, > >> >> I am interested in being part of ovirt for gsoc 2017. > >> >> I have looked into ovirt project ideas and the project that I find > most > >> >> suitable is Configuring the backup storage in ovirt. > >> >> > >> >> Since the ovirt online docs have sufficient info for getting started > >> >> for > >> >> development so I don't have a questions about that but I want to > >> >> clarify one > >> >> doubt that did the previous year mentioned project on ovirt gsoc page > >> >> are > >> >> also available to work on? > >> >> I will also appreciate any discussion about the project or question > >> >> from > >> >> mentor side.Even some guideline for start working is welcome. > >> >> > >> >> Thanks, > >> >> Shubham > >> >> > >> >> _______________________________________________ > >> >> Users mailing list > >> >> Users(a)ovirt.org > >> >> http://lists.ovirt.org/mailman/listinfo/users > >> >> > > > > >

1 0

migrate to hosted engine
by Devin A. Bougie 13 Mar '17

13 Mar '17

Hi, All. We have an ovirt 4.1 cluster setup using multiple paths to a single iSCSI LUN for the data storage domain. I would now like to migrate to a hosted engine. I setup the new engine VM, shutdown and backed-up the old VM, and restored to the new VM using engine-backup. After updating DNS to change our engine's FQDN to point to the hosted engine, everything seems to work properly. However, when rebooting the entire cluster, the engine VM doesn't come up automatically. Is there anything that now needs to be done to tell the cluster that it's now using a hosted engine? I started with a standard engine setup, as I didn't see a way to specify multiple paths to a single iSCSI LUN when using "hosted-engine --deploy." Any tips would be greatly appreciated. Many thanks, Devin

3 2

Re: [ovirt-users] Extend disk in engine appliance in ovirt 4.1.0?
by Simone Tiraboschi 13 Mar '17

13 Mar '17

On Mon, Mar 13, 2017 at 11:31 AM, gflwqs gflwqs <gflwqs(a)gmail.com> wrote: > Hi Simone, > Since there is a lot of stuff in the setup log that i don't want to send > to the internet, are there anything specific that i could pick out from the > log? > > Here is the lsbl and fdisk output from the engine: > > [root@engine ~]# lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sr0 11:0 1 1024M 0 rom > vda 253:0 0 90G 0 disk > └─vda1 253:1 0 10G 0 part / > > OK, so the disk is 90 GB but the partition is just 10 GB. Could you please check if you have cloud-utils-growpart package inside your engine VM? If not, could you please report the version of ovirt-engine-appliance you used at setup time? > [root@engine ~]# fdisk -l /dev/vda > > Disk /dev/vda: 96.6 GB, 96636764160 bytes, 188743680 sectors > Units = sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disk label type: dos > Disk identifier: 0x0009e1bc > > Device Boot Start End Blocks Id System > /dev/vda1 * 2048 20971519 10484736 83 Linux > > 2017-03-13 10:44 GMT+01:00 Simone Tiraboschi <stirabos(a)redhat.com>: > >> >> >> On Mon, Mar 13, 2017 at 8:26 AM, gflwqs gflwqs <gflwqs(a)gmail.com> wrote: >> >>> Hi, I have recently installed a new ovirt environment from scratch. >>> The version is 4.1.0. >>> I made a 90GB disk for the hosted engine. >>> When i had installed it and log into the engine i see that it has only >>> configured 10GB of those 90GB. >>> So there is 80GB of unconfigured disk unused. >>> >> >> Can you please attach your hosted-engine-setup log file? >> >> Could you please check if the disk is 90GB but the main disk partition is >> just 10 GB? >> >> cloud-utils-growpart should grow the partition on the first boot, if the >> issue is there we need to understand why it didn't triggered in your case. >> >> >>> That is fine as long as it had used lvm but it has not meaning i have to >>> take the engine down to extend the disk? >>> >>> My questions is: >>> 1. Why does it not configure all disk? >>> 2. Why is it not using lvm? >>> 3. What should i do to extend the disk? >>> >> >> It depends on where the issue is (is on the disk size or just on the >> partition size?). >> >> >>> >>> Regards >>> Christian >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> >

1 0

VDSM hang
by paf1＠email.cz 13 Mar '17

13 Mar '17

This is a multi-part message in MIME format. --------------C04C9E032CD3A8F03847AFCD Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hello everybody, for production usage i'm testing ovirt with gluster. All components seems to be running fine but whenever I'm testing huge workload, then node freez. Not the main OS, but VDSM mgmt and attached services, VMs eg. *mgmt * oVirt - 4.1.0.4 centos 7.3-1611 *nodes* ( installed from ovirt image /"ovirt-node-ng-installer-ovirt-4.1-2017030804.iso" )/ OS Version:== RHEL - 7 - 3.1611.el7.centos OS Description:== oVirt Node 4.1.0 Kernel Version:== 3.10.0 - 514.10.2.el7.x86_64 KVM Version:== 2.6.0 - 28.el7_3.3.1 LIBVIRT Version:== libvirt-2.0.0-10.el7_3.5 VDSM Version:== vdsm-4.19.4-1.el7.centos SPICE Version:== 0.12.4 - 20.el7_3 GlusterFS Version:== glusterfs-3.8.9-1.el7 ( LVM thinprovisioning in replica 2 - created from ovirt GUI ) concurently running - huge import from export domain ( net workload ) - sequential write to VMs local disk ( gluster replica sequential workload ) - VMs database huge select ( random IOps ) - huge old snapshot delete ( random IOps ) In this configuration / workload is runnig one hour eg, with no exceptions , with 70-80% disk load, but in some point VDSM freez all jobs for a timeout and VM's are in "uknown" status . The whole system revitalize then automaticaly in cca 20min time frame ( except the import and snapshot deleting(rollback) ) engine.log - focus 10:39:07 time ( Failed in 'HSMGetAllTasksStatusesVDS' method ) ======== n child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete 2017-03-10 10:39:01,727+01 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (DefaultQuartzScheduler2) [759c8e1f] Command 'RemoveSnapshotSingleDiskLive' (id: 'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command id: '33df2c1e-6ce3-44fd-a39b-d111883b4c4e' type:'DestroyImage' to complete 2017-03-10 10:39:03,929+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 446d0cd3 2017-03-10 10:39:04,343+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 446d0cd3 2017-03-10 10:39:04,353+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 69ea1fda 2017-03-10 10:39:05,128+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 69ea1fda 2017-03-10 10:39:07,163+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [759c8e1f] Failed in 'HSMGetAllTasksStatusesVDS' method 2017-03-10 10:39:07,178+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [759c8e1f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM 2kvm2 command HSMGetAllTasksStatusesVDS failed: Connection timed out 2017-03-10 10:39:07,182+01 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] BaseAsyncTask::onTaskEndSuccess: Task 'f594bf69-619b-4d1b-8f6d-a9826997e478' (Parent Command 'ImportVm', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2017-03-10 10:39:07,182+01 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler2) [759c8e1f] Task with DB Task ID 'a05c7c07-9b98-4ab2-ac7b-9e70a75ba7b7' and VDSM Task ID '7c60369f-70a3-4a6a-80c9-4753ac9ed372' is in state Polling. End action for command 8deb3fe3-4a83-4605-816c-ffdc63fd9ac1 will proceed when all the entity's tasks are completed. 2017-03-10 10:39:07,182+01 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] SPMAsyncTask::PollTask: Polling task 'f351e8f6-6dd7-49aa-bf54-650d84fc6352' (Parent Command 'DestroyImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'. 2017-03-10 10:39:07,182+01 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] BaseAsyncTask::logEndTaskFailure: Task 'f351e8f6-6dd7-49aa-bf54-650d84fc6352' (Parent Command 'DestroyImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Connection timed out, code = 100', -- Exception: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Connection timed out, code = 100' 2017-03-10 10:39:07,184+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] CommandAsyncTask::endActionIfNecessary: All tasks of command '33df2c1e-6ce3-44fd-a39b-d111883b4c4e' has ended -> executing 'endAction' 2017-03-10 10:39:07,185+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: '33df2c1e-6ce3-44fd-a39b-d111883b4c4e'): calling endAction '. 2017-03-10 10:39:07,185+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'DestroyImage', 2017-03-10 10:39:07,192+01 INFO [org.ovirt.engine.core.bll.storage.disk.image.DestroyImageCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] Command [id=33df2c1e-6ce3-44fd-a39b-d111883b4c4e]: Updating status to 'FAILED', The command end method logic will be executed by one of its parent commands. 2017-03-10 10:39:07,192+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'DestroyImage' completed, handling the result. 2017-03-10 10:39:07,192+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'DestroyImage' succeeded, clearing tasks. 2017-03-10 10:39:07,192+01 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'f351e8f6-6dd7-49aa-bf54-650d84fc6352' 2017-03-10 10:39:07,193+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', taskId='f351e8f6-6dd7-49aa-bf54-650d84fc6352'}), log id: 2b7080c2 2017-03-10 10:39:07,194+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] START, HSMClearTaskVDSCommand(HostName = 2kvm2, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='905375e1-6de4-4fdf-b69c-b2d546f869c8', taskId='f351e8f6-6dd7-49aa-bf54-650d84fc6352'}), log id: 2edff460 2017-03-10 10:39:08,208+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] FINISH, HSMClearTaskVDSCommand, log id: 2edff460 2017-03-10 10:39:08,208+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] FINISH, SPMClearTaskVDSCommand, log id: 2b7080c2 2017-03-10 10:39:08,213+01 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] BaseAsyncTask::removeTaskFromDB: Removed task 'f351e8f6-6dd7-49aa-bf54-650d84fc6352' from DataBase 2017-03-10 10:39:08,213+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '33df2c1e-6ce3-44fd-a39b-d111883b4c4e' 2017-03-10 10:39:10,142+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 2e7278cb 2017-03-10 10:39:11,513+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 2e7278cb 2017-03-10 10:39:11,523+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 43704ef2 2017-03-10 10:39:11,777+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler9) [67e1d8ed] Command 'RemoveSnapshot' (id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete 2017-03-10 10:39:11,789+01 WARN [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [759c8e1f] Child command 'DESTROY_IMAGE' failed, proceeding to verify 2017-03-10 10:39:11,789+01 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [759c8e1f] Executing Live Merge command step 'DESTROY_IMAGE_CHECK' 2017-03-10 10:39:11,832+01 INFO [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (pool-5-thread-7) [4856f570] Running command: DestroyImageCheckCommand internal: true. 2017-03-10 10:39:11,833+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand] (pool-5-thread-7) [4856f570] START, SPMGetVolumeInfoVDSCommand( SPMGetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]', runAsync='true', storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', storageDomainId='1603cd90-92ef-4c03-922c-cecb282fd00e', imageGroupId='7543338a-3ca6-4698-bb50-c14f0bd71428', imageId='50b592f7-bfba-4398-879c-8d6a19a2c000'}), log id: 2c8031f8 2017-03-10 10:39:11,833+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand] (pool-5-thread-7) [4856f570] Executing GetVolumeInfo using the current SPM 2017-03-10 10:39:11,834+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (pool-5-thread-7) [4856f570] START, GetVolumeInfoVDSCommand(HostName = 2kvm2, GetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]', runAsync='true', hostId='905375e1-6de4-4fdf-b69c-b2d546f869c8', storagePoolId='00000001-0001-0001-0001-000000000311', storageDomainId='1603cd90-92ef-4c03-922c-cecb282fd00e', imageGroupId='7543338a-3ca6-4698-bb50-c14f0bd71428', imageId='50b592f7-bfba-4398-879c-8d6a19a2c000'}), log id: 79ca86cc 2017-03-10 10:39:11,846+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 43704ef2 2017-03-10 10:39:16,858+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 6542adcd 2017-03-10 10:39:17,394+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 6542adcd 2017-03-10 10:39:17,406+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 44ec33ed 2017-03-10 10:39:18,598+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 44ec33ed 2017-03-10 10:39:21,865+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot' (id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete 2017-03-10 10:39:21,881+01 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (DefaultQuartzScheduler6) [4856f570] Command 'RemoveSnapshotSingleDiskLive' (id: 'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command id: 'b1d63b8e-19d3-4d64-8fa8-4eb3e2d1a8fc' type:'DestroyImageCheck' to complete 2017-03-10 10:39:23,611+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler6) [4856f570] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 4c2fc22d 2017-03-10 10:39:24,616+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] START, GlusterTasksListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 1f169371 2017-03-10 10:39:24,618+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler6) [4856f570] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 4c2fc22d 2017-03-10 10:39:24,629+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler6) [4856f570] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 2ac55735 2017-03-10 10:39:24,822+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] FINISH, GlusterTasksListVDSCommand, return: [], log id: 1f169371 2017-03-10 10:39:26,836+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler6) [4856f570] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 2ac55735 2017-03-10 10:39:31,849+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 2e8dbcd1 2017-03-10 10:39:31,932+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot' (id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete 2017-03-10 10:39:31,944+01 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (DefaultQuartzScheduler6) [4856f570] Command 'RemoveSnapshotSingleDiskLive' (id: 'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command id: 'b1d63b8e-19d3-4d64-8fa8-4eb3e2d1a8fc' type:'DestroyImageCheck' to complete 2017-03-10 10:39:33,213+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 2e8dbcd1 2017-03-10 10:39:33,226+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 1fb3f9e3 2017-03-10 10:39:34,375+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 1fb3f9e3 2017-03-10 10:39:39,392+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 1e0b8eeb 2017-03-10 10:39:40,753+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 1e0b8eeb 2017-03-10 10:39:40,763+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 35b04b33 2017-03-10 10:39:41,952+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 35b04b33 2017-03-10 10:39:41,991+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot' (id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete gluster ( nothing in logs ) ====== ## "etc-glusterfs-glusterd.vol.log" [2017-03-10 10:13:52.599019] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:16:48.639635] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 The message "I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1" repeated 3 times between [2017-03-10 10:16:48.639635] and [2017-03-10 10:17:55.659379] [2017-03-10 10:18:56.875516] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:19:57.204689] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:21:56.576879] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:21:57.772857] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:24:00.617931] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:30:04.918080] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:31:06.128638] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:32:07.325672] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:32:12.433586] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:32:13.544909] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:35:10.039213] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:37:19.905314] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:37:20.174209] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:38:12.635460] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 [2017-03-10 10:40:14.169864] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1 ## "rhev-data-center-mnt-glusterSD-localhost:_slow1.log" [2017-03-10 09:43:40.346785] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61849: dentry not found in 43e6968f-9c2a-40d8-8074-caf1a36f60cf [2017-03-10 09:43:40.347076] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61879: dentry not found in 902a6e3d-b7aa-439f-8262-cdc1b7f9f022 [2017-03-10 09:43:40.347145] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61935: dentry not found in 846bbcfc-f2b3-4ab6-af44-aeaa10b39318 [2017-03-10 09:43:40.347211] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61922: dentry not found in 66ad3bc5-26c7-4360-b33b-a084e3305cf8 [2017-03-10 09:43:40.351571] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61834: dentry not found in 3b8278e1-40e5-4363-b21e-7bffcd024c62 [2017-03-10 09:43:40.352449] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61870: dentry not found in 282f4c05-e09a-48e0-96a3-52e079ff2f73 [2017-03-10 09:50:38.829325] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta (hash=slow1-replicate-0/cache=slow1-replicate-0) [2017-03-10 09:50:42.221775] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta (hash=slow1-replicate-0/cache=slow1-replicate-0) [2017-03-10 09:50:45.956432] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/3cef54b4-45b9-4f5b-82c2-fcc8def06a37/85287865-38f0-45df-9e6c-1294913cbb88.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/3cef54b4-45b9-4f5b-82c2-fcc8def06a37/85287865-38f0-45df-9e6c-1294913cbb88.meta (hash=slow1-replicate-0/cache=slow1-replicate-0) [2017-03-10 09:50:40.349563] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta (hash=slow1-replicate-0/cache=slow1-replicate-0) [2017-03-10 09:50:44.503866] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta (hash=slow1-replicate-0/cache=slow1-replicate-0) [2017-03-10 09:59:46.860762] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.6684: dentry not found in d1e65eea-8758-4407-ac2e-3605dc661364 [2017-03-10 10:02:22.500865] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.8767: dentry not found in e228bb28-9602-4f8e-8323-7434d77849fc [2017-03-10 10:04:03.103839] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.9787: dentry not found in 6be71632-aa36-4975-b673-1357e0355027 [2017-03-10 10:06:02.406385] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/2a9c1c6a-f045-4dce-a47b-95a2267eef72/6f264695-0669-4b49-a2f6-e6c92482f2fb.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/2a9c1c6a-f045-4dce-a47b-95a2267eef72/6f264695-0669-4b49-a2f6-e6c92482f2fb.meta (hash=slow1-replicate-0/cache=slow1-replicate-0) ... no other record messages ======== several times occured: Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd. Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd. Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd. Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd. Mar 10 09:10:01 2kvm2 systemd: Started Session 274 of user root. Mar 10 09:10:01 2kvm2 systemd: Starting Session 274 of user root. Mar 10 09:20:02 2kvm2 systemd: Started Session 275 of user root. Mar 10 09:20:02 2kvm2 systemd: Starting Session 275 of user root. Mar 10 09:22:59 2kvm2 sanlock[1673]: 2017-03-10 09:22:59+0100 136031 [2576]: s3 delta_renew long write time 11 sec Mar 10 09:24:03 2kvm2 kernel: kswapd1: page allocation failure: order:2, mode:0x104020 Mar 10 09:24:03 2kvm2 kernel: CPU: 42 PID: 265 Comm: kswapd1 Tainted: G I ------------ 3.10.0-514.10.2.el7.x86_64 #1 Mar 10 09:24:03 2kvm2 kernel: Hardware name: Supermicro X10DRC/X10DRi-LN4+, BIOS 1.0a 08/29/2014 Mar 10 09:24:03 2kvm2 kernel: 0000000000104020 00000000f7228dc9 ffff88301f4839d8 ffffffff816864ef Mar 10 09:24:03 2kvm2 kernel: ffff88301f483a68 ffffffff81186ba0 000068fc00000000 0000000000000000 Mar 10 09:24:03 2kvm2 kernel: fffffffffffffffc 0010402000000000 ffff88301567ae80 00000000f7228dc9 Mar 10 09:24:03 2kvm2 kernel: Call Trace: Mar 10 09:24:03 2kvm2 kernel: <IRQ> [<ffffffff816864ef>] dump_stack+0x19/0x1b Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81186ba0>] warn_alloc_failed+0x110/0x180 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81682083>] __alloc_pages_slowpath+0x6b7/0x725 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8118b155>] __alloc_pages_nodemask+0x405/0x420 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811cf30a>] alloc_pages_current+0xaa/0x170 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81185a7e>] __get_free_pages+0xe/0x50 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811dabae>] kmalloc_order_trace+0x2e/0xa0 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811dd381>] __kmalloc+0x221/0x240 Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02f83fa>] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x] Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02f92f7>] bnx2x_rx_int+0x227/0x17b0 [bnx2x] Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81033669>] ? sched_clock+0x9/0x10 Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02fc72d>] bnx2x_poll+0x1dd/0x260 [bnx2x] Mar 10 09:24:03 2kvm2 kernel: [<ffffffff815705e0>] net_rx_action+0x170/0x380 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8108f2cf>] __do_softirq+0xef/0x280 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8169859c>] call_softirq+0x1c/0x30 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8102d365>] do_softirq+0x65/0xa0 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8108f665>] irq_exit+0x115/0x120 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81699138>] do_IRQ+0x58/0xf0 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8168e2ad>] common_interrupt+0x6d/0x6d Mar 10 09:24:03 2kvm2 kernel: <EOI> [<ffffffff81189a73>] ? free_hot_cold_page+0x103/0x160 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81189b16>] free_hot_cold_page_list+0x46/0xa0 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81195193>] shrink_page_list+0x543/0xb00 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81195dda>] shrink_inactive_list+0x1fa/0x630 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81196975>] shrink_lruvec+0x385/0x770 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81196dd6>] shrink_zone+0x76/0x1a0 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8119807c>] balance_pgdat+0x48c/0x5e0 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81198343>] kswapd+0x173/0x450 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b17d0>] ? wake_up_atomic_t+0x30/0x30 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811981d0>] ? balance_pgdat+0x5e0/0x5e0 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b06ff>] kthread+0xcf/0xe0 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81696a58>] ret_from_fork+0x58/0x90 Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140 Mar 10 09:24:03 2kvm2 kernel: kswapd1: page allocation failure: order:2, mode:0x104020 Mar 10 09:24:03 2kvm2 kernel: CPU: 42 PID: 265 Comm: kswapd1 Tainted: G I ------------ 3.10.0-514.10.2.el7.x86_64 #1 Mar 10 09:24:03 2kvm2 kernel: Hardware name: Supermicro X10DRC/X10DRi-LN4+, BIOS 1.0a 08/29/2014 Mar 10 09:24:03 2kvm2 kernel: 0000000000104020 00000000f7228dc9 ffff88301f4839d8 ffffffff816864ef and in critical time again Mar 10 10:37:53 2kvm2 sanlock[1673]: 2017-03-10 10:37:53+0100 140524 [1673]: s3 check_our_lease warning 73 last_success 140451 Mar 10 10:37:54 2kvm2 sanlock[1673]: 2017-03-10 10:37:54+0100 140525 [1673]: s3 check_our_lease warning 74 last_success 140451 Mar 10 10:37:54 2kvm2 wdmd[1732]: test warning now 140526 ping 140516 close 0 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1 Mar 10 10:37:54 2kvm2 kernel: watchdog watchdog0: watchdog did not stop! Mar 10 10:37:54 2kvm2 wdmd[1732]: /dev/watchdog0 closed unclean Mar 10 10:37:55 2kvm2 sanlock[1673]: 2017-03-10 10:37:55+0100 140526 [1673]: s3 check_our_lease warning 75 last_success 140451 Mar 10 10:37:55 2kvm2 wdmd[1732]: test warning now 140527 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1 Mar 10 10:37:56 2kvm2 sanlock[1673]: 2017-03-10 10:37:56+0100 140527 [1673]: s3 check_our_lease warning 76 last_success 140451 Mar 10 10:37:56 2kvm2 wdmd[1732]: test warning now 140528 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1 Mar 10 10:37:57 2kvm2 sanlock[1673]: 2017-03-10 10:37:57+0100 140528 [1673]: s3 check_our_lease warning 77 last_success 140451 Mar 10 10:37:57 2kvm2 wdmd[1732]: test warning now 140529 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1 Mar 10 10:37:58 2kvm2 sanlock[1673]: 2017-03-10 10:37:58+0100 140529 [1673]: s3 check_our_lease warning 78 last_success 140451 Mar 10 10:37:58 2kvm2 wdmd[1732]: test warning now 140530 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1 Mar 10 10:37:59 2kvm2 sanlock[1673]: 2017-03-10 10:37:59+0100 140530 [1673]: s3 check_our_lease warning 79 last_success 140451 Mar 10 10:37:59 2kvm2 wdmd[1732]: test failed rem 55 now 140531 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1 Mar 10 10:38:00 2kvm2 sanlock[1673]: 2017-03-10 10:38:00+0100 140531 [1673]: s3 check_our_lease failed 80 Mar 10 10:38:00 2kvm2 sanlock[1673]: 2017-03-10 10:38:00+0100 140531 [1673]: s3 all pids clear Mar 10 10:38:01 2kvm2 wdmd[1732]: /dev/watchdog0 reopen Mar 10 10:38:10 2kvm2 journal: Cannot start job (query, none) for domain TEST-LBS_EBSAPP; current job is (query, none) owned by (3284 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (62s, 0s) Mar 10 10:38:10 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Mar 10 10:38:11 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 40590, 0, 0) at 0x3acdd88>: unexpected eof Mar 10 10:38:40 2kvm2 journal: Cannot start job (query, none) for domain TEST1-LBS_ATRYA; current job is (query, none) owned by (3288 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (47s, 0s) Mar 10 10:38:40 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Mar 10 10:38:41 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 40592, 0, 0) at 0x3fd5b90>: unexpected eof Mar 10 10:39:10 2kvm2 journal: Cannot start job (query, none) for domain TEST-LBS_EBSAPP; current job is (query, none) owned by (3284 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (122s, 0s) Mar 10 10:39:10 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Mar 10 10:39:10 2kvm2 journal: Cannot start job (query, none) for domain TEST1-LBS_ATRYA; current job is (query, none) owned by (3288 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (77s, 0s) Mar 10 10:39:10 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Mar 10 10:39:11 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 40594, 0, 0) at 0x2447290>: unexpected eof Mar 10 10:39:23 2kvm2 sanlock[1673]: 2017-03-10 10:39:23+0100 140615 [2576]: s3 delta_renew write time 140 error -202 Mar 10 10:39:23 2kvm2 sanlock[1673]: 2017-03-10 10:39:23+0100 140615 [2576]: s3 renewal error -202 delta_length 144 last_success 140451 Mar 10 10:39:40 2kvm2 journal: Cannot start job (query, none) for domain TEST-LBS_EBSAPP; current job is (query, none) owned by (3284 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (152s, 0s) Mar 10 10:39:40 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Mar 10 10:39:40 2kvm2 journal: Cannot start job (query, none) for domain TEST1-LBS_ATRYA; current job is (query, none) owned by (3288 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (107s, 0s) Mar 10 10:39:40 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Mar 10 10:39:41 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 40596, 0, 0) at 0x2472ef0>: unexpected eof Mar 10 10:39:49 2kvm2 kernel: INFO: task qemu-img:42107 blocked for more than 120 seconds. Mar 10 10:39:49 2kvm2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 10 10:39:49 2kvm2 kernel: qemu-img D ffff88010dad3e30 0 42107 3631 0x00000080 Mar 10 10:39:49 2kvm2 kernel: ffff88010dad3b30 0000000000000082 ffff8814491f4e70 ffff88010dad3fd8 Mar 10 10:39:49 2kvm2 kernel: ffff88010dad3fd8 ffff88010dad3fd8 ffff8814491f4e70 ffff88301f096c40 Mar 10 10:39:49 2kvm2 kernel: 0000000000000000 7fffffffffffffff ffff88181f186c00 ffff88010dad3e30 Mar 10 10:39:49 2kvm2 kernel: Call Trace: Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168bbb9>] schedule+0x29/0x70 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81689609>] schedule_timeout+0x239/0x2d0 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168b15e>] io_schedule_timeout+0xae/0x130 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168b1f8>] io_schedule+0x18/0x20 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8124d9e5>] wait_on_sync_kiocb+0x35/0x80 Mar 10 10:39:49 2kvm2 kernel: [<ffffffffa0a36091>] fuse_direct_IO+0x231/0x380 [fuse] Mar 10 10:39:49 2kvm2 kernel: [<ffffffff812a6ddd>] ? cap_inode_need_killpriv+0x2d/0x40 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff812a8cb6>] ? security_inode_need_killpriv+0x16/0x20 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81219e3f>] ? dentry_needs_remove_privs.part.13+0x1f/0x30 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81182a2d>] generic_file_direct_write+0xcd/0x190 Mar 10 10:39:49 2kvm2 kernel: [<ffffffffa0a36905>] fuse_file_aio_write+0x185/0x340 [fuse] Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811fdabd>] do_sync_write+0x8d/0xd0 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811fe32d>] vfs_write+0xbd/0x1e0 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811ff002>] SyS_pwrite64+0x92/0xc0 Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81696b09>] system_call_fastpath+0x16/0x1b Mar 10 10:39:49 2kvm2 kernel: INFO: task qemu-img:42111 blocked for more than 120 seconds. Mar 10 10:39:49 2kvm2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 10 10:39:49 2kvm2 kernel: qemu-img D ffff8818a76e7e30 0 42111 3632 0x00000080 Mar 10 10:39:49 2kvm2 kernel: ffff8818a76e7b30 0000000000000082 ffff88188aaeaf10 ffff8818a76e7fd8 Mar 10 10:39:49 2kvm2 kernel: ffff8818a76e7fd8 ffff8818a76e7fd8 ffff88188aaeaf10 ffff88301f156c40 memory ======= # cat /proc/meminfo MemTotal: 197983472 kB MemFree: 834228 kB MemAvailable: 165541204 kB Buffers: 45548 kB Cached: 159596272 kB SwapCached: 119872 kB Active: 40803264 kB Inactive: 148022076 kB Active(anon): 26594112 kB Inactive(anon): 2626384 kB Active(file): 14209152 kB Inactive(file): 145395692 kB Unevictable: 50488 kB Mlocked: 50488 kB SwapTotal: 4194300 kB SwapFree: 3612188 kB Dirty: 624 kB Writeback: 0 kB AnonPages: 29185032 kB Mapped: 85176 kB Shmem: 25908 kB Slab: 6203384 kB SReclaimable: 5857240 kB SUnreclaim: 346144 kB KernelStack: 19184 kB PageTables: 86100 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 103186036 kB Committed_AS: 52300288 kB VmallocTotal: 34359738367 kB VmallocUsed: 1560580 kB VmallocChunk: 34257341440 kB HardwareCorrupted: 0 kB AnonHugePages: 5566464 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 431292 kB DirectMap2M: 19382272 kB DirectMap1G: 183500800 kB can anybody help me with this ?? I've got a small tip about swap problem ( in messages), but not shure ..... The similar problem occured in older versions in gluster/ovirt testing ( in huge workload freeez - but not fatal overload ) regards Paf1 --------------C04C9E032CD3A8F03847AFCD Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000066"> <font face="Ubuntu">Hello everybody,<br> <br> for production usage i'm testing ovirt with gluster.<br> All components seems to be running fine but whenever I'm testing huge workload, then node freez. Not the main OS, but VDSM mgmt and attached services, VMs eg.<br> <br> <b>mgmt </b><br> oVirt - 4.1.0.4<br> centos 7.3-1611<br> <br> <br> <b>nodes</b> ( installed from ovirt image <i>"ovirt-node-ng-installer-ovirt-4.1-2017030804.iso" )</i><br> </font><br> <div class="row"> <div class="col-md-12"> <div class="row"> <div class="col-md-2"> <div class="col-md-10">OS Version:<span id="SubTabHostGeneralSoftwareView_formPanel_col0_row0_value" class="GOJECEMBACD"> == RHEL - 7 - 3.1611.el7.centos</span></div> </div> </div> </div> </div> <div class="row"> <div class="col-md-12"> <div class="row"> <div class="col-md-2"> <div class="col-md-10">OS Description:<span id="SubTabHostGeneralSoftwareView_formPanel_col0_row1_value" class="GOJECEMBACD">== oVirt Node 4.1.0</span></div> </div> </div> </div> </div> <div class="row"> <div class="col-md-12"> <div class="row"> <div class="col-md-2"> <div class="col-md-10">Kernel Version:<span id="SubTabHostGeneralSoftwareView_formPanel_col0_row2_value" class="GOJECEMBACD">== 3.10.0 - 514.10.2.el7.x86_64</span></div> </div> </div> </div> </div> <div class="row"> <div class="col-md-12"> <div class="row"> <div class="col-md-2"> <div class="col-md-10">KVM Version:<span id="SubTabHostGeneralSoftwareView_formPanel_col0_row3_value" class="GOJECEMBACD">== 2.6.0 - 28.el7_3.3.1</span></div> </div> </div> </div> </div> <div class="row"> <div class="col-md-12"> <div class="row"> <div class="col-md-2"> <div class="col-md-10">LIBVIRT Version:<span id="SubTabHostGeneralSoftwareView_formPanel_col0_row4_value" class="GOJECEMBACD">== libvirt-2.0.0-10.el7_3.5</span></div> </div> </div> </div> </div> <div class="row"> <div class="col-md-12"> <div class="row"> <div class="col-md-2"> <div class="col-md-10">VDSM Version:<span id="SubTabHostGeneralSoftwareView_formPanel_col0_row5_value" class="GOJECEMBACD">== vdsm-4.19.4-1.el7.centos</span></div> </div> </div> </div> </div> <div class="row"> <div class="col-md-12"> <div class="row"> <div class="col-md-2"> <div class="col-md-10">SPICE Version:<span id="SubTabHostGeneralSoftwareView_formPanel_col0_row6_value" class="GOJECEMBACD">== 0.12.4 - 20.el7_3</span></div> </div> </div> </div> </div> <div class="row"> <div class="col-md-12"> <div class="row"> <div class="col-md-2"> <div class="col-md-10">GlusterFS Version:<span id="SubTabHostGeneralSoftwareView_formPanel_col0_row7_value" class="GOJECEMBACD">== glusterfs-3.8.9-1.el7 ( LVM thinprovisioning in replica 2 - created from ovirt GUI )<br> </span></div> </div> </div> </div> </div> <div class="row"> <div class="col-md-12"> <div class="row"> <div class="col-md-2"> <div class="GOJECEMBPBD" id="SubTabHostGeneralSoftwareView_formPanel_col0_row8_label"><br> </div> </div> </div> </div> </div> <font face="Ubuntu">concurently running<br> - huge import from export domain ( net workload )<br> - sequential write to VMs local disk ( gluster replica sequential workload )<br> - VMs database huge select ( random IOps )<br> - huge old snapshot delete ( random IOps )<br> <br> In this configuration / workload is runnig one hour eg, with no exceptions , with 70-80% disk load, but in some point VDSM freez all jobs for a timeout and VM's are in "uknown" status .<br> The whole system revitalize then automaticaly in cca 20min time frame ( except the import and snapshot deleting(rollback) )<br> <br> engine.log - focus 10:39:07 time ( </font><font face="Ubuntu"><font face="Ubuntu">Failed in 'HSMGetAllTasksStatusesVDS' method )<br> </font>========<br> <br> <font size="-1">n child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete<br> 2017-03-10 10:39:01,727+01 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (DefaultQuartzScheduler2) [759c8e1f] Command 'RemoveSnapshotSingleDiskLive' (id: 'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command id: '33df2c1e-6ce3-44fd-a39b-d111883b4c4e' type:'DestroyImage' to complete<br> 2017-03-10 10:39:03,929+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 446d0cd3<br> 2017-03-10 10:39:04,343+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 446d0cd3<br> 2017-03-10 10:39:04,353+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 69ea1fda<br> 2017-03-10 10:39:05,128+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 69ea1fda<br> 2017-03-10 10:39:07,163+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [759c8e1f] Failed in 'HSMGetAllTasksStatusesVDS' method<br> 2017-03-10 10:39:07,178+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [759c8e1f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM 2kvm2 command HSMGetAllTasksStatusesVDS failed: Connection timed out<br> 2017-03-10 10:39:07,182+01 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] BaseAsyncTask::onTaskEndSuccess: Task 'f594bf69-619b-4d1b-8f6d-a9826997e478' (Parent Command 'ImportVm', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully.<br> 2017-03-10 10:39:07,182+01 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler2) [759c8e1f] Task with DB Task ID 'a05c7c07-9b98-4ab2-ac7b-9e70a75ba7b7' and VDSM Task ID '7c60369f-70a3-4a6a-80c9-4753ac9ed372' is in state Polling. End action for command 8deb3fe3-4a83-4605-816c-ffdc63fd9ac1 will proceed when all the entity's tasks are completed.<br> 2017-03-10 10:39:07,182+01 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] SPMAsyncTask::PollTask: Polling task 'f351e8f6-6dd7-49aa-bf54-650d84fc6352' (Parent Command 'DestroyImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'.<br> 2017-03-10 10:39:07,182+01 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] BaseAsyncTask::logEndTaskFailure: Task 'f351e8f6-6dd7-49aa-bf54-650d84fc6352' (Parent Command 'DestroyImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure:<br> -- Result: 'cleanSuccess'<br> -- Message: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Connection timed out, code = 100',<br> -- Exception: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Connection timed out, code = 100'<br> 2017-03-10 10:39:07,184+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] CommandAsyncTask::endActionIfNecessary: All tasks of command '33df2c1e-6ce3-44fd-a39b-d111883b4c4e' has ended -> executing 'endAction'<br> 2017-03-10 10:39:07,185+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler2) [759c8e1f] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: '33df2c1e-6ce3-44fd-a39b-d111883b4c4e'): calling endAction '.<br> 2017-03-10 10:39:07,185+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'DestroyImage',<br> 2017-03-10 10:39:07,192+01 INFO [org.ovirt.engine.core.bll.storage.disk.image.DestroyImageCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] Command [id=33df2c1e-6ce3-44fd-a39b-d111883b4c4e]: Updating status to 'FAILED', The command end method logic will be executed by one of its parent commands.<br> 2017-03-10 10:39:07,192+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'DestroyImage' completed, handling the result.<br> 2017-03-10 10:39:07,192+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'DestroyImage' succeeded, clearing tasks.<br> 2017-03-10 10:39:07,192+01 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'f351e8f6-6dd7-49aa-bf54-650d84fc6352'<br> 2017-03-10 10:39:07,193+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', taskId='f351e8f6-6dd7-49aa-bf54-650d84fc6352'}), log id: 2b7080c2<br> 2017-03-10 10:39:07,194+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] START, HSMClearTaskVDSCommand(HostName = 2kvm2, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='905375e1-6de4-4fdf-b69c-b2d546f869c8', taskId='f351e8f6-6dd7-49aa-bf54-650d84fc6352'}), log id: 2edff460<br> 2017-03-10 10:39:08,208+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] FINISH, HSMClearTaskVDSCommand, log id: 2edff460<br> 2017-03-10 10:39:08,208+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] FINISH, SPMClearTaskVDSCommand, log id: 2b7080c2<br> 2017-03-10 10:39:08,213+01 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] BaseAsyncTask::removeTaskFromDB: Removed task 'f351e8f6-6dd7-49aa-bf54-650d84fc6352' from DataBase<br> 2017-03-10 10:39:08,213+01 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [759c8e1f] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '33df2c1e-6ce3-44fd-a39b-d111883b4c4e'<br> 2017-03-10 10:39:10,142+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 2e7278cb<br> 2017-03-10 10:39:11,513+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 2e7278cb<br> 2017-03-10 10:39:11,523+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 43704ef2<br> 2017-03-10 10:39:11,777+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler9) [67e1d8ed] Command 'RemoveSnapshot' (id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete<br> 2017-03-10 10:39:11,789+01 WARN [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [759c8e1f] Child command 'DESTROY_IMAGE' failed, proceeding to verify<br> 2017-03-10 10:39:11,789+01 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [759c8e1f] Executing Live Merge command step 'DESTROY_IMAGE_CHECK'<br> 2017-03-10 10:39:11,832+01 INFO [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (pool-5-thread-7) [4856f570] Running command: DestroyImageCheckCommand internal: true.<br> 2017-03-10 10:39:11,833+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand] (pool-5-thread-7) [4856f570] START, SPMGetVolumeInfoVDSCommand( SPMGetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]', runAsync='true', storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', storageDomainId='1603cd90-92ef-4c03-922c-cecb282fd00e', imageGroupId='7543338a-3ca6-4698-bb50-c14f0bd71428', imageId='50b592f7-bfba-4398-879c-8d6a19a2c000'}), log id: 2c8031f8<br> 2017-03-10 10:39:11,833+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand] (pool-5-thread-7) [4856f570] Executing GetVolumeInfo using the current SPM<br> 2017-03-10 10:39:11,834+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (pool-5-thread-7) [4856f570] START, GetVolumeInfoVDSCommand(HostName = 2kvm2, GetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]', runAsync='true', hostId='905375e1-6de4-4fdf-b69c-b2d546f869c8', storagePoolId='00000001-0001-0001-0001-000000000311', storageDomainId='1603cd90-92ef-4c03-922c-cecb282fd00e', imageGroupId='7543338a-3ca6-4698-bb50-c14f0bd71428', imageId='50b592f7-bfba-4398-879c-8d6a19a2c000'}), log id: 79ca86cc<br> 2017-03-10 10:39:11,846+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 43704ef2<br> 2017-03-10 10:39:16,858+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 6542adcd<br> 2017-03-10 10:39:17,394+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 6542adcd<br> 2017-03-10 10:39:17,406+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 44ec33ed<br> 2017-03-10 10:39:18,598+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 44ec33ed<br> 2017-03-10 10:39:21,865+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot' (id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete<br> 2017-03-10 10:39:21,881+01 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (DefaultQuartzScheduler6) [4856f570] Command 'RemoveSnapshotSingleDiskLive' (id: 'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command id: 'b1d63b8e-19d3-4d64-8fa8-4eb3e2d1a8fc' type:'DestroyImageCheck' to complete<br> 2017-03-10 10:39:23,611+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler6) [4856f570] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 4c2fc22d<br> 2017-03-10 10:39:24,616+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] START, GlusterTasksListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 1f169371<br> 2017-03-10 10:39:24,618+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler6) [4856f570] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 4c2fc22d<br> 2017-03-10 10:39:24,629+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler6) [4856f570] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 2ac55735<br> 2017-03-10 10:39:24,822+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] FINISH, GlusterTasksListVDSCommand, return: [], log id: 1f169371<br> 2017-03-10 10:39:26,836+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler6) [4856f570] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 2ac55735<br> 2017-03-10 10:39:31,849+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 2e8dbcd1<br> 2017-03-10 10:39:31,932+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot' (id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete<br> 2017-03-10 10:39:31,944+01 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (DefaultQuartzScheduler6) [4856f570] Command 'RemoveSnapshotSingleDiskLive' (id: 'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command id: 'b1d63b8e-19d3-4d64-8fa8-4eb3e2d1a8fc' type:'DestroyImageCheck' to complete<br> 2017-03-10 10:39:33,213+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 2e8dbcd1<br> 2017-03-10 10:39:33,226+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 1fb3f9e3<br> 2017-03-10 10:39:34,375+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 1fb3f9e3<br> 2017-03-10 10:39:39,392+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] START, GlusterServersListVDSCommand(HostName = 2kvm1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 1e0b8eeb<br> 2017-03-10 10:39:40,753+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] FINISH, GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 1e0b8eeb<br> 2017-03-10 10:39:40,763+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] START, GlusterVolumesListVDSCommand(HostName = 2kvm1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 35b04b33<br> 2017-03-10 10:39:41,952+01 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] FINISH, GlusterVolumesListVDSCommand, return: {8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad}, log id: 35b04b33<br> 2017-03-10 10:39:41,991+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot' (id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508' type:'RemoveSnapshotSingleDiskLive' to complete<br> <br> </font><br> gluster ( nothing in logs )<br> ======<br> <br> <br> </font><font face="Ubuntu">## "etc-glusterfs-glusterd.vol.log"</font><font face="Ubuntu"><font size="-1"><br> [2017-03-10 10:13:52.599019] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:16:48.639635] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> The message "I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1" repeated 3 times between [2017-03-10 10:16:48.639635] and [2017-03-10 10:17:55.659379]<br> [2017-03-10 10:18:56.875516] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:19:57.204689] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:21:56.576879] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:21:57.772857] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:24:00.617931] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:30:04.918080] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:31:06.128638] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:32:07.325672] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:32:12.433586] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:32:13.544909] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:35:10.039213] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:37:19.905314] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:37:20.174209] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:38:12.635460] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1<br> [2017-03-10 10:40:14.169864] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume slow1</font><br> <br> <br> ## "rhev-data-center-mnt-glusterSD-localhost:_slow1.log"<br> <font size="-1">[2017-03-10 09:43:40.346785] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61849: dentry not found in 43e6968f-9c2a-40d8-8074-caf1a36f60cf<br> [2017-03-10 09:43:40.347076] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61879: dentry not found in 902a6e3d-b7aa-439f-8262-cdc1b7f9f022<br> [2017-03-10 09:43:40.347145] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61935: dentry not found in 846bbcfc-f2b3-4ab6-af44-aeaa10b39318<br> [2017-03-10 09:43:40.347211] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61922: dentry not found in 66ad3bc5-26c7-4360-b33b-a084e3305cf8<br> [2017-03-10 09:43:40.351571] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61834: dentry not found in 3b8278e1-40e5-4363-b21e-7bffcd024c62<br> [2017-03-10 09:43:40.352449] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61870: dentry not found in 282f4c05-e09a-48e0-96a3-52e079ff2f73<br> [2017-03-10 09:50:38.829325] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta (hash=slow1-replicate-0/cache=slow1-replicate-0)<br> [2017-03-10 09:50:42.221775] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta (hash=slow1-replicate-0/cache=slow1-replicate-0)<br> [2017-03-10 09:50:45.956432] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/3cef54b4-45b9-4f5b-82c2-fcc8def06a37/85287865-38f0-45df-9e6c-1294913cbb88.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/3cef54b4-45b9-4f5b-82c2-fcc8def06a37/85287865-38f0-45df-9e6c-1294913cbb88.meta (hash=slow1-replicate-0/cache=slow1-replicate-0)<br> [2017-03-10 09:50:40.349563] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta (hash=slow1-replicate-0/cache=slow1-replicate-0)<br> [2017-03-10 09:50:44.503866] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta (hash=slow1-replicate-0/cache=slow1-replicate-0)<br> [2017-03-10 09:59:46.860762] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.6684: dentry not found in d1e65eea-8758-4407-ac2e-3605dc661364<br> [2017-03-10 10:02:22.500865] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.8767: dentry not found in e228bb28-9602-4f8e-8323-7434d77849fc<br> [2017-03-10 10:04:03.103839] W [MSGID: 101159] [inode.c:1214:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.9787: dentry not found in 6be71632-aa36-4975-b673-1357e0355027<br> [2017-03-10 10:06:02.406385] I [MSGID: 109066] [dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming /1603cd90-92ef-4c03-922c-cecb282fd00e/images/2a9c1c6a-f045-4dce-a47b-95a2267eef72/6f264695-0669-4b49-a2f6-e6c92482f2fb.meta.new (hash=slow1-replicate-0/cache=slow1-replicate-0) => /1603cd90-92ef-4c03-922c-cecb282fd00e/images/2a9c1c6a-f045-4dce-a47b-95a2267eef72/6f264695-0669-4b49-a2f6-e6c92482f2fb.meta (hash=slow1-replicate-0/cache=slow1-replicate-0)</font><br> <font size="-1">... no other record</font><br> <br> <br> messages<br> ========<br> <br> several times occured:<br> <br> <font size="-1">Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd.<br> Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd.<br> Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd.<br> Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd.<br> Mar 10 09:10:01 2kvm2 systemd: Started Session 274 of user root.<br> Mar 10 09:10:01 2kvm2 systemd: Starting Session 274 of user root.<br> Mar 10 09:20:02 2kvm2 systemd: Started Session 275 of user root.<br> Mar 10 09:20:02 2kvm2 systemd: Starting Session 275 of user root.<br> Mar 10 09:22:59 2kvm2 sanlock[1673]: 2017-03-10 09:22:59+0100 136031 [2576]: s3 delta_renew long write time 11 sec<br> Mar 10 09:24:03 2kvm2 kernel: kswapd1: page allocation failure: order:2, mode:0x104020<br> Mar 10 09:24:03 2kvm2 kernel: CPU: 42 PID: 265 Comm: kswapd1 Tainted: G I ------------ 3.10.0-514.10.2.el7.x86_64 #1<br> Mar 10 09:24:03 2kvm2 kernel: Hardware name: Supermicro X10DRC/X10DRi-LN4+, BIOS 1.0a 08/29/2014<br> Mar 10 09:24:03 2kvm2 kernel: 0000000000104020 00000000f7228dc9 ffff88301f4839d8 ffffffff816864ef<br> Mar 10 09:24:03 2kvm2 kernel: ffff88301f483a68 ffffffff81186ba0 000068fc00000000 0000000000000000<br> Mar 10 09:24:03 2kvm2 kernel: fffffffffffffffc 0010402000000000 ffff88301567ae80 00000000f7228dc9<br> Mar 10 09:24:03 2kvm2 kernel: Call Trace:<br> Mar 10 09:24:03 2kvm2 kernel: <IRQ> [<ffffffff816864ef>] dump_stack+0x19/0x1b<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81186ba0>] warn_alloc_failed+0x110/0x180<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81682083>] __alloc_pages_slowpath+0x6b7/0x725<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8118b155>] __alloc_pages_nodemask+0x405/0x420<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811cf30a>] alloc_pages_current+0xaa/0x170<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81185a7e>] __get_free_pages+0xe/0x50<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811dabae>] kmalloc_order_trace+0x2e/0xa0<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811dd381>] __kmalloc+0x221/0x240<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02f83fa>] bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x]<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02f92f7>] bnx2x_rx_int+0x227/0x17b0 [bnx2x]<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81033669>] ? sched_clock+0x9/0x10<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02fc72d>] bnx2x_poll+0x1dd/0x260 [bnx2x]<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff815705e0>] net_rx_action+0x170/0x380<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8108f2cf>] __do_softirq+0xef/0x280<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8169859c>] call_softirq+0x1c/0x30<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8102d365>] do_softirq+0x65/0xa0<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8108f665>] irq_exit+0x115/0x120<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81699138>] do_IRQ+0x58/0xf0<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8168e2ad>] common_interrupt+0x6d/0x6d<br> Mar 10 09:24:03 2kvm2 kernel: <EOI> [<ffffffff81189a73>] ? free_hot_cold_page+0x103/0x160<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81189b16>] free_hot_cold_page_list+0x46/0xa0<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81195193>] shrink_page_list+0x543/0xb00<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81195dda>] shrink_inactive_list+0x1fa/0x630<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81196975>] shrink_lruvec+0x385/0x770<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81196dd6>] shrink_zone+0x76/0x1a0<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8119807c>] balance_pgdat+0x48c/0x5e0<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81198343>] kswapd+0x173/0x450<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b17d0>] ? wake_up_atomic_t+0x30/0x30<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811981d0>] ? balance_pgdat+0x5e0/0x5e0<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b06ff>] kthread+0xcf/0xe0<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81696a58>] ret_from_fork+0x58/0x90<br> Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140<br> Mar 10 09:24:03 2kvm2 kernel: kswapd1: page allocation failure: order:2, mode:0x104020<br> Mar 10 09:24:03 2kvm2 kernel: CPU: 42 PID: 265 Comm: kswapd1 Tainted: G I ------------ 3.10.0-514.10.2.el7.x86_64 #1<br> Mar 10 09:24:03 2kvm2 kernel: Hardware name: Supermicro X10DRC/X10DRi-LN4+, BIOS 1.0a 08/29/2014<br> Mar 10 09:24:03 2kvm2 kernel: 0000000000104020 00000000f7228dc9 ffff88301f4839d8 ffffffff816864ef<br> </font><br> <br> and in critical time again<br> <br> <font size="-1">Mar 10 10:37:53 2kvm2 sanlock[1673]: 2017-03-10 10:37:53+0100 140524 [1673]: s3 check_our_lease warning 73 last_success 140451<br> Mar 10 10:37:54 2kvm2 sanlock[1673]: 2017-03-10 10:37:54+0100 140525 [1673]: s3 check_our_lease warning 74 last_success 140451<br> Mar 10 10:37:54 2kvm2 wdmd[1732]: test warning now 140526 ping 140516 close 0 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br> Mar 10 10:37:54 2kvm2 kernel: watchdog watchdog0: watchdog did not stop!<br> Mar 10 10:37:54 2kvm2 wdmd[1732]: /dev/watchdog0 closed unclean<br> Mar 10 10:37:55 2kvm2 sanlock[1673]: 2017-03-10 10:37:55+0100 140526 [1673]: s3 check_our_lease warning 75 last_success 140451<br> Mar 10 10:37:55 2kvm2 wdmd[1732]: test warning now 140527 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br> Mar 10 10:37:56 2kvm2 sanlock[1673]: 2017-03-10 10:37:56+0100 140527 [1673]: s3 check_our_lease warning 76 last_success 140451<br> Mar 10 10:37:56 2kvm2 wdmd[1732]: test warning now 140528 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br> Mar 10 10:37:57 2kvm2 sanlock[1673]: 2017-03-10 10:37:57+0100 140528 [1673]: s3 check_our_lease warning 77 last_success 140451<br> Mar 10 10:37:57 2kvm2 wdmd[1732]: test warning now 140529 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br> Mar 10 10:37:58 2kvm2 sanlock[1673]: 2017-03-10 10:37:58+0100 140529 [1673]: s3 check_our_lease warning 78 last_success 140451<br> Mar 10 10:37:58 2kvm2 wdmd[1732]: test warning now 140530 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br> Mar 10 10:37:59 2kvm2 sanlock[1673]: 2017-03-10 10:37:59+0100 140530 [1673]: s3 check_our_lease warning 79 last_success 140451<br> Mar 10 10:37:59 2kvm2 wdmd[1732]: test failed rem 55 now 140531 ping 140516 close 140526 renewal 140451 expire 140531 client 1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br> Mar 10 10:38:00 2kvm2 sanlock[1673]: 2017-03-10 10:38:00+0100 140531 [1673]: s3 check_our_lease failed 80<br> Mar 10 10:38:00 2kvm2 sanlock[1673]: 2017-03-10 10:38:00+0100 140531 [1673]: s3 all pids clear<br> Mar 10 10:38:01 2kvm2 wdmd[1732]: /dev/watchdog0 reopen<br> Mar 10 10:38:10 2kvm2 journal: Cannot start job (query, none) for domain TEST-LBS_EBSAPP; current job is (query, none) owned by (3284 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (62s, 0s)<br> Mar 10 10:38:10 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)<br> Mar 10 10:38:11 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 40590, 0, 0) at 0x3acdd88>: unexpected eof<br> Mar 10 10:38:40 2kvm2 journal: Cannot start job (query, none) for domain TEST1-LBS_ATRYA; current job is (query, none) owned by (3288 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (47s, 0s)<br> Mar 10 10:38:40 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)<br> Mar 10 10:38:41 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 40592, 0, 0) at 0x3fd5b90>: unexpected eof<br> Mar 10 10:39:10 2kvm2 journal: Cannot start job (query, none) for domain TEST-LBS_EBSAPP; current job is (query, none) owned by (3284 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (122s, 0s)<br> Mar 10 10:39:10 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)<br> Mar 10 10:39:10 2kvm2 journal: Cannot start job (query, none) for domain TEST1-LBS_ATRYA; current job is (query, none) owned by (3288 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (77s, 0s)<br> Mar 10 10:39:10 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)<br> Mar 10 10:39:11 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 40594, 0, 0) at 0x2447290>: unexpected eof<br> Mar 10 10:39:23 2kvm2 sanlock[1673]: 2017-03-10 10:39:23+0100 140615 [2576]: s3 delta_renew write time 140 error -202<br> Mar 10 10:39:23 2kvm2 sanlock[1673]: 2017-03-10 10:39:23+0100 140615 [2576]: s3 renewal error -202 delta_length 144 last_success 140451<br> Mar 10 10:39:40 2kvm2 journal: Cannot start job (query, none) for domain TEST-LBS_EBSAPP; current job is (query, none) owned by (3284 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (152s, 0s)<br> Mar 10 10:39:40 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)<br> Mar 10 10:39:40 2kvm2 journal: Cannot start job (query, none) for domain TEST1-LBS_ATRYA; current job is (query, none) owned by (3288 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (107s, 0s)<br> Mar 10 10:39:40 2kvm2 journal: Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)<br> Mar 10 10:39:41 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 40596, 0, 0) at 0x2472ef0>: unexpected eof<br> Mar 10 10:39:49 2kvm2 kernel: INFO: task qemu-img:42107 blocked for more than 120 seconds.<br> Mar 10 10:39:49 2kvm2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br> Mar 10 10:39:49 2kvm2 kernel: qemu-img D ffff88010dad3e30 0 42107 3631 0x00000080<br> Mar 10 10:39:49 2kvm2 kernel: ffff88010dad3b30 0000000000000082 ffff8814491f4e70 ffff88010dad3fd8<br> Mar 10 10:39:49 2kvm2 kernel: ffff88010dad3fd8 ffff88010dad3fd8 ffff8814491f4e70 ffff88301f096c40<br> Mar 10 10:39:49 2kvm2 kernel: 0000000000000000 7fffffffffffffff ffff88181f186c00 ffff88010dad3e30<br> Mar 10 10:39:49 2kvm2 kernel: Call Trace:<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168bbb9>] schedule+0x29/0x70<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81689609>] schedule_timeout+0x239/0x2d0<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168b15e>] io_schedule_timeout+0xae/0x130<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168b1f8>] io_schedule+0x18/0x20<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8124d9e5>] wait_on_sync_kiocb+0x35/0x80<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffffa0a36091>] fuse_direct_IO+0x231/0x380 [fuse]<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff812a6ddd>] ? cap_inode_need_killpriv+0x2d/0x40<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff812a8cb6>] ? security_inode_need_killpriv+0x16/0x20<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81219e3f>] ? dentry_needs_remove_privs.part.13+0x1f/0x30<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81182a2d>] generic_file_direct_write+0xcd/0x190<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffffa0a36905>] fuse_file_aio_write+0x185/0x340 [fuse]<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811fdabd>] do_sync_write+0x8d/0xd0<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811fe32d>] vfs_write+0xbd/0x1e0<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811ff002>] SyS_pwrite64+0x92/0xc0<br> Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81696b09>] system_call_fastpath+0x16/0x1b<br> Mar 10 10:39:49 2kvm2 kernel: INFO: task qemu-img:42111 blocked for more than 120 seconds.<br> Mar 10 10:39:49 2kvm2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br> Mar 10 10:39:49 2kvm2 kernel: qemu-img D ffff8818a76e7e30 0 42111 3632 0x00000080<br> Mar 10 10:39:49 2kvm2 kernel: ffff8818a76e7b30 0000000000000082 ffff88188aaeaf10 ffff8818a76e7fd8<br> Mar 10 10:39:49 2kvm2 kernel: ffff8818a76e7fd8 ffff8818a76e7fd8 ffff88188aaeaf10 ffff88301f156c40<br> </font><br> memory<br> =======<br> <br> # cat /proc/meminfo <br> <font size="-1">MemTotal: 197983472 kB<br> MemFree: 834228 kB<br> MemAvailable: 165541204 kB<br> Buffers: 45548 kB<br> Cached: 159596272 kB<br> SwapCached: 119872 kB<br> Active: 40803264 kB<br> Inactive: 148022076 kB<br> Active(anon): 26594112 kB<br> Inactive(anon): 2626384 kB<br> Active(file): 14209152 kB<br> Inactive(file): 145395692 kB<br> Unevictable: 50488 kB<br> Mlocked: 50488 kB<br> SwapTotal: 4194300 kB<br> SwapFree: 3612188 kB<br> Dirty: 624 kB<br> Writeback: 0 kB<br> AnonPages: 29185032 kB<br> Mapped: 85176 kB<br> Shmem: 25908 kB<br> Slab: 6203384 kB<br> SReclaimable: 5857240 kB<br> SUnreclaim: 346144 kB<br> KernelStack: 19184 kB<br> PageTables: 86100 kB<br> NFS_Unstable: 0 kB<br> Bounce: 0 kB<br> WritebackTmp: 0 kB<br> CommitLimit: 103186036 kB<br> Committed_AS: 52300288 kB<br> VmallocTotal: 34359738367 kB<br> VmallocUsed: 1560580 kB<br> VmallocChunk: 34257341440 kB<br> HardwareCorrupted: 0 kB<br> AnonHugePages: 5566464 kB<br> HugePages_Total: 0<br> HugePages_Free: 0<br> HugePages_Rsvd: 0<br> HugePages_Surp: 0<br> Hugepagesize: 2048 kB<br> DirectMap4k: 431292 kB<br> DirectMap2M: 19382272 kB<br> DirectMap1G: 183500800 kB</font><br> <br> <br> can anybody help me with this ??<br> I've got a small tip about swap problem ( in messages), but not shure .....<br> The similar problem occured in older versions in gluster/ovirt testing ( in huge workload freeez - but not fatal overload )<br> <br> regards<br> Paf1<br> <br> <br> </font> </body> </html> --------------C04C9E032CD3A8F03847AFCD--

3 4

Extend disk in engine appliance in ovirt 4.1.0?
by gflwqs gflwqs 13 Mar '17

13 Mar '17

Hi, I have recently installed a new ovirt environment from scratch. The version is 4.1.0. I made a 90GB disk for the hosted engine. When i had installed it and log into the engine i see that it has only configured 10GB of those 90GB. So there is 80GB of unconfigured disk unused. That is fine as long as it had used lvm but it has not meaning i have to take the engine down to extend the disk? My questions is: 1. Why does it not configure all disk? 2. Why is it not using lvm? 3. What should i do to extend the disk? Regards Christian

2 1

HE in bad stauts, will not start following storage issue - HELP
by Ian Neilsen 12 Mar '17

12 Mar '17

Hi All I had a storage issue with my gluster volumes running under ovirt hosted. I now cannot start the hosted engine manager vm from "hosted-engine --vm-start". I've scoured the net to find a way, but can't seem to find anything concrete. Running Centos7, ovirt 4.0 and gluster 3.8.9 How do I recover the engine manager. Im at a loss! Engine Status = score between nodes was 0 for all, now node 1 is reading 3400, but all others are 0 {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Logs from agent.log ================== INFO::2017-03-09 19:32:52,600::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Global maintenance detected INFO::2017-03-09 19:32:52,603::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM INFO::2017-03-09 19:32:54,820::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage INFO::2017-03-09 19:32:54,821::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server INFO::2017-03-09 19:32:59,194::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server INFO::2017-03-09 19:32:59,211::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain INFO::2017-03-09 19:32:59,328::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images INFO::2017-03-09 19:32:59,328::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images INFO::2017-03-09 19:33:01,748::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Reloading vm.conf from the shared storage domain INFO::2017-03-09 19:33:01,748::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE WARNING::2017-03-09 19:33:04,056::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE ERROR::2017-03-09 19:33:04,058::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf ovirt-ha-agent logs ================ ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf vdsm ====== vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof ovirt-ha-broker ============ ovirt-ha-broker cpu_load_no_engine.EngineHealth ERROR Failed to getVmStats: 'pid' -- Ian Neilsen Mobile: 0424 379 762 Linkedin: http://au.linkedin.com/in/ianneilsen Twitter : ineilsen

3 4

host-only network
by qinglong.dong＠horebdata.cn 12 Mar '17

12 Mar '17

This is a multi-part message in MIME format. ------=_001_NextPart562105536544_=---- Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: base64 SGksIGFsbA0KICAgICAgICBJIGhhdmUgbm90aWNlZCB0aGF0IGt2bSBzdXBwb3J0cyBob3N0LW9u bHkgbmV0d29yayBtb2RlLiBTbyBJIHdhbnQgdG8ga25vdyBob3cgdG8gY3JlYXRlIGEgaG9zdC1v bmx5IHZpbmMgZm9yIGEgdmlydHVhbCBtYWNoaW5lIGluIG92aXJ0Lg0KICAgICAgICBBbnlvbmUg Y2FuIGhlbHA/IFRoYW5rcyENCg== ------=_001_NextPart562105536544_=---- Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charse= t=3DISO-8859-1"><style>body { line-height: 1.5; }body { font-size: 10.5pt;= font-family: ????; color: rgb(0, 0, 0); line-height: 1.5; }body { font-si= ze: 10.5pt; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>H= i, all<div>    <span style=3D"font-size: 10.5pt; line-heigh= t: 1.5; background-color: window;">    I have noticed that kvm s= upports host-only network mode. So I want to know how to create a host-onl= y vinc for a virtual machine in ovirt.</span></div><div>   &nbsp= ;<span style=3D"font-size: 10.5pt; line-height: 1.5; background-color: win= dow;">    Anyone can help? Thanks!</span></div></body></html> ------=_001_NextPart562105536544_=------

2 1

Re: [ovirt-users] Users Digest, Vol 66, Issue 86
by algotech 10 Mar '17

10 Mar '17

--------------7FAFD472380337AE680E33F3 Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit It looks that any memory eager running and swap fails afterwards will write here who's response of this On 03/10/2017 01:56 PM, users-request(a)ovirt.org wrote: > Send Users mailing list submissions to > users(a)ovirt.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.ovirt.org/mailman/listinfo/users > or, via email, send a message with subject or body 'help' to > users-request(a)ovirt.org > > You can reach the person managing the list at > users-owner(a)ovirt.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Users digest..." > > > Today's Topics: > > 1. Error on Node upgrade 2 (FERNANDO FREDIANI) > 2. Re: oVirt VM backup and restore (Gianluca Cecchi) > 3. Re: VDSM hang (Nir Soffer) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 10 Mar 2017 09:37:00 -0300 > From: FERNANDO FREDIANI <fernando.frediani(a)upx.com> > To: "users(a)ovirt.org" <users(a)ovirt.org> > Subject: [ovirt-users] Error on Node upgrade 2 > Message-ID: <3a306136-25aa-4533-4267-f1b8be58fd87(a)upx.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > I am not sure if another email I sent went through but has anyone got > problems when upgrading a running oVirt-node-ng from 4.1.0 to 4.1.1. > > Is the only solution a complete reinstall of the node ? > > Thanks > > Fernando > > > > ------------------------------ > > Message: 2 > Date: Fri, 10 Mar 2017 13:52:35 +0100 > From: Gianluca Cecchi <gianluca.cecchi(a)gmail.com> > To: Juan Hern?ndez <jhernand(a)redhat.com> > Cc: "users(a)ovirt.org" <users(a)ovirt.org> > Subject: Re: [ovirt-users] oVirt VM backup and restore > Message-ID: > <CAG2kNCwA3-jubtjcz=bzwX+V5-a5V85=HKi5pr=447Ad-tRD4w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Thu, Mar 9, 2017 at 11:23 AM, Juan Hern?ndez <jhernand(a)redhat.com> wrote: > >>> Very good point Gialuca, you are right, the 'persist_memorystate' >> flag >>> is 'true' by default, and that makes the pause longer. Would you be >> so >>> kind to add it to the 'vm_backup.py' example that is part of version >> 4 >>> of the SDK? >>> >>> >>> https://github.com/oVirt/ovirt-engine-sdk/blob/master/ >> sdk/examples/vm_backup.py#L143-L151 >>> <https://github.com/oVirt/ovirt-engine-sdk/blob/master/ >> sdk/examples/vm_backup.py#L143-L151> >>> (Note that that Gibhub is just a mirror, the change needs to be >> submited >>> using gerrit.ovirt.org <http://gerrit.ovirt.org>). >>> >>> >>> >>> I already verified (on a 4.1 infra) that changing vm_backup.py >>> downloaded yesterday from master this way (apart connection paramters): >>> >>> $ diff vm_backup.py vm_backup.py.orig >>> 150d149 >>> < persist_memorystate=False, >>> >>> I get the backup result and snapshot is correctly without memory saved >>> (and no pause at VM side) >>> >> [snip] >> I see that you already have a gerrit.ovirt.org, so it shouldn't be >> difficult. The initial setup should be like this: >> >> $ git config --global user.name youruser >> $ git config --global user.email your@email >> $ git clone ssh://youruser@gerrit.ovirt.org:29418/ovirt-engine-sdk >> $ gitdir=$(git rev-parse --git-dir); scp -p -P 29418 >> youruser@gerrit.ovirt.org:hooks/commit-msg ${gitdir}/hooks/ >> >> Then, to submit the patch: >> >> $ cd ovirt-engine-sdk >> $ Edit the vm_backup.py file, and make your changes. >> $ git commit -a -s >> $ git push origin HEAD:refs/for/master >> >> > Ok. I found the time to try and apparently it worked as expected. > The master (you? jenkins CI? ;-) should see my change... > Just learnt (a little...) another thing > > sys admins often try to put an eye inside devs field but the reverse seldom > happens ;-) >

1 0

PHX outage report 10.03.2017
by Evgheni Dereveanchin 10 Mar '17

10 Mar '17

Hi everyone, Tonight we experienced a hardware fault on one of our PHX storage servers. The faulty server was used to provide storage for multiple production VMs. Since automatic failover did not happen they became unavailable. The outage lasted between 09.03.2017 20:36 UTC and 10.03.2017 09:15 UTC Unavailable services included all of oVirt's CI infrastructure, mailing lists and package repositories. Services in other datecenters such as gerrit.ovirt.org and www.ovirt.org were not affected. We brought storage back up and this allowed for VMs to be restarted. If you see tests that failed or didn't run during this period please re-trigger them. If there are still persisting issues please report them to the tracker ticket that has a more detailed root cause analysis: https://ovirt-jira.atlassian.net/browse/OVIRT-1244 Sorry for the inconvenience caused. We are working on improving reliability of the environment to avoid similar incidents from happening in the future. -- Regards, Evgheni Dereveanchin

1 0

oVirt VM backup and restore
by raphael awadallah 10 Mar '17

10 Mar '17

Dears Engineers, I hope you are doing great, I wish you can help me with a tutorial that shows how to take a scheduled backups for virtual machines running on oVirt node 4 and how to restore them. Best regards

8 18

Re: [ovirt-users] Unable to add permissions for LDAP users
by Richard Neuboeck 10 Mar '17

10 Mar '17

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --2LvNVti3HDOn1MhIrbe5B2snDhjNOnQPn Content-Type: multipart/mixed; boundary="cUgn38JlIV4VKxpoXvonH2a8Rp00918Nu"; protected-headers="v1" From: Richard Neuboeck <hawk(a)tbi.univie.ac.at> To: users <users(a)ovirt.org> Message-ID: <ce3a6463-b08d-6af6-8adc-66f236ce0bed(a)tbi.univie.ac.at> Subject: Re: [ovirt-users] Unable to add permissions for LDAP users References: <71057a13-ee3d-aba0-b005-9fcbc906f375(a)tbi.univie.ac.at> In-Reply-To: <71057a13-ee3d-aba0-b005-9fcbc906f375(a)tbi.univie.ac.at> --cUgn38JlIV4VKxpoXvonH2a8Rp00918Nu Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 03/10/2017 09:46 AM, Ondra Machacek wrote: > So what's your provider 389ds or FreeIPA? >=20 > Note that both use differrent unique ID. IPA is using 'ipaUniqueID', > and 389ds is using 'nsuniqueid'. DId you tried both? Thanks for pointing that out! It works perfectly if I use IPA. I didn't know they have different identifiers (though it might have been obvious to me since there is a separate IPA option...). I clung to the thought that FreeIPA uses 389ds internally. Thanks a lot! Richard >=20 > I can successfully run a search and also login > from the setup script. >=20 > After running the setup I rebootet the Engine VM to make sure > everything is restarted. >=20 > In the web UI configuration for 'System Permissions' I'm able to > find users from LDAP but when I try to 'Add' a selected user the UI= > shows me this error: 'User admin@internal-authz failed to grant > permission for Role SuperUser on System to User/Group <UNKNOWN>.'. >=20 > In then engine.log the following lines are generated: > 2017-03-09 14:02:49,308+01 INFO > [org.ovirt.engine.core.bll.AddSystemPermissionCommand] > (org.ovirt.thread.pool-6-thread-4) > [1ebae5e0-e5f6-49ba-ac80-95266c582893] Running command: > AddSystemPermissionCommand internal: false. Entities affected : ID= : > aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group > MANIPULATE_PERMISSIONS with role type USER, ID: > aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group > ADD_USERS_AND_GROUPS_FROM_DIRECTORY with role type USER > 2017-03-09 14:02:49,319+01 ERROR > [org.ovirt.engine.core.bll.AddSystemPermissionCommand] > (org.ovirt.thread.pool-6-thread-4) > [1ebae5e0-e5f6-49ba-ac80-95266c582893] Transaction rolled-back for > command 'org.ovirt.engine.core.bll.AddSystemPermissionCommand'. > 2017-03-09 14:02:49,328+01 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirect= or] > (org.ovirt.thread.pool-6-thread-4) > [1ebae5e0-e5f6-49ba-ac80-95266c582893] EVENT_ID: > USER_ADD_SYSTEM_PERMISSION_FAILED(867), Correlation ID: > 1ebae5e0-e5f6-49ba-ac80-95266c582893, Call Stack: null, Custom Even= t > ID: -1, Message: User admin@internal-authz failed to grant > permission for Role SuperUser on System to User/Group <UNKNOWN>. >=20 >=20 > So far I've re-run the ldap-setup routine. I made sure all newly > generated files in /etc/ovirt-engine/[aaa|extensions.d] are owned b= y > ovirt:ovirt (instead of root) and have 0600 as permission (instead > of 0644). That didn't change anything. >=20 > I've also found an older bug report but for oVirt 3.5 > https://bugzilla.redhat.com/show_bug.cgi?id=3D1121954 > <https://bugzilla.redhat.com/show_bug.cgi?id=3D1121954> > That didn't reveal any new either. >=20 > Any ideas what I could try next? >=20 > Thanks! > Cheers > Richard >=20 >=20 >=20 >=20 > On 10/06/2016 04:36 PM, Ondra Machacek wrote: > > On 10/06/2016 01:47 PM, Michael Burch wrote: > >> I'm using the latest ovirt on CentOS7 with the aaa-ldap > extension. > >> I can > >> successfully authenticate as an LDAP user. I can also login as > >> admin@internal and search for, find, and select LDAP users but I= > >> cannot > >> add permissions for them. Each time I get the error "User > >> admin@internal-authz failed to grant permission for Role > UserRole on > >> System to User/Group <UNKNOWN>." > > > > This error usually means bad unique attribute used. > > > >> > >> > >> I have no control over the LDAP server, which uses custom > >> objectClasses > >> and uses groupOfNames instead of PosixGroups. I assume I need > to set > >> sequence variables to accommodate our group configuration but I'= m > >> at a > >> loss as to where to begin. the The config I have is as follows: > >> > >> > >> include =3D <rfc2307-generic.properties> > >> > >> vars.server =3D labauth.lan.lab.org <http://labauth.lan.lab.org>= > >> > >> pool.authz.auth.type =3D none > >> pool.default.serverset.type =3D single > >> pool.default.serverset.single.server =3D ${global:vars.server} > >> pool.default.ssl.startTLS =3D true > >> pool.default.ssl.insecure =3D true > >> > >> pool.default.connection-options.connectTimeoutMillis =3D 10000 > >> pool.default.connection-options.responseTimeoutMillis =3D 90000 > >> sequence-init.init.100-my-basedn-init-vars =3D my-basedn-init-va= rs > >> sequence.my-basedn-init-vars.010.description =3D set baseDN > >> sequence.my-basedn-init-vars.010.type =3D var-set > >> sequence.my-basedn-init-vars.010.var-set.variable =3D simple_bas= eDN > >> sequence.my-basedn-init-vars.010.var-set.value =3D o=3DLANLAB > >> > >> sequence-init.init.101-my-objectclass-init-vars =3D > >> my-objectclass-init-vars > >> sequence.my-objectclass-init-vars.020.description =3D set > objectClass > >> sequence.my-objectclass-init-vars.020.type =3D var-set > >> sequence.my-objectclass-init-vars.020.var-set.variable =3D > >> simple_filterUserObject > >> sequence.my-objectclass-init-vars.020.var-set.value =3D > >> (objectClass=3DlabPerson)(uid=3D*) > >> > >> search.default.search-request.derefPolicy =3D NEVER > >> > >> sequence-init.init.900-local-init-vars =3D local-init-vars > >> sequence.local-init-vars.010.description =3D override name space= > >> sequence.local-init-vars.010.type =3D var-set > >> sequence.local-init-vars.010.var-set.variable =3D > >> simple_namespaceDefault > >> sequence.local-init-vars.010.var-set.value =3D * > > > > What's this^ for? I think it's unusable. > > > >> > >> sequence.local-init-vars.020.description =3D apply filter to use= rs > >> sequence.local-init-vars.020.type =3D var-set > >> sequence.local-init-vars.020.var-set.variable =3D > >> simple_filterUserObject > >> sequence.local-init-vars.020.var-set.value =3D > >> ${seq:simple_filterUserObject}(employeeStatus=3D3) > >> > >> sequence.local-init-vars.030.description =3D apply filter to gro= ups > >> sequence.local-init-vars.030.type =3D var-set > >> sequence.local-init-vars.030.var-set.variable =3D > >> simple_filterGroupObject > >> sequence.local-init-vars.030.var-set.value =3D > >> (objectClass=3DgroupOfUniqueNames) > > > > This looks as hard to maintain file. I would suggest you to inser= t > > into this file just following: > > > > include =3D <rfc2307-mycustom.properties> > > > > vars.server =3D labauth.lan.lab.org <http://labauth.lan.lab.org>= > > > > pool.authz.auth.type =3D none > > pool.default.serverset.type =3D single > > pool.default.serverset.single.server =3D ${global:vars.server} > > pool.default.ssl.startTLS =3D true > > pool.default.ssl.insecure =3D true > > > > pool.default.connection-options.connectTimeoutMillis =3D 10000 > > pool.default.connection-options.responseTimeoutMillis =3D 90000 > > > > # Set custom base DN > > sequence-init.init.100-my-basedn-init-vars =3D my-basedn-init-va= rs > > sequence.my-basedn-init-vars.010.description =3D set baseDN > > sequence.my-basedn-init-vars.010.type =3D var-set > > sequence.my-basedn-init-vars.010.var-set.variable =3D simple_bas= eDN > > sequence.my-basedn-init-vars.010.var-set.value =3D o=3DLANLAB > > > > And then create in directory > > '/usr/share/ovirt-engine-extension-aaa-ldap/profiles/' file > > 'rfc2307-mycustom.properties' with content: > > > > include =3D <rfc2307.properties> > > > > sequence-init.init.100-rfc2307-mycustom-init-vars =3D > > rfc2307-mycustom-init-vars > > sequence.rfc2307-mycustom-init-vars.010.description =3D set > unique attr > > sequence.rfc2307-mycustom-init-vars.010.type =3D var-set > > sequence.rfc2307-mycustom-init-vars.010.var-set.variable =3D > > rfc2307_attrsUniqueId > > sequence.rfc2307-mycustom-init-vars.010.var-set.value =3D > FIND_THIS_ONE > > > > sequence.rfc2307-mycustom-init-vars.020.type =3D var-set > > sequence.rfc2307-mycustom-init-vars.020.var-set.variable =3D > > simple_filterUserObject > > sequence.rfc2307-mycustom-init-vars.020.var-set.value =3D > > > (objectClass=3DlabPerson)(employeeStatus=3D3)(${seq:simple_attrsUse= rName}=3D*) > > > > > > > > The FIND_*THIS_ONE* replace with the unique attribute of > labPerson(I > > guess). It can be extended attribute(+,++). > > > > $ LDAPTLS_REQCERT=3Dnever ldapsearch -ZZ -x -b 'o=3DLANLAB' -H > > ldap://labauth.lan.lab.org <http://labauth.lan.lab.org> > 'objectClass=3DlabPerson' > > > > maybe (or even with two +): > > $ LDAPTLS_REQCERT=3Dnever ldapsearch -ZZ -x -b 'o=3DLANLAB' -H > > ldap://labauth.lan.lab.org <http://labauth.lan.lab.org> > 'objectClass=3DlabPerson' + > > > > The question is if even your implementation has unique > attribute, does > > it? > > > > Also may you share what's your LDAP provider? And maybe if you > share > > content of some user it would help as well. > > > >> > >> > >> > >> > >> _______________________________________________ > >> Users mailing list > >> Users(a)ovirt.org <mailto:Users@ovirt.org> > >> http://lists.ovirt.org/mailman/listinfo/users > <http://lists.ovirt.org/mailman/listinfo/users> > >> > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org <mailto:Users@ovirt.org> > > http://lists.ovirt.org/mailman/listinfo/users > <http://lists.ovirt.org/mailman/listinfo/users> >=20 >=20 > -- > /dev/null >=20 >=20 --=20 /dev/null --cUgn38JlIV4VKxpoXvonH2a8Rp00918Nu-- --2LvNVti3HDOn1MhIrbe5B2snDhjNOnQPn Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCgAGBQJYwnegAAoJEA7XCanqEVqIODIQAM4IumlowJYvAoH/+vVTaUT1 pxzYVsOFYHhs32f1in9aqJ37esa1CXngUcB1gHm1QcBixq1FEH1iVEF1Iy22nyXM gNoOh4R7meoUzqpdlcqYpdTdo3F3v6KgWDRuz9yxH912MZzheXA9BGZ0qhdbQH3S a1eo2UHNm6wpyS+6CCgUirM8fbrjdb6aTGTuXNQ1px3GaqFFHkQvlWPlENatSxGw G3IqVoPHhuDNsFDyF6dG6oF2iTbQdzlvk84x3aOztX/nGnxV1dY00/73qEp+8aXK 0krTp0hjiGI0uRiOHChl0DSUuO9VvlTmmqjxYCN1z2ZYHZkMaK590v6YtXmT42tM VRIsqt6J5HpyLLiQWmCNPPkXf/s7iuG1zAmO+fi4vd5x6gQlW7ao6lLFFRqB8dVr FuJyv7DW6PcBNjNiG0zCLbGb431t2MK9AaSxF+qb8KmTluDBqQ2trdijM43J+3Jk /ptoXfpSzWmX2HkDLuaJL0NeQ6F5vqGUZRgu9IkXS2RAvQwLXflm/cvMKsj32EkZ 39gF5k1ViYAdtk9YUwBObor6uxnb0UpdrvZyA5dUEHsYaP6Yw+hVNqi0tloTN9wL Q+J6CdHK+lRkRppddoEdSmnAtBqhN5/woZEqrBZ1E1jFRUw7KTefbaeUpDbpWQvS 0csSL7D0IScsBJE2kLSA =73cu -----END PGP SIGNATURE----- --2LvNVti3HDOn1MhIrbe5B2snDhjNOnQPn--

1 0

Unable to add permissions for LDAP users
by Michael Burch 10 Mar '17

10 Mar '17

--_000_SN1PR10MB071807A20FF1DCCB62983C19D5C70SN1PR10MB0718namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I'm using the latest ovirt on CentOS7 with the aaa-ldap extension. I can su= ccessfully authenticate as an LDAP user. I can also login as admin@internal= and search for, find, and select LDAP users but I cannot add permissions f= or them. Each time I get the error "User admin@internal-authz failed to gra= nt permission for Role UserRole on System to User/Group <UNKNOWN>." I have no control over the LDAP server, which uses custom objectClasses and= uses groupOfNames instead of PosixGroups. I assume I need to set sequence = variables to accommodate our group configuration but I'm at a loss as to wh= ere to begin. the The config I have is as follows: include =3D <rfc2307-generic.properties> vars.server =3D labauth.lan.lab.org pool.authz.auth.type =3D none pool.default.serverset.type =3D single pool.default.serverset.single.server =3D ${global:vars.server} pool.default.ssl.startTLS =3D true pool.default.ssl.insecure =3D true pool.default.connection-options.connectTimeoutMillis =3D 10000 pool.default.connection-options.responseTimeoutMillis =3D 90000 sequence-init.init.100-my-basedn-init-vars =3D my-basedn-init-vars sequence.my-basedn-init-vars.010.description =3D set baseDN sequence.my-basedn-init-vars.010.type =3D var-set sequence.my-basedn-init-vars.010.var-set.variable =3D simple_baseDN sequence.my-basedn-init-vars.010.var-set.value =3D o=3DLANLAB sequence-init.init.101-my-objectclass-init-vars =3D my-objectclass-init-var= s sequence.my-objectclass-init-vars.020.description =3D set objectClass sequence.my-objectclass-init-vars.020.type =3D var-set sequence.my-objectclass-init-vars.020.var-set.variable =3D simple_filterUse= rObject sequence.my-objectclass-init-vars.020.var-set.value =3D (objectClass=3DlabP= erson)(uid=3D*) search.default.search-request.derefPolicy =3D NEVER sequence-init.init.900-local-init-vars =3D local-init-vars sequence.local-init-vars.010.description =3D override name space sequence.local-init-vars.010.type =3D var-set sequence.local-init-vars.010.var-set.variable =3D simple_namespaceDefault sequence.local-init-vars.010.var-set.value =3D * sequence.local-init-vars.020.description =3D apply filter to users sequence.local-init-vars.020.type =3D var-set sequence.local-init-vars.020.var-set.variable =3D simple_filterUserObject sequence.local-init-vars.020.var-set.value =3D ${seq:simple_filterUserObjec= t}(employeeStatus=3D3) sequence.local-init-vars.030.description =3D apply filter to groups sequence.local-init-vars.030.type =3D var-set sequence.local-init-vars.030.var-set.variable =3D simple_filterGroupObject sequence.local-init-vars.030.var-set.value =3D (objectClass=3DgroupOfUnique= Names) --_000_SN1PR10MB071807A20FF1DCCB62983C19D5C70SN1PR10MB0718namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none;"></style> </head> <body dir=3D"ltr"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;"> <p>I'm using the latest ovirt on CentOS7 with the aaa-ldap extension. I can= successfully authenticate as an LDAP user. I can also login as admin@inter= nal and search for, find, and select LDAP users but I cannot add permission= s for them. Each time I get the error "<span>User admin@internal-authz failed to grant permission for= Role UserRole on System to User/Group <UNKNOWN>.</span>"</p> <p><br> </p> <p>I have no control over the LDAP server, which uses custom objectClasses = and uses groupOfNames instead of PosixGroups. I assume I need to set sequen= ce variables to accommodate our group configuration but I'm at a loss as to= where to begin. the The config I have is as follows:</p> <p><br> </p> <p></p> <div>include =3D <rfc2307-generic.properties><br> <br> vars.server =3D labauth.lan.lab.org<br> <br> pool.authz.auth.type =3D none<br> pool.default.serverset.type =3D single<br> pool.default.serverset.single.server =3D ${global:vars.server}<br> pool.default.ssl.startTLS =3D true<br> pool.default.ssl.insecure =3D true<br> <br> pool.default.connection-options.connectTimeoutMillis =3D 10000<br> pool.default.connection-options.responseTimeoutMillis =3D 90000<br> sequence-init.init.100-my-basedn-init-vars =3D my-basedn-init-vars<br> sequence.my-basedn-init-vars.010.description =3D set baseDN<br> sequence.my-basedn-init-vars.010.type =3D var-set<br> sequence.my-basedn-init-vars.010.var-set.variable =3D simple_baseDN<br> sequence.my-basedn-init-vars.010.var-set.value =3D o=3DLANLAB<br> <br> sequence-init.init.101-my-objectclass-init-vars =3D my-objectclass-init-var= s<br> sequence.my-objectclass-init-vars.020.description =3D set objectClass<br> sequence.my-objectclass-init-vars.020.type =3D var-set<br> sequence.my-objectclass-init-vars.020.var-set.variable =3D simple_filterUse= rObject<br> sequence.my-objectclass-init-vars.020.var-set.value =3D (objectClass=3DlabP= erson)(uid=3D*)<br> <br> search.default.search-request.derefPolicy =3D NEVER<br> <br> sequence-init.init.900-local-init-vars =3D local-init-vars<br> sequence.local-init-vars.010.description =3D override name space<br> sequence.local-init-vars.010.type =3D var-set<br> sequence.local-init-vars.010.var-set.variable =3D simple_namespaceDefault<b= r> sequence.local-init-vars.010.var-set.value =3D *<br> <br> sequence.local-init-vars.020.description =3D apply filter to users<br> sequence.local-init-vars.020.type =3D var-set<br> sequence.local-init-vars.020.var-set.variable =3D simple_filterUserObject<b= r> sequence.local-init-vars.020.var-set.value =3D ${seq:simple_filterUserObjec= t}(employeeStatus=3D3)<br> <br> sequence.local-init-vars.030.description =3D apply filter to groups<br> sequence.local-init-vars.030.type =3D var-set<br> sequence.local-init-vars.030.var-set.variable =3D simple_filterGroupObject<= br> sequence.local-init-vars.030.var-set.value =3D (objectClass=3DgroupOfUnique= Names)<br> <br> <br> </div> <p></p> </div> </body> </html> --_000_SN1PR10MB071807A20FF1DCCB62983C19D5C70SN1PR10MB0718namp_--

3 3

HE vm fails to migrate due to host CPU incompatibility even after changing cluster CPU type
by Stefano Stagnaro 09 Mar '17

09 Mar '17

Hi guys, I've started an oVirt 4.1 HE deployment on a Broadwell based server. Then I added to HE a second, older host based on Nehalem. I've downgraded the cluster CPU type to Nehalem to accommodate host2 and it finally reached score 3400. However, when I try to migrate HE vm it fails with the following error: 2017-03-03 20:19:51,814 ERROR (migsrc/b0d38435) [virt.vm] (vmId='b0d38435-5774-4ca9-ad24-70b57b5bc25d') unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: pclmuldq, fma, pcid, x2apic, movbe, tsc-deadline, aes, xsave, avx, fsgsbase, bmi1, hle, avx2, smep, bmi2, erms, invpcid, rtm, rdseed, adx, smap, 3dnowprefetch; try using 'Broadwell-noTSX' CPU model (migration:265) I believe the problem is in the HE vm XML where the cpu is still configured as Broadwell. how can I change this specific setting without losing the deployment? Please find all the relevant logs at the following link: https://www.dropbox.com/sh/njl9aofhdw10ses/AADf2Ql4GKVIKcbgLivbmjC2a Besides that, I believe this is a wrong behavior because HE should follow cluster properties (otherwise do not reach score 3400); do you believe is it worth opening a issue on bugzilla? Thank you, Stefano.

3 3

cannot exceed maximum memory size
by Marcelo Leandro 09 Mar '17

09 Mar '17

Hello , I am have an script that create VM , after upgrade to 4.1 I have this erro when trying create a new VM: detail: Cannot add VM. Memory size (20480MB) cannot exceed maximum memory size (4096MB). my commando fore create VM : api.vms.add(params.VM(name="teste", memory=memo , cpu=cpu, cluster=api.clusters.get(cluster), template=api.templates.get('Blank'), type_="server")) I dont how specific the maximum memory size. Thanks Marcelo Leandro

6 8

DWH reports in 4.X ?
by Eduardo Mayoral 09 Mar '17

09 Mar '17

Hi, I see the DWH reports disappeared in 3.5 or maybe 3.6, there seems to be no substitute por 4.X Is there some alternative to have some pre-made reports from the DWH on 4.X ? Maybe not something on the oVirt project itself but from someone on the community? Thanks in advance! -- Eduardo Mayoral Jimeno (emayoral(a)arsys.es) Administrador de sistemas. Departamento de Plataformas. Arsys internet. +34 941 620 145 ext. 5153

2 3

[ANN] oVirt 4.1.1 Second Release Candidate
by Sandro Bonazzola 09 Mar '17

09 Mar '17

The oVirt Project is pleased to announce the availability of the Second Release Candidate of oVirt 4.1.1 for testing, as of March 9th, 2017 This is pre-release software. Please take a look at our community page[1] to know how to ask questions and interact with developers and users. All issues or bugs should be reported via oVirt Bugzilla[2]. This pre-release should not to be used in production. This update is the second release candidate of the first in a series of stabilization updates to the 4.1 series. 4.1.1 brings 32 enhancements and more than 370 bugfixes, including more than 150 high or urgent severity fixes, on top of oVirt 4.1 series This release is available now for: * Fedora 24 (tech preview) * Red Hat Enterprise Linux 7.3 or later * CentOS Linux (or similar) 7.3 or later This release supports Hypervisor Hosts running: * Red Hat Enterprise Linux 7.3 or later * CentOS Linux (or similar) 7.3 or later * Fedora 24 (tech preview) See the release notes draft [3] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Live and oVirt node have been already built [4] - oVirt Engine Appliance had been already built Additional Resources: * Read more about the oVirt 4.1.1 release highlights: http://www.ovirt.org/release/4.1.1/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] https://www.ovirt.org/community/ [2] https://bugzilla.redhat.com/enter_bug.cgi?classification=oVirt [3] http://www.ovirt.org/release/4.1.1/ [4] http://resources.ovirt.org/pub/ovirt-4.1-pre/iso/ -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

"patched upgrade" from 4.0 (not appliance) => 4.1 (appliance) fails
by Nelson Lameiras 09 Mar '17

09 Mar '17

------=_Part_17575365_626819882.1489055431520 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello,=20 I'm trying to upgrade our ovirt-engine from 4.0 (centos manually installed)= to 4.1.0 (appliance) using "hosted-engine --upgrade-appliance".=20 Our test setup :=20 2 hosts with centos 7.3 with ovirt4.0 (latest updates)=20 1 ovirt-engine 4.0 (latest update)=20 This method uses a manual backup from current engine and injects it to new = appliance. Since "old" ovirt-engine is a manually installed 4.0 (latest ver= sion) as opposed to an appliance, upgrade process warns that it doesn't rec= ognise the current engine with a somewhat deprecated message :=20 [WARNING] Unable to detect engine version=20 [WARNING] Unsupported upgrade path=20 This procedure has been designed and tested only for upgrading the engine V= M from ['3.6'] to ['4.0'].=20 Any other usage is highly experimental and potentially dangerous:=20 Current engine: unknown=20 Selected appliance: 4.1-20170201.1.el7.centos=20 Do you want to abort the upgrade process? (Yes, No) [Yes]: no=20 [WARNING] Proceeding on an unsupported and highly experimental path=20 This fails, according to a bug already posted on https://bugzilla.redhat.co= m/show_bug.cgi?id=3D1420283 which is resolved in future 4.1.1=20 So I tried to manually patch the file corrected in gerrit https://gerrit.ov= irt.org/#/c/71965/ and - after multiple tries - it still fails at the end w= ith the following error :=20 ...=20 [ INFO ] Running engine-setup on the appliance=20 |- Preparing to restore:=20 |- - Unpacking file '/root/engine_backup.tar.gz'=20 |- FATAL: Backup was created by version '4.0' and can not be restored using= the installed version 4.1=20 |- HE_APPLIANCE_ENGINE_RESTORE_FAIL=20 [ ERROR ] Engine backup restore failed on the appliance=20 [ ERROR ] Failed to execute stage 'Closing up': engine-backup failed restor= ing the engine backup on the appliance Please check its log on the applianc= e.=20 When checking upgrade log, there's not much more error information other th= an :=20 2017-03-08 18:08:30 INFO otopi.plugins.gr_he_common.engine.health health._c= loseup:127 Running engine-setup on the appliance=20 2017-03-08 18:08:44 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:204 DIALOG:SEND |- Preparing to restore:=20 2017-03-08 18:08:44 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:204 DIALOG:SEND |- - Unpacking file '/root/engine_backup.tar.gz'=20 2017-03-08 18:08:44 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:204 DIALOG:SEND |- FATAL: Backup was created by version '4.0' and can no= t be restored using the installed version 4.1=20 2017-03-08 18:08:44 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:204 DIALOG:SEND |- HE_APPLIANCE_ENGINE_RESTORE_FAIL=20 2017-03-08 18:08:44 ERROR otopi.plugins.gr_he_common.engine.health health._= closeup:154 Engine backup restore failed on the appliance=20 2017-03-08 18:08:44 DEBUG otopi.context context._executeMethod:142 method e= xception=20 Traceback (most recent call last):=20 File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _exe= cuteMethod=20 method['method']()=20 File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-common/= engine/health.py", line 158, in _closeup=20 'engine-backup failed restoring the engine backup '=20 RuntimeError: engine-backup failed restoring the engine backup on the appli= ance=20 Please check its log on the appliance.=20 2017-03-08 18:08:44 ERROR otopi.context context._executeMethod:151 Failed t= o execute stage 'Closing up': engine-backup failed restoring the engine bac= kup on the appliance=20 Please check its log on the appliance.=20 2017-03-08 18:08:44 DEBUG otopi.context context.dumpEnvironment:760 ENVIRON= MENT DUMP - BEGIN=20 In "new" engine setup log, the only thing which catched my eye is :=20 2017-03-08 18:08:45 DEBUG otopi.context context.dumpEnvironment:774 ENVIRON= MENT DUMP - END=20 2017-03-08 18:08:45 DEBUG otopi.context context._executeMethod:128 Stage bo= ot METHOD otopi.plugins.otopi.packagers.dnfpackager.Plugin._boot=20 2017-03-08 18:08:45 DEBUG otopi.plugins.otopi.packagers.dnfpackager dnfpack= ager._boot:163 Cannot initialize minidnf=20 Traceback (most recent call last):=20 File "/usr/share/otopi/plugins/otopi/packagers/dnfpackager.py", line 150, i= n _boot=20 constants.PackEnv.DNF_DISABLED_PLUGINS=20 File "/usr/share/otopi/plugins/otopi/packagers/dnfpackager.py", line 60, in= _getMiniDNF=20 from otopi import minidnf=20 File "/usr/lib/python2.7/site-packages/otopi/minidnf.py", line 16, in <modu= le>=20 import dnf=20 ImportError: No module named dnf=20 2017-03-08 18:08:45 DEBUG otopi.context context.dumpEnvironment:760 ENVIRON= MENT DUMP - BEGIN=20 I'm not sure if my patched file ( runvm.py ) should be enough to make it wo= rk, but maybe this can bring some new information to you.=20 Really hopping that 4.1.1 oVirt will resolve this issue as we are eager to = start using appliances.=20 (I can provide you with more/full log files if necessary)=20 cordialement, regards,=20 =09Nelson LAMEIRAS=20 Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee= r=20 Tel: +33 5 32 09 09 70=20 nelson.lameiras(a)lyra-network.com=20 www.lyra-network.com | www.payzen.eu=20 =09 =09 =09 Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20 ------=_Part_17575365_626819882.1489055431520 Content-Type: multipart/related; boundary="----=_Part_17575366_2065010947.1489055431520" ------=_Part_17575366_2065010947.1489055431520 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: arial, helvetica, sans-serif; font-s= ize: 12pt; color: #000000"><div>Hello,<br></div><div><br data-mce-bogus=3D"= 1"></div><div>I'm trying to upgrade our ovirt-engine from 4.0 (centos manua= lly installed) to 4.1.0 (appliance) using "hosted-engine --upgrade-applianc= e". </div><div><br data-mce-bogus=3D"1"></div><div>Our test setup : </div><= div>2 hosts with centos 7.3 with ovirt4.0 (latest updates)<br data-mce-bogu= s=3D"1"></div><div>1 ovirt-engine 4.0 (latest update)<br data-mce-bogus=3D"= 1"></div><div><br data-mce-bogus=3D"1"></div><div>This method uses a manual= backup from current engine and injects it to new appliance. Since "old" ov= irt-engine is a manually installed 4.0 (latest version) as opposed to an ap= pliance, upgrade process warns that it doesn't recognise the current engine= with a somewhat deprecated message :<br data-mce-bogus=3D"1"></div><div><b= r data-mce-bogus=3D"1"></div><div>[WARNING] Unable to detect engine version= <br>[WARNING] Unsupported upgrade path<br>     &nb= sp;    This procedure has been designed and tested only for = upgrading the engine VM from ['3.6'] to ['4.0'].<br>   &nbsp= ;      Any other usage is highly experimental and = potentially dangerous:<br>        &= nbsp;   Current engine: unknown<br>     =        Selected appliance: 4.1-20170201.1.el7= .centos<br>          Do you wa= nt to abort the upgrade process? (Yes, No) [Yes]: no<br>[WARNING] Proceedin= g on an unsupported and highly experimental path<br></div><div><br data-mce= -bogus=3D"1"></div><div>This fails, according to a bug already posted on ht= tps://bugzilla.redhat.com/show_bug.cgi?id=3D1420283 which is resolved in fu= ture 4.1.1<br data-mce-bogus=3D"1"></div><div><br data-mce-bogus=3D"1"></di= v><div>So I tried to manually patch the file corrected in gerrit https://ge= rrit.ovirt.org/#/c/71965/ and - after multiple tries - it still fails at th= e end with the following error : <br data-mce-bogus=3D"1"></div><div><br da= ta-mce-bogus=3D"1"></div><div>...<br data-mce-bogus=3D"1"></div><div>[ INFO=   ] Running engine-setup on the appliance<br>    &= nbsp;     |- Preparing to restore:<br>  &nbsp= ;       |- - Unpacking file '/root/engine_bac= kup.tar.gz'<br>          |- FA= TAL: Backup was created by version '4.0' and can not be restored using the = installed version 4.1<br>        &n= bsp; |- HE_APPLIANCE_ENGINE_RESTORE_FAIL<br>[ ERROR ] Engine backup restore= failed on the appliance<br>[ ERROR ] Failed to execute stage 'Closing up':= engine-backup failed restoring the engine backup on the appliance Please c= heck its log on the appliance.</div><div><br data-mce-bogus=3D"1"></div><di= v>When checking upgrade log, there's not much more error information other = than :<br data-mce-bogus=3D"1"></div><div><br data-mce-bogus=3D"1"></div><d= iv>2017-03-08 18:08:30 INFO otopi.plugins.gr_he_common.engine.health health= ._closeup:127 Running engine-setup on the appliance<br>2017-03-08 18:08:44 = DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND&n= bsp;           &nbsp= ;    |- Preparing to restore:<br>2017-03-08 18:08:44 DEBUG o= topi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND &nb= sp;            =    |- - Unpacking file '/root/engine_backup.tar.gz'<br>2017-03-08= 18:08:44 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIA= LOG:SEND           &= nbsp;     |- FATAL: Backup was created by version '4.0'= and can not be restored using the installed version 4.1<br>2017-03-08 18:0= 8:44 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:S= END            =      |- HE_APPLIANCE_ENGINE_RESTORE_FAIL<br>2017-03-08 = 18:08:44 ERROR otopi.plugins.gr_he_common.engine.health health._closeup:154= Engine backup restore failed on the appliance<br>2017-03-08 18:08:44 DEBUG= otopi.context context._executeMethod:142 method exception<br>Traceback (mo= st recent call last):<br>  File "/usr/lib/python2.7/site-packages/otop= i/context.py", line 132, in _executeMethod<br>    method['me= thod']()<br>  File "/usr/share/ovirt-hosted-engine-setup/scripts/../pl= ugins/gr-he-common/engine/health.py", line 158, in _closeup<br>  =   'engine-backup failed restoring the engine backup '<br>RuntimeError:= engine-backup failed restoring the engine backup on the appliance<br>Pleas= e check its log on the appliance.<br>2017-03-08 18:08:44 ERROR otopi.contex= t context._executeMethod:151 Failed to execute stage 'Closing up': engine-b= ackup failed restoring the engine backup on the appliance<br>Please check i= ts log on the appliance.<br>2017-03-08 18:08:44 DEBUG otopi.context context= .dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN<br></div><div><br data-mce-bo= gus=3D"1"></div><div>In "new" engine setup log, the only thing which catche= d my eye is :<br data-mce-bogus=3D"1"></div><div><br data-mce-bogus=3D"1"><= /div><div>2017-03-08 18:08:45 DEBUG otopi.context context.dumpEnvironment:7= 74 ENVIRONMENT DUMP - END<br>2017-03-08 18:08:45 DEBUG otopi.context contex= t._executeMethod:128 Stage boot METHOD otopi.plugins.otopi.packagers.dnfpac= kager.Plugin._boot<br>2017-03-08 18:08:45 DEBUG otopi.plugins.otopi.package= rs.dnfpackager dnfpackager._boot:163 Cannot initialize minidnf<br>Traceback= (most recent call last):<br>  File "/usr/share/otopi/plugins/otopi/pa= ckagers/dnfpackager.py", line 150, in _boot<br>    constants= .PackEnv.DNF_DISABLED_PLUGINS<br>  File "/usr/share/otopi/plugins/otop= i/packagers/dnfpackager.py", line 60, in _getMiniDNF<br>    = from otopi import minidnf<br>  File "/usr/lib/python2.7/site-packages/= otopi/minidnf.py", line 16, in <module><br>    import = dnf<br>ImportError: No module named dnf<br>2017-03-08 18:08:45 DEBUG otopi.= context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN<br></div><div>= <br data-mce-bogus=3D"1"></div><div>I'm not sure if my patched file (<a hre= f=3D"https://gerrit.ovirt.org/#/c/71965/3/src/plugins/gr-he-upgradeapplianc= e/vm/runvm.py" onclick=3D"gwt_uid_92(event,1)" data-mce-href=3D"https://ger= rit.ovirt.org/#/c/71965/3/src/plugins/gr-he-upgradeappliance/vm/runvm.py">r= unvm.py</a>) should be enough to make it work, but maybe this can bring som= e new information to you.<br data-mce-bogus=3D"1"></div><div>Really hopping= that 4.1.1 oVirt will resolve this issue as we are eager to start using ap= pliances.<br data-mce-bogus=3D"1"></div><div><br></div><div>(I can provide = you with more/full log files if necessary)<br data-mce-bogus=3D"1"></div><d= iv><br data-mce-bogus=3D"1"></div><div data-marker=3D"__SIG_PRE__"><div>cor= dialement, regards,</div><div><br data-mce-bogus=3D"1"></div><table style= =3D"margin: 0px; table-layout: fixed; border: none; width: 345pt; padding: = 0px;" data-mce-style=3D"margin: 0px; table-layout: fixed; border: none; wid= th: 345pt; padding: 0px;" width=3D"460" cellspacing=3D"0" cellpadding=3D"0"= border=3D"0"><tbody><tr><td style=3D"margin: 0; font-size: 11px; font-fami= ly: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 86.25pt; te= xt-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-family:= Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 86.25pt; text-= align: left;" width=3D"115" valign=3D"top"><a href=3D"https://www.lyra-netw= ork.com/" style=3D"border: none; text-decoration: none; width: 100%; displa= y: block; clear: both;" data-mce-href=3D"https://www.lyra-network.com/" dat= a-mce-style=3D"border: none; text-decoration: none; width: 100%; display: b= lock; clear: both;"><img src=3D"cid:2d73caa90af1eb7aeb4cb1bde4b870be1e954fc= 6@zimbra" alt=3D"" data-mce-src=3D"/home/admin(a)lyra-network.com/corporate-d= ocuments/element-signature_logo_lyra_115x94.jpg" doc=3D"corporate-documents= /element-signature_logo_lyra_115x94.jpg" height=3D"94" width=3D"115" border= =3D"0"></a><br data-mce-bogus=3D"1"></td><td style=3D"margin: 0; font-size:= 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; wi= dth: 258.75pt;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 258.75pt;" widt= h=3D"345" valign=3D"top"><table style=3D"margin: 0; width: 258.75pt; table-= layout: fixed; border: none;" data-mce-style=3D"margin: 0; width: 258.75pt;= table-layout: fixed; border: none;" width=3D"345" cellspacing=3D"0" cellpa= dding=3D"0" border=3D"0"><tbody><tr><td style=3D"font-size: 16px; font-fami= ly: Arial,sans-serif; color: #2b3c7a; line-height: 18px; font-weight: bold;= " data-mce-style=3D"font-size: 16px; font-family: Arial,sans-serif; color: = #2b3c7a; line-height: 18px; font-weight: bold;">Nelson LAMEIRAS</td></tr><t= r><td style=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11= px; line-height: 16px;" data-mce-style=3D"font-family: Arial,sans-serif; co= lor: #2b3c7a; font-size: 11px; line-height: 16px;">Ing=C3=A9nieur Syst=C3= =A8mes et R=C3=A9seaux<span style=3D"font-family: Arial,sans-serif; color: = #337ab7; font-size: 11px;" data-mce-style=3D"font-family: Arial,sans-serif;= color: #337ab7; font-size: 11px;"> / Systems and Networks engineer</span><= /td></tr><tr><td style=3D"line-height: 16px; font-family: 'Arial',sans-seri= f; color: #2b3c7a; font-size: 11px;" data-mce-style=3D"line-height: 16px; f= ont-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;"><span sty= le=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px;" data= -mce-style=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11p= x;">Tel: +33 5 32 09 09 70</span></td></tr><tr><td style=3D"line-height: 16= px; font-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data= -mce-style=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2= b3c7a; font-size: 11px;"><a href=3D"mailto:nelson.lameiras@lyra-network.com= " style=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: 16p= x; text-decoration: none;" data-mce-href=3D"mailto:nelson.lameiras@lyra-net= work.com" data-mce-style=3D"font-size: 11px; font-family: Arial,sans-serif;= line-height: 16px; text-decoration: none;">nelson.lameiras(a)lyra-network.co= m</a><br data-mce-bogus=3D"1"></td></tr><tr><td style=3D"line-height: 16px;= font-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mc= e-style=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c= 7a; font-size: 11px;"><a href=3D"https://www.lyra-network.com/" style=3D"fo= nt-size: 11px; font-family: Arial,sans-serif; line-height: 16px; border: no= ne; text-decoration: none;" data-mce-href=3D"https://www.lyra-network.com/"= data-mce-style=3D"font-size: 11px; font-family: Arial,sans-serif; line-hei= ght: 16px; border: none; text-decoration: none;">www.lyra-network.com</a> <= a href=3D"https://payzen.eu" style=3D"font-size: 11px; font-family: Arial,s= ans-serif; line-height: 16px; border: none; text-decoration: none;" data-mc= e-href=3D"https://payzen.eu" data-mce-style=3D"font-size: 11px; font-family= : Arial,sans-serif; line-height: 16px; border: none; text-decoration: none;= "> | www.payzen.eu</a><br data-mce-bogus=3D"1"></td></tr></tbody></table></= td></tr></tbody></table><table style=3D"margin: 0; table-layout: fixed; bor= der: none; width: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed;= border: none; width: 345pt;" width=3D"460" cellspacing=3D"0" cellpadding= =3D"0"><tbody><tr><td style=3D"margin: 0; font-size: 11px; font-family: Ari= al,sans-serif; color: #2b3c7a; line-height: 16px; width: 24pt; text-align: = left;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Arial,san= s-serif; color: #2b3c7a; line-height: 16px; width: 24pt; text-align: left;"= width=3D"32" valign=3D"top"><a href=3D"https://www.youtube.com/channel/UCr= Vl1CO_Jlu3KbiRH-tQ_vA" style=3D"border: none; text-decoration: none; margin= : 0; float: left; display: inline-block; width: 32px;" data-mce-href=3D"htt= ps://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" data-mce-style=3D"bo= rder: none; text-decoration: none; margin: 0; float: left; display: inline-= block; width: 32px;"><img src=3D"cid:3ab05240a7d61766c65061d295bb6ab763776f= 35@zimbra" alt=3D"" data-mce-src=3D"/home/admin(a)lyra-network.com/corporate-= documents/element-signature_logo_YouTube_32x28.jpg" doc=3D"corporate-docume= nts/element-signature_logo_YouTube_32x28.jpg" height=3D"28" width=3D"32" bo= rder=3D"0"></a><br data-mce-bogus=3D"1"></td><td style=3D"margin: 0; font-s= ize: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px= ; width: 30.75pt; text-align: left;" data-mce-style=3D"margin: 0; font-size= : 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; w= idth: 30.75pt; text-align: left;" width=3D"41" valign=3D"top"><a href=3D"ht= tps://www.linkedin.com/company/lyra-network_2" style=3D"border: none; text-= decoration: none; margin: 0; float: left; display: inline-block; width: 41p= x;" data-mce-href=3D"https://www.linkedin.com/company/lyra-network_2" data-= mce-style=3D"border: none; text-decoration: none; margin: 0; float: left; d= isplay: inline-block; width: 41px;"><img src=3D"cid:8abb8ba852f9dac790947a0= 613276631ef5d9907@zimbra" data-mce-src=3D"/home/admin(a)lyra-network.com/corp= orate-documents/element-signature_logo_LinkedIn_41x28.jpg" doc=3D"corporate= -documents/element-signature_logo_LinkedIn_41x28.jpg" height=3D"28" width= =3D"41" border=3D"0"></a><br data-mce-bogus=3D"1"></td><td style=3D"margin:= 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-he= ight: 16px; width: 31.5pt; text-align: left;" data-mce-style=3D"margin: 0; = font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height= : 16px; width: 31.5pt; text-align: left;" width=3D"42" valign=3D"top"><a hr= ef=3D"https://twitter.com/LyraNetwork" style=3D"border: none; text-decorati= on: none; margin: 0; float: left; display: inline-block; width: 42px;" data= -mce-href=3D"https://twitter.com/LyraNetwork" data-mce-style=3D"border: non= e; text-decoration: none; margin: 0; float: left; display: inline-block; wi= dth: 42px;"><img src=3D"cid:ebd5093b575f672e757e59e7c978d7b5d1ecfef7@zimbra= " alt=3D"" data-mce-src=3D"/home/admin(a)lyra-network.com/corporate-documents= /element-signature_logo_Twitter_42x28.jpg" doc=3D"corporate-documents/eleme= nt-signature_logo_Twitter_42x28.jpg" height=3D"28" width=3D"42" border=3D"0= "></a><br data-mce-bogus=3D"1"></td><td style=3D"margin: 0; font-size: 11px= ; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: = 258.75pt; text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; = font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 25= 8.75pt; text-align: left;" width=3D"345" valign=3D"top"><a href=3D"https://= payzen.eu" style=3D"border: none; text-decoration: none; margin: 0; float: = left; display: inline-block; width: 61px;" data-mce-href=3D"https://payzen.= eu" data-mce-style=3D"border: none; text-decoration: none; margin: 0; float= : left; display: inline-block; width: 61px;"><img src=3D"cid:3c61f70c1921e2= a4d5c92140344a2f50072f670d@zimbra" alt=3D"" data-mce-src=3D"/home/admin@lyr= a-network.com/corporate-documents/element-signature_payzen_61x28.jpg" doc= =3D"corporate-documents/element-signature_payzen_61x28.jpg" height=3D"28" w= idth=3D"61" border=3D"0"></a><br data-mce-bogus=3D"1"></td></tr></tbody></t= able><table style=3D"margin: 0; table-layout: fixed; border: none; width: 3= 45pt;" data-mce-style=3D"margin: 0; table-layout: fixed; border: none; widt= h: 345pt;" width=3D"460" cellspacing=3D"0" cellpadding=3D"0"><tbody><tr><td= ><hr style=3D"background: #293b79 none repeat scroll 0% 0%; width: 100%; he= ight: 4px; color: #293b79; margin: 0px;" data-mce-style=3D"background: #293= b79 none repeat scroll 0% 0%; width: 100%; height: 4px; color: #293b79; mar= gin: 0px;"></td></tr></tbody></table><table style=3D"margin: 0; table-layou= t: fixed; border: none; width: 345pt;" data-mce-style=3D"margin: 0; table-l= ayout: fixed; border: none; width: 345pt;" width=3D"460" cellspacing=3D"0" = cellpadding=3D"0"><tbody><tr><td style=3D"margin: 0; font-size: 11px; font-= family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt= ; text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-fam= ily: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; t= ext-align: left;"><div>Lyra Network, 109 rue de l'innovation, 31670 Lab=C3= =A8ge, FRANCE</div></td></tr></tbody></table><div><br></div></div></div></b= ody></html> ------=_Part_17575366_2065010947.1489055431520 Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg Content-Disposition: attachment; filename=element-signature_logo_lyra_115x94.jpg Content-Transfer-Encoding: base64 Content-ID: <2d73caa90af1eb7aeb4cb1bde4b870be1e954fc6@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S 3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/ Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L +zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0 x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2 9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7 gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK 8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1 PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36 RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV +GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H 7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3 LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8 qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4 XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t 1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5 pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z ------=_Part_17575366_2065010947.1489055431520 Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg Content-Disposition: attachment; filename=element-signature_logo_YouTube_32x28.jpg Content-Transfer-Encoding: base64 Content-ID: <3ab05240a7d61766c65061d295bb6ab763776f35@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1 YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ 4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1 5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv DAAwAMAP/9k= ------=_Part_17575366_2065010947.1489055431520 Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg Content-Disposition: attachment; filename=element-signature_logo_LinkedIn_41x28.jpg Content-Transfer-Encoding: base64 Content-ID: <8abb8ba852f9dac790947a0613276631ef5d9907@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33 zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i 9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4 d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6 rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z ------=_Part_17575366_2065010947.1489055431520 Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg Content-Disposition: attachment; filename=element-signature_logo_Twitter_42x28.jpg Content-Transfer-Encoding: base64 Content-ID: <ebd5093b575f672e757e59e7c978d7b5d1ecfef7@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u 3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/ LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb 9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e +rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4 5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z ------=_Part_17575366_2065010947.1489055431520 Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg Content-Transfer-Encoding: base64 Content-ID: <3c61f70c1921e2a4d5c92140344a2f50072f670d@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc 2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b 3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8 Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q== ------=_Part_17575366_2065010947.1489055431520-- ------=_Part_17575365_626819882.1489055431520--

2 1

oVirt backup and VM pauses.
by Misak Khachatryan 09 Mar '17

09 Mar '17

Hello, I'm using oVirt and recently added backup functionality using this script: https://github.com/wefixit-AT/oVirtBackup This script runs every day from cron, but i noticed very bad behavior with the machines. Script on some stage pauses them, and it leads to problems. I have NTP stratum 2 server as a virtual machine, which I added to pool.ntp.org. When machine pauses, it loses time sync for a while and NTP.ORG monitoring system removes this machine from pool. And it take about a 24 hours to get good score to be included int the pool again, then next backup time arrives, and so on. So my question is - any suggestion to prevent this behavior or any other backup solution without pausing available? Best regards, Misak Khachatryan

3 3