Hi,<br>I shared the logs yesterday, please see my most recent post for the links.<br><br>Thanks<br>Frank<div class="quote" style="line-height: 1.5"><br><br>-------- Message original --------<br>Objet : Re: [ovirt-users] Re: VMs unexpectidly restarted<br>De : Sahina Bose <sabose@redhat.com><br>À : fsoyer@systea.fr<br>Cc : users <users@ovirt.org>,Gobinda Das <godas@redhat.com><br><br><br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Sun, Oct 28, 2018 at 5:17 PM fsoyer <fsoyer@systea.fr> wrote:<br>><br>><br>> Well guys,<br>> I can say now that I have a real problem, maybe between ovirt and gluster storage, but I can't be sure. Yesterday, I wanted to clone a VM (named "crij2") from a snapshot, but (this is another problem I think) the UI never gave me the popup (blank window with the cursor with a message 400 after a timeout). So I decided to export it, then import it.<br>> The export/import finally works, but when it was working, some VMs became randomly unresponsives, and one restarted on error. At this time, the engine was on "ginger" node. Copy of the event log :<br>> 27 oct. 2018 20:32:12 VM crij2 started on Host victor.local.systea.fr<br>> 27 oct. 2018 20:31:37 VM crij2 was started by admin@internal-authz (Host: victor.local.systea.fr).<br>> 27 oct. 2018 20:26:39 Vm crij2 was imported successfully to Data Center Default, Cluster Default<br>> 27 oct. 2018 20:22:53 VM logcollector is not responding.<br>> 27 oct. 2018 20:22:10 VM Sogov3 is not responding.<br>> 27 oct. 2018 20:17:53 VM cerbere4 is not responding.<br>> 27 oct. 2018 20:17:49 VM cerbere3 is not responding.<br>> 27 oct. 2018 20:17:48 VM logcollector is not responding.<br>> 27 oct. 2018 20:16:38 VM Sogov3 is not responding.<br>> 27 oct. 2018 20:16:38 VM cerbere4 is not responding.<br>> 27 oct. 2018 20:16:38 VM op2drugs1 is not responding.<br>> 27 oct. 2018 20:16:33 VM cerbere3 is not responding.<br>> 27 oct. 2018 20:07:30 VM op2drugs1 is not responding.<br>> 27 oct. 2018 20:06:14 VM cerbere3 is not responding.<br>> 27 oct. 2018 20:02:27 VM cerbere3 is not responding.<br>> 27 oct. 2018 20:01:11 VM logcollector is not responding.<br>> 27 oct. 2018 20:00:56 VM zabbix is not responding.<br>> 27 oct. 2018 19:57:42 VM zabbix is not responding.<br>> 27 oct. 2018 19:57:42 VM cerbere3 is not responding.<br>> 27 oct. 2018 19:57:42 VM logcollector is not responding.<br>> 27 oct. 2018 19:54:40 VM zabbix is not responding.<br>> 27 oct. 2018 19:53:25 VM cerbere3 is not responding.<br>> 27 oct. 2018 19:53:25 VM cerbere4 is not responding.<br>> 27 oct. 2018 19:48:29 Starting to import Vm crij2 to Data Center Default, Cluster Default<br>> 27 oct. 2018 19:47:41 Refresh image list succeeded for domain(s): ISO (ISO file type)<br>> 27 oct. 2018 19:46:46 VM crij2 was renamed from crij2 to crij2_ok by admin.<br>> 27 oct. 2018 19:46:46 VM crij2 configuration was updated by admin@internal-authz.<br>> 27 oct. 2018 19:46:12 Refresh image list succeeded for domain(s): ISO (ISO file type)<br>> 27 oct. 2018 19:42:36 Refresh image list succeeded for domain(s): ISO (ISO file type)<br>> 27 oct. 2018 19:37:22 Vm crij2 was exported successfully to EXPORT<br>> 27 oct. 2018 19:36:04 VM HostedEngine is not responding.<br>> 27 oct. 2018 19:33:03 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:32:48 VM altern8 is not responding.<br>> 27 oct. 2018 19:32:48 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:31:03 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:30:48 VM altern8 is not responding.<br>> 27 oct. 2018 19:30:48 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:28:37 VM Sogov3 is not responding.<br>> 27 oct. 2018 19:28:07 VM altern8 is not responding.<br>> 27 oct. 2018 19:28:07 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:28:07 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:25:10 VM Mint19 is not responding.<br>> 27 oct. 2018 19:25:10 VM zabbix is not responding.<br>> 27 oct. 2018 19:24:55 VM HostedEngine is not responding.<br>> 27 oct. 2018 19:23:33 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:23:18 VM altern8 is not responding.<br>> 27 oct. 2018 19:23:18 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:21:52 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:20:06 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:19:51 VM Sogov3 is not responding.<br>> 27 oct. 2018 19:18:26 Host ginger.local.systea.fr power management was verified successfully.<br>> 27 oct. 2018 19:18:26 Status of host ginger.local.systea.fr was set to Up.<br>> 27 oct. 2018 19:18:25 Manually synced the storage devices from host ginger.local.systea.fr<br>> 27 oct. 2018 19:17:51 Executing power management status on Host ginger.local.systea.fr using Proxy Host victor.local.systea.fr and Fence Agent ipmilan:10.0.0.225.<br>> 27 oct. 2018 19:17:39 Host ginger.local.systea.fr is not responding. It will stay in Connecting state for a grace period of 82 seconds and after that an attempt to fence the host will be issued.<br>> 27 oct. 2018 19:17:21 VM altern8 is not responding.<br>> 27 oct. 2018 19:17:21 Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host ginger.local.systea.fr, Error: Network error during communication with the Host.).<br>> 27 oct. 2018 19:17:21 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:17:20 VM HostedEngine is not responding.<br>> 27 oct. 2018 19:17:20 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:17:19 VDSM ginger.local.systea.fr command SpmStatusVDS failed: Connection timeout for host 'ginger.local.systea.fr', last response arrived 17279 ms ago.<br>> 27 oct. 2018 19:16:16 Failed to update VMs/Templates OVF data for Storage Domain DATA02 in Data Center Default. 27 oct. 2018 19:16:16<br>> Failed to update OVF disks 85d67951-d610-49b3-aaab-a81850621e35, OVF data isn't updated on those OVF stores (Data Center Default, Storage Domain DATA02).<br>> 27 oct. 2018 19:16:16 VDSM command SetVolumeDescriptionVDS failed: Resource timeout: ()<br>> 27 oct. 2018 19:16:16 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:16:16 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:14:46 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:14:46 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:13:18 Host ginger.local.systea.fr power management was verified successfully.<br>> 27 oct. 2018 19:13:18 Status of host ginger.local.systea.fr was set to Up.<br>> 27 oct. 2018 19:13:03 Manually synced the storage devices from host ginger.local.systea.fr<br>> 27 oct. 2018 19:12:51 VM altern8 is not responding.<br>> 27 oct. 2018 19:12:51 VM HostedEngine is not responding.<br>> 27 oct. 2018 19:12:51 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:12:48 Executing power management status on Host ginger.local.systea.fr using Proxy Host victor.local.systea.fr and Fence Agent ipmilan:10.0.0.225.<br>> 27 oct. 2018 19:12:44 Host ginger.local.systea.fr does not enforce SELinux. Current status: DISABLED<br>> 27 oct. 2018 19:12:36 Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host ginger.local.systea.fr, Error: Network error during communication with the Host.).<br>> 27 oct. 2018 19:12:28 Host ginger.local.systea.fr is not responding. It will stay in Connecting state for a grace period of 82 seconds and after that an attempt to fence the host will be issued.<br>> 27 oct. 2018 19:12:28 VDSM ginger.local.systea.fr command SpmStatusVDS failed: Connection timeout for host 'ginger.local.systea.fr', last response arrived 25225 ms ago.<br>> 27 oct. 2018 19:10:06 VM altern8 is not responding.<br>> 27 oct. 2018 19:10:06 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:10:06 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:08:49 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:08:45 Refresh image list succeeded for domain(s): ISO (ISO file type)<br>> 27 oct. 2018 19:08:34 VM altern8 is not responding.<br>> 27 oct. 2018 19:08:34 VM patjoub1 is not responding.<br>> 27 oct. 2018 19:08:34 VM HostedEngine is not responding.<br>> 27 oct. 2018 19:04:01 VM op2drugs1 is not responding.<br>> 27 oct. 2018 19:01:08 VM HostedEngine is not responding.<br>> 27 oct. 2018 19:00:53 VM zabbix is not responding.<br>> 27 oct. 2018 19:00:01 Trying to restart VM npi2 on Host victor.local.systea.fr<br>> 27 oct. 2018 18:59:14 Trying to restart VM npi2 on Host victor.local.systea.fr<br>> 27 oct. 2018 18:59:13 Highly Available VM np2 failed. It will be restarted automatically.<br>> 27 oct. 2018 18:59:13 VM npi2 is down with error. Exit message: VM has been terminated on the host.<br>> 27 oct. 2018 18:59:05 VM altern8 is not responding.<br>> 27 oct. 2018 18:58:44 Storage domain DATA02 experienced a high latency of 6.16279 seconds from host ginger.local.systea.fr. This may cause performance and functional issues. Please consult your Storage Administrator.<br>> 27 oct. 2018 18:57:19 VM altern8 is not responding.<br>> 27 oct. 2018 18:57:19 VM patjoub1 is not responding.<br>> 27 oct. 2018 18:57:19 VM HostedEngine is not responding.<br>> 27 oct. 2018 18:57:19 VM op2drugs1 is not responding.<br>> 27 oct. 2018 18:55:56 VM altern8 is not responding.<br>> 27 oct. 2018 18:55:41 VM op2drugs1 is not responding.<br>> 27 oct. 2018 18:55:00 VM altern8 is not responding.<br>> 27 oct. 2018 18:54:45 VM op2drugs1 is not responding.<br>> 27 oct. 2018 18:52:21 VM Sogov3 is not responding.<br>> 27 oct. 2018 18:52:21 VM npi2 is not responding.<br>> 27 oct. 2018 18:50:50 VM altern8 is not responding.<br>> 27 oct. 2018 18:50:47 VM zabbix is not responding.<br>> 27 oct. 2018 18:48:16 VM op2drugs1 is not responding.<br>> 27 oct. 2018 18:48:03 VM altern8 is not responding.<br>> 27 oct. 2018 18:48:03 VM HostedEngine is not responding.<br>> 27 oct. 2018 18:45:48 Starting export Vm crij2 to EXPORT<br>> 27 oct. 2018 18:42:57 Refresh image list succeeded for domain(s): ISO (ISO file type)<br>> 27 oct. 2018 18:40:44 Refresh image list succeeded for domain(s): ISO (ISO file type)<br>> 27 oct. 2018 18:40:04 VM crij2 is down. Exit message: User shut down from within the guest<br>> 27 oct. 2018 18:39:25 User admin@internal-authz got disconnected from VM crij2.<br>> I checked the network and gluster since it works but saw absolutly nothing special. The storage network was not too sollicited (bwm-ng indicated max 50MB/s on bond1). Gluster log no errors too (even if the engine reported some timeouts).<br>><br>> This morning I started to search why and wanted to submit to you some logs on this thread, but I found something that had not caught my attention before, so I ask about it before all.<br>><br>> I recall the configuration :<br>> 3 hosts with gluster (replica 2 + arbiter). The volumes are on a separate network (bond1 is an aggregation of 2 Gb cards while ovirmgmt is on bond0, 2 NICs in backup mode).<br>> For now, I have only declared the first 2 nodes in the engine GUI as ovirt nodes, because the arbiter is a small machine with a smaller CPU (and only 8Gb memory), that needed to downgrade the cluster from Sandybridge to Nehalem. Maybe it was an error. The storagenetwork on bond1 was declared too in the GYUI, but not yet as a gluster storage.<br>><br>> The Gluster volumes themselves were declared on the storage network by using names indicated in /etc/hosts on bond1 network. Here is a volume status :<br>> # gluster volume status<br>> Status of volume: DATA01<br>> Gluster process                             TCP Port  RDMA Port  Online  Pid<br>> ------------------------------------------------------------------------------<br>> Brick victorstorage.local.systea.fr:/home/d<br>> ata01/data01/brick                          49152     0          Y       2489<br>> Brick gingerstorage.local.systea.fr:/home/d<br>> ata01/data01/brick                          49152     0          Y       2531<br>> Brick eskarinastorage.local.systea.fr:/home<br>> /data01/data01/brick                        49153     0          Y       28119<br>> Self-heal Daemon on localhost               N/A       N/A        Y       24859<br>> Self-heal Daemon on eskarinastorage.local.s<br>> ystea.fr                                    N/A       N/A        Y       30725<br>> Self-heal Daemon on victorstorage.local.sys<br>> tea.fr                                      N/A       N/A        Y       2810<br>><br>> Task Status of Volume DATA01<br>> ------------------------------------------------------------------------------<br>> There are no active volume tasks<br>><br>> Status of volume: DATA02<br>> Gluster process                             TCP Port  RDMA Port  Online  Pid<br>> ------------------------------------------------------------------------------<br>> Brick victorstorage.local.systea.fr:/home/d<br>> ata02/data02/brick                          49153     0          Y       2553<br>> Brick gingerstorage.local.systea.fr:/home/d<br>> ata02/data02/brick                          49153     0          Y       2561<br>> Brick eskarinastorage.local.systea.fr:/home<br>> /data01/data02/brick                        49154     0          Y       28204<br>> Self-heal Daemon on localhost               N/A       N/A        Y       24859<br>> Self-heal Daemon on eskarinastorage.local.s<br>> ystea.fr                                    N/A       N/A        Y       30725<br>> Self-heal Daemon on victorstorage.local.sys<br>> tea.fr                                      N/A       N/A        Y       2810<br>><br>> Task Status of Volume DATA02<br>> ------------------------------------------------------------------------------<br>> There are no active volume tasks<br>><br>> Status of volume: ENGINE<br>> Gluster process                             TCP Port  RDMA Port  Online  Pid<br>> ------------------------------------------------------------------------------<br>> Brick victorstorage.local.systea.fr:/home/d<br>> ata02/engine/brick                          49154     0          Y       2571<br>> Brick gingerstorage.local.systea.fr:/home/d<br>> ata02/engine/brick                          49154     0          Y       2610<br>> Brick eskarinastorage.local.systea.fr:/home<br>> /data01/engine/brick                        49152     0          Y       28013<br>> Self-heal Daemon on localhost               N/A       N/A        Y       24859<br>> Self-heal Daemon on eskarinastorage.local.s<br>> ystea.fr                                    N/A       N/A        Y       30725<br>> Self-heal Daemon on victorstorage.local.sys<br>> tea.fr                                      N/A       N/A        Y       2810<br>><br>> Task Status of Volume ENGINE<br>> ------------------------------------------------------------------------------<br>> There are no active volume tasks<br>><br>> Status of volume: EXPORT<br>> Gluster process                             TCP Port  RDMA Port  Online  Pid<br>> ------------------------------------------------------------------------------<br>> Brick victorstorage.local.systea.fr:/home/d<br>> ata01/export/brick                          49155     0          Y       2588<br>> Brick gingerstorage.local.systea.fr:/home/d<br>> ata01/export/brick                          49155     0          Y       2629<br>> Brick eskarinastorage.local.systea.fr:/home<br>> /data01/export/brick                        49156     0          Y       28384<br>> Self-heal Daemon on localhost               N/A       N/A        Y       24859<br>> Self-heal Daemon on eskarinastorage.local.s<br>> ystea.fr                                    N/A       N/A        Y       30725<br>> Self-heal Daemon on victorstorage.local.sys<br>> tea.fr                                      N/A       N/A        Y       2810<br>><br>> Task Status of Volume EXPORT<br>> ------------------------------------------------------------------------------<br>> There are no active volume tasks<br>><br>> Status of volume: ISO<br>> Gluster process                             TCP Port  RDMA Port  Online  Pid<br>> ------------------------------------------------------------------------------<br>> Brick victorstorage.local.systea.fr:/home/d<br>> ata01/iso/brick                             49156     0          Y       2595<br>> Brick gingerstorage.local.systea.fr:/home/d<br>> ata01/iso/brick                             49156     0          Y       2636<br>> Brick eskarinastorage.local.systea.fr:/home<br>> /data01/iso/brick                           49155     0          Y       28292<br>> Self-heal Daemon on localhost               N/A       N/A        Y       24859<br>> Self-heal Daemon on eskarinastorage.local.s<br>> ystea.fr                                    N/A       N/A        Y       30725<br>> Self-heal Daemon on victorstorage.local.sys<br>> tea.fr                                      N/A       N/A        Y       2810<br>><br>> Task Status of Volume ISO<br>> -------------------------------<br>> But, a df on the nodes shows that all volumes except ENGINE was mounted on ovirmgmt network (hosts names without "storage") :<br>><br>> gingerstorage.local.systea.fr:/ENGINE   5,0T    226G  4,7T   5% /rhev/data-center/mnt/glusterSD/gingerstorage.local.systea.fr:_ENGINE<br>> victor.local.systea.fr:/DATA01          1,3T    425G  862G  33% /rhev/data-center/mnt/glusterSD/victor.local.systea.fr:_DATA01<br>> victor.local.systea.fr:/DATA02          5,0T    226G  4,7T   5% /rhev/data-center/mnt/glusterSD/victor.local.systea.fr:_DATA02<br>> victor.local.systea.fr:/ISO             1,3T    425G  862G  33% /rhev/data-center/mnt/glusterSD/victor.local.systea.fr:_ISO<br>> victor.local.systea.fr:/EXPORT          1,3T    425G  862G  33% /rhev/data-center/mnt/glusterSD/victor.local.systea.fr:_EXPORT<br>><br>> I can't remember how it was declared at install time, maybe I had not seen that, but if I tried to had a domain now, gluster managed, effectively it proposes to me only the nodes by their ovirmgmt names, not storage names.<br>><br>> Names are only known in the /etc/hosts of all nodes + engine, there is no DNS for this local addresses.<br>><br>> So : in your opinion, can this configuration be a (the) source of my problems ? And have you an idea how I could correct this now, without loosing anything ?<br><br>I don't think this is the cause of your issues.<br>Are there errors in vdsm logs? Do you have issues with storage latency<br>(can you check the gluster volume profile output?)<br><br>><br>> Thanks for all suggestions.<br>><br>> --<br>><br>> Regards,<br>><br>> Frank<br>><br>><br>> Le Jeudi, Octobre 18, 2018 23:13 CEST, Nir Soffer <nsoffer@redhat.com> a écrit:<br>>  On Thu, Oct 18, 2018 at 3:43 PM fsoyer <fsoyer@systea.fr> wrote:Hi,<br>> I forgot to look in the /var/log/messages file on the host ! What a shame :/<br>> Here is the messages file at the time of the error : https://gist.github.com/fsoyer/4d1247d4c3007a8727459efd23d89737<br>> At the sasme time, the second host as no particular messages in its log<br><br></blockquote></div>