Hi Matthew, I'm also using a HA TrueNAS as the storage. I have NFS as well as iscsi shares and did do some in place upgrade. The failover went more or less smooth, it was more of an issue on the TrueNas side where the different vlans didn't come up. This caused the engine to take down the storage domain and things took some time until everything was up again. The VMs in ovirt did go into paused mode and started to work again as soon as the failover was done. I was failing over by rebooting one of the TrueNas nodes and this took some time for the other node to take over. I was thinking about asking the TN guys if there is a command or procedure to speed up the failover. In all I didn't stop any VMs although the VMs paused. Depending on the critically of the VMs you might want to move to another storage. Sven -----Ursprüngliche Nachricht----- Von: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag von Matthew Trent Gesendet: Montag, 5. Juni 2017 23:48 An: users <users(a)ovirt.org&gt; Betreff: [ovirt-users] Seamless SAN HA failovers with oVirt? I'm using two TrueNAS HA SANs (FreeBSD-based ZFS) to provide storage via NFS to 7 oVirt boxes and about 25 VMs. For SAN system upgrades I've always scheduled a maintenance window, shut down all the oVirt stuff, upgraded the SANs, and spun everything back up. It's pretty disruptive, but I assumed that was the thing to do. However, in talking with the TrueNAS vendor they said the majority of their customers are using VMWare and they almost always do TrueNAS updates in production. They just upgrade one head of the TrueNAS HA pair then failover to the other head and upgrade it too. There's a 30-ish second pause in I/O while the disk arrays are taken over by the other HA head, but VMWare just tolerates it and continues without skipping a beat. They say this is standard procedure in the SAN world and virtualization systems should tolerate 30-60 seconds of I/O pause for HA failovers seamlessly. It sounds great to me, but I wanted to pick this lists' brain -- is anyone doing this with oVirt? Are you able to failover your HA SAN with 30-60 seconds of no I/O without oVirt freaking out? If not, are there any tunables relating to this? I see the default NFS mount options look fairly tolerant (proto=tcp,timeo=600,retrans=6), but are there VDSM or sanlock or some other oVirt timeouts that will kick in and start putting storage domains into error states, fencing hosts or something before that? I've never timed anything, but I want to say my past experience is that ovirt hosted engine started showing errors almost immediately when we've had SAN issues in the past. Thanks! -- Matthew Trent Network Engineer Lewis County IT Services _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

I think its not related to something on the trueNAS side. if you are using iscsi multipath you should be using round-robin , if one of the paths goes down you still have the other path with your information., so no sanlock . unfortunately if you want iscsi mpath on ovirt, its prefered to edit the config by hand and test. also, with multipath, you can tell the os to 'stop using' one of the paths( represented as a disk). so, for example: multipath -ll should* be looking like this: 36589cfc000000341111968eacf965e3c dm-17 FreeNAS ,iSCSI Disk size=50G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=enabled |- 31:0:0:5 sdl 8:176 failed faulty running |- 32:0:0:5 sdm 8:192 failed faulty running |- 35:0:0:5 sdo 8:224 failed faulty running `- 34:0:0:5 sdn 8:208 failed faulty running and working correctly like this: 36589cfc000000ee205ed6757fa724bac dm-2 FreeNAS ,iSCSI Disk size=5.5T features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 13:0:0:10 sdi 8:128 active ready running |- 15:0:0:10 sdk 8:160 active ready running |- 28:0:0:10 sdg 8:96 active ready running `- 29:0:0:10 sdj 8:144 active ready running (yes, they are different ones, I wont disconnect one path just to show an example =) ) hope I clarified a bit. cant tell how it would work on nfs or if it works at all. 2017-06-06 10:06 GMT-03:00 Chris Adams <cma(a)cmadams.net&gt;:

Im saying you can do it with multipath and not rely on truenas/freenas. with an active/active configuration on the virt side...instead of active/passive on the storage side. 2017-06-06 10:44 GMT-03:00 Chris Adams <cma(a)cmadams.net&gt;:

Chris, if you have active-active with multipath: you upgrade one system, reboot it, check it came active again, then upgrade the other. -seamless. -no service interruption. -not locked to any storage solution. multipath was designed exactly for that. 2017-06-06 11:03 GMT-03:00 Chris Adams <cma(a)cmadams.net&gt;:

Thanks for the replies, all! Yep, Chris is right. TrueNAS HA is active/passive and there isn't a way around that when failing between heads. Sven: In my experience with iX support, they have directed me to reboot the active node to initiate failover. There's "hactl takeover" and "hactl giveback" commends, but reboot seems to be their preferred method. VMs going into a paused state and resuming when storage is back online sounds great. As long as oVirt's pause/resume isn't significantly slower than the 30-or-so seconds the TrueNAS takes to complete its failover, that's a pretty tolerable interruption for my needs. So my next questions are: 1) Assuming the SAN failover DOES work correctly, can anyone comment on their experience with oVirt pausing/thawing VMs in an NFS-based active/passive SAN failover scenario? Does it work reliably without intervention? Is it reasonably fast? 2) Is there anything else in the oVirt stack that might cause it to "freak out" rather than gracefully pause/unpause VMs? 2a) Particularly: I'm running hosted engine on the same TrueNAS storage. Does that change anything WRT to timeouts and oVirt's HA and fencing and sanlock and such? 2b) Is there a limit to how long oVirt will wait for storage before doing something more drastic than just pausing VMs? -- Matthew Trent Network Engineer Lewis County IT Services 360.740.1247 - Helpdesk 360.740.3343 - Direct line ________________________________________ From: users-bounces(a)ovirt.org <users-bounces(a)ovirt.org&gt; on behalf of Chris Adams <cma(a)cmadams.net&gt; Sent: Tuesday, June 6, 2017 7:21 AM To: users(a)ovirt.org Subject: Re: [ovirt-users] Seamless SAN HA failovers with oVirt? Once upon a time, Juan Pablo <pablo.localhost(a)gmail.com&gt; said: Yes, but that's still not how a TrueNAS (and most other low- to mid-range SANs) works, so is not relevant. The TrueNAS only has a single active node talking to the hard drives at a time, because having two nodes talking to the same storage at the same time is a hard problem to solve (typically requires custom hardware with active cache coherency and such). You can (and should) use multipath between servers and a TrueNAS, and that protects against NIC, cable, and switch failures, but does not help with a controller failure/reboot/upgrade. Multipath is also used to provide better bandwidth sharing between links than ethernet LAGs. -- Chris Adams <cma(a)cmadams.net&gt; _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Hey Matthew, I think it's VDSM that handles the pausing & resuming of the VMs. An analogous small-scale scenario...the Gluster layer for one of our smaller oVirt clusters temporarily lost quorum the other week, locking all I/O for about 30 minutes. The VMs all went into pause & then resumed automatically when quorum was restored. To my surprise/relief, not a single one of the 10 odd VMs reported any errors. YMMV Doug On 6 June 2017 at 13:45, Matthew Trent <Matthew.Trent(a)lewiscountywa.gov&gt; wrote: -- Doug