I setup an iscsi domain in hopes of moving away from NFS.  I created a new logical network for this on a storage vlan. both hosts and storage server grabbed a good IP on this vlan. 

ISCSI Domain connected ok and I started moving disks over from the NFS domain.  Transfers were completing so I let it do its thing.  

This morning, after the transfers completed, three things occurred in quick succession:

1. Affinity Rules Enforcement Manager started.
2. Invalid status on Data Center. Setting status to Non Responsive.
3. Storage Pool Manager runs on Host brian.xxx.net (Address: brian.xxx.net), Data Center.
4. Storage Domain domiscsi was deactivated by system because it's not visible by any of the hosts.

Try to attach it again and it fails.  Have another coffee and try to attach again to get logs, and this time it attaches just fine. 

domain disconnected vdsm log: https://pastebin.com/pfGerEE5

Successful attach operation vdsm log: https://pastebin.com/c0U07QXg

host Brian can't seem to keep the storage domain https://pastebin.com/ySAqiqA3

both hosts seem to have connected sessions when looking from the storage server:

image.png

Seems like it might be one problematic host, because engine keeps trying to reboot it to get it to reconnect to the storage domain.  Maybe after a few tries it just deems the domain is bad.  Not sure!

Any help is appreciated!