Re: [ovirt-users] Storage Domain Not Found After upgrade from 3.3.2 to 3.4

Hi Paul, I am still looking into this log, but from a quick first assessment, it looks like (for some reason I don't know yet...) there is a storage domain that is missing. This is visible in the following error traceback in the vdsm log: Thread-29::ERROR::2014-04-27 12:43:05,825::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae m Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',) However, I do not see any information in the vdsm log itself about what this domain is (yet - it may be there and I am still looking). The reason vdsm is trying to access this storage domain (9a4a80a1-5377-4a94-ade3-e58183e916ae) is that it appears to be part of the storage pool (datacenter) according to the pool's metadata, as seen in the following lines from the initial connection to the datacenter, when vdsm first starts up: Thread-13::DEBUG::2014-04-27 12:42:05,604::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897ff2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454'] Is it possible you had another domain attached to this datacenter before the upgrade that is somehow on the root vg partitions and gets destroyed along the upgrade process, and then reverting brings them back? If you have this system currently back to 3.3.2 it should be up, could you run the following commands on the ovirt-node: vdsClient -s 0 getStorageDomainsList <- This lists all storage domains' ids that vdsm (the ovirt-node) can currently see (hopefully the domain in question's ID should be listed there) vdsClient -s 0 getStorageDomainInfo 9a4a80a1-5377-4a94-ade3-e58183e916ae <- displays information about the storage domain. If this succeeds we should know a bit more about the domain Thanks, Gadi Ickowicz ----- Original Message ----- From: regpm(a)mccleary.me.uk To: "Gadi Ickowicz" <gickowic(a)redhat.com&gt;, users(a)ovirt.org Sent: Sunday, April 27, 2014 3:10:27 PM Subject: Re: [ovirt-users] Storage Domain Not Found After upgrade from 3.3.2 to 3.4 Hi, Yes, you're correct, Gadi. I have a single Ovirt engine, which has two Datacenters; one is local storage on the Engine server and the other is local storage on the Ovirt Node. I've renamed the servers in the attached vdsm log output (from the ovirt-node) to ovirt-engine (10.50.0.18) and ovirt-node (10.50.0.19). BDS_DataCentre1 is on the ovirt-engine server and this works fine after the upgrade. BDS_DataCentre2 is on the ovirt-node and this is the one that fails to activate due to the storage domains not being accessible. The Master storage domain on the ovirt-node is s2data1_s2mgt_boot1. There are two other storage domains as well: s2data1_s2mgt_app1 and s2data1_s2usr_boot1. The underlying filesystems are mounted fine, and as I said, if I restore the server (root vg partitions; the storage domain filesystems are not touched) then it works fine again. So the upgrade is not playing nicely for some reason, but it's not clear to me from the log what the issue is. Thanks, Paul On 27/04/2014 07:56, Gadi Ickowicz wrote: > Hi, > > Could you please attach the vdsm log as a file (it is easier to read) for the failing node? > > Also - I am a bit confused regarding what exactly your setup is - Do you have only a single engine (the all-in-one) and 2 dcs, one for the all in one and one for the oVirt node, which is the one that is failing? > > Thanks, > Gadi Ickowicz > > ----- Original Message ----- > From: regpm(a)mccleary.me.uk > To: users(a)ovirt.org > Sent: Friday, April 25, 2014 10:46:35 PM > Subject: [ovirt-users] Storage Domain Not Found After upgrade from 3.3.2 to 3.4 > > Hi, > > I have an All-in-one installation engine node and an Ovirt node. They > each use local storage and are thus configured in separate Data > Centres. I upgraded both from 3.3.2 to 3.4 and this completed without > error. I completed the engine-setup upgrade and this complete ok. I > rebooted both the servers and the Ovirt engine node worked fine and I > could start it's VMs. The Ovirt Node's datacenter is not activating, > which seems to be due to none of the storage domains coming online. > I've checked and the storage is mounted and available fine on the Ovirt > node. > > Looking at the vdsm.log I can see errors stating that it can't find the > storage domain. I have restored the entire node back to the pre-upgrade > state and it works fine. It breaks again when I upgrade it. The > approach I used was to put the Ovirt node in maintenance mode and run > yum update. Anybody have similar issues or understand the log errors below? > > < SNIP SNIP original log output> > > > Thanks, Paul > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users