[Users] Storage domain issue iSCSI
Hi Everyone, I'm having an issue following a power cut, none of my 3 nodes (All Fedora 17 / oVirt 3.1) are able to attach the master Domain. in /var/log/messages I see the following on each node: Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4 Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR {'status': {'message': "Wrong Master domain or its version: 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb, pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}} I've tried reboots/restarts/node re-installs I can see the PV and the iSCSI sessions fine from the shell. As this is the master, none of my nodes will start. Any help would be great. Kind Regards Simon
Hi Simon, We could use some more information in order to understand the problem, could you please attach the vdsm logs? Thanks, Yeela ----- Original Message -----
From: "Simon Donnellan" <f@baconwho.re> To: users@ovirt.org Sent: Thursday, November 29, 2012 7:35:18 PM Subject: [Users] Storage domain issue iSCSI
Hi Everyone,
I'm having an issue following a power cut, none of my 3 nodes (All Fedora 17 / oVirt 3.1) are able to attach the master Domain.
in /var/log/messages I see the following on each node:
Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4 Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR {'status': {'message': "Wrong Master domain or its version: 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb, pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
I've tried reboots/restarts/node re-installs
I can see the PV and the iSCSI sessions fine from the shell.
As this is the master, none of my nodes will start.
Any help would be great.
Kind Regards
Simon
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Yeela, Thanks for the reply, I've attached a vdsm.log file containing an attempt to activate the host, then activate the iSCSI storage domain. Thanks Simon On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan <ykaplan@redhat.com> wrote:
Hi Simon, We could use some more information in order to understand the problem, could you please attach the vdsm logs? Thanks, Yeela
----- Original Message -----
From: "Simon Donnellan" <f@baconwho.re> To: users@ovirt.org Sent: Thursday, November 29, 2012 7:35:18 PM Subject: [Users] Storage domain issue iSCSI
Hi Everyone,
I'm having an issue following a power cut, none of my 3 nodes (All Fedora 17 / oVirt 3.1) are able to attach the master Domain.
in /var/log/messages I see the following on each node:
Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4 Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR {'status': {'message': "Wrong Master domain or its version: 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb, pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
I've tried reboots/restarts/node re-installs
I can see the PV and the iSCSI sessions fine from the shell.
As this is the master, none of my nodes will start.
Any help would be great.
Kind Regards
Simon
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Your log indicates connection only to nfs storage: /home/iso /share/MD0_DATA/VMs can you tell me their UUIDs? Also a DC is allowed to contain only one storage type (nfs / block), which does not fit with your question regarding the iSCSI SD, since you have attached nfs SD to your DC. Please check this again and return with more details... ----- Original Message -----
From: "Simon Donnellan" <f@baconwho.re> To: "Yeela Kaplan" <ykaplan@redhat.com> Cc: users@ovirt.org Sent: Thursday, November 29, 2012 10:08:17 PM Subject: Re: [Users] Storage domain issue iSCSI
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an attempt to activate the host, then activate the iSCSI storage domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan < ykaplan@redhat.com > wrote:
Hi Simon, We could use some more information in order to understand the problem, could you please attach the vdsm logs? Thanks, Yeela
----- Original Message -----
From: "Simon Donnellan" < f@baconwho.re > To: users@ovirt.org Sent: Thursday, November 29, 2012 7:35:18 PM Subject: [Users] Storage domain issue iSCSI
Hi Everyone,
I'm having an issue following a power cut, none of my 3 nodes (All Fedora 17 / oVirt 3.1) are able to attach the master Domain.
in /var/log/messages I see the following on each node:
Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4 Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR {'status': {'message': "Wrong Master domain or its version: 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb, pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
I've tried reboots/restarts/node re-installs
I can see the PV and the iSCSI sessions fine from the shell.
As this is the master, none of my nodes will start.
Any help would be great.
Kind Regards
Simon
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a multi-part message in MIME format. --------------060402000606000203070701 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit I think the error is clear. Engine was expecting a master storage domain metadata format version 3, while the master storage metadata gave version 4. I am wondering if the master storage domain metadata was corrupted during the power off. See: the error came from. Thread-54::ERROR::2012-11-29 20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain) Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4 See: 'MASTER_VERSION=4' below: Thread-49::DEBUG::2012-11-29 20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=2', u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad', 'DESCRIPTION=512gb2', 'CLASS=Data', 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175', 'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4'_,_ 'ROLE=Master', 'POOL_DESCRIPTION=UB1', u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active', 'POOL_SPM_LVER=541', '_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e', 'POOL_SPM_ID=-1'] 2012-11-30 4:08, Simon Donnellan:
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an attempt to activate the host, then activate the iSCSI storage domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan <ykaplan@redhat.com <mailto:ykaplan@redhat.com>> wrote:
Hi Simon, We could use some more information in order to understand the problem, could you please attach the vdsm logs? Thanks, Yeela
----- Original Message ----- > From: "Simon Donnellan" <f@baconwho.re <mailto:f@baconwho.re>> > To: users@ovirt.org <mailto:users@ovirt.org> > Sent: Thursday, November 29, 2012 7:35:18 PM > Subject: [Users] Storage domain issue iSCSI > > > Hi Everyone, > > I'm having an issue following a power cut, none of my 3 nodes (All > Fedora 17 / oVirt 3.1) are able to attach the master Domain. > > in /var/log/messages I see the following on each node: > > Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master > domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected > version 3 it is version 4 > Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR > Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error > Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR > {'status': {'message': "Wrong Master domain or its version: > 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb, > pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}} > > I've tried reboots/restarts/node re-installs > > I can see the PV and the iSCSI sessions fine from the shell. > > As this is the master, none of my nodes will start. > > Any help would be great. > > Kind Regards > > Simon > > _______________________________________________ > Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/users >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- --- ?? Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming@cn.ibm.com or shuming@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC --------------060402000606000203070701 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">I think the error is clear. Engine was expecting a master storage domain metadata format version 3, while the master storage metadata gave version 4. I am wondering if the master storage domain metadata was corrupted during the power off.<br> <br> See: the error came from.<br> Thread-54::ERROR::2012-11-29 20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain) Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4<br> <br> See: 'MASTER_VERSION=4' below:<br> <br> Thread-49::DEBUG::2012-11-29 20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=2', u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad', 'DESCRIPTION=512gb2', 'CLASS=Data', 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175', 'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4'<u>,</u> 'ROLE=Master', 'POOL_DESCRIPTION=UB1', u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active', 'POOL_SPM_LVER=541', '_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e', 'POOL_SPM_ID=-1']<br> <br> <br> 2012-11-30 4:08, Simon Donnellan:<br> </div> <blockquote cite="mid:CACLk-0yisx1+LtcOLmR9R_ojKEpDQZ=NmbJhx2SRBqtdue5hDQ@mail.gmail.com" type="cite">Hi Yeela,<br> <br> Thanks for the reply, I've attached a vdsm.log file containing an attempt to activate the host, then activate the iSCSI storage domain.<br> <br> Thanks<br> <br> Simon<br> <div class="gmail_extra"><br> <br> <div class="gmail_quote"> On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan <span dir="ltr"><<a moz-do-not-send="true" href="mailto:ykaplan@redhat.com" target="_blank">ykaplan@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hi Simon,<br> We could use some more information in order to understand the problem,<br> could you please attach the vdsm logs?<br> Thanks,<br> Yeela<br> <div> <div class="h5"><br> ----- Original Message -----<br> > From: "Simon Donnellan" <<a moz-do-not-send="true" href="mailto:f@baconwho.re">f@baconwho.re</a>><br> > To: <a moz-do-not-send="true" href="mailto:users@ovirt.org">users@ovirt.org</a><br> > Sent: Thursday, November 29, 2012 7:35:18 PM<br> > Subject: [Users] Storage domain issue iSCSI<br> ><br> ><br> > Hi Everyone,<br> ><br> > I'm having an issue following a power cut, none of my 3 nodes (All<br> > Fedora 17 / oVirt 3.1) are able to attach the master Domain.<br> ><br> > in /var/log/messages I see the following on each node:<br> ><br> > Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master<br> > domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected<br> > version 3 it is version 4<br> > Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR<br> > Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error<br> > Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR<br> > {'status': {'message': "Wrong Master domain or its version:<br> > 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,<br> > pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}<br> ><br> > I've tried reboots/restarts/node re-installs<br> ><br> > I can see the PV and the iSCSI sessions fine from the shell.<br> ><br> > As this is the master, none of my nodes will start.<br> ><br> > Any help would be great.<br> ><br> > Kind Regards<br> ><br> > Simon<br> ><br> </div> </div> > _______________________________________________<br> > Users mailing list<br> > <a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> > <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br> ><br> </blockquote> </div> <br> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre class="moz-signature" cols="72">-- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: <a class="moz-txt-link-abbreviated" href="mailto:shuming@cn.ibm.com">shuming@cn.ibm.com</a> or <a class="moz-txt-link-abbreviated" href="mailto:shuming@linux.vnet.ibm.com">shuming@linux.vnet.ibm.com</a> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC</pre> </body> </html> --------------060402000606000203070701--
Hi Yeela, Shu, Many thanks for your replies. I'm aware of the one type rule, the two NFS shares you noticed are the ISO share and an export store. (I'm unable to find a way to create these as iSCSI type) The UUID of the master iSCSI domain is f1b40ecc-b6a9-44e7- 92cb-0fdf445e3175 it's name in the gui is "512gb2" I too believe there is a meta data corruption, is there any way to get my systems back up and running? Kind Regards Simon On Sun, Dec 2, 2012 at 3:13 PM, Shu Ming <shuming@linux.vnet.ibm.com> wrote:
I think the error is clear. Engine was expecting a master storage domain metadata format version 3, while the master storage metadata gave version 4. I am wondering if the master storage domain metadata was corrupted during the power off.
See: the error came from. Thread-54::ERROR::2012-11-29 20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain) Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4
See: 'MASTER_VERSION=4' below:
Thread-49::DEBUG::2012-11-29 20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=2', u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad', 'DESCRIPTION=512gb2', 'CLASS=Data', 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175', 'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4'*,*'ROLE=Master', 'POOL_DESCRIPTION=UB1', u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active', 'POOL_SPM_LVER=541', '_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e', 'POOL_SPM_ID=-1']
2012-11-30 4:08, Simon Donnellan:
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an attempt to activate the host, then activate the iSCSI storage domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan <ykaplan@redhat.com> wrote:
Hi Simon, We could use some more information in order to understand the problem, could you please attach the vdsm logs? Thanks, Yeela
----- Original Message -----
From: "Simon Donnellan" <f@baconwho.re> To: users@ovirt.org Sent: Thursday, November 29, 2012 7:35:18 PM Subject: [Users] Storage domain issue iSCSI
Hi Everyone,
I'm having an issue following a power cut, none of my 3 nodes (All Fedora 17 / oVirt 3.1) are able to attach the master Domain.
in /var/log/messages I see the following on each node:
Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4 Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR {'status': {'message': "Wrong Master domain or its version: 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb, pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
I've tried reboots/restarts/node re-installs
I can see the PV and the iSCSI sessions fine from the shell.
As this is the master, none of my nodes will start.
Any help would be great.
Kind Regards
Simon
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
-- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming@cn.ibm.com or shuming@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC
Just a clarification, the UUID of the pool is f1b40ecc-b6a9-44e7-92cb-0fdf445e3175 and the UUID of the msd it is looking for is 68d8b0e2-c348-4cfe-a896-08c62d491dfb (according to the logs you sent me). The problem really is in the msd version (Thanks Shu) but I need more details in order to solve the issue, can you also attach the engine logs? thanks. ----- Original Message -----
From: "Simon Donnellan" <f@baconwho.re> To: "Shu Ming" <shuming@linux.vnet.ibm.com> Cc: "Yeela Kaplan" <ykaplan@redhat.com>, users@ovirt.org Sent: Sunday, December 2, 2012 5:55:00 PM Subject: Re: [Users] Storage domain issue iSCSI
Hi Yeela, Shu,
Many thanks for your replies.
I'm aware of the one type rule, the two NFS shares you noticed are the ISO share and an export store. (I'm unable to find a way to create these as iSCSI type)
The UUID of the master iSCSI domain is f1b40ecc-b6a9-44e7- 92cb-0fdf445e3175
it's name in the gui is "512gb2"
I too believe there is a meta data corruption, is there any way to get my systems back up and running?
Kind Regards
Simon
On Sun, Dec 2, 2012 at 3:13 PM, Shu Ming < shuming@linux.vnet.ibm.com
wrote:
I think the error is clear. Engine was expecting a master storage domain metadata format version 3, while the master storage metadata gave version 4. I am wondering if the master storage domain metadata was corrupted during the power off.
See: the error came from. Thread-54::ERROR::2012-11-29 20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain) Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4
See: 'MASTER_VERSION=4' below:
Thread-49::DEBUG::2012-11-29 20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=2', u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad', 'DESCRIPTION=512gb2', 'CLASS=Data', 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175', 'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4' , 'ROLE=Master', 'POOL_DESCRIPTION=UB1', u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active', 'POOL_SPM_LVER=541', '_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e', 'POOL_SPM_ID=-1']
2012-11-30 4:08, Simon Donnellan:
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an attempt to activate the host, then activate the iSCSI storage domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan < ykaplan@redhat.com > wrote:
Hi Simon, We could use some more information in order to understand the problem, could you please attach the vdsm logs? Thanks, Yeela
----- Original Message -----
From: "Simon Donnellan" < f@baconwho.re > To: users@ovirt.org Sent: Thursday, November 29, 2012 7:35:18 PM Subject: [Users] Storage domain issue iSCSI
Hi Everyone,
I'm having an issue following a power cut, none of my 3 nodes (All Fedora 17 / oVirt 3.1) are able to attach the master Domain.
in /var/log/messages I see the following on each node:
Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4 Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR {'status': {'message': "Wrong Master domain or its version: 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb, pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
I've tried reboots/restarts/node re-installs
I can see the PV and the iSCSI sessions fine from the shell.
As this is the master, none of my nodes will start.
Any help would be great.
Kind Regards
Simon
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming@cn.ibm.com or shuming@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC
Hi Yeela, Attached are the engine logs. Kind Regards Simon On Sun, Dec 2, 2012 at 4:46 PM, Yeela Kaplan <ykaplan@redhat.com> wrote:
Just a clarification, the UUID of the pool is f1b40ecc-b6a9-44e7-92cb-0fdf445e3175 and the UUID of the msd it is looking for is 68d8b0e2-c348-4cfe-a896-08c62d491dfb (according to the logs you sent me). The problem really is in the msd version (Thanks Shu) but I need more details in order to solve the issue, can you also attach the engine logs? thanks.
----- Original Message -----
From: "Simon Donnellan" <f@baconwho.re> To: "Shu Ming" <shuming@linux.vnet.ibm.com> Cc: "Yeela Kaplan" <ykaplan@redhat.com>, users@ovirt.org Sent: Sunday, December 2, 2012 5:55:00 PM Subject: Re: [Users] Storage domain issue iSCSI
Hi Yeela, Shu,
Many thanks for your replies.
I'm aware of the one type rule, the two NFS shares you noticed are the ISO share and an export store. (I'm unable to find a way to create these as iSCSI type)
The UUID of the master iSCSI domain is f1b40ecc-b6a9-44e7- 92cb-0fdf445e3175
it's name in the gui is "512gb2"
I too believe there is a meta data corruption, is there any way to get my systems back up and running?
Kind Regards
Simon
On Sun, Dec 2, 2012 at 3:13 PM, Shu Ming < shuming@linux.vnet.ibm.com
wrote:
I think the error is clear. Engine was expecting a master storage domain metadata format version 3, while the master storage metadata gave version 4. I am wondering if the master storage domain metadata was corrupted during the power off.
See: the error came from. Thread-54::ERROR::2012-11-29 20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain) Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4
See: 'MASTER_VERSION=4' below:
Thread-49::DEBUG::2012-11-29 20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=2',
u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0',
'TYPE=ISCSI', 'LOGBLKSIZE=512', 'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad', 'DESCRIPTION=512gb2', 'CLASS=Data', 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175', 'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4' , 'ROLE=Master', 'POOL_DESCRIPTION=UB1',
u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active',
'POOL_SPM_LVER=541', '_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e', 'POOL_SPM_ID=-1']
2012-11-30 4:08, Simon Donnellan:
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an attempt to activate the host, then activate the iSCSI storage domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan < ykaplan@redhat.com > wrote:
Hi Simon, We could use some more information in order to understand the problem, could you please attach the vdsm logs? Thanks, Yeela
----- Original Message -----
From: "Simon Donnellan" < f@baconwho.re > To: users@ovirt.org Sent: Thursday, November 29, 2012 7:35:18 PM Subject: [Users] Storage domain issue iSCSI
Hi Everyone,
I'm having an issue following a power cut, none of my 3 nodes (All Fedora 17 / oVirt 3.1) are able to attach the master Domain.
in /var/log/messages I see the following on each node:
Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4 Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR {'status': {'message': "Wrong Master domain or its version: 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb, pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
I've tried reboots/restarts/node re-installs
I can see the PV and the iSCSI sessions fine from the shell.
As this is the master, none of my nodes will start.
Any help would be great.
Kind Regards
Simon
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming@cn.ibm.com or shuming@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC
Simon, I need to see the logs at the same time from both engine and vdsm, they are a few days apart. Can you get me the right logs? thanks. ----- Original Message -----
From: "Simon Donnellan" <f@baconwho.re> To: "Yeela Kaplan" <ykaplan@redhat.com> Cc: users@ovirt.org Sent: Tuesday, December 4, 2012 12:02:08 AM Subject: Re: [Users] Storage domain issue iSCSI
Hi Yeela,
Attached are the engine logs.
Kind Regards
Simon
On Sun, Dec 2, 2012 at 4:46 PM, Yeela Kaplan < ykaplan@redhat.com > wrote:
Just a clarification, the UUID of the pool is f1b40ecc-b6a9-44e7-92cb-0fdf445e3175 and the UUID of the msd it is looking for is 68d8b0e2-c348-4cfe-a896-08c62d491dfb (according to the logs you sent me). The problem really is in the msd version (Thanks Shu) but I need more details in order to solve the issue, can you also attach the engine logs? thanks.
----- Original Message -----
From: "Simon Donnellan" < f@baconwho.re >
To: "Shu Ming" < shuming@linux.vnet.ibm.com > Cc: "Yeela Kaplan" < ykaplan@redhat.com >, users@ovirt.org Sent: Sunday, December 2, 2012 5:55:00 PM Subject: Re: [Users] Storage domain issue iSCSI
Hi Yeela, Shu,
Many thanks for your replies.
I'm aware of the one type rule, the two NFS shares you noticed are the ISO share and an export store. (I'm unable to find a way to create these as iSCSI type)
The UUID of the master iSCSI domain is f1b40ecc-b6a9-44e7- 92cb-0fdf445e3175
it's name in the gui is "512gb2"
I too believe there is a meta data corruption, is there any way to get my systems back up and running?
Kind Regards
Simon
On Sun, Dec 2, 2012 at 3:13 PM, Shu Ming < shuming@linux.vnet.ibm.com
wrote:
I think the error is clear. Engine was expecting a master storage domain metadata format version 3, while the master storage metadata gave version 4. I am wondering if the master storage domain metadata was corrupted during the power off.
See: the error came from. Thread-54::ERROR::2012-11-29 20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain) Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4
See: 'MASTER_VERSION=4' below:
Thread-49::DEBUG::2012-11-29 20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=2', u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad', 'DESCRIPTION=512gb2', 'CLASS=Data', 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175', 'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4' , 'ROLE=Master', 'POOL_DESCRIPTION=UB1', u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active', 'POOL_SPM_LVER=541', '_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e', 'POOL_SPM_ID=-1']
2012-11-30 4:08, Simon Donnellan:
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an attempt to activate the host, then activate the iSCSI storage domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan < ykaplan@redhat.com
wrote:
Hi Simon, We could use some more information in order to understand the problem, could you please attach the vdsm logs? Thanks, Yeela
----- Original Message -----
From: "Simon Donnellan" < f@baconwho.re > To: users@ovirt.org Sent: Thursday, November 29, 2012 7:35:18 PM Subject: [Users] Storage domain issue iSCSI
Hi Everyone,
I'm having an issue following a power cut, none of my 3 nodes (All Fedora 17 / oVirt 3.1) are able to attach the master Domain.
in /var/log/messages I see the following on each node:
Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version 3 it is version 4 Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR {'status': {'message': "Wrong Master domain or its version: 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb, pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
I've tried reboots/restarts/node re-installs
I can see the PV and the iSCSI sessions fine from the shell.
As this is the master, none of my nodes will start.
Any help would be great.
Kind Regards
Simon
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming@cn.ibm.com or shuming@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC
participants (3)
-
Shu Ming -
Simon Donnellan -
Yeela Kaplan