11:35 a.m.
Hi Everyone,
I'm having an issue following a power cut, none of my 3 nodes (All Fedora
17 / oVirt 3.1) are able to attach the master Domain.
in /var/log/messages I see the following on each node:
Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master
domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected version
3 it is version 4
Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR
Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error
Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR {'status':
{'message': "Wrong Master domain or its version:
'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,
pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
I've tried reboots/restarts/node re-installs
I can see the PV and the iSCSI sessions fine from the shell.
As this is the master, none of my nodes will start.
Any help would be great.
Kind Regards
Simon
12:03 p.m.
Hi Simon,
We could use some more information in order to understand the problem,
could you please attach the vdsm logs?
Thanks,
Yeela
----- Original Message -----
From: "Simon Donnellan" <f(a)baconwho.re>
To: users(a)ovirt.org
Sent: Thursday, November 29, 2012 7:35:18 PM
Subject: [Users] Storage domain issue iSCSI
Hi Everyone,
I'm having an issue following a power cut, none of my 3 nodes (All
Fedora 17 / oVirt 3.1) are able to attach the master Domain.
in /var/log/messages I see the following on each node:
Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master
domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected
version 3 it is version 4
Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR
Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error
Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR
{'status': {'message': "Wrong Master domain or its version:
'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,
pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
I've tried reboots/restarts/node re-installs
I can see the PV and the iSCSI sessions fine from the shell.
As this is the master, none of my nodes will start.
Any help would be great.
Kind Regards
Simon
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
2:08 p.m.
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an attempt
to activate the host, then activate the iSCSI storage domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan <ykaplan(a)redhat.com> wrote:
Hi Simon,
We could use some more information in order to understand the problem,
could you please attach the vdsm logs?
Thanks,
Yeela
----- Original Message -----
> From: "Simon Donnellan" <f(a)baconwho.re>
> To: users(a)ovirt.org
> Sent: Thursday, November 29, 2012 7:35:18 PM
> Subject: [Users] Storage domain issue iSCSI
>
>
> Hi Everyone,
>
> I'm having an issue following a power cut, none of my 3 nodes (All
> Fedora 17 / oVirt 3.1) are able to attach the master Domain.
>
> in /var/log/messages I see the following on each node:
>
> Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master
> domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected
> version 3 it is version 4
> Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR
> Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error
> Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR
> {'status': {'message': "Wrong Master domain or its version:
> 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,
> pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
>
> I've tried reboots/restarts/node re-installs
>
> I can see the PV and the iSCSI sessions fine from the shell.
>
> As this is the master, none of my nodes will start.
>
> Any help would be great.
>
> Kind Regards
>
> Simon
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
9:03 a.m.
Your log indicates connection only to nfs storage:
/home/iso
/share/MD0_DATA/VMs
can you tell me their UUIDs?
Also a DC is allowed to contain only one storage type (nfs / block),
which does not fit with your question regarding the iSCSI SD, since you have attached nfs
SD to your DC.
Please check this again and return with more details...
----- Original Message -----
From: "Simon Donnellan" <f(a)baconwho.re>
To: "Yeela Kaplan" <ykaplan(a)redhat.com>
Cc: users(a)ovirt.org
Sent: Thursday, November 29, 2012 10:08:17 PM
Subject: Re: [Users] Storage domain issue iSCSI
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an
attempt to activate the host, then activate the iSCSI storage
domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan < ykaplan(a)redhat.com >
wrote:
Hi Simon,
We could use some more information in order to understand the
problem,
could you please attach the vdsm logs?
Thanks,
Yeela
----- Original Message -----
> From: "Simon Donnellan" < f(a)baconwho.re >
> To: users(a)ovirt.org
> Sent: Thursday, November 29, 2012 7:35:18 PM
> Subject: [Users] Storage domain issue iSCSI
>
>
> Hi Everyone,
>
> I'm having an issue following a power cut, none of my 3 nodes (All
> Fedora 17 / oVirt 3.1) are able to attach the master Domain.
>
> in /var/log/messages I see the following on each node:
>
> Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested
> master
> domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected
> version 3 it is version 4
> Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR
> Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error
> Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR
> {'status': {'message': "Wrong Master domain or its version:
> 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,
> pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
>
> I've tried reboots/restarts/node re-installs
>
> I can see the PV and the iSCSI sessions fine from the shell.
>
> As this is the master, none of my nodes will start.
>
> Any help would be great.
>
> Kind Regards
>
> Simon
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
9:13 a.m.
This is a multi-part message in MIME format.
--------------060402000606000203070701
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
I think the error is clear. Engine was expecting a master storage
domain metadata format version 3, while the master storage metadata gave
version 4. I am wondering if the master storage domain metadata was
corrupted during the power off.
See: the error came from.
Thread-54::ERROR::2012-11-29
20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain) Requested
master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have
expected version 3 it is version 4
See: 'MASTER_VERSION=4' below:
Thread-49::DEBUG::2012-11-29
20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh)
read lines (VGTagMetadataRW)=['VERSION=2',
u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0',
'TYPE=ISCSI', 'LOGBLKSIZE=512',
'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3',
'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512',
'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad', 'DESCRIPTION=512gb2',
'CLASS=Data', 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175',
'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4'_,_
'ROLE=Master', 'POOL_DESCRIPTION=UB1',
u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active',
'POOL_SPM_LVER=541',
'_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e', 'POOL_SPM_ID=-1']
2012-11-30 4:08, Simon Donnellan:
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an
attempt to activate the host, then activate the iSCSI storage domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan <ykaplan(a)redhat.com
<mailto:ykaplan@redhat.com>> wrote:
Hi Simon,
We could use some more information in order to understand the problem,
could you please attach the vdsm logs?
Thanks,
Yeela
----- Original Message -----
> From: "Simon Donnellan" <f(a)baconwho.re
<mailto:f@baconwho.re>>
> To: users(a)ovirt.org <mailto:users@ovirt.org>
> Sent: Thursday, November 29, 2012 7:35:18 PM
> Subject: [Users] Storage domain issue iSCSI
>
>
> Hi Everyone,
>
> I'm having an issue following a power cut, none of my 3 nodes (All
> Fedora 17 / oVirt 3.1) are able to attach the master Domain.
>
> in /var/log/messages I see the following on each node:
>
> Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested
master
> domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected
> version 3 it is version 4
> Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR
> Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error
> Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR
> {'status': {'message': "Wrong Master domain or its
version:
> 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,
> pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
>
> I've tried reboots/restarts/node re-installs
>
> I can see the PV and the iSCSI sessions fine from the shell.
>
> As this is the master, none of my nodes will start.
>
> Any help would be great.
>
> Kind Regards
>
> Simon
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
---
?? Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or
shuming(a)linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193,
PRC
--------------060402000606000203070701
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">I think the error is clear.
Engine was
expecting a master storage domain metadata format version 3, while
the master storage metadata gave version 4. I am wondering if
the master storage domain metadata was corrupted during the power
off.<br>
<br>
See: the error came from.<br>
Thread-54::ERROR::2012-11-29
20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain)
Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does
not have expected version 3 it is version 4<br>
<br>
See: 'MASTER_VERSION=4' below:<br>
<br>
Thread-49::DEBUG::2012-11-29
20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh)
read lines (VGTagMetadataRW)=['VERSION=2',
u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0',
'TYPE=ISCSI', 'LOGBLKSIZE=512',
'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3',
'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=',
'PHYBLKSIZE=512',
'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad',
'DESCRIPTION=512gb2', 'CLASS=Data',
'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175',
'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60',
'MASTER_VERSION=4'<u>,</u>
'ROLE=Master', 'POOL_DESCRIPTION=UB1',
u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active',
'POOL_SPM_LVER=541',
'_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e',
'POOL_SPM_ID=-1']<br>
<br>
<br>
2012-11-30 4:08, Simon Donnellan:<br>
</div>
<blockquote
cite="mid:CACLk-0yisx1+LtcOLmR9R_ojKEpDQZ=NmbJhx2SRBqtdue5hDQ@mail.gmail.com"
type="cite">Hi Yeela,<br>
<br>
Thanks for the reply, I've attached a vdsm.log file containing an
attempt to activate the host, then activate the iSCSI storage
domain.<br>
<br>
Thanks<br>
<br>
Simon<br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ykaplan@redhat.com"
target="_blank">ykaplan(a)redhat.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Simon,<br>
We could use some more information in order to understand
the problem,<br>
could you please attach the vdsm logs?<br>
Thanks,<br>
Yeela<br>
<div>
<div class="h5"><br>
----- Original Message -----<br>
> From: "Simon Donnellan" <<a
moz-do-not-send="true"
href="mailto:f@baconwho.re">f@baconwho.re</a>><br>
> To: <a moz-do-not-send="true"
href="mailto:users@ovirt.org">users@ovirt.org</a><br>
> Sent: Thursday, November 29, 2012 7:35:18 PM<br>
> Subject: [Users] Storage domain issue iSCSI<br>
><br>
><br>
> Hi Everyone,<br>
><br>
> I'm having an issue following a power cut, none of
my 3 nodes (All<br>
> Fedora 17 / oVirt 3.1) are able to attach the
master Domain.<br>
><br>
> in /var/log/messages I see the following on each
node:<br>
><br>
> Nov 29 17:15:41 hades vdsm Storage.StoragePool
ERROR Requested master<br>
> domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does
not have expected<br>
> version 3 it is version 4<br>
> Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR<br>
>
Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected
error<br>
> Nov 29 17:15:41 hades vdsm
Storage.Dispatcher.Protect ERROR<br>
> {'status': {'message': "Wrong Master domain
or its
version:<br>
> 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,<br>
> pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'",
'code': 324}}<br>
><br>
> I've tried reboots/restarts/node re-installs<br>
><br>
> I can see the PV and the iSCSI sessions fine from
the shell.<br>
><br>
> As this is the master, none of my nodes will start.<br>
><br>
> Any help would be great.<br>
><br>
> Kind Regards<br>
><br>
> Simon<br>
><br>
</div>
</div>
> _______________________________________________<br>
> Users mailing list<br>
> <a moz-do-not-send="true"
href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> <a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626 Tieline: 9051626 E-mail: <a
class="moz-txt-link-abbreviated"
href="mailto:shuming@cn.ibm.com">shuming@cn.ibm.com</a> or <a
class="moz-txt-link-abbreviated"
href="mailto:shuming@linux.vnet.ibm.com">shuming@linux.vnet.ibm.com</a>
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193,
PRC</pre>
</body>
</html>
--------------060402000606000203070701--
9:55 a.m.
Hi Yeela, Shu,
Many thanks for your replies.
I'm aware of the one type rule, the two NFS shares you noticed are the ISO
share and an export store. (I'm unable to find a way to create these as
iSCSI type)
The UUID of the master iSCSI domain is f1b40ecc-b6a9-44e7-
92cb-0fdf445e3175
it's name in the gui is "512gb2"
I too believe there is a meta data corruption, is there any way to get my
systems back up and running?
Kind Regards
Simon
On Sun, Dec 2, 2012 at 3:13 PM, Shu Ming <shuming(a)linux.vnet.ibm.com> wrote:
I think the error is clear. Engine was expecting a master storage
domain metadata format version 3, while the master storage metadata gave
version 4. I am wondering if the master storage domain metadata was
corrupted during the power off.
See: the error came from.
Thread-54::ERROR::2012-11-29
20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain) Requested
master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected
version 3 it is version 4
See: 'MASTER_VERSION=4' below:
Thread-49::DEBUG::2012-11-29
20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh) read
lines (VGTagMetadataRW)=['VERSION=2',
u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0',
'TYPE=ISCSI', 'LOGBLKSIZE=512',
'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3',
'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512',
'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad', 'DESCRIPTION=512gb2',
'CLASS=Data', 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175',
'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60',
'MASTER_VERSION=4'*,*'ROLE=Master', 'POOL_DESCRIPTION=UB1',
u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active',
'POOL_SPM_LVER=541',
'_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e',
'POOL_SPM_ID=-1']
2012-11-30 4:08, Simon Donnellan:
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an attempt
to activate the host, then activate the iSCSI storage domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan <ykaplan(a)redhat.com> wrote:
> Hi Simon,
> We could use some more information in order to understand the problem,
> could you please attach the vdsm logs?
> Thanks,
> Yeela
>
> ----- Original Message -----
> > From: "Simon Donnellan" <f(a)baconwho.re>
> > To: users(a)ovirt.org
> > Sent: Thursday, November 29, 2012 7:35:18 PM
> > Subject: [Users] Storage domain issue iSCSI
> >
> >
> > Hi Everyone,
> >
> > I'm having an issue following a power cut, none of my 3 nodes (All
> > Fedora 17 / oVirt 3.1) are able to attach the master Domain.
> >
> > in /var/log/messages I see the following on each node:
> >
> > Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested master
> > domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected
> > version 3 it is version 4
> > Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR
> > Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error
> > Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR
> > {'status': {'message': "Wrong Master domain or its
version:
> > 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,
> > pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
> >
> > I've tried reboots/restarts/node re-installs
> >
> > I can see the PV and the iSCSI sessions fine from the shell.
> >
> > As this is the master, none of my nodes will start.
> >
> > Any help would be great.
> >
> > Kind Regards
> >
> > Simon
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
_______________________________________________
Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or
shuming(a)linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193,
PRC
10:46 a.m.
Just a clarification, the UUID of the pool is f1b40ecc-b6a9-44e7-92cb-0fdf445e3175
and the UUID of the msd it is looking for is 68d8b0e2-c348-4cfe-a896-08c62d491dfb
(according to the logs you sent me).
The problem really is in the msd version (Thanks Shu) but I need more details in order to
solve the issue,
can you also attach the engine logs?
thanks.
----- Original Message -----
From: "Simon Donnellan" <f(a)baconwho.re>
To: "Shu Ming" <shuming(a)linux.vnet.ibm.com>
Cc: "Yeela Kaplan" <ykaplan(a)redhat.com>, users(a)ovirt.org
Sent: Sunday, December 2, 2012 5:55:00 PM
Subject: Re: [Users] Storage domain issue iSCSI
Hi Yeela, Shu,
Many thanks for your replies.
I'm aware of the one type rule, the two NFS shares you noticed are
the ISO share and an export store. (I'm unable to find a way to
create these as iSCSI type)
The UUID of the master iSCSI domain is f1b40ecc-b6a9-44e7-
92cb-0fdf445e3175
it's name in the gui is "512gb2"
I too believe there is a meta data corruption, is there any way to
get my systems back up and running?
Kind Regards
Simon
On Sun, Dec 2, 2012 at 3:13 PM, Shu Ming < shuming(a)linux.vnet.ibm.com
> wrote:
I think the error is clear. Engine was expecting a master storage
domain metadata format version 3, while the master storage metadata
gave version 4. I am wondering if the master storage domain metadata
was corrupted during the power off.
See: the error came from.
Thread-54::ERROR::2012-11-29
20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain)
Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does
not have expected version 3 it is version 4
See: 'MASTER_VERSION=4' below:
Thread-49::DEBUG::2012-11-29
20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh)
read lines (VGTagMetadataRW)=['VERSION=2',
u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0',
'TYPE=ISCSI', 'LOGBLKSIZE=512',
'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3',
'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512',
'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad',
'DESCRIPTION=512gb2', 'CLASS=Data',
'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175',
'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4' ,
'ROLE=Master', 'POOL_DESCRIPTION=UB1',
u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active',
'POOL_SPM_LVER=541',
'_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e',
'POOL_SPM_ID=-1']
2012-11-30 4:08, Simon Donnellan:
Hi Yeela,
Thanks for the reply, I've attached a vdsm.log file containing an
attempt to activate the host, then activate the iSCSI storage
domain.
Thanks
Simon
On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan < ykaplan(a)redhat.com >
wrote:
Hi Simon,
We could use some more information in order to understand the
problem,
could you please attach the vdsm logs?
Thanks,
Yeela
----- Original Message -----
> From: "Simon Donnellan" < f(a)baconwho.re >
> To: users(a)ovirt.org
> Sent: Thursday, November 29, 2012 7:35:18 PM
> Subject: [Users] Storage domain issue iSCSI
>
>
> Hi Everyone,
>
> I'm having an issue following a power cut, none of my 3 nodes (All
> Fedora 17 / oVirt 3.1) are able to attach the master Domain.
>
> in /var/log/messages I see the following on each node:
>
> Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested
> master
> domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected
> version 3 it is version 4
> Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR
> Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error
> Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR
> {'status': {'message': "Wrong Master domain or its version:
> 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,
> pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
>
> I've tried reboots/restarts/node re-installs
>
> I can see the PV and the iSCSI sessions fine from the shell.
>
> As this is the master, none of my nodes will start.
>
> Any help would be great.
>
> Kind Regards
>
> Simon
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or
shuming(a)linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun
Software Park, Haidian District, Beijing 100193, PRC
4:02 p.m.
Hi Yeela,
Attached are the engine logs.
Kind Regards
Simon
On Sun, Dec 2, 2012 at 4:46 PM, Yeela Kaplan <ykaplan(a)redhat.com> wrote:
Just a clarification, the UUID of the pool is
f1b40ecc-b6a9-44e7-92cb-0fdf445e3175
and the UUID of the msd it is looking for is
68d8b0e2-c348-4cfe-a896-08c62d491dfb (according to the logs you sent me).
The problem really is in the msd version (Thanks Shu) but I need more
details in order to solve the issue,
can you also attach the engine logs?
thanks.
----- Original Message -----
> From: "Simon Donnellan" <f(a)baconwho.re>
> To: "Shu Ming" <shuming(a)linux.vnet.ibm.com>
> Cc: "Yeela Kaplan" <ykaplan(a)redhat.com>, users(a)ovirt.org
> Sent: Sunday, December 2, 2012 5:55:00 PM
> Subject: Re: [Users] Storage domain issue iSCSI
>
> Hi Yeela, Shu,
>
> Many thanks for your replies.
>
> I'm aware of the one type rule, the two NFS shares you noticed are
> the ISO share and an export store. (I'm unable to find a way to
> create these as iSCSI type)
>
> The UUID of the master iSCSI domain is f1b40ecc-b6a9-44e7-
> 92cb-0fdf445e3175
>
> it's name in the gui is "512gb2"
>
> I too believe there is a meta data corruption, is there any way to
> get my systems back up and running?
>
> Kind Regards
>
> Simon
>
>
>
>
> On Sun, Dec 2, 2012 at 3:13 PM, Shu Ming < shuming(a)linux.vnet.ibm.com
> > wrote:
>
>
>
>
> I think the error is clear. Engine was expecting a master storage
> domain metadata format version 3, while the master storage metadata
> gave version 4. I am wondering if the master storage domain metadata
> was corrupted during the power off.
>
> See: the error came from.
> Thread-54::ERROR::2012-11-29
> 20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain)
> Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does
> not have expected version 3 it is version 4
>
> See: 'MASTER_VERSION=4' below:
>
> Thread-49::DEBUG::2012-11-29
> 20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh)
> read lines (VGTagMetadataRW)=['VERSION=2',
>
u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0',
> 'TYPE=ISCSI', 'LOGBLKSIZE=512',
> 'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3',
> 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=',
'PHYBLKSIZE=512',
> 'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad',
> 'DESCRIPTION=512gb2', 'CLASS=Data',
> 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175',
> 'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4'
,
> 'ROLE=Master', 'POOL_DESCRIPTION=UB1',
>
u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active',
> 'POOL_SPM_LVER=541',
> '_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e',
> 'POOL_SPM_ID=-1']
>
>
> 2012-11-30 4:08, Simon Donnellan:
>
>
>
>
> Hi Yeela,
>
> Thanks for the reply, I've attached a vdsm.log file containing an
> attempt to activate the host, then activate the iSCSI storage
> domain.
>
> Thanks
>
> Simon
>
>
>
>
> On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan < ykaplan(a)redhat.com >
> wrote:
>
>
> Hi Simon,
> We could use some more information in order to understand the
> problem,
> could you please attach the vdsm logs?
> Thanks,
> Yeela
>
>
>
> ----- Original Message -----
> > From: "Simon Donnellan" < f(a)baconwho.re >
> > To: users(a)ovirt.org
> > Sent: Thursday, November 29, 2012 7:35:18 PM
> > Subject: [Users] Storage domain issue iSCSI
> >
> >
> > Hi Everyone,
> >
> > I'm having an issue following a power cut, none of my 3 nodes (All
> > Fedora 17 / oVirt 3.1) are able to attach the master Domain.
> >
> > in /var/log/messages I see the following on each node:
> >
> > Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested
> > master
> > domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have expected
> > version 3 it is version 4
> > Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR
> > Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error
> > Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR
> > {'status': {'message': "Wrong Master domain or its
version:
> > 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,
> > pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
> >
> > I've tried reboots/restarts/node re-installs
> >
> > I can see the PV and the iSCSI sessions fine from the shell.
> >
> > As this is the master, none of my nodes will start.
> >
> > Any help would be great.
> >
> > Kind Regards
> >
> > Simon
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
>
>
> _______________________________________________
> Users mailing list Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
> --
> ---
> 舒明 Shu Ming
> Open Virtualization Engineerning; CSTL, IBM Corp.
> Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or
> shuming(a)linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun
> Software Park, Haidian District, Beijing 100193, PRC
>
9:29 a.m.
Simon,
I need to see the logs at the same time from both engine and vdsm,
they are a few days apart.
Can you get me the right logs?
thanks.
----- Original Message -----
From: "Simon Donnellan" <f(a)baconwho.re>
To: "Yeela Kaplan" <ykaplan(a)redhat.com>
Cc: users(a)ovirt.org
Sent: Tuesday, December 4, 2012 12:02:08 AM
Subject: Re: [Users] Storage domain issue iSCSI
Hi Yeela,
Attached are the engine logs.
Kind Regards
Simon
On Sun, Dec 2, 2012 at 4:46 PM, Yeela Kaplan < ykaplan(a)redhat.com >
wrote:
Just a clarification, the UUID of the pool is
f1b40ecc-b6a9-44e7-92cb-0fdf445e3175
and the UUID of the msd it is looking for is
68d8b0e2-c348-4cfe-a896-08c62d491dfb (according to the logs you sent
me).
The problem really is in the msd version (Thanks Shu) but I need more
details in order to solve the issue,
can you also attach the engine logs?
thanks.
----- Original Message -----
> From: "Simon Donnellan" < f(a)baconwho.re >
> To: "Shu Ming" < shuming(a)linux.vnet.ibm.com >
> Cc: "Yeela Kaplan" < ykaplan(a)redhat.com >, users(a)ovirt.org
> Sent: Sunday, December 2, 2012 5:55:00 PM
> Subject: Re: [Users] Storage domain issue iSCSI
>
> Hi Yeela, Shu,
>
> Many thanks for your replies.
>
> I'm aware of the one type rule, the two NFS shares you noticed are
> the ISO share and an export store. (I'm unable to find a way to
> create these as iSCSI type)
>
> The UUID of the master iSCSI domain is f1b40ecc-b6a9-44e7-
> 92cb-0fdf445e3175
>
> it's name in the gui is "512gb2"
>
> I too believe there is a meta data corruption, is there any way to
> get my systems back up and running?
>
> Kind Regards
>
> Simon
>
>
>
>
> On Sun, Dec 2, 2012 at 3:13 PM, Shu Ming <
> shuming(a)linux.vnet.ibm.com
> > wrote:
>
>
>
>
> I think the error is clear. Engine was expecting a master storage
> domain metadata format version 3, while the master storage metadata
> gave version 4. I am wondering if the master storage domain
> metadata
> was corrupted during the power off.
>
> See: the error came from.
> Thread-54::ERROR::2012-11-29
> 20:06:02,491::sp::1532::Storage.StoragePool::(getMasterDomain)
> Requested master domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does
> not have expected version 3 it is version 4
>
> See: 'MASTER_VERSION=4' below:
>
> Thread-49::DEBUG::2012-11-29
> 20:05:58,337::persistentDict::234::Storage.PersistentDict::(refresh)
> read lines (VGTagMetadataRW)=['VERSION=2',
>
u'PV0=pv:36001405c2f5e9d2d3be7d41a8db27dd6,uuid:b62d1B-zFVl-LKrH-fekH-vprs-znOZ-IF6jJy,pestart:0,pecount:4093,mapoffset:0',
> 'TYPE=ISCSI', 'LOGBLKSIZE=512',
> 'SDUUID=68d8b0e2-c348-4cfe-a896-08c62d491dfb', 'LEASERETRIES=3',
> 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=',
'PHYBLKSIZE=512',
> 'VGUUID=q7sGQ7-1G03-s9mh-dIyx-NSmo-P0zk-23c3Ad',
> 'DESCRIPTION=512gb2', 'CLASS=Data',
> 'POOL_UUID=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175',
> 'IOOPTIMEOUTSEC=10', 'LEASETIMESEC=60', 'MASTER_VERSION=4'
,
> 'ROLE=Master', 'POOL_DESCRIPTION=UB1',
>
u'POOL_DOMAINS=afec8026-ccac-4366-bb4b-2150d8731e4c:Active,c2b01420-fc73-4ccc-a560-3e1c5aa28a9f:Active,45fa93a8-1761-4522-bafb-c5d3ab45f731:Attached,68d8b0e2-c348-4cfe-a896-08c62d491dfb:Active',
> 'POOL_SPM_LVER=541',
> '_SHA_CKSUM=d54ce32f30c8040449f2a91ccc6e115e35894a5e',
> 'POOL_SPM_ID=-1']
>
>
> 2012-11-30 4:08, Simon Donnellan:
>
>
>
>
> Hi Yeela,
>
> Thanks for the reply, I've attached a vdsm.log file containing an
> attempt to activate the host, then activate the iSCSI storage
> domain.
>
> Thanks
>
> Simon
>
>
>
>
> On Thu, Nov 29, 2012 at 6:03 PM, Yeela Kaplan < ykaplan(a)redhat.com
> >
> wrote:
>
>
> Hi Simon,
> We could use some more information in order to understand the
> problem,
> could you please attach the vdsm logs?
> Thanks,
> Yeela
>
>
>
> ----- Original Message -----
> > From: "Simon Donnellan" < f(a)baconwho.re >
> > To: users(a)ovirt.org
> > Sent: Thursday, November 29, 2012 7:35:18 PM
> > Subject: [Users] Storage domain issue iSCSI
> >
> >
> > Hi Everyone,
> >
> > I'm having an issue following a power cut, none of my 3 nodes
> > (All
> > Fedora 17 / oVirt 3.1) are able to attach the master Domain.
> >
> > in /var/log/messages I see the following on each node:
> >
> > Nov 29 17:15:41 hades vdsm Storage.StoragePool ERROR Requested
> > master
> > domain 68d8b0e2-c348-4cfe-a896-08c62d491dfb does not have
> > expected
> > version 3 it is version 4
> > Nov 29 17:15:41 hades vdsm TaskManager.Task ERROR
> > Task=`f06fd1bb-46d1-47d7-80ca-c2e01becdc51`::Unexpected error
> > Nov 29 17:15:41 hades vdsm Storage.Dispatcher.Protect ERROR
> > {'status': {'message': "Wrong Master domain or its
version:
> > 'SD=68d8b0e2-c348-4cfe-a896-08c62d491dfb,
> > pool=f1b40ecc-b6a9-44e7-92cb-0fdf445e3175'", 'code': 324}}
> >
> > I've tried reboots/restarts/node re-installs
> >
> > I can see the PV and the iSCSI sessions fine from the shell.
> >
> > As this is the master, none of my nodes will start.
> >
> > Any help would be great.
> >
> > Kind Regards
> >
> > Simon
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
>
>
> _______________________________________________
> Users mailing list Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
> --
> ---
> 舒明 Shu Ming
> Open Virtualization Engineerning; CSTL, IBM Corp.
> Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or
> shuming(a)linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun
> Software Park, Haidian District, Beijing 100193, PRC
>
4158
days inactive
4164
days old
8 comments
3 participants
participants (3)
-
Shu Ming -
Simon Donnellan -
Yeela Kaplan