- Users - oVirt List Archives

Re: [Users] networking: basic vlan help
by Juan Pablo Lorier 27 Jan '14

27 Jan '14

This is a multi-part message in MIME format. --------------010004070404070407050707 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi Itamar, I don't know if I get your post right, but to me, it seems that if so many users hit the same rock, it should mean that this should be documented somewhere visible and in my opinion, push on getting bug 1049476 <https://bugzilla.redhat.com/show_bug.cgi?id=1049476> solved asap. Regards, --------------010004070404070407050707 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi Itamar, I don't know if I get your post right, but to me, it seems that if so many users hit the same rock, it should mean that this should be documented somewhere visible and in my opinion, push on getting bug <a name="b1049476" href="https://bugzilla.redhat.com/show_bug.cgi?id=1049476">1049476</a> solved asap. Regards, </body> </html> --------------010004070404070407050707--

4 4

[Users] Reboot causes poweroff of VM 3.4 Beta
by Jon Archer 27 Jan '14

27 Jan '14

Hi, Seem to be suffering an issue in 3.4 where if a vm is rebooted it actually shuts down, this occurs for all guests regardless of OS installed within. Anyone seen this? Jon

4 10

[Users] Issues starting hosted engine VM
by Andrew Lau 27 Jan '14

27 Jan '14

Hi, With the great help from sbonazzo, I managed to step past the initial bug with the hosted-engine-setup but appear to have run into another show stopper. I ran through the install process successfully up to the stage where it completed and the engine VM was to be shutdown. (The engine has already been installed on the VM and the host has been connected to the engine). The issue starts here that the host finds itself not able to start the VM up again. VDSM Logs: http://www.fpaste.org/69592/00427141/ ovirt-hosted-engine-ha agent.log http://www.fpaste.org/69595/43609139/ It seems to keep failing to start the VM.. when I restart the agent I can see the score drop to 0 after 3 boot attempts. The interesting thing seems to be in the VDSM Logs "'Virtual machine does not exist', 'code': 1}}" I'm not sure where else to look. Suggestions? Cheers, Andrew

6 25

[Users] Power saving cluster policy (3.4 test day)
by Lior Vernia 27 Jan '14

27 Jan '14

Hello, Just wanted to drop a line here about this feature I tested last week, which I thought was cool. I only tested it with two hosts and a single VM, using one of the hosts as a reserve, but it was quite satisfying to see the hosts being powered off and on as the VM was being run and stopped. This could be interesting to look at for those of you running a moderately-sized deployment, if often the load on your hosts is low. Yours, Lior.

3 2

[Users] sanlock leases on VM disks
by José Luis Sanz Boixader 27 Jan '14

27 Jan '14

I have an oVirt testing setup with 3 hosts running for a few weeks: CentOS 6.4, oVirt 3.3.1, VDSM 4.13.0, iSCSI based storage domain. I have just realized that sanlock has no leases on VM disks, so nothing prevents vdsm/libvirt from starting a VM on two different hosts, corrupting disk data. I know that something has to go wrong on oVirt engine to do it, but I've manually forced some errors ("Setting Host state to Non-Operational", "VM xxxx is not responding") for a "Highly available" VM and oVirt engine started that VM on another host. oVirt engine was not aware, but the VM was running on two hosts. I think this is a job for libvirt/sanlock/wdmd, but libvirt is not receiving "lease" tags for disks when creating domains. I think it should. What's left in my config? What am I doing wrong? Thanks

2 3

[Users] oVirt 3.4 test day: iscsi multipath
by Francesco Romani 27 Jan '14

27 Jan '14

Hi everyone, During the last oVirt 3.4 test day I played a bit with ISCSI multipath: http://www.ovirt.org/Feature/iSCSI-Multipath#Configure_iSCSI_Multipathing It took me longer than expected to properly set up things but, once done, everything looked fine at the config level (both seen from engine and the actual OS configuration) and worked well on my tests, including physical disconnections mid-I/O. Well done! :) -- Francesco Romani RedHat Engineering Virtualization R & D IRC: fromani

1 0

[Users] multiple storage domains
by Michal Skrivanek 27 Jan '14

27 Jan '14

Hey, I was trying to play with multiple storage domain during the 3.4 test day and even though it took me ages to set storage up correctly, the feature itself seems to be working just fine. Much easier to select what kind of storage I want to use…just Shared vs Local storage pool. VM could use volumes from different domains and I didn't have to worry about the right type (other than local vs shared, obviously:) Thanks, michal

1 0

[Users] CANCELED: Feature Review: Gluster Volume Snapshots
by Blue Jeans Network 27 Jan '14

27 Jan '14

------=_Part_794_1616979341.1390801575818 Content-Type: multipart/related; boundary="----=_Part_795_166231402.1390801575818" ------=_Part_795_166231402.1390801575818 Content-Type: multipart/alternative; boundary="----=_Part_796_2122432314.1390801575818" ------=_Part_796_2122432314.1390801575818 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Shubhendu Ram Tripathi (shtripat(a)redhat.com) has cancelled the following meeting: Meeting Title: Feature Review: Gluster Volume Snapshots Meeting Time: Monday January 27, 2014 6:30 p.m. IST / 1 hr ************************************************ Message/Description: Cancelled ************************************************ Blue Jeans Network 2012 ------=_Part_796_2122432314.1390801575818 Content-Type: text/html;charset="utf-8" Content-Transfer-Encoding: 7bit <table width="100%"> <tr bgcolor="#d8d8d8"> <td align="center"> <table width="600px" bgcolor="#FFFFFF" style="margin-top:20px;padding:0px 50px 10px 50px;margin-bottom:20px;text-align:left;"> <tr> <td style="padding-right:10px;padding-bottom:20px;"> <a href="http://bluejeans.com"> <img src="https://static.bluejeans.com/z1/static/images/bjn/BlueJeans_Logo.png" alt="Blue Jeans" style="border:none;"> </a> </td> </tr> <tr> <td style="font-size: 20px;color: #333333; padding-bottom:30px;"> <a href="mailto:shtripat@redhat.com">Shubhendu Ram Tripathi</a> has cancelled the following meeting: </td> </tr> <tr> <td style="padding-top:10px; font-size: 16px;"> Meeting Title: Feature Review: Gluster Volume Snapshots </td> </tr> <tr> <td style="padding-top:10px; padding-bottom: 10px; font-size: 16px;"> Meeting Time: Monday January 27, 2014 • 6:30 p.m. IST / 1 hr </td> </tr> <tr> <td> <hr size="1" noshade="noshade" style="line-height:1px;color:#CCCCCC;"> </td> </tr> <tr> <td style="style="padding-top:10px;""> Message/Description: Cancelled </td> </tr> <tr> <td style="line-height:1px;color:#CCCCCC;"> <hr size="1" noshade="noshade" style="line-height:1px;color:#CCCCCC;border-color:#CCCCCC;"> </td> </tr> <tr> <td style="font-size:11px;text-align:left"> ©Blue Jeans Network 2013 </td> </tr> </table> </td> </tr> </table> ------=_Part_796_2122432314.1390801575818 Content-Type: text/calendar; method=REQUEST; charset="utf-8" Content-Transfer-Encoding: 7bit BEGIN:VCALENDAR VERSION:2.0 METHOD:CANCEL PRODID:-//PYVOBJECT//NONSGML Version 1//EN BEGIN:VTIMEZONE TZID:Asia/Kolkata TZURL:http://tzurl.org/zoneinfo-outlook/Asia/Kolkata X-LIC-LOCATION:Asia/Kolkata BEGIN:STANDARD TZOFFSETFROM:+0530 TZOFFSETTO:+0530 TZNAME:IST DTSTART:19700101T000000 END:STANDARD END:VTIMEZONE BEGIN:VEVENT UID:0H44B_sN0PzTAnM2@H6D3JmW4G7v71tX DTSTART;TZID=Asia/Kolkata:20140127T183000 DTEND;TZID=Asia/Kolkata:20140127T193000 ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=DECLINED:mailto:u sers(a)ovirt.org CREATED:20140121T113648Z DESCRIPTION:\nShubhendu Ram Tripathi (shtripat(a)redhat.com) has cancelled t his meeting.\n\n************************************************\n\nMessag e/Description: Cancelled\n\n\n******************************************** ****\n\nBlue Jeans Network 2012\n DTSTAMP:20140127T054615Z LAST-MODIFIED:20140127T054615Z LOCATION: ORGANIZER:mailto:shtripat@redhat.com SEQUENCE:1 STATUS:CANCELLED SUMMARY:Feature Review: Gluster Volume Snapshots TRANSP:OPAQUE END:VEVENT END:VCALENDAR ------=_Part_796_2122432314.1390801575818-- ------=_Part_795_166231402.1390801575818-- ------=_Part_794_1616979341.1390801575818 Content-Type: application/ics; name=invite.ics Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=invite.ics BEGIN:VCALENDAR VERSION:2.0 METHOD:CANCEL PRODID:-//PYVOBJECT//NONSGML Version 1//EN BEGIN:VTIMEZONE TZID:Asia/Kolkata TZURL:http://tzurl.org/zoneinfo-outlook/Asia/Kolkata X-LIC-LOCATION:Asia/Kolkata BEGIN:STANDARD TZOFFSETFROM:+0530 TZOFFSETTO:+0530 TZNAME:IST DTSTART:19700101T000000 END:STANDARD END:VTIMEZONE BEGIN:VEVENT UID:0H44B_sN0PzTAnM2@H6D3JmW4G7v71tX DTSTART;TZID=Asia/Kolkata:20140127T183000 DTEND;TZID=Asia/Kolkata:20140127T193000 ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=DECLINED:mailto:u sers(a)ovirt.org CREATED:20140121T113648Z DESCRIPTION:\nShubhendu Ram Tripathi (shtripat(a)redhat.com) has cancelled t his meeting.\n\n************************************************\n\nMessag e/Description: Cancelled\n\n\n******************************************** ****\n\nBlue Jeans Network 2012\n DTSTAMP:20140127T054615Z LAST-MODIFIED:20140127T054615Z LOCATION: ORGANIZER:mailto:shtripat@redhat.com SEQUENCE:1 STATUS:CANCELLED SUMMARY:Feature Review: Gluster Volume Snapshots TRANSP:OPAQUE END:VEVENT END:VCALENDAR ------=_Part_794_1616979341.1390801575818--

1 0

[Users] Cancelled: Feature Review: Gluster Volume Snapshot
by Shubhendu Tripathi 27 Jan '14

27 Jan '14

------=_Part_8859059_641577150.1390801417195 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit The following meeting has been cancelled: Subject: Feature Review: Gluster Volume Snapshot Organizer: "Shubhendu Tripathi" <shtripat(a)redhat.com> Location: "Vijayanagara - BLR" <vijayanagara-blr(a)redhat.com> Resources: "Vijayanagara - BLR" <vijayanagara-blr(a)redhat.com> (Vijayanagara - BLR) Time: Monday, January 27, 2014, 6:30:00 PM - 7:30:00 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Invitees: arch(a)ovirt.org; users(a)ovirt.org; rnachimu(a)redhat.com; don(a)techcetera.com; otavio.ferranti(a)eldorado.org.br; lucas.lima(a)suinfra.ufrgs.br; pagupta(a)redhat.com; ehildesh(a)redhat.com; srevivo(a)redhat.com; Karli.Sjoberg(a)slu.se; pportant(a)redhat.com ... *~*~*~*~*~*~*~*~*~* Hi, The meeting is cancelled as the feature is being dropped from the current release because of other high priority features being taken up. Extremely sorry for the last moment cancellation. Will schedule the feature review later once we take up this feature. Thanks and Regards, Shubhendu Tripathi ------=_Part_8859059_641577150.1390801417195 Content-Type: text/calendar; charset=utf-8; method=CANCEL; name=meeting.ics Content-Transfer-Encoding: quoted-printable BEGIN:VCALENDAR PRODID:Zimbra-Calendar-Provider VERSION:2.0 METHOD:CANCEL BEGIN:VTIMEZONE TZID:Asia/Kolkata BEGIN:STANDARD DTSTART:16010101T000000 TZOFFSETTO:+0530 TZOFFSETFROM:+0530 TZNAME:IST END:STANDARD END:VTIMEZONE BEGIN:VEVENT UID:3440764a-ef38-4491-a023-f369dcfe9a18 SUMMARY:Cancelled: Feature Review: Gluster Volume Snapshot COMMENT:The following meeting has been cancelled: LOCATION:"Vijayanagara - BLR" <vijayanagara-blr(a)redhat.com> ATTENDEE;ROLE=3DREQ-PARTICIPANT;PARTSTAT=3DNEEDS-ACTION;RSVP=3DTRUE:mailto:= arch@ov irt.org ATTENDEE;ROLE=3DREQ-PARTICIPANT;PARTSTAT=3DNEEDS-ACTION;RSVP=3DTRUE:mailto:= users@o virt.org ATTENDEE;CN=3DVijayanagara - BLR;CUTYPE=3DRESOURCE;ROLE=3DNON-PARTICIPANT;P= ARTSTAT =3DACCEPTED;RSVP=3DTRUE:mailto:vijayanagara-blr@redhat.com ATTENDEE;CN=3DRamesh Nachimuthu;PARTSTAT=3DACCEPTED:mailto:rnachimu@redhat.= com ATTENDEE;CN=3DDon Harris;PARTSTAT=3DTENTATIVE:mailto:don@techcetera.com ATTENDEE;CN=3DOtavio Luiz Ferranti;PARTSTAT=3DDECLINED:mailto:otavio.ferran= ti@el dorado.org.br ATTENDEE;CN=3DLucas Lima;PARTSTAT=3DACCEPTED:mailto:lucas.lima@suinfra.ufrg= s.br ATTENDEE;CN=3DPankaj Gupta;PARTSTAT=3DACCEPTED:mailto:pagupta@redhat.com ATTENDEE;CN=3DEldan Hildesheim;PARTSTAT=3DTENTATIVE:mailto:ehildesh@redhat.= com ATTENDEE;CN=3DShai Revivo;PARTSTAT=3DTENTATIVE:mailto:srevivo@redhat.com ATTENDEE;CN=3D"Karli Sj=C3=B6berg";PARTSTAT=3DDECLINED:mailto:Karli.Sjoberg= @slu.se ATTENDEE;CN=3DPeter Portante;PARTSTAT=3DDECLINED:mailto:pportant@redhat.com ATTENDEE;CUTYPE=3DINDIVIDUAL;PARTSTAT=3DDECLINED:mailto:dbishop@controlscan= .com ATTENDEE;CN=3DRic Wheeler;PARTSTAT=3DTENTATIVE:mailto:rwheeler@redhat.com ATTENDEE;CN=3DPavithra;PARTSTAT=3DACCEPTED:mailto:psriniva@redhat.com ATTENDEE;CN=3DSatya Vempati;PARTSTAT=3DACCEPTED:mailto:svempati@tintri.com ATTENDEE;CN=3DEldad Marciano;PARTSTAT=3DACCEPTED:mailto:emarcian@redhat.com ORGANIZER;CN=3DShubhendu Tripathi:mailto:shtripat@redhat.com DTSTART;TZID=3D"Asia/Kolkata":20140127T183000 DTEND;TZID=3D"Asia/Kolkata":20140127T193000 STATUS:CANCELLED CLASS:PUBLIC X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY TRANSP:OPAQUE LAST-MODIFIED:20140127T054337Z DTSTAMP:20140127T054337Z SEQUENCE:1 DESCRIPTION:The following meeting has been cancelled:\n\nSubject: Feature R= e view: Gluster Volume Snapshot \nOrganizer: "Shubhendu Tripathi" <shtripat@= re dhat.com> \n\nLocation: "Vijayanagara - BLR" <vijayanagara-blr(a)redhat.com>= \ nResources: "Vijayanagara - BLR" <vijayanagara-blr(a)redhat.com> (Vijayanaga= ra - BLR) \nTime: Monday\, January 27\, 2014\, 6:30:00 PM - 7:30:00 PM GMT += 05 :30 Chennai\, Kolkata\, Mumbai\, New Delhi\n \nInvitees: arch(a)ovirt.org\; = us ers(a)ovirt.org\; rnachimu(a)redhat.com\; don(a)techcetera.com\; otavio.ferranti= @e ldorado.org.br\; lucas.lima(a)suinfra.ufrgs.br\; pagupta(a)redhat.com\; ehilde= sh @redhat.com\; srevivo(a)redhat.com\; Karli.Sjoberg(a)slu.se\; pportant(a)redhat.= co m ... \n\n\n*~*~*~*~*~*~*~*~*~*\n\nHi\,\n\nThe meeting is cancelled as the= f eature is being dropped from the current release because of other high pri= or ity features being taken up.\nExtremely sorry for the last moment cancella= ti on.\n\nWill schedule the feature review later once we take up this feature= .\ n\nThanks and Regards\,\nShubhendu Tripathi END:VEVENT END:VCALENDAR ------=_Part_8859059_641577150.1390801417195--

1 0

[Users] sanlock can't read empty 'ids' file
by Ted Miller 27 Jan '14

27 Jan '14

--------------000306010201050105090700 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit After some indirect path to getting here, I believe that the cause of my problem is with sanlock. (See thread on 'Data Center Status on the GUI flips between "Non Responsive" and "Contending" ' for how I got here). My sanlock.log file is full of: 2014-01-26 21:36:19-0500 36474 [2905]: s4576 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0 2014-01-26 21:36:19-0500 36474 [26767]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512 2014-01-26 21:36:19-0500 36474 [26767]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids 2014-01-26 21:36:20-0500 36475 [2905]: s4576 add_lockspace fail result -90 2014-01-26 21:36:26-0500 36481 [2906]: s4577 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0 2014-01-26 21:36:26-0500 36481 [26809]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512 2014-01-26 21:36:26-0500 36481 [26809]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids 2014-01-26 21:36:27-0500 36482 [2906]: s4577 add_lockspace fail result -90 The only other thing is after rebooting, when it says: 2014-01-26 11:28:53-0500 29 [2897]: sanlock daemon started 2.8 host 09423427-2f70-4f68-9696-e41a93937ef0.office2a.l 2014-01-26 11:29:22-0500 57 [2905]: s1 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0 2014-01-26 11:29:22-0500 57 [3569]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c001000 result 0:0 match len 512 2014-01-26 11:29:22-0500 57 [3569]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids 2014-01-26 11:29:23-0500 58 [2905]: s1 add_lockspace fail result -90 2014-01-26 11:29:32-0500 67 [2905]: s2 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0 2014-01-26 11:29:32-0500 67 [3731]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512 2014-01-26 11:29:32-0500 67 [3731]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids 2014-01-26 11:29:33-0500 68 [2905]: s2 add_lockspace fail result -90 2014-01-26 11:29:42-0500 77 [2905]: s3 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0 2014-01-26 11:29:42-0500 77 [3806]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512 2014-01-26 11:29:42-0500 77 [3806]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids 2014-01-26 11:29:43-0500 78 [2905]: s3 add_lockspace fail result -90 Background: Setup is 2 hosts on Centos 6.5 with gluster storage, engine on separate Centos 6.5, all up to date. Originally set up running: Data Center: default Cluster: default Master storage domain: VM log had entries like: 2013-12-18 23:20:39-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -103:0 match res 2013-12-18 23:20:39-0500 4510 [3318]: s1 delta_renew read rv -103 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids 2013-12-18 23:20:39-0500 4510 [3318]: s1 renewal error -103 delta_length 9 last_success 4481 2013-12-18 23:20:40-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res 2013-12-18 23:20:40-0500 4510 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids 2013-12-18 23:20:40-0500 4510 [3318]: s1 renewal error -107 delta_length 0 last_success 4481 2013-12-18 23:20:40-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res 2013-12-18 23:20:40-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids 2013-12-18 23:20:40-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481 2013-12-18 23:20:41-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res 2013-12-18 23:20:41-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids 2013-12-18 23:20:41-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481 storage 'VM' had a Split-Brain. In trying to fix split-brain, file dom_md/ids was erased. I never revived the Data Center 'default' after that. log is full of: 2014-01-16 17:23:21-0500 62902 [29262]: open error -13 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids 2014-01-16 17:23:21-0500 62902 [29262]: s3 open_disk /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids error -13 2014-01-16 17:23:22-0500 62903 [3368]: s3 add_lockspace fail result -19 2014-01-16 17:23:31-0500 62912 [3368]: s4 lockspace 9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88:1:/rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids:0 Jan. 16: Created new: Data Center: mill; Cluster: one; Master storage domain: 'VM2' All was working. Logs looked like this: 2013-12-18 23:20:39-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -103:0 match res 2013-12-18 23:20:39-0500 4510 [3318]: s1 delta_renew read rv -103 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids 2013-12-18 23:20:39-0500 4510 [3318]: s1 renewal error -103 delta_length 9 last_success 4481 2013-12-18 23:20:40-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res 2013-12-18 23:20:40-0500 4510 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids 2013-12-18 23:20:40-0500 4510 [3318]: s1 renewal error -107 delta_length 0 last_success 4481 2013-12-18 23:20:40-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res 2013-12-18 23:20:40-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids 2013-12-18 23:20:40-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481 2013-12-18 23:20:41-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res 2013-12-18 23:20:41-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids 2013-12-18 23:20:41-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481 For a while, very little was logged at all, and this were working fine. Then, Jan 22 logs went from: 2014-01-22 00:44:56-0500 521397 [30933]: leader3 m 0 v 0 ss 0 nh 0 mh 0 oi 0 og 0 lv 0 2014-01-22 00:44:56-0500 521397 [30933]: leader4 sn rn ts 0 cs 0 2014-01-22 00:44:56-0500 521397 [30933]: s123 delta_renew verify_leader error -223 2014-01-22 00:44:56-0500 521397 [30933]: s123 renewal error -223 delta_length 0 last_success 521369 2014-01-22 00:44:57-0500 521398 [3362]: s123 kill 32599 sig 15 count 9 2014-01-22 00:44:57-0500 521398 [30933]: verify_leader 2 wrong magic 0 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids 2014-01-22 00:44:57-0500 521398 [30933]: leader1 delta_renew error -223 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c host_id 2 2014-01-22 00:44:57-0500 521398 [30933]: leader2 path /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids offset 0 2014-01-22 00:44:57-0500 521398 [30933]: leader3 m 0 v 0 ss 0 nh 0 mh 0 oi 0 og 0 lv 0 to: 2014-01-22 00:45:21-0500 521422 [17199]: 0322a407 aio collect 0 0x7f87ac0008c0:0x7f87ac0008d0:0x7f87ac101000 result 0:0 match len 512 2014-01-22 00:45:21-0500 521422 [17199]: read_sectors delta_leader offset 512 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids 2014-01-22 00:45:22-0500 521423 [3368]: s125 add_lockspace fail result -90 2014-01-22 00:45:25-0500 521426 [3368]: s126 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:2:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0 2014-01-22 00:45:25-0500 521426 [17243]: 0322a407 aio collect 0 0x7f87ac0008c0:0x7f87ac0008d0:0x7f87ac101000 result 0:0 match len 512 2014-01-22 00:45:25-0500 521426 [17243]: read_sectors delta_leader offset 512 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids 2014-01-22 00:45:26-0500 521427 [3368]: s126 add_lockspace fail result -90 2014-01-22 00:45:36-0500 521437 [3368]: s127 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:2:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0 and the Data Center has been down ever since. Best I can figure is that something is supposed to be in dom_md/ids, but that file is empty: ls /rhev/data-center/mnt/glusterSD/10.41.65.2\:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ -l total 1029 -rw-rw---- 1 vdsm kvm 0 Jan 22 00:44 ids -rw-rw---- 1 vdsm kvm 0 Jan 16 18:50 inbox -rw-rw---- 1 vdsm kvm 2097152 Jan 21 18:20 leases -rw-r--r-- 1 vdsm kvm 491 Jan 21 18:20 metadata -rw-rw---- 1 vdsm kvm 0 Jan 16 18:50 outbox Any hints as to how to put whatever is needed into 'ids', or reinitialize the sanlock system--or a better diagnosis and solution--gladly accepted. Ted Miller Elkhart, IN, USA --------------000306010201050105090700 Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> </head> <body text="#000000" bgcolor="#FFFFFF"> After some indirect path to getting here, I believe that the cause of my problem is with sanlock.  (See thread on 'Data Center Status on the GUI flips between "Non Responsive" and "Contending" '  for how I got here). My sanlock.log file is full of: <pre>2014-01-26 21:36:19-0500 36474 [2905]: s4576 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre> <pre>2014-01-26 21:36:19-0500 36474 [26767]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512</pre> <pre>2014-01-26 21:36:19-0500 36474 [26767]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre> <pre>2014-01-26 21:36:20-0500 36475 [2905]: s4576 add_lockspace fail result -90</pre> <pre>2014-01-26 21:36:26-0500 36481 [2906]: s4577 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre> <pre>2014-01-26 21:36:26-0500 36481 [26809]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512</pre> <pre>2014-01-26 21:36:26-0500 36481 [26809]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre> <pre>2014-01-26 21:36:27-0500 36482 [2906]: s4577 add_lockspace fail result -90</pre> The only other thing is after rebooting, when it says: <pre>2014-01-26 11:28:53-0500 29 [2897]: sanlock daemon started 2.8 host 09423427-2f70-4f68-9696-e41a93937ef0.office2a.l</pre> <pre>2014-01-26 11:29:22-0500 57 [2905]: s1 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre> <pre>2014-01-26 11:29:22-0500 57 [3569]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c001000 result 0:0 match len 512</pre> <pre>2014-01-26 11:29:22-0500 57 [3569]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre> <pre>2014-01-26 11:29:23-0500 58 [2905]: s1 add_lockspace fail result -90</pre> <pre>2014-01-26 11:29:32-0500 67 [2905]: s2 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre> <pre>2014-01-26 11:29:32-0500 67 [3731]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512</pre> <pre>2014-01-26 11:29:32-0500 67 [3731]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre> <pre>2014-01-26 11:29:33-0500 68 [2905]: s2 add_lockspace fail result -90</pre> <pre>2014-01-26 11:29:42-0500 77 [2905]: s3 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre> <pre>2014-01-26 11:29:42-0500 77 [3806]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512</pre> <pre>2014-01-26 11:29:42-0500 77 [3806]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre> <pre>2014-01-26 11:29:43-0500 78 [2905]: s3 add_lockspace fail result -90</pre> Background: Setup is 2 hosts on Centos 6.5 with gluster storage, engine on separate Centos 6.5, all up to date. Originally set up running: <blockquote>Data Center: default Cluster: default Master storage domain: VM </blockquote> log had entries like: <blockquote> <pre>2013-12-18 23:20:39-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -103:0 match res</pre> <pre>2013-12-18 23:20:39-0500 4510 [3318]: s1 delta_renew read rv -103 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre> <pre>2013-12-18 23:20:39-0500 4510 [3318]: s1 renewal error -103 delta_length 9 last_success 4481</pre> <pre>2013-12-18 23:20:40-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre> <pre>2013-12-18 23:20:40-0500 4510 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre> <pre>2013-12-18 23:20:40-0500 4510 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre> <pre>2013-12-18 23:20:40-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre> <pre>2013-12-18 23:20:40-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre> <pre>2013-12-18 23:20:40-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre> <pre>2013-12-18 23:20:41-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre> <pre>2013-12-18 23:20:41-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre> <pre>2013-12-18 23:20:41-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre> </blockquote> storage 'VM' had a Split-Brain.  In trying to fix split-brain, file dom_md/ids was erased.  I never revived the Data Center 'default' after that. <blockquote>log is full of: <pre>2014-01-16 17:23:21-0500 62902 [29262]: open error -13 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre> <pre>2014-01-16 17:23:21-0500 62902 [29262]: s3 open_disk /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids error -13</pre> <pre>2014-01-16 17:23:22-0500 62903 [3368]: s3 add_lockspace fail result -19</pre> <pre>2014-01-16 17:23:31-0500 62912 [3368]: s4 lockspace 9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88:1:/rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids:0</pre> </blockquote> Jan. 16: Created new:  Data Center: mill; Cluster: one;  Master storage domain: 'VM2'  All was working. <blockquote>Logs looked like this: <pre>2013-12-18 23:20:39-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -103:0 match res</pre> <pre>2013-12-18 23:20:39-0500 4510 [3318]: s1 delta_renew read rv -103 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre> <pre>2013-12-18 23:20:39-0500 4510 [3318]: s1 renewal error -103 delta_length 9 last_success 4481</pre> <pre>2013-12-18 23:20:40-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre> <pre>2013-12-18 23:20:40-0500 4510 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre> <pre>2013-12-18 23:20:40-0500 4510 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre> <pre>2013-12-18 23:20:40-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre> <pre>2013-12-18 23:20:40-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre> <pre>2013-12-18 23:20:40-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre> <pre>2013-12-18 23:20:41-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre> <pre>2013-12-18 23:20:41-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre> <pre>2013-12-18 23:20:41-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre> </blockquote> For a while, very little was logged at all, and this were working fine.  Then, Jan 22 logs went from: <blockquote> <pre>2014-01-22 00:44:56-0500 521397 [30933]: leader3 m 0 v 0 ss 0 nh 0 mh 0 oi 0 og 0 lv 0</pre> <pre>2014-01-22 00:44:56-0500 521397 [30933]: leader4 sn  rn  ts 0 cs 0</pre> <pre>2014-01-22 00:44:56-0500 521397 [30933]: s123 delta_renew verify_leader error -223</pre> <pre>2014-01-22 00:44:56-0500 521397 [30933]: s123 renewal error -223 delta_length 0 last_success 521369</pre> <pre>2014-01-22 00:44:57-0500 521398 [3362]: s123 kill 32599 sig 15 count 9</pre> <pre>2014-01-22 00:44:57-0500 521398 [30933]: verify_leader 2 wrong magic 0 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre> <pre>2014-01-22 00:44:57-0500 521398 [30933]: leader1 delta_renew error -223 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c host_id 2</pre> <pre>2014-01-22 00:44:57-0500 521398 [30933]: leader2 path /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids offset 0</pre> <pre>2014-01-22 00:44:57-0500 521398 [30933]: leader3 m 0 v 0 ss 0 nh 0 mh 0 oi 0 og 0 lv 0</pre> </blockquote> to: <blockquote> <pre>2014-01-22 00:45:21-0500 521422 [17199]: 0322a407 aio collect 0 0x7f87ac0008c0:0x7f87ac0008d0:0x7f87ac101000 result 0:0 match len 512</pre> <pre>2014-01-22 00:45:21-0500 521422 [17199]: read_sectors delta_leader offset 512 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre> <pre>2014-01-22 00:45:22-0500 521423 [3368]: s125 add_lockspace fail result -90</pre> <pre>2014-01-22 00:45:25-0500 521426 [3368]: s126 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:2:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre> <pre>2014-01-22 00:45:25-0500 521426 [17243]: 0322a407 aio collect 0 0x7f87ac0008c0:0x7f87ac0008d0:0x7f87ac101000 result 0:0 match len 512</pre> <pre>2014-01-22 00:45:25-0500 521426 [17243]: read_sectors delta_leader offset 512 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre> <pre>2014-01-22 00:45:26-0500 521427 [3368]: s126 add_lockspace fail result -90</pre> <pre>2014-01-22 00:45:36-0500 521437 [3368]: s127 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:2:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre> </blockquote> and the Data Center has been down ever since. Best I can figure is that something is supposed to be in dom_md/ids, but that file is empty: <pre>ls /rhev/data-center/mnt/glusterSD/10.41.65.2\:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ -l</pre> <pre>total 1029</pre> <pre>-rw-rw---- 1 vdsm kvm       0 Jan 22 00:44 ids</pre> <pre>-rw-rw---- 1 vdsm kvm       0 Jan 16 18:50 inbox</pre> <pre>-rw-rw---- 1 vdsm kvm 2097152 Jan 21 18:20 leases</pre> <pre>-rw-r--r-- 1 vdsm kvm     491 Jan 21 18:20 metadata</pre> <pre>-rw-rw---- 1 vdsm kvm       0 Jan 16 18:50 outbox </pre> Any hints as to how to put whatever is needed into 'ids', or reinitialize the sanlock system--or a better diagnosis and solution--gladly accepted. Ted Miller Elkhart, IN, USA </body> </html> --------------000306010201050105090700--

1 0