Re: Ovirt + Gluster : How do I gain access to the file systems of the VMs
by Hanson Turner
Hi Guys,
I've an answer... Here's how I did it...
First, I needed kpartx ... so
#apt-get install kpartx
Then setup a loopback device for the raw hdd image
#losetup /dev/loop4 [IMAGE FILE]
#kpartx -a /dev/loop4
This allowed me to mount the various partitions included in the VM.
There you can modify the configs, make backups etc.
Thanks,
Hanson
On 06/19/2018 09:31 AM, Hanson Turner wrote:
>
> Hi Sahina,
>
> Thanks for your reply, I can copy the files off without issue. Using
> either a remote mount gluster, or just use the node and scp the files
> to where I want them.
>
> I was asking how to/do I mount the VM's disk in a way to be able to
> pull/modify files that are on the HDD of the VM.
>
> Thanks,
>
> Hason
>
>
> On 06/19/2018 05:02 AM, Sahina Bose wrote:
>>
>>
>> On Mon, Jun 18, 2018 at 5:12 PM, Hanson Turner
>> <hanson(a)andrewswireless.net <mailto:hanson@andrewswireless.net>> wrote:
>>
>> Hi Guys,
>>
>> My engine has corrupted, and while waiting for help, I'd like to
>> see if I can pull some data off the VM's to re purpose back onto
>> dedicated hardware.
>>
>> Our setup is/was a gluster based storage system for VM's. The
>> gluster data storage I'm assuming is okay, I think the hosted
>> engine is hosed, and needs restored, but that's another thread.
>>
>> I can copy the raw disk file off of the gluster data domain.
>> What's the best way to mount it short of importing it into
>> another gluster domain?
>>
>> With vmware, we can grab the disk file and move it from server to
>> server without issue. You can mount and explore contents with
>> workstation.
>>
>>
>> If you want to copy the image file. you can mount the gluster volume
>> and copy it.
>> using mount -t glusterfs <gluster-server>:/<volumename> <path-to-mount>
>>
>>
>> What do we have available to us for ovirt?
>>
>> Thanks,
>>
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org>
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> <mailto:users-leave@ovirt.org>
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> <https://www.ovirt.org/site/privacy-policy/>
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> <https://www.ovirt.org/community/about/community-guidelines/>
>> List Archives:
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/2AF5K2JERYH...
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/2AF5K2JERYH...>
>>
>>
>
6 years, 5 months
General failure
by nicolas＠devels.es
Hi,
We're running oVirt 4.1.9 (we cannot upgrade at this time) and we're
having a major problem in our infrastructure. On friday, a snapshots
were automatically created on more than 200 VMs and as this was just a
test task, all of them were deleted at the same time, which seems to
have corrupted several VMs.
When trying to delete a snapshot on some of the VMs, a "General error"
is thrown with a NullPointerException in the engine log (attached).
But the worst part is that when some of these machines is powered off
and then powered on, the VMs are corrupt...
VM myvm is down with error. Exit message: Bad volume specification
{u'index': 0, u'domainID': u'110ea376-d789-40a1-b9f6-6b40c31afe01',
'reqsize': '0', u'format': u'cow', u'bootOrder': u'1', u'address':
{u'function': u'0x0', u'bus': u'0x00', u'domain': u'0x0000', u'type':
u'pci', u'slot': u'0x06'}, u'volumeID':
u'1fd0f9aa-6505-45d2-a17e-859bd5dd4290', 'apparentsize': '23622320128',
u'imageID': u'65519220-68e1-462a-99b3-f0763c78eae2', u'discard': False,
u'specParams': {}, u'readonly': u'false', u'iface': u'virtio',
u'optional': u'false', u'deviceId':
u'65519220-68e1-462a-99b3-f0763c78eae2', 'truesize': '23622320128',
u'poolID': u'75bf8f48-970f-42bc-8596-f8ab6efb2b63', u'device': u'disk',
u'shared': u'false', u'propagateErrors': u'off', u'type': u'disk'}.
We're really frustrated by now and don't know how to procceed... We have
a DB backup (with engine-backup) from thursday which would have a "sane"
DB definition without all the snapshots, as they were all created on
friday. Would it be safe to restore this backup?
Any help is really appreciated...
Thanks.
6 years, 5 months
glusterfs-fuse consuming large amounts of ram
by Edward Clay
It looks like we are experiencing a bug in the version of glusterfs
included with ovirt 4.2.3. It looks like glusterfs 3.12.x has an issue
where it consumes large amounts of ram which has caused our HV report
storage errors and pause VMs.
https://bugzilla.redhat.com/show_bug.cgi?id=1496379
Is there a safe way to get glusterfs-fuse v3.13.x installed with ovirt
4.2.3 or do we have to live with this issue until future updates to
ovirt/glusterfs are released?
$ ssh hv1.domain.com "sudo grep glusterfs
/var/log/messages-* | grep -i kill "
/var/log/messages-20180610:Jun 6 13:49:54 hv1 kernel: Out of memory:
Kill process 15353 (glusterfs) score 630 or sacrifice child
/var/log/messages-20180610:Jun 6 13:49:54 hv1 kernel: Killed process
15353 (glusterfs) total-vm:33800604kB, anon-rss:31896632kB,
file-rss:840kB, shmem-rss:0kB
/var/log/messages-20180617:Jun 17 00:24:16 hv1 kernel: Out of memory:
Kill process 4072 (glusterfs) score 678 or sacrifice child
/var/log/messages-20180617:Jun 17 00:24:16 hv1 kernel: Killed process
4072 (glusterfs) total-vm:36159900kB, anon-rss:34338508kB,
file-rss:888kB, shmem-rss:0kB
We see this same issue occur on all hv in our cluster.
Edward Clay
Systems Administrator
The Hut Group<http://www.thehutgroup.com/>
Tel:
Email: edward.clay(a)uk2group.com<mailto:edward.clay@uk2group.com>
For the purposes of this email, the "company" means The Hut Group Limited, a company registered in England and Wales (company number 6539496) whose registered office is at Fifth Floor, Voyager House, Chicago Avenue, Manchester Airport, M90 3DQ and/or any of its respective subsidiaries.
Confidentiality Notice
This e-mail is confidential and intended for the use of the named recipient only. If you are not the intended recipient please notify us by telephone immediately on +44(0)1606 811888 or return it to us by e-mail. Please then delete it from your system and note that any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views or opinions are solely those of the author and do not necessarily represent those of the company.
Encryptions and Viruses
Please note that this e-mail and any attachments have not been encrypted. They may therefore be liable to be compromised. Please also note that it is your responsibility to scan this e-mail and any attachments for viruses. We do not, to the extent permitted by law, accept any liability (whether in contract, negligence or otherwise) for any virus infection and/or external compromise of security and/or confidentiality in relation to transmissions sent by e-mail.
Monitoring
Activity and use of the company's systems is monitored to secure its effective use and operation and for other lawful business purposes. Communications using these systems will also be monitored and may be recorded to secure effective use and operation and for other lawful business purposes.
hgvyjuv
6 years, 5 months
Slow on deleting file in VM with gluster as storage
by nha.pham
Hi all,
I had an issue with delete file in VM.
I used Ovirt with Gluster as Storage. I create new VM is fast, install everything is fast, also create and copy file.
But when I delete large number of file in VM, it really slow.
OS in VM is windows server 2012 r2.
Ovirt 4.2.3
Gluster 3.12.9
Could you please help me on this case.
Thanks.
6 years, 5 months
Ovirt + Gluster : How do I gain access to the file systems of the VMs
by Hanson Turner
Hi Guys,
My engine has corrupted, and while waiting for help, I'd like to see if
I can pull some data off the VM's to re purpose back onto dedicated
hardware.
Our setup is/was a gluster based storage system for VM's. The gluster
data storage I'm assuming is okay, I think the hosted engine is hosed,
and needs restored, but that's another thread.
I can copy the raw disk file off of the gluster data domain. What's the
best way to mount it short of importing it into another gluster domain?
With vmware, we can grab the disk file and move it from server to server
without issue. You can mount and explore contents with workstation.
What do we have available to us for ovirt?
Thanks,
6 years, 5 months
glusterfs backup-volfile-servers lost
by g.vasilopoulos＠uoc.gr
I have a 4 node setup (centos 7) with hosted engine on glusterfs (replica 3 arbiter1). Gluster fs is like this
ohost01 104G brick (real data)
ohost02 104g brick (real data)
ohost04 104g brick (arbiter)
ohost05 104g partition used as nfs-storage.
hosted engine is on gluster. I also have an fc domain of 3,6 TB
mount of gluster is like this
storage=172.16.224.10:/engine
mnt_options=backup-volfile-servers=172.16.224.11:172.16.224.13
172.16.224.10 is ohost01 storage network
172.16.224.12 is ohost02 storage network
172.16.224.13 is ohost04 storage network
Today I upgraded all nodes. I did it like this:
hosted-engine was running on ohost05 at the time
Put ohost04 (arbiter) on maintenance and did upgrade (ok)
Same with ohost02
Ohost01 was spm so I did ohost05 spm then put ohost01 on maintenance and then upgraded it. I notticed that engine VM paused during the process (which usualy does not happen) as I have backup-volfile-servers mount option. But today I notticed that this option is ignored. On the hosts I also noted that mount is like this
172.16.224.10:/engine on /rhev/data-center/mnt/glusterSD/172.16.224.10:_engine type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
so reduduncy is gone from gluster and I cannot figure out why.
If I restart ohost01 (after maintenance) hosted engine get paused until ohost01 comes back up.
How can I solve this issue
6 years, 5 months
Lost host after upgrade/reboot
by Nicolas Ecarnot
Hello,
TL;DR : engine stops talking with rebooted host.
[oVirt 4.2.3.5-1.el7.centos]
- From the web gui, upgrading a host, allowing the reboot checkbox checked
- upgrade is OK (/var/log/yum.log is showing successful updates + the
Ansible host deploy log is also OK)
- reboot is OK (clean, SSH OK...)
- the host eventually appears as "Install failed"
- the engine.log is telling :
> 2018-06-19 10:02:24,896+02 ERROR
> [org.ovirt.engine.core.bll.SshHostRebootCommand]
> (EE-ManagedThreadFactory-commandCoordinator-Thread-7) [6e32b3ac] SSH
> reboot command failed on host 'serv-hv-prds06': SSH session timeout
> host 'root@ serv-hv-prds06' Stdout: Stderr: 2018-06-19
> 10:02:25,028+02 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-commandCoordinator-Thread-7) [6e32b3ac]
> EVENT_ID: SYSTEM_FAILED_SSH_HOST_RESTART(198), A restart usin g SSH
> initiated by the engine to Host serv-hv-prds06 has failed. 2018-06-19
> 10:02:25,185+02 INFO
> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (EE-ManagedThreadFactory-commandCoordinator-Thread-7) [6e32b3ac]
> START, SetVdsStatusVDSCommand(HostName = serv-hv-prds06,
> SetVdsStatusVDSCom
> mandParameters:{hostId='9c1566a4-8432-4de6-b30d-fd3b8e5fafca',
> status='InstallFailed', nonOperationalReason='NONE',
> stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
> 833f9bd 2018-06-19 10:02:25,191+02 INFO
> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (EE-ManagedThreadFactory-commandCoordinator-Thread-7) [6e32b3ac]
> FINISH, SetVdsStatusVDSCommand, log id: 833f9bd 2018-06-19
> 10:02:25,191+02 ERROR
> [org.ovirt.engine.core.bll.hostdeploy.UpgradeHostInternalCommand]
> (EE-ManagedThreadFactory-commandCoordinator-Thread-7) [6e32b3ac]
> Engine failed to restart via ssh host 'serv-hv-prds06' ('9c1566a4-
> 8432-4de6-b30d-fd3b8e5fafca') after upgrade 2018-06-19
> 10:02:25,256+02 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-commandCoordinator-Thread-7)
> [8b7c6e7d-1a22-407c-818b-849e67b94051] EVENT_ID:
> HOST_UPGRADE_FAILED(841 ), Failed to upgrade Host serv-hv-prds06
> (User: necarnot@sdis.isere.fr(a)SDIS38-authz). 2018-06-19
> 10:02:30,755+02 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-engineScheduled-Thread-69)
> [8b7c6e7d-1a22-407c-818b-849e67b94051] EVENT_ID:
> HOST_UPGRADE_FAILED(841), Failed to upgrade Host serv-hv-prds06
> (User: necarnot@sdis.isere.fr(a)SDIS38-authz).
- Manually activating the host puts it back on track without issue
The usual SSH communications between the engine and the host are usually
very sound (VM migrations, maintenance...).
On this oVirt DC, I reproduced this issue twice on 2 different hosts.
In this engine log above, you see that I'm using my account to manage
this engine, as I 'm doing for years with no issue.
I'll try the exact same path with admin@internal to see what could
change, but I don't see the link.
What other logs could I give you to debug this?
Regards,
--
Nicolas ECARNOT
6 years, 5 months
RHEL 7.5 as an oVirt host - missing packages
by jasonmicron＠gmail.com
Hello everyone,
I believe there is a potential issue with the RHEL 7 yum repos and at least one missing RPM to convert a pre-existing RHEL 7.5 system into an oVirt host node.
I have the following YUM repos enabled in my RHEL 7 system, according to the documentation:
rhel-7-server-rpms
rhel-7-server-optional-rpms
rhel-7-server-extras-rpms
I installed the oVirt 4.2 RPM:
ovirt-release42-4.2.3.1-1.el7.noarch
I then go to the engine GUI to add the new host, and during the installation process I am informed that there is a YUM transaction error with the following package being unavailable:
python-qpid-proton (as a dependency install for the python-gofer-proton package)
I dug around Red Hat's YUM repos and discovered that this package only exists in the rhel-7-server-satellite-tools-6.{1..3}-rpms YUM repos from the official Red Hat site.
I tried enabling this repo (specifically the rhel-7-server-satellite-tools-6.3-rpms repo) to see which version of the package would be installed, and it appears to be a custom spin for Satellite in the release version:
=============================================================================================================
Package Arch Version Repository Size
=============================================================================================================
Installing:
python-gofer-proton noarch 2.7.7-3.el7sat lab1-rhel-7-server-satellite-tools-6.3-rpms 54 k
Installing for dependencies:
python-gofer noarch 2.7.7-3.el7sat lab1-rhel-7-server-satellite-tools-6.3-rpms 132 k
python-qpid-proton x86_64 0.16.0-7.el7 lab1-rhel-7-server-satellite-tools-6.3-rpms 216 k
qpid-proton-c x86_64 0.16.0-7.el7 lab1-rhel-7-server-satellite-tools-6.3-rpms 131 k
Transaction Summary
==============================================================================================================
Install 1 Package (+3 Dependent packages)
Since I shouldn't even require this repo according to the official oVirt install documentation for hosts, and the fact that these packages appear to be custom spins (el7sat) versions, I was curious if this is OK to use for oVirt or if I have encountered a dependency bug with RHEL 7.5 and oVirt 4.2?
Any advice would be welcomed.
6 years, 5 months
