Can't import some VMs after storage domain detach and reattach to new datacenter.
Can you attach engine and vdsm logs? On Sun, Jun 23, 2019 at 11:29 AM m black <mblack0@yandex.ru> wrote:
Hi.
I have a problem with importing some VMs after importing storage domain in new datacenter.
I have 5 servers with oVirt version 4.1.7, hosted-engine setup and datacenter with iscsi, fc and nfs storages. Also i have 3 servers with oVirt 4.3.4, hosted-engine and nfs storage.
I've set iscsi and fc storages to maintenance and detached them successfully on 4.1.7 datacenter. Then i've imported these storage domains via Import Domain in 4.3.4 datacenter successfully.
After storage domains were imported to new 4.3.4 datacenter i've tried to import VMs from VM Import tab on storages.
On the FC storage it was good, all VMs imported and started, all VMs in place.
And with iSCSI storage i've got problems: On the iSCSI storage some VMs imported and started, but some of them missing, some of missing VMs disks are showing at Disk Import, i've tried to import disks from Disk Import tab and got error - 'Failed to register disk'. Tried to scan disks with 'Scan Disks' in storage domain, also tried 'Update OVF' - no result.
What caused this? What can i do to recover missing VMs? What logs to examine? Can it be storage domain disk corruption?
Please, help.
Thank you.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MF5IUXURKIQZNN...
Sorry for the delay. vdsm logs are huge, i'll cut when i moved storage domains and attach the logs.
I have seen similar situation , when a VM had one disk on 1 domain and 2nd disk on another storage domain. Are you sure that all disks of the problematic VMs were moved to the iSCSI storage domain ? Best Regards,Strahil Nikolov В неделя, 23 юни 2019 г., 11:28:56 ч. Гринуич+3, m black <mblack0@yandex.ru> написа: Hi. I have a problem with importing some VMs after importing storage domain in new datacenter. I have 5 servers with oVirt version 4.1.7, hosted-engine setup and datacenter with iscsi, fc and nfs storages. Also i have 3 servers with oVirt 4.3.4, hosted-engine and nfs storage. I've set iscsi and fc storages to maintenance and detached them successfully on 4.1.7 datacenter.Then i've imported these storage domains via Import Domain in 4.3.4 datacenter successfully. After storage domains were imported to new 4.3.4 datacenter i've tried to import VMs from VM Import tab on storages. On the FC storage it was good, all VMs imported and started, all VMs in place. And with iSCSI storage i've got problems:On the iSCSI storage some VMs imported and started, but some of them missing, some of missing VMs disks are showing at Disk Import, i've tried to import disks from Disk Import tab and got error - 'Failed to register disk'.Tried to scan disks with 'Scan Disks' in storage domain, also tried 'Update OVF' - no result. What caused this? What can i do to recover missing VMs? What logs to examine?Can it be storage domain disk corruption? Please, help. Thank you. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MF5IUXURKIQZNN...
Before detaching storages i've checked that all disks on these storages. After detaching in the old datacenter there were left empty storage domains with only disk of hosted-engine. Rechecked this today.
Take a look at the corresponding .meta file for the disks you can not import. I had the very same problem and it was caused by DISKTYPE=1 in .meta. When changed to DISKTYPE=DATA I was able to import disks correctly. Not the whole VM though.. -- Dmitry Filonov Linux Administrator SBGrid Core | Harvard Medical School 250 Longwood Ave, SGM-114 Boston, MA 02115 On Sun, Jun 23, 2019 at 4:29 AM m black <mblack0@yandex.ru> wrote:
Hi.
I have a problem with importing some VMs after importing storage domain in new datacenter.
I have 5 servers with oVirt version 4.1.7, hosted-engine setup and datacenter with iscsi, fc and nfs storages. Also i have 3 servers with oVirt 4.3.4, hosted-engine and nfs storage.
I've set iscsi and fc storages to maintenance and detached them successfully on 4.1.7 datacenter. Then i've imported these storage domains via Import Domain in 4.3.4 datacenter successfully.
After storage domains were imported to new 4.3.4 datacenter i've tried to import VMs from VM Import tab on storages.
On the FC storage it was good, all VMs imported and started, all VMs in place.
And with iSCSI storage i've got problems: On the iSCSI storage some VMs imported and started, but some of them missing, some of missing VMs disks are showing at Disk Import, i've tried to import disks from Disk Import tab and got error - 'Failed to register disk'. Tried to scan disks with 'Scan Disks' in storage domain, also tried 'Update OVF' - no result.
What caused this? What can i do to recover missing VMs? What logs to examine? Can it be storage domain disk corruption?
Please, help.
Thank you.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MF5IUXURKIQZNN...
That's interesting. Where can i find meta for block storage? On NFS storage these files are located next to the disk image.
On Tue, Jun 25, 2019 at 9:40 AM <mblack0@yandex.ru> wrote:
That's interesting.
Where can i find meta for block storage?
In block storage metadata is kept in the "metadata" logical volume. To find metadata for particular volume, you need to to look at the logical volume tags: lvs -o tags vg-name/lv-name Which will output tags like MD_42 The metadata for this volume is at slot number 42. To find the metadata, you need to calculate the offset in the metadata logical volume. On storage domain V4 format, the offset is: offset=$((42 * 512)) On storage domain V5 format, the offset is: offset=$((1048576 + 42 * 8192)) You can read the metadata like this: dd if=/dev/vg-name/metadata bs=512 count=1 skip=$offset conv=skip_bytes iflag=direct If you need help you can copy the metadata logical volume and share it here: dd if=/dev/vg-name/metadata bs=1M count=17 iflag=direct of=metadata-$(date +%Y-%m-%d-%H-%M) xz metadata-* Nir On NFS storage these files are located next to the disk image.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KX2MVSAKJOPAVT...
On Mon, Jun 24, 2019 at 7:47 PM Dmitry Filonov <filonov@hkl.hms.harvard.edu> wrote:
Take a look at the corresponding .meta file for the disks you can not import. I had the very same problem and it was caused by DISKTYPE=1 in .meta.
I want more info on this. We think that the only value ever used for DISKTYPE is 2. Do you have any info on how these disks were created? Maybe by some ancient version? Nir
When changed to DISKTYPE=DATA I was able to import disks correctly. Not the whole VM though..
-- Dmitry Filonov Linux Administrator SBGrid Core | Harvard Medical School 250 Longwood Ave, SGM-114 Boston, MA 02115
On Sun, Jun 23, 2019 at 4:29 AM m black <mblack0@yandex.ru> wrote:
Hi.
I have a problem with importing some VMs after importing storage domain in new datacenter.
I have 5 servers with oVirt version 4.1.7, hosted-engine setup and datacenter with iscsi, fc and nfs storages. Also i have 3 servers with oVirt 4.3.4, hosted-engine and nfs storage.
I've set iscsi and fc storages to maintenance and detached them successfully on 4.1.7 datacenter. Then i've imported these storage domains via Import Domain in 4.3.4 datacenter successfully.
After storage domains were imported to new 4.3.4 datacenter i've tried to import VMs from VM Import tab on storages.
On the FC storage it was good, all VMs imported and started, all VMs in place.
And with iSCSI storage i've got problems: On the iSCSI storage some VMs imported and started, but some of them missing, some of missing VMs disks are showing at Disk Import, i've tried to import disks from Disk Import tab and got error - 'Failed to register disk'. Tried to scan disks with 'Scan Disks' in storage domain, also tried 'Update OVF' - no result.
What caused this? What can i do to recover missing VMs? What logs to examine? Can it be storage domain disk corruption?
Please, help.
Thank you.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MF5IUXURKIQZNN...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BMXCPVIHKKQ3T7...
Hi Nir - in my case these VMs were migrated from VirtualBox to oVirt using some of the VMWare provided tool and then virt-v2v to convert images. Here's the example of the meta file - DOMAIN=92be9db3-eab4-47ed-9ee9-87b8616b7c8c VOLTYPE=LEAF CTIME=1529005629 MTIME=1529005629 IMAGE=f0d0b3b3-5a31-4c9f-b551-90586bf946a5 DISKTYPE=1 PUUID=00000000-0000-0000-0000-000000000000 LEGALITY=LEGAL POOL_UUID= SIZE=41943040 FORMAT=RAW TYPE=SPARSE DESCRIPTION=generated by virt-v2v 1.36.10rhel_7,release_6.el7_5.2,libvirt EOF These disks worked fine on 4.2.3.8 but I wasn't able to import them into 4.3.4.3 unless I changed DISKTYPE line manually. Fil -- Dmitry Filonov Linux Administrator SBGrid Core | Harvard Medical School 250 Longwood Ave, SGM-114 Boston, MA 02115 On Tue, Jun 25, 2019 at 5:41 AM Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Jun 24, 2019 at 7:47 PM Dmitry Filonov < filonov@hkl.hms.harvard.edu> wrote:
Take a look at the corresponding .meta file for the disks you can not import. I had the very same problem and it was caused by DISKTYPE=1 in .meta.
I want more info on this. We think that the only value ever used for DISKTYPE is 2.
Do you have any info on how these disks were created? Maybe by some ancient version?
Nir
When changed to DISKTYPE=DATA I was able to import disks correctly. Not the whole VM though..
-- Dmitry Filonov Linux Administrator SBGrid Core | Harvard Medical School 250 Longwood Ave, SGM-114 Boston, MA 02115
On Sun, Jun 23, 2019 at 4:29 AM m black <mblack0@yandex.ru> wrote:
Hi.
I have a problem with importing some VMs after importing storage domain in new datacenter.
I have 5 servers with oVirt version 4.1.7, hosted-engine setup and datacenter with iscsi, fc and nfs storages. Also i have 3 servers with oVirt 4.3.4, hosted-engine and nfs storage.
I've set iscsi and fc storages to maintenance and detached them successfully on 4.1.7 datacenter. Then i've imported these storage domains via Import Domain in 4.3.4 datacenter successfully.
After storage domains were imported to new 4.3.4 datacenter i've tried to import VMs from VM Import tab on storages.
On the FC storage it was good, all VMs imported and started, all VMs in place.
And with iSCSI storage i've got problems: On the iSCSI storage some VMs imported and started, but some of them missing, some of missing VMs disks are showing at Disk Import, i've tried to import disks from Disk Import tab and got error - 'Failed to register disk'. Tried to scan disks with 'Scan Disks' in storage domain, also tried 'Update OVF' - no result.
What caused this? What can i do to recover missing VMs? What logs to examine? Can it be storage domain disk corruption?
Please, help.
Thank you.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MF5IUXURKIQZNN...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BMXCPVIHKKQ3T7...
On Tue, Jun 25, 2019 at 3:15 PM Dmitry Filonov <filonov@hkl.hms.harvard.edu> wrote:
Hi Nir -
in my case these VMs were migrated from VirtualBox to oVirt using some of the VMWare provided tool and then virt-v2v to convert images. Here's the example of the meta file -
DOMAIN=92be9db3-eab4-47ed-9ee9-87b8616b7c8c VOLTYPE=LEAF CTIME=1529005629 MTIME=1529005629 IMAGE=f0d0b3b3-5a31-4c9f-b551-90586bf946a5 DISKTYPE=1 PUUID=00000000-0000-0000-0000-000000000000 LEGALITY=LEGAL POOL_UUID= SIZE=41943040 FORMAT=RAW TYPE=SPARSE DESCRIPTION=generated by virt-v2v 1.36.10rhel_7,release_6.el7_5.2,libvirt EOF
These disks worked fine on 4.2.3.8 but I wasn't able to import them into 4.3.4.3 unless I changed DISKTYPE line manually.
Do you have engine and vdsm logs from the time you imported this vm? Which engine version was used during the import? Nir
Sorry, don't have any logs from back then. That was some time ago and it was easy to fix so I didn't bother keeping logs. DISKTYPE and DESCRIPTION were the only two lines I had to fix to get disks imported nicely. If you like I can probably re-create situation by creating a VM, then unregistering it, changing the .meta file and try re-importing it back. -- Dmitry Filonov Linux Administrator SBGrid Core | Harvard Medical School 250 Longwood Ave, SGM-114 Boston, MA 02115 On Tue, Jun 25, 2019 at 10:42 AM Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Jun 25, 2019 at 3:15 PM Dmitry Filonov < filonov@hkl.hms.harvard.edu> wrote:
Hi Nir -
in my case these VMs were migrated from VirtualBox to oVirt using some of the VMWare provided tool and then virt-v2v to convert images. Here's the example of the meta file -
DOMAIN=92be9db3-eab4-47ed-9ee9-87b8616b7c8c VOLTYPE=LEAF CTIME=1529005629 MTIME=1529005629 IMAGE=f0d0b3b3-5a31-4c9f-b551-90586bf946a5 DISKTYPE=1 PUUID=00000000-0000-0000-0000-000000000000 LEGALITY=LEGAL POOL_UUID= SIZE=41943040 FORMAT=RAW TYPE=SPARSE DESCRIPTION=generated by virt-v2v 1.36.10rhel_7,release_6.el7_5.2,libvirt EOF
These disks worked fine on 4.2.3.8 but I wasn't able to import them into 4.3.4.3 unless I changed DISKTYPE line manually.
Do you have engine and vdsm logs from the time you imported this vm?
Which engine version was used during the import?
Nir
On Tue, Jun 25, 2019 at 8:39 PM Dmitry Filonov <filonov@hkl.hms.harvard.edu> wrote:
Sorry, don't have any logs from back then. That was some time ago and it was easy to fix so I didn't bother keeping logs. DISKTYPE and DESCRIPTION were the only two lines I had to fix to get disks imported nicely. If you like I can probably re-create situation by creating a VM, then unregistering it, changing the .meta file and try re-importing it back.
Sure, reproducing it easy. But we want to support only valid value created by older version of oVirt. if such volumes actually exists in the field, the system should handle the import transparently, translating the disk type to "DATA". Are you sure the metadata was not modified outside of oVirt? Nir
On Tue, Jun 25, 2019 at 10:42 AM Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Jun 25, 2019 at 3:15 PM Dmitry Filonov < filonov@hkl.hms.harvard.edu> wrote:
Hi Nir -
in my case these VMs were migrated from VirtualBox to oVirt using some of the VMWare provided tool and then virt-v2v to convert images. Here's the example of the meta file -
DOMAIN=92be9db3-eab4-47ed-9ee9-87b8616b7c8c VOLTYPE=LEAF CTIME=1529005629 MTIME=1529005629 IMAGE=f0d0b3b3-5a31-4c9f-b551-90586bf946a5 DISKTYPE=1 PUUID=00000000-0000-0000-0000-000000000000 LEGALITY=LEGAL POOL_UUID= SIZE=41943040 FORMAT=RAW TYPE=SPARSE DESCRIPTION=generated by virt-v2v 1.36.10rhel_7,release_6.el7_5.2,libvirt EOF
These disks worked fine on 4.2.3.8 but I wasn't able to import them into 4.3.4.3 unless I changed DISKTYPE line manually.
Do you have engine and vdsm logs from the time you imported this vm?
Which engine version was used during the import?
Nir
I have at least one of those VMs with DISKTYPE=1 on my old oVirt farm. Am pretty sure nobody ever touched metadata files outside of oVirt, but the disks were not created by oVirt but rather produced by virt-v2v and then imported into older version of oVirt. That happened several years ago and I wasn't involved with the project so don't have details. Fil -- Dmitry Filonov Linux Administrator SBGrid Core | Harvard Medical School 250 Longwood Ave, SGM-114 Boston, MA 02115 On Tue, Jun 25, 2019 at 3:35 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Jun 25, 2019 at 8:39 PM Dmitry Filonov < filonov@hkl.hms.harvard.edu> wrote:
Sorry, don't have any logs from back then. That was some time ago and it was easy to fix so I didn't bother keeping logs. DISKTYPE and DESCRIPTION were the only two lines I had to fix to get disks imported nicely. If you like I can probably re-create situation by creating a VM, then unregistering it, changing the .meta file and try re-importing it back.
Sure, reproducing it easy.
But we want to support only valid value created by older version of oVirt. if such volumes actually exists in the field, the system should handle the import transparently, translating the disk type to "DATA".
Are you sure the metadata was not modified outside of oVirt?
Nir
On Tue, Jun 25, 2019 at 10:42 AM Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Jun 25, 2019 at 3:15 PM Dmitry Filonov < filonov@hkl.hms.harvard.edu> wrote:
Hi Nir -
in my case these VMs were migrated from VirtualBox to oVirt using some of the VMWare provided tool and then virt-v2v to convert images. Here's the example of the meta file -
DOMAIN=92be9db3-eab4-47ed-9ee9-87b8616b7c8c VOLTYPE=LEAF CTIME=1529005629 MTIME=1529005629 IMAGE=f0d0b3b3-5a31-4c9f-b551-90586bf946a5 DISKTYPE=1 PUUID=00000000-0000-0000-0000-000000000000 LEGALITY=LEGAL POOL_UUID= SIZE=41943040 FORMAT=RAW TYPE=SPARSE DESCRIPTION=generated by virt-v2v 1.36.10rhel_7,release_6.el7_5.2,libvirt EOF
These disks worked fine on 4.2.3.8 but I wasn't able to import them into 4.3.4.3 unless I changed DISKTYPE line manually.
Do you have engine and vdsm logs from the time you imported this vm?
Which engine version was used during the import?
Nir
Hi Nir, Trying to revive this old thread, I ran in to a similar issue. I imported a few VMs from an old VMWARE version into ovirt 4.3.3 using OVAs generated on the VMWARE side. After adjusting a bit the OVAs I was able to do the import. A year later I have tried to export those VMs from 4.3.3 to a new Hyperconverged cluster that uses oVirt 4.3.10. If I try to import a VM with only 1 disk I need to modify the DISKTYPE from 1 to DATA, and Need to add the alias on the DESCRIPTION field as it does not appear on the v5 (ovirt 4.3.10). Original settings: Disk1.- [root@ovirt1 images]# strings a88461c4-61cb-4dbc-806e-484ce6fd5b4d/247d61c6-46a1-4c48-bb2d-55f6b8743692.meta DOMAIN=523debef-f166-407d-a8d8-9cfd6d20ebb7 VOLTYPE=LEAF CTIME=1574205481 MTIME=1574205481 IMAGE=a88461c4-61cb-4dbc-806e-484ce6fd5b4d DISKTYPE=1 PUUID=00000000-0000-0000-0000-000000000000 LEGALITY=LEGAL POOL_UUID= SIZE=209715200 FORMAT=RAW TYPE=SPARSE DESCRIPTION=generated by virt-v2v 1.38.2rhel_7,release_12.el7_6.2,libvirt Disk2.- [root@ovirt1 images]# strings 75044386-90db-4c16-94ad-0f376fa7ceea/d17a4bf4-e633-4e08-aa7c-42b22a2c8769.meta CAP=268435456000 CTIME=1575689283 DESCRIPTION=generated by virt-v2v 1.38.2rhel_7,release_12.el7_6.2,libvirt DISKTYPE=1 DOMAIN=523debef-f166-407d-a8d8-9cfd6d20ebb7 FORMAT=RAW GEN=0 IMAGE=75044386-90db-4c16-94ad-0f376fa7ceea LEGALITY=LEGAL PUUID=00000000-0000-0000-0000-000000000000 TYPE=PREALLOCATED VOLTYPE=LEAF I changed DISKTYPE=1 to DISKTYPE=DATA and DESCRIPTION=DESCRIPTION=generated by virt-v2v 1.38.2rhel_7,release_12.el7_6.2,libvirt TO DESCRIPTION={"DiskAlias":"MYVM-DISK1","DiskDescription":""} and for Disk2 DESCRIPTION={"DiskAlias":"MYVM-DISK2","DiskDescription":""} VMs with only 1 disk on their configuration only migrates the disk and only able to import the disk, not the VM from an NFS sharead data domain. So I had to create the VM and attach the imported disk and worked fine. Then I tranfered the disk from the NFS domain to the VMSTORE Gluster domain... up to here all good. Problem when I tried to do the same with a VM with 2 disks, up to starting the VM all was good, however migrating the storage to the VMSTORE Gluster domain the second disk stays in migration forever, issue related to deleting auto generated snapshot of the second disk. In order to half way solve the issue I had to follow up on: https://access.redhat.com/solutions/396753 https://access.redhat.com/solutions/1347513 https://access.redhat.com/solutions/3991591 /usr/share/ovirt-engine/setup/dbutils/ https://lists.ovirt.org/pipermail/users/2016-May/039898.html but this caused a lot of downtime, so right now 1 disk of the VM is on VMSTORE DOMAIN (Gluster), and one disk on the migration data domain (NFS). Seems I can't transfer the second disk to VMSTORE Domain. Any ideas? Thanks, AQ
participants (7)
-
adrianquintero＠gmail.com -
Benny Zlotnik -
Dmitry Filonov -
m black -
mblack0＠yandex.ru
-
Nir Soffer -
Strahil Nikolov