The issues is with machine that have already been imported once. I started cloudspin with a two drive consumer NAS, and it was slow and crappy so I migrated all of the machines that were on that drive to the gluster, which went off with few issues. A couple days ago the gluster that held the engine went into split brain hell, and crashed corrupting the engine. I rebuilt the engine, and the machine that were created on the gluster had no problems importing, it’s the machines that had already been imported once that have the issue. There is nothing in the vdsm logs related to this, it never makes it past the engine. Here is the error. 2015-02-07 08:46:21,445 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GlusterVolumesListVDSCommand(HostName = node3.cloudspin.me, HostId = 1d81be66-49b7-41a6-81ca-1c8dd55dcf3c), log id: 46f08f27 2015-02-07 08:46:21,487 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GlusterVolumesListVDSCommand, return: {a823009f-aa52-4b4b-b2bf-093ece3e8f02=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d475d4ca, e8ca5fa4-37ca-4cd9-a87b-e7c20243e021=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b3d17fb9, af67b75f-c8e1-44ec-aa42-69aed9bd9226=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@55088de0}, log id: 46f08f27 2015-02-07 08:46:26,499 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-7) [e4169ff] Lock Acquired to object EngineLock [exclusiveLocks= key: aks-test1 value: VM_NAME , sharedLocks= key: e4c0cf2d-61ed-4cbf-9c43-81e9a565755f value: REMOTE_VM ] 2015-02-07 08:46:26,516 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GlusterVolumesListVDSCommand(HostName = node3.cloudspin.me, HostId = 1d81be66-49b7-41a6-81ca-1c8dd55dcf3c), log id: 22aa4324
HERE IS THE ERROR<<<<<<<<< 2015-02-07 08:46:26,562 WARN [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-7) [e4169ff] CanDoAction of action ImportVmFromConfiguration failed for user donny@cloudspin. Reasons: VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST 2015-02-07 08:46:26,570 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-7) [e4169ff] Lock freed to object EngineLock [exclusiveLocks= key: aks-test1 value: VM_NAME , sharedLocks= key: e4c0cf2d-61ed-4cbf-9c43-81e9a565755f value: REMOTE_VM ] 2015-02-07 08:46:26,596 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GlusterVolumesListVDSCommand, return: {a823009f-aa52-4b4b-b2bf-093ece3e8f02=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@3cecea52, e8ca5fa4-37ca-4cd9-a87b-e7c20243e021=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@bdea2c2c, af67b75f-c8e1-44ec-aa42-69aed9bd9226=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4f63b2b3}, log id: 22aa4324
Thank you for your assistance Donny D -----Original Message----- From: Maor Lipchuk [mailto:mlipchuk@redhat.com] Sent: Saturday, February 7, 2015 4:08 AM To: DONALD DAVIS Cc: Allon Mureinik Subject: Re: Storage issues Hi Donald, can you please share your engine and vdsm logs. Have you done any move operation with the disks? That was the bug that was solved in 3.5.1 Thakns, Maor ----- Original Message -----
From: "Allon Mureinik" <amureini@redhat.com> To: "DONALD DAVIS" <donny@5dindustries.com> Cc: "Maor Lipchuk" <mlipchuk@redhat.com> Sent: Friday, February 6, 2015 7:16:47 PM Subject: Re: Storage issues
Hi Donald,
What version are you using (oVirt 3.5.0 presumably)?
Maor - isn't this a bug we solved in 3.5.1?
-Allon ----- Original Message -----
From: "DONALD DAVIS" <donny@5dindustries.com> To: amureini@redhat.com Sent: Friday, February 6, 2015 6:53:27 PM Subject: Storage issues
Hi,
I need some help getting my users vm’s imported back into the system after the failure yesterday. I reattached the storage and half of the vm’s imported without issue. The other half of the vms give this error
Error while executing action: Cannot import VM. Storage Domain doesn't exist
Funny part is, I only had one storage domain… and I imported it back into the engine… I’m confused.
Donny Davis
cloudspin.me
Hi Donald, Can you please elaborate more on the process, I'm not sure I understood the scenario clearly. Regards, Maor ----- Original Message -----
From: "DONALD DAVIS" <donny@5dindustries.com> To: "Maor Lipchuk" <mlipchuk@redhat.com> Cc: "Allon Mureinik" <amureini@redhat.com>, users@ovirt.org Sent: Saturday, February 7, 2015 5:48:06 PM Subject: RE: Storage issues
The issues is with machine that have already been imported once. I started cloudspin with a two drive consumer NAS, and it was slow and crappy so I migrated all of the machines that were on that drive to the gluster, which went off with few issues.
A couple days ago the gluster that held the engine went into split brain hell, and crashed corrupting the engine.
Is engine setup was on gluster, or the Storage domains which was used for the VM's disks?
I rebuilt the engine, and the machine that were created on the gluster had no problems importing, it’s the machines that had already been imported once that have the issue.
How they were imported?
There is nothing in the vdsm logs related to this, it never makes it past the engine. Here is the error.
2015-02-07 08:46:21,445 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GlusterVolumesListVDSCommand(HostName = node3.cloudspin.me, HostId = 1d81be66-49b7-41a6-81ca-1c8dd55dcf3c), log id: 46f08f27 2015-02-07 08:46:21,487 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GlusterVolumesListVDSCommand, return: {a823009f-aa52-4b4b-b2bf-093ece3e8f02=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d475d4ca, e8ca5fa4-37ca-4cd9-a87b-e7c20243e021=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b3d17fb9, af67b75f-c8e1-44ec-aa42-69aed9bd9226=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@55088de0}, log id: 46f08f27 2015-02-07 08:46:26,499 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-7) [e4169ff] Lock Acquired to object EngineLock [exclusiveLocks= key: aks-test1 value: VM_NAME , sharedLocks= key: e4c0cf2d-61ed-4cbf-9c43-81e9a565755f value: REMOTE_VM ] 2015-02-07 08:46:26,516 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GlusterVolumesListVDSCommand(HostName = node3.cloudspin.me, HostId = 1d81be66-49b7-41a6-81ca-1c8dd55dcf3c), log id: 22aa4324
HERE IS THE ERROR<<<<<<<<< 2015-02-07 08:46:26,562 WARN [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-7) [e4169ff] CanDoAction of action ImportVmFromConfiguration failed for user donny@cloudspin. Reasons: VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST 2015-02-07 08:46:26,570 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-7) [e4169ff] Lock freed to object EngineLock [exclusiveLocks= key: aks-test1 value: VM_NAME , sharedLocks= key: e4c0cf2d-61ed-4cbf-9c43-81e9a565755f value: REMOTE_VM ] 2015-02-07 08:46:26,596 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GlusterVolumesListVDSCommand, return: {a823009f-aa52-4b4b-b2bf-093ece3e8f02=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@3cecea52, e8ca5fa4-37ca-4cd9-a87b-e7c20243e021=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@bdea2c2c, af67b75f-c8e1-44ec-aa42-69aed9bd9226=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4f63b2b3}, log id: 22aa4324
Thank you for your assistance
Donny D
-----Original Message----- From: Maor Lipchuk [mailto:mlipchuk@redhat.com] Sent: Saturday, February 7, 2015 4:08 AM To: DONALD DAVIS Cc: Allon Mureinik Subject: Re: Storage issues
Hi Donald, can you please share your engine and vdsm logs. Have you done any move operation with the disks? That was the bug that was solved in 3.5.1
Thakns, Maor
----- Original Message -----
From: "Allon Mureinik" <amureini@redhat.com> To: "DONALD DAVIS" <donny@5dindustries.com> Cc: "Maor Lipchuk" <mlipchuk@redhat.com> Sent: Friday, February 6, 2015 7:16:47 PM Subject: Re: Storage issues
Hi Donald,
What version are you using (oVirt 3.5.0 presumably)?
Maor - isn't this a bug we solved in 3.5.1?
-Allon ----- Original Message -----
From: "DONALD DAVIS" <donny@5dindustries.com> To: amureini@redhat.com Sent: Friday, February 6, 2015 6:53:27 PM Subject: Storage issues
Hi,
I need some help getting my users vm’s imported back into the system after the failure yesterday. I reattached the storage and half of the vm’s imported without issue. The other half of the vms give this error
Error while executing action: Cannot import VM. Storage Domain doesn't exist
Funny part is, I only had one storage domain… and I imported it back into the engine… I’m confused.
Donny Davis
cloudspin.me
Well, I have everything fixed, and by fixed I mean all the users vm's won't import. My gluster experiment is over, and I have moved the storage to an actual storage device. Hopefully everyone isn't too mad Donny D -----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Maor Lipchuk Sent: Monday, February 9, 2015 3:15 AM To: DONALD DAVIS Cc: users@ovirt.org Subject: Re: [ovirt-users] Storage issues Hi Donald, Can you please elaborate more on the process, I'm not sure I understood the scenario clearly. Regards, Maor ----- Original Message -----
From: "DONALD DAVIS" <donny@5dindustries.com> To: "Maor Lipchuk" <mlipchuk@redhat.com> Cc: "Allon Mureinik" <amureini@redhat.com>, users@ovirt.org Sent: Saturday, February 7, 2015 5:48:06 PM Subject: RE: Storage issues
The issues is with machine that have already been imported once. I started cloudspin with a two drive consumer NAS, and it was slow and crappy so I migrated all of the machines that were on that drive to the gluster, which went off with few issues.
A couple days ago the gluster that held the engine went into split brain hell, and crashed corrupting the engine.
Is engine setup was on gluster, or the Storage domains which was used for the VM's disks?
I rebuilt the engine, and the machine that were created on the gluster had no problems importing, it’s the machines that had already been imported once that have the issue.
How they were imported?
There is nothing in the vdsm logs related to this, it never makes it past the engine. Here is the error.
2015-02-07 08:46:21,445 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GlusterVolumesListVDSCommand(HostName = node3.cloudspin.me, HostId = 1d81be66-49b7-41a6-81ca-1c8dd55dcf3c), log id: 46f08f27 2015-02-07 08:46:21,487 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GlusterVolumesListVDSCommand, return: {a823009f-aa52-4b4b-b2bf-093ece3e8f02=org.ovirt.engine.core.common.bus inessentities.gluster.GlusterVolumeEntity@d475d4ca, e8ca5fa4-37ca-4cd9-a87b-e7c20243e021=org.ovirt.engine.core.common.busi nessentities.gluster.GlusterVolumeEntity@b3d17fb9, af67b75f-c8e1-44ec-aa42-69aed9bd9226=org.ovirt.engine.core.common.busi nessentities.gluster.GlusterVolumeEntity@55088de0}, log id: 46f08f27 2015-02-07 08:46:26,499 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-7) [e4169ff] Lock Acquired to object EngineLock [exclusiveLocks= key: aks-test1 value: VM_NAME , sharedLocks= key: e4c0cf2d-61ed-4cbf-9c43-81e9a565755f value: REMOTE_VM ] 2015-02-07 08:46:26,516 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GlusterVolumesListVDSCommand(HostName = node3.cloudspin.me, HostId = 1d81be66-49b7-41a6-81ca-1c8dd55dcf3c), log id: 22aa4324
HERE IS THE ERROR<<<<<<<<< 2015-02-07 08:46:26,562 WARN [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-7) [e4169ff] CanDoAction of action ImportVmFromConfiguration failed for user donny@cloudspin. Reasons: VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NO T_EXIST 2015-02-07 08:46:26,570 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-7) [e4169ff] Lock freed to object EngineLock [exclusiveLocks= key: aks-test1 value: VM_NAME , sharedLocks= key: e4c0cf2d-61ed-4cbf-9c43-81e9a565755f value: REMOTE_VM ] 2015-02-07 08:46:26,596 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GlusterVolumesListVDSCommand, return: {a823009f-aa52-4b4b-b2bf-093ece3e8f02=org.ovirt.engine.core.common.bus inessentities.gluster.GlusterVolumeEntity@3cecea52, e8ca5fa4-37ca-4cd9-a87b-e7c20243e021=org.ovirt.engine.core.common.busi nessentities.gluster.GlusterVolumeEntity@bdea2c2c, af67b75f-c8e1-44ec-aa42-69aed9bd9226=org.ovirt.engine.core.common.busi nessentities.gluster.GlusterVolumeEntity@4f63b2b3}, log id: 22aa4324
Thank you for your assistance
Donny D
-----Original Message----- From: Maor Lipchuk [mailto:mlipchuk@redhat.com] Sent: Saturday, February 7, 2015 4:08 AM To: DONALD DAVIS Cc: Allon Mureinik Subject: Re: Storage issues
Hi Donald, can you please share your engine and vdsm logs. Have you done any move operation with the disks? That was the bug that was solved in 3.5.1
Thakns, Maor
----- Original Message -----
From: "Allon Mureinik" <amureini@redhat.com> To: "DONALD DAVIS" <donny@5dindustries.com> Cc: "Maor Lipchuk" <mlipchuk@redhat.com> Sent: Friday, February 6, 2015 7:16:47 PM Subject: Re: Storage issues
Hi Donald,
What version are you using (oVirt 3.5.0 presumably)?
Maor - isn't this a bug we solved in 3.5.1?
-Allon ----- Original Message -----
From: "DONALD DAVIS" <donny@5dindustries.com> To: amureini@redhat.com Sent: Friday, February 6, 2015 6:53:27 PM Subject: Storage issues
Hi,
I need some help getting my users vm’s imported back into the system after the failure yesterday. I reattached the storage and half of the vm’s imported without issue. The other half of the vms give this error
Error while executing action: Cannot import VM. Storage Domain doesn't exist
Funny part is, I only had one storage domain… and I imported it back into the engine… I’m confused.
Donny Davis
cloudspin.me
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
DONALD DAVIS -
Donny Davis -
Maor Lipchuk