[Users] VM wont restart after some NFS snapshot restore.
I have some VM's that live on NFS share. Basically, I had to revert the VM disk to a backup from a few days ago. So I powered the VM down, copied over the following files 039a8482-c267-4051-b1e6-1c1dee49b3d7 039a8482-c267-4051-b1e6-1c1dee49b3d7.lease 039a8482-c267-4051-b1e6-1c1dee49b3d7.meta 8d48505d-846d-49a7-8b50-d972ee051145 8d48505d-846d-49a7-8b50-d972ee051145.lease 8d48505d-846d-49a7-8b50-d972ee051145.meta and now when I try to power the VM, it complains 2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 (User: admin@internal). 2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 on Host tss-tusk-ovirt-01-ovirtmgmt.tusk.tufts.edu. 2013-Oct-02, 22:02:38 VM zabbix-prod-01 is down. Exit message: 'truesize'. Any ideas on how I could resolve this? Perhaps a better way of approaching the restore on a filesystem level? I see the following the vsdm.log Thread-7843::ERROR::2013-10-02 22:02:37,548::vm::716::vm.Vm::(_startUnderlyingVm) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 678, in _startUnderlyingVm self._run() File "/usr/share/vdsm/libvirtvm.py", line 1467, in _run devices = self.buildConfDevices() File "/usr/share/vdsm/vm.py", line 515, in buildConfDevices self._normalizeVdsmImg(drv) File "/usr/share/vdsm/vm.py", line 408, in _normalizeVdsmImg drv['truesize'] = res['truesize'] KeyError: 'truesize' Thread-7843::DEBUG::2013-10-02 22:02:37,553::vm::1065::vm.Vm::(setDownStatus) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::Changed state to Down: 'truesize' Any help would be really nice, thanks! -- Usman
Hi Usman, can you paste somewhere the content of the meta files? $ cat 039a8482-c267-4051-b1e6-1c1dee49b3d7.meta 8d48505d-846d-49a7-8b50-d972ee051145.meta could you also provide the absolute path to those files? (in the vdsm host) Thanks, -- Federico ----- Original Message -----
From: "Usman Aslam" <usman@linkshift.com> To: users@ovirt.org Sent: Thursday, October 3, 2013 4:29:43 AM Subject: [Users] VM wont restart after some NFS snapshot restore.
I have some VM's that live on NFS share. Basically, I had to revert the VM disk to a backup from a few days ago. So I powered the VM down, copied over the following files
039a8482-c267-4051-b1e6-1c1dee49b3d7 039a8482-c267-4051-b1e6-1c1dee49b3d7.lease 039a8482-c267-4051-b1e6-1c1dee49b3d7.meta 8d48505d-846d-49a7-8b50-d972ee051145 8d48505d-846d-49a7-8b50-d972ee051145.lease 8d48505d-846d-49a7-8b50-d972ee051145.meta
and now when I try to power the VM, it complains
2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 (User: admin@internal). 2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 on Host tss-tusk-ovirt-01-ovirtmgmt.tusk.tufts.edu . 2013-Oct-02, 22:02:38 VM zabbix-prod-01 is down. Exit message: 'truesize'.
Any ideas on how I could resolve this? Perhaps a better way of approaching the restore on a filesystem level?
I see the following the vsdm.log
Thread-7843::ERROR::2013-10-02 22:02:37,548::vm::716::vm.Vm::(_startUnderlyingVm) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 678, in _startUnderlyingVm self._run() File "/usr/share/vdsm/libvirtvm.py", line 1467, in _run devices = self.buildConfDevices() File "/usr/share/vdsm/vm.py", line 515, in buildConfDevices self._normalizeVdsmImg(drv) File "/usr/share/vdsm/vm.py", line 408, in _normalizeVdsmImg drv['truesize'] = res['truesize'] KeyError: 'truesize' Thread-7843::DEBUG::2013-10-02 22:02:37,553::vm::1065::vm.Vm::(setDownStatus) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::Changed state to Down: 'truesize'
Any help would be really nice, thanks! -- Usman
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Federico, The files reside on this mount on the hypervisor /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__vm_tusk__vm/fa3279ec-2912-45ac-b7bc-9fe89151ed99/images/79ccd989-3033-4e6a-80da-ba210c94225a and are symlinked as described below [root@xyz-02 430cd986-6488-403b-8d46-29abbc3eba38]# pwd /rhev/data-center/430cd986-6488-403b-8d46-29abbc3eba38 [root@xyz-02 430cd986-6488-403b-8d46-29abbc3eba38]# ll total 12 lrwxrwxrwx 1 vdsm kvm 120 Oct 3 12:35 ee2ae498-6e45-448d-8f91-0efca377dcf6 -> /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__iso_tusk__iso/ee2ae498-6e45-448d-8f91-0efca377dcf6 lrwxrwxrwx 1 vdsm kvm 118 Oct 3 12:35 fa3279ec-2912-45ac-b7bc-9fe89151ed99 -> /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__vm_tusk__vm/fa3279ec-2912-45ac-b7bc-9fe89151ed99 lrwxrwxrwx 1 vdsm kvm 118 Oct 3 12:35 mastersd -> /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__vm_tusk__vm/fa3279ec-2912-45ac-b7bc-9fe89151ed99 I did a diff and the contents of the the *Original *meta file (that works and VM starts but have bad file system) and the *Backup *meta file (the files being restored from nfs snapshot) *are exactly the same*. Contents are listed blow. Also the files sizes for all six related files are exactly the same. [root@xyz-02 images]# cat 79ccd989-3033-4e6a-80da-ba210c94225a/039a8482-c267-4051-b1e6-1c1dee49b3d7.meta DOMAIN=fa3279ec-2912-45ac-b7bc-9fe89151ed99 VOLTYPE=SHARED CTIME=1368457020 FORMAT=RAW IMAGE=59b6a429-bd11-40c6-a218-78df840725c6 DISKTYPE=2 PUUID=00000000-0000-0000-0000-000000000000 LEGALITY=LEGAL MTIME=1368457020 POOL_UUID= DESCRIPTION=Active VM TYPE=SPARSE SIZE=104857600 EOF [root@tss-tusk-ovirt-02 images]# cat 79ccd989-3033-4e6a-80da-ba210c94225a/8d48505d-846d-49a7-8b50-d972ee051145.meta DOMAIN=fa3279ec-2912-45ac-b7bc-9fe89151ed99 CTIME=1370303194 FORMAT=COW DISKTYPE=2 LEGALITY=LEGAL SIZE=104857600 VOLTYPE=LEAF DESCRIPTION= IMAGE=79ccd989-3033-4e6a-80da-ba210c94225a PUUID=039a8482-c267-4051-b1e6-1c1dee49b3d7 MTIME=1370303194 POOL_UUID= TYPE=SPARSE EOF Any help would be greatly appreciated! Thanks, Usman On Fri, Oct 4, 2013 at 9:50 AM, Federico Simoncelli <fsimonce@redhat.com>wrote:
Hi Usman, can you paste somewhere the content of the meta files?
$ cat 039a8482-c267-4051-b1e6-1c1dee49b3d7.meta 8d48505d-846d-49a7-8b50-d972ee051145.meta
could you also provide the absolute path to those files? (in the vdsm host)
Thanks, -- Federico
----- Original Message -----
From: "Usman Aslam" <usman@linkshift.com> To: users@ovirt.org Sent: Thursday, October 3, 2013 4:29:43 AM Subject: [Users] VM wont restart after some NFS snapshot restore.
I have some VM's that live on NFS share. Basically, I had to revert the VM disk to a backup from a few days ago. So I powered the VM down, copied over the following files
039a8482-c267-4051-b1e6-1c1dee49b3d7 039a8482-c267-4051-b1e6-1c1dee49b3d7.lease 039a8482-c267-4051-b1e6-1c1dee49b3d7.meta 8d48505d-846d-49a7-8b50-d972ee051145 8d48505d-846d-49a7-8b50-d972ee051145.lease 8d48505d-846d-49a7-8b50-d972ee051145.meta
and now when I try to power the VM, it complains
2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 (User: admin@internal). 2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 on Host tss-tusk-ovirt-01-ovirtmgmt.tusk.tufts.edu . 2013-Oct-02, 22:02:38 VM zabbix-prod-01 is down. Exit message: 'truesize'.
Any ideas on how I could resolve this? Perhaps a better way of approaching the restore on a filesystem level?
I see the following the vsdm.log
Thread-7843::ERROR::2013-10-02 22:02:37,548::vm::716::vm.Vm::(_startUnderlyingVm) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 678, in _startUnderlyingVm self._run() File "/usr/share/vdsm/libvirtvm.py", line 1467, in _run devices = self.buildConfDevices() File "/usr/share/vdsm/vm.py", line 515, in buildConfDevices self._normalizeVdsmImg(drv) File "/usr/share/vdsm/vm.py", line 408, in _normalizeVdsmImg drv['truesize'] = res['truesize'] KeyError: 'truesize' Thread-7843::DEBUG::2013-10-02 22:02:37,553::vm::1065::vm.Vm::(setDownStatus) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::Changed state to Down: 'truesize'
Any help would be really nice, thanks! -- Usman
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Usman Aslam 401.6.99.66.55 usman@linkshift.com
Usman, the only thing that comes to my mind is something related to: http://gerrit.ovirt.org/13529 which means that in some way the restored volumes are either inaccessible (permissions?) or their metadata is corrupted (but it doesn't seem so). There is probably another Traceback in the logs that should give us more information. Could you post somewhere the entire vdsm log? Thanks. -- Federico ----- Original Message -----
From: "Usman Aslam" <usman@linkshift.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: users@ovirt.org Sent: Friday, October 4, 2013 4:49:08 PM Subject: Re: [Users] VM wont restart after some NFS snapshot restore.
Federico,
The files reside on this mount on the hypervisor /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__vm_tusk__vm/fa3279ec-2912-45ac-b7bc-9fe89151ed99/images/79ccd989-3033-4e6a-80da-ba210c94225a
and are symlinked as described below
[root@xyz-02 430cd986-6488-403b-8d46-29abbc3eba38]# pwd /rhev/data-center/430cd986-6488-403b-8d46-29abbc3eba38 [root@xyz-02 430cd986-6488-403b-8d46-29abbc3eba38]# ll total 12 lrwxrwxrwx 1 vdsm kvm 120 Oct 3 12:35 ee2ae498-6e45-448d-8f91-0efca377dcf6 -> /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__iso_tusk__iso/ee2ae498-6e45-448d-8f91-0efca377dcf6 lrwxrwxrwx 1 vdsm kvm 118 Oct 3 12:35 fa3279ec-2912-45ac-b7bc-9fe89151ed99 -> /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__vm_tusk__vm/fa3279ec-2912-45ac-b7bc-9fe89151ed99 lrwxrwxrwx 1 vdsm kvm 118 Oct 3 12:35 mastersd -> /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__vm_tusk__vm/fa3279ec-2912-45ac-b7bc-9fe89151ed99
I did a diff and the contents of the the *Original *meta file (that works and VM starts but have bad file system) and the *Backup *meta file (the files being restored from nfs snapshot) *are exactly the same*. Contents are listed blow. Also the files sizes for all six related files are exactly the same.
[root@xyz-02 images]# cat 79ccd989-3033-4e6a-80da-ba210c94225a/039a8482-c267-4051-b1e6-1c1dee49b3d7.meta DOMAIN=fa3279ec-2912-45ac-b7bc-9fe89151ed99 VOLTYPE=SHARED CTIME=1368457020 FORMAT=RAW IMAGE=59b6a429-bd11-40c6-a218-78df840725c6 DISKTYPE=2 PUUID=00000000-0000-0000-0000-000000000000 LEGALITY=LEGAL MTIME=1368457020 POOL_UUID= DESCRIPTION=Active VM TYPE=SPARSE SIZE=104857600 EOF [root@tss-tusk-ovirt-02 images]# cat 79ccd989-3033-4e6a-80da-ba210c94225a/8d48505d-846d-49a7-8b50-d972ee051145.meta DOMAIN=fa3279ec-2912-45ac-b7bc-9fe89151ed99 CTIME=1370303194 FORMAT=COW DISKTYPE=2 LEGALITY=LEGAL SIZE=104857600 VOLTYPE=LEAF DESCRIPTION= IMAGE=79ccd989-3033-4e6a-80da-ba210c94225a PUUID=039a8482-c267-4051-b1e6-1c1dee49b3d7 MTIME=1370303194 POOL_UUID= TYPE=SPARSE EOF
Any help would be greatly appreciated!
Thanks, Usman
On Fri, Oct 4, 2013 at 9:50 AM, Federico Simoncelli <fsimonce@redhat.com>wrote:
Hi Usman, can you paste somewhere the content of the meta files?
$ cat 039a8482-c267-4051-b1e6-1c1dee49b3d7.meta 8d48505d-846d-49a7-8b50-d972ee051145.meta
could you also provide the absolute path to those files? (in the vdsm host)
Thanks, -- Federico
----- Original Message -----
From: "Usman Aslam" <usman@linkshift.com> To: users@ovirt.org Sent: Thursday, October 3, 2013 4:29:43 AM Subject: [Users] VM wont restart after some NFS snapshot restore.
I have some VM's that live on NFS share. Basically, I had to revert the VM disk to a backup from a few days ago. So I powered the VM down, copied over the following files
039a8482-c267-4051-b1e6-1c1dee49b3d7 039a8482-c267-4051-b1e6-1c1dee49b3d7.lease 039a8482-c267-4051-b1e6-1c1dee49b3d7.meta 8d48505d-846d-49a7-8b50-d972ee051145 8d48505d-846d-49a7-8b50-d972ee051145.lease 8d48505d-846d-49a7-8b50-d972ee051145.meta
and now when I try to power the VM, it complains
2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 (User: admin@internal). 2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 on Host tss-tusk-ovirt-01-ovirtmgmt.tusk.tufts.edu . 2013-Oct-02, 22:02:38 VM zabbix-prod-01 is down. Exit message: 'truesize'.
Any ideas on how I could resolve this? Perhaps a better way of approaching the restore on a filesystem level?
I see the following the vsdm.log
Thread-7843::ERROR::2013-10-02 22:02:37,548::vm::716::vm.Vm::(_startUnderlyingVm) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 678, in _startUnderlyingVm self._run() File "/usr/share/vdsm/libvirtvm.py", line 1467, in _run devices = self.buildConfDevices() File "/usr/share/vdsm/vm.py", line 515, in buildConfDevices self._normalizeVdsmImg(drv) File "/usr/share/vdsm/vm.py", line 408, in _normalizeVdsmImg drv['truesize'] = res['truesize'] KeyError: 'truesize' Thread-7843::DEBUG::2013-10-02 22:02:37,553::vm::1065::vm.Vm::(setDownStatus) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::Changed state to Down: 'truesize'
Any help would be really nice, thanks! -- Usman
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Usman Aslam 401.6.99.66.55 usman@linkshift.com
Federico, You're a life savor.. Its was a permissions issues, somehow, the owner and group was assigned as root. I matched permission with the original data and it works! Cant believe I missed something so simple. Thanks you! Usman On Fri, Oct 4, 2013 at 11:07 AM, Federico Simoncelli <fsimonce@redhat.com>wrote:
Usman, the only thing that comes to my mind is something related to:
which means that in some way the restored volumes are either inaccessible (permissions?) or their metadata is corrupted (but it doesn't seem so).
There is probably another Traceback in the logs that should give us more information.
Could you post somewhere the entire vdsm log?
Thanks. -- Federico
----- Original Message -----
From: "Usman Aslam" <usman@linkshift.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: users@ovirt.org Sent: Friday, October 4, 2013 4:49:08 PM Subject: Re: [Users] VM wont restart after some NFS snapshot restore.
Federico,
The files reside on this mount on the hypervisor /rhev/data-center/mnt/xyz-02.tufts.edu:
_vol_tusk__vm_tusk__vm/fa3279ec-2912-45ac-b7bc-9fe89151ed99/images/79ccd989-3033-4e6a-80da-ba210c94225a
and are symlinked as described below
[root@xyz-02 430cd986-6488-403b-8d46-29abbc3eba38]# pwd /rhev/data-center/430cd986-6488-403b-8d46-29abbc3eba38 [root@xyz-02 430cd986-6488-403b-8d46-29abbc3eba38]# ll total 12 lrwxrwxrwx 1 vdsm kvm 120 Oct 3 12:35
ee2ae498-6e45-448d-8f91-0efca377dcf6
-> /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__iso_tusk__iso/ee2ae498-6e45-448d-8f91-0efca377dcf6 lrwxrwxrwx 1 vdsm kvm 118 Oct 3 12:35 fa3279ec-2912-45ac-b7bc-9fe89151ed99 -> /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__vm_tusk__vm/fa3279ec-2912-45ac-b7bc-9fe89151ed99 lrwxrwxrwx 1 vdsm kvm 118 Oct 3 12:35 mastersd -> /rhev/data-center/mnt/xyz-02.tufts.edu: _vol_tusk__vm_tusk__vm/fa3279ec-2912-45ac-b7bc-9fe89151ed99
I did a diff and the contents of the the *Original *meta file (that works and VM starts but have bad file system) and the *Backup *meta file (the files being restored from nfs snapshot) *are exactly the same*. Contents are listed blow. Also the files sizes for all six related files are exactly the same.
[root@xyz-02 images]# cat
79ccd989-3033-4e6a-80da-ba210c94225a/039a8482-c267-4051-b1e6-1c1dee49b3d7.meta
DOMAIN=fa3279ec-2912-45ac-b7bc-9fe89151ed99 VOLTYPE=SHARED CTIME=1368457020 FORMAT=RAW IMAGE=59b6a429-bd11-40c6-a218-78df840725c6 DISKTYPE=2 PUUID=00000000-0000-0000-0000-000000000000 LEGALITY=LEGAL MTIME=1368457020 POOL_UUID= DESCRIPTION=Active VM TYPE=SPARSE SIZE=104857600 EOF [root@tss-tusk-ovirt-02 images]# cat
DOMAIN=fa3279ec-2912-45ac-b7bc-9fe89151ed99 CTIME=1370303194 FORMAT=COW DISKTYPE=2 LEGALITY=LEGAL SIZE=104857600 VOLTYPE=LEAF DESCRIPTION= IMAGE=79ccd989-3033-4e6a-80da-ba210c94225a PUUID=039a8482-c267-4051-b1e6-1c1dee49b3d7 MTIME=1370303194 POOL_UUID= TYPE=SPARSE EOF
Any help would be greatly appreciated!
Thanks, Usman
On Fri, Oct 4, 2013 at 9:50 AM, Federico Simoncelli <fsimonce@redhat.com>wrote:
Hi Usman, can you paste somewhere the content of the meta files?
$ cat 039a8482-c267-4051-b1e6-1c1dee49b3d7.meta 8d48505d-846d-49a7-8b50-d972ee051145.meta
could you also provide the absolute path to those files? (in the vdsm host)
Thanks, -- Federico
----- Original Message -----
From: "Usman Aslam" <usman@linkshift.com> To: users@ovirt.org Sent: Thursday, October 3, 2013 4:29:43 AM Subject: [Users] VM wont restart after some NFS snapshot restore.
I have some VM's that live on NFS share. Basically, I had to revert
79ccd989-3033-4e6a-80da-ba210c94225a/8d48505d-846d-49a7-8b50-d972ee051145.meta the
VM
disk to a backup from a few days ago. So I powered the VM down, copied over the following files
039a8482-c267-4051-b1e6-1c1dee49b3d7 039a8482-c267-4051-b1e6-1c1dee49b3d7.lease 039a8482-c267-4051-b1e6-1c1dee49b3d7.meta 8d48505d-846d-49a7-8b50-d972ee051145 8d48505d-846d-49a7-8b50-d972ee051145.lease 8d48505d-846d-49a7-8b50-d972ee051145.meta
and now when I try to power the VM, it complains
2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 (User: admin@internal). 2013-Oct-02, 22:02:38 Failed to run VM zabbix-prod-01 on Host tss-tusk-ovirt-01-ovirtmgmt.tusk.tufts.edu . 2013-Oct-02, 22:02:38 VM zabbix-prod-01 is down. Exit message: 'truesize'.
Any ideas on how I could resolve this? Perhaps a better way of approaching the restore on a filesystem level?
I see the following the vsdm.log
Thread-7843::ERROR::2013-10-02 22:02:37,548::vm::716::vm.Vm::(_startUnderlyingVm) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 678, in _startUnderlyingVm self._run() File "/usr/share/vdsm/libvirtvm.py", line 1467, in _run devices = self.buildConfDevices() File "/usr/share/vdsm/vm.py", line 515, in buildConfDevices self._normalizeVdsmImg(drv) File "/usr/share/vdsm/vm.py", line 408, in _normalizeVdsmImg drv['truesize'] = res['truesize'] KeyError: 'truesize' Thread-7843::DEBUG::2013-10-02 22:02:37,553::vm::1065::vm.Vm::(setDownStatus) vmId=`8e8764ad-6b4c-48d8-9a19-fa5cf77208ef`::Changed state to Down: 'truesize'
Any help would be really nice, thanks! -- Usman
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Usman Aslam 401.6.99.66.55 usman@linkshift.com
-- Usman Aslam 401.6.99.66.55 usman@linkshift.com
participants (2)
-
Federico Simoncelli -
Usman Aslam