Re: [ovirt-users] disk not bootable

Ops... forgot the link: http://pastebin.com/LereJgyw The requested infor is in the pastebin.

Regards, -- Fernando Fuentes ffuentes(a)txweather.org http://www.txweather.org On Mon, Jul 18, 2016, at 03:16 PM, Fernando Fuentes wrote: Nir, After some playing around with pvscan I was able to get all of the need it information. Please see: -- Fernando Fuentes ffuentes(a)txweather.org http://www.txweather.org On Mon, Jul 18, 2016, at 02:30 PM, Nir Soffer wrote: On Mon, Jul 18, 2016 at 6:48 PM, Fernando Fuentes <ffuentes(a)darktcp.net&gt; wrote: > Nir, > > As requested: > > [root@gamma ~]# lsblk > NAME MAJ:MIN RM > SIZE RO TYPE MOUNTPOINT > sda 8:0 0 > 557G 0 disk > ├─sda1 8:1 0 > 500M 0 part /boot > └─sda2 8:2 0 > 556.5G 0 part > ├─vg_gamma-lv_root (dm-0) 253:0 0 > 50G 0 lvm / > ├─vg_gamma-lv_swap (dm-1) 253:1 0 > 4G 0 lvm [SWAP] > └─vg_gamma-lv_home (dm-2) 253:2 0 > 502.4G 0 lvm /home > sr0 11:0 1 > 1024M 0 rom > sdb 8:16 0 > 2T 0 disk > └─36589cfc000000881b9b93c2623780840 (dm-4) 253:4 0 > 2T 0 mpath > sdc 8:32 0 > 2T 0 disk > └─36589cfc00000050564002c7e51978316 (dm-3) 253:3 0 > 2T 0 mpath > ├─3ccb7b67--8067--4315--9656--d68ba10975ba-metadata (dm-5) 253:5 0 > 512M 0 lvm > ├─3ccb7b67--8067--4315--9656--d68ba10975ba-outbox (dm-6) 253:6 0 > 128M 0 lvm > ├─3ccb7b67--8067--4315--9656--d68ba10975ba-leases (dm-7) 253:7 0 > 2G 0 lvm > ├─3ccb7b67--8067--4315--9656--d68ba10975ba-ids (dm-8) 253:8 0 > 128M 0 lvm > ├─3ccb7b67--8067--4315--9656--d68ba10975ba-inbox (dm-9) 253:9 0 > 128M 0 lvm > └─3ccb7b67--8067--4315--9656--d68ba10975ba-master (dm-10) 253:10 0 > 1G 0 lvm > sdd 8:48 0 > 4T 0 disk > └─36589cfc00000059ccab70662b71c47ef (dm-11) 253:11 0 > 4T 0 mpath > ├─4861322b--352f--41c6--890a--5cbf1c2c1f01-metadata (dm-12) 253:12 0 > 512M 0 lvm > ├─4861322b--352f--41c6--890a--5cbf1c2c1f01-ids (dm-13) 253:13 0 > 128M 0 lvm > ├─4861322b--352f--41c6--890a--5cbf1c2c1f01-leases (dm-14) 253:14 0 > 2G 0 lvm > ├─4861322b--352f--41c6--890a--5cbf1c2c1f01-outbox (dm-15) 253:15 0 > 128M 0 lvm > ├─4861322b--352f--41c6--890a--5cbf1c2c1f01-inbox (dm-16) 253:16 0 > 128M 0 lvm > └─4861322b--352f--41c6--890a--5cbf1c2c1f01-master (dm-17) 253:17 0 > 1G 0 lvm So you have 2 storage domains: - 3ccb7b67-8067-4315-9656-d68ba10975ba - 4861322b-352f-41c6-890a-5cbf1c2c1f01 But most likely both of them are not active now. Can you share the output of: iscsiadm -m session On a system connected to iscsi storage you will see something like: # iscsiadm -m session tcp: [5] 10.35.0.99:3260,1 iqn.2003-01.org.dumbo.target1 (non-flash) The special lvs (ids, leases, ...) should be active, and you should see also regular disks lvs (used for snapshot for vm disks). Here is an example from machine connected to active iscsi domain: # lvs LV VG Attr LSize 27c4c795-bca4-4d7b-9b40-cda9098790f5 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi------- 1.00g 35be1f52-5b28-4c90-957a-710dbbb8f13f 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi------- 8.00g 36d9b41b-4b01-4fc2-8e93-ccf79af0f766 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi------- 128.00m 4fda3b44-27a5-4ce4-b8c3-66744aa9937b 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi------- 2.12g c2e78f72-d499-44f0-91f5-9930a599dc87 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi------- 128.00m d49919b4-30fc-440f-9b21-3367ddfdf396 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi------- 4.00g f3b10280-43ed-4772-b122-18c92e098171 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi------- 1.00g f409cc48-8248-4239-a4ea-66b0b1084416 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi------- 1.00g ids 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi-ao---- 128.00m inbox 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi-a----- 128.00m leases 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi-a----- 2.00g master 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi-a----- 1.00g metadata 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi-a----- 512.00m outbox 5f35b5c0-17d7-4475-9125-e97f1cdb06f9 -wi-a----- 128.00m > [root@gamma ~]# > > Regards, > > -- > Fernando Fuentes > ffuentes(a)txweather.org > http://www.txweather.org > > On Mon, Jul 18, 2016, at 07:43 AM, Nir Soffer wrote: >> Can you share output of lsblk on this host? >> >> On Mon, Jul 18, 2016 at 3:52 AM, Fernando Fuentes <ffuentes(a)darktcp.net&gt; >> wrote: >> > Nir, >> > >> > That's odd. gamma is my iscsi host, its in up state and it has active >> > VM's. >> > What am I missing? >> > >> > Regards, >> > >> > -- >> > Fernando Fuentes >> > ffuentes(a)txweather.org >> > http://www.txweather.org >> > >> > On Sun, Jul 17, 2016, at 07:24 PM, Nir Soffer wrote: >> >> On Sun, Jul 17, 2016 at 1:24 AM, Fernando Fuentes >> >> <ffuentes(a)darktcp.net&gt; >> >> wrote: >> >> > Nir, >> >> > >> >> > Ok I got the uuid but I am getting the same results as before. >> >> > Nothing comes up. >> >> > >> >> > [root@gamma ~]# pvscan --cache >> >> > [root@gamma ~]# lvs -o vg_name,lv_name,tags | grep >> >> > 3b7d9349-9eb1-42f8-9e04-7bbb97c02b98 >> >> > [root@gamma ~]# >> >> > >> >> > without the grep all I get is: >> >> > >> >> > [root@gamma ~]# lvs -o vg_name,lv_name,tags >> >> > VG LV LV Tags >> >> > vg_gamma lv_home >> >> > vg_gamma lv_root >> >> > vg_gamma lv_swap >> >> >> >> You are not connected to the iscsi storage domain. >> >> >> >> Please try this from a host in up state in engine. >> >> >> >> Nir >> >> >> >> > >> >> > On the other hand an fdisk shows a bunch of disks and here is one >> >> > example: >> >> > >> >> > Disk /dev/mapper/36589cfc00000050564002c7e51978316: 2199.0 GB, >> >> > 2199023255552 bytes >> >> > 255 heads, 63 sectors/track, 267349 cylinders >> >> > Units = cylinders of 16065 * 512 = 8225280 bytes >> >> > Sector size (logical/physical): 512 bytes / 32768 bytes >> >> > I/O size (minimum/optimal): 32768 bytes / 1048576 bytes >> >> > Disk identifier: 0x00000000 >> >> > >> >> > >> >> > Disk /dev/mapper/36589cfc000000881b9b93c2623780840: 2199.0 GB, >> >> > 2199023255552 bytes >> >> > 255 heads, 63 sectors/track, 267349 cylinders >> >> > Units = cylinders of 16065 * 512 = 8225280 bytes >> >> > Sector size (logical/physical): 512 bytes / 32768 bytes >> >> > I/O size (minimum/optimal): 32768 bytes / 1048576 bytes >> >> > Disk identifier: 0x00000000 >> >> > >> >> > >> >> > Disk /dev/mapper/3ccb7b67--8067--4315--9656--d68ba10975ba-metadata: >> >> > 536 >> >> > MB, 536870912 bytes >> >> > 255 heads, 63 sectors/track, 65 cylinders >> >> > Units = cylinders of 16065 * 512 = 8225280 bytes >> >> > Sector size (logical/physical): 512 bytes / 32768 bytes >> >> > I/O size (minimum/optimal): 32768 bytes / 1048576 bytes >> >> > Disk identifier: 0x00000000 >> >> > >> >> > Disk /dev/mapper/4861322b--352f--41c6--890a--5cbf1c2c1f01-master: >> >> > 1073 >> >> > MB, 1073741824 bytes >> >> > 255 heads, 63 sectors/track, 130 cylinders >> >> > Units = cylinders of 16065 * 512 = 8225280 bytes >> >> > Sector size (logical/physical): 512 bytes / 32768 bytes >> >> > I/O size (minimum/optimal): 32768 bytes / 1048576 bytes >> >> > Disk identifier: 0x00000000 >> >> > >> >> > Regards, >> >> > >> >> > -- >> >> > Fernando Fuentes >> >> > ffuentes(a)txweather.org >> >> > http://www.txweather.org >> >> > >> >> > On Sat, Jul 16, 2016, at 04:25 PM, Fernando Fuentes wrote: >> >> >> Nir, >> >> >> >> >> >> Ok ill look for it here in a few. >> >> >> Thanks for your reply and help! >> >> >> >> >> >> -- >> >> >> Fernando Fuentes >> >> >> ffuentes(a)txweather.org >> >> >> http://www.txweather.org >> >> >> >> >> >> On Sat, Jul 16, 2016, at 04:16 PM, Nir Soffer wrote: >> >> >> > On Fri, Jul 15, 2016 at 3:50 PM, Fernando Fuentes >> >> >> > <ffuentes(a)darktcp.net&gt; >> >> >> > wrote: >> >> >> > > Nir, >> >> >> > > >> >> >> > > I try to follow your steps but I cant seem to find the ID of >> >> >> > > the >> >> >> > > template. >> >> >> > >> >> >> > The image-uuid of the template is displayed in the Disks tab in >> >> >> > engine. >> >> >> > >> >> >> > To find the volume-uuid on block storage, you can do: >> >> >> > >> >> >> > pvscan --cache >> >> >> > lvs -o vg_name,lv_name,tags | grep image-uuid >> >> >> > >> >> >> > > >> >> >> > > Regards, >> >> >> > > >> >> >> > > -- >> >> >> > > Fernando Fuentes >> >> >> > > ffuentes(a)txweather.org >> >> >> > > http://www.txweather.org >> >> >> > > >> >> >> > > On Sun, Jul 10, 2016, at 02:15 PM, Nir Soffer wrote: >> >> >> > >> On Thu, Jul 7, 2016 at 7:46 PM, Melissa Mesler >> >> >> > >> <melissa(a)justmelly.com&gt; >> >> >> > >> wrote: >> >> >> > >> > All, I did a test for Fernando in our ovirt environment. I >> >> >> > >> > created a vm >> >> >> > >> > called win7melly in the nfs domain. I then migrated it to >> >> >> > >> > the iscsi >> >> >> > >> > domain. It booted without any issue. So it has to be >> >> >> > >> > something with the >> >> >> > >> > templates. I have attached the vdsm log for the host the vm >> >> >> > >> > resides on. >> >> >> > >> >> >> >> > >> The log show a working vm, so it does not help much. >> >> >> > >> >> >> >> > >> I think that the template you copied from the nfs domain to >> >> >> > >> the block >> >> >> > >> domain is >> >> >> > >> corrupted, or the volume metadata are incorrect. >> >> >> > >> >> >> >> > >> If I understand this correctly, this started when Fernando >> >> >> > >> could not copy >> >> >> > >> the vm >> >> >> > >> disk to the block storage, and I guess the issue was that the >> >> >> > >> template >> >> >> > >> was missing >> >> >> > >> on that storage domain. I assume that he copied the template >> >> >> > >> to the >> >> >> > >> block storage >> >> >> > >> domain by opening the templates tab, selecting the template, >> >> >> > >> and choosing >> >> >> > >> copy >> >> >> > >> from the menu. >> >> >> > >> >> >> >> > >> Lets compare the template on both nfs and block storage >> >> >> > >> domain. >> >> >> > >> >> >> >> > >> 1. Find the template on the nfs storage domain, using the >> >> >> > >> image uuid in >> >> >> > >> engine. >> >> >> > >> >> >> >> > >> It should be at >> >> >> > >> >> >> >> > >> >> >> >> > >> /rhev/data-center/mnt/server:_path/domain-uuid/images/image-uuid/volume-uuid >> >> >> > >> >> >> >> > >> 2. Please share the output of: >> >> >> > >> >> >> >> > >> cat /path/to/volume.meta >> >> >> > >> qemu-img info /path/to/volume >> >> >> > >> qemu-img check /path/to/volume >> >> >> > >> >> >> >> > >> 4. Find the template on the block storage domain >> >> >> > >> >> >> >> > >> You should have an lv using the same volume uuid and the >> >> >> > >> image-uuid >> >> >> > >> should be in the lv tag. >> >> >> > >> >> >> >> > >> Find it using: >> >> >> > >> >> >> >> > >> lvs -o vg_name,lv_name,tags | grep volume-uuid >> >> >> > >> >> >> >> > >> 5. Activate the lv >> >> >> > >> >> >> >> > >> lvchange -ay vg_name/lv_name >> >> >> > >> >> >> >> > >> 6. Share the output of >> >> >> > >> >> >> >> > >> qemu-img info /dev/vg_name/lv_name >> >> >> > >> qemu-img check /dev/vg_name/lv_name >> >> >> > >> >> >> >> > >> 7. Deactivate the lv >> >> >> > >> >> >> >> > >> lvchange -an vg_name/lv_name >> >> >> > >> >> >> >> > >> 8. Find the lv metadata >> >> >> > >> >> >> >> > >> The metadata is stored in /dev/vg_name/metadata. To find the >> >> >> > >> correct >> >> >> > >> block, >> >> >> > >> find the tag named MD_N in the lv tags you found in step 4 >> >> >> > >> >> >> >> > >> The block we need is located at offset N from start of volume. >> >> >> > >> >> >> >> > >> 9. Share the output of: >> >> >> > >> >> >> >> > >> dd if=/dev/vg_name/metadata bs=512 skip=N count=1 >> >> >> > >> iflag=direct >> >> >> > >> >> >> >> > >> The output of this command should show the image-uuid. >> >> >> > >> >> >> >> > >> Nir >> >> >> > >> >> >> >> > >> > >> >> >> > >> > - MeLLy >> >> >> > >> > >> >> >> > >> > On Mon, Jul 4, 2016, at 11:52 PM, Fernando Fuentes wrote: >> >> >> > >> >> Nir, >> >> >> > >> >> >> >> >> > >> >> That's exactly how I did it Nir. >> >> >> > >> >> I will test tomorrow with a new Windows VM and report back. >> >> >> > >> >> >> >> >> > >> >> Regards, >> >> >> > >> >> >> >> >> > >> >> -- >> >> >> > >> >> Fernando Fuentes >> >> >> > >> >> ffuentes(a)txweather.org >> >> >> > >> >> http://www.txweather.org >> >> >> > >> >> >> >> >> > >> >> On Mon, Jul 4, 2016, at 10:48 AM, Nir Soffer wrote: >> >> >> > >> >> > On Mon, Jul 4, 2016 at 6:43 PM, Francesco Romani >> >> >> > >> >> > <fromani(a)redhat.com&gt; >> >> >> > >> >> > wrote: >> >> >> > >> >> > > ----- Original Message ----- >> >> >> > >> >> > >> From: "Nir Soffer" <nsoffer(a)redhat.com&gt; >> >> >> > >> >> > >> To: "Fernando Fuentes" <ffuentes(a)darktcp.net&gt; >> >> >> > >> >> > >> Cc: "Francesco Romani" <fromani(a)redhat.com&gt;, "users" >> >> >> > >> >> > >> <users(a)ovirt.org&gt; >> >> >> > >> >> > >> Sent: Saturday, July 2, 2016 11:18:01 AM >> >> >> > >> >> > >> Subject: Re: [ovirt-users] disk not bootable >> >> >> > >> >> > >> >> >> >> > >> >> > >> On Sat, Jul 2, 2016 at 1:33 AM, Fernando Fuentes >> >> >> > >> >> > >> <ffuentes(a)darktcp.net&gt; >> >> >> > >> >> > >> wrote: >> >> >> > >> >> > >> > Nir, >> >> >> > >> >> > >> > >> >> >> > >> >> > >> > Ok I ran another test and this one I moved from NFS >> >> >> > >> >> > >> > domain to iSCSI and >> >> >> > >> >> > >> > stop working than I moved it back and still unable >> >> >> > >> >> > >> > to run... Windows VM >> >> >> > >> >> > >> > is saying "no available boot disk" >> >> >> > >> >> > >> > VM: Win7-Test >> >> >> > >> >> > >> > Host: Zeta >> >> >> > >> >> > >> > Info as requested: http://pastebin.com/1fSi3auz >> >> >> > >> >> > >> >> >> >> > >> >> > >> We need a working xml to compare to. >> >> >> > >> >> > > >> >> >> > >> >> > > [snip expected changes] >> >> >> > >> >> > > >> >> >> > >> >> > > >> >> >> > >> >> > >> <entry name="manufacturer">oVirt</entry> >> >> >> > >> >> > >> <entry name="product">oVirt Node</entry> >> >> >> > >> >> > >> <entry name="version">6-5.el6.centos.11.2</entry> >> >> >> > >> >> > >> - <entry >> >> >> > >> >> > >> name="serial">C938F077-55E2-3E50-A694-9FCB7661FD89</entry> >> >> >> > >> >> > >> + <entry >> >> >> > >> >> > >> name="serial">735C7A01-1F16-3CF0-AF8C-A99823E95AC0</entry> >> >> >> > >> >> > >> >> >> >> > >> >> > >> Not expected - maybe this is confusing windows? >> >> >> > >> >> > >> >> >> >> > >> >> > >> Francesco, why vm serial has changed after moving >> >> >> > >> >> > >> disks from one storage >> >> >> > >> >> > >> domain >> >> >> > >> >> > >> to another? >> >> >> > >> >> > > >> >> >> > >> >> > > We put in serial either >> >> >> > >> >> > > 1. the UUID Engine send to us >> >> >> > >> >> > > 2. the host UUID as returned by our getHostUUID utility >> >> >> > >> >> > > function >> >> >> > >> >> > > >> >> >> > >> >> > > the latter is unlikely to change, even after this disk >> >> >> > >> >> > > move. >> >> >> > >> >> > >> >> >> > >> >> > Fernando, can you describe exactly how you moved the >> >> >> > >> >> > disk? >> >> >> > >> >> > >> >> >> > >> >> > I assume that you selected the vm in the virtual machines >> >> >> > >> >> > tab, then >> >> >> > >> >> > selected >> >> >> > >> >> > disks from the sub tab, then selected move, and selected >> >> >> > >> >> > the target >> >> >> > >> >> > storage domain. >> >> >> > >> >> > >> >> >> > >> >> > Also, can you reproduce this with a new vm? (create vm >> >> >> > >> >> > with disk nfs, >> >> >> > >> >> > stop vm, >> >> >> > >> >> > move disk to iscsi, start vm). >> >> >> > >> >> > >> >> >> > >> >> > > So the first suspect in line is Engine >> >> >> > >> >> > > >> >> >> > >> >> > > Arik, do you know if Engine is indeed supposed to >> >> >> > >> >> > > change the UUID in this flow? >> >> >> > >> >> > > That seems very surprising. >> >> >> > >> >> > > >> >> >> > >> >> > > Thanks and bests, >> >> >> > >> >> > > >> >> >> > >> >> > > -- >> >> >> > >> >> > > Francesco Romani >> >> >> > >> >> > > RedHat Engineering Virtualization R & D >> >> >> > >> >> > > Phone: 8261328 >> >> >> > >> >> > > IRC: fromani >> >> >> > >> >> _______________________________________________ >> >> >> > >> >> Users mailing list >> >> >> > >> >> Users(a)ovirt.org >> >> >> > >> >> http://lists.ovirt.org/mailman/listinfo/users >> >> >> > >> > >> >> >> > >> > _______________________________________________ >> >> >> > >> > Users mailing list >> >> >> > >> > Users(a)ovirt.org >> >> >> > >> > http://lists.ovirt.org/mailman/listinfo/users >> >> >> > >> > >> >> >> _______________________________________________ >> >> >> Users mailing list >> >> >> Users(a)ovirt.org >> >> >> http://lists.ovirt.org/mailman/listinfo/users >> >> > _______________________________________________ >> >> > Users mailing list >> >> > Users(a)ovirt.org >> >> > http://lists.ovirt.org/mailman/listinfo/users