<div dir="ltr">I&#39;m sure I have worked through this before, but I&#39;ve been banging my head against this one for a while now, and I think I&#39;m too close to the issue.<div><br></div><div>Basically, my hosted-engine won&#39;t start anymore.  I do recall attempting to migrate it to a new gluster-based NFS share recently, but I could have sworn that was successful and working.  I _think_ I have some sort of id issue with storage/volume/whatever id&#39;s, but I need some help digging through it if someone would be so kind.</div><div><br></div><div>I have the following error in /var/log/libvirt/qemu/HostedEngine.log:</div><div><br></div><div>----------<br></div><div>2015-07-02T00:01:13.080952Z qemu-kvm: -drive file=/var/run/vdsm/storage/4e3017eb-d062-4ad1-9df8-7057fcee412c/5ead7b5d-50e8-4d6c-a0e5-bbe6d93dd836/eeb2d821-a432-4df6-8856-fdb14df260e3,if=none,id=drive-virtio-disk0,format=raw,serial=5ead7b5d-50e8-4d6c-a0e5-bbe6d93dd836,cache=none,werror=stop,rerror=stop,aio=threads: could not open disk image /var/run/vdsm/storage/4e3017eb-d062-4ad1-9df8-7057fcee412c/5ead7b5d-50e8-4d6c-a0e5-bbe6d93dd836/eeb2d821-a432-4df6-8856-fdb14df260e3: Could not refresh total sector count: Input/output error<br></div><div>----------<br></div><div><br></div><div>I also am including a few command outputs that I&#39;m sure might help:</div><div><br></div><div>----------<br></div><div><div>[root@orldc-dev-vnode02 4e3017eb-d062-4ad1-9df8-7057fcee412c]# ls -la /var/run/vdsm/storage/4e3017eb-d062-4ad1-9df8-7057fcee412c/</div><div>total 8</div><div>drwxr-xr-x. 2 vdsm kvm  80 Jul  1 20:04 .</div><div>drwxr-xr-x. 3 vdsm kvm  60 Jul  1 20:04 ..</div><div>lrwxrwxrwx. 1 vdsm kvm 128 Jul  1 20:04 1d80a60c-8f26-4448-9460-2c7b00ff75bf -&gt; /rhev/data-center/mnt/ovirt-gluster-nfs:_engine/4e3017eb-d062-4ad1-9df8-7057fcee412c/images/1d80a60c-8f26-4448-9460-2c7b00ff75bf</div><div>lrwxrwxrwx. 1 vdsm kvm 128 Jul  1 20:04 23ac8897-b0c7-41d6-a7de-19f46ed78400 -&gt; /rhev/data-center/mnt/ovirt-gluster-nfs:_engine/4e3017eb-d062-4ad1-9df8-7057fcee412c/images/23ac8897-b0c7-41d6-a7de-19f46ed78400</div></div><div><br></div><div>----------<br></div><div><div>[root@orldc-dev-vnode02 4e3017eb-d062-4ad1-9df8-7057fcee412c]# cat /etc/ovirt-hosted-engine/vm.conf</div><div>vmId=6b7329f9-518a-4488-b1c4-2cd809f2f580</div><div>memSize=5120</div><div>display=vnc</div><div>devices={index:2,iface:ide,address:{ controller:0, target:0,unit:0, bus:1, type:drive},specParams:{},readonly:true,deviceId:77924fc2-c5c9-408b-97d3-cd0b0d11a62c,path:/home/tmp/CentOS-6.6-x86_64-minimal.iso,device:cdrom,shared:false,type:disk}</div><div>devices={index:0,iface:virtio,format:raw,poolID:00000000-0000-0000-0000-000000000000,volumeID:eeb2d821-a432-4df6-8856-fdb14df260e3,imageID:5ead7b5d-50e8-4d6c-a0e5-bbe6d93dd836,specParams:{},readonly:false,domainID:4e3017eb-d062-4ad1-9df8-7057fcee412c,optional:false,deviceId:5ead7b5d-50e8-4d6c-a0e5-bbe6d93dd836,address:{bus:0x00, slot:0x06, domain:0x0000, type:pci, function:0x0},device:disk,shared:exclusive,propagateErrors:off,type:disk,bootOrder:1}</div><div>devices={device:scsi,model:virtio-scsi,type:controller}</div><div>devices={nicModel:pv,macAddr:00:16:3e:0e:d0:68,linkActive:true,network:ovirtmgmt,filter:vdsm-no-mac-spoofing,specParams:{},deviceId:f70ba622-6ac8-4c06-a005-0ebd940a15b2,address:{bus:0x00, slot:0x03, domain:0x0000, type:pci, function:0x0},device:bridge,type:interface}</div><div>devices={device:console,specParams:{},type:console,deviceId:98386e6c-ae56-4b6d-9bfb-c72bbd299ad1,alias:console0}</div><div>vmName=HostedEngine</div><div>spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir</div><div>smp=2</div><div>cpuType=Westmere</div><div>emulatedMachine=pc</div></div><div>----------<br></div><div><br></div><div><div>[root@orldc-dev-vnode02 4e3017eb-d062-4ad1-9df8-7057fcee412c]# cat /etc/ovirt-hosted-engine/hosted-engine.conf</div><div>fqdn=orldc-dev-vengine01.*******</div><div>vm_disk_id=5ead7b5d-50e8-4d6c-a0e5-bbe6d93dd836</div><div>vmid=6b7329f9-518a-4488-b1c4-2cd809f2f580</div><div>storage=ovirt-gluster-nfs:/engine</div><div>conf=/etc/ovirt-hosted-engine/vm.conf</div><div>service_start_time=0</div><div>host_id=1</div><div>console=vnc</div><div>domainType=nfs3</div><div>spUUID=379cf161-d5b1-4c20-bb71-e3ca5d2ccd6b</div><div>sdUUID=4e3017eb-d062-4ad1-9df8-7057fcee412c</div><div>connectionUUID=0d1b50ac-cf3f-4cd7-90df-3c3a6d11a984</div><div>ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem</div><div>ca_subject=&quot;C=EN, L=Test, O=Test, CN=Test&quot;</div><div>vdsm_use_ssl=true</div><div>gateway=10.16.3.1</div><div>bridge=ovirtmgmt</div><div>metadata_volume_UUID=dd9f373c-d161-4fa0-aab1-3cb52305dba7</div><div>metadata_image_UUID=23ac8897-b0c7-41d6-a7de-19f46ed78400</div><div>lockspace_volume_UUID=d9bacbf6-c2f4-4f74-a91f-3a3a52f255bf</div><div>lockspace_image_UUID=1d80a60c-8f26-4448-9460-2c7b00ff75bf</div><div><br></div><div># The following are used only for iSCSI storage</div><div>iqn=</div><div>portal=</div><div>user=</div><div>password=</div><div>port=</div></div><div>----------</div><div><br></div><div>(mount output for the NFS share this should be running on (gluster-based):</div><div><br></div><div>ovirt-gluster-nfs:/engine on /rhev/data-center/mnt/ovirt-gluster-nfs:_engine type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nolock,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.16.3.30,mountvers=3,mountport=38465,mountproto=tcp,local_lock=all,addr=10.16.3.30)<br></div><div><br></div><div>----------</div><div><br></div><div><div>[root@orldc-dev-vnode02 4e3017eb-d062-4ad1-9df8-7057fcee412c]# tree /rhev/data-center/mnt/ovirt-gluster-nfs\:_engine/</div><div>/rhev/data-center/mnt/ovirt-gluster-nfs:_engine/</div><div>├── 4e3017eb-d062-4ad1-9df8-7057fcee412c</div><div>│   ├── dom_md</div><div>│   │   ├── ids</div><div>│   │   ├── inbox</div><div>│   │   ├── leases</div><div>│   │   ├── metadata</div><div>│   │   └── outbox</div><div>│   ├── ha_agent</div><div>│   │   ├── hosted-engine.lockspace -&gt; /rhev/data-center/mnt/ovirt-gluster-nfs:_engine/4e3017eb-d062-4ad1-9df8-7057fcee412c/images/1d80a60c-8f26-4448-9460-2c7b00ff75bf/d9bacbf6-c2f4-4f74-a91f-3a3a52f255bf</div><div>│   │   └── hosted-engine.metadata -&gt; /rhev/data-center/mnt/ovirt-gluster-nfs:_engine/4e3017eb-d062-4ad1-9df8-7057fcee412c/images/23ac8897-b0c7-41d6-a7de-19f46ed78400/dd9f373c-d161-4fa0-aab1-3cb52305dba7</div><div>│   ├── images</div><div>│   │   ├── 1d80a60c-8f26-4448-9460-2c7b00ff75bf</div><div>│   │   │   ├── d9bacbf6-c2f4-4f74-a91f-3a3a52f255bf</div><div>│   │   │   ├── d9bacbf6-c2f4-4f74-a91f-3a3a52f255bf.lease</div><div>│   │   │   └── d9bacbf6-c2f4-4f74-a91f-3a3a52f255bf.meta</div><div>│   │   ├── 23ac8897-b0c7-41d6-a7de-19f46ed78400</div><div>│   │   │   ├── dd9f373c-d161-4fa0-aab1-3cb52305dba7 -&gt; /rhev/data-center/mnt/ovirt-gluster-nfs:_engine/4e3017eb-d062-4ad1-9df8-7057fcee412c/images/23ac8897-b0c7-41d6-a7de-19f46ed78400</div><div>│   │   │   ├── dd9f373c-d161-4fa0-aab1-3cb52305dba7.lease</div><div>│   │   │   └── dd9f373c-d161-4fa0-aab1-3cb52305dba7.meta</div><div>│   │   ├── 3278c444-d92a-4cb9-87d6-9669c6e4993e</div><div>│   │   │   ├── 1a4b6a5d-9c8f-4d54-91a7-3dd87377c362</div><div>│   │   │   ├── 1a4b6a5d-9c8f-4d54-91a7-3dd87377c362.lease</div><div>│   │   │   └── 1a4b6a5d-9c8f-4d54-91a7-3dd87377c362.meta</div><div>│   │   ├── 5ead7b5d-50e8-4d6c-a0e5-bbe6d93dd836</div><div>│   │   │   ├── eeb2d821-a432-4df6-8856-fdb14df260e3.lease</div><div>│   │   │   └── eeb2d821-a432-4df6-8856-fdb14df260e3.meta</div><div>│   │   ├── 6064179f-2720-4db9-a7c4-a97e044c2238</div><div>│   │   │   ├── 05afaa26-95af-4226-9a00-5383d8937a83</div><div>│   │   │   ├── 05afaa26-95af-4226-9a00-5383d8937a83.lease</div><div>│   │   │   └── 05afaa26-95af-4226-9a00-5383d8937a83.meta</div><div>│   │   └── bb9d9a37-4f91-4973-ba9e-72ee81aed0b6</div><div>│   │       ├── 5acb27b3-62c5-46ac-8978-576a8a4a0399</div><div>│   │       ├── 5acb27b3-62c5-46ac-8978-576a8a4a0399.lease</div><div>│   │       └── 5acb27b3-62c5-46ac-8978-576a8a4a0399.meta</div><div>│   └── master</div><div>│       ├── tasks</div><div>│       │   └── fef13299-0e7f-4c7a-a399-092a1235faab</div><div>│       │       ├── fef13299-0e7f-4c7a-a399-092a1235faab.job.0</div><div>│       │       ├── fef13299-0e7f-4c7a-a399-092a1235faab.recover.0</div><div>│       │       ├── fef13299-0e7f-4c7a-a399-092a1235faab.recover.1</div><div>│       │       ├── fef13299-0e7f-4c7a-a399-092a1235faab.result</div><div>│       │       └── fef13299-0e7f-4c7a-a399-092a1235faab.task</div><div>│       └── vms</div><div>└── __DIRECT_IO_TEST__</div><div><br></div><div>16 directories, 28 files</div></div><div>----------<br></div><div><br></div><div><div>[root@orldc-dev-vnode02 4e3017eb-d062-4ad1-9df8-7057fcee412c]# tree /var/run/vdsm/storage/4e3017eb-d062-4ad1-9df8-7057fcee412c</div><div>/var/run/vdsm/storage/4e3017eb-d062-4ad1-9df8-7057fcee412c</div><div>├── 1d80a60c-8f26-4448-9460-2c7b00ff75bf -&gt; /rhev/data-center/mnt/ovirt-gluster-nfs:_engine/4e3017eb-d062-4ad1-9df8-7057fcee412c/images/1d80a60c-8f26-4448-9460-2c7b00ff75bf</div><div>└── 23ac8897-b0c7-41d6-a7de-19f46ed78400 -&gt; /rhev/data-center/mnt/ovirt-gluster-nfs:_engine/4e3017eb-d062-4ad1-9df8-7057fcee412c/images/23ac8897-b0c7-41d6-a7de-19f46ed78400</div><div><br></div><div>2 directories, 0 files</div></div><div>----------<br></div><div><br></div><div><br></div><div>Can you anyone point me in the right direction here?  Eventually, I&#39;d like to get this hosted-engine running in HA mode (I had started this process with ctdb/gluster/etc. via the well-known guide out there), but I&#39;m thinking that at this point, I should get this up and running again, export everything, and perhaps just build a new hosted-engine and import things into (if that is possible).  Right now, though, I just need to get this hosted-engine run.</div><div><br></div><div>Many many thanks!</div><div><br></div><div>Chris</div></div>