GetStorageDeviceListVDS failed
Good morning oVirt community, I need some assistance. I am running a 3 host, hosted engine, gluster environment. The hosts are running centos 7.3 and the engine is version 4.1.2.3 Since yesterday, every 2 hours, the engine reports the error below on host 2 and 3. event ID 10802 VDSM <host> command GetStorageDeviceListVDS failed: 'gluster_vg_sdb-/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error' I've attached logs from both hosts. Hosts 3 is the SPM and holds the arbiter brick of all 4 gluster volumes. As always, your help is appreciated. Joel
Could you provide the supervdsm.log from either host2 or host3. Were the packages on these hosts updated? What's the version of python-blivet? Is this different from host1? On Wed, May 24, 2017 at 6:26 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Good morning oVirt community,
I need some assistance.
I am running a 3 host, hosted engine, gluster environment. The hosts are running centos 7.3 and the engine is version 4.1.2.3
Since yesterday, every 2 hours, the engine reports the error below on host 2 and 3.
event ID 10802
VDSM <host> command GetStorageDeviceListVDS failed: 'gluster_vg_sdb-/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error'
I've attached logs from both hosts. Hosts 3 is the SPM and holds the arbiter brick of all 4 gluster volumes.
As always, your help is appreciated.
Joel
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hello Sahina, Thanks for the response. Attached are the requested supervsdm logs from both hosts. Joel On May 25, 2017 5:41 AM, "Sahina Bose" <sabose@redhat.com> wrote:
Could you provide the supervdsm.log from either host2 or host3.
Were the packages on these hosts updated? What's the version of python-blivet? Is this different from host1?
On Wed, May 24, 2017 at 6:26 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Good morning oVirt community,
I need some assistance.
I am running a 3 host, hosted engine, gluster environment. The hosts are running centos 7.3 and the engine is version 4.1.2.3
Since yesterday, every 2 hours, the engine reports the error below on host 2 and 3.
event ID 10802
VDSM <host> command GetStorageDeviceListVDS failed: 'gluster_vg_sdb-/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error'
I've attached logs from both hosts. Hosts 3 is the SPM and holds the arbiter brick of all 4 gluster volumes.
As always, your help is appreciated.
Joel
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I almost forgot about python-blivet. All 3 hosts are on version 0.61.15.59-1. The hosts and engine were updated using the snapshot repo last week. I updated them just now. I just noticed that host 2 and required the same 20 updates but host 1 required an additional update, ovirt-engine-appliance. On May 25, 2017 10:40 AM, "Joel Diaz" <mrjoeldiaz@gmail.com> wrote: Hello Sahina, Thanks for the response. Attached are the requested supervsdm logs from both hosts. Joel On May 25, 2017 5:41 AM, "Sahina Bose" <sabose@redhat.com> wrote:
Could you provide the supervdsm.log from either host2 or host3.
Were the packages on these hosts updated? What's the version of python-blivet? Is this different from host1?
On Wed, May 24, 2017 at 6:26 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Good morning oVirt community,
I need some assistance.
I am running a 3 host, hosted engine, gluster environment. The hosts are running centos 7.3 and the engine is version 4.1.2.3
Since yesterday, every 2 hours, the engine reports the error below on host 2 and 3.
event ID 10802
VDSM <host> command GetStorageDeviceListVDS failed: 'gluster_vg_sdb-/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error'
I've attached logs from both hosts. Hosts 3 is the SPM and holds the arbiter brick of all 4 gluster volumes.
As always, your help is appreciated.
Joel
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Thu, May 25, 2017 at 8:29 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
I almost forgot about python-blivet. All 3 hosts are on version 0.61.15.59-1.
The hosts and engine were updated using the snapshot repo last week. I updated them just now. I just noticed that host 2 and required the same 20 updates but host 1 required an additional update, ovirt-engine-appliance.
On May 25, 2017 10:40 AM, "Joel Diaz" <mrjoeldiaz@gmail.com> wrote:
Hello Sahina,
Thanks for the response.
Attached are the requested supervsdm logs from both hosts.
Seems to be an error returned by lvm module. " lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype" - seems to throw this error. Can you check? Do you have stale devicemapper entries on these hosts?
Joel
On May 25, 2017 5:41 AM, "Sahina Bose" <sabose@redhat.com> wrote:
Could you provide the supervdsm.log from either host2 or host3.
Were the packages on these hosts updated? What's the version of python-blivet? Is this different from host1?
On Wed, May 24, 2017 at 6:26 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Good morning oVirt community,
I need some assistance.
I am running a 3 host, hosted engine, gluster environment. The hosts are running centos 7.3 and the engine is version 4.1.2.3
Since yesterday, every 2 hours, the engine reports the error below on host 2 and 3.
event ID 10802
VDSM <host> command GetStorageDeviceListVDS failed: 'gluster_vg_sdb-/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error'
I've attached logs from both hosts. Hosts 3 is the SPM and holds the arbiter brick of all 4 gluster volumes.
As always, your help is appreciated.
Joel
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Sahina, I believe you're correct regarding the stale devicemapper entries. Before this started happening, I added a new iscsi storage domain. Host 1 couldn't communicate with it and the engine kept setting host 1 as non-operational. I removed the iscsi domain to get host 1 working again but the other two host must still think the iscsi storage is attached. Below is the ouput from both hosts. root@<host2> ~]# lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype /dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error /dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511562240: Input/output error /dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511619584: Input/output error /dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536805376: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536862720: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134152192: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134209536: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147418112 <%28214%29%20741-8112>: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147475456 <%28214%29%20747-5456>: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134152192: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134209536: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073676288: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073733632: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134152192: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134209536: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073676288: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073733632: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 4096: Input/output error /dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 0: Input/output error /dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870846464: Input/output error /dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870903808: Input/output error /dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 4096: Input/output error LVM2_VG_NAME=centos_ovirt-hyp-02 LVM2_LV_NAME=home LVM2_LV_UUID=WmKd5N-uhew-w2uu-P5Qy-Xpeq-quDz-LrLM3n LVM2_LV_SIZE=63627264.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=centos_ovirt-hyp-02 LVM2_LV_NAME=root LVM2_LV_UUID=qeFOoZ-3WR0-LX2J-BZXh-oo9u-mqXJ-NRJkq3 LVM2_LV_SIZE=52428800.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=centos_ovirt-hyp-02 LVM2_LV_NAME=swap LVM2_LV_UUID=FQ6NIv-N5bc-uo6Q-IdZS-i6XL-Y4TQ-5mOkSd LVM2_LV_SIZE=8192000.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_data LVM2_LV_UUID=nhoFx5-aHEd-ZUe4-eEuI-FItR-I0nJ-JS1wD3 LVM2_LV_SIZE=681574400.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_engine LVM2_LV_UUID=QAyxti-wHCB-LqEF-HVdK-S1KM-i99p-QBhCun LVM2_LV_SIZE=26214400.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_export LVM2_LV_UUID=Gb4qNx-djQV-85oW-l8Q8-3AfH-zvoa-Y4EFie LVM2_LV_SIZE=104857600.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_iso LVM2_LV_UUID=9zHjQd-QW21-Nu5C-8ggR-y3R7-TgsV-b1ydGp LVM2_LV_SIZE=52428800.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_thinpool_sdb LVM2_LV_UUID=F8J1Uh-So1O-2cju-su1W-BFxI-cgfe-QBYpuT LVM2_LV_SIZE=838860800.00 LVM2_LV_ATTR=twi-aotz-- LVM2_SEGTYPE=thin-pool LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tdata] LVM2_LV_UUID=gkWCVU-yYuE-iqvU-9p79-k8Oq-EK4g-kBg6py LVM2_LV_SIZE=838860800.00 LVM2_LV_ATTR=Twi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tmeta] LVM2_LV_UUID=6VDVJo-AAgI-HcrP-ANQM-qvgo-GMOp-HAYeQM LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[lvol0_pmspare] LVM2_LV_UUID=STxeFv-0sl2-fCqz-jm4q-kVB5-sek1-BwSVNc LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi------- LVM2_SEGTYPE=linear [root@<host3> ~]# lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype /dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error /dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511562240: Input/output error /dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511619584: Input/output error /dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536805376: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536862720: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134152192: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134209536: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073676288: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073733632: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147418112 <%28214%29%20741-8112>: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147475456 <%28214%29%20747-5456>: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134152192: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134209536: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134152192: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134209536: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073676288: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073733632: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 4096: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 0: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 21608988672: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 21609046016: Input/output error /dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 4096: Input/output error /dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 0: Input/output error /dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870846464: Input/output error /dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870903808: Input/output error /dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 4096: Input/output error LVM2_VG_NAME=centos_ovirt-hyp-03 LVM2_LV_NAME=home LVM2_LV_UUID=d4pR6F-JD7f-JR7F-RjYv-EkB9-ka7b-4BVJqZ LVM2_LV_SIZE=63832064.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=centos_ovirt-hyp-03 LVM2_LV_NAME=root LVM2_LV_UUID=Dk3ihf-qtXB-8h7m-0zul-d9wo-oTGd-4gijh1 LVM2_LV_SIZE=52428800.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=centos_ovirt-hyp-03 LVM2_LV_NAME=swap LVM2_LV_UUID=RVT0Wl-f3Kx-vxha-CIFA-nK1M-IEeU-CNyFFu LVM2_LV_SIZE=8192000.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_data LVM2_LV_UUID=ILg0Ym-l0M3-Zxg7-heDz-VsDF-yfHJ-jy8gJG LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_engine LVM2_LV_UUID=NnIR9J-4PIP-ksQA-PdpX-1epc-X5iT-Wk9qiO LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_export LVM2_LV_UUID=z6hJeU-wb4z-WzHv-jHKv-2sB5-BUoR-XK9zRf LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_iso LVM2_LV_UUID=kj9fMh-SBym-erPu-sgOe-m6kU-TWsc-LmRP65 LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_thinpool_sdb LVM2_LV_UUID=FGu1eb-T273-fEY0-pThE-lXXc-zBCs-OfEJkJ LVM2_LV_SIZE=73400320.00 LVM2_LV_ATTR=twi-aotz-- LVM2_SEGTYPE=thin-pool LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tdata] LVM2_LV_UUID=CnqJbS-o7XA-4XD0-0nQ5-t3Wu-iRfW-Nc337j LVM2_LV_SIZE=73400320.00 LVM2_LV_ATTR=Twi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tmeta] LVM2_LV_UUID=f9KiCq-UUMN-eve1-lqLK-0lAI-VRP5-ASKGPt LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi-ao---- LVM2_SEGTYPE=linear LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[lvol0_pmspare] LVM2_LV_UUID=C24xEs-njob-0E7Q-vwJG-q814-FmMQ-ebrb33 LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi------- LVM2_SEGTYPE=linear How can I clean that up? Thank you, Joel On May 25, 2017 11:49 AM, "Sahina Bose" <sabose@redhat.com> wrote: On Thu, May 25, 2017 at 8:29 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
I almost forgot about python-blivet. All 3 hosts are on version 0.61.15.59-1.
The hosts and engine were updated using the snapshot repo last week. I updated them just now. I just noticed that host 2 and required the same 20 updates but host 1 required an additional update, ovirt-engine-appliance.
On May 25, 2017 10:40 AM, "Joel Diaz" <mrjoeldiaz@gmail.com> wrote:
Hello Sahina,
Thanks for the response.
Attached are the requested supervsdm logs from both hosts.
Seems to be an error returned by lvm module. " lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype" - seems to throw this error. Can you check? Do you have stale devicemapper entries on these hosts?
Joel
On May 25, 2017 5:41 AM, "Sahina Bose" <sabose@redhat.com> wrote:
Could you provide the supervdsm.log from either host2 or host3.
Were the packages on these hosts updated? What's the version of python-blivet? Is this different from host1?
On Wed, May 24, 2017 at 6:26 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Good morning oVirt community,
I need some assistance.
I am running a 3 host, hosted engine, gluster environment. The hosts are running centos 7.3 and the engine is version 4.1.2.3
Since yesterday, every 2 hours, the engine reports the error below on host 2 and 3.
event ID 10802
VDSM <host> command GetStorageDeviceListVDS failed: 'gluster_vg_sdb-/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error'
I've attached logs from both hosts. Hosts 3 is the SPM and holds the arbiter brick of all 4 gluster volumes.
As always, your help is appreciated.
Joel
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Good morning, Any advice on how to remove those stale device mapper entries? Thank you, On Thu, May 25, 2017 at 12:52 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Sahina,
I believe you're correct regarding the stale devicemapper entries.
Before this started happening, I added a new iscsi storage domain. Host 1 couldn't communicate with it and the engine kept setting host 1 as non-operational. I removed the iscsi domain to get host 1 working again but the other two host must still think the iscsi storage is attached.
Below is the ouput from both hosts.
root@<host2> ~]# lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511562240: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511619584: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536805376: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536862720: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147418112 <%28214%29%20741-8112>: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147475456 <%28214%29%20747-5456>: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073676288: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073733632: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073676288: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073733632: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 4096: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870846464: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870903808: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 4096: Input/output error
LVM2_VG_NAME=centos_ovirt-hyp-02 LVM2_LV_NAME=home LVM2_LV_UUID=WmKd5N-uhew-w2uu-P5Qy-Xpeq-quDz-LrLM3n LVM2_LV_SIZE=63627264.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=centos_ovirt-hyp-02 LVM2_LV_NAME=root LVM2_LV_UUID=qeFOoZ-3WR0-LX2J-BZXh-oo9u-mqXJ-NRJkq3 LVM2_LV_SIZE=52428800.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=centos_ovirt-hyp-02 LVM2_LV_NAME=swap LVM2_LV_UUID=FQ6NIv-N5bc-uo6Q-IdZS-i6XL-Y4TQ-5mOkSd LVM2_LV_SIZE=8192000.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_data LVM2_LV_UUID=nhoFx5-aHEd-ZUe4-eEuI-FItR-I0nJ-JS1wD3 LVM2_LV_SIZE=681574400.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_engine LVM2_LV_UUID=QAyxti-wHCB-LqEF-HVdK-S1KM-i99p-QBhCun LVM2_LV_SIZE=26214400.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_export LVM2_LV_UUID=Gb4qNx-djQV-85oW-l8Q8-3AfH-zvoa-Y4EFie LVM2_LV_SIZE=104857600.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_iso LVM2_LV_UUID=9zHjQd-QW21-Nu5C-8ggR-y3R7-TgsV-b1ydGp LVM2_LV_SIZE=52428800.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_thinpool_sdb LVM2_LV_UUID=F8J1Uh-So1O-2cju-su1W-BFxI-cgfe-QBYpuT LVM2_LV_SIZE=838860800.00 LVM2_LV_ATTR=twi-aotz-- LVM2_SEGTYPE=thin-pool
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tdata] LVM2_LV_UUID=gkWCVU-yYuE-iqvU-9p79-k8Oq-EK4g-kBg6py LVM2_LV_SIZE=838860800.00 LVM2_LV_ATTR=Twi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tmeta] LVM2_LV_UUID=6VDVJo-AAgI-HcrP-ANQM-qvgo-GMOp-HAYeQM LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[lvol0_pmspare] LVM2_LV_UUID=STxeFv-0sl2-fCqz-jm4q-kVB5-sek1-BwSVNc LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi------- LVM2_SEGTYPE=linear
[root@<host3> ~]# lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511562240: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511619584: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536805376: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536862720: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073676288: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073733632: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147418112 <%28214%29%20741-8112>: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147475456 <%28214%29%20747-5456>: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073676288: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073733632: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 21608988672: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 21609046016: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 4096: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870846464: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870903808: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 4096: Input/output error
LVM2_VG_NAME=centos_ovirt-hyp-03 LVM2_LV_NAME=home LVM2_LV_UUID=d4pR6F-JD7f-JR7F-RjYv-EkB9-ka7b-4BVJqZ LVM2_LV_SIZE=63832064.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=centos_ovirt-hyp-03 LVM2_LV_NAME=root LVM2_LV_UUID=Dk3ihf-qtXB-8h7m-0zul-d9wo-oTGd-4gijh1 LVM2_LV_SIZE=52428800.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=centos_ovirt-hyp-03 LVM2_LV_NAME=swap LVM2_LV_UUID=RVT0Wl-f3Kx-vxha-CIFA-nK1M-IEeU-CNyFFu LVM2_LV_SIZE=8192000.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_data LVM2_LV_UUID=ILg0Ym-l0M3-Zxg7-heDz-VsDF-yfHJ-jy8gJG LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_engine LVM2_LV_UUID=NnIR9J-4PIP-ksQA-PdpX-1epc-X5iT-Wk9qiO LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_export LVM2_LV_UUID=z6hJeU-wb4z-WzHv-jHKv-2sB5-BUoR-XK9zRf LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_iso LVM2_LV_UUID=kj9fMh-SBym-erPu-sgOe-m6kU-TWsc-LmRP65 LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_thinpool_sdb LVM2_LV_UUID=FGu1eb-T273-fEY0-pThE-lXXc-zBCs-OfEJkJ LVM2_LV_SIZE=73400320.00 LVM2_LV_ATTR=twi-aotz-- LVM2_SEGTYPE=thin-pool
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tdata] LVM2_LV_UUID=CnqJbS-o7XA-4XD0-0nQ5-t3Wu-iRfW-Nc337j LVM2_LV_SIZE=73400320.00 LVM2_LV_ATTR=Twi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tmeta] LVM2_LV_UUID=f9KiCq-UUMN-eve1-lqLK-0lAI-VRP5-ASKGPt LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[lvol0_pmspare] LVM2_LV_UUID=C24xEs-njob-0E7Q-vwJG-q814-FmMQ-ebrb33 LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi------- LVM2_SEGTYPE=linear
How can I clean that up?
Thank you,
Joel
On May 25, 2017 11:49 AM, "Sahina Bose" <sabose@redhat.com> wrote:
On Thu, May 25, 2017 at 8:29 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
I almost forgot about python-blivet. All 3 hosts are on version 0.61.15.59-1.
The hosts and engine were updated using the snapshot repo last week. I updated them just now. I just noticed that host 2 and required the same 20 updates but host 1 required an additional update, ovirt-engine-appliance.
On May 25, 2017 10:40 AM, "Joel Diaz" <mrjoeldiaz@gmail.com> wrote:
Hello Sahina,
Thanks for the response.
Attached are the requested supervsdm logs from both hosts.
Seems to be an error returned by lvm module.
" lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype" - seems to throw this error. Can you check?
Do you have stale devicemapper entries on these hosts?
Joel
On May 25, 2017 5:41 AM, "Sahina Bose" <sabose@redhat.com> wrote:
Could you provide the supervdsm.log from either host2 or host3.
Were the packages on these hosts updated? What's the version of python-blivet? Is this different from host1?
On Wed, May 24, 2017 at 6:26 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Good morning oVirt community,
I need some assistance.
I am running a 3 host, hosted engine, gluster environment. The hosts are running centos 7.3 and the engine is version 4.1.2.3
Since yesterday, every 2 hours, the engine reports the error below on host 2 and 3.
event ID 10802
VDSM <host> command GetStorageDeviceListVDS failed: 'gluster_vg_sdb-/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error'
I've attached logs from both hosts. Hosts 3 is the SPM and holds the arbiter brick of all 4 gluster volumes.
As always, your help is appreciated.
Joel
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I'm not an lvm expert, a quick search indicates either reboot or using dmsetup to clear the entries Alternately, you can check on https://www.redhat.com/archives/linux-lvm/ or ask on the list. On Mon, May 29, 2017 at 10:03 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Good morning,
Any advice on how to remove those stale device mapper entries?
Thank you,
On Thu, May 25, 2017 at 12:52 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Sahina,
I believe you're correct regarding the stale devicemapper entries.
Before this started happening, I added a new iscsi storage domain. Host 1 couldn't communicate with it and the engine kept setting host 1 as non-operational. I removed the iscsi domain to get host 1 working again but the other two host must still think the iscsi storage is attached.
Below is the ouput from both hosts.
root@<host2> ~]# lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511562240: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511619584: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536805376: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536862720: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147418112 <%28214%29%20741-8112>: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147475456 <%28214%29%20747-5456>: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073676288: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073733632: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073676288: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073733632: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 4096: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870846464: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870903808: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 4096: Input/output error
LVM2_VG_NAME=centos_ovirt-hyp-02 LVM2_LV_NAME=home LVM2_LV_UUID=WmKd5N-uhew-w2uu-P5Qy-Xpeq-quDz-LrLM3n LVM2_LV_SIZE=63627264.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=centos_ovirt-hyp-02 LVM2_LV_NAME=root LVM2_LV_UUID=qeFOoZ-3WR0-LX2J-BZXh-oo9u-mqXJ-NRJkq3 LVM2_LV_SIZE=52428800.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=centos_ovirt-hyp-02 LVM2_LV_NAME=swap LVM2_LV_UUID=FQ6NIv-N5bc-uo6Q-IdZS-i6XL-Y4TQ-5mOkSd LVM2_LV_SIZE=8192000.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_data LVM2_LV_UUID=nhoFx5-aHEd-ZUe4-eEuI-FItR-I0nJ-JS1wD3 LVM2_LV_SIZE=681574400.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_engine LVM2_LV_UUID=QAyxti-wHCB-LqEF-HVdK-S1KM-i99p-QBhCun LVM2_LV_SIZE=26214400.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_export LVM2_LV_UUID=Gb4qNx-djQV-85oW-l8Q8-3AfH-zvoa-Y4EFie LVM2_LV_SIZE=104857600.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_iso LVM2_LV_UUID=9zHjQd-QW21-Nu5C-8ggR-y3R7-TgsV-b1ydGp LVM2_LV_SIZE=52428800.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_thinpool_sdb LVM2_LV_UUID=F8J1Uh-So1O-2cju-su1W-BFxI-cgfe-QBYpuT LVM2_LV_SIZE=838860800.00 LVM2_LV_ATTR=twi-aotz-- LVM2_SEGTYPE=thin-pool
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tdata] LVM2_LV_UUID=gkWCVU-yYuE-iqvU-9p79-k8Oq-EK4g-kBg6py LVM2_LV_SIZE=838860800.00 LVM2_LV_ATTR=Twi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tmeta] LVM2_LV_UUID=6VDVJo-AAgI-HcrP-ANQM-qvgo-GMOp-HAYeQM LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[lvol0_pmspare] LVM2_LV_UUID=STxeFv-0sl2-fCqz-jm4q-kVB5-sek1-BwSVNc LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi------- LVM2_SEGTYPE=linear
[root@<host3> ~]# lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511562240: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 1099511619584: Input/output error
/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536805376: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 536862720: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/metadata: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/outbox: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073676288: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 1073733632: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/xleases: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147418112 <%28214%29%20741-8112>: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 2147475456 <%28214%29%20747-5456>: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/leases: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/ids: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134152192: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 134209536: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/inbox: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073676288: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 1073733632: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/master: read failed after 0 of 4096 at 4096: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 0: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 21608988672: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 21609046016: Input/output error
/dev/81f7db35-cff2-4592-bad7-55f437bb09ee/d7e958f7-e307-4c33-95ea-f98532ad6fd0: read failed after 0 of 4096 at 4096: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870846464: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 536870903808: Input/output error
/dev/mapper/36589cfc000000b8e821765d5febefda2: read failed after 0 of 4096 at 4096: Input/output error
LVM2_VG_NAME=centos_ovirt-hyp-03 LVM2_LV_NAME=home LVM2_LV_UUID=d4pR6F-JD7f-JR7F-RjYv-EkB9-ka7b-4BVJqZ LVM2_LV_SIZE=63832064.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=centos_ovirt-hyp-03 LVM2_LV_NAME=root LVM2_LV_UUID=Dk3ihf-qtXB-8h7m-0zul-d9wo-oTGd-4gijh1 LVM2_LV_SIZE=52428800.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=centos_ovirt-hyp-03 LVM2_LV_NAME=swap LVM2_LV_UUID=RVT0Wl-f3Kx-vxha-CIFA-nK1M-IEeU-CNyFFu LVM2_LV_SIZE=8192000.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_data LVM2_LV_UUID=ILg0Ym-l0M3-Zxg7-heDz-VsDF-yfHJ-jy8gJG LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_engine LVM2_LV_UUID=NnIR9J-4PIP-ksQA-PdpX-1epc-X5iT-Wk9qiO LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=-wi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_export LVM2_LV_UUID=z6hJeU-wb4z-WzHv-jHKv-2sB5-BUoR-XK9zRf LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_lv_iso LVM2_LV_UUID=kj9fMh-SBym-erPu-sgOe-m6kU-TWsc-LmRP65 LVM2_LV_SIZE=10485760.00 LVM2_LV_ATTR=Vwi-aotz-- LVM2_SEGTYPE=thin
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=gluster_thinpool_sdb LVM2_LV_UUID=FGu1eb-T273-fEY0-pThE-lXXc-zBCs-OfEJkJ LVM2_LV_SIZE=73400320.00 LVM2_LV_ATTR=twi-aotz-- LVM2_SEGTYPE=thin-pool
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tdata] LVM2_LV_UUID=CnqJbS-o7XA-4XD0-0nQ5-t3Wu-iRfW-Nc337j LVM2_LV_SIZE=73400320.00 LVM2_LV_ATTR=Twi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[gluster_thinpool_sdb_tmeta] LVM2_LV_UUID=f9KiCq-UUMN-eve1-lqLK-0lAI-VRP5-ASKGPt LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi-ao---- LVM2_SEGTYPE=linear
LVM2_VG_NAME=gluster_vg_sdb LVM2_LV_NAME=[lvol0_pmspare] LVM2_LV_UUID=C24xEs-njob-0E7Q-vwJG-q814-FmMQ-ebrb33 LVM2_LV_SIZE=16777216.00 LVM2_LV_ATTR=ewi------- LVM2_SEGTYPE=linear
How can I clean that up?
Thank you,
Joel
On May 25, 2017 11:49 AM, "Sahina Bose" <sabose@redhat.com> wrote:
On Thu, May 25, 2017 at 8:29 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
I almost forgot about python-blivet. All 3 hosts are on version 0.61.15.59-1.
The hosts and engine were updated using the snapshot repo last week. I updated them just now. I just noticed that host 2 and required the same 20 updates but host 1 required an additional update, ovirt-engine-appliance.
On May 25, 2017 10:40 AM, "Joel Diaz" <mrjoeldiaz@gmail.com> wrote:
Hello Sahina,
Thanks for the response.
Attached are the requested supervsdm logs from both hosts.
Seems to be an error returned by lvm module.
" lvm lvs -a --unit k --nosuffix --nameprefixes --unquoted --noheadings -ovg_name,lv_name,lv_uuid,lv_size,lv_attr,segtype" - seems to throw this error. Can you check?
Do you have stale devicemapper entries on these hosts?
Joel
On May 25, 2017 5:41 AM, "Sahina Bose" <sabose@redhat.com> wrote:
Could you provide the supervdsm.log from either host2 or host3.
Were the packages on these hosts updated? What's the version of python-blivet? Is this different from host1?
On Wed, May 24, 2017 at 6:26 PM, Joel Diaz <mrjoeldiaz@gmail.com> wrote:
Good morning oVirt community,
I need some assistance.
I am running a 3 host, hosted engine, gluster environment. The hosts are running centos 7.3 and the engine is version 4.1.2.3
Since yesterday, every 2 hours, the engine reports the error below on host 2 and 3.
event ID 10802
VDSM <host> command GetStorageDeviceListVDS failed: 'gluster_vg_sdb-/dev/mapper/36589cfc000000f05aea0f2b50f8d76e5: read failed after 0 of 4096 at 0: Input/output error'
I've attached logs from both hosts. Hosts 3 is the SPM and holds the arbiter brick of all 4 gluster volumes.
As always, your help is appreciated.
Joel
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Joel Diaz -
Sahina Bose