I've spent the weekend trying to get to the bottom of this issue. Adding a Host fails: From RHVM VDSM rhv1 command failed: Connection reset by peer Could not get hardware information for host rhv1 VDSM rhv1 command failed: Failed to read hardware information Host rhv1 installed Network changes were saved on host rhv1 Installing Host rhv1. Stage: Termination. Installing Host rhv1. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-deploy- 20161010115606-192.168.21.71-24d39274.log'. Installing Host rhv1. Stage: Pre-termination. Installing Host rhv1. Starting ovirt-vmconsole-host-sshd. Installing Host rhv1. Starting vdsm. Installing Host rhv1. Stopping libvirtd. Installing Host rhv1. Stage: Closing up. Installing Host rhv1. Setting kernel arguments. Installing Host rhv1. Stage: Transaction commit. Installing Host rhv1. Enrolling serial console certificate. Installing Host rhv1. Enrolling certificate. Installing Host rhv1. Stage: Misc configuration. This was in the /var/log/vdsm/vdsm.log on the host trying to be added: jsonrpc.Executor/2::ERROR::2016-10-10 11:57:10,276::API::1340::vds::(getHardwareInfo) failed to retrieve hardware info Traceback (most recent call last): File "/usr/share/vdsm/API.py", line 1337, in getHardwareInfo hw = supervdsm.getProxy().getHardwareInfo() File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in <lambda> **kwargs) File "<string>", line 2, in getHardwareInfo File "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod kind, result = conn.recv() EOFError and then VDSM fails to start. Looking at the source code... def getHardwareInfoStructure(): dmiInfo = getAllDmidecodeInfo() sysStruct = {} for k1, k2 in (('system', 'Manufacturer'), ('system', 'Product Name'), ('system', 'Version'), ('system', 'Serial Number'), ('system', 'UUID'), ('system', 'Family')): val = dmiInfo.get(k1, {}).get(k2, None) if val not in [None, 'Not Specified']: sysStruct[(k1 + k2).replace(' ', '')] = val return sysStruct Running dmidecode from command line I get.. System Information Manufacturer: Supermicro Product Name: H8DM8-2 Version: 1234567890 Serial Number: 1234567890 UUID: 00020003-0004-0005-0006-000700080009 Wake-up Type: Power Switch SKU Number: To Be Filled By O.E.M. Family: To Be Filled By O.E.M. Q: Is the string in Family the source of my problems?? Q: Any work arounds?? -- David Pinkerton Consultant Red Hat Asia Pacific Pty. Ltd. Level 11, Canberra House 40 Marcus Clarke Street Canberra 2600 ACT Mobile: +61-488-904-232 Email: david.pinkerton(a)redhat.com Web: http://apac.redhat.com/ <http://apac.redhat.com/> _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On Mon, Oct 10, 2016 at 5:05 AM, Charles Kozler <ckozleriii(a)gmail.com&gt; wrote: > Possibly stupid question but are you doing this on a base empty > centos/rhel 7? > > On Oct 9, 2016 9:48 PM, "David Pinkerton" <dpinkert(a)redhat.com&gt; wrote: > >> >> I've spent the weekend trying to get to the bottom of this issue. >> >> Adding a Host fails: >> >> From RHVM >> >> >> VDSM rhv1 command failed: Connection reset by peer >> Could not get hardware information for host rhv1 >> VDSM rhv1 command failed: Failed to read hardware information >> Host rhv1 installed >> Network changes were saved on host rhv1 >> Installing Host rhv1. Stage: Termination. >> Installing Host rhv1. Retrieving installation logs to: >> '/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-2016101 >> 0115606-192.168.21.71-24d39274.log'. >> Installing Host rhv1. Stage: Pre-termination. >> Installing Host rhv1. Starting ovirt-vmconsole-host-sshd. >> Installing Host rhv1. Starting vdsm. >> Installing Host rhv1. Stopping libvirtd. >> Installing Host rhv1. Stage: Closing up. >> Installing Host rhv1. Setting kernel arguments. >> Installing Host rhv1. Stage: Transaction commit. >> Installing Host rhv1. Enrolling serial console certificate. >> Installing Host rhv1. Enrolling certificate. >> Installing Host rhv1. Stage: Misc configuration. >> >> >> >> This was in the /var/log/vdsm/vdsm.log on the host trying to be added: >> >> jsonrpc.Executor/2::ERROR::2016-10-10 11:57:10,276::API::1340::vds::(getHardwareInfo) >> failed to retrieve hardware info >> Traceback (most recent call last): >> File "/usr/share/vdsm/API.py", line 1337, in getHardwareInfo >> hw = supervdsm.getProxy().getHardwareInfo() >> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, >> in __call__ >> return callMethod() >> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, >> in <lambda> >> **kwargs) >> File "<string>", line 2, in getHardwareInfo >> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in >> _callmethod >> kind, result = conn.recv() >> EOFError >> > If a request to supervdsm fails with EOFError, something bad happened supervdsm and we would see the exception in the supervdsm log. Can you share supervdsm.log? Nir

From the keywords it look like my dmi info is corrupted. I have download a

On Tue, Oct 11, 2016 at 11:59 PM, David Pinkerton <dpinkert(a)redhat.com&gt; wrote: > Logs attached According vdsm.log and supervdsm.log, each time vdsm try to call getHardwareInfo, supervdsm show the start of the call and then it show no logs for 10 seconds, and than we see the startup message. So it seems that supervdsm is crashing each time it try to invoke dmidecode code. To dig deeper, I suggest you try to run the relevant code from the shell. If this code crash, we will see the details in the shell, and we can also run the python shell in gdb to debug this. Try this: 1. Open a python shell as root $ sudo python 2. In the shell, type this >>> from vdsm import dmidecodeUtil >>> dmidecodeUtil.getHardwareInfoStructure() If at this point the the python shell crash, please try: 1. Install python debug-info packages: $ sudo debuginfo-install -y python 2. Start python in gdb $ sudo gdb python 3. In the gdb shell, run python (gdb) run Python shell will show, type the code above again. If this crash in gdb, please type this in the gdb shell: (gdb) thread apply all bt full Nir > > On Mon, Oct 10, 2016 at 4:59 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >> On Mon, Oct 10, 2016 at 5:05 AM, Charles Kozler <ckozleriii(a)gmail.com&gt; >> wrote: >>> >>> Possibly stupid question but are you doing this on a base empty >>> centos/rhel 7? >>> >>> >>> On Oct 9, 2016 9:48 PM, "David Pinkerton" <dpinkert(a)redhat.com&gt; wrote: >>>> >>>> >>>> I've spent the weekend trying to get to the bottom of this issue. >>>> >>>> Adding a Host fails: >>>> >>>> From RHVM >>>> >>>> >>>> VDSM rhv1 command failed: Connection reset by peer >>>> Could not get hardware information for host rhv1 >>>> VDSM rhv1 command failed: Failed to read hardware information >>>> Host rhv1 installed >>>> Network changes were saved on host rhv1 >>>> Installing Host rhv1. Stage: Termination. >>>> Installing Host rhv1. Retrieving installation logs to: >>>> '/var/log/ovirt-engine/host-deploy/ovirt-host-deploy- 20161010115606-192.168.21.71-24d39274.log'. >>>> Installing Host rhv1. Stage: Pre-termination. >>>> Installing Host rhv1. Starting ovirt-vmconsole-host-sshd. >>>> Installing Host rhv1. Starting vdsm. >>>> Installing Host rhv1. Stopping libvirtd. >>>> Installing Host rhv1. Stage: Closing up. >>>> Installing Host rhv1. Setting kernel arguments. >>>> Installing Host rhv1. Stage: Transaction commit. >>>> Installing Host rhv1. Enrolling serial console certificate. >>>> Installing Host rhv1. Enrolling certificate. >>>> Installing Host rhv1. Stage: Misc configuration. >>>> >>>> >>>> >>>> This was in the /var/log/vdsm/vdsm.log on the host trying to be added: >>>> >>>> jsonrpc.Executor/2::ERROR::2016-10-10 >>>> 11:57:10,276::API::1340::vds::(getHardwareInfo) failed to retrieve hardware >>>> info >>>> Traceback (most recent call last): >>>> File "/usr/share/vdsm/API.py", line 1337, in getHardwareInfo >>>> hw = supervdsm.getProxy().getHardwareInfo() >>>> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, in >>>> __call__ >>>> return callMethod() >>>> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in >>>> <lambda> >>>> **kwargs) >>>> File "<string>", line 2, in getHardwareInfo >>>> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in >>>> _callmethod >>>> kind, result = conn.recv() >>>> EOFError >> >> >> If a request to supervdsm fails with EOFError, something bad happened >> supervdsm and we would see the exception in the supervdsm log. >> >> Can you share supervdsm.log? >> >> Nir > > > > > -- > > David Pinkerton > Consultant > Red Hat Asia Pacific Pty. Ltd. > Level 11, Canberra House > 40 Marcus Clarke Street > Canberra 2600 ACT > > Mobile: +61-488-904-232 > Email: david.pinkerton(a)redhat.com > Web: http://apac.redhat.com/ > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >

On Thu, Oct 13, 2016 at 11:52:17AM +1100, David Pinkerton wrote: > Nir, > > Looks like its crashing on the dmidecode system call. > > I've attached the output from gbd as well as a dmidecode text dump, > dmidecode binary dump and each keywords run individually. > > >From the keywords it look like my dmi info is corrupted. I have download a > AMI dmi editor but this only allows access to limited fields. Do you know > another tools to rewrite the dmi info? I don't. But whatever is inside your dmi, dmidecode must not crash. Which version of python-dmidecode do you have installed? Would you open a bug against it?

I believe that its maintainers would appriace a simple reproducer, that does not involve ovirt or Vdsm. See if you can simplify the code in def __leafDict(d): ret = {} for k, v in d.iteritems(): if isinstance(v, dict): ret.update(__leafDict(v)) else: ret[k] = v return ret def getAllDmidecodeInfo(): import dmidecode myLeafDict = {} for k in ('system', 'bios', 'cache', 'processor', 'chassis', 'memory'): myLeafDict[k] = __leafDict(getattr(dmidecode, k)()) return myLeafDict _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On 13/10/16 09:01 +0300, Dan Kenigsberg wrote: > On Thu, Oct 13, 2016 at 11:52:17AM +1100, David Pinkerton wrote: > >> Nir, >> >> Looks like its crashing on the dmidecode system call. >> >> I've attached the output from gbd as well as a dmidecode text dump, >> dmidecode binary dump and each keywords run individually. >> >> >From the keywords it look like my dmi info is corrupted. I have >> download a >> AMI dmi editor but this only allows access to limited fields. Do you >> know >> another tools to rewrite the dmi info? >> > > I don't. But whatever is inside your dmi, dmidecode must not crash. > Which version of python-dmidecode do you have installed? > Would you open a bug against it? > This is really unfortunate - I've reproduced the issue with the attached dump and it's python-dmidecode that crashes. The issue is actually fixed upstream, but the version at least in RHEL does not contain the fix. RHEL version: python-dmidecode-3.10.13-11.el7.x86_64 works with (actual upstream): python-dmidecode-3.12.2-1.el7.x86_64 (actually it's ~6 line change in dmioem.c) VDSM output: # vdsClient 0 getVdsHardwareInfo systemFamily = 'To Be Filled By O.E.M.' systemManufacturer = 'Supermicro' systemProductName = 'H8DM8-2' systemSerialNumber = '1234567890' systemUUID = '00020003-0004-0005-0006-000700080009' systemVersion = '1234567890' Although the upstream version of python-dmidecode is able to deal with improper DMI tables, I can't say what else will/will not behave correctly. mpolednik I believe that its maintainers would appriace a simple reproducer, that > does not involve ovirt or Vdsm. See if you can simplify the code in > > def __leafDict(d): > ret = {} > for k, v in d.iteritems(): > if isinstance(v, dict): > ret.update(__leafDict(v)) > else: > ret[k] = v > return ret > > > def getAllDmidecodeInfo(): > import dmidecode > > myLeafDict = {} > for k in ('system', 'bios', 'cache', 'processor', 'chassis', 'memory'): > myLeafDict[k] = __leafDict(getattr(dmidecode, k)()) > return myLeafDict > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >

On 13/10/16 09:01 +0300, Dan Kenigsberg wrote: > > On Thu, Oct 13, 2016 at 11:52:17AM +1100, David Pinkerton wrote: >> >> Nir, >> >> Looks like its crashing on the dmidecode system call. >> >> I've attached the output from gbd as well as a dmidecode text dump, >> dmidecode binary dump and each keywords run individually. >> >> >From the keywords it look like my dmi info is corrupted. I have >> download a >> AMI dmi editor but this only allows access to limited fields. Do you >> know >> another tools to rewrite the dmi info? > > > I don't. But whatever is inside your dmi, dmidecode must not crash. > Which version of python-dmidecode do you have installed? > Would you open a bug against it? This is really unfortunate - I've reproduced the issue with the attached dump

and it's python-dmidecode that crashes. The issue is actually fixed upstream, but the version at least in RHEL does not contain the fix.

RHEL version: python-dmidecode-3.10.13-11.el7.x86_64 works with (actual upstream): python-dmidecode-3.12.2-1.el7.x86_64 (actually it's ~6 line change in dmioem.c) VDSM output: # vdsClient 0 getVdsHardwareInfo systemFamily = 'To Be Filled By O.E.M.' systemManufacturer = 'Supermicro' systemProductName = 'H8DM8-2' systemSerialNumber = '1234567890' systemUUID = '00020003-0004-0005-0006-000700080009' systemVersion = '1234567890' Although the upstream version of python-dmidecode is able to deal with improper DMI tables, I can't say what else will/will not behave correctly. mpolednik > I believe that its maintainers would appriace a simple reproducer, that > does not involve ovirt or Vdsm. See if you can simplify the code in > > def __leafDict(d): > ret = {} > for k, v in d.iteritems(): > if isinstance(v, dict): > ret.update(__leafDict(v)) > else: > ret[k] = v > return ret > > > def getAllDmidecodeInfo(): > import dmidecode > > myLeafDict = {} > for k in ('system', 'bios', 'cache', 'processor', 'chassis', 'memory'): > myLeafDict[k] = __leafDict(getattr(dmidecode, k)()) > return myLeafDict > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users