Re: [ovirt-users] virt-in-virt problem: DHCP failing for a container in a oVirt VM

7 Jun 2016

      Oops, missed that comment...

When I try an operation with virsh, it asks me for credentials that it doesn't accept...

[root@ovirt-node-03 ~]# virsh list
Please enter your authentication name: root
Please enter your password:
error: failed to connect to the hypervisor
error: no valid connection
error: authentication failed: authentication failed

I have tried both 'root' (as above) as well as my credentials I use with oVirt (AD auth), but neither work.

So I searched thru the vdsm logs for the VM's name, and got lines begging with "jsonrpc.Executor/5::DEBUG::2016-06-06 16:45:38,287::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.migrationCreate' in bridge with" with the JSON data thereafter... So this is the JSON pretty-printed:

{  
   u'guestFQDN':u'',
   u'acpiEnable':u'true',
   u'emulatedMachine':u'pc-i440fx-rhel7.2.0',
   u'afterMigrationStatus':u'',
   u'spiceSecureChannels':u'smain,
   sinputs,
   scursor,
   splayback,
   srecord,
   sdisplay,
   susbredir,
   ssmartcard',
   u'vmId':u'1a49b65b-9950-4b29-8c72-139d5263e11d',
   u'memGuaranteedSize':8192,
   u'transparentHugePages':u'true',
   u'displaySecurePort':u'5901',
   u'spiceSslCipherSuite':u'DEFAULT',
   u'cpuType':u'Penryn',
   u'smp':u'4',
   u'pauseCode':u'NOERR',
   u'migrationDest':u'libvirt',
   u'guestNumaNodes':[  
      {  
         u'nodeIndex':0,
         u'cpus':u'0,
         1,
         2,
         3         ', u'         memory':u'8192'
      }
   ],
   u'smartcardEnable':u'false',
   u'numaTune':{  
      u'nodeset':u'0',
      u'mode':u'interleave'
   },
   u'custom':{  
      u'macspoof':u'true',
      u'device_a8c2dcc6-b27b-48ed-9e22-f1e1a36423aedevice_ba4d1ef4-ce41-4b08-aacb-4172828b13c9device_2149d02d-33d4-436a-8eab-2bdfd19bdd12':      u"VmDevice:{  
         id='VmDeviceId:{  
            deviceId='2149d02d-33d4-436a-8eab-2bdfd19bdd12',
            vmId='1a49b65b-9950-4b29-8c72-139d5263e11d'
         }         ', device='         unix',
         type='CHANNEL',
         bootOrder='0',
         specParams='         [  

         ]         ', address='         {  
            bus=0,
            controller=0,
            type=virtio-serial,
            port=2
         }         ', managed='         false         ', plugged='         true         ', readOnly='         false         ', deviceAlias='         channel1',
         customProperties='         [  

         ]         ', snapshotId='         null         ', logicalName='         null         ', usingScsiReservation='         false         '}", u'         device_a8c2dcc6-b27b-48ed-9e22-f1e1a36423aedevice_ba4d1ef4-ce41-4b08-aacb-4172828b13c9device_2149d02d-33d4-436a-8eab-2bdfd19bdd12device_cd207626-438b-462d-b80e-5735ec9c1a7f':         u"VmDevice:{  
            id='VmDeviceId:{  
               deviceId='cd207626-438b-462d-b80e-5735ec9c1a7f',
               vmId='1a49b65b-9950-4b29-8c72-139d5263e11d'
            }            ', device='            spicevmc',
            type='CHANNEL',
            bootOrder='0',
            specParams='            [  

            ]            ', address='            {  
               bus=0,
               controller=0,
               type=virtio-serial,
               port=3
            }            ', managed='            false            ', plugged='            true            ', readOnly='            false            ', deviceAlias='            channel2',
            customProperties='            [  

            ]            ', snapshotId='            null            ', logicalName='            null            ', usingScsiReservation='            false            '}", u'            device_a8c2dcc6-b27b-48ed-9e22-f1e1a36423ae':            u"VmDevice:{  
               id='VmDeviceId:{  
                  deviceId='a8c2dcc6-b27b-48ed-9e22-f1e1a36423ae',
                  vmId='1a49b65b-9950-4b29-8c72-139d5263e11d'
               }               ', device='               ide',
               type='CONTROLLER',
               bootOrder='0',
               specParams='               [  

               ]               ', address='               {  
                  slot=0x01,
                  bus=0x00,
                  domain=0x0000,
                  type=pci,
                  function=0x1
               }               ', managed='               false               ', plugged='               true               ', readOnly='               false               ', deviceAlias='               ide',
               customProperties='               [  

               ]               ', snapshotId='               null               ', logicalName='               null               ', usingScsiReservation='               false               '}", u'               device_a8c2dcc6-b27b-48ed-9e22-f1e1a36423aedevice_ba4d1ef4-ce41-4b08-aacb-4172828b13c9':               u"VmDevice:{  
                  id='VmDeviceId:{  
                     deviceId='ba4d1ef4-ce41-4b08-aacb-4172828b13c9',
                     vmId='1a49b65b-9950-4b29-8c72-139d5263e11d'
                  }                  ', device='                  unix',
                  type='CHANNEL',
                  bootOrder='0',
                  specParams='                  [  

                  ]                  ', address='                  {  
                     bus=0,
                     controller=0,
                     type=virtio-serial,
                     port=1
                  }                  ', managed='                  false                  ', plugged='                  true                  ', readOnly='                  false                  ', deviceAlias='                  channel0',
                  customProperties='                  [  

                  ]                  ', snapshotId='                  null                  ', logicalName='                  null                  ', usingScsiReservation='                  false                  '}"}, u'                  vmType':u'kvm',
                  u'_srcDomXML':                  u'<domain type=\'kvm\' id=\'6\'>\n   <name>proxmox-01</name>\n  <uuid>1a49b65b-9950-4b29-8c72-139d5263e11d</uuid>\n  <metadata xmlns:                  ovirt="http:                  //ovirt.org/vm/tune/1.0">\n    <ovirt:                  qos/>\n  </metadata>\n  <maxMemory slots=\'16\' unit=\'KiB\'>4294967296</maxMemory>\n  <memory unit=\'KiB\'>8388608</memory>\n  <currentMemory unit=\'KiB\'>8388608</currentMemory>\n  <vcpu placement=\'static\' current=\'4\'>16</vcpu>\n  <cputune>\n    <shares>1020</shares>\n  </cputune>\n  <numatune>\n    <memory mode=\'interleave\' nodeset=\'0\'/>\n  </numatune>\n  <resource>\n    <partition>/machine</partition>\n  </resource>\n  <sysinfo type=\'smbios\'>\n    <system>\n      <entry name=\'manufacturer\'>oVirt</entry>\n      <entry name=\'product\'>oVirt Node</entry>\n      <entry name=\'version\'>7-2.1511.el7.centos.2.10</entry>\n      <entry name=\'serial\'>271BC900-7252-11DD-8001-002185350C3C</entry>\n      <entry name=\'uuid\'>1a49b65b-9950-4b29-8c72-139d5263e11d</entry>\n    </system>\n  </sysinfo>\n  <os>\n    <type arch=\'x86_64\' machine=\'pc-i440fx-rhel7.2.0\'>hvm</type>\n    <bootmenu enable=\'yes\'/>\n    <smbios mode=\'sysinfo\'/>\n  </os>\n  <features>\n    <acpi/>\n  </features>\n  <cpu mode=\'custom\' match=\'exact\'>\n    <model fallback=\'allow\'>Penryn</model>\n    <topology sockets=\'16\' cores=\'1\' threads=\'1\'/>\n    <numa>\n      <cell id=\'0\' cpus=\'0-3\' memory=\'8388608\' unit=\'KiB\'/>\n    </numa>\n  </cpu>\n  <clock offset=\'variable\' adjustment=\'0\' basis=\'utc\'>\n    <timer name=\'rtc\' tickpolicy=\'catchup\'/>\n    <timer name=\'pit\' tickpolicy=\'delay\'/>\n    <timer name=\'hpet\' present=\'no\'/>\n  </clock>\n  <on_poweroff>destroy</on_poweroff>\n  <on_reboot>restart</on_reboot>\n  <on_crash>destroy</on_crash>\n  <devices>\n    <emulator>/usr/libexec/qemu-kvm</emulator>\n    <disk type=\'file\' device=\'cdrom\'>\n      <driver name=\'qemu\' type=\'raw\'/>\n      <source startupPolicy=\'optional\'/>\n      <backingStore/>\n      <target dev=\'hdc\' bus=\'ide\'/>\n      <readonly/>\n      <serial></serial>\n      <boot order=\'2\'/>\n      <alias name=\'ide0-1-0\'/>\n      <address type=\'drive\' controller=\'0\' bus=\'1\' target=\'0\' unit=\'0\'/>\n    </disk>\n    <disk type=\'file\' device=\'disk\' snapshot=\'no\'>\n      <driver name=\'qemu\' type=\'raw\' cache=\'none\' error_policy=\'stop\' io=\'threads\'/>\n      <source file=\'/rhev/data-center/00000001-0001-0001-0001-000000000138/4e2358ab-6b51-4d13-9e36-ac5cb271618a/images/aeae1aeb-047e-493f-a70c-f471f2e7245c/278ed347-c79a-4168-b578-1562a08ac551\'/>\n      <backingStore/>\n      <target dev=\'vda\' bus=\'virtio\'/>\n      <serial>aeae1aeb-047e-493f-a70c-f471f2e7245c</serial>\n      <boot order=\'1\'/>\n      <alias name=\'virtio-disk0\'/>\n      <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x05\' function=\'0x0\'/>\n    </disk>\n    <controller type=\'virtio-serial\' index=\'0\' ports=\'16\'>\n      <alias name=\'virtio-serial0\'/>\n      <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x04\' function=\'0x0\'/>\n    </controller>\n    <controller type=\'usb\' index=\'0\'>\n      <alias name=\'usb\'/>\n      <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x01\' function=\'0x2\'/>\n     </controller>\n    <controller type=\'pci\' index=\'0\' model=\'pci-root\'>\n      <alias name=\'pci.0\'/>\n    </controller>\n    <controller type=\'ide\' index=\'0\'>\n      <alias name=\'ide\'/>\n      <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x01\' function=\'0x1\'/>\n    </controller>\n    <interface type=\'bridge\'>\n      <mac address=\'00:1                  a:4                  a:16:01:56                  \'/>\n      <source bridge=\'207-net\'/>\n      <target dev=\'vnet0\'/>\n      <model type=\'virtio\'/>\n      <filterref filter=\'vdsm-no-mac-spoofing\'/>\n      <link state=\'up\'/>\n      <alias name=\'net0\'/>\n      <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x0\'/>\n    </interface>\n    <channel type=\'unix\'>\n      <source mode=\'bind\' path=\'/var/lib/libvirt/qemu/channels/1a49b65b-9950-4b29-8c72-139d5263e11d.com.redhat.rhevm.vdsm\'/>\n      <target type=\'virtio\' name=\'com.redhat.rhevm.vdsm\' state=\'disconnected\'/>\n      <alias name=\'channel0\'/>\n      <address type=\'virtio-serial\' controller=\'0\' bus=\'0\' port=\'1\'/>\n    </channel>\n    <channel type=\'unix\'>\n      <source mode=\'bind\' path=\'/var/lib/libvirt/qemu/channels/1a49b65b-9950-4b29-8c72-139d5263e11d.org.qemu.guest_agent.0\'/>\n      <target type=\'virtio\' name=\'org.qemu.guest_agent.0\' state=\'disconnected\'/>\n      <alias name=\'channel1\'/>\n      <address type=\'virtio-serial\' controller=\'0\' bus=\'0\' port=\'2\'/>\n    </channel>\n    <channel type=\'spicevmc\'>\n      <target type=\'virtio\' name=\'com.redhat.spice.0\' state=\'disconnected\'/>\n      <alias name=\'channel2\'/>\n      <address type=\'virtio-serial\' controller=\'0\' bus=\'0\' port=\'3\'/>\n    </channel>\n    <input type=\'mouse\' bus=\'ps2\'/>\n    <input type=\'keyboard\' bus=\'ps2\'/>\n    <graphics type=\'spice\' port=\'5900\' tlsPort=\'5901\' autoport=\'yes\' listen=\'0\' passwdValidTo=\'2016-05-18T15:01:33                  \' connected=\'disconnect\'>\n      <listen type=\'address\' address=\'0\'/>\n     </graphics>\n    <video>\n      <model type=\'qxl\' ram=\'65536\' vram=\'32768\' vgamem=\'16384\' heads=\'1\'/>\n      <alias name=\'video0\'/>\n      <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x0\'/>\n    </video>\n    <memballoon model=\'none\'>\n      <alias name=\'balloon0\'/>\n    </memballoon>\n  </devices>\n</domain>\n',
                  u'memSize':8192,
                  u'smpCoresPerSocket':u'1',
                  u'vmName':u'proxmox-01',
                  u'nice':u'0',
                  u'username':u'Unknown',
                  u'maxMemSize':4194304,
                  u'bootMenuEnable':u'true',
                  u'copyPasteEnable':u'true',
                  u'displayIp':u'0',
                  u'keyboardLayout':u'en-us',
                  u'displayPort':u'5900',
                  u'guestDiskMapping':{  

                  },
                  u'clientIp':u'',
                  u'fileTransferEnable':u'true',
                  u'nicModel':u'rtl8139,
                  pv',
                  u'elapsedTimeOffset':1662384.113245964,
                  u'kvmEnable':u'true',
                  u'devices':[  
                     {  
                        u'device':u'memballoon',
                        u'specParams':{  
                           u'model':u'none'
                        },
                        u'type':u'balloon',
                        u'alias':u'balloon0'
                     },
                     {  
                        u'alias':u'virtio-serial0',
                        u'specParams':{  

                        },
                        u'deviceId':u'c12cfd04-eab1-45ff-85b9-5156ee96aae2',
                        u'address':{  
                           u'slot':u'0x04',
                           u'bus':u'0x00',
                           u'domain':u'0x0000',
                           u'type':u'pci',
                           u'function':u'0x0'
                        },
                        u'device':u'virtio-serial',
                        u'type':u'controller'
                     },
                     {  
                        u'alias':u'video0',
                        u'specParams':{  
                           u'vram':u'32768',
                           u'heads':u'1'
                        },
                        u'deviceId':u'e85bdb50-2e29-4729-87b0-ccd5836a1a67',
                        u'address':{  
                           u'slot':u'0x02',
                           u'bus':u'0x00',
                           u'domain':u'0x0000',
                           u'type':u'pci',
                           u'function':u'0x0'
                        },
                        u'device':u'qxl',
                        u'type':u'video'
                     },
                     {  
                        u'tlsPort':u'5901',
                        u'specParams':{  
                           u'fileTransferEnable':u'true',
                           u'copyPasteEnable':u'true',
                           u'displayIp':u'0'
                        },
                        u'deviceId':u'6bfcef8d-6158-48e7-8ac8-211042f956b2',
                        u'device':u'spice',
                        u'type':u'graphics',
                        u'port':u'5900'
                     },
                     {  
                        u'nicModel':u'pv',
                        u'macAddr':                        u'00:1                        a:4                        a:16:01:56                        ', u'                        linkActive':True,
                        u'network':u'207-net',
                        u'alias':u'net0',
                        u'filter':u'vdsm-no-mac-spoofing',
                        u'specParams':{  
                           u'inbound':{  

                           },
                           u'outbound':{  

                           }
                        },
                        u'deviceId':u'8a039a7d-8ad6-4ef0-b074-29ba2d520c0a',
                        u'address':{  
                           u'slot':u'0x03',
                           u'bus':u'0x00',
                           u'domain':u'0x0000',
                           u'type':u'pci',
                           u'function':u'0x0'
                        },
                        u'device':u'bridge',
                        u'type':u'interface',
                        u'name':u'vnet0'
                     },
                     {  
                        u'index':u'2',
                        u'iface':u'ide',
                        u'name':u'hdc',
                        u'alias':u'ide0-1-0',
                        u'bootOrder':u'2',
                        u'specParams':{  
                           u'path':u''
                        },
                        u'readonly':u'True',
                        u'deviceId':u'f70d7996-3948-4d0e-8f43-7530d15fbeb3',
                        u'address':{  
                           u'bus':u'1',
                           u'controller':u'0',
                           u'type':u'drive',
                           u'target':u'0',
                           u'unit':u'0'
                        },
                        u'device':u'cdrom',
                        u'shared':u'false',
                        u'path':u'',
                        u'type':u'disk'
                     },
                     {  
                        u'poolID':u'00000001-0001-0001-0001-000000000138',
                        u'volumeInfo':{  
                           u'domainID':u'4e2358ab-6b51-4d13-9e36-ac5cb271618a',
                           u'volType':u'path',
                           u'leaseOffset':0,
                           u'volumeID':u'278ed347-c79a-4168-b578-1562a08ac551',
                           u'leasePath':                           u'/rhev/data-center/mnt/glusterSD/ovirt-node-01.nec-labs.com:_vmdata/4e2358ab-6b51-4d13-9e36-ac5cb271618a/images/aeae1aeb-047e-493f-a70c-f471f2e7245c/278ed347-c79a-4168-b578-1562a08ac551.lease',
                           u'imageID':u'aeae1aeb-047e-493f-a70c-f471f2e7245c',
                           u'path':                           u'/rhev/data-center/mnt/glusterSD/ovirt-node-01.nec-labs.com:_vmdata/4e2358ab-6b51-4d13-9e36-ac5cb271618a/images/aeae1aeb-047e-493f-a70c-f471f2e7245c/278ed347-c79a-4168-b578-1562a08ac551'
                        },
                        u'index':0,
                        u'iface':u'virtio',
                        u'apparentsize':u'536870912000',
                        u'alias':u'virtio-disk0',
                        u'imageID':u'aeae1aeb-047e-493f-a70c-f471f2e7245c',
                        u'readonly':u'False',
                        u'shared':u'false',
                        u'truesize':u'11320029184',
                        u'type':u'disk',
                        u'domainID':u'4e2358ab-6b51-4d13-9e36-ac5cb271618a',
                        u'reqsize':u'0',
                        u'format':u'raw',
                        u'deviceId':u'aeae1aeb-047e-493f-a70c-f471f2e7245c',
                        u'address':{  
                           u'slot':u'0x05',
                           u'bus':u'0x00',
                           u'domain':u'0x0000',
                           u'type':u'pci',
                           u'function':u'0x0'
                        },
                        u'device':u'disk',
                        u'path':u'/rhev/data-center/00000001-0001-0001-0001-000000000138/4e2358ab-6b51-4d13-9e36-ac5cb271618a/images/aeae1aeb-047e-493f-a70c-f471f2e7245c/278ed347-c79a-4168-b578-1562a08ac551',
                        u'propagateErrors':u'off',
                        u'optional':u'false',
                        u'name':u'vda',
                        u'bootOrder':u'1',
                        u'volumeID':u'278ed347-c79a-4168-b578-1562a08ac551',
                        u'specParams':{  

                        },
                        u'volumeChain':[  
                           {  
                              u'domainID':u'4e2358ab-6b51-4d13-9e36-ac5cb271618a',
                              u'volType':u'path',
                              u'leaseOffset':0,
                              u'volumeID':u'278ed347-c79a-4168-b578-1562a08ac551',
                              u'leasePath':                              u'/rhev/data-center/mnt/glusterSD/ovirt-node-01.nec-labs.com:_vmdata/4e2358ab-6b51-4d13-9e36-ac5cb271618a/images/aeae1aeb-047e-493f-a70c-f471f2e7245c/278ed347-c79a-4168-b578-1562a08ac551.lease',
                              u'imageID':u'aeae1aeb-047e-493f-a70c-f471f2e7245c',
                              u'path':                              u'/rhev/data-center/mnt/glusterSD/ovirt-node-01.nec-labs.com:_vmdata/4e2358ab-6b51-4d13-9e36-ac5cb271618a/images/aeae1aeb-047e-493f-a70c-f471f2e7245c/278ed347-c79a-4168-b578-1562a08ac551'
                           }
                        ]
                     },
                     {  
                        u'device':u'usb',
                        u'alias':u'usb',
                        u'type':u'controller',
                        u'address':{  
                           u'slot':u'0x01',
                           u'bus':u'0x00',
                           u'domain':u'0x0000',
                           u'type':u'pci',
                           u'function':u'0x2'
                        }
                     },
                     {  
                        u'device':u'ide',
                        u'alias':u'ide',
                        u'type':u'controller',
                        u'address':{  
                           u'slot':u'0x01',
                           u'bus':u'0x00',
                           u'domain':u'0x0000',
                           u'type':u'pci',
                           u'function':u'0x1'
                        }
                     },
                     {  
                        u'device':u'unix',
                        u'alias':u'channel0',
                        u'type':u'channel',
                        u'address':{  
                           u'bus':u'0',
                           u'controller':u'0',
                           u'type':u'virtio-serial',
                           u'port':u'1'
                        }
                     },
                     {  
                        u'device':u'unix',
                        u'alias':u'channel1',
                        u'type':u'channel',
                        u'address':{  
                           u'bus':u'0',
                           u'controller':u'0',
                           u'type':u'virtio-serial',
                           u'port':u'2'
                        }
                     },
                     {  
                        u'device':u'spicevmc',
                        u'alias':u'channel2',
                        u'type':u'channel',
                        u'address':{  
                           u'bus':u'0',
                           u'controller':u'0',
                           u'type':u'virtio-serial',
                           u'port':u'3'
                        }
                     }
                  ],
                  u'display':u'qxl',
                  u'status':u'Up',
                  u'timeOffset':u'0',
                  u'maxVCpus':u'16',
                  u'guestIPs':u'',
                  u'statusTime':u'12906968250',
                  u'maxMemSlots':16
               }

-----Original Message-----
From: Yedidyah Bar David [mailto:didi@redhat.com] 
Sent: Tuesday, June 07, 2016 2:03 AM
To: Will Dennis
Cc: users@ovirt.org; Michal Skrivanek
Subject: Re: [ovirt-users] virt-in-virt problem: DHCP failing for a container in a oVirt VM

On Tue, Jun 7, 2016 at 12:04 AM, Will Dennis <wdennis@nec-labs.com> wrote:
...
Sorry, got away from this for a while attending to other work, but back at it now... Anyone have any ideas on how I may further troubleshoot this issue?
Edward later asked you to post the domxml.
You can find it with 'virsh dumpxml vm' or search vdsm logs.

Best,
-- 
Didi