[ovirt-users] Help! My hosted engine lost his nic!
Simone Tiraboschi
stirabos at redhat.com
Fri Dec 2 16:14:28 UTC 2016
On Fri, Dec 2, 2016 at 5:03 PM, Cristian Mammoli <c.mammoli at apra.it> wrote:
> Out of desperation i added the missing section int the OVF I extracted and
> "tarred" it back to the storage domain. The nic now is back...
>
Unfortunately I've to tell you that the engine will periodically rewrite
the OVF_STORE based on the configuration of the VM in its DB so in
principle also this one is just a temporary solution.
>
> I don't know if the query you suggested still make sense now... :/
> Anyway I don't have any nic in the gui even now
>
> [root at ovengine ~]# sudo -u postgres psql engine -c "select * from
> vm_device where type='interface' and vm_id='497f5e4a-0c76-441a-b72e
> -724d7092d07e'"
> could not change directory to "/root"
> device_id | vm_id | type
> | device | address |
> boot_order | spec_params | is_managed | is_plugged | is_readonly |
> _create_date | _update_date | alias | custom_properties | snaps
> hot_id | logical_name | is_using_scsi_reservation
> --------------------------------------+---------------------
> -----------------+-----------+--------+---------------------
> -----------------------------------------+------------+-----
> --------+------------+------------+-------------+-----------
> --------------------+--------------+-------+-------------------+------
> -------+--------------+---------------------------
> 6207e0d7-4dc9-406d-ab99-3facf45788f4 | 497f5e4a-0c76-441a-b72e-724d7092d07e
> | interface | bridge | {slot=0x04, bus=0x00, domain=0x0000, type=pci,
> function=0x0} | 0 | { } | f | t | f
> | 2016-12-02 01:46:42.999885+01 | | net0 |
> |
> | | f
> (1 row)
>
But this seams fine now.
Adding Roy on this.
>
> [root at ovengine ~]# sudo -u postgres psql engine -c "select * from vms
> where vm_guid='497f5e4a-0c76-441a-b72e-724d7092d07e'"
> could not change directory to "/root"
> vm_name | mem_size_mb | num_of_io_threads | nice_level | cpu_shares
> | vmt_guid | os | description |
> free_text_comment | vds_group_id | creation_date
> | auto_startup | is_stateless | is_smartcard_enabled |
> is_delete_protected | sso_method |
> dedicated_vm_for_vds | fail_back | default_boot_sequence | vm_type |
> vm_pool_spice_proxy | vds_group_name | transparent_hugepages |
> trusted_service | storage_pool_id | storage_pool_name
> | vds_group_description | vds_group_spice_proxy | vmt_name |
> vmt_mem_size_mb | vmt_os |
> vmt_creation_date | vmt_child_count | vmt_num_of_sockets |
> vmt_cpu_per_socket | vmt_threads_per_cpu | vmt_num_of_cpus |
> vmt_description | status | vm_ip | vm_ip_inet_array | vm_host
> | vm_pid | last_start_time | guest_cur_user_name | console_cur_user_name |
> guest_os | co
> nsole_user_id | guest_agent_nics_hash | run_on_vds |
> migrating_to_vds | app_list | vm_pool_name | vm_pool_id |
> vm_guid | num_of_monitors | single_qxl_pci |
> allow_console_reconnect | is_initialized | num_of_sockets | cpu_per_socket
> | threads_per_cpu | usb_policy | acpi_enable | session | num_of_cpus |
> quota_id | quota_name | quota_enforcement_type | kvm_enable | boot_sequence
> | utc_diff | last_
> vds_run_on | client_ip | guest_requested_memory | time_zone | cpu_user |
> cpu_sys | memory_usage_history |
> cpu_usage_history
> | network_usage_history | elapsed_time |
> usage_network_percent | disks_usage |
> usage_mem_percent | migration_progress_percent | usage_cpu_percent |
> run_on_vds_name | vds_group_cpu_name | de
> fault_display_type | priority | iso_path | origin |
> vds_group_compatibility_version | initrd_url | kernel_url | kernel_params
> | pause_status | exit_message | exit_status | migration_support |
> predefined_properties | userdefined_properties | min_allocated_mem |
> hash | cpu_pinning | db_generatio
> n | host_cpu_flags | tunnel_migration | vnc_keyboard_layout |
> is_run_and_pause | created_by_user_id | last_watchdog_event |
> last_watchdog_action | is_run_once | vm_fqdn | cpu_name |
> emulated_machine | current_cd | reason | exit_reason | instance_type_id |
> image_type_id | architecture | original_temp
> late_id | original_template_name | last_stop_time |
> migration_downtime | template_version_number | serial_number_policy |
> custom_serial_number | is_boot_menu_enabled | guest_cpu_count |
> next_run_config_exists | numatune_mode | is_spice_file_transfer_enabled |
> is_spice_copy_paste_enabled |
> cpu_profile_id | is_auto_converge | is_migrate_compressed |
> custom_emulated_machine | custom_cpu_name | spice_port | spice_tls_port |
> spice_ip | vnc_port | vnc_ip | guest_agent_status |
> guest_mem_buffered | guest_mem_cached | guest_mem_free |
> small_icon_id |
> large_icon_id | provider_id | console_disconnect_action |
> guest_timezone_offset | guest_timezone_name | guestos_arch |
> guestos_codename | guestos_distribution | guestos_kernel_version |
> guestos_type | guestos_version
> --------------+-------------+-------------------+-----------
> -+------------+--------------------------------------+----+-
> ------------+-------------------+---------------------------
> -----------+----------------------------+--------------+----
> ----------+----------------------+---------------------+-------------+
> ----------------------+-----------+-----------------------+-
> --------+---------------------+----------------+------------
> -----------+-----------------+------------------------------
> --------+-------------------+----------------------------+--
> -----------------------------+----------+-----------------+--------+--
> ----------------------+-----------------+-------------------
> -+--------------------+---------------------+---------------
> --+-----------------+--------+---------------+--------------
> ----+-------------------+--------+-----------------+--------
> -------------+-----------------------+----------------------------+---
> --------------+-----------------------+---------------------
> -----------------+------------------+-----------------------
> ------------------------------------------------------------
> ------------------------------------------------------------
> ------------------------------------+--------------+------------+-----
> ---------------------------------+-----------------+--------
> --------+-------------------------+----------------+--------
> --------+----------------+-----------------+------------+---
> ----------+---------+-------------+----------+------------+-
> -----------------------+------------+---------------+----------+------
> -----------+-----------+------------------------+-----------
> +----------+---------+--------------------------------------
> ------------------------------------------------------------
> -----------------------+------------------------------------
> ----------------------------------------------------------------------
> +-----------------------------------------------------------
> ----------------------+--------------+----------------------
> -+----------------------------------------------------------
> ------------+-------------------+---------------------------
> -+-------------------+-----------------+--------------------------+---
> -------------------+----------+----------+--------+---------
> ------------------------+------------+------------+---------
> ------+--------------+--------------+-------------+---------
> ----------+-----------------------+------------------------+
> -------------------+----------------------+-------------+-------------
> --+----------------+------------------+---------------------
> +------------------+--------------------+-------------------
> --+----------------------+-------------+-------------------+
> ----------+------------------+------------+--------+--------
> -----+------------------+---------------+--------------+--------------
> --------+------------------------+--------------------------
> --+--------------------+-------------------------+----------
> ------------+----------------------+----------------------+-
> ----------------+------------------------+---------------+--
> ------------------------------+-----------------------------+---------
> -----------------------------+------------------+-----------
> ------------+-------------------------+-----------------+---
> ---------+----------------+----------+----------+-----------
> -----+--------------------+--------------------+------------
> ------+----------------+--------------------------------------+-------
> -------------------------------+-------------+--------------
> -------------+-----------------------+---------------------+
> --------------+------------------+----------------------+---
> -------------------------+--------------+-----------------
> HostedEngine | 6144 | 0 | 0 | 0
> | 00000000-0000-0000-0000-000000000000 | 0 | |
> | 00000002-0002-0002-0002-0000000000ca | 2015-11-03 16:54:06.536+01
> | f | f | f | f
> | guest_agent |
> | f | 0 | 1 |
> | Default | t | f |
> 00000001-0001-0001-0001-000000000296 | Default | The default
> server cluster | http://ovengine.omme.net:3128 | Blank |
> 1024 | 0 | 2
> 008-03-31 23:00:00+02 | 11 | 1 |
> 1 | 1 | 1 | Blank template | 1
> | 192.168.42.27 | {192.168.42.27} | ovengine.omme.net | |
> | root | | 3.10.0-327.36.3.el7.x86_64 |
> | 1234360272 | 572aa833-37fb-4c4b-9576-9d367def2d04
> | | kernel-3.10.0-229.14.1.el7,ker
> nel-3.10.0-327.36.3.el7,cloud-init-0.7.5-10.el7.centos.1,
> kernel-3.10.0-229.20.1.el7,kernel-3.10.0-327.4.4.el7,
> ovirt-guest-agent-common-1.0.12-3.el7 | | | 497f
> 5e4a-0c76-441a-b72e-724d7092d07e | 1 | f | f
> | f | 2 | 1 | 1
> | 1 | t | 0 | 2 | | |
> 0 | t | 0 | 0 |
> | | | Etc/GMT | 16 | 1
> | 37,37,37,37,37,37,38,38,37,37,38,38,38,38,38,38,38,38,38,38,
> 37,37,37,37,37,37,37,37,37,37,37,37,37,37,37,37,37,37,37,37 |
> 23,16,16,34,22,25,26,22,12,11,12,25,11,13,12,14,12,11,8,12,9
> ,8,7,13,10,10,9,9,8,8,9,13,9,5,6,9,8,11,11,8
> | 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
> | 55056 | 0 | [{"path":"/","total":"84416028
> 672","used":"7691350016","fs":"ext4"}] | 37 |
> 0 | 8 | kvm02 | Intel SandyBridge
> Family |
> 0 | 1 | | 6 | 3.6
> | | | | 5 | |
> 0 | 1 | | | 6144
> | -1125579430873805323 | | 2
> 2 | f | | | f
> | | | | f |
> ovengine.omme.net | | | | |
> -1 | | | 1 |
> | | 2016-11-29 11:37:07.501+01 |
> 1000 | 1 | |
> | f | 2 | f | interleave | t
> | t | 0000000e
> -000e-000e-000e-000000000039 | | | pc
> | | | | | 5900 |
> kvm02.omme.net | 0 | 102080 |
> 846088 | 3916916 | 1807edab-f180-4268-b364-c6cc9b65b602 | 0330e1
> 6e-25c4-458d-8741-17c92d160d6a | | LOCK_SCREEN
> | 60 | Europe/Rome | 1 | Core
> | centos | 3.10.0-327.36.3.el7.x86_64 | Linux |
> 7.2.1511
> (1 row)
>
> [root at ovengine ~]#
>
>
> Il 01/12/2016 18:11, Simone Tiraboschi ha scritto:
>
>>
>>
>> On Thu, Dec 1, 2016 at 5:16 PM, Cristian Mammoli <c.mammoli at apra.it
>> <mailto:c.mammoli at apra.it>> wrote:
>>
>> Here it is:
>> http://cloud.apra.it/index.php/s/4cdcde8cafdb7a1c2c2374b02dce118e
>> <http://cloud.apra.it/index.php/s/4cdcde8cafdb7a1c2c2374b02dce118e>
>>
>> I tarred all the agent.log on both servers.
>>
>> The engine was running on kvm01 and got shutdown on kvm01 around
>> 10:35 AM on 29 November. But I think that's not the problem, it is
>> supposed to shut down if the host can't reach the gateway.
>> Probably the nic problem was already there but got triggered on reboot
>>
>> Btw I kept digging: I extracted the ovf from which vm.conf is
>> generated:
>>
>> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>> OVF_STORE volume path:
>> /rhev/data-center/mnt/blockSD/2c3585cc-b7bc-4881-85b3-aa6514
>> 991a26/images/9c5e2121-f1a3-4886-964c-c74fdfbbb3c1/ff76505
>> 5-09c5-4b05-9cc7-5277b15c5d08
>>
>> # tar xvf
>> /rhev/data-center/mnt/blockSD/2c3585cc-b7bc-4881-85b3-aa6514
>> 991a26/images/9c5e2121-f1a3-4886-964c-c74fdfbbb3c1/ff76505
>> 5-09c5-4b05-9cc7-5277b15c5d08
>> 497f5e4a-0c76-441a-b72e-724d7092d07e.ovf
>> info.json
>>
>> In the ovf file there is no Nic section...
>>
>>
>> Ciao Cristian,
>> do you see any interface for the engine VM in the engine admin portal?
>>
>> Could you please execute this on the engine VM and share its output?
>> sudo -u postgres psql engine -c "select * from vm_device where
>> type='interface' and vm_id='497f5e4a-0c76-441a-b72e-724d7092d07e'"
>> sudo -u postgres psql engine -c "select * from vms where
>> vm_guid='497f5e4a-0c76-441a-b72e-724d7092d07e'"
>>
>> thanks
>>
>> I uploaded the ovf on the same share as the logs
>>
>> Ty
>>
>>
>> Il 01/12/2016 15:26, Yedidyah Bar David ha scritto:
>>
>> On Thu, Dec 1, 2016 at 1:08 PM, Cristian Mammoli
>> <c.mammoli at apra.it <mailto:c.mammoli at apra.it>> wrote:
>>
>> Hi, I upgraded an oVirt installation a month ago to the
>> latest 3.6.7. Before
>> it was 3.6.0 if I remember correctly.
>> Everything went fine so far for a month or so.
>>
>> A couple of days ago the the default gateway got rebooted
>> and the physical
>> server hosting the HE decided to shut down the vm because
>> it could not ping
>> the gateway.
>> The other host restarted the hevm but it now has *no nic*.
>> As a workaround I attached a virtio nic via virsh but
>> every time the vm gets
>> restarted the nic get lost
>>
>> After a bit of troubleshooting and digging this is what I
>> found:
>>
>> This is the /var/run/ovirt-hosted-engine-ha/vm.conf which,
>> as far as I
>> understand, gets extracted from the HE storage domain
>>
>> emulatedMachine=pc
>> vmId=497f5e4a-0c76-441a-b72e-724d7092d07e
>> smp=2
>> memSize=6144
>> spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback
>> ,srecord,ssmartcard,susbredir
>> vmName=HostedEngine
>> display=vnc
>> devices={index:0,iface:virtio,format:raw,bootOrder:1,address
>> :{slot:0x06,bus:0x00,domain:0x0000,type:pci,function:0x0},
>> volumeID:bb3218ba-cbe9-4cd0-b50b-931deae992f7,imageID:d65b82e2-2ad1-
>> 4f4f-bfad-0277c37f2808,readonly:false,domainID:2c3585cc-
>> b7bc-4881-85b3-aa6514991a26,deviceId:d65b82e2-2ad1-4f4f-
>> bfad-0277c37f2808,poolID:00000000-0000-0000-0000-
>> 000000000000,device:disk,
>> shared:exclusive,propagateErrors:off,type:disk}
>> devices={index:2,iface:ide,shared:false,readonly:true,device
>> Id:8c3179ac-b322-4f5c-9449-c52e3665e0ae,address:{controll
>> er:0,target:0,unit:0,bus:1,type:drive},device:cdrom,path:,type:disk}
>> devices={device:cirrus,alias:video0,type:video,deviceId:a994
>> 68b6-02d4-4a77-8f94-e5df806030f6,address:{slot:0x02,bus:
>> 0x00,domain:0x0000,type:pci,function:0x0}}
>> devices={device:virtio-serial,type:controller,deviceId:b7580
>> 676-19fb-462f-a61e-677b65ad920a,address:{slot:0x03,bus:0x00,
>> domain:0x0000,type:pci,function:0x0}}
>> devices={device:usb,type:controller,deviceId:c63092b3-7bd8-
>> 4b54-bcd3-51f34dce478a,address:{slot:0x01,bus:0x00,domain:
>> 0x0000,type:pci,function:0x2}}
>> devices={device:ide,type:controller,deviceId:c77c2c01-6ccc-
>> 404b-b8d6-5a7f0631a52f,address:{slot:0x01,bus:0x00,domain:
>> 0x0000,type:pci,function:0x1}}
>>
>> As you can see there is no nic, and there is no nic in the
>> qemu-kvm
>> command-line:
>> qemu 23290 1 14 00:23 ? 01:44:26
>> /usr/libexec/qemu-kvm -name
>> HostedEngine -S -machine
>> pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu
>> qemu64,-svm -m 6144 -realtime mlock=off -s
>> mp 2,sockets=2,cores=1,threads=1 -uuid
>> 497f5e4a-0c76-441a-b72e-724d7092d07e
>> -smbios type=1,manufacturer=oVirt,product=oVirt
>> Node,version=7-2.1511.el7.centos.2.10,serial=4C4C4544-004B-
>> 571
>> 0-8044-B9C04F5A3732,uuid=497f5e4a-0c76-441a-b72e-724d7092d07e
>> -no-user-config -nodefaults -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-Host
>> edEngine/monitor.sock,serve
>> r,nowait -mon chardev=charmonitor,id=monitor,mode=control
>> -rtc
>> base=2016-11-30T23:23:26,driftfix=slew -global
>> kvm-pit.lost_tick_policy=discard -no-hpet -no-reboot -boot
>> strict=on -device
>> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
>> virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,
>> addr=0x3
>> -drive
>> file=/var/run/vdsm/storage/2c3585cc-b7bc-4881-85b3-aa6514
>> 991a26/d65b82e2-2ad1-4f4f-bfad-0277c37f2808/bb3218ba-cbe9-
>> 4cd0-b50b-931deae992f7,if=none,id=drive-virtio-disk0,for
>> mat=raw,serial=d65b82e2-2ad1-4f4f-bfad-0277c37f2808,cache=none,werror=st
>> op,rerror=stop,aio=native -device
>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virti
>> o-disk0,id=virtio-disk0,bootindex=1
>> -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw
>> -device
>> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>> -chardev
>> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4
>> 97f5e4a-0c76-441a-b72e-724d7092d07e.com.redhat.rhevm
>> .vdsm,server,nowait -device
>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel
>> 0,id=channel0,name=com.redhat.rhevm.vdsm
>> -chardev socket,id=charchannel1,path=/var/lib/libvirt/qem
>> u/channels/497f5e4a-0c76-441a-b72e-724d7092d07e.org.qemu.gue
>> st_agent.0,server,nowait
>> -device
>> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel
>> 1,id=channel1,name=org.qemu.guest
>> _agent.0 -chardev
>> socket,id=charchannel2,path=/var/lib/libvirt/qemu/channels/4
>> 97f5e4a-0c76-441a-b72e-724d7092d07e.org.ovirt.hosted-engine-
>> setup.0,server,nowait
>> -device virtserialport,bus
>> =virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,
>> name=org.ovirt.hosted-engine-setup.0
>> -vnc 0:0,password -device
>> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -msg
>> timestamp=on
>>
>> I extracted the vm.conf from the storage domain and the
>> nic is there:
>> mId=497f5e4a-0c76-441a-b72e-724d7092d07e
>> memSize=6144
>> display=vnc
>> devices={index:2,iface:ide,address:{ controller:0,
>> target:0,unit:0, bus:1,
>> type:drive},specParams:{},readonly:true,deviceId:857b98b3-
>> cf43-4c2d-8061-e7f105234a65,path:,device:cdrom,shared
>> :false,type:disk}
>> devices={index:0,iface:virtio,format:raw,poolID:00000000-000
>> 0-0000-0000-000000000000,volumeID:bb3218ba-cbe9-4cd0-b50b-
>> 931deae992f7,imageID:d65b82e2-2ad1-4f4f-bfad-0277c37f2808,specParams
>> :{},readonly:false,domainID:2c3585cc-b7bc-4881-85b3-aa651499
>> 1a26,optional:false,deviceId:d65b82e2-2ad1-4f4f-bfad-
>> 0277c37f2808,address:{bus:0x00,
>> slot:0x06, domain:0x0000, type:pci, funct
>> ion:0x0},device:disk,shared:exclusive,propagateErrors:off,ty
>> pe:disk,bootOrder:1}
>> devices={device:scsi,model:virtio-scsi,type:controller}
>> devices={nicModel:pv,macAddr:00:16:3e:7d:d8:27,linkActive:tr
>> ue,network:ovirtmgmt,filter:vdsm-no-mac-spoofing,specParams:
>> {},deviceId:5be8a089-9f51-46dc-a8bd-28422985aa35,address:{bus:0x00
>> , slot:0x03, domain:0x0000, type:pci,
>> function:0x0},device:bridge,type:interface}
>> devices={device:console,specParams:{},type:console,deviceId:
>> 1644f556-a4ff-4c93-8945-5aa165de2a85,alias:console0}
>> vmName=HostedEngine
>> spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback
>> ,srecord,ssmartcard,susbredir
>> smp=2
>> cpuType=SandyBridge
>> emulatedMachine=pc
>>
>> The local vm.conf gets continuosly overwritten but for
>> some reason the nic
>> line gets lost in the process.
>>
>> Can you please check/share
>> /var/log/ovirt-hosted-engine-ha/agent.log?
>> Preferably all of it (including backups)? Thanks.
>>
>>
>> -- Mammoli Cristian
>> System administrator
>> T. +39 0731 22911 <tel:%2B39%200731%2022911>
>> Via Brodolini 6 | 60035 Jesi (an)
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org <mailto:Users at ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users
>> <http://lists.ovirt.org/mailman/listinfo/users>
>>
>>
>>
> --
> Mammoli Cristian
> System administrator
> T. +39 0731 22911
> Via Brodolini 6 | 60035 Jesi (an)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161202/4215affe/attachment-0001.html>
More information about the Users
mailing list