8:15 p.m.
Fedora 17 RPMs, hosts complets install and reboots, but never connects
back to engine. I just see thousands of the following in the logs:
----
MainThread::INFO::2012-06-21 13:07:26,304::vdsm::71::vds::(run) I am the
actual vdsm 4.9-0
MainThread::ERROR::2012-06-21 13:07:26,656::vdsm::74::vds::(run) Traceback
(most recent call last):
File "/usr/share/vdsm/vdsm", line 72, in run
serve_clients(log)
File "/usr/share/vdsm/vdsm", line 40, in serve_clients
cif = clientIF.clientIF(log)
File "/usr/share/vdsm/clientIF.py", line 115, in __init__
self._createLibvirtNetworks()
File "/usr/share/vdsm/clientIF.py", line 152, in _createLibvirtNetworks
nf = netinfo.NetInfo()
File "/usr/share/vdsm/netinfo.py", line 268, in __init__
_netinfo = get()
File "/usr/share/vdsm/netinfo.py", line 215, in get
for bridge in bridges() ])
KeyError: 'ovirtmgmt'
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::76::vds::(run) VDSM main
thread ended. Waiting for 1 other threads...
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
<_MainThread(MainThread, started 140487047427904)>
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
<Thread(libvirtEventLoop, started daemon 140486974248704)>
----
ovirtmgmt is up and running:
-bash-4.2# brctl show
bridge name bridge id STP enabled interfaces
ovirtmgmt 8000.a0369f0a15f8 no bond0
<>
Nathan Stratton
nathan at robotics.net
http://www.robotics.net
2:43 a.m.
Is there a way to manually test and debug VDSM?
<>
Nathan Stratton
nathan at robotics.net
http://www.robotics.net
On Thu, 21 Jun 2012, Nathan Stratton wrote:
Fedora 17 RPMs, hosts complets install and reboots, but never connects back
to engine. I just see thousands of the following in the logs:
----
MainThread::INFO::2012-06-21 13:07:26,304::vdsm::71::vds::(run) I am the
actual vdsm 4.9-0
MainThread::ERROR::2012-06-21 13:07:26,656::vdsm::74::vds::(run) Traceback
(most recent call last):
File "/usr/share/vdsm/vdsm", line 72, in run
serve_clients(log)
File "/usr/share/vdsm/vdsm", line 40, in serve_clients
cif = clientIF.clientIF(log)
File "/usr/share/vdsm/clientIF.py", line 115, in __init__
self._createLibvirtNetworks()
File "/usr/share/vdsm/clientIF.py", line 152, in _createLibvirtNetworks
nf = netinfo.NetInfo()
File "/usr/share/vdsm/netinfo.py", line 268, in __init__
_netinfo = get()
File "/usr/share/vdsm/netinfo.py", line 215, in get
for bridge in bridges() ])
KeyError: 'ovirtmgmt'
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::76::vds::(run) VDSM main
thread ended. Waiting for 1 other threads...
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
<_MainThread(MainThread, started 140487047427904)>
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
<Thread(libvirtEventLoop, started daemon 140486974248704)>
----
ovirtmgmt is up and running:
-bash-4.2# brctl show
bridge name bridge id STP enabled interfaces
ovirtmgmt 8000.a0369f0a15f8 no bond0
> <>
Nathan Stratton
nathan at robotics.net
http://www.robotics.net
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:41 p.m.
On Thu, Jun 21, 2012 at 12:15:25PM -0500, Nathan Stratton wrote:
Fedora 17 RPMs, hosts complets install and reboots, but never
connects back to engine. I just see thousands of the following in
the logs:
----
MainThread::INFO::2012-06-21 13:07:26,304::vdsm::71::vds::(run) I am
the actual vdsm 4.9-0
MainThread::ERROR::2012-06-21 13:07:26,656::vdsm::74::vds::(run)
Traceback (most recent call last):
File "/usr/share/vdsm/vdsm", line 72, in run
serve_clients(log)
File "/usr/share/vdsm/vdsm", line 40, in serve_clients
cif = clientIF.clientIF(log)
File "/usr/share/vdsm/clientIF.py", line 115, in __init__
self._createLibvirtNetworks()
File "/usr/share/vdsm/clientIF.py", line 152, in _createLibvirtNetworks
nf = netinfo.NetInfo()
File "/usr/share/vdsm/netinfo.py", line 268, in __init__
_netinfo = get()
File "/usr/share/vdsm/netinfo.py", line 215, in get
for bridge in bridges() ])
KeyError: 'ovirtmgmt'
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::76::vds::(run) VDSM
main thread ended. Waiting for 1 other threads...
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
<_MainThread(MainThread, started 140487047427904)>
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
<Thread(libvirtEventLoop, started daemon 140486974248704)>
----
ovirtmgmt is up and running:
-bash-4.2# brctl show
bridge name bridge id STP enabled interfaces
ovirtmgmt 8000.a0369f0a15f8 no bond0
Which version of vdsm is installed on your host?
What does `rpm -V vdsm` say?
File "/usr/share/vdsm/netinfo.py", line 215, in get
for bridge in bridges() ])
was removed long ago. I suspect that you are experiencing the pains of
f17's ifconfig changed output
Dan.
5:16 p.m.
Hi Nathan,
On 06/21/2012 01:15 PM, Nathan Stratton wrote:
Fedora 17 RPMs, hosts complets install and reboots, but never connects
back to engine. I just see thousands of the following in the logs:
----
MainThread::INFO::2012-06-21 13:07:26,304::vdsm::71::vds::(run) I am
the actual vdsm 4.9-0
4.9-0? Looks like a previous version of vdsm which we had a network
error. Can you please try 4.10.0 ?
MainThread::ERROR::2012-06-21 13:07:26,656::vdsm::74::vds::(run)
Traceback (most recent call last):
File "/usr/share/vdsm/vdsm", line 72, in run
serve_clients(log)
File "/usr/share/vdsm/vdsm", line 40, in serve_clients
cif = clientIF.clientIF(log)
File "/usr/share/vdsm/clientIF.py", line 115, in __init__
self._createLibvirtNetworks()
File "/usr/share/vdsm/clientIF.py", line 152, in _createLibvirtNetworks
nf = netinfo.NetInfo()
File "/usr/share/vdsm/netinfo.py", line 268, in __init__
_netinfo = get()
File "/usr/share/vdsm/netinfo.py", line 215, in get
for bridge in bridges() ])
KeyError: 'ovirtmgmt'
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::76::vds::(run) VDSM
main thread ended. Waiting for 1 other threads...
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
<_MainThread(MainThread, started 140487047427904)>
MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
<Thread(libvirtEventLoop, started daemon 140486974248704)>
----
ovirtmgmt is up and running:
-bash-4.2# brctl show
bridge name bridge id STP enabled interfaces
ovirtmgmt 8000.a0369f0a15f8 no bond0
> <>
Nathan Stratton
nathan at robotics.net
http://www.robotics.net
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Cheers
Douglas
10:57 p.m.
Thanks! That was it, now I think I have some issue with sudo or someting.
If I run vdsm as root it connects to the engine and the host comes up, but
if it is run normally I get:
MainThread::INFO::2012-06-22 15:46:50,424::vdsm::70::vds::(run) I am the
actual vdsm 4.10-1
MainThread::DEBUG::2012-06-22
15:46:50,675::resourceManager::379::ResourceManager::(registerNamespace)
Registering namespace 'Storage'
MainThread::DEBUG::2012-06-22
15:46:50,675::threadPool::45::Misc.ThreadPool::(__init__) Enter -
numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
MainThread::DEBUG::2012-06-22
15:46:50,690::__init__::1249::Storage.Misc.excCmd::(_log) '/usr/bin/sudo
-n /usr/bin/kill -9 2106' (cwd None)
MainThread::DEBUG::2012-06-22
15:46:50,696::supervdsm::103::SuperVdsmProxy::(_killSupervdsm) Could not
kill old Super Vdsm Permission denied
MainThread::DEBUG::2012-06-22
15:46:50,697::supervdsm::91::SuperVdsmProxy::(_launchSupervdsm) Launching
Super Vdsm
MainThread::DEBUG::2012-06-22
15:46:50,697::__init__::1249::Storage.Misc.excCmd::(_log) '/usr/bin/sudo
-n /usr/bin/python /usr/share/vdsm/supervdsmServer.py
5b882e55-ee35-4daa-86ab-be408d99e8b3 3032' (cwd None)
MainThread::ERROR::2012-06-22 15:46:50,702::clientIF::210::vds::(_initIRS)
Error initializing IRS
Traceback (most recent call last):
File "/usr/share/vdsm/clientIF.py", line 208, in _initIRS
self.irs = Dispatcher(HSM())
File "/usr/share/vdsm/storage/hsm.py", line 296, in __init__
if not multipath.isEnabled():
File "/usr/share/vdsm/storage/multipath.py", line 85, in isEnabled
svdsm = supervdsm.getProxy()
File "/usr/share/vdsm/supervdsm.py", line 135, in getProxy
_g_singletonSupervdsmInstance = SuperVdsmProxy()
File "/usr/share/vdsm/supervdsm.py", line 82, in __init__
self._restartSupervdsm()
File "/usr/share/vdsm/supervdsm.py", line 123, in _restartSupervdsm
self._launchSupervdsm()
File "/usr/share/vdsm/supervdsm.py", line 94, in _launchSupervdsm
misc.execCmd(superVdsmCmd, sync=False, sudo=True)
File "/usr/share/vdsm/storage/misc.py", line 199, in execCmd
p = BetterPopen(command, close_fds=True, cwd=cwd, env=env)
File "/usr/lib/python2.7/site-packages/vdsm/betterPopen/__init__.py",
line 46, in __init__
stderr=PIPE)
File "/usr/lib64/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/site-packages/vdsm/betterPopen/__init__.py",
line 60, in _execute_child
cwd, env)
OSError: Permission denied
MainThread::DEBUG::2012-06-22
15:46:50,712::__init__::1249::Storage.Misc.excCmd::(_log) '/usr/bin/pgrep
-xf ksmd' (cwd None)
MainThread::DEBUG::2012-06-22
15:46:50,725::__init__::1249::Storage.Misc.excCmd::(_log) SUCCESS: <err> =
''; <rc> = 0
MainThread::INFO::2012-06-22
15:46:50,726::vmChannels::139::vds::(settimeout) Setting channels' timeout
to 30 seconds.
VM Channels Listener::INFO::2012-06-22
15:46:50,728::vmChannels::127::vds::(run) Starting VM channels listener
thread.
MainThread::ERROR::2012-06-22
15:46:50,731::clientIF::151::vds::(_prepareBindings) Unable to load the
rest server module. Please make sure it is installed.
MainThread::INFO::2012-06-22 15:55:12,339::vmChannels::135::vds::(stop) VM
channels listener was stopped.
MainThread::INFO::2012-06-22 15:55:12,339::vdsm::75::vds::(run) VDSM main
thread ended. Waiting for 15 other threads...
MainThread::INFO::2012-06-22 15:55:12,340::vdsm::78::vds::(run)
<_MainThread(MainThread, started 140294687758144)>
MainThread::INFO::2012-06-22 15:55:12,340::vdsm::78::vds::(run)
<Thread(libvirtEventLoop, started daemon 140294603454208)>
MainThread::INFO::2012-06-22 15:55:12,340::vdsm::78::vds::(run)
<WorkerThread(Thread-8, started daemon 140293965870848)>
MainThread::INFO::2012-06-22 15:55:12,340::vdsm::78::vds::(run)
<WorkerThread(Thread-5, started daemon 140294469170944)>
MainThread::INFO::2012-06-22 15:55:12,340::vdsm::78::vds::(run)
<WorkerThread(Thread-4, started daemon 140294477563648)>
MainThread::INFO::2012-06-22 15:55:12,340::vdsm::78::vds::(run)
<WorkerThread(Thread-10, started daemon 140293949085440)>
MainThread::INFO::2012-06-22 15:55:12,340::vdsm::78::vds::(run)
<WorkerThread(Thread-3, started daemon 140294485956352)>
MainThread::INFO::2012-06-22 15:55:12,340::vdsm::78::vds::(run)
<Thread(Thread-11, started daemon 140293940692736)>
MainThread::INFO::2012-06-22 15:55:12,340::vdsm::78::vds::(run)
<WorkerThread(Thread-9, started daemon 140293957478144)>
MainThread::INFO::2012-06-22 15:55:12,341::vmChannels::135::vds::(stop) VM
channels listener was stopped.
MainThread::INFO::2012-06-22 15:55:12,341::vdsm::78::vds::(run)
<Listener(VM Channels Listener, started daemon 140293428999936)>
MainThread::INFO::2012-06-22 15:55:12,341::vdsm::78::vds::(run)
<WorkerThread(Thread-1, started daemon 140294502741760)>
MainThread::INFO::2012-06-22 15:55:12,341::vdsm::78::vds::(run)
<KsmMonitorThread(KsmMonitor, started daemon 140293923907328)>
MainThread::INFO::2012-06-22 15:55:12,341::vdsm::78::vds::(run)
<WorkerThread(Thread-6, started daemon 140294460778240)>
MainThread::INFO::2012-06-22 15:55:12,341::vdsm::78::vds::(run)
<WorkerThread(Thread-7, started daemon 140294452385536)>
MainThread::INFO::2012-06-22 15:55:12,341::vdsm::78::vds::(run)
<WorkerThread(Thread-2, started daemon 140294494349056)>
MainThread::INFO::2012-06-22 15:55:19,304::vdsm::70::vds::(run) I am the
actual vdsm 4.10-1
MainThread::DEBUG::2012-06-22
15:55:19,557::resourceManager::379::ResourceManager::(registerNamespace)
Registering namespace 'Storage'
MainThread::DEBUG::2012-06-22
15:55:19,557::threadPool::45::Misc.ThreadPool::(__init__) Enter -
numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
MainThread::DEBUG::2012-06-22
15:55:19,573::__init__::1249::Storage.Misc.excCmd::(_log) '/usr/bin/sudo
-n /usr/bin/kill -9 2106' (cwd None)
MainThread::DEBUG::2012-06-22
15:55:19,578::supervdsm::103::SuperVdsmProxy::(_killSupervdsm) Could not
kill old Super Vdsm Permission denied
MainThread::DEBUG::2012-06-22
15:55:19,579::supervdsm::91::SuperVdsmProxy::(_launchSupervdsm) Launching
Super Vdsm
MainThread::DEBUG::2012-06-22
15:55:19,580::__init__::1249::Storage.Misc.excCmd::(_log) '/usr/bin/sudo
-n /usr/bin/python /usr/share/vdsm/supervdsmServer.py
8fc32c0c-f502-4d3c-9de0-fc1bf2ba53cd 3485' (cwd None)
MainThread::ERROR::2012-06-22 15:55:19,584::clientIF::210::vds::(_initIRS)
Error initializing IRS
Traceback (most recent call last):
File "/usr/share/vdsm/clientIF.py", line 208, in _initIRS
self.irs = Dispatcher(HSM())
File "/usr/share/vdsm/storage/hsm.py", line 296, in __init__
if not multipath.isEnabled():
File "/usr/share/vdsm/storage/multipath.py", line 85, in isEnabled
svdsm = supervdsm.getProxy()
File "/usr/share/vdsm/supervdsm.py", line 135, in getProxy
_g_singletonSupervdsmInstance = SuperVdsmProxy()
File "/usr/share/vdsm/supervdsm.py", line 82, in __init__
self._restartSupervdsm()
File "/usr/share/vdsm/supervdsm.py", line 123, in _restartSupervdsm
self._launchSupervdsm()
File "/usr/share/vdsm/supervdsm.py", line 94, in _launchSupervdsm
misc.execCmd(superVdsmCmd, sync=False, sudo=True)
File "/usr/share/vdsm/storage/misc.py", line 199, in execCmd
p = BetterPopen(command, close_fds=True, cwd=cwd, env=env)
File "/usr/lib/python2.7/site-packages/vdsm/betterPopen/__init__.py",
line 46, in __init__
stderr=PIPE)
File "/usr/lib64/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/site-packages/vdsm/betterPopen/__init__.py",
line 60, in _execute_child
cwd, env)
OSError: Permission denied
MainThread::DEBUG::2012-06-22
15:55:19,594::__init__::1249::Storage.Misc.excCmd::(_log) '/usr/bin/pgrep
-xf ksmd' (cwd None)
MainThread::DEBUG::2012-06-22
15:55:19,606::__init__::1249::Storage.Misc.excCmd::(_log) SUCCESS: <err> =
''; <rc> = 0
MainThread::INFO::2012-06-22
15:55:19,607::vmChannels::139::vds::(settimeout) Setting channels' timeout
to 30 seconds.
VM Channels Listener::INFO::2012-06-22
15:55:19,608::vmChannels::127::vds::(run) Starting VM channels listener
thread.
MainThread::ERROR::2012-06-22
15:55:19,612::clientIF::151::vds::(_prepareBindings) Unable to load the
rest server module. Please make sure it is installed.
<>
Nathan Stratton
nathan at robotics.net
http://www.robotics.net
On Fri, 22 Jun 2012, Douglas Landgraf wrote:
Hi Nathan,
On 06/21/2012 01:15 PM, Nathan Stratton wrote:
>
> Fedora 17 RPMs, hosts complets install and reboots, but never connects back
> to engine. I just see thousands of the following in the logs:
>
> ----
>
> MainThread::INFO::2012-06-21 13:07:26,304::vdsm::71::vds::(run) I am the
> actual vdsm 4.9-0
4.9-0? Looks like a previous version of vdsm which we had a network error.
Can you please try 4.10.0 ?
> MainThread::ERROR::2012-06-21 13:07:26,656::vdsm::74::vds::(run) Traceback
> (most recent call last):
> File "/usr/share/vdsm/vdsm", line 72, in run
> serve_clients(log)
> File "/usr/share/vdsm/vdsm", line 40, in serve_clients
> cif = clientIF.clientIF(log)
> File "/usr/share/vdsm/clientIF.py", line 115, in __init__
> self._createLibvirtNetworks()
> File "/usr/share/vdsm/clientIF.py", line 152, in _createLibvirtNetworks
> nf = netinfo.NetInfo()
> File "/usr/share/vdsm/netinfo.py", line 268, in __init__
> _netinfo = get()
> File "/usr/share/vdsm/netinfo.py", line 215, in get
> for bridge in bridges() ])
> KeyError: 'ovirtmgmt'
>
> MainThread::INFO::2012-06-21 13:07:26,657::vdsm::76::vds::(run) VDSM main
> thread ended. Waiting for 1 other threads...
> MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
> <_MainThread(MainThread, started 140487047427904)>
> MainThread::INFO::2012-06-21 13:07:26,657::vdsm::79::vds::(run)
> <Thread(libvirtEventLoop, started daemon 140486974248704)>
>
> ----
>
> ovirtmgmt is up and running:
>
> -bash-4.2# brctl show
> bridge name bridge id STP enabled interfaces
> ovirtmgmt 8000.a0369f0a15f8 no bond0
>
>> <>
> Nathan Stratton
> nathan at robotics.net
> http://www.robotics.net
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--
Cheers
Douglas
9:58 a.m.
On Fri, Jun 22, 2012 at 02:57:40PM -0500, Nathan Stratton wrote:
Thanks! That was it, now I think I have some issue with sudo or
someting. If I run vdsm as root
Please refrain from doing that. It make vdsm create all sort of files
owned by root (vdsm.log, /var/run/vdsm/vdsm.pid, probably others) which
a future non-root vdsm process would find hard to override. I suspect
that this is your problem now.
Try running vdsm from the commandline (as vdsm user!) for hints where
startup fails.
it connects to the engine and the
host comes up, but if it is run normally I get:
MainThread::INFO::2012-06-22 15:46:50,424::vdsm::70::vds::(run) I am
the actual vdsm 4.10-1
4535
days inactive
4538
days old
5 comments
3 participants
participants (3)
-
Dan Kenigsberg -
Douglas Landgraf -
Nathan Stratton