Hi All, I need your help. Everytime the vdsm start, the server become *read-only file system*. My OS installed in SAN storage. I also got this error when i add the hosts "Host installation failed. Network error during communication with the host." Here are some errors i got in /var/log/messages: Oct 24 14:24:27 init: libvirtd main process (3394) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3410) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3416) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3420) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3425) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3434) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3439) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3443) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3447) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3457) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3461) terminated with status 1 Oct 24 14:24:27 init: libvirtd respawning too fast, stopped Oct 24 14:24:54 kernel: sd 1:0:0:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: sd 1:0:2:0: rejecting I/O to offline device Oct 24 14:24:54 device-mapper: multipath: Failing path 8:64. Oct 24 14:24:54 multipathd: 8:64: mark as failed Oct 24 14:24:54 multipathd: 3600507680281077d680000000000001c: remaining active paths: 5 Oct 24 14:24:54 kernel: sd 2:0:0:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: sd 2:0:1:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: sd 2:0:1:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:96. Oct 24 14:24:54 kernel: sd 2:0:2:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:112. Oct 24 14:24:54 kernel: sd 1:0:0:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:32. Oct 24 14:24:54 kernel: sd 1:0:1:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:48. Oct 24 14:24:54 kernel: sd 2:0:0:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:80. Oct 24 14:24:54 kernel: sd 2:0:2:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:112. Oct 24 14:24:54 kernel: sd 2:0:1:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:96. Oct 24 14:24:54 kernel: sd 1:0:2:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:64. Oct 24 14:24:54 kernel: Buffer I/O error on device dm-2, logical block 132850 Oct 24 14:24:54 kernel: lost page write due to I/O error on dm-2 Oct 24 14:24:54 kernel: Buffer I/O error on device dm-2, logical block 577540 Oct 24 14:24:54 kernel: lost page write due to I/O error on dm-2 Hope you can help me. Thanks
On 24.10.2014 09:02, John Michael Mercado wrote:
Hi All, Hi John,
I need your help. Everytime the vdsm start, the server become *read-only file system*. My OS installed in SAN storage. Reading your logs I can hardly believe this is a VDSM issue but rather something more basic:
I also got this error when i add the hosts "Host installation failed. Network error during communication with the host."
Here are some errors i got in /var/log/messages:
Oct 24 14:24:27 init: libvirtd main process (3394) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3410) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3416) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3420) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3425) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3434) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3439) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3443) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3447) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3457) terminated with status 1 Oct 24 14:24:27 init: libvirtd main process ended, respawning Oct 24 14:24:27 init: libvirtd main process (3461) terminated with status 1 Oct 24 14:24:27 init: libvirtd respawning too fast, stopped Oct 24 14:24:54 kernel: sd 1:0:0:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: sd 1:0:2:0: rejecting I/O to offline device Oct 24 14:24:54 device-mapper: multipath: Failing path 8:64. Oct 24 14:24:54 multipathd: 8:64: mark as failed Oct 24 14:24:54 multipathd: 3600507680281077d680000000000001c: remaining active Can you detail why the path is failing? paths: 5 Oct 24 14:24:54 kernel: sd 2:0:0:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: sd 2:0:1:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: sd 2:0:1:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:96. Oct 24 14:24:54 kernel: sd 2:0:2:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:112. Oct 24 14:24:54 kernel: sd 1:0:0:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:32. Oct 24 14:24:54 kernel: sd 1:0:1:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:48. Oct 24 14:24:54 kernel: sd 2:0:0:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:80. Oct 24 14:24:54 kernel: sd 2:0:2:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:112. Oct 24 14:24:54 kernel: sd 2:0:1:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:96. Oct 24 14:24:54 kernel: sd 1:0:2:0: rejecting I/O to offline device Oct 24 14:24:54 kernel: device-mapper: multipath: Failing path 8:64. Oct 24 14:24:54 kernel: Buffer I/O error on device dm-2, logical block 132850 Oct 24 14:24:54 kernel: lost page write due to I/O error on dm-2 Oct 24 14:24:54 kernel: Buffer I/O error on device dm-2, logical block 577540 Oct 24 14:24:54 kernel: lost page write due to I/O error on dm-2 Is dm-2 the failed path by any chance? Otherwise dm-2 seems quite a low number, this can be your root LV? However, I would investigate your multipath setup. Also, try to write some files (dd if=/dev/zero of=/path/to/shared/storage bs=4m).
Hope you can help me.
Thanks
-- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767
On Fri, Oct 24, 2014 at 03:01:40PM +0800, John Michael Mercado wrote:
Hi All,
I need your help. Everytime the vdsm start, the server become *read-only file system*. My OS installed in SAN storage.
Could you share more details about your platfor? Is it el6? Which vdsm version is running there? Did it work better with a previous version? Do you have anything in /var/log/vdsm/vdsm.log? and supervdsm.log? Vdsm may be restoring the host network configuration to some outdated values that break the connection to you SAN storage. Dan.
Hi Dan, Yes it is el6. VDSM version currently working without problem vdsm-4.14.11.2-0.el6.x86_64 but for the version vdsm-4.14.17-0.el6.x86_64 it's failing. I used ovirt-engine-3.4.3-1.el6.noarch version. Please see attached file vdsm.log and supervdsm.log On Fri, Oct 24, 2014 at 6:47 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Oct 24, 2014 at 03:01:40PM +0800, John Michael Mercado wrote:
Hi All,
I need your help. Everytime the vdsm start, the server become *read-only file system*. My OS installed in SAN storage.
Could you share more details about your platfor? Is it el6? Which vdsm version is running there? Did it work better with a previous version?
Do you have anything in /var/log/vdsm/vdsm.log? and supervdsm.log?
Vdsm may be restoring the host network configuration to some outdated values that break the connection to you SAN storage.
Dan.
I also have another setup and i used ovirt-engine-3.5.0.1-1.el6.noarch and vdsm-4.16.7-1.gitdb83943.el6.x86_64 but i still encounter the same error. Here is the vdsm.log MainThread::DEBUG::2014-10-23 18:20:33,717::resourceManager::421::Storage.ResourceManager::(registerNamespace) Registering namespace 'Storage' MainThread::DEBUG::2014-10-23 18:20:33,717::threadPool::35::Storage.ThreadPool::(__init__) Enter - numThreads: 10, waitTimeout: 3, maxTasks: 500 MainThread::DEBUG::2014-10-23 18:20:33,722::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt MainThread::DEBUG::2014-10-23 18:20:33,844::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm MainThread::DEBUG::2014-10-23 18:20:33,870::multipath::151::Storage.Multipath::(isEnabled) multipath Defaulting to False MainThread::DEBUG::2014-10-23 18:20:33,871::misc::456::Storage.Misc::(rotateFiles) dir: /etc, prefixName: multipath.conf, versions: 5 MainThread::DEBUG::2014-10-23 18:20:33,871::misc::477::Storage.Misc::(rotateFiles) versions found: [] MainThread::DEBUG::2014-10-23 18:20:33,872::multipath::169::Storage.Misc.excCmd::(setupMultipath) /usr/bin/sudo -n /bin/cp /tmp/tmp2_UKnY /etc/multipath.conf (cwd None) MainThread::DEBUG::2014-10-23 18:20:33,888::multipath::169::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:33,890::multipath::175::Storage.Misc.excCmd::(setupMultipath) /usr/bin/sudo -n /sbin/multipath -F (cwd None) MainThread::DEBUG::2014-10-23 18:20:33,907::multipath::175::Storage.Misc.excCmd::(setupMultipath) FAILED: <err> = ''; <rc> = 1 MainThread::DEBUG::2014-10-23 18:20:33,907::multipath::178::Storage.Misc.excCmd::(setupMultipath) /usr/bin/sudo -n /usr/bin/vdsm-tool service-reload multipathd (cwd None) MainThread::DEBUG::2014-10-23 18:20:54,163::multipath::178::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:54,163::hsm::418::Storage.Misc.excCmd::(__validateLvmLockingType) /usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type (cwd None) MainThread::DEBUG::2014-10-23 18:20:54,331::hsm::418::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:54,332::hsm::442::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' MainThread::DEBUG::2014-10-23 18:20:54,333::hsm::474::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] MainThread::DEBUG::2014-10-23 18:20:54,334::hsm::475::Storage.HSM::(__cleanStorageRepository) Mount list: [] MainThread::DEBUG::2014-10-23 18:20:54,334::hsm::477::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers MainThread::DEBUG::2014-10-23 18:20:54,338::hsm::520::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' storageRefresh::DEBUG::2014-10-23 18:20:54,339::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) MainThread::INFO::2014-10-23 18:20:54,341::dispatcher::46::Storage.Dispatcher::(__init__) Starting StorageDispatcher... MainThread::INFO::2014-10-24 14:24:27,283::vdsm::131::vds::(run) (PID: 3372) I am the actual vdsm 4.16.7-1.gitdb83943.el6 ovirt-node-mk2-05.i.gov.ph (2.6.32-431.29.2.el6.x86_64) MainThread::DEBUG::2014-10-24 14:24:27,284::resourceManager::421::Storage.ResourceManager::(registerNamespace) Registering namespace 'Storage' MainThread::DEBUG::2014-10-24 14:24:27,285::threadPool::35::Storage.ThreadPool::(__init__) Enter - numThreads: 10, waitTimeout: 3, maxTasks: 500 MainThread::DEBUG::2014-10-24 14:24:27,288::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt MainThread::WARNING::2014-10-24 14:24:27,289::fileUtils::149::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists MainThread::DEBUG::2014-10-24 14:24:27,394::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm MainThread::DEBUG::2014-10-24 14:24:27,418::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving MainThread::DEBUG::2014-10-24 14:24:27,418::hsm::418::Storage.Misc.excCmd::(__validateLvmLockingType) /usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type (cwd None) MainThread::DEBUG::2014-10-24 14:24:27,604::hsm::418::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:27,604::hsm::442::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' MainThread::DEBUG::2014-10-24 14:24:27,607::hsm::474::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] MainThread::DEBUG::2014-10-24 14:24:27,607::hsm::475::Storage.HSM::(__cleanStorageRepository) Mount list: [] MainThread::DEBUG::2014-10-24 14:24:27,607::hsm::477::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers MainThread::DEBUG::2014-10-24 14:24:27,611::hsm::520::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' MainThread::INFO::2014-10-24 14:24:27,614::dispatcher::46::Storage.Dispatcher::(__init__) Starting StorageDispatcher... storageRefresh::DEBUG::2014-10-24 14:24:27,614::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) storageRefresh::DEBUG::2014-10-24 14:24:47,671::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method storageRefresh::DEBUG::2014-10-24 14:24:47,673::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) storageRefresh::DEBUG::2014-10-24 14:24:47,679::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method storageRefresh::DEBUG::2014-10-24 14:24:47,680::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds storageRefresh::DEBUG::2014-10-24 14:24:47,684::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) MainThread::DEBUG::2014-10-24 14:24:47,761::task::595::Storage.TaskManager.Task::(_updateState) Task=`377cdbc9-9686-4f97-a846-2a2a0f87053f`::moving from state init -> state preparing MainThread::INFO::2014-10-24 14:24:47,761::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF object at 0x2d4a690>>) MainThread::INFO::2014-10-24 14:24:47,761::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None MainThread::DEBUG::2014-10-24 14:24:47,761::task::1191::Storage.TaskManager.Task::(prepare) Task=`377cdbc9-9686-4f97-a846-2a2a0f87053f`::finished: None MainThread::DEBUG::2014-10-24 14:24:47,761::task::595::Storage.TaskManager.Task::(_updateState) Task=`377cdbc9-9686-4f97-a846-2a2a0f87053f`::moving from state preparing -> state finished MainThread::DEBUG::2014-10-24 14:24:47,762::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} MainThread::DEBUG::2014-10-24 14:24:47,762::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} MainThread::DEBUG::2014-10-24 14:24:47,762::task::993::Storage.TaskManager.Task::(_decref) Task=`377cdbc9-9686-4f97-a846-2a2a0f87053f`::ref 0 aborting False MainThread::INFO::2014-10-24 14:24:47,763::momIF::49::MOM::(__init__) Starting up MOM MainThread::INFO::2014-10-24 14:24:47,767::vmchannels::192::vds::(settimeout) Setting channels' timeout to 30 seconds. VM Channels Listener::INFO::2014-10-24 14:24:47,769::vmchannels::175::vds::(run) Starting VM channels listener thread. clientIFinit::DEBUG::2014-10-24 14:24:47,772::libvirtconnection::150::root::(get) trying to connect libvirt MainThread::DEBUG::2014-10-24 14:24:47,783::protocoldetector::131::vds.MultiProtocolAcceptor::(add_detector) adding detector: <rpc.BindingXMLRPC.XmlDetector instance at 0x2e439e0> MainThread::DEBUG::2014-10-24 14:24:47,876::protocoldetector::131::vds.MultiProtocolAcceptor::(add_detector) adding detector: <yajsonrpc.stompReactor.StompDetector instance at 0x2fca050> JsonRpcServer::DEBUG::2014-10-24 14:24:47,880::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request Detector thread::DEBUG::2014-10-24 14:24:47,883::protocoldetector::70::vds.MultiProtocolAcceptor::(serve_forever) Acceptor running Detector thread::DEBUG::2014-10-24 14:24:47,884::protocoldetector::72::vds.MultiProtocolAcceptor::(serve_forever) Using required_size=7 storageRefresh::DEBUG::2014-10-24 14:24:49,733::misc::751::Storage.SamplingMethod::(__call__) Returning last result storageRefresh::DEBUG::2014-10-24 14:24:49,770::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None) and Here is the supervdsm.log MainThread::DEBUG::2014-10-23 18:17:50,485::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:17:50,486::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:17:50,516::supervdsmServer::411::SuperVdsm.Server::(main) Making sure I'm root - SuperVdsm MainThread::DEBUG::2014-10-23 18:17:50,517::supervdsmServer::420::SuperVdsm.Server::(main) Parsing cmd args MainThread::DEBUG::2014-10-23 18:17:50,517::supervdsmServer::423::SuperVdsm.Server::(main) Cleaning old socket /var/run/vdsm/svdsm.sock MainThread::DEBUG::2014-10-23 18:17:50,517::supervdsmServer::427::SuperVdsm.Server::(main) Setting up keep alive thread MainThread::DEBUG::2014-10-23 18:17:50,517::supervdsmServer::433::SuperVdsm.Server::(main) Creating remote object manager MainThread::DEBUG::2014-10-23 18:17:50,519::fileUtils::192::Storage.fileUtils::(chown) Changing owner for /var/run/vdsm/svdsm.sock, to (36:36) MainThread::DEBUG::2014-10-23 18:17:50,519::supervdsmServer::444::SuperVdsm.Server::(main) Started serving super vdsm object sourceRoute::DEBUG::2014-10-23 18:17:50,519::sourceroutethread::75::root::(_subscribeToInotifyLoop) sourceRouteThread.subscribeToInotifyLoop started MainThread::DEBUG::2014-10-23 18:20:12,945::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:12,945::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:12,945::vdsm-restore-net-config::55::root::(unified_restoration) Removing all networks ({}) and bonds ({}) in running config. MainThread::DEBUG::2014-10-23 18:20:12,945::libvirtconnection::150::root::(get) trying to connect libvirt MainThread::DEBUG::2014-10-23 18:20:12,959::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,961::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,962::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,964::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,974::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/bonding_masters (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,976::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,976::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/L5P6SkRXZIa3zqz/bonding/mode (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,979::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,980::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/L5P6SkRXZIa3zqz/bonding/mode (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,982::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,983::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/L5P6SkRXZIa3zqz/bonding/mode (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,985::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,986::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/L5P6SkRXZIa3zqz/bonding/mode (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,989::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,990::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/L5P6SkRXZIa3zqz/bonding/mode (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,992::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,993::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/L5P6SkRXZIa3zqz/bonding/mode (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,995::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,996::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/L5P6SkRXZIa3zqz/bonding/mode (cwd None) MainThread::DEBUG::2014-10-23 18:20:12,997::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:12,999::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/bonding_masters (cwd None) MainThread::DEBUG::2014-10-23 18:20:13,035::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:13,036::api::621::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{}, bondings:{}, options:{'_inRollback': True, 'connectivityCheck': False} MainThread::DEBUG::2014-10-23 18:20:13,036::api::625::root::(setupNetworks) Validating configuration MainThread::DEBUG::2014-10-23 18:20:13,040::api::637::setupNetworks::(setupNetworks) Applying... MainThread::DEBUG::2014-10-23 18:20:13,040::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,040::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,041::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-23 18:20:13,043::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:13,053::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-23 18:20:13,055::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::INFO::2014-10-23 18:20:13,063::netconfpersistence::158::root::(_clearDisk) Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainThread::DEBUG::2014-10-23 18:20:13,063::netconfpersistence::166::root::(_clearDisk) No existent config to clear. MainThread::INFO::2014-10-23 18:20:13,063::netconfpersistence::182::root::(save) Saved new config RunningConfig({}, {}) to /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainThread::DEBUG::2014-10-23 18:20:13,065::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,065::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,116::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,116::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,117::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,117::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,120::vdsm-restore-net-config::68::root::(unified_restoration) Calling setupNetworks with networks ({}) and bond ({}). MainThread::DEBUG::2014-10-23 18:20:13,121::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-23 18:20:13,123::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:13,124::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-23 18:20:13,126::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:13,135::api::621::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{}, bondings:{}, options:{'_inRollback': True, 'connectivityCheck': False} MainThread::DEBUG::2014-10-23 18:20:13,135::api::625::root::(setupNetworks) Validating configuration MainThread::DEBUG::2014-10-23 18:20:13,139::api::637::setupNetworks::(setupNetworks) Applying... MainThread::DEBUG::2014-10-23 18:20:13,139::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,139::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-23 18:20:13,139::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-23 18:20:13,141::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-23 18:20:13,150::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-23 18:20:13,153::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::INFO::2014-10-23 18:20:13,161::netconfpersistence::158::root::(_clearDisk) Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainThread::DEBUG::2014-10-23 18:20:13,161::netconfpersistence::166::root::(_clearDisk) No existent config to clear. MainThread::INFO::2014-10-23 18:20:13,161::netconfpersistence::182::root::(save) Saved new config RunningConfig({}, {}) to /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainProcess::DEBUG::2014-10-23 18:20:33,869::supervdsmServer::101::SuperVdsm.ServerCallback::(wrapper) call readMultipathConf with () {} MainProcess::DEBUG::2014-10-23 18:20:33,870::supervdsmServer::108::SuperVdsm.ServerCallback::(wrapper) return readMultipathConf with ['# multipath.conf written by anaconda', '', 'defaults {', '\tuser_friendly_names yes', '}', 'blacklist {', '\tdevnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"', '\tdevnode "^hd[a-z]"', '\tdevnode "^dcssblk[0-9]*"', '\tdevice {', '\t\tvendor "DGC"', '\t\tproduct "LUNZ"', '\t}', '\tdevice {', '\t\tvendor "IBM"', '\t\tproduct "S/390.*"', '\t}', "\t# don't count normal SATA devices as multipaths", '\tdevice {', '\t\tvendor "ATA"', '\t}', "\t# don't count 3ware devices as multipaths", '\tdevice {', '\t\tvendor "3ware"', '\t}', '\tdevice {', '\t\tvendor "AMCC"', '\t}', '\t# nor highpoint devices', '\tdevice {', '\t\tvendor "HPT"', '\t}', '\twwid "F0102BJF"', '\tdevice {', '\t\tvendor IBM_SATA', '\t\tproduct DEVICE_81Y3658', '\t}', '\twwid "*"', '}', 'blacklist_exceptions {', '\twwid "3600507680281077d680000000000001c"', '}', 'multipaths {', '\tmultipath {', '\t\tuid 0', '\t\tgid 0', '\t\twwid "3600507680281077d680000000000001c"', '\t\tmode 0600', '\t}', '}'] MainThread::DEBUG::2014-10-24 14:21:44,088::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:21:44,088::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:21:44,128::supervdsmServer::411::SuperVdsm.Server::(main) Making sure I'm root - SuperVdsm MainThread::DEBUG::2014-10-24 14:21:44,129::supervdsmServer::420::SuperVdsm.Server::(main) Parsing cmd args MainThread::DEBUG::2014-10-24 14:21:44,129::supervdsmServer::423::SuperVdsm.Server::(main) Cleaning old socket /var/run/vdsm/svdsm.sock MainThread::DEBUG::2014-10-24 14:21:44,129::supervdsmServer::427::SuperVdsm.Server::(main) Setting up keep alive thread MainThread::DEBUG::2014-10-24 14:21:44,129::supervdsmServer::433::SuperVdsm.Server::(main) Creating remote object manager MainThread::DEBUG::2014-10-24 14:21:44,131::fileUtils::192::Storage.fileUtils::(chown) Changing owner for /var/run/vdsm/svdsm.sock, to (36:36) MainThread::DEBUG::2014-10-24 14:21:44,131::supervdsmServer::444::SuperVdsm.Server::(main) Started serving super vdsm object sourceRoute::DEBUG::2014-10-24 14:21:44,131::sourceroutethread::75::root::(_subscribeToInotifyLoop) sourceRouteThread.subscribeToInotifyLoop started MainThread::DEBUG::2014-10-24 14:24:06,571::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,571::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,571::vdsm-restore-net-config::55::root::(unified_restoration) Removing all networks ({}) and bonds ({}) in running config. MainThread::DEBUG::2014-10-24 14:24:06,571::libvirtconnection::150::root::(get) trying to connect libvirt MainThread::DEBUG::2014-10-24 14:24:06,591::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,594::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,595::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,597::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,607::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/bonding_masters (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,610::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,610::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/WUR1ZiCBIS9eo8U/bonding/mode (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,612::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,613::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/WUR1ZiCBIS9eo8U/bonding/mode (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,615::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,616::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/WUR1ZiCBIS9eo8U/bonding/mode (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,618::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,619::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/WUR1ZiCBIS9eo8U/bonding/mode (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,621::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,622::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/WUR1ZiCBIS9eo8U/bonding/mode (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,624::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,625::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/WUR1ZiCBIS9eo8U/bonding/mode (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,626::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,628::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/WUR1ZiCBIS9eo8U/bonding/mode (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,630::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,631::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/bonding_masters (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,671::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,672::api::621::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{}, bondings:{}, options:{'_inRollback': True, 'connectivityCheck': False} MainThread::DEBUG::2014-10-24 14:24:06,672::api::625::root::(setupNetworks) Validating configuration MainThread::DEBUG::2014-10-24 14:24:06,676::api::637::setupNetworks::(setupNetworks) Applying... MainThread::DEBUG::2014-10-24 14:24:06,676::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,676::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,676::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,678::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,688::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,690::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::INFO::2014-10-24 14:24:06,699::netconfpersistence::158::root::(_clearDisk) Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainThread::DEBUG::2014-10-24 14:24:06,699::netconfpersistence::166::root::(_clearDisk) No existent config to clear. MainThread::INFO::2014-10-24 14:24:06,699::netconfpersistence::182::root::(save) Saved new config RunningConfig({}, {}) to /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainThread::DEBUG::2014-10-24 14:24:06,700::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,700::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,705::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,705::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,706::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,706::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,709::vdsm-restore-net-config::68::root::(unified_restoration) Calling setupNetworks with networks ({}) and bond ({}). MainThread::DEBUG::2014-10-24 14:24:06,710::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,712::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,713::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,715::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,724::api::621::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{}, bondings:{}, options:{'_inRollback': True, 'connectivityCheck': False} MainThread::DEBUG::2014-10-24 14:24:06,724::api::625::root::(setupNetworks) Validating configuration MainThread::DEBUG::2014-10-24 14:24:06,727::api::637::setupNetworks::(setupNetworks) Applying... MainThread::DEBUG::2014-10-24 14:24:06,728::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,728::netconfpersistence::134::root::(_getConfigs) Non-existing config set. MainThread::DEBUG::2014-10-24 14:24:06,728::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,730::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::DEBUG::2014-10-24 14:24:06,740::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainThread::DEBUG::2014-10-24 14:24:06,742::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainThread::INFO::2014-10-24 14:24:06,750::netconfpersistence::158::root::(_clearDisk) Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainThread::DEBUG::2014-10-24 14:24:06,750::netconfpersistence::166::root::(_clearDisk) No existent config to clear. MainThread::INFO::2014-10-24 14:24:06,750::netconfpersistence::182::root::(save) Saved new config RunningConfig({}, {}) to /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainProcess::DEBUG::2014-10-24 14:24:27,417::supervdsmServer::101::SuperVdsm.ServerCallback::(wrapper) call readMultipathConf with () {} MainProcess::DEBUG::2014-10-24 14:24:27,417::supervdsmServer::108::SuperVdsm.ServerCallback::(wrapper) return readMultipathConf with ['# RHEV REVISION 1.0', '', 'defaults {', ' polling_interval 5', ' getuid_callout "/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"', ' no_path_retry fail', ' user_friendly_names no', ' flush_on_last_del yes', ' fast_io_fail_tmo 5', ' dev_loss_tmo 30', ' max_fds 4096', '}', '', 'devices {', 'device {', ' vendor "HITACHI"', ' product "DF.*"', ' getuid_callout "/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"', '}', 'device {', ' vendor "COMPELNT"', ' product "Compellent Vol"', ' no_path_retry fail', '}', '}'] MainProcess|storageRefresh::DEBUG::2014-10-24 14:24:49,745::supervdsmServer::101::SuperVdsm.ServerCallback::(wrapper) call hbaRescan with () {} MainProcess|storageRefresh::INFO::2014-10-24 14:24:49,746::hba::54::Storage.HBA::(rescan) Rescanning HBAs MainProcess|storageRefresh::DEBUG::2014-10-24 14:24:49,747::hba::56::Storage.HBA::(rescan) Issuing lip /sys/class/fc_host/host1/issue_lip MainProcess|storageRefresh::DEBUG::2014-10-24 14:24:49,758::hba::56::Storage.HBA::(rescan) Issuing lip /sys/class/fc_host/host2/issue_lip MainProcess|storageRefresh::DEBUG::2014-10-24 14:24:49,769::supervdsmServer::108::SuperVdsm.ServerCallback::(wrapper) return hbaRescan with None On Fri, Oct 24, 2014 at 6:47 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Oct 24, 2014 at 03:01:40PM +0800, John Michael Mercado wrote:
Hi All,
I need your help. Everytime the vdsm start, the server become *read-only file system*. My OS installed in SAN storage.
Could you share more details about your platfor? Is it el6? Which vdsm version is running there? Did it work better with a previous version?
Do you have anything in /var/log/vdsm/vdsm.log? and supervdsm.log?
Vdsm may be restoring the host network configuration to some outdated values that break the connection to you SAN storage.
Dan.
participants (3)
-
Dan Kenigsberg -
Daniel Helgenberger -
John Michael Mercado