supervdsmd IOError to /dev/stdout
After an upgrade to 4.0 I have a single host that cannot start supervdsmd because of IOError on /dev/stdout. All other hosts upgraded correctly. In the systemd unit I have to hack StandardOutput=null. Any thing I have overlooked? The hosts are all identical and it is just this one that has this weird behaviour. -- Richard Chan
On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan <richard@treeboxsolutions.com> wrote:
After an upgrade to 4.0 I have a single host that cannot start supervdsmd because of IOError on /dev/stdout. All other hosts upgraded correctly.
Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1?
In the systemd unit I have to hack StandardOutput=null.
Any thing I have overlooked? The hosts are all identical and it is just this one that has this weird behaviour.
Any logs that you can share? Y.
-- Richard Chan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Yaniv, On this one node, it happened from 3.6 -> 4.0. It persisted after 4.0->4.1. Current: vdsm-4.19.24-1.el7.centos.x86_64 In systemd override I now have to have: StandardOutput=null Jul 29 01:19:21 xxxxxxxx systemd[1]: Starting Auxiliary vdsm service for running helper functions as root... Jul 29 01:19:21 xxxxxxxx python2[39124]: detected unhandled Python exception in '/usr/share/vdsm/supervdsmServer' Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: Traceback (most recent call last): Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/share/vdsm/supervdsmServer", line 45, in <module> Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: level=logging.DEBUG) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: hdlr = FileHandler(filename, mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: StreamHandler.__init__(self, self._open()) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: stream = open(self.baseFilename, self.mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: IOError: [Errno 6] No such device or address: '/dev/stdout' Jul 29 01:19:21 xxxxxxxx systemd[1]: supervdsmd.service: main process exited, code=exited, status=1/FAILURE On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < richard@treeboxsolutions.com> wrote:
After an upgrade to 4.0 I have a single host that cannot start supervdsmd because of IOError on /dev/stdout. All other hosts upgraded correctly.
Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1?
In the systemd unit I have to hack StandardOutput=null.
Any thing I have overlooked? The hosts are all identical and it is just this one that has this weird behaviour.
Any logs that you can share? Y.
-- Richard Chan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Richard Chan Chief Architect TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com Co.Reg.No. 201100585R
Hi, Seems like I already bumped in such issue awhile ago - https://bugzilla.redhat.com/show_bug.cgi?id=1216880 but now I see what's wrong - it happens after failing to read /etc/vdsm/svdsm.logger.conf for some reason then we have a bug in our fallback which tries to run logging.basicConfig(filename='/dev/stdout', filemode='w+', level=logging.DEBUG) and this fails while running as systemd daemon sorry about that. check why it can't load /etc/vdsm/svdsm.logger.conf and update us after fixing the access issue check if you can run logging.config.fileConfig("/etc/vdsm/svdsm.logger.conf" , disable_existing_loggers=False) Then supervdsmd should start properly I'll reopen the bugzilla and fix the wrong fallback to syslog print Thanks! On Sun, Jul 30, 2017 at 3:23 PM Richard Chan <richard@treeboxsolutions.com> wrote:
Hi Yaniv,
On this one node, it happened from 3.6 -> 4.0. It persisted after 4.0->4.1. Current: vdsm-4.19.24-1.el7.centos.x86_64
In systemd override I now have to have: StandardOutput=null
Jul 29 01:19:21 xxxxxxxx systemd[1]: Starting Auxiliary vdsm service for running helper functions as root... Jul 29 01:19:21 xxxxxxxx python2[39124]: detected unhandled Python exception in '/usr/share/vdsm/supervdsmServer' Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: Traceback (most recent call last): Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/share/vdsm/supervdsmServer", line 45, in <module> Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: level=logging.DEBUG) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: hdlr = FileHandler(filename, mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: StreamHandler.__init__(self, self._open()) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: stream = open(self.baseFilename, self.mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: IOError: [Errno 6] No such device or address: '/dev/stdout' Jul 29 01:19:21 xxxxxxxx systemd[1]: supervdsmd.service: main process exited, code=exited, status=1/FAILURE
On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < richard@treeboxsolutions.com> wrote:
After an upgrade to 4.0 I have a single host that cannot start supervdsmd because of IOError on /dev/stdout. All other hosts upgraded correctly.
Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1?
In the systemd unit I have to hack StandardOutput=null.
Any thing I have overlooked? The hosts are all identical and it is just this one that has this weird behaviour.
Any logs that you can share? Y.
-- Richard Chan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Richard Chan Chief Architect
TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com
Co.Reg.No. 201100585R _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Yaniv Bronhaim.
Hi, Could this be the cause of the access issue (old system upgraded from 3.4 all the way to 4.1)? ## change in python class name?? [handler_logfile] -class=logUtils.UserGroupEnforcingHandler +class=vdsm.logUtils.UserGroupEnforcingHandler When I copied svdsm.logger.conf.rpmnew over svdsm.logger.conf, it could start up. Thanks! On Tue, Aug 1, 2017 at 5:02 PM, Yaniv Bronheim <ybronhei@redhat.com> wrote:
Hi,
Seems like I already bumped in such issue awhile ago - https://bugzilla.redhat.com/show_bug.cgi?id=1216880 but now I see what's wrong - it happens after failing to read /etc/vdsm/svdsm.logger.conf for some reason then we have a bug in our fallback which tries to run logging.basicConfig(filename='/dev/stdout', filemode='w+', level=logging.DEBUG)
and this fails while running as systemd daemon
sorry about that. check why it can't load /etc/vdsm/svdsm.logger.conf and update us after fixing the access issue check if you can run logging.config.fileConfig("/etc/vdsm/svdsm.logger.conf" , disable_existing_loggers=False) Then supervdsmd should start properly
I'll reopen the bugzilla and fix the wrong fallback to syslog print
Thanks!
On Sun, Jul 30, 2017 at 3:23 PM Richard Chan <richard@treeboxsolutions.com> wrote:
Hi Yaniv,
On this one node, it happened from 3.6 -> 4.0. It persisted after 4.0->4.1. Current: vdsm-4.19.24-1.el7.centos.x86_64
In systemd override I now have to have: StandardOutput=null
Jul 29 01:19:21 xxxxxxxx systemd[1]: Starting Auxiliary vdsm service for running helper functions as root... Jul 29 01:19:21 xxxxxxxx python2[39124]: detected unhandled Python exception in '/usr/share/vdsm/supervdsmServer' Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: Traceback (most recent call last): Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/share/vdsm/supervdsmServer", line 45, in <module> Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: level=logging.DEBUG) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: hdlr = FileHandler(filename, mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: StreamHandler.__init__(self, self._open()) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: stream = open(self.baseFilename, self.mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: IOError: [Errno 6] No such device or address: '/dev/stdout' Jul 29 01:19:21 xxxxxxxx systemd[1]: supervdsmd.service: main process exited, code=exited, status=1/FAILURE
On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < richard@treeboxsolutions.com> wrote:
After an upgrade to 4.0 I have a single host that cannot start supervdsmd because of IOError on /dev/stdout. All other hosts upgraded correctly.
Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1?
In the systemd unit I have to hack StandardOutput=null.
Any thing I have overlooked? The hosts are all identical and it is just this one that has this weird behaviour.
Any logs that you can share? Y.
-- Richard Chan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Richard Chan Chief Architect
TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com
Co.Reg.No. 201100585R _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Yaniv Bronhaim.
-- Richard Chan Chief Architect TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com Co.Reg.No. 201100585R
On Tue, Aug 1, 2017 at 3:07 PM Richard Chan <richard@treeboxsolutions.com> wrote:
Hi,
Could this be the cause of the access issue (old system upgraded from 3.4 all the way to 4.1)? ## change in python class name??
[handler_logfile] -class=logUtils.UserGroupEnforcingHandler +class=vdsm.logUtils.UserGroupEnforcingHandler
When I copied svdsm.logger.conf.rpmnew over svdsm.logger.conf, it could start up.
If this is the case, I don't think we should change anything. Logging to syslog by default is problematic, we don't want to spam systlog with unimportant debug logs. Nir
Thanks!
On Tue, Aug 1, 2017 at 5:02 PM, Yaniv Bronheim <ybronhei@redhat.com> wrote:
Hi,
Seems like I already bumped in such issue awhile ago - https://bugzilla.redhat.com/show_bug.cgi?id=1216880 but now I see what's wrong - it happens after failing to read /etc/vdsm/svdsm.logger.conf for some reason then we have a bug in our fallback which tries to run logging.basicConfig(filename='/dev/stdout', filemode='w+', level=logging.DEBUG)
and this fails while running as systemd daemon
sorry about that. check why it can't load /etc/vdsm/svdsm.logger.conf and update us after fixing the access issue check if you can run logging.config.fileConfig("/etc/vdsm/svdsm.logger.conf" , disable_existing_loggers=False) Then supervdsmd should start properly
I'll reopen the bugzilla and fix the wrong fallback to syslog print
Thanks!
On Sun, Jul 30, 2017 at 3:23 PM Richard Chan < richard@treeboxsolutions.com> wrote:
Hi Yaniv,
On this one node, it happened from 3.6 -> 4.0. It persisted after 4.0->4.1. Current: vdsm-4.19.24-1.el7.centos.x86_64
In systemd override I now have to have: StandardOutput=null
Jul 29 01:19:21 xxxxxxxx systemd[1]: Starting Auxiliary vdsm service for running helper functions as root... Jul 29 01:19:21 xxxxxxxx python2[39124]: detected unhandled Python exception in '/usr/share/vdsm/supervdsmServer' Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: Traceback (most recent call last): Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/share/vdsm/supervdsmServer", line 45, in <module> Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: level=logging.DEBUG) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: hdlr = FileHandler(filename, mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: StreamHandler.__init__(self, self._open()) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: stream = open(self.baseFilename, self.mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: IOError: [Errno 6] No such device or address: '/dev/stdout' Jul 29 01:19:21 xxxxxxxx systemd[1]: supervdsmd.service: main process exited, code=exited, status=1/FAILURE
On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < richard@treeboxsolutions.com> wrote:
After an upgrade to 4.0 I have a single host that cannot start supervdsmd because of IOError on /dev/stdout. All other hosts upgraded correctly.
Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1?
In the systemd unit I have to hack StandardOutput=null.
Any thing I have overlooked? The hosts are all identical and it is just this one that has this weird behaviour.
Any logs that you can share? Y.
-- Richard Chan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Richard Chan Chief Architect
TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com
Co.Reg.No. 201100585R _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Yaniv Bronhaim.
-- Richard Chan Chief Architect
TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com
Co.Reg.No. 201100585R _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Did it change the file permissions when you copied svdsm.logger.conf.rpmnew? can you reproduce it? On Tue, Aug 1, 2017 at 3:06 PM Richard Chan <richard@treeboxsolutions.com> wrote:
Hi,
Could this be the cause of the access issue (old system upgraded from 3.4 all the way to 4.1)? ## change in python class name??
[handler_logfile] -class=logUtils.UserGroupEnforcingHandler +class=vdsm.logUtils.UserGroupEnforcingHandler
When I copied svdsm.logger.conf.rpmnew over svdsm.logger.conf, it could start up.
Thanks!
On Tue, Aug 1, 2017 at 5:02 PM, Yaniv Bronheim <ybronhei@redhat.com> wrote:
Hi,
Seems like I already bumped in such issue awhile ago - https://bugzilla.redhat.com/show_bug.cgi?id=1216880 but now I see what's wrong - it happens after failing to read /etc/vdsm/svdsm.logger.conf for some reason then we have a bug in our fallback which tries to run logging.basicConfig(filename='/dev/stdout', filemode='w+', level=logging.DEBUG)
and this fails while running as systemd daemon
sorry about that. check why it can't load /etc/vdsm/svdsm.logger.conf and update us after fixing the access issue check if you can run logging.config.fileConfig("/etc/vdsm/svdsm.logger.conf" , disable_existing_loggers=False) Then supervdsmd should start properly
I'll reopen the bugzilla and fix the wrong fallback to syslog print
Thanks!
On Sun, Jul 30, 2017 at 3:23 PM Richard Chan < richard@treeboxsolutions.com> wrote:
Hi Yaniv,
On this one node, it happened from 3.6 -> 4.0. It persisted after 4.0->4.1. Current: vdsm-4.19.24-1.el7.centos.x86_64
In systemd override I now have to have: StandardOutput=null
Jul 29 01:19:21 xxxxxxxx systemd[1]: Starting Auxiliary vdsm service for running helper functions as root... Jul 29 01:19:21 xxxxxxxx python2[39124]: detected unhandled Python exception in '/usr/share/vdsm/supervdsmServer' Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: Traceback (most recent call last): Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/share/vdsm/supervdsmServer", line 45, in <module> Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: level=logging.DEBUG) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: hdlr = FileHandler(filename, mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: StreamHandler.__init__(self, self._open()) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: stream = open(self.baseFilename, self.mode) Jul 29 01:19:21 xxxxxxxx daemonAdapter[39124]: IOError: [Errno 6] No such device or address: '/dev/stdout' Jul 29 01:19:21 xxxxxxxx systemd[1]: supervdsmd.service: main process exited, code=exited, status=1/FAILURE
On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < richard@treeboxsolutions.com> wrote:
After an upgrade to 4.0 I have a single host that cannot start supervdsmd because of IOError on /dev/stdout. All other hosts upgraded correctly.
Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1?
In the systemd unit I have to hack StandardOutput=null.
Any thing I have overlooked? The hosts are all identical and it is just this one that has this weird behaviour.
Any logs that you can share? Y.
-- Richard Chan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Richard Chan Chief Architect
TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com
Co.Reg.No. 201100585R _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Yaniv Bronhaim.
-- Richard Chan Chief Architect
TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com
Co.Reg.No. 201100585R
-- Yaniv Bronhaim.
Did it change the file permissions when you copied svdsm.logger.conf.rpmnew? No, the file permission were the same root:root 0644 Yes reproducible with 3.6 config file - it is definitely due to the 3.6 svdsm.logger.conf: when I copied the rpmnew file from 4.1, it works again.
participants (4)
-
Nir Soffer -
Richard Chan -
Yaniv Bronheim -
Yaniv Kaul