Gluster storage domain error after upgrading to 3.6
Hello, I have an Ovirt installation with only 1 host and self-hosted engine. My Master Data storage domain is GlusterFS type. After upgrading to Ovirt 3.6 data storage domain and default dataceter are down. The error in vdsm.log is: Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin g Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non e Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/share/vdsm/storage/storageServer.py", line 323, in options backup_servers_option = self._get_backup_servers_option() File "/usr/share/vdsm/storage/storageServer.py", line 340, in _get_backup_servers_option servers.remove(self._volfileserver) ValueError: list.remove(x): x not in list Thread-6585::DEBUG::2015-11-04 23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain} Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state preparing -> state finished
My error is related to this message: http://lists.ovirt.org/pipermail/users/2015-August/034316.html Il 05/11/2015 0.28, Stefano Danzi ha scritto:
Hello, I have an Ovirt installation with only 1 host and self-hosted engine. My Master Data storage domain is GlusterFS type.
After upgrading to Ovirt 3.6 data storage domain and default dataceter are down. The error in vdsm.log is:
Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin g Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non e Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/share/vdsm/storage/storageServer.py", line 323, in options backup_servers_option = self._get_backup_servers_option() File "/usr/share/vdsm/storage/storageServer.py", line 340, in _get_backup_servers_option servers.remove(self._volfileserver) ValueError: list.remove(x): x not in list Thread-6585::DEBUG::2015-11-04 23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain} Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state preparing -> state finished _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
To temporary solve the problem I patched storageserver.py as suggested on link above. I can't find a related issue on bugzilla. Il 05/11/2015 11.43, Stefano Danzi ha scritto:
My error is related to this message:
http://lists.ovirt.org/pipermail/users/2015-August/034316.html
Il 05/11/2015 0.28, Stefano Danzi ha scritto:
Hello, I have an Ovirt installation with only 1 host and self-hosted engine. My Master Data storage domain is GlusterFS type.
After upgrading to Ovirt 3.6 data storage domain and default dataceter are down. The error in vdsm.log is:
Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin g Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non e Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/share/vdsm/storage/storageServer.py", line 323, in options backup_servers_option = self._get_backup_servers_option() File "/usr/share/vdsm/storage/storageServer.py", line 340, in _get_backup_servers_option servers.remove(self._volfileserver) ValueError: list.remove(x): x not in list Thread-6585::DEBUG::2015-11-04 23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain} Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state preparing -> state finished _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
בתאריך 5 בנוב׳ 2015 8:18 אחה״צ, "Stefano Danzi" <s.danzi@hawai.it> כתב:
To temporary solve the problem I patched storageserver.py as suggested on
link above. I would not patch the code but change the configuration.
I can't find a related issue on bugzilla.
Would you file bug about this?
Il 05/11/2015 11.43, Stefano Danzi ha scritto:
My error is related to this message:
http://lists.ovirt.org/pipermail/users/2015-August/034316.html
Il 05/11/2015 0.28, Stefano Danzi ha scritto:
Hello, I have an Ovirt installation with only 1 host and self-hosted engine. My Master Data storage domain is GlusterFS type.
After upgrading to Ovirt 3.6 data storage domain and default dataceter
The error in vdsm.log is:
Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin g Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating
are down. directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non
e Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/share/vdsm/storage/storageServer.py", line 323, in options backup_servers_option = self._get_backup_servers_option() File "/usr/share/vdsm/storage/storageServer.py", line 340, in _get_backup_servers_option servers.remove(self._volfileserver) ValueError: list.remove(x): x not in list Thread-6585::DEBUG::2015-11-04 23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain} Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state preparing -> state finished _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a multi-part message in MIME format. --------------020002040508080703030705 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit I pathced the code for "emergency".... I can't find how change confguration. But I think that's a bug: - All work in ovirt 3.5, after upgrade stop working - The log show a python exception. I think a thing: If there are chages on configuration requirements I have to be warned during upgrade, or I have to find a specific error in log. ...remove something that non exist from a list, and leave a cryptic python exception as error log, isn't the better solution... Il 06/11/2015 8.12, Nir Soffer ha scritto:
בתאריך 5 בנוב׳ 2015 8:18 אחה״צ, "Stefano Danzi" <s.danzi@hawai.it <mailto:s.danzi@hawai.it>> כתב:
To temporary solve the problem I patched storageserver.py as
suggested on link above.
I would not patch the code but change the configuration.
I can't find a related issue on bugzilla.
Would you file bug about this?
Il 05/11/2015 11.43, Stefano Danzi ha scritto:
My error is related to this message:
http://lists.ovirt.org/pipermail/users/2015-August/034316.html
Il 05/11/2015 0.28, Stefano Danzi ha scritto:
Hello, I have an Ovirt installation with only 1 host and self-hosted engine. My Master Data storage domain is GlusterFS type.
After upgrading to Ovirt 3.6 data storage domain and default
The error in vdsm.log is:
Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin g Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating
e Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/share/vdsm/storage/storageServer.py", line 323, in
backup_servers_option = self._get_backup_servers_option() File "/usr/share/vdsm/storage/storageServer.py", line 340, in
_get_backup_servers_option
servers.remove(self._volfileserver) ValueError: list.remove(x): x not in list Thread-6585::DEBUG::2015-11-04
23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain}
Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state
dataceter are down. directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non options preparing -> state finished
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--------------020002040508080703030705 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> I pathced the code for "emergency".... <br> I can't find how change confguration.<br> <br> But I think that's a bug: <br> <br> - All work in ovirt 3.5, after upgrade stop working<br> - The log show a python exception.<br> <br> I think a thing:<br> <br> If there are chages on configuration requirements I have to be warned during upgrade, or I have to find a specific error in log.<br> ...remove something that non exist from a list, and leave a cryptic python exception as error log, isn't the better solution...<br> <br> <div class="moz-cite-prefix">Il 06/11/2015 8.12, Nir Soffer ha scritto:<br> </div> <blockquote cite="mid:CAMRbyyt0ui6FOH8TbeDcONNJ0TLwuaWPDN79NY22064eK7jQcg@mail.gmail.com" type="cite"> <p dir="ltr"><br> בתאריך 5 בנוב׳ 2015 8:18 אחה״צ, "Stefano Danzi" <<a moz-do-not-send="true" href="mailto:s.danzi@hawai.it"><a class="moz-txt-link-abbreviated" href="mailto:s.danzi@hawai.it">s.danzi@hawai.it</a></a>> כתב:<br> ><br> > To temporary solve the problem I patched storageserver.py as suggested on link above.</p> <p dir="ltr">I would not patch the code but change the configuration.</p> <p dir="ltr">> I can't find a related issue on bugzilla.</p> <p dir="ltr">Would you file bug about this?</p> <p dir="ltr">><br> ><br> > Il 05/11/2015 11.43, Stefano Danzi ha scritto:<br> >><br> >> My error is related to this message:<br> >><br> >> <a moz-do-not-send="true" href="http://lists.ovirt.org/pipermail/users/2015-August/034316.html">http://lists.ovirt.org/pipermail/users/2015-August/034316.html</a><br> >><br> >> Il 05/11/2015 0.28, Stefano Danzi ha scritto:<br> >>><br> >>> Hello,<br> >>> I have an Ovirt installation with only 1 host and self-hosted engine.<br> >>> My Master Data storage domain is GlusterFS type.<br> >>><br> >>> After upgrading to Ovirt 3.6 data storage domain and default dataceter are down.<br> >>> The error in vdsm.log is:<br> >>><br> >>> Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin<br> >>> g<br> >>> Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi<br> >>> st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password':<br> >>> '********', u'port': u''}], options=None)<br> >>> Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non<br> >>> e<br> >>> Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists<br> >>> Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer<br> >>> Traceback (most recent call last):<br> >>> File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer<br> >>> conObj.connect()<br> >>> File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect<br> >>> self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)<br> >>> File "/usr/share/vdsm/storage/storageServer.py", line 323, in options<br> >>> backup_servers_option = self._get_backup_servers_option()<br> >>> File "/usr/share/vdsm/storage/storageServer.py", line 340, in _get_backup_servers_option<br> >>> servers.remove(self._volfileserver)<br> >>> ValueError: list.remove(x): x not in list<br> >>> Thread-6585::DEBUG::2015-11-04 23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain}<br> >>> Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]}<br> >>> Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]}<br> >>> Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state preparing -> state finished<br> >>> _______________________________________________<br> >>> Users mailing list<br> >>> <a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> >>> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> >>><br> >><br> >> _______________________________________________<br> >> Users mailing list<br> >> <a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> >> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> >><br> ><br> > _______________________________________________<br> > Users mailing list<br> > <a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> > <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> </p> </blockquote> <br> <pre class="moz-signature" cols="72"> </pre> </body> </html> --------------020002040508080703030705--
On Fri, Nov 6, 2015 at 10:38 AM, Stefano Danzi <s.danzi@hawai.it> wrote:
I pathced the code for "emergency"....
A safer way is to downgrade vdsm to the previous version.
I can't find how change confguration.
1. Put the gluster domain in maintenance - select the domain in the storage tab - in the "data center" sub tab, click maintenance - the domain will turn to locked, and then to maintenance mode 2. Edit the domain 3. Change the address to the same one configured in gluster (ovirt01...) 4. Activate the domain
But I think that's a bug:
- All work in ovirt 3.5, after upgrade stop working
Yes, it should keep working after an upgrade However, using straightforward setup, like using same server address in both ovirt and gluster will increase the chance that things will continue to work after an upgrade.
- The log show a python exception.
This is good, and makes debugging this issue easy.
I think a thing:
If there are chages on configuration requirements I have to be warned during upgrade, or I have to find a specific error in log.
Correct
...remove something that non exist from a list,
The code assumes that the server address is in the list, so removing it is correct. This assumption is wrong, we will have to change to code to handle this case.
and leave a cryptic python exception as error log, isn't the better solution...
The traceback in the log is very important, without it would be very hard to debug this issue.
Il 06/11/2015 8.12, Nir Soffer ha scritto:
בתאריך 5 בנוב׳ 2015 8:18 אחה״צ, "Stefano Danzi" <s.danzi@hawai.it> כתב:
To temporary solve the problem I patched storageserver.py as suggested on link above.
I would not patch the code but change the configuration.
I can't find a related issue on bugzilla.
Would you file bug about this?
Il 05/11/2015 11.43, Stefano Danzi ha scritto:
My error is related to this message:
http://lists.ovirt.org/pipermail/users/2015-August/034316.html
Il 05/11/2015 0.28, Stefano Danzi ha scritto:
Hello, I have an Ovirt installation with only 1 host and self-hosted engine. My Master Data storage domain is GlusterFS type.
After upgrading to Ovirt 3.6 data storage domain and default dataceter are down. The error in vdsm.log is:
Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin g Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non e Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/share/vdsm/storage/storageServer.py", line 323, in options backup_servers_option = self._get_backup_servers_option() File "/usr/share/vdsm/storage/storageServer.py", line 340, in _get_backup_servers_option servers.remove(self._volfileserver) ValueError: list.remove(x): x not in list Thread-6585::DEBUG::2015-11-04 23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain} Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state preparing -> state finished _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
בתאריך 5 בנוב׳ 2015 1:47 לפנה״צ, "Stefano Danzi" <s.danzi@hawai.it> כתב:
Hello, I have an Ovirt installation with only 1 host and self-hosted engine. My Master Data storage domain is GlusterFS type.
After upgrading to Ovirt 3.6 data storage domain and default dataceter
are down.
The error in vdsm.log is:
Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin g Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None)
Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating
The error below suggest that ovirt and gluster are not configured in the same way, one using a domain name and the other ip address. Can you share the output of gluster volme info On one of the bricks, or on the host (you will need to use --remote-host) Nir directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non
e Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/share/vdsm/storage/storageServer.py", line 323, in options backup_servers_option = self._get_backup_servers_option() File "/usr/share/vdsm/storage/storageServer.py", line 340, in _get_backup_servers_option servers.remove(self._volfileserver) ValueError: list.remove(x): x not in list Thread-6585::DEBUG::2015-11-04 23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain} Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state preparing -> state finished _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a multi-part message in MIME format. --------------060004060906010303090709 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit oVirt is configured to use ovirtbk-mount.hawai.lan but gluster ovirt01.hawai.lan. ovirtbk-mount.hawai.lan is an alias of ovirt01 and is in /etc/hosts Il 06/11/2015 8.01, Nir Soffer ha scritto:
בתאריך 5 בנוב׳ 2015 1:47 לפנה״צ, "Stefano Danzi" <s.danzi@hawai.it <mailto:s.danzi@hawai.it>> כתב:
Hello, I have an Ovirt installation with only 1 host and self-hosted engine. My Master Data storage domain is GlusterFS type.
After upgrading to Ovirt 3.6 data storage domain and default
dataceter are down.
The error in vdsm.log is:
Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin g Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None)
The error below suggest that ovirt and gluster are not configured in the same way, one using a domain name and the other ip address.
Can you share the output of gluster volme info
On one of the bricks, or on the host (you will need to use --remote-host)
Nir
Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non e Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/share/vdsm/storage/storageServer.py", line 323, in options backup_servers_option = self._get_backup_servers_option() File "/usr/share/vdsm/storage/storageServer.py", line 340, in _get_backup_servers_option servers.remove(self._volfileserver) ValueError: list.remove(x): x not in list Thread-6585::DEBUG::2015-11-04 23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain} Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]} Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state preparing -> state finished _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--------------060004060906010303090709 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> oVirt is configured to use ovirtbk-mount.hawai.lan but gluster ovirt01.hawai.lan.<br> ovirtbk-mount.hawai.lan is an alias of ovirt01 and is in /etc/hosts<br> <br> <div class="moz-cite-prefix">Il 06/11/2015 8.01, Nir Soffer ha scritto:<br> </div> <blockquote cite="mid:CAMRbyyvBr3yNRG+ZJF6Bh5641oz7bX9_=2Go5qV16+OOuyN1Jg@mail.gmail.com" type="cite"> <p dir="ltr"><br> בתאריך 5 בנוב׳ 2015 1:47 לפנה״צ, "Stefano Danzi" <<a moz-do-not-send="true" href="mailto:s.danzi@hawai.it"><a class="moz-txt-link-abbreviated" href="mailto:s.danzi@hawai.it">s.danzi@hawai.it</a></a>> כתב:<br> ><br> > Hello,<br> > I have an Ovirt installation with only 1 host and self-hosted engine.<br> > My Master Data storage domain is GlusterFS type.<br> ><br> > After upgrading to Ovirt 3.6 data storage domain and default dataceter are down.<br> > The error in vdsm.log is:<br> ><br> > Thread-6585::DEBUG::2015-11-04 23:55:00,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state init -> state preparin<br> > g<br> > Thread-6585::INFO::2015-11-04 23:55:00,173::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conLi<br> > st=[{u'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499', u'connection': u'ovirtbk-mount.hawai.lan:data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password':<br> > '********', u'port': u''}], options=None)</p> <p dir="ltr">The error below suggest that ovirt and gluster are not configured in the same way, one using a domain name and the other ip address.</p> <p dir="ltr">Can you share the output of<br> gluster volme info</p> <p dir="ltr">On one of the bricks, or on the host (you will need to use --remote-host)</p> <p dir="ltr">Nir</p> <p dir="ltr">> Thread-6585::DEBUG::2015-11-04 23:55:00,235::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data mode: Non<br> > e<br> > Thread-6585::WARNING::2015-11-04 23:55:00,235::fileUtils::152::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/glusterSD/ovirtbk-mount.hawai.lan:data already exists<br> > Thread-6585::ERROR::2015-11-04 23:55:00,235::hsm::2465::Storage.HSM::(connectStorageServer) Could not connect to storageServer<br> > Traceback (most recent call last):<br> > File "/usr/share/vdsm/storage/hsm.py", line 2462, in connectStorageServer<br> > conObj.connect()<br> > File "/usr/share/vdsm/storage/storageServer.py", line 224, in connect<br> > self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)<br> > File "/usr/share/vdsm/storage/storageServer.py", line 323, in options<br> > backup_servers_option = self._get_backup_servers_option()<br> > File "/usr/share/vdsm/storage/storageServer.py", line 340, in _get_backup_servers_option<br> > servers.remove(self._volfileserver)<br> > ValueError: list.remove(x): x not in list<br> > Thread-6585::DEBUG::2015-11-04 23:55:00,235::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {46f55a31-f35f-465c-b3e2-df45c05e06a7: storage.nfsSD.findDomain}<br> > Thread-6585::INFO::2015-11-04 23:55:00,236::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]}<br> > Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`86e72580-fa76-4347-b919-a73970d12682`::finished: {'statuslist': [{'status': 100, 'id': u'dc0eaef1-8494-4e35-abea-80a4e7f37499'}]}<br> > Thread-6585::DEBUG::2015-11-04 23:55:00,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`86e72580-fa76-4347-b919-a73970d12682`::moving from state preparing -> state finished<br> > _______________________________________________<br> > Users mailing list<br> > <a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> > <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> </p> </blockquote> <br> <pre class="moz-signature" cols="72"> </pre> </body> </html> --------------060004060906010303090709--
participants (2)
-
Nir Soffer -
Stefano Danzi