12:27 a.m.
Hi All,
I've got some issues with connecting my oVirt Cluster to my Ceph Cluster via iSCSI.
There are two issues, and I don't know if one is causing the other, if they are
related at all, or if they are two separate, unrelated issues. Let me explain.
The Situation
-------------
- I have a working three node Ceph Cluster (Ceph Quincy on Rocky Linux 8.6)
- The Ceph Cluster has four Storage Pools of between 4 and 8 TB each
- The Ceph Cluster has three iSCSI Gateways
- There is a single iSCSI Target on the Ceph Cluster
- The iSCSI Target has all three iSCSI Gateways attached
- The iSCSI Target has all four Storage Pools attached
- The four Storage Pools have been assigned LUNs 0-3
- I have set up (Discovery) CHAP Authorisation on the iSCSI Target
- I have a working three node self-hosted oVirt Cluster (oVirt v4.5.3 on Rocky Linux 8.6)
- The oVirt Cluster has (in addition to the hosted_storage Storage Domain) three GlusterFS
Storage Domains
- I can ping all three Ceph Cluster Nodes to/from all three oVirt Hosts
- The iSCSI Target on the Ceph Cluster has all three oVirt Hosts Initiators attached
- Each Initiator has all four Ceph Storage Pools attached
- I have set up CHAP Authorisation on the iSCSI Target's Initiators
- The Ceph Cluster Admin Portal reports that all three Initiators are
"logged_in"
- I have previous connected Ceph iSCSI LUNs to the oVirt Cluster successfully (as an
experiment), but had to remove and re-instate them for the "final" version(?).
- The oVirt Admin Portal (ie HostedEngine) reports that Initiators are 1 & 2 (ie oVirt
Hosts 1 & 2) are "logged_in" to all three iSCSI Gateways
- The oVirt Admin Portal reports that Initiator 3 (ie oVirt Host 3) is
"logged_in" to iSCSI Gateways 1 & 2
- I can "force" Initiator 3 to become "logged_in" to iSCSI Gateway 3,
but when I do this it is *not* persistent
- oVirt Hosts 1 & 2 can/have discovered all three iSCSI Gateways
- oVirt Hosts 1 & 2 can/have discovered all four LUNs/Targets on all three iSCSI
Gateways
- oVirt Host 3 can only discover 2 of the iSCSI Gateways
- For Target/LUN 0 oVirt Host 3 can only "see" the LUN provided by iSCSI Gateway
1
- For Targets/LUNs 1-3 oVirt Host 3 can only "see" the LUNs provided by iSCSI
Gateways 1 & 2
- oVirt Host 3 can *not* "see" any of the Targets/LUNs provided by iSCSI Gateway
3
- When I create a new oVirt Storage Domain for any of the four LUNs:
- I am presented with a message saying "The following LUNs are already in
use..."
- I am asked to "Approve operation" via a checkbox, which I do
- As I watch the oVirt Admin Portal I can see the new iSCSI Storage Domain appear in the
Storage Domain list, and then after a few minutes it is removed
- After those few minutes I am presented with this failure message: "Error while
executing action New SAN Storage Domain: Network error during communication with the
Host."
- I have looked in the engine.log and all I could find that was relevant (as far as I
know) was this:
~~~
2022-11-28 19:59:20,506+11 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1)
[77b0c12d] Command 'CreateStorageDomainVDSCommand(HostName = ovirt_node_1.mynet.local,
CreateStorageDomainVDSCommandParameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71',
storageDomain='StorageDomainStatic:{name='data',
id='2a14e4bd-c273-40a0-9791-6d683d145558'}',
args='s0OGKR-80PH-KVPX-Fi1q-M3e4-Jsh7-gv337P'})' execution failed:
VDSGenericException: VDSNetworkException: Message timeout which can be caused by
communication issues
2022-11-28 19:59:20,507+11 ERROR
[org.ovirt.engine.core.bll.storage.domain.AddSANStorageDomainCommand] (default task-1)
[77b0c12d] Command
'org.ovirt.engine.core.bll.storage.domain.AddSANStorageDomainCommand' failed:
EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be caused by
communication issues (Failed with error VDS_NETWORK_ERROR and code 5022)
~~~
I cannot see/detect any "communication issue" - but then again I'm not 100%
sure what I should be looking for
I have looked on-line for an answer, and apart from not being able to get past Red
Hat's "wall" to see the solutions that they have, all I could find that was
relevant was this:
https://lists.ovirt.org/archives/list/devel@ovirt.org/thread/AVLORQNOLJHR...
. If this *is* relevant then there is not enough context here for me to proceed (ie/eg
*where* (which host/vm) should that command be run?).
I also found (for a previous version of oVirt) notes about modifying the Postgres DB
manual to resolve a similar issue. While I am more than comfortable doing this (I've
been an SQL DBA for well over 20 years) this seems like asking for trouble - at least
until I hear back from the oVirt Devs that this is OK to do - and of course, I'll need
the relevant commands / locations / authorisations to get into the DB.
Questions
---------
- Are the two issues (oVirt Host 3 not having a full picture of the Ceph iSCSI environment
and the oVirt iSCSI Storage Domain creation failure) related?
- Do I need to "refresh" the iSCSI info on the oVirt Hosts, and if so, how do I
do this?
- Do I need to "flush" the old LUNs from the oVirt Cluster, and if so, how do I
do this?
- Where else should I be looking for info in the logs (& which logs)?
- Does *anyone* have any other ideas how to resolve the situation - especially when using
the Ceph iSCII Gateways?
Thanks in advance
Cheers
Dulux-Oz
6:21 a.m.
Matthew, good morning.
Is iSCSI Target configured with ACL?
Do all Gateways have the same amount of active sessions? It could be that
some Gateway has crashed the sessions (specifically gateway 3).
If you are not actually using Storage Domain on iSCSI, I recommend the
following:
1- Logout through oVirt
2- Check if there is still an initiator in the multipath on each oVirt Host
3- Log out of all sessions and delete through iscsiadm on each oVirt Host
4- Check if there is still an active session in CEPH
5- Restart all Gateway Daemons in CEPH, it may take a while if there is a
stuck session
6- Try to perform Discovery again through oVirt
Em ter., 29 de nov. de 2022 às 03:28, Matthew J Black <
matthew(a)peregrineit.net> escreveu:
Hi All,
I've got some issues with connecting my oVirt Cluster to my Ceph Cluster
via iSCSI. There are two issues, and I don't know if one is causing the
other, if they are related at all, or if they are two separate, unrelated
issues. Let me explain.
The Situation
-------------
- I have a working three node Ceph Cluster (Ceph Quincy on Rocky Linux 8.6)
- The Ceph Cluster has four Storage Pools of between 4 and 8 TB each
- The Ceph Cluster has three iSCSI Gateways
- There is a single iSCSI Target on the Ceph Cluster
- The iSCSI Target has all three iSCSI Gateways attached
- The iSCSI Target has all four Storage Pools attached
- The four Storage Pools have been assigned LUNs 0-3
- I have set up (Discovery) CHAP Authorisation on the iSCSI Target
- I have a working three node self-hosted oVirt Cluster (oVirt v4.5.3 on
Rocky Linux 8.6)
- The oVirt Cluster has (in addition to the hosted_storage Storage Domain)
three GlusterFS Storage Domains
- I can ping all three Ceph Cluster Nodes to/from all three oVirt Hosts
- The iSCSI Target on the Ceph Cluster has all three oVirt Hosts
Initiators attached
- Each Initiator has all four Ceph Storage Pools attached
- I have set up CHAP Authorisation on the iSCSI Target's Initiators
- The Ceph Cluster Admin Portal reports that all three Initiators are
"logged_in"
- I have previous connected Ceph iSCSI LUNs to the oVirt Cluster
successfully (as an experiment), but had to remove and re-instate them for
the "final" version(?).
- The oVirt Admin Portal (ie HostedEngine) reports that Initiators are 1 &
2 (ie oVirt Hosts 1 & 2) are "logged_in" to all three iSCSI Gateways
- The oVirt Admin Portal reports that Initiator 3 (ie oVirt Host 3) is
"logged_in" to iSCSI Gateways 1 & 2
- I can "force" Initiator 3 to become "logged_in" to iSCSI Gateway 3,
but
when I do this it is *not* persistent
- oVirt Hosts 1 & 2 can/have discovered all three iSCSI Gateways
- oVirt Hosts 1 & 2 can/have discovered all four LUNs/Targets on all three
iSCSI Gateways
- oVirt Host 3 can only discover 2 of the iSCSI Gateways
- For Target/LUN 0 oVirt Host 3 can only "see" the LUN provided by iSCSI
Gateway 1
- For Targets/LUNs 1-3 oVirt Host 3 can only "see" the LUNs provided by
iSCSI Gateways 1 & 2
- oVirt Host 3 can *not* "see" any of the Targets/LUNs provided by iSCSI
Gateway 3
- When I create a new oVirt Storage Domain for any of the four LUNs:
- I am presented with a message saying "The following LUNs are already
in use..."
- I am asked to "Approve operation" via a checkbox, which I do
- As I watch the oVirt Admin Portal I can see the new iSCSI Storage
Domain appear in the Storage Domain list, and then after a few minutes it
is removed
- After those few minutes I am presented with this failure message:
"Error while executing action New SAN Storage Domain: Network error during
communication with the Host."
- I have looked in the engine.log and all I could find that was relevant
(as far as I know) was this:
~~~
2022-11-28 19:59:20,506+11 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand]
(default task-1) [77b0c12d] Command 'CreateStorageDomainVDSCommand(HostName
= ovirt_node_1.mynet.local,
CreateStorageDomainVDSCommandParameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71',
storageDomain='StorageDomainStatic:{name='data',
id='2a14e4bd-c273-40a0-9791-6d683d145558'}',
args='s0OGKR-80PH-KVPX-Fi1q-M3e4-Jsh7-gv337P'})' execution failed:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
2022-11-28 19:59:20,507+11 ERROR
[org.ovirt.engine.core.bll.storage.domain.AddSANStorageDomainCommand]
(default task-1) [77b0c12d] Command
'org.ovirt.engine.core.bll.storage.domain.AddSANStorageDomainCommand'
failed: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues (Failed with error VDS_NETWORK_ERROR and
code 5022)
~~~
I cannot see/detect any "communication issue" - but then again I'm not
100% sure what I should be looking for
I have looked on-line for an answer, and apart from not being able to get
past Red Hat's "wall" to see the solutions that they have, all I could
find
that was relevant was this:
https://lists.ovirt.org/archives/list/devel@ovirt.org/thread/AVLORQNOLJHR...
. If this *is* relevant then there is not enough context here for me to
proceed (ie/eg *where* (which host/vm) should that command be run?).
I also found (for a previous version of oVirt) notes about modifying the
Postgres DB manual to resolve a similar issue. While I am more than
comfortable doing this (I've been an SQL DBA for well over 20 years) this
seems like asking for trouble - at least until I hear back from the oVirt
Devs that this is OK to do - and of course, I'll need the relevant commands
/ locations / authorisations to get into the DB.
Questions
---------
- Are the two issues (oVirt Host 3 not having a full picture of the Ceph
iSCSI environment and the oVirt iSCSI Storage Domain creation failure)
related?
- Do I need to "refresh" the iSCSI info on the oVirt Hosts, and if so, how
do I do this?
- Do I need to "flush" the old LUNs from the oVirt Cluster, and if so, how
do I do this?
- Where else should I be looking for info in the logs (& which logs)?
- Does *anyone* have any other ideas how to resolve the situation -
especially when using the Ceph iSCII Gateways?
Thanks in advance
Cheers
Dulux-Oz
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MCOJD4R6PS4...
3:57 a.m.
Hi Murlio,
Thanks for that.
Yes, iSCSI Target is configured with ACL
Yes, all the Gateways have the same amount of sessions.
Yes, I followed your 6 step-suggestion.
So what happened is - and I won't go into all the boring details of how, why, and how
- I discovered the remains of another iSCSI Target on the 2nd Gateway. Installing and
using tergetcli I removed this old target, which allowed me to then remove and then
re-attach the 2nd Gateway to the "real" target (using the Ceph GUI - resetting
the gateways to be identical once again), and then ran through the 6-steps. I now have all
three oVirt Hosts logged into all three Ceph iSCSI Gateways, with each of the four LUNs
reporting connectivity to all three Gateways, etc.
So that's the Gateways-connectivity issue resolved.
However, I still have the other issue: I still come up with the "The following LUNs
are already in use..." message and I still have the same results - ie log messages
re: connectivity issues with the new (iSCSI) Storage domain briefly appearing and then
being removed again (see 1st post in this thread).
This happens with all four LUNs no matter which of the three Gateways I point oVirt Admin
GUI at. (Yes, I tried this twelve times to cover the twelve possible combinations).
So, anyone got any further ideas? :-)
Cheers
Dulux-Oz
10:36 p.m.
Hi All,
So I just tried to create an iSCSI Storage Domain again, and got the same results.
I grabbed the relevant(?) part of the engine.log file and placed it into Dropbox:
https://www.dropbox.com/s/0kt9z21zkewvn3j/extract_engine.log?dl=0
I can see the ERRORs, but I can't figure out what's causing them, because I
can't see anything wrong with the system.
Would someone be so kind as to take a look and let me know what I'm missing, please?
Also, if there's other logs I should be looking at could someone please let me know
that as well - I had a root-around and I couldn't see anything else relevant, but
then, I'm not one of the devs who knows the system inside and out.
Thanks in advance
Dulux-Oz
1:04 a.m.
Hi All,
So, on further investigation/experimentation, it seems that each individual iSCSI Storage
Domain is being created (there is visual evidence from watching the Ceph Monitoring
Portal, and LVM volumes are being created on the selected LUN/Host) but it *appears* that,
for some reason, vdsm is "timing out" (or, at least, that's my
interpretation of what is happening) and thus resetting the Storage Domains back to the
pre-creation state.
So it appears that the fact that there is a previous LUN on the Ceph Pool/Disk is *not* my
underlying issue.
So my question is: How would it be if I adjusted the following settings (via
engine-config), which I obtained from this thread:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/LRZLORSREVEY...
- TimeoutToResetVdsInSeconds
- VDSAttemptsToResetCount
- VdsRecoveryTimeoutInMintues
- VdsRefreshRate
- vdsTimeout
Anyone have any opinions (on this list and/or other settings)?
Cheers
Dulux-Oz
3:45 p.m.
Before taking that approach.Have you tried to run iscsiadm manually from host3 in order to
identify if it sees the targets and is able to login?
I have experience only with HA iSCSI and I'm not sure I understand this 3 gateway
setup. Does this mean that all Ceph hosts play the role of the iscsi target ?
Also, run tcpdump on the 3rd (problematic ) oVirt node and initiate a new iSCSI
setup.Maybe it can give a clue what is going on.
Best Regards,Strahil Nikolov
On Fri, Dec 2, 2022 at 9:04, Matthew J Black<matthew(a)peregrineit.net> wrote: Hi
All,
So, on further investigation/experimentation, it seems that each individual iSCSI Storage
Domain is being created (there is visual evidence from watching the Ceph Monitoring
Portal, and LVM volumes are being created on the selected LUN/Host) but it *appears* that,
for some reason, vdsm is "timing out" (or, at least, that's my
interpretation of what is happening) and thus resetting the Storage Domains back to the
pre-creation state.
So it appears that the fact that there is a previous LUN on the Ceph Pool/Disk is *not* my
underlying issue.
So my question is: How would it be if I adjusted the following settings (via
engine-config), which I obtained from this thread:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/LRZLORSREVEY...
- TimeoutToResetVdsInSeconds
- VDSAttemptsToResetCount
- VdsRecoveryTimeoutInMintues
- VdsRefreshRate
- vdsTimeout
Anyone have any opinions (on this list and/or other settings)?
Cheers
Dulux-Oz
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z73EIUTOEQE...
3:09 a.m.
Yes - as I said, I've got all three oVirt Hosts talking to all three Ceph iSCSI
Gateways, and can see all four LUNs (via each of the three Gateways). Just to be 100%
clear: yes, this means that all three oVirt Hosts are logged into the Ceph Cluster (via
all three of the iSCSI Gateways).
Yes, I believe that the way it works (but *please* do not take this as gospel) is that
each iSCSI Gateway - and it is recommended that a Ceph Cluster have 2-4 of them) - plays
the part of of the Target ie there is one Target with each of the 3 Gateways as
"subordinate", as are the four LUNs and the three oVirt Initiators, which in
turn show each LUN as subordinate to them, as well. Very confusing on initial look-see,
but it sort of makes sense, I suppose.
However, please see my next/other post in this thread - and thanks for the help.
3:13 a.m.
Hi All,
OK, so an update
TL;DR: I got it all working.
So it turns out that I had a typo in my /etc/multipath/conf.d/host.conf file, and because
all three Ceph iSCSI Gateways are identical (except for IP, Hostname, etc) I did a basic
copy-paste of the one file to all three Gateways. After discovering this (and fixing it),
I have now attached my four LUNs to the oVirt Cluster and everything *seems* AOK, both
from the oVirt-side and the Ceph-side, so I think we're all good and so we can mark
this thread as solved.
Thanks to everyone who had a hand it making suggestions and helping out - I really do
appreciate it
Cheers
Dulux-Oz
886
days inactive
892
days old
7 comments
3 participants
participants (3)
-
Matthew J Black -
Murilo Morais -
Strahil Nikolov