vdsm is started on node1 now. Now I have this error : root@node1 ~]# vdsClient -s 0 getConnectedStoragePoolsList Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2538, in <module> code, message = commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClient.py", line 1150, in getConnectedStoragePoolsList pools = self.s.getConnectedStoragePoolsList() File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1235, in request self.send_content(h, request_body) File "/usr/lib64/python2.6/xmlrpclib.py", line 1349, in send_content connection.endheaders() File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders self._send_output() File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output self.send(msg) File "/usr/lib64/python2.6/httplib.py", line 739, in send self.connect() File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 195, in connect cert_reqs=self.cert_reqs) File "/usr/lib64/python2.6/ssl.py", line 342, in wrap_socket suppress_ragged_eofs=suppress_ragged_eofs) File "/usr/lib64/python2.6/ssl.py", line 120, in __init__ self.do_handshake() File "/usr/lib64/python2.6/ssl.py", line 279, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed [root@node1 ~]# 2014-04-11 17:18 GMT+02:00 Dafna Ron <dron@redhat.com>:
Can you please restart vdsm on the second host and see if it can connect to the pool and if the errors are gone once you restart?
Thanks,
Dafna
On 04/11/2014 04:00 PM, James James wrote:
I am using a iscsi storage
On node2 : [root@node2 ~]# vdsClient -s 0 getConnectedStoragePoolsList 9ca706b7-bc4f-4f88-bd80-a1203e4bd2ce
On node1 : [root@node1 ~]# vdsClient -s 0 getConnectedStoragePoolsList Connection to ovirt-node1.example.com:54321 <http://ovirt-node1.example. com:54321> refused
2014-04-11 16:47 GMT+02:00 Dafna Ron <dron@redhat.com <mailto: dron@redhat.com>>:
This is the error:
Thread-26::ERROR::2014-04-11 15:52:39,788::task::866::TaskManager.Task::(_setError) Task=`e69fe141-ea0c-4ec0-8c7b-a597ce1bc3af`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3546, in fenceSpmStorage pool.invalidateMetadata() AttributeError: 'StoragePool' object has no attribute 'invalidateMetadata'
What storage are you using (iscsi/nfs)
can you please run:
vdsClient -s 0 getConnectedStoragePoolsList
On 04/11/2014 02:56 PM, James James wrote:
Here is the second host log ...
2014-04-11 15:43 GMT+02:00 Dafna Ron <dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>:
can you attach the engine log and the vdsm log from the second host?
The SPM cannot be released because the master storage domain is not visible from the second host.
Dafna
On 04/11/2014 02:38 PM, James James wrote:
I try to follow the Daria's advices .. I restarted my engine to clear the cache ....
but now I am facing a new problem. node1 is the SPM and I have this error message :
Manual fence did not revoke the selected SPM (node1) since the master storage domain was not active or could not use another host for the fence operation.
....
2014-04-11 15:16 GMT+02:00 Dafna Ron <dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>:
I think there might be a chance it's related to issues we were seeing with engine cache.
James, would you mind testing that? restart of ovirt-engine process should clear the cache. once you restart, log in to the webadmin and if the host is still spm and none responsive, can you try confirm host reboot again?
If that still fails, can you please attach the engine log?
Thanks. Dafna
On 04/11/2014 02:13 PM, Itamar Heim wrote:
On 04/11/2014 04:10 PM, James James wrote:
Nothing new. node1 can't release SPM ... :(
allon/federico - thoughts? confirm host shutdown should release SPM for a non-responsive node
2014-04-11 14:49 GMT+02:00 Dafna Ron <dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>>:
confirm host has been rebooted should release the SPM :)
On 04/11/2014 01:23 PM, James James wrote:
2014-04-11 14:08 GMT+02:00 Dafna Ron <dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>>>:
James, Please answer the user's list as well as to me so that other people can participate as well :)
Oups ... I will do that ..
did you try to press the "confirm host has been rebooted" (right click)
Yes but same problem. node1 cannot be in maintenance mode.
node1 is SPM . I will make node1 release the SPM ressource.
On 04/11/2014 12:41 PM, James James wrote:
2014-04-11 13:16 GMT+02:00 Dafna Ron <dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>
<mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>>>>:
what is the error message you get when you try to put the host in maintenance?
I have this message :
Error while executing action: Cannot switch Host to Maintenance mode. Host is Storage Pool Manager and is in Non Responsive state. - If power management is configured, engine will try to fence the host automatically. - Otherwise, either bring the node back up, or release the SPM resource. To do so, verify that the node is really down by right clicking on the host and confirm that the node was shutdown manually.
are there any running vm's reported?
No there is no VM running on this host
Try to press the "confirm host has been rebooted" button and than see if you can put the host in maintenance.
If that fails, select the host, in the general tab you will get the re-install link.
I am running ovirt 3.4.0-1. I down't know where is the re-install link but I can't see it.
try to re-install, when install fails the host should change status to failed installation.
On 04/11/2014 12:09 PM, James James wrote:
No, I can't ....
2014-04-11 12:12 GMT+02:00 Dafna Ron <dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>>
<mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>> <mailto:dron@redhat.com <mailto:dron@redhat.com> <mailto:dron@redhat.com <mailto:dron@redhat.com>>>>>>>>:
can you put the host in maintenance?
On 04/11/2014 10:43 AM, James James wrote:
I can't delete the old node because it is in Non Responsive state. The remove button is stil blank .
In the engine.log I've got this log : 2014-04-11 11:40:45,911 ERROR
[org.ovirt.engine.core.__vdsbroker.vdsbroker.__ GetCapabilitiesVDSCommand] (DefaultQuartzScheduler___Worker-88) Command GetCapabilitiesVDSCommand(__HostName = node1, HostId = 36fb6df3-c2c2-4133-86ac-__fe50b99ee2e3, vds=Host[node1]) execution failed. Exception: VDSNetworkException:
sun.security.provider.__certpath.__SunCertPathBuilderException: unable to find valid certification path to requested target 2014-04-11 11:40:48,943 ERROR
[org.ovirt.engine.core.__vdsbroker.vdsbroker.__ GetCapabilitiesVDSCommand] (DefaultQuartzScheduler___Worker-12) Command GetCapabilitiesVDSCommand(__HostName = node1, HostId = 36fb6df3-c2c2-4133-86ac-__fe50b99ee2e3, vds=Host[node1]) execution failed. Exception: VDSNetworkException:
sun.security.provider.__certpath.__SunCertPathBuilderException: unable to find valid certification path to requested target
2014-04-11 11:06 GMT+02:00 Alon Bar-Lev < alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>>>
<mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>>>>
<mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com> <mailto:alonbl@redhat.com <mailto:alonbl@redhat.com>>>>>>>>>__:
----- Original Message ----- > From: "James James" <jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>>>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com> <mailto:jreg2k@gmail.com <mailto:jreg2k@gmail.com>>>