Re: [Users] very odd permission problem

On 6-9-2013 12:34, Alessandro Bianchi wrote:
Hi all
I'm running 3.2 on several Fedora 18 nodes
One of them has a local storage running 4 VMs
Today the UPS crashed and host was rebboted after UPS replacement
None of the VM's were able to be started
I tried to put the Host in maintenance and reinstalled it, but this didn't give any result
Digging into the logs I discovered the following error:
The first was of this kind (on every VM)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2630, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirtError: errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not use private key file qemu-kvm: failed to initialize spice server
Thread-564::DEBUG::2013-09-06 11:31:32,814::vm::1065::vm.Vm::(setDownStatus) vmId=`49d84915-490b-497d-a3f8-c7dac7485281`::Changed state to Down: errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not use private key file qemu-kvm: failed to initialize spice server
The private key was marked 440 as permission owned by vdsm user and kvm group
I had to change it to 444 to allow everyone to read it
After that I had for every VM the following error:
could not open disk image /rhev/data-center/3935800a-abe4-406d-84a1-4c3c0b915cce/6818de31-5cda-41d0-a41a-681230a409ba/images/54144c03-5057-462e-8275-6ab386ae8c5a/01298998-32d5-44c2-b5d1-91be1316ed19: Permission denied
Disks were owned by vdsm:kvm with 660 permission
I had to relax this to 666 to enable the VMs to start
Has anyone faced this kind f problem before?
Yes, me.
Any hint about what may have caused this odd problem?
yum update.
I updated one of my hosts and after that that host couldn't start VMs anymore with exact the same errors. See thread 'Starting VM error' by Shaun Glass. I tried a couple of things but not making world readable those files. Will probably restore a backup and try it. I added the virt-preview repo for F18 and updated qemu/libvirt which also solved the problem. The difference between the updated and not updated host were really minimal. See the thead for logs.
Regards,
Joop Thank you for your very quick answer
I suspected the same thing ! I'll update libvirt and revert the permission changes Best regards Alessandro

Alessandro Bianchi wrote:
On 6-9-2013 12:34, Alessandro Bianchi wrote:
Hi all
I'm running 3.2 on several Fedora 18 nodes
One of them has a local storage running 4 VMs
Today the UPS crashed and host was rebboted after UPS replacement
None of the VM's were able to be started
I tried to put the Host in maintenance and reinstalled it, but this didn't give any result
Digging into the logs I discovered the following error:
The first was of this kind (on every VM)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2630, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirtError: errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not use private key file qemu-kvm: failed to initialize spice server
Thread-564::DEBUG::2013-09-06 11:31:32,814::vm::1065::vm.Vm::(setDownStatus) vmId=`49d84915-490b-497d-a3f8-c7dac7485281`::Changed state to Down: errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not use private key file qemu-kvm: failed to initialize spice server
The private key was marked 440 as permission owned by vdsm user and kvm group
I had to change it to 444 to allow everyone to read it
After that I had for every VM the following error:
could not open disk image /rhev/data-center/3935800a-abe4-406d-84a1-4c3c0b915cce/6818de31-5cda-41d0-a41a-681230a409ba/images/54144c03-5057-462e-8275-6ab386ae8c5a/01298998-32d5-44c2-b5d1-91be1316ed19:
Permission denied
Disks were owned by vdsm:kvm with 660 permission
I had to relax this to 666 to enable the VMs to start
Has anyone faced this kind f problem before?
Yes, me.
Any hint about what may have caused this odd problem?
yum update.
I updated one of my hosts and after that that host couldn't start VMs anymore with exact the same errors. See thread 'Starting VM error' by Shaun Glass. I tried a couple of things but not making world readable those files. Will probably restore a backup and try it. I added the virt-preview repo for F18 and updated qemu/libvirt which also solved the problem. The difference between the updated and not updated host were really minimal. See the thead for logs.
Regards,
Joop Thank you for your very quick answer
I suspected the same thing !
I'll update libvirt and revert the permission changes
That will give you way way newer libvirt/qemu than you probably want. I would keep the permission changes and hope that one of the following updates to either libvirt/qemu fixes this problem. Joop

On Fri, Sep 06, 2013 at 04:05:13PM +0200, Joop wrote:
Alessandro Bianchi wrote:
On 6-9-2013 12:34, Alessandro Bianchi wrote:
Hi all
I'm running 3.2 on several Fedora 18 nodes
One of them has a local storage running 4 VMs
Today the UPS crashed and host was rebboted after UPS replacement
None of the VM's were able to be started
I tried to put the Host in maintenance and reinstalled it, but this didn't give any result
Digging into the logs I discovered the following error:
The first was of this kind (on every VM)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2630, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirtError: errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not use private key file qemu-kvm: failed to initialize spice server
Thread-564::DEBUG::2013-09-06 11:31:32,814::vm::1065::vm.Vm::(setDownStatus) vmId=`49d84915-490b-497d-a3f8-c7dac7485281`::Changed state to Down: errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not use private key file qemu-kvm: failed to initialize spice server
The private key was marked 440 as permission owned by vdsm user and kvm group
I had to change it to 444 to allow everyone to read it
After that I had for every VM the following error:
could not open disk image /rhev/data-center/3935800a-abe4-406d-84a1-4c3c0b915cce/6818de31-5cda-41d0-a41a-681230a409ba/images/54144c03-5057-462e-8275-6ab386ae8c5a/01298998-32d5-44c2-b5d1-91be1316ed19:
Permission denied
Disks were owned by vdsm:kvm with 660 permission
I had to relax this to 666 to enable the VMs to start
Has anyone faced this kind f problem before?
Yes, me.
Any hint about what may have caused this odd problem?
yum update.
I updated one of my hosts and after that that host couldn't start VMs anymore with exact the same errors. See thread 'Starting VM error' by Shaun Glass. I tried a couple of things but not making world readable those files. Will probably restore a backup and try it. I added the virt-preview repo for F18 and updated qemu/libvirt which also solved the problem. The difference between the updated and not updated host were really minimal. See the thead for logs.
Regards,
Joop Thank you for your very quick answer
I suspected the same thing !
I'll update libvirt and revert the permission changes
That will give you way way newer libvirt/qemu than you probably want. I would keep the permission changes and hope that one of the following updates to either libvirt/qemu fixes this problem.
Joop, I'm sorry that I have many requests and few answers, but if indeed the problem is related to a version of libvirt/qemu, would yould you try to reproduce it outside ovirt? I mean, in your working/non-working hosts, could you create a vdsm:kvm- owned image, and try to run it from virsh (using vdsm@ovirt user and the ever-so-secret password listed in vdsm/libvirt_password)? What happens if you chown your image to vdsm:qemu? (keeping mode as 660) What's `groups qemu` on your hosts? Could you attach gdb to the short-living qemu process, and run getgroups(2) on it? Dan.

Alessandro Bianchi wrote:
On 6-9-2013 12:34=2C Alessandro Bianchi wrote:
Hi all
I'm running 3.2 on several Fedora 18 nodes
One of them has a local storage running 4 VMs
Today the UPS crashed and host was rebboted after UPS replacement
None of the VM's were able to be started
I tried to put the Host in maintenance and reinstalled it=2C but thi= s didn't give any result
Digging into the logs I discovered the following error:
The first was of this kind (on every VM)
File "/usr/lib64/python2.7/site-packages/libvirt.py"=2C line 2630= =2C in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed'= =2C conn=3Dself) libvirtError: errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not use private key file qemu-kvm: failed to initialize spice server
Thread-564::DEBUG::2013-09-06 11:31:32=2C814::vm::1065::vm.Vm::(setDownStatus) vmId=3D`49d84915-490b-497d-a3f8-c7dac7485281`::Changed state to Down= : errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could no= t use private key file qemu-kvm: failed to initialize spice server
The private key was marked 440 as permission owned by vdsm user and kvm group
I had to change it to 444 to allow everyone to read it
After that I had for every VM the following error:
could not open disk image /rhev/data-center/3935800a-abe4-406d-84a1-4c3c0b915cce/6818de31-5cda= -41d0-a41a-681230a409ba/images/54144c03-5057-462e-8275-6ab386ae8c5a/0129899= 8-32d5-44c2-b5d1-91be1316ed19:
Permission denied
Disks were owned by vdsm:kvm with 660 permission
I had to relax this to 666 to enable the VMs to start
Has anyone faced this kind f problem before?
Yes=2C me.
Any hint about what may have caused this odd problem?
yum update.
I updated one of my hosts and after that that host couldn't start VMs anymore with exact the same errors. See thread 'Starting VM error' by Shaun Glass. I tried a couple of things but not making world readable those files. Will probably restore a backup and try it. I added the virt-preview repo for F18 and updated qemu/libvirt which also solved the problem. The difference between the updated and not updated host were really minimal. See the thead for logs.
Regards=2C
Joop Thank you for your very quick answer
I suspected the same thing !
I'll update libvirt and revert the permission changes
That will give you way way newer libvirt/qemu than you probably want. I would keep the permission changes and hope that one of the following updates to either libvirt/qemu fixes this problem. =20 Joop=2C I'm sorry that I have many requests and few answers=2C but if ind= eed
Date: Fri=2C 6 Sep 2013 22:05:05 +0100 From: danken@redhat.com To: jvdwege@xs4all.nl CC: users@ovirt.org Subject: Re: [Users] very odd permission problem =20 On Fri=2C Sep 06=2C 2013 at 04:05:13PM +0200=2C Joop wrote: the problem is related to a version of libvirt/qemu=2C would yould you tr= y to reproduce it outside ovirt? =20 I mean=2C in your working/non-working hosts=2C could you create a vdsm:kv= m- owned image=2C and try to run it from virsh (using vdsm@ovirt user and th= e ever-so-secret password listed in vdsm/libvirt_password)? =20 What happens if you chown your image to vdsm:qemu? (keeping mode as 660) =20 What's `groups qemu` on your hosts? =20 Could you attach gdb to the short-living qemu process=2C and run getgroups(2) on it? =20 Dan. > > >>>errore interno process exited while connecting to monitor:
((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could no= t use private key file I just ran into this myself on a fresh 3.2.2 in= stall. Enabling the virt-preview repo and doing a yum update fixed the spic= e-warning issue and the VMs started right up. That might help you Joop. An= other issue=2C ovirt-engine-sdk is newer on the fedora repos than in the ov= irt-repo. The fedora one caused issues (I forget which error at the moment)= =2C so I had to disable the fedora repos=2C remove ovirt-engine-sdk=2C and =
--_37a82616-95c5-4643-a2ad-2d119d724eb4_ Content-Type: multipart/alternative; boundary="_9ac12577-111e-4331-a941-8ee28a6f041e_" --_9ac12577-111e-4331-a941-8ee28a6f041e_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable =20 then reinstall it from the ovirt-repo. =20
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users =
>=3B >=3B >=3B>=3B>=3Bmonitor: ((null):5034): Spice-Warning **: = reds.c:3247:reds_init_ssl:<br>>=3B >=3B >=3B>=3B>=3BCould not use=
--_9ac12577-111e-4331-a941-8ee28a6f041e_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <style><!-- .hmmessage P { margin:0px=3B padding:0px } body.hmmessage { font-size: 12pt=3B font-family:Calibri } --></style></head> <body class=3D'hmmessage'><div dir=3D'ltr'><br> =3B<BR><div>>=3B Date= : Fri=2C 6 Sep 2013 22:05:05 +0100<br>>=3B From: danken@redhat.com<br>>= =3B To: jvdwege@xs4all.nl<br>>=3B CC: users@ovirt.org<br>>=3B Subject: = Re: [Users] very odd permission problem<br>>=3B <br>>=3B On Fri=2C Sep = 06=2C 2013 at 04:05:13PM +0200=2C Joop wrote:<br>>=3B >=3B Alessandro B= ianchi wrote:<br>>=3B >=3B >=3B>=3BOn 6-9-2013 12:34=2C Alessandro = Bianchi wrote:<br>>=3B >=3B >=3B>=3B>=3BHi all<br>>=3B >=3B &= gt=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BI'm running 3.2 on sev= eral Fedora 18 nodes<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B &= gt=3B>=3B>=3BOne of them has a local storage running 4 VMs<br>>=3B &g= t=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BToday the UPS cr= ashed and host was rebboted after UPS replacement<br>>=3B >=3B >=3B&g= t=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BNone of the VM's were able to= be started<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>= =3B>=3BI tried to put the Host in maintenance and reinstalled it=2C but t= his<br>>=3B >=3B >=3B>=3B>=3Bdidn't give any result<br>>=3B >= =3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BDigging into the = logs I discovered the following error:<br>>=3B >=3B >=3B>=3B>=3B<= br>>=3B >=3B >=3B>=3B>=3BThe first was of this kind (on every VM)= <br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3B F= ile "/usr/lib64/python2.7/site-packages/libvirt.py"=2C line 2630=2C in<br>&= gt=3B >=3B >=3B>=3B>=3BcreateXML<br>>=3B >=3B >=3B>=3B>= =3B if ret is None:raise libvirtError('virDomainCreateXML() failed'=2C<= br>>=3B >=3B >=3B>=3B>=3Bconn=3Dself)<br>>=3B >=3B >=3B>= =3B>=3BlibvirtError: errore interno process exited while connecting to<br= private key file<br>>=3B >=3B >=3B>=3B>=3Bqemu-kvm: failed to in= itialize spice server<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B = >=3B>=3B>=3BThread-564::DEBUG::2013-09-06<br>>=3B >=3B >=3B>= =3B>=3B11:31:32=2C814::vm::1065::vm.Vm::(setDownStatus)<br>>=3B >=3B = >=3B>=3B>=3BvmId=3D`49d84915-490b-497d-a3f8-c7dac7485281`::Changed st= ate to Down:<br>>=3B >=3B >=3B>=3B>=3Berrore interno process exit= ed while connecting to monitor:<br>>=3B >=3B >=3B>=3B>=3B((null):= 5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not<br>>=3B >= =3B >=3B>=3B>=3Buse private key file<br>>=3B >=3B >=3B>=3B>= =3Bqemu-kvm: failed to initialize spice server<br>>=3B >=3B >=3B>= =3B>=3B<br>>=3B >=3B >=3B>=3B>=3BThe private key was marked 440= as permission owned by vdsm user and<br>>=3B >=3B >=3B>=3B>=3Bkv= m group<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B&g= t=3BI had to change it to 444 to allow everyone to read it<br>>=3B >=3B= >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BAfter that I had for= every VM the following error:<br>>=3B >=3B >=3B>=3B>=3B<br>>= =3B >=3B >=3B>=3B>=3Bcould not open disk image<br>>=3B >=3B >= =3B>=3B>=3B/rhev/data-center/3935800a-abe4-406d-84a1-4c3c0b915cce/6818d= e31-5cda-41d0-a41a-681230a409ba/images/54144c03-5057-462e-8275-6ab386ae8c5a= /01298998-32d5-44c2-b5d1-91be1316ed19:<br>>=3B >=3B >=3B>=3B>=3B<= br>>=3B >=3B >=3B>=3B>=3BPermission denied<br>>=3B >=3B >= =3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BDisks were owned by vdsm= :kvm with 660 permission<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >= =3B >=3B>=3B>=3BI had to relax this to 666 to enable the VMs to start= <br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BHas= anyone faced this kind f problem before?<br>>=3B >=3B >=3B>=3B>= =3B<br>>=3B >=3B >=3B>=3BYes=2C me.<br>>=3B >=3B >=3B>=3B&g= t=3BAny hint about what may have caused this odd problem?<br>>=3B >=3B = >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3Byum update.<br>>=3B >= =3B >=3B>=3B<br>>=3B >=3B >=3B>=3BI updated one of my hosts and= after that that host couldn't start VMs<br>>=3B >=3B >=3B>=3Banymo= re with exact the same errors. See thread 'Starting VM error' by<br>>=3B = >=3B >=3B>=3BShaun Glass. I tried a couple of things but not making w= orld readable<br>>=3B >=3B >=3B>=3Bthose files. Will probably resto= re a backup and try it.<br>>=3B >=3B >=3B>=3BI added the virt-previ= ew repo for F18 and updated qemu/libvirt which<br>>=3B >=3B >=3B>= =3Balso solved the problem.<br>>=3B >=3B >=3B>=3BThe difference bet= ween the updated and not updated host were really<br>>=3B >=3B >=3B&g= t=3Bminimal. See the thead for logs.<br>>=3B >=3B >=3B>=3B<br>>= =3B >=3B >=3B>=3BRegards=2C<br>>=3B >=3B >=3B>=3B<br>>=3B &= gt=3B >=3B>=3BJoop<br>>=3B >=3B >=3BThank you for your very quick= answer<br>>=3B >=3B >=3B<br>>=3B >=3B >=3BI suspected the same= thing !<br>>=3B >=3B >=3B<br>>=3B >=3B >=3BI'll update libvirt= and revert the permission changes<br>>=3B >=3B >=3B<br>>=3B >=3B= That will give you way way newer libvirt/qemu than you probably<br>>=3B = >=3B want. I would keep the permission changes and hope that one of the<b= r>>=3B >=3B following updates to either libvirt/qemu fixes this problem= .<br>>=3B <br>>=3B Joop=2C I'm sorry that I have many requests and few = answers=2C but if indeed<br>>=3B the problem is related to a version of l= ibvirt/qemu=2C would yould you try<br>>=3B to reproduce it outside ovirt?= <br>>=3B <br>>=3B I mean=2C in your working/non-working hosts=2C could = you create a vdsm:kvm-<br>>=3B owned image=2C and try to run it from virs= h (using vdsm@ovirt user and the<br>>=3B ever-so-secret password listed i= n vdsm/libvirt_password)?<br>>=3B <br>>=3B What happens if you chown yo= ur image to vdsm:qemu? (keeping mode as 660)<br>>=3B <br>>=3B What's `g= roups qemu` on your hosts?<br>>=3B <br>>=3B Could you attach gdb to the= short-living qemu process=2C and run<br>>=3B getgroups(2) on it?<br>>= =3B <br>>=3B Dan.</div><div> =3B</div><div>>=3B >=3B >=3B>=3B= >=3Berrore interno process exited while connecting to monitor:<br>>=3B = >=3B >=3B>=3B>=3B((null):5034): Spice-Warning **: reds.c:3247:reds_= init_ssl: Could not<br>>=3B >=3B >=3B>=3B>=3Buse private key file= </div><div> =3B</div><div>I just ran into this myself on a fresh 3.2.2 = install. Enabling the virt-preview repo and doing a yum update =3Bfixed= the spice-warning issue and the VMs started right up. That might help you = Joop.</div><div> =3B</div><div> =3B</div><div>Another issue=2C = =3Bovirt-engine-sdk is newer on the fedora repos than in the ovirt-repo. Th= e fedora one caused issues (I forget which error at the moment)=2C so I had= to disable the fedora repos=2C remove ovirt-engine-sdk=2C and then reinsta= ll it from the ovirt-repo.</div><div> =3B</div><div> =3B</div><div>=  =3B</div><div><br>>=3B _____________________________________________= __<br>>=3B Users mailing list<br>>=3B Users@ovirt.org<br>>=3B http://= lists.ovirt.org/mailman/listinfo/users<br></div> </div></body> </html>= --_9ac12577-111e-4331-a941-8ee28a6f041e_-- --_37a82616-95c5-4643-a2ad-2d119d724eb4_ Content-Type: text/plain Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="virt_preview.txt" VXBkYXRpbmc6DQogaXB4ZS1yb21zLXFlbXUgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBub2FyY2ggICAgICAgICAgICAgICAgICAgICAgMjAxMzA1MTctMi5naXRjNGJjZTQz LmZjMTggICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAg ICAgICAgICAgICA4MDYgaw0KIGxpYmlzY3NpICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuNy4wLTUuZmMxOCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAg ICAgICAgICAgICAgICAgICAgIDUxIGsNCiBsaWJ2aXJ0ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAgICAgICAxLjEuMC0x LmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2 aWV3ICAgICAgICAgICAgICAgICAgICAgICAzMiBrDQogbGlidmlydC1jbGllbnQgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAg MS4xLjAtMS5mYzE4ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZp cnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAgICA0LjggTQ0KIGxpYnZpcnQtZGFlbW9uICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAg ICAgICAgIDEuMS4wLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZl ZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAgICAgICAgICAgMi4xIE0NCiBsaWJ2aXJ0LWRh ZW1vbi1jb25maWctbmV0d29yayAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAg ICAgICAgICAgICAgICAxLjEuMC0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAgICAzMSBrDQogbGli dmlydC1kYWVtb24tY29uZmlnLW53ZmlsdGVyICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQg ICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5mYzE4ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAgICAgMzUg aw0KIGxpYnZpcnQtZGFlbW9uLWRyaXZlci1pbnRlcmZhY2UgICAgICAgICAgICAgICAgICAgICAg eDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuMS4wLTEuZmMxOCAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAgICAgICAg ICAgIDcxIGsNCiBsaWJ2aXJ0LWRhZW1vbi1kcml2ZXItbGlieGwgICAgICAgICAgICAgICAgICAg ICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAgICAgICAxLjEuMC0xLmZjMTggICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAg ICAgICAgICAgIDExMiBrDQogbGlidmlydC1kYWVtb24tZHJpdmVyLWx4YyAgICAgICAgICAgICAg ICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5mYzE4ICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAg ICAgICAgICAgICAgICAgICAxMTYgaw0KIGxpYnZpcnQtZGFlbW9uLWRyaXZlci1uZXR3b3JrICAg ICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuMS4wLTEu ZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZp ZXcgICAgICAgICAgICAgICAgICAgICAgIDg0IGsNCiBsaWJ2aXJ0LWRhZW1vbi1kcml2ZXItbm9k ZWRldiAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAgICAgICAx LjEuMC0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEtdmly dC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAgICA3MSBrDQogbGlidmlydC1kYWVtb24tZHJp dmVyLW53ZmlsdGVyICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAg ICAgICAgMS4xLjAtMS5mYzE4ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVk b3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAgICAgOTcgaw0KIGxpYnZpcnQtZGFl bW9uLWRyaXZlci1xZW11ICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAg ICAgICAgICAgICAgIDEuMS4wLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAgICAgICAgICAgMzgyIGsNCiBsaWJ2 aXJ0LWRhZW1vbi1kcml2ZXItc2VjcmV0ICAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAg ICAgICAgICAgICAgICAgICAgICAxLjEuMC0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAgICA2NiBr DQogbGlidmlydC1kYWVtb24tZHJpdmVyLXN0b3JhZ2UgICAgICAgICAgICAgICAgICAgICAgICB4 ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5mYzE4ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAg ICAxMTAgaw0KIGxpYnZpcnQtZGFlbW9uLWRyaXZlci11bWwgICAgICAgICAgICAgICAgICAgICAg ICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuMS4wLTEuZmMxOCAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAg ICAgICAgICAgIDc5IGsNCiBsaWJ2aXJ0LWRhZW1vbi1kcml2ZXIteGVuICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAgICAgICAxLjEuMC0xLmZjMTggICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAg ICAgICAgICAgICAgICAgIDExNiBrDQogbGlidmlydC1sb2NrLXNhbmxvY2sgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5m YzE4ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmll dyAgICAgICAgICAgICAgICAgICAgICAgNzggaw0KIGxpYnZpcnQtcHl0aG9uICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEu MS4wLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0 LXByZXZpZXcgICAgICAgICAgICAgICAgICAgICAgMjIyIGsNCiBxZW11LWNvbW1vbiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAg ICAgICAyOjEuNS4xLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRv cmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAgIDIxNyBrDQogcWVtdS1pbWcgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAg ICAgICAgICAgICAgMjoxLjUuMS0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAgICA0NjAgaw0KIHFlbXUt a3ZtICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAg ICAgICAgICAgICAgICAgICAgIDI6MS41LjEtMS5mYzE4ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAgICAgICAgICAgIDM4IGsN CiBxZW11LXN5c3RlbS14ODYgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4 Nl82NCAgICAgICAgICAgICAgICAgICAgICAyOjEuNS4xLTEuZmMxOCAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAg IDMuMyBNDQogc2VhYmlvcy1iaW4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBub2FyY2ggICAgICAgICAgICAgICAgICAgICAgMS43LjIuMi0xLmZjMTggICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAg ICAgICAgICAgNzAgaw0KIHNnYWJpb3MtYmluICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgbm9hcmNoICAgICAgICAgICAgICAgICAgICAgIDE6MC4yMDExMDYyMnN2bi01 LmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAg ICAgICAgICAgICAgICAgNi4yIGsNCkluc3RhbGxpbmcgZm9yIGRlcGVuZGVuY2llczoNCiBhdC1z cGkyLWF0ayAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAg ICAgICAgICAgICAgICAgICAgICAyLjYuMi0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBmZWRvcmEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA3MiBr DQogYXQtc3BpMi1jb3JlICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4 ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMi42LjItMS5mYzE4ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgZmVkb3JhICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAxNDcgaw0KIGNhaXJvLWdvYmplY3QgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuMTIuOC0yLmZjMTggICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYSAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDI0IGsNCiBjb2xvci1maWxlc3lzdGVtICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIG5vYXJjaCAgICAgICAgICAgICAgICAgICAgICAxLTEwICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIDQuOSBrDQogY29sb3JkICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMC4xLjI1LTEu ZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAyMzggaw0KIGZ1c2UtbGlicyAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDIu OS4yLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYSAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDkwIGsNCiBndGszICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAg ICAgICAzLjYuMi0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRv cmEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDMuMSBNDQogbGNtczIgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAg ICAgICAgICAgICAgMi40LTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgZmVkb3JhICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAxMzggaw0KIGxpYlhl dmllICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAg ICAgICAgICAgICAgICAgICAgIDEuMC4zLTQuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIGZlZG9yYSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDE3IGsN CiBsaWJndXNiICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4 Nl82NCAgICAgICAgICAgICAgICAgICAgICAwLjEuNC0xLmZjMTggICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBmZWRvcmEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAzMyBrDQogbGlidmlydC1kYWVtb24tZHJpdmVyLXZib3ggICAgICAgICAgICAgICAgICAgICAg ICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5mYzE4ICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAg ICAgICAgICAxODkgaw0KIHNlYXZnYWJpb3MtYmluICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgbm9hcmNoICAgICAgICAgICAgICAgICAgICAgIDEuNy4yLjItMS5mYzE4ICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAg ICAgICAgICAgICAgICAgIDI1IGsNCiBzaGFyZWQtY29sb3ItcHJvZmlsZXMgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIG5vYXJjaCAgICAgICAgICAgICAgICAgICAgICAwLjEuNi0xLmZj MTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIDcwNSBrDQogdnRlMyAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMC4z NC4yLTMuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAzMjkgaw== --_37a82616-95c5-4643-a2ad-2d119d724eb4_--

Alessandro Bianchi wrote:
On 6-9-2013 12:34=2C Alessandro Bianchi wrote:
Hi all
I'm running 3.2 on several Fedora 18 nodes
One of them has a local storage running 4 VMs
Today the UPS crashed and host was rebboted after UPS replacement
None of the VM's were able to be started
I tried to put the Host in maintenance and reinstalled it=2C but thi= s didn't give any result
Digging into the logs I discovered the following error:
The first was of this kind (on every VM)
File "/usr/lib64/python2.7/site-packages/libvirt.py"=2C line 2630= =2C in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed'= =2C conn=3Dself) libvirtError: errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not use private key file qemu-kvm: failed to initialize spice server
Thread-564::DEBUG::2013-09-06 11:31:32=2C814::vm::1065::vm.Vm::(setDownStatus) vmId=3D`49d84915-490b-497d-a3f8-c7dac7485281`::Changed state to Down= : errore interno process exited while connecting to monitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could no= t use private key file qemu-kvm: failed to initialize spice server
The private key was marked 440 as permission owned by vdsm user and kvm group
I had to change it to 444 to allow everyone to read it
After that I had for every VM the following error:
could not open disk image /rhev/data-center/3935800a-abe4-406d-84a1-4c3c0b915cce/6818de31-5cda= -41d0-a41a-681230a409ba/images/54144c03-5057-462e-8275-6ab386ae8c5a/0129899= 8-32d5-44c2-b5d1-91be1316ed19:
Permission denied
Disks were owned by vdsm:kvm with 660 permission
I had to relax this to 666 to enable the VMs to start
Has anyone faced this kind f problem before?
Yes=2C me.
Any hint about what may have caused this odd problem?
yum update.
I updated one of my hosts and after that that host couldn't start VMs anymore with exact the same errors. See thread 'Starting VM error' by Shaun Glass. I tried a couple of things but not making world readable those files. Will probably restore a backup and try it. I added the virt-preview repo for F18 and updated qemu/libvirt which also solved the problem. The difference between the updated and not updated host were really minimal. See the thead for logs.
Regards=2C
Joop Thank you for your very quick answer
I suspected the same thing !
I'll update libvirt and revert the permission changes
That will give you way way newer libvirt/qemu than you probably want. I would keep the permission changes and hope that one of the following updates to either libvirt/qemu fixes this problem. =20 Joop=2C I'm sorry that I have many requests and few answers=2C but if ind= eed
Date: Fri=2C 6 Sep 2013 22:05:05 +0100 From: danken@redhat.com To: jvdwege@xs4all.nl CC: users@ovirt.org Subject: Re: [Users] very odd permission problem =20 On Fri=2C Sep 06=2C 2013 at 04:05:13PM +0200=2C Joop wrote: the problem is related to a version of libvirt/qemu=2C would yould you tr= y to reproduce it outside ovirt? =20 I mean=2C in your working/non-working hosts=2C could you create a vdsm:kv= m- owned image=2C and try to run it from virsh (using vdsm@ovirt user and th= e ever-so-secret password listed in vdsm/libvirt_password)? =20 What happens if you chown your image to vdsm:qemu? (keeping mode as 660) =20 What's `groups qemu` on your hosts? =20 Could you attach gdb to the short-living qemu process=2C and run getgroups(2) on it? =20 Dan. > > >>>errore interno process exited while connecting to monitor:
((null):5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could no= t use private key file I just ran into this myself on a fresh 3.2.2 in= stall. Enabling the virt-preview repo and doing a yum update fixed the spic= e-warning issue and the VMs started right up. That might help you Joop. An= other issue=2C ovirt-engine-sdk is newer on the fedora repos than in the ov= irt-repo. The fedora one caused issues (I forget which error at the moment)= =2C so I had to disable the fedora repos=2C remove ovirt-engine-sdk=2C and =
--_13e46ecb-40fa-403f-b2ac-0b4bd23d6779_ Content-Type: multipart/alternative; boundary="_b101886b-ffcb-4ef0-9e73-502cb1206a0c_" --_b101886b-ffcb-4ef0-9e73-502cb1206a0c_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable =20 From: cybertimber2000@hotmail.com To: danken@redhat.com=3B jvdwege@xs4all.nl CC: users@ovirt.org Subject: RE: [Users] very odd permission problem Date: Fri=2C 6 Sep 2013 21:35:51 -0400 =0A= =0A= =0A= =20 then reinstall it from the ovirt-repo. *Forgot to note that I attached a te= xt file of what updated once I added the virt-preview repo =20
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users =
>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BThread= -564::DEBUG::2013-09-06<br>>=3B >=3B >=3B>=3B>=3B11:31:32=2C814::= vm::1065::vm.Vm::(setDownStatus)<br>>=3B >=3B >=3B>=3B>=3BvmId=3D= `49d84915-490b-497d-a3f8-c7dac7485281`::Changed state to Down:<br>>=3B &g= t=3B >=3B>=3B>=3Berrore interno process exited while connecting to mo= nitor:<br>>=3B >=3B >=3B>=3B>=3B((null):5034): Spice-Warning **: = reds.c:3247:reds_init_ssl: Could not<br>>=3B >=3B >=3B>=3B>=3Buse=
>=3B >=3B >=3BI suspected the same thing !<br>>=3B >=3B >=3B<b= r>>=3B >=3B >=3BI'll update libvirt and revert the permission changes= <br>>=3B >=3B >=3B<br>>=3B >=3B That will give you way way newer =
--_b101886b-ffcb-4ef0-9e73-502cb1206a0c_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <style><!-- .hmmessage P { margin:0px=3B padding:0px } body.hmmessage { font-size: 12pt=3B font-family:Calibri } --></style></head> <body class=3D'hmmessage'><div dir=3D'ltr'><br> =3B<BR><div><hr id=3D"s= topSpelling">From: cybertimber2000@hotmail.com<br>To: danken@redhat.com=3B = jvdwege@xs4all.nl<br>CC: users@ovirt.org<br>Subject: RE: [Users] very odd p= ermission problem<br>Date: Fri=2C 6 Sep 2013 21:35:51 -0400<br><br>=0A= =0A= <style><!--=0A= .ExternalClass .ecxhmmessage P {=0A= padding:0px=3B=0A= }=0A= =0A= .ExternalClass body.ecxhmmessage {=0A= font-size:12pt=3B=0A= font-family:Calibri=3B=0A= }=0A= =0A= --></style>=0A= <div dir=3D"ltr"><br> =3B<br><div>>=3B Date: Fri=2C 6 Sep 2013 22:05:= 05 +0100<br>>=3B From: danken@redhat.com<br>>=3B To: jvdwege@xs4all.nl<= br>>=3B CC: users@ovirt.org<br>>=3B Subject: Re: [Users] very odd permi= ssion problem<br>>=3B <br>>=3B On Fri=2C Sep 06=2C 2013 at 04:05:13PM += 0200=2C Joop wrote:<br>>=3B >=3B Alessandro Bianchi wrote:<br>>=3B &g= t=3B >=3B>=3BOn 6-9-2013 12:34=2C Alessandro Bianchi wrote:<br>>=3B &= gt=3B >=3B>=3B>=3BHi all<br>>=3B >=3B >=3B>=3B>=3B<br>>= =3B >=3B >=3B>=3B>=3BI'm running 3.2 on several Fedora 18 nodes<br>= >=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BOne of = them has a local storage running 4 VMs<br>>=3B >=3B >=3B>=3B>=3B<= br>>=3B >=3B >=3B>=3B>=3BToday the UPS crashed and host was rebbo= ted after UPS replacement<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >= =3B >=3B>=3B>=3BNone of the VM's were able to be started<br>>=3B &g= t=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BI tried to put t= he Host in maintenance and reinstalled it=2C but this<br>>=3B >=3B >= =3B>=3B>=3Bdidn't give any result<br>>=3B >=3B >=3B>=3B>=3B<b= r>>=3B >=3B >=3B>=3B>=3BDigging into the logs I discovered the fo= llowing error:<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B&= gt=3B>=3BThe first was of this kind (on every VM)<br>>=3B >=3B >=3B= >=3B>=3B<br>>=3B >=3B >=3B>=3B>=3B File "/usr/lib64/python2.= 7/site-packages/libvirt.py"=2C line 2630=2C in<br>>=3B >=3B >=3B>= =3B>=3BcreateXML<br>>=3B >=3B >=3B>=3B>=3B if ret is None:r= aise libvirtError('virDomainCreateXML() failed'=2C<br>>=3B >=3B >=3B&= gt=3B>=3Bconn=3Dself)<br>>=3B >=3B >=3B>=3B>=3BlibvirtError: er= rore interno process exited while connecting to<br>>=3B >=3B >=3B>= =3B>=3Bmonitor: ((null):5034): Spice-Warning **: reds.c:3247:reds_init_ss= l:<br>>=3B >=3B >=3B>=3B>=3BCould not use private key file<br>>= =3B >=3B >=3B>=3B>=3Bqemu-kvm: failed to initialize spice server<br= private key file<br>>=3B >=3B >=3B>=3B>=3Bqemu-kvm: failed to in= itialize spice server<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B = >=3B>=3B>=3BThe private key was marked 440 as permission owned by vds= m user and<br>>=3B >=3B >=3B>=3B>=3Bkvm group<br>>=3B >=3B &g= t=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BI had to change it to 4= 44 to allow everyone to read it<br>>=3B >=3B >=3B>=3B>=3B<br>>= =3B >=3B >=3B>=3B>=3BAfter that I had for every VM the following er= ror:<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>= =3Bcould not open disk image<br>>=3B >=3B >=3B>=3B>=3B/rhev/data-= center/3935800a-abe4-406d-84a1-4c3c0b915cce/6818de31-5cda-41d0-a41a-681230a= 409ba/images/54144c03-5057-462e-8275-6ab386ae8c5a/01298998-32d5-44c2-b5d1-9= 1be1316ed19:<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>= =3B>=3BPermission denied<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B &g= t=3B >=3B>=3B>=3BDisks were owned by vdsm:kvm with 660 permission<br>= >=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B>=3B>=3BI had t= o relax this to 666 to enable the VMs to start<br>>=3B >=3B >=3B>= =3B>=3B<br>>=3B >=3B >=3B>=3B>=3BHas anyone faced this kind f p= roblem before?<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B >=3B >=3B&= gt=3BYes=2C me.<br>>=3B >=3B >=3B>=3B>=3BAny hint about what may = have caused this odd problem?<br>>=3B >=3B >=3B>=3B>=3B<br>>=3B= >=3B >=3B>=3Byum update.<br>>=3B >=3B >=3B>=3B<br>>=3B >= =3B >=3B>=3BI updated one of my hosts and after that that host couldn't= start VMs<br>>=3B >=3B >=3B>=3Banymore with exact the same errors.= See thread 'Starting VM error' by<br>>=3B >=3B >=3B>=3BShaun Glass= . I tried a couple of things but not making world readable<br>>=3B >=3B= >=3B>=3Bthose files. Will probably restore a backup and try it.<br>>= =3B >=3B >=3B>=3BI added the virt-preview repo for F18 and updated qe= mu/libvirt which<br>>=3B >=3B >=3B>=3Balso solved the problem.<br>&= gt=3B >=3B >=3B>=3BThe difference between the updated and not updated= host were really<br>>=3B >=3B >=3B>=3Bminimal. See the thead for l= ogs.<br>>=3B >=3B >=3B>=3B<br>>=3B >=3B >=3B>=3BRegards=2C<= br>>=3B >=3B >=3B>=3B<br>>=3B >=3B >=3B>=3BJoop<br>>=3B &= gt=3B >=3BThank you for your very quick answer<br>>=3B >=3B >=3B<br= libvirt/qemu than you probably<br>>=3B >=3B want. I would keep the perm= ission changes and hope that one of the<br>>=3B >=3B following updates = to either libvirt/qemu fixes this problem.<br>>=3B <br>>=3B Joop=2C I'm= sorry that I have many requests and few answers=2C but if indeed<br>>=3B= the problem is related to a version of libvirt/qemu=2C would yould you try= <br>>=3B to reproduce it outside ovirt?<br>>=3B <br>>=3B I mean=2C in= your working/non-working hosts=2C could you create a vdsm:kvm-<br>>=3B o= wned image=2C and try to run it from virsh (using vdsm@ovirt user and the<b= r>>=3B ever-so-secret password listed in vdsm/libvirt_password)?<br>>= =3B <br>>=3B What happens if you chown your image to vdsm:qemu? (keeping = mode as 660)<br>>=3B <br>>=3B What's `groups qemu` on your hosts?<br>&g= t=3B <br>>=3B Could you attach gdb to the short-living qemu process=2C an= d run<br>>=3B getgroups(2) on it?<br>>=3B <br>>=3B Dan.</div><div>&nb= sp=3B</div><div>>=3B >=3B >=3B>=3B>=3Berrore interno process exit= ed while connecting to monitor:<br>>=3B >=3B >=3B>=3B>=3B((null):= 5034): Spice-Warning **: reds.c:3247:reds_init_ssl: Could not<br>>=3B >= =3B >=3B>=3B>=3Buse private key file</div><div> =3B</div><div>I j= ust ran into this myself on a fresh 3.2.2 install. Enabling the virt-previe= w repo and doing a yum update =3Bfixed the spice-warning issue and the = VMs started right up. That might help you Joop.</div><div> =3B</div><di= v> =3B</div><div>Another issue=2C =3Bovirt-engine-sdk is newer on t= he fedora repos than in the ovirt-repo. The fedora one caused issues (I for= get which error at the moment)=2C so I had to disable the fedora repos=2C r= emove ovirt-engine-sdk=2C and then reinstall it from the ovirt-repo.</div><= div> =3B</div><div>*Forgot to note that I attached a text file of what&= nbsp=3Bupdated once I added the virt-preview =3Brepo =3B</div><div>=  =3B</div><div> =3B</div><div><br>>=3B __________________________= _____________________<br>>=3B Users mailing list<br>>=3B Users@ovirt.or= g<br>>=3B http://lists.ovirt.org/mailman/listinfo/users<br></div> = </div></div> </div></body> </html>= --_b101886b-ffcb-4ef0-9e73-502cb1206a0c_-- --_13e46ecb-40fa-403f-b2ac-0b4bd23d6779_ Content-Type: text/plain Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="virt_preview.txt" VXBkYXRpbmc6DQogaXB4ZS1yb21zLXFlbXUgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBub2FyY2ggICAgICAgICAgICAgICAgICAgICAgMjAxMzA1MTctMi5naXRjNGJjZTQz LmZjMTggICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAg ICAgICAgICAgICA4MDYgaw0KIGxpYmlzY3NpICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuNy4wLTUuZmMxOCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAg ICAgICAgICAgICAgICAgICAgIDUxIGsNCiBsaWJ2aXJ0ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAgICAgICAxLjEuMC0x LmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2 aWV3ICAgICAgICAgICAgICAgICAgICAgICAzMiBrDQogbGlidmlydC1jbGllbnQgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAg MS4xLjAtMS5mYzE4ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZp cnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAgICA0LjggTQ0KIGxpYnZpcnQtZGFlbW9uICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAg ICAgICAgIDEuMS4wLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZl ZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAgICAgICAgICAgMi4xIE0NCiBsaWJ2aXJ0LWRh ZW1vbi1jb25maWctbmV0d29yayAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAg ICAgICAgICAgICAgICAxLjEuMC0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAgICAzMSBrDQogbGli dmlydC1kYWVtb24tY29uZmlnLW53ZmlsdGVyICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQg ICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5mYzE4ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAgICAgMzUg aw0KIGxpYnZpcnQtZGFlbW9uLWRyaXZlci1pbnRlcmZhY2UgICAgICAgICAgICAgICAgICAgICAg eDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuMS4wLTEuZmMxOCAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAgICAgICAg ICAgIDcxIGsNCiBsaWJ2aXJ0LWRhZW1vbi1kcml2ZXItbGlieGwgICAgICAgICAgICAgICAgICAg ICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAgICAgICAxLjEuMC0xLmZjMTggICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAg ICAgICAgICAgIDExMiBrDQogbGlidmlydC1kYWVtb24tZHJpdmVyLWx4YyAgICAgICAgICAgICAg ICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5mYzE4ICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAg ICAgICAgICAgICAgICAgICAxMTYgaw0KIGxpYnZpcnQtZGFlbW9uLWRyaXZlci1uZXR3b3JrICAg ICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuMS4wLTEu ZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZp ZXcgICAgICAgICAgICAgICAgICAgICAgIDg0IGsNCiBsaWJ2aXJ0LWRhZW1vbi1kcml2ZXItbm9k ZWRldiAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAgICAgICAx LjEuMC0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEtdmly dC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAgICA3MSBrDQogbGlidmlydC1kYWVtb24tZHJp dmVyLW53ZmlsdGVyICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAg ICAgICAgMS4xLjAtMS5mYzE4ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVk b3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAgICAgOTcgaw0KIGxpYnZpcnQtZGFl bW9uLWRyaXZlci1xZW11ICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAg ICAgICAgICAgICAgIDEuMS4wLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAgICAgICAgICAgMzgyIGsNCiBsaWJ2 aXJ0LWRhZW1vbi1kcml2ZXItc2VjcmV0ICAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAg ICAgICAgICAgICAgICAgICAgICAxLjEuMC0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAgICA2NiBr DQogbGlidmlydC1kYWVtb24tZHJpdmVyLXN0b3JhZ2UgICAgICAgICAgICAgICAgICAgICAgICB4 ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5mYzE4ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAg ICAxMTAgaw0KIGxpYnZpcnQtZGFlbW9uLWRyaXZlci11bWwgICAgICAgICAgICAgICAgICAgICAg ICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuMS4wLTEuZmMxOCAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAg ICAgICAgICAgIDc5IGsNCiBsaWJ2aXJ0LWRhZW1vbi1kcml2ZXIteGVuICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAgICAgICAxLjEuMC0xLmZjMTggICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAg ICAgICAgICAgICAgICAgIDExNiBrDQogbGlidmlydC1sb2NrLXNhbmxvY2sgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5m YzE4ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmll dyAgICAgICAgICAgICAgICAgICAgICAgNzggaw0KIGxpYnZpcnQtcHl0aG9uICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEu MS4wLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0 LXByZXZpZXcgICAgICAgICAgICAgICAgICAgICAgMjIyIGsNCiBxZW11LWNvbW1vbiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAg ICAgICAyOjEuNS4xLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRv cmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAgIDIxNyBrDQogcWVtdS1pbWcgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAg ICAgICAgICAgICAgMjoxLjUuMS0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAgICAgICAgICA0NjAgaw0KIHFlbXUt a3ZtICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAg ICAgICAgICAgICAgICAgICAgIDI6MS41LjEtMS5mYzE4ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAgICAgICAgICAgICAgICAgIDM4IGsN CiBxZW11LXN5c3RlbS14ODYgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4 Nl82NCAgICAgICAgICAgICAgICAgICAgICAyOjEuNS4xLTEuZmMxOCAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBmZWRvcmEtdmlydC1wcmV2aWV3ICAgICAgICAgICAgICAgICAgICAg IDMuMyBNDQogc2VhYmlvcy1iaW4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBub2FyY2ggICAgICAgICAgICAgICAgICAgICAgMS43LjIuMi0xLmZjMTggICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAg ICAgICAgICAgNzAgaw0KIHNnYWJpb3MtYmluICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgbm9hcmNoICAgICAgICAgICAgICAgICAgICAgIDE6MC4yMDExMDYyMnN2bi01 LmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAg ICAgICAgICAgICAgICAgNi4yIGsNCkluc3RhbGxpbmcgZm9yIGRlcGVuZGVuY2llczoNCiBhdC1z cGkyLWF0ayAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAg ICAgICAgICAgICAgICAgICAgICAyLjYuMi0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBmZWRvcmEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA3MiBr DQogYXQtc3BpMi1jb3JlICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4 ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMi42LjItMS5mYzE4ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgZmVkb3JhICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAxNDcgaw0KIGNhaXJvLWdvYmplY3QgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDEuMTIuOC0yLmZjMTggICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYSAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDI0IGsNCiBjb2xvci1maWxlc3lzdGVtICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIG5vYXJjaCAgICAgICAgICAgICAgICAgICAgICAxLTEwICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIDQuOSBrDQogY29sb3JkICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMC4xLjI1LTEu ZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAyMzggaw0KIGZ1c2UtbGlicyAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAgICAgICAgICAgICAgICAgICAgIDIu OS4yLTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYSAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDkwIGsNCiBndGszICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4Nl82NCAgICAgICAgICAgICAgICAg ICAgICAzLjYuMi0xLmZjMTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRv cmEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDMuMSBNDQogbGNtczIgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAg ICAgICAgICAgICAgMi40LTEuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgZmVkb3JhICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAxMzggaw0KIGxpYlhl dmllICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeDg2XzY0ICAg ICAgICAgICAgICAgICAgICAgIDEuMC4zLTQuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIGZlZG9yYSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDE3IGsN CiBsaWJndXNiICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHg4 Nl82NCAgICAgICAgICAgICAgICAgICAgICAwLjEuNC0xLmZjMTggICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBmZWRvcmEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAzMyBrDQogbGlidmlydC1kYWVtb24tZHJpdmVyLXZib3ggICAgICAgICAgICAgICAgICAgICAg ICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMS4xLjAtMS5mYzE4ICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhLXZpcnQtcHJldmlldyAgICAgICAgICAgICAg ICAgICAgICAxODkgaw0KIHNlYXZnYWJpb3MtYmluICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgbm9hcmNoICAgICAgICAgICAgICAgICAgICAgIDEuNy4yLjItMS5mYzE4ICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZlZG9yYS12aXJ0LXByZXZpZXcgICAgICAg ICAgICAgICAgICAgICAgIDI1IGsNCiBzaGFyZWQtY29sb3ItcHJvZmlsZXMgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIG5vYXJjaCAgICAgICAgICAgICAgICAgICAgICAwLjEuNi0xLmZj MTggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmZWRvcmEgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIDcwNSBrDQogdnRlMyAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB4ODZfNjQgICAgICAgICAgICAgICAgICAgICAgMC4z NC4yLTMuZmMxOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmVkb3JhICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAzMjkgaw== --_13e46ecb-40fa-403f-b2ac-0b4bd23d6779_--
participants (4)
-
Alessandro Bianchi
-
Dan Kenigsberg
-
Joop
-
Nicholas Kesick