7:07 a.m.
Hello,
We're planning a move from our old building towards a new one a few
meters away.
In a similar way of Martijn
(https://www.mail-archive.com/users@ovirt.org/msg33182.html), I have
maintenance planed on our storage side.
Say an oVirt DC is using a SAN's LUN via iSCSI (Equallogic).
This SAN allows me to setup block replication between two SANs, seen by
oVirt as one (Dell is naming it SyncRep).
Then switch all the iSCSI accesses to the replicated LUN.
When doing this, the iSCSI stack of each oVirt host notices the
de-connection, tries to reconnect, and succeeds.
Amongst our hosts, this happens between 4 and 15 seconds.
When this happens fast enough, oVirt engine and the VMs don't even
notice, and they keep running happily.
When this takes more than 4 seconds, there are 2 cases :
1 - The hosts and/or oVirt and/or the SPM (I actually don't know)
notices that there is a storage failure, and pauses the VMs.
When the iSCSI stack reconnects, the VMs are automatically recovered
from pause, and this all takes less than 30 seconds. That is very
acceptable for us, as this action is extremely rare.
2 - Same storage failure, VMs paused, and some VMs stay in pause mode
forever.
Manual "run" action is mandatory.
When done, everything recovers correctly.
This is also quite acceptable, but here come my questions :
My questions : (!)
- *WHAT* process or piece of code or what oVirt parts is responsible for
deciding when to UN-pause a VM, and at what conditions?
That would help me to understand why some cases are working even more
smoothly than others.
- Are there related timeouts I could play with in engine-config options?
- [a bit off-topic] Is it safe to increase some iSCSI timeouts of
buffer-sizes in the hope this kind of disconnection would get un-noticed?
--
Nicolas ECARNOT
7:30 a.m.
Hi,
you are aware of the fact that eql sync replication is just about
replication, no single piece of high availability? i am not even sure if
it does ip failover itself. so better think about minutes of
interruptions than seconds.
anyway, dont count on ovirts pause/unpause. theres a real chance that it
will go horrible wrong. a scheduled maint. window where everything gets
shut down whould be best practice
Juergen
Am 5/30/2016 um 3:07 PM schrieb Nicolas Ecarnot:
Hello,
We're planning a move from our old building towards a new one a few
meters away.
In a similar way of Martijn
(https://www.mail-archive.com/users@ovirt.org/msg33182.html), I have
maintenance planed on our storage side.
Say an oVirt DC is using a SAN's LUN via iSCSI (Equallogic).
This SAN allows me to setup block replication between two SANs, seen by
oVirt as one (Dell is naming it SyncRep).
Then switch all the iSCSI accesses to the replicated LUN.
When doing this, the iSCSI stack of each oVirt host notices the
de-connection, tries to reconnect, and succeeds.
Amongst our hosts, this happens between 4 and 15 seconds.
When this happens fast enough, oVirt engine and the VMs don't even
notice, and they keep running happily.
When this takes more than 4 seconds, there are 2 cases :
1 - The hosts and/or oVirt and/or the SPM (I actually don't know)
notices that there is a storage failure, and pauses the VMs.
When the iSCSI stack reconnects, the VMs are automatically recovered
from pause, and this all takes less than 30 seconds. That is very
acceptable for us, as this action is extremely rare.
2 - Same storage failure, VMs paused, and some VMs stay in pause mode
forever.
Manual "run" action is mandatory.
When done, everything recovers correctly.
This is also quite acceptable, but here come my questions :
My questions : (!)
- *WHAT* process or piece of code or what oVirt parts is responsible for
deciding when to UN-pause a VM, and at what conditions?
That would help me to understand why some cases are working even more
smoothly than others.
- Are there related timeouts I could play with in engine-config options?
- [a bit off-topic] Is it safe to increase some iSCSI timeouts of
buffer-sizes in the hope this kind of disconnection would get un-noticed?
7:59 a.m.
Le 30/05/2016 15:30, InterNetX - Juergen Gotteswinter a écrit :
Hi,
you are aware of the fact that eql sync replication is just about
replication, no single piece of high availability? i am not even sure if
it does ip failover itself. so better think about minutes of
interruptions than seconds.
Hi Juergen,
I'm absolutely aware that there is no HA discussed here, at least in my
mind.
It does ip fail-over, but I'm not even blindly trusting it enough,
that's why I'm doing numerous tests and measures.
I'm gladly surprised by how the iSCSI stack is reacting, and its log
files are readable enough for me to decide.
Actually, I was more worrying about the iSCSI reconnection storm, but
googling about it does not seem to get any warnings.
anyway, dont count on ovirts pause/unpause. theres a real chance that
it
will go horrible wrong. a scheduled maint. window where everything gets
shut down whould be best practice
Indeed, this would the best choice, if I had it.
--
Nicolas ECARNOT
8:07 a.m.
Am 5/30/2016 um 3:59 PM schrieb Nicolas Ecarnot:
Le 30/05/2016 15:30, InterNetX - Juergen Gotteswinter a écrit :
> Hi,
>
> you are aware of the fact that eql sync replication is just about
> replication, no single piece of high availability? i am not even sure if
> it does ip failover itself. so better think about minutes of
> interruptions than seconds.
Hi Juergen,
I'm absolutely aware that there is no HA discussed here, at least in my
mind.
It does ip fail-over, but I'm not even blindly trusting it enough,
that's why I'm doing numerous tests and measures.
I'm gladly surprised by how the iSCSI stack is reacting, and its log
files are readable enough for me to decide.
Actually, I was more worrying about the iSCSI reconnection storm, but
googling about it does not seem to get any warnings.
This works pretty well with the Eql Boxes, except you use the EQL
without Hit Kit. With installed HitKit on each Client i dont think that
this will cause problems.
> anyway, dont count on ovirts pause/unpause. theres a real chance that it
> will go horrible wrong. a scheduled maint. window where everything gets
> shut down whould be best practice
Indeed, this would the best choice, if I had it.
1:09 p.m.
On Mon, May 30, 2016 at 4:07 PM, Nicolas Ecarnot <nicolas(a)ecarnot.net> wrote:
Hello,
We're planning a move from our old building towards a new one a few meters
away.
In a similar way of Martijn
(https://www.mail-archive.com/users@ovirt.org/msg33182.html), I have
maintenance planed on our storage side.
Say an oVirt DC is using a SAN's LUN via iSCSI (Equallogic).
This SAN allows me to setup block replication between two SANs, seen by
oVirt as one (Dell is naming it SyncRep).
Then switch all the iSCSI accesses to the replicated LUN.
When doing this, the iSCSI stack of each oVirt host notices the
de-connection, tries to reconnect, and succeeds.
Amongst our hosts, this happens between 4 and 15 seconds.
When this happens fast enough, oVirt engine and the VMs don't even notice,
and they keep running happily.
When this takes more than 4 seconds, there are 2 cases :
1 - The hosts and/or oVirt and/or the SPM (I actually don't know) notices
that there is a storage failure, and pauses the VMs.
When the iSCSI stack reconnects, the VMs are automatically recovered from
pause, and this all takes less than 30 seconds. That is very acceptable for
us, as this action is extremely rare.
2 - Same storage failure, VMs paused, and some VMs stay in pause mode
forever.
Manual "run" action is mandatory.
When done, everything recovers correctly.
This is also quite acceptable, but here come my questions :
My questions : (!)
- *WHAT* process or piece of code or what oVirt parts is responsible for
deciding when to UN-pause a VM, and at what conditions?
Vms get paused by qemu, when you get ENOSPC or some other IO error.
This probably happens when a vm is writing to storage, and all paths to storage
are faulty - with current configuration, the scsi layer will fail
after 5 seconds,
and if no path is available, the write will fail.
If vdsm storage monitoring system detected the issue, the storage domain
will become invalid. When the storage domain will become valid again, we
try to resume all vms paused because of IO errors.
Storage monitoring is done every 10 seconds in normal conditions, but in
current release, there can be delays of up to couple of minutes in
extreme conditions,
for example, 50 storage domains and doing lot of io. So basically, the
storage domain
monitor may miss an error on storage, never become invalid, and would
never become valid again and the vm will have to be resumed manually.
See https://bugzilla.redhat.com/1081962
In ovirt 4.0 monitoring should be improved, and will always monitor
storage every
10 seconds, but even this cannot guarantee that we will detect all
storage errors
For example, if the storage outage is shorter then 10 seconds. But I
guess that chance
that storage outage was shorter then 10 seconds, but long enough to cause a vm
to pause is very low.
That would help me to understand why some cases are working even
more
smoothly than others.
- Are there related timeouts I could play with in engine-config options?
Nothing on the engine side...
- [a bit off-topic] Is it safe to increase some iSCSI timeouts of
buffer-sizes in the hope this kind of disconnection would get un-noticed?
But you may modify multipath configuration on the host.
We use now this multipath configuration (/etc/multipath.conf):
# VDSM REVISION 1.3
defaults {
polling_interval 5
no_path_retry fail
user_friendly_names no
flush_on_last_del yes
fast_io_fail_tmo 5
dev_loss_tmo 30
max_fds 4096
deferred_remove yes
}
devices {
device {
all_devs yes
no_path_retry fail
}
}
This enforces failing of io request on devices that by default will queue such
requests for long or unlimited time. Queuing requests is very bad for vdsm, and
cause various commands to block for minutes during storage outage,
failing various
flows in vdsm and the ui.
See https://bugzilla.redhat.com/880738
However, in your case, using queuing may be the best way to do the switch
from one storage to another in the smoothest way.
You may try this setting:
devices {
device {
all_devs yes
no_path_retry 30
}
}
This will queue io requests for 30 seconds before failing.
Using this normally would be a bad idea with vdsm, since during storage outage,
vdsm may block for 30 seconds when no paths is available, and is not designed
for this behavior, but blocking from time to time for short time should be ok.
I think that modifying the configuration and reloading multipathd service should
be enough to use the new settings, but I'm not sure if this changes
existing sessions
or open devices.
Adding Ben to add more info about this.
Nir
1:21 p.m.
Le 30/05/2016 21:09, Nir Soffer wrote... SOME VERY VALUABLE ANSWERS!
Thank you very much Nir, as your answers will give me food for thought
for the weeks to come.
It's late here, I'll begin checking all this tomorrow, but just a note :
This enforces failing of io request on devices that by default will
queue such
requests for long or unlimited time. Queuing requests is very bad for vdsm, and
cause various commands to block for minutes during storage outage,
failing various
flows in vdsm and the ui.
See https://bugzilla.redhat.com/880738
Though we own a Redhat customer active subscription, I logged in and yet
I can not access the BZ above.
I'm sure you can help :)
--
Nicolas ECARNOT
7:38 a.m.
On 30 May 2016, at 21:21, Nicolas Ecarnot <nicolas(a)ecarnot.net>
wrote:
Le 30/05/2016 21:09, Nir Soffer wrote... SOME VERY VALUABLE ANSWERS!
Thank you very much Nir, as your answers will give me food for thought for the weeks to
come.
It's late here, I'll begin checking all this tomorrow, but just a note :
> This enforces failing of io request on devices that by default will queue such
> requests for long or unlimited time. Queuing requests is very bad for vdsm, and
> cause various commands to block for minutes during storage outage,
> failing various
> flows in vdsm and the ui.
> See https://bugzilla.redhat.com/880738
Though we own a Redhat customer active subscription, I logged in and yet I can not access
the BZ above.
I'm sure you can help :)
Hi Nicolas,
ugh, there are procedural issues with that bug it seems.
But in a nutshell, it is shipped fixed by [1] (so 3.6 release)
and the actual fixes are [2] and [3]
alternatively it’s described in ovirt bug [4]
Thanks,
michal
[1] https://rhn.redhat.com/errata/RHBA-2016-0362.html
[2] https://gerrit.ovirt.org/#/c/44855/
[3] https://gerrit.ovirt.org/#/c/42189/
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1225162
--
Nicolas ECARNOT
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:51 a.m.
On Mon, May 30, 2016 at 10:09:25PM +0300, Nir Soffer wrote:
On Mon, May 30, 2016 at 4:07 PM, Nicolas Ecarnot
<nicolas(a)ecarnot.net> wrote:
> Hello,
>
> We're planning a move from our old building towards a new one a few meters
> away.
>
>
>
> In a similar way of Martijn
> (https://www.mail-archive.com/users@ovirt.org/msg33182.html), I have
> maintenance planed on our storage side.
>
> Say an oVirt DC is using a SAN's LUN via iSCSI (Equallogic).
> This SAN allows me to setup block replication between two SANs, seen by
> oVirt as one (Dell is naming it SyncRep).
> Then switch all the iSCSI accesses to the replicated LUN.
>
> When doing this, the iSCSI stack of each oVirt host notices the
> de-connection, tries to reconnect, and succeeds.
> Amongst our hosts, this happens between 4 and 15 seconds.
>
> When this happens fast enough, oVirt engine and the VMs don't even notice,
> and they keep running happily.
>
> When this takes more than 4 seconds, there are 2 cases :
>
> 1 - The hosts and/or oVirt and/or the SPM (I actually don't know) notices
> that there is a storage failure, and pauses the VMs.
> When the iSCSI stack reconnects, the VMs are automatically recovered from
> pause, and this all takes less than 30 seconds. That is very acceptable for
> us, as this action is extremely rare.
>
> 2 - Same storage failure, VMs paused, and some VMs stay in pause mode
> forever.
> Manual "run" action is mandatory.
> When done, everything recovers correctly.
> This is also quite acceptable, but here come my questions :
>
> My questions : (!)
> - *WHAT* process or piece of code or what oVirt parts is responsible for
> deciding when to UN-pause a VM, and at what conditions?
Vms get paused by qemu, when you get ENOSPC or some other IO error.
This probably happens when a vm is writing to storage, and all paths to storage
are faulty - with current configuration, the scsi layer will fail
after 5 seconds,
and if no path is available, the write will fail.
If vdsm storage monitoring system detected the issue, the storage domain
will become invalid. When the storage domain will become valid again, we
try to resume all vms paused because of IO errors.
Storage monitoring is done every 10 seconds in normal conditions, but in
current release, there can be delays of up to couple of minutes in
extreme conditions,
for example, 50 storage domains and doing lot of io. So basically, the
storage domain
monitor may miss an error on storage, never become invalid, and would
never become valid again and the vm will have to be resumed manually.
See https://bugzilla.redhat.com/1081962
In ovirt 4.0 monitoring should be improved, and will always monitor
storage every
10 seconds, but even this cannot guarantee that we will detect all
storage errors
For example, if the storage outage is shorter then 10 seconds. But I
guess that chance
that storage outage was shorter then 10 seconds, but long enough to cause a vm
to pause is very low.
> That would help me to understand why some cases are working even more
> smoothly than others.
> - Are there related timeouts I could play with in engine-config options?
Nothing on the engine side...
> - [a bit off-topic] Is it safe to increase some iSCSI timeouts of
> buffer-sizes in the hope this kind of disconnection would get un-noticed?
But you may modify multipath configuration on the host.
We use now this multipath configuration (/etc/multipath.conf):
# VDSM REVISION 1.3
defaults {
polling_interval 5
no_path_retry fail
user_friendly_names no
flush_on_last_del yes
fast_io_fail_tmo 5
dev_loss_tmo 30
max_fds 4096
deferred_remove yes
}
devices {
device {
all_devs yes
no_path_retry fail
}
}
This enforces failing of io request on devices that by default will queue such
requests for long or unlimited time. Queuing requests is very bad for vdsm, and
cause various commands to block for minutes during storage outage,
failing various
flows in vdsm and the ui.
See https://bugzilla.redhat.com/880738
However, in your case, using queuing may be the best way to do the switch
from one storage to another in the smoothest way.
You may try this setting:
devices {
device {
all_devs yes
no_path_retry 30
}
}
This will queue io requests for 30 seconds before failing.
Using this normally would be a bad idea with vdsm, since during storage outage,
vdsm may block for 30 seconds when no paths is available, and is not designed
for this behavior, but blocking from time to time for short time should be ok.
I think that modifying the configuration and reloading multipathd service should
be enough to use the new settings, but I'm not sure if this changes
existing sessions
or open devices.
Adding Ben to add more info about this.
Reloading the multipathd service will update this setting on all
existing devices. Outside of vdsm, multipath settings like this are
fairly common. So, from multipath's point of view this is completely
reasonable.
-Ben
Nir
10:32 a.m.
--_000_14647123411738133camelacroniscom_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
UGxlYXNlIG5vdGUgdGhhdCBpdCdzIG5lY2Vzc2FyeSB0byBhZGQgYSBtYWdpYyBsaW5lICcjIFZE
U00gUFJJVkFURScgYXMgc2Vjb25kIGxpbmUgaW4gL2V0Yy9tdWx0aXBhdGguY29uZi4gT3RoZXJ3
aXNlIHZkc20gd291bGQgb3ZlcndyaXRlIHlvdXIgc2V0dGluZ3MuDQpUaHVzLCAvZXRjL211bHRp
cGF0aC5jb25mIHNob3VsZCBzdGFydCB3aXRoIHRoZSBmb2xsb3dpbmcgdHdvIGxpbmVzOg0KIyBW
RFNNIFJFVklTSU9OIDEuMw0KIyBWRFNNIFBSSVZBVEUNCg0KDQpPbiBNb24sIDIwMTYtMDUtMzAg
YXQgMjI6MDkgKzAzMDAsIE5pciBTb2ZmZXIgd3JvdGU6DQoNCkJ1dCB5b3UgbWF5IG1vZGlmeSBt
dWx0aXBhdGggY29uZmlndXJhdGlvbiBvbiB0aGUgaG9zdC4NCg0KV2UgdXNlIG5vdyB0aGlzIG11
bHRpcGF0aCBjb25maWd1cmF0aW9uICgvZXRjL211bHRpcGF0aC5jb25mKToNCg0KIyBWRFNNIFJF
VklTSU9OIDEuMw0KDQpkZWZhdWx0cyB7DQogICAgcG9sbGluZ19pbnRlcnZhbCAgICAgICAgICAg
IDUNCiAgICBub19wYXRoX3JldHJ5ICAgICAgICAgICAgICAgZmFpbA0KICAgIHVzZXJfZnJpZW5k
bHlfbmFtZXMgICAgICAgICBubw0KICAgIGZsdXNoX29uX2xhc3RfZGVsICAgICAgICAgICB5ZXMN
CiAgICBmYXN0X2lvX2ZhaWxfdG1vICAgICAgICAgICAgNQ0KICAgIGRldl9sb3NzX3RtbyAgICAg
ICAgICAgICAgICAzMA0KICAgIG1heF9mZHMgICAgICAgICAgICAgICAgICAgICA0MDk2DQogICAg
ZGVmZXJyZWRfcmVtb3ZlICAgICAgICAgICAgIHllcw0KfQ0KDQpkZXZpY2VzIHsNCiAgICBkZXZp
Y2Ugew0KICAgICAgICBhbGxfZGV2cyAgICAgICAgICAgICAgICB5ZXMNCiAgICAgICAgbm9fcGF0
aF9yZXRyeSAgICAgICAgICAgZmFpbA0KICAgIH0NCn0NCg0KVGhpcyBlbmZvcmNlcyBmYWlsaW5n
IG9mIGlvIHJlcXVlc3Qgb24gZGV2aWNlcyB0aGF0IGJ5IGRlZmF1bHQgd2lsbCBxdWV1ZSBzdWNo
DQpyZXF1ZXN0cyBmb3IgbG9uZyBvciB1bmxpbWl0ZWQgdGltZS4gUXVldWluZyByZXF1ZXN0cyBp
cyB2ZXJ5IGJhZCBmb3IgdmRzbSwgYW5kDQpjYXVzZSB2YXJpb3VzIGNvbW1hbmRzIHRvIGJsb2Nr
IGZvciBtaW51dGVzIGR1cmluZyBzdG9yYWdlIG91dGFnZSwNCmZhaWxpbmcgdmFyaW91cw0KZmxv
d3MgaW4gdmRzbSBhbmQgdGhlIHVpLg0KU2VlIGh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS84
ODA3MzgNCg0KSG93ZXZlciwgaW4geW91ciBjYXNlLCB1c2luZyBxdWV1aW5nIG1heSBiZSB0aGUg
YmVzdCB3YXkgdG8gZG8gdGhlIHN3aXRjaA0KZnJvbSBvbmUgc3RvcmFnZSB0byBhbm90aGVyIGlu
IHRoZSBzbW9vdGhlc3Qgd2F5Lg0KDQpZb3UgbWF5IHRyeSB0aGlzIHNldHRpbmc6DQoNCmRldmlj
ZXMgew0KICAgIGRldmljZSB7DQogICAgICAgIGFsbF9kZXZzICAgICAgICAgICAgICAgIHllcw0K
ICAgICAgICBub19wYXRoX3JldHJ5ICAgICAgICAgICAzMA0KICAgIH0NCn0NCg0KVGhpcyB3aWxs
IHF1ZXVlIGlvIHJlcXVlc3RzIGZvciAzMCBzZWNvbmRzIGJlZm9yZSBmYWlsaW5nLg0KVXNpbmcg
dGhpcyBub3JtYWxseSB3b3VsZCBiZSBhIGJhZCBpZGVhIHdpdGggdmRzbSwgc2luY2UgZHVyaW5n
IHN0b3JhZ2Ugb3V0YWdlLA0KdmRzbSBtYXkgYmxvY2sgZm9yIDMwIHNlY29uZHMgd2hlbiBubyBw
YXRocyBpcyBhdmFpbGFibGUsIGFuZCBpcyBub3QgZGVzaWduZWQNCmZvciB0aGlzIGJlaGF2aW9y
LCBidXQgYmxvY2tpbmcgZnJvbSB0aW1lIHRvIHRpbWUgZm9yIHNob3J0IHRpbWUgc2hvdWxkIGJl
IG9rLg0KDQpJIHRoaW5rIHRoYXQgbW9kaWZ5aW5nIHRoZSBjb25maWd1cmF0aW9uIGFuZCByZWxv
YWRpbmcgbXVsdGlwYXRoZCBzZXJ2aWNlIHNob3VsZA0KYmUgZW5vdWdoIHRvIHVzZSB0aGUgbmV3
IHNldHRpbmdzLCBidXQgSSdtIG5vdCBzdXJlIGlmIHRoaXMgY2hhbmdlcw0KZXhpc3Rpbmcgc2Vz
c2lvbnMNCm9yIG9wZW4gZGV2aWNlcy4NCg0KDQoNCg==
--_000_14647123411738133camelacroniscom_
Content-Type: text/html; charset="utf-8"
Content-ID: <206222F74072F048B72C6E762EF4BDE5(a)acronis.com>
Content-Transfer-Encoding: base64
PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i
dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5Pg0KPGRpdj5QbGVhc2Ug
bm90ZSB0aGF0IGl0J3MgbmVjZXNzYXJ5IHRvIGFkZCBhIG1hZ2ljIGxpbmUgJyMgVkRTTSBQUklW
QVRFJyBhcyBzZWNvbmQgbGluZSBpbiAvZXRjL211bHRpcGF0aC5jb25mLiBPdGhlcndpc2UgdmRz
bSB3b3VsZCBvdmVyd3JpdGUgeW91ciBzZXR0aW5ncy48L2Rpdj4NCjxkaXY+VGh1cywgL2V0Yy9t
dWx0aXBhdGguY29uZiBzaG91bGQgc3RhcnQgd2l0aCB0aGUgZm9sbG93aW5nIHR3byBsaW5lczo8
L2Rpdj4NCjxkaXY+DQo8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj48L2Jsb2NrcXVvdGU+DQo8L2Rp
dj4NCjxkaXY+IyBWRFNNIFJFVklTSU9OIDEuMzwvZGl2Pg0KPGRpdj4jIFZEU00gUFJJVkFURTwv
ZGl2Pg0KPGRpdj48YnI+DQo8L2Rpdj4NCjxkaXY+PGJyPg0KPC9kaXY+DQo8ZGl2Pk9uIE1vbiwg
MjAxNi0wNS0zMCBhdCAyMjowOSAmIzQzOzAzMDAsIE5pciBTb2ZmZXIgd3JvdGU6PC9kaXY+DQo8
YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4NCjxwcmU+QnV0IHlvdSBtYXkgbW9kaWZ5IG11bHRpcGF0
aCBjb25maWd1cmF0aW9uIG9uIHRoZSBob3N0Lg0KDQpXZSB1c2Ugbm93IHRoaXMgbXVsdGlwYXRo
IGNvbmZpZ3VyYXRpb24gKC9ldGMvbXVsdGlwYXRoLmNvbmYpOg0KDQojIFZEU00gUkVWSVNJT04g
MS4zDQoNCmRlZmF1bHRzIHsNCiAgICBwb2xsaW5nX2ludGVydmFsICAgICAgICAgICAgNQ0KICAg
IG5vX3BhdGhfcmV0cnkgICAgICAgICAgICAgICBmYWlsDQogICAgdXNlcl9mcmllbmRseV9uYW1l
cyAgICAgICAgIG5vDQogICAgZmx1c2hfb25fbGFzdF9kZWwgICAgICAgICAgIHllcw0KICAgIGZh
c3RfaW9fZmFpbF90bW8gICAgICAgICAgICA1DQogICAgZGV2X2xvc3NfdG1vICAgICAgICAgICAg
ICAgIDMwDQogICAgbWF4X2ZkcyAgICAgICAgICAgICAgICAgICAgIDQwOTYNCiAgICBkZWZlcnJl
ZF9yZW1vdmUgICAgICAgICAgICAgeWVzDQp9DQoNCmRldmljZXMgew0KICAgIGRldmljZSB7DQog
ICAgICAgIGFsbF9kZXZzICAgICAgICAgICAgICAgIHllcw0KICAgICAgICBub19wYXRoX3JldHJ5
ICAgICAgICAgICBmYWlsDQogICAgfQ0KfQ0KDQpUaGlzIGVuZm9yY2VzIGZhaWxpbmcgb2YgaW8g
cmVxdWVzdCBvbiBkZXZpY2VzIHRoYXQgYnkgZGVmYXVsdCB3aWxsIHF1ZXVlIHN1Y2gNCnJlcXVl
c3RzIGZvciBsb25nIG9yIHVubGltaXRlZCB0aW1lLiBRdWV1aW5nIHJlcXVlc3RzIGlzIHZlcnkg
YmFkIGZvciB2ZHNtLCBhbmQNCmNhdXNlIHZhcmlvdXMgY29tbWFuZHMgdG8gYmxvY2sgZm9yIG1p
bnV0ZXMgZHVyaW5nIHN0b3JhZ2Ugb3V0YWdlLA0KZmFpbGluZyB2YXJpb3VzDQpmbG93cyBpbiB2
ZHNtIGFuZCB0aGUgdWkuDQpTZWUgPGEgaHJlZj0iaHR0cHM6Ly9idWd6aWxsYS5yZWRoYXQuY29t
Lzg4MDczOCI+aHR0cHM6Ly9idWd6aWxsYS5yZWRoYXQuY29tLzg4MDczODwvYT4NCg0KSG93ZXZl
ciwgaW4geW91ciBjYXNlLCB1c2luZyBxdWV1aW5nIG1heSBiZSB0aGUgYmVzdCB3YXkgdG8gZG8g
dGhlIHN3aXRjaA0KZnJvbSBvbmUgc3RvcmFnZSB0byBhbm90aGVyIGluIHRoZSBzbW9vdGhlc3Qg
d2F5Lg0KDQpZb3UgbWF5IHRyeSB0aGlzIHNldHRpbmc6DQoNCmRldmljZXMgew0KICAgIGRldmlj
ZSB7DQogICAgICAgIGFsbF9kZXZzICAgICAgICAgICAgICAgIHllcw0KICAgICAgICBub19wYXRo
X3JldHJ5ICAgICAgICAgICAzMA0KICAgIH0NCn0NCg0KVGhpcyB3aWxsIHF1ZXVlIGlvIHJlcXVl
c3RzIGZvciAzMCBzZWNvbmRzIGJlZm9yZSBmYWlsaW5nLg0KVXNpbmcgdGhpcyBub3JtYWxseSB3
b3VsZCBiZSBhIGJhZCBpZGVhIHdpdGggdmRzbSwgc2luY2UgZHVyaW5nIHN0b3JhZ2Ugb3V0YWdl
LA0KdmRzbSBtYXkgYmxvY2sgZm9yIDMwIHNlY29uZHMgd2hlbiBubyBwYXRocyBpcyBhdmFpbGFi
bGUsIGFuZCBpcyBub3QgZGVzaWduZWQNCmZvciB0aGlzIGJlaGF2aW9yLCBidXQgYmxvY2tpbmcg
ZnJvbSB0aW1lIHRvIHRpbWUgZm9yIHNob3J0IHRpbWUgc2hvdWxkIGJlIG9rLg0KDQpJIHRoaW5r
IHRoYXQgbW9kaWZ5aW5nIHRoZSBjb25maWd1cmF0aW9uIGFuZCByZWxvYWRpbmcgbXVsdGlwYXRo
ZCBzZXJ2aWNlIHNob3VsZA0KYmUgZW5vdWdoIHRvIHVzZSB0aGUgbmV3IHNldHRpbmdzLCBidXQg
SSdtIG5vdCBzdXJlIGlmIHRoaXMgY2hhbmdlcw0KZXhpc3Rpbmcgc2Vzc2lvbnMNCm9yIG9wZW4g
ZGV2aWNlcy4NCg0KDQo8L3ByZT4NCjwvYmxvY2txdW90ZT4NCjwvYm9keT4NCjwvaHRtbD4NCg==
--_000_14647123411738133camelacroniscom_--
2884
days inactive
2885
days old
8 comments
6 participants
participants (6)
-
Benjamin Marzinski -
InterNetX - Juergen Gotteswinter -
Michal Skrivanek -
Nicolas Ecarnot -
Nir Soffer -
Pavel Gashev