From stockhausen at collogia.de Thu Apr 3 06:13:38 2014 Content-Type: multipart/mixed; boundary="===============8266169886473504678==" MIME-Version: 1.0 From: Markus Stockhausen To: users at ovirt.org Subject: [Users] TSC clocksource gets lost after live migration Date: Thu, 03 Apr 2014 10:13:35 +0000 Message-ID: <12EF8D94C6F8734FB2FF37B9FBEDD17358615C53@EXCHANGE.collogia.de> --===============8266169886473504678== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ------=3D_NextPartTM-000-bf1f3448-4cf7-4f81-b33a-060c7d113f51 Content-Type: multipart/alternative; boundary=3D"_000_12EF8D94C6F8734FB2FF37B9FBEDD17358615C53EXCHANGEcollogi_" --_000_12EF8D94C6F8734FB2FF37B9FBEDD17358615C53EXCHANGEcollogi_ Content-Type: text/plain; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable Hello, we have an up to date ovirt 3.4 installation. Inside we are running SLES11 = =3D SP3 VMs (Kernel 3.0.76-0.11). After live migration of these VMs they all of a s= =3D udden do not react any longer and CPU usage of the VM goes to 100%. We identified kvm-clock source to be the culprit and therefore switched to = =3D another clocksource. We ended with hpet but are not happy with that as our inital g= =3D oal was to use the more simple designed TSC clocksoure. The reason behind that is the question I have for you experts. Our hosts all have the constant_tsc CPU flag available. Just to mention the= =3D se are not identical hosts. We have a mix of Xeon 5500 and 5600 machines. E.G. [root(a)colovn01 ~]# cat /proc/cpuinfo | grep constant_tsc | wc -l 8 When we start the VM the client sees TSC as available clocksource: colvm53:~ # cat /sys/devices/system/clocksource/clocksource0/available_cloc= =3D ksource kvm-clock tsc hpet acpi_pm After the first live migration to another host that also has constant_tsc (= =3D see above) that flag is lost inside the VM. colvm53:~ # cat /sys/devices/system/clocksource/clocksource0/available_cloc= =3D ksource kvm-clock hpet acpi_pm Any ideas? Markus --_000_12EF8D94C6F8734FB2FF37B9FBEDD17358615C53EXCHANGEcollogi_ Content-Type: text/html; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hello,

we have an up to date ovirt 3.4 installation. Inside we are running SLES11 = =3D SP3
VMs (Kernel 3.0.76-0.11). After live migration of these VMs they all of a s= =3D udden
do not react any longer and CPU usage of the VM goes to 100%.

We identified kvm-clock source to be the culprit and therefore switched to = =3D another
clocksource. We ended with hpet but are not happy with that as our inital g= =3D oal
was to use the more simple designed TSC clocksoure.

The reason behind that is the question I have for you experts.

Our hosts all have the constant_tsc CPU flag available. Just to mention the= =3D se
are not identical hosts. We have a mix of Xeon 5500 and 5600 machines. E.G.= =3D
[root(a)colovn01 ~]# cat /proc/cpuinfo | grep constant_tsc | wc -l
8

When we start the VM the client sees TSC as available clocksource:

colvm53:~ # cat /sys/devices/system/clocksource/clocksource0/available_cloc= =3D ksource
kvm-clock tsc hpet acpi_pm

After the first live migration to another host that also has constant_tsc (= =3D see above)
that flag is lost inside the VM.

colvm53:~ # cat /sys/devices/system/clocksource/clocksource0/available_cloc= =3D ksource
kvm-clock hpet acpi_pm

Any ideas?

Markus


--_000_12EF8D94C6F8734FB2FF37B9FBEDD17358615C53EXCHANGEcollogi_-- ------=3D_NextPartTM-000-bf1f3448-4cf7-4f81-b33a-060c7d113f51 Content-Type: text/plain; name=3D"InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=3D"InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmlich erhalten haben, informieren Sie bitte sofort den Absender u= nd vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. =C3=9Cber das Internet versandte E-Mails k=C3=B6nnen unter fremden Namen er= stellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserkl=C3=A4rung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln Vorstand: Kadir Akin Dr. Michael H=C3=B6hnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht K=C3=B6ln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln executive board: Kadir Akin Dr. Michael H=C3=B6hnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=3D_NextPartTM-000-bf1f3448-4cf7-4f81-b33a-060c7d113f51-- --===============8266169886473504678== Content-Type: multipart/mixed MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tLS0tLT1fTmV4dFBhcnRUTS0wMDAtYmYxZjM0NDgtNGNmNy00ZjgxLWIzM2EtMDYwYzdkMTEz ZjUxCkNvbnRlbnQtVHlwZTogbXVsdGlwYXJ0L2FsdGVybmF0aXZlOwoJYm91bmRhcnk9Il8wMDBf MTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJFREQxNzM1ODYxNUM1M0VYQ0hBTkdFY29sbG9naV8i CgotLV8wMDBfMTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJFREQxNzM1ODYxNUM1M0VYQ0hBTkdF Y29sbG9naV8KQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFyc2V0PSJpc28tODg1OS0xIgpD b250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgpIZWxsbywKCndlIGhh dmUgYW4gdXAgdG8gZGF0ZSBvdmlydCAzLjQgaW5zdGFsbGF0aW9uLiBJbnNpZGUgd2UgYXJlIHJ1 bm5pbmcgU0xFUzExID0KU1AzClZNcyAoS2VybmVsIDMuMC43Ni0wLjExKS4gQWZ0ZXIgbGl2ZSBt aWdyYXRpb24gb2YgdGhlc2UgVk1zIHRoZXkgYWxsIG9mIGEgcz0KdWRkZW4KZG8gbm90IHJlYWN0 IGFueSBsb25nZXIgYW5kIENQVSB1c2FnZSBvZiB0aGUgVk0gZ29lcyB0byAxMDAlLgoKV2UgaWRl bnRpZmllZCBrdm0tY2xvY2sgc291cmNlIHRvIGJlIHRoZSBjdWxwcml0IGFuZCB0aGVyZWZvcmUg c3dpdGNoZWQgdG8gPQphbm90aGVyCmNsb2Nrc291cmNlLiBXZSBlbmRlZCB3aXRoIGhwZXQgYnV0 IGFyZSBub3QgaGFwcHkgd2l0aCB0aGF0IGFzIG91ciBpbml0YWwgZz0Kb2FsCndhcyB0byB1c2Ug dGhlIG1vcmUgc2ltcGxlIGRlc2lnbmVkIFRTQyBjbG9ja3NvdXJlLgoKVGhlIHJlYXNvbiBiZWhp bmQgdGhhdCBpcyB0aGUgcXVlc3Rpb24gSSBoYXZlIGZvciB5b3UgZXhwZXJ0cy4KCk91ciBob3N0 cyBhbGwgaGF2ZSB0aGUgY29uc3RhbnRfdHNjIENQVSBmbGFnIGF2YWlsYWJsZS4gSnVzdCB0byBt ZW50aW9uIHRoZT0Kc2UKYXJlIG5vdCBpZGVudGljYWwgaG9zdHMuIFdlIGhhdmUgYSBtaXggb2Yg WGVvbiA1NTAwIGFuZCA1NjAwIG1hY2hpbmVzLiBFLkcuCltyb290QGNvbG92bjAxIH5dIyBjYXQg L3Byb2MvY3B1aW5mbyB8IGdyZXAgY29uc3RhbnRfdHNjIHwgd2MgLWwKOAoKV2hlbiB3ZSBzdGFy dCB0aGUgVk0gdGhlIGNsaWVudCBzZWVzIFRTQyBhcyBhdmFpbGFibGUgY2xvY2tzb3VyY2U6Cgpj b2x2bTUzOn4gIyBjYXQgL3N5cy9kZXZpY2VzL3N5c3RlbS9jbG9ja3NvdXJjZS9jbG9ja3NvdXJj ZTAvYXZhaWxhYmxlX2Nsb2M9Cmtzb3VyY2UKa3ZtLWNsb2NrIHRzYyBocGV0IGFjcGlfcG0KCkFm dGVyIHRoZSBmaXJzdCBsaXZlIG1pZ3JhdGlvbiB0byBhbm90aGVyIGhvc3QgdGhhdCBhbHNvIGhh cyBjb25zdGFudF90c2MgKD0Kc2VlIGFib3ZlKQp0aGF0IGZsYWcgaXMgbG9zdCBpbnNpZGUgdGhl IFZNLgoKY29sdm01Mzp+ICMgY2F0IC9zeXMvZGV2aWNlcy9zeXN0ZW0vY2xvY2tzb3VyY2UvY2xv Y2tzb3VyY2UwL2F2YWlsYWJsZV9jbG9jPQprc291cmNlCmt2bS1jbG9jayBocGV0IGFjcGlfcG0K CkFueSBpZGVhcz8KCk1hcmt1cwoKCgotLV8wMDBfMTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJF REQxNzM1ODYxNUM1M0VYQ0hBTkdFY29sbG9naV8KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNo YXJzZXQ9Imlzby04ODU5LTEiCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmlu dGFibGUKCjxodG1sIGRpcj0zRCJsdHIiPgo8aGVhZD4KPG1ldGEgaHR0cC1lcXVpdj0zRCJDb250 ZW50LVR5cGUiIGNvbnRlbnQ9M0QidGV4dC9odG1sOyBjaGFyc2V0PTNEaXNvLTg4NTktPQoxIj4K PHN0eWxlIGlkPTNEIm93YVBhcmFTdHlsZSIgdHlwZT0zRCJ0ZXh0L2NzcyI+UCB7bWFyZ2luLXRv cDowO21hcmdpbi1ib3R0b206PQowO308L3N0eWxlPgo8L2hlYWQ+Cjxib2R5IG9jc2k9M0QiMCIg ZnBzdHlsZT0zRCIxIj4KPGRpdiBzdHlsZT0zRCJkaXJlY3Rpb246IGx0cjtmb250LWZhbWlseTog VGFob21hO2NvbG9yOiAjMDAwMDAwO2ZvbnQtc2l6ZTogPQoxMHB0OyI+SGVsbG8sPGJyPgo8YnI+ CndlIGhhdmUgYW4gdXAgdG8gZGF0ZSBvdmlydCAzLjQgaW5zdGFsbGF0aW9uLiBJbnNpZGUgd2Ug YXJlIHJ1bm5pbmcgU0xFUzExID0KU1AzPGJyPgpWTXMgKEtlcm5lbCAzLjAuNzYtMC4xMSkuIEFm dGVyIGxpdmUgbWlncmF0aW9uIG9mIHRoZXNlIFZNcyB0aGV5IGFsbCBvZiBhIHM9CnVkZGVuPGJy PgpkbyBub3QgcmVhY3QgYW55IGxvbmdlciBhbmQgQ1BVIHVzYWdlIG9mIHRoZSBWTSBnb2VzIHRv IDEwMCUuPGJyPgo8YnI+CldlIGlkZW50aWZpZWQga3ZtLWNsb2NrIHNvdXJjZSB0byBiZSB0aGUg Y3VscHJpdCBhbmQgdGhlcmVmb3JlIHN3aXRjaGVkIHRvID0KYW5vdGhlcjxicj4KY2xvY2tzb3Vy Y2UuIFdlIGVuZGVkIHdpdGggaHBldCBidXQgYXJlIG5vdCBoYXBweSB3aXRoIHRoYXQgYXMgb3Vy IGluaXRhbCBnPQpvYWw8YnI+CndhcyB0byB1c2UgdGhlIG1vcmUgc2ltcGxlIGRlc2lnbmVkIFRT QyBjbG9ja3NvdXJlLiA8YnI+Cjxicj4KVGhlIHJlYXNvbiBiZWhpbmQgdGhhdCBpcyB0aGUgcXVl c3Rpb24gSSBoYXZlIGZvciB5b3UgZXhwZXJ0cy48YnI+Cjxicj4KT3VyIGhvc3RzIGFsbCBoYXZl IHRoZSBjb25zdGFudF90c2MgQ1BVIGZsYWcgYXZhaWxhYmxlLiBKdXN0IHRvIG1lbnRpb24gdGhl PQpzZTxicj4KYXJlIG5vdCBpZGVudGljYWwgaG9zdHMuIFdlIGhhdmUgYSBtaXggb2YgWGVvbiA1 NTAwIGFuZCA1NjAwIG1hY2hpbmVzLiBFLkcuPQo8YnI+Cltyb290QGNvbG92bjAxIH5dIyBjYXQg L3Byb2MvY3B1aW5mbyB8IGdyZXAgY29uc3RhbnRfdHNjIHwgd2MgLWw8YnI+Cjg8YnI+Cjxicj4K V2hlbiB3ZSBzdGFydCB0aGUgVk0gdGhlIGNsaWVudCBzZWVzIFRTQyBhcyBhdmFpbGFibGUgY2xv Y2tzb3VyY2U6PGJyPgo8YnI+CmNvbHZtNTM6fiAjIGNhdCAvc3lzL2RldmljZXMvc3lzdGVtL2Ns b2Nrc291cmNlL2Nsb2Nrc291cmNlMC9hdmFpbGFibGVfY2xvYz0Ka3NvdXJjZTxicj4Ka3ZtLWNs b2NrIHRzYyBocGV0IGFjcGlfcG08YnI+Cjxicj4KQWZ0ZXIgdGhlIGZpcnN0IGxpdmUgbWlncmF0 aW9uIHRvIGFub3RoZXIgaG9zdCB0aGF0IGFsc28gaGFzIGNvbnN0YW50X3RzYyAoPQpzZWUgYWJv dmUpPGJyPgp0aGF0IGZsYWcgaXMgbG9zdCBpbnNpZGUgdGhlIFZNLjxicj4KPGJyPgpjb2x2bTUz On4gIyBjYXQgL3N5cy9kZXZpY2VzL3N5c3RlbS9jbG9ja3NvdXJjZS9jbG9ja3NvdXJjZTAvYXZh aWxhYmxlX2Nsb2M9Cmtzb3VyY2U8YnI+Cmt2bS1jbG9jayBocGV0IGFjcGlfcG08YnI+Cjxicj4K QW55IGlkZWFzPzxicj4KPGJyPgpNYXJrdXM8YnI+Cjxicj4KPGJyPgo8L2Rpdj4KPC9ib2R5Pgo8 L2h0bWw+CgotLV8wMDBfMTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJFREQxNzM1ODYxNUM1M0VY Q0hBTkdFY29sbG9naV8tLQoKLS0tLS0tPV9OZXh0UGFydFRNLTAwMC1iZjFmMzQ0OC00Y2Y3LTRm ODEtYjMzYS0wNjBjN2QxMTNmNTEKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOwoJbmFtZT0iSW50 ZXJTY2FuX0Rpc2NsYWltZXIudHh0IgpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA3Yml0CkNv bnRlbnQtRGlzcG9zaXRpb246IGF0dGFjaG1lbnQ7CglmaWxlbmFtZT0iSW50ZXJTY2FuX0Rpc2Ns YWltZXIudHh0IgoKKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKgpEaWVzZSBFLU1haWwgZW50aMOkbHQgdmVy dHJhdWxpY2hlIHVuZC9vZGVyIHJlY2h0bGljaCBnZXNjaMO8dHp0ZQpJbmZvcm1hdGlvbmVuLiBX ZW5uIFNpZSBuaWNodCBkZXIgcmljaHRpZ2UgQWRyZXNzYXQgc2luZCBvZGVyIGRpZXNlIEUtTWFp bAppcnJ0w7xtbGljaCBlcmhhbHRlbiBoYWJlbiwgaW5mb3JtaWVyZW4gU2llIGJpdHRlIHNvZm9y dCBkZW4gQWJzZW5kZXIgdW5kCnZlcm5pY2h0ZW4gU2llIGRpZXNlIE1haWwuIERhcyB1bmVybGF1 YnRlIEtvcGllcmVuIHNvd2llIGRpZSB1bmJlZnVndGUKV2VpdGVyZ2FiZSBkaWVzZXIgTWFpbCBp c3QgbmljaHQgZ2VzdGF0dGV0LgoKw5xiZXIgZGFzIEludGVybmV0IHZlcnNhbmR0ZSBFLU1haWxz IGvDtm5uZW4gdW50ZXIgZnJlbWRlbiBOYW1lbiBlcnN0ZWxsdCBvZGVyCm1hbmlwdWxpZXJ0IHdl cmRlbi4gRGVzaGFsYiBpc3QgZGllc2UgYWxzIEUtTWFpbCB2ZXJzY2hpY2t0ZSBOYWNocmljaHQg a2VpbmUKcmVjaHRzdmVyYmluZGxpY2hlIFdpbGxlbnNlcmtsw6RydW5nLgoKQ29sbG9naWEKVW50 ZXJuZWhtZW5zYmVyYXR1bmcgQUcKVWJpZXJyaW5nIDExCkQtNTA2NzggS8O2bG4KClZvcnN0YW5k OgpLYWRpciBBa2luCkRyLiBNaWNoYWVsIEjDtmhuZXJiYWNoCgpWb3JzaXR6ZW5kZXIgZGVzIEF1 ZnNpY2h0c3JhdGVzOgpIYW5zIEtyaXN0aWFuIExhbmd2YQoKUmVnaXN0ZXJnZXJpY2h0OiBBbXRz Z2VyaWNodCBLw7ZsbgpSZWdpc3Rlcm51bW1lcjogSFJCIDUyIDQ5NwoKVGhpcyBlLW1haWwgbWF5 IGNvbnRhaW4gY29uZmlkZW50aWFsIGFuZC9vciBwcml2aWxlZ2VkIGluZm9ybWF0aW9uLiBJZiB5 b3UKYXJlIG5vdCB0aGUgaW50ZW5kZWQgcmVjaXBpZW50IChvciBoYXZlIHJlY2VpdmVkIHRoaXMg ZS1tYWlsIGluIGVycm9yKQpwbGVhc2Ugbm90aWZ5IHRoZSBzZW5kZXIgaW1tZWRpYXRlbHkgYW5k IGRlc3Ryb3kgdGhpcyBlLW1haWwuIEFueQp1bmF1dGhvcml6ZWQgY29weWluZywgZGlzY2xvc3Vy ZSBvciBkaXN0cmlidXRpb24gb2YgdGhlIG1hdGVyaWFsIGluIHRoaXMKZS1tYWlsIGlzIHN0cmlj dGx5IGZvcmJpZGRlbi4KCmUtbWFpbHMgc2VudCBvdmVyIHRoZSBpbnRlcm5ldCBtYXkgaGF2ZSBi ZWVuIHdyaXR0ZW4gdW5kZXIgYSB3cm9uZyBuYW1lIG9yCmJlZW4gbWFuaXB1bGF0ZWQuIFRoYXQg aXMgd2h5IHRoaXMgbWVzc2FnZSBzZW50IGFzIGFuIGUtbWFpbCBpcyBub3QgYQpsZWdhbGx5IGJp bmRpbmcgZGVjbGFyYXRpb24gb2YgaW50ZW50aW9uLgoKQ29sbG9naWEKVW50ZXJuZWhtZW5zYmVy YXR1bmcgQUcKVWJpZXJyaW5nIDExCkQtNTA2NzggS8O2bG4KCmV4ZWN1dGl2ZSBib2FyZDoKS2Fk aXIgQWtpbgpEci4gTWljaGFlbCBIw7ZobmVyYmFjaAoKUHJlc2lkZW50IG9mIHRoZSBzdXBlcnZp c29yeSBib2FyZDoKSGFucyBLcmlzdGlhbiBMYW5ndmEKClJlZ2lzdHJ5IG9mZmljZTogZGlzdHJp Y3QgY291cnQgQ29sb2duZQpSZWdpc3RlciBudW1iZXI6IEhSQiA1MiA0OTcKCioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioKCi0tLS0tLT1fTmV4dFBhcnRUTS0wMDAtYmYxZjM0NDgtNGNmNy00ZjgxLWIzM2Et MDYwYzdkMTEzZjUxLS0KCg== --===============8266169886473504678==-- From michal.skrivanek at redhat.com Fri Apr 4 09:33:25 2014 Content-Type: multipart/mixed; boundary="===============8530820087776768257==" MIME-Version: 1.0 From: Michal Skrivanek To: users at ovirt.org Subject: Re: [Users] TSC clocksource gets lost after live migration Date: Fri, 04 Apr 2014 15:33:17 +0200 Message-ID: <56307342-39C8-4D67-B92E-E68201B68F89@redhat.com> In-Reply-To: 12EF8D94C6F8734FB2FF37B9FBEDD17358615C53@EXCHANGE.collogia.de --===============8530820087776768257== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_BA43FE41-733B-497D-BFE0-FAD8F2CAE57A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dwindows-1252 Hi, this is more for the KVM folks I suppose=3D85can you get the qemu process = =3D cmdline please? Thanks, michal On 3 Apr 2014, at 12:13, Markus Stockhausen wrote: > Hello, >=3D20 > we have an up to date ovirt 3.4 installation. Inside we are running =3D SLES11 SP3 > VMs (Kernel 3.0.76-0.11). After live migration of these VMs they all =3D of a sudden > do not react any longer and CPU usage of the VM goes to 100%. >=3D20 > We identified kvm-clock source to be the culprit and therefore =3D switched to another > clocksource. We ended with hpet but are not happy with that as our =3D inital goal > was to use the more simple designed TSC clocksoure.=3D20 >=3D20 > The reason behind that is the question I have for you experts. >=3D20 > Our hosts all have the constant_tsc CPU flag available. Just to =3D mention these > are not identical hosts. We have a mix of Xeon 5500 and 5600 machines. = =3D E.G. > [root(a)colovn01 ~]# cat /proc/cpuinfo | grep constant_tsc | wc -l > 8 >=3D20 > When we start the VM the client sees TSC as available clocksource: >=3D20 > colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource > kvm-clock tsc hpet acpi_pm >=3D20 > After the first live migration to another host that also has =3D constant_tsc (see above) > that flag is lost inside the VM. >=3D20 > colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource > kvm-clock hpet acpi_pm >=3D20 > Any ideas? >=3D20 > Markus >=3D20 >=3D20 > =3D _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --Apple-Mail=3D_BA43FE41-733B-497D-BFE0-FAD8F2CAE57A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dwindows-1252 Hi,
this is more for the KVM folks I =3D suppose=3D85can you get the qemu process cmdline =3D please?

Thanks,
michal

On 3 Apr 2014, at 12:13, Markus Stockhausen wrote:

Hello,

we have an up to date =3D ovirt 3.4 installation. Inside we are running SLES11 SP3
VMs (Kernel =3D 3.0.76-0.11). After live migration of these VMs they all of a =3D sudden
do not react any longer and CPU usage of the VM goes to =3D 100%.

We identified kvm-clock source to be the culprit and =3D therefore switched to another
clocksource. We ended with hpet but are = =3D not happy with that as our inital goal
was to use the more simple =3D designed TSC clocksoure. 

The reason behind = =3D that is the question I have for you experts.

Our hosts all have =3D the constant_tsc CPU flag available. Just to mention these
are not =3D identical hosts. We have a mix of Xeon 5500 and 5600 machines. =3D E.G.
[root(a)colovn01 ~]# cat /proc/cpuinfo | grep constant_tsc | wc =3D -l
8

When we start the VM the client sees TSC as available =3D clocksource:

colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-= =3D clock tsc hpet acpi_pm

After the first live migration to another =3D host that also has constant_tsc (see above)
that flag is lost inside =3D the VM.

colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-= =3D clock hpet acpi_pm

Any =3D ideas?

Markus


<InterScan_Disclaimer.txt&g= =3D t;_______________________________________________
Users =3D mailing list
Users(a)ovirt.org
http://lists.ovirt= .=3D org/mailman/listinfo/users

=3D --Apple-Mail=3D_BA43FE41-733B-497D-BFE0-FAD8F2CAE57A-- --===============8530820087776768257== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fQkE0M0ZFNDEtNzMzQi00OTdELUJGRTAtRkFEOEYyQ0FFNTdBCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD13aW5kb3dzLTEyNTIKCkhpLAp0aGlzIGlzIG1vcmUgZm9yIHRoZSBL Vk0gZm9sa3MgSSBzdXBwb3NlPTg1Y2FuIHlvdSBnZXQgdGhlIHFlbXUgcHJvY2VzcyA9CmNtZGxp bmUgcGxlYXNlPwoKVGhhbmtzLAptaWNoYWwKCk9uIDMgQXByIDIwMTQsIGF0IDEyOjEzLCBNYXJr dXMgU3RvY2toYXVzZW4gd3JvdGU6Cgo+IEhlbGxvLAo+PTIwCj4gd2UgaGF2ZSBhbiB1cCB0byBk YXRlIG92aXJ0IDMuNCBpbnN0YWxsYXRpb24uIEluc2lkZSB3ZSBhcmUgcnVubmluZyA9ClNMRVMx MSBTUDMKPiBWTXMgKEtlcm5lbCAzLjAuNzYtMC4xMSkuIEFmdGVyIGxpdmUgbWlncmF0aW9uIG9m IHRoZXNlIFZNcyB0aGV5IGFsbCA9Cm9mIGEgc3VkZGVuCj4gZG8gbm90IHJlYWN0IGFueSBsb25n ZXIgYW5kIENQVSB1c2FnZSBvZiB0aGUgVk0gZ29lcyB0byAxMDAlLgo+PTIwCj4gV2UgaWRlbnRp ZmllZCBrdm0tY2xvY2sgc291cmNlIHRvIGJlIHRoZSBjdWxwcml0IGFuZCB0aGVyZWZvcmUgPQpz d2l0Y2hlZCB0byBhbm90aGVyCj4gY2xvY2tzb3VyY2UuIFdlIGVuZGVkIHdpdGggaHBldCBidXQg YXJlIG5vdCBoYXBweSB3aXRoIHRoYXQgYXMgb3VyID0KaW5pdGFsIGdvYWwKPiB3YXMgdG8gdXNl IHRoZSBtb3JlIHNpbXBsZSBkZXNpZ25lZCBUU0MgY2xvY2tzb3VyZS49MjAKPj0yMAo+IFRoZSBy ZWFzb24gYmVoaW5kIHRoYXQgaXMgdGhlIHF1ZXN0aW9uIEkgaGF2ZSBmb3IgeW91IGV4cGVydHMu Cj49MjAKPiBPdXIgaG9zdHMgYWxsIGhhdmUgdGhlIGNvbnN0YW50X3RzYyBDUFUgZmxhZyBhdmFp bGFibGUuIEp1c3QgdG8gPQptZW50aW9uIHRoZXNlCj4gYXJlIG5vdCBpZGVudGljYWwgaG9zdHMu IFdlIGhhdmUgYSBtaXggb2YgWGVvbiA1NTAwIGFuZCA1NjAwIG1hY2hpbmVzLiA9CkUuRy4KPiBb cm9vdEBjb2xvdm4wMSB+XSMgY2F0IC9wcm9jL2NwdWluZm8gfCBncmVwIGNvbnN0YW50X3RzYyB8 IHdjIC1sCj4gOAo+PTIwCj4gV2hlbiB3ZSBzdGFydCB0aGUgVk0gdGhlIGNsaWVudCBzZWVzIFRT QyBhcyBhdmFpbGFibGUgY2xvY2tzb3VyY2U6Cj49MjAKPiBjb2x2bTUzOn4gIyBjYXQgPQovc3lz L2RldmljZXMvc3lzdGVtL2Nsb2Nrc291cmNlL2Nsb2Nrc291cmNlMC9hdmFpbGFibGVfY2xvY2tz b3VyY2UKPiBrdm0tY2xvY2sgdHNjIGhwZXQgYWNwaV9wbQo+PTIwCj4gQWZ0ZXIgdGhlIGZpcnN0 IGxpdmUgbWlncmF0aW9uIHRvIGFub3RoZXIgaG9zdCB0aGF0IGFsc28gaGFzID0KY29uc3RhbnRf dHNjIChzZWUgYWJvdmUpCj4gdGhhdCBmbGFnIGlzIGxvc3QgaW5zaWRlIHRoZSBWTS4KPj0yMAo+ IGNvbHZtNTM6fiAjIGNhdCA9Ci9zeXMvZGV2aWNlcy9zeXN0ZW0vY2xvY2tzb3VyY2UvY2xvY2tz b3VyY2UwL2F2YWlsYWJsZV9jbG9ja3NvdXJjZQo+IGt2bS1jbG9jayBocGV0IGFjcGlfcG0KPj0y MAo+IEFueSBpZGVhcz8KPj0yMAo+IE1hcmt1cwo+PTIwCj49MjAKPiA9CjxJbnRlclNjYW5fRGlz Y2xhaW1lci50eHQ+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18KPiBVc2VycyBtYWlsaW5nIGxpc3QKPiBVc2Vyc0BvdmlydC5vcmcKPiBodHRwOi8vbGlzdHMu b3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKCgotLUFwcGxlLU1haWw9X0JBNDNGRTQx LTczM0ItNDk3RC1CRkUwLUZBRDhGMkNBRTU3QQpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBx dW90ZWQtcHJpbnRhYmxlCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOwoJY2hhcnNldD13aW5kb3dz LTEyNTIKCjxodG1sPjxoZWFkPjxiYXNlIGhyZWY9M0QieC1tc2c6Ly8xNTc1LyI+PC9oZWFkPjxi b2R5IHN0eWxlPTNEIndvcmQtd3JhcDogPQpicmVhay13b3JkOyAtd2Via2l0LW5ic3AtbW9kZTog c3BhY2U7IC13ZWJraXQtbGluZS1icmVhazogPQphZnRlci13aGl0ZS1zcGFjZTsgIj5IaSw8ZGl2 PnRoaXMgaXMgbW9yZSBmb3IgdGhlIEtWTSBmb2xrcyBJID0Kc3VwcG9zZT04NWNhbiB5b3UgZ2V0 IHRoZSBxZW11IHByb2Nlc3MgY21kbGluZSA9CnBsZWFzZT88L2Rpdj48ZGl2Pjxicj48L2Rpdj48 ZGl2PlRoYW5rcyw8L2Rpdj48ZGl2Pm1pY2hhbDwvZGl2PjxkaXY+PGJyPjxkPQppdj48ZGl2Pk9u IDMgQXByIDIwMTQsIGF0IDEyOjEzLCBNYXJrdXMgU3RvY2toYXVzZW4gd3JvdGU6PC9kaXY+PGJy ID0KY2xhc3M9M0QiQXBwbGUtaW50ZXJjaGFuZ2UtbmV3bGluZSI+PGJsb2NrcXVvdGUgdHlwZT0z RCJjaXRlIj48c3BhbiA9CmNsYXNzPTNEIkFwcGxlLXN0eWxlLXNwYW4iIHN0eWxlPTNEImJvcmRl ci1jb2xsYXBzZTogc2VwYXJhdGU7ID0KZm9udC1mYW1pbHk6IEhlbHZldGljYTsgZm9udC1zdHls ZTogbm9ybWFsOyBmb250LXZhcmlhbnQ6IG5vcm1hbDsgPQpmb250LXdlaWdodDogbm9ybWFsOyBs ZXR0ZXItc3BhY2luZzogbm9ybWFsOyBsaW5lLWhlaWdodDogbm9ybWFsOyA9Cm9ycGhhbnM6IDI7 IHRleHQtYWxpZ246IC13ZWJraXQtYXV0bzsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zv cm06ID0Kbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd2lkb3dzOiAyOyB3b3JkLXNwYWNpbmc6 IDBweDsgPQotd2Via2l0LWJvcmRlci1ob3Jpem9udGFsLXNwYWNpbmc6IDBweDsgLXdlYmtpdC1i b3JkZXItdmVydGljYWwtc3BhY2luZzogPQowcHg7IC13ZWJraXQtdGV4dC1kZWNvcmF0aW9ucy1p bi1lZmZlY3Q6IG5vbmU7IC13ZWJraXQtdGV4dC1zaXplLWFkanVzdDogPQphdXRvOyAtd2Via2l0 LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IGZvbnQtc2l6ZTogbWVkaXVtOyAiPjxkaXYgb2NzaT0z RCIwIj0KIGZwc3R5bGU9M0QiMSI+PGRpdiBzdHlsZT0zRCJkaXJlY3Rpb246IGx0cjsgZm9udC1m YW1pbHk6IFRhaG9tYTsgY29sb3I6ID0KcmdiKDAsIDAsIDApOyBmb250LXNpemU6IDEwcHQ7ICI+ SGVsbG8sPGJyPjxicj53ZSBoYXZlIGFuIHVwIHRvIGRhdGUgPQpvdmlydCAzLjQgaW5zdGFsbGF0 aW9uLiBJbnNpZGUgd2UgYXJlIHJ1bm5pbmcgU0xFUzExIFNQMzxicj5WTXMgKEtlcm5lbCA9CjMu MC43Ni0wLjExKS4gQWZ0ZXIgbGl2ZSBtaWdyYXRpb24gb2YgdGhlc2UgVk1zIHRoZXkgYWxsIG9m IGEgPQpzdWRkZW48YnI+ZG8gbm90IHJlYWN0IGFueSBsb25nZXIgYW5kIENQVSB1c2FnZSBvZiB0 aGUgVk0gZ29lcyB0byA9CjEwMCUuPGJyPjxicj5XZSBpZGVudGlmaWVkIGt2bS1jbG9jayBzb3Vy Y2UgdG8gYmUgdGhlIGN1bHByaXQgYW5kID0KdGhlcmVmb3JlIHN3aXRjaGVkIHRvIGFub3RoZXI8 YnI+Y2xvY2tzb3VyY2UuIFdlIGVuZGVkIHdpdGggaHBldCBidXQgYXJlID0Kbm90IGhhcHB5IHdp dGggdGhhdCBhcyBvdXIgaW5pdGFsIGdvYWw8YnI+d2FzIHRvIHVzZSB0aGUgbW9yZSBzaW1wbGUg PQpkZXNpZ25lZCBUU0MgY2xvY2tzb3VyZS48c3BhbiA9CmNsYXNzPTNEIkFwcGxlLWNvbnZlcnRl ZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxicj48YnI+VGhlIHJlYXNvbiBiZWhpbmQgPQp0aGF0IGlz IHRoZSBxdWVzdGlvbiBJIGhhdmUgZm9yIHlvdSBleHBlcnRzLjxicj48YnI+T3VyIGhvc3RzIGFs bCBoYXZlID0KdGhlIGNvbnN0YW50X3RzYyBDUFUgZmxhZyBhdmFpbGFibGUuIEp1c3QgdG8gbWVu dGlvbiB0aGVzZTxicj5hcmUgbm90ID0KaWRlbnRpY2FsIGhvc3RzLiBXZSBoYXZlIGEgbWl4IG9m IFhlb24gNTUwMCBhbmQgNTYwMCBtYWNoaW5lcy4gPQpFLkcuPGJyPltyb290QGNvbG92bjAxIH5d IyBjYXQgL3Byb2MvY3B1aW5mbyB8IGdyZXAgY29uc3RhbnRfdHNjIHwgd2MgPQotbDxicj44PGJy Pjxicj5XaGVuIHdlIHN0YXJ0IHRoZSBWTSB0aGUgY2xpZW50IHNlZXMgVFNDIGFzIGF2YWlsYWJs ZSA9CmNsb2Nrc291cmNlOjxicj48YnI+Y29sdm01Mzp+ICMgY2F0ID0KL3N5cy9kZXZpY2VzL3N5 c3RlbS9jbG9ja3NvdXJjZS9jbG9ja3NvdXJjZTAvYXZhaWxhYmxlX2Nsb2Nrc291cmNlPGJyPmt2 bS09CmNsb2NrIHRzYyBocGV0IGFjcGlfcG08YnI+PGJyPkFmdGVyIHRoZSBmaXJzdCBsaXZlIG1p Z3JhdGlvbiB0byBhbm90aGVyID0KaG9zdCB0aGF0IGFsc28gaGFzIGNvbnN0YW50X3RzYyAoc2Vl IGFib3ZlKTxicj50aGF0IGZsYWcgaXMgbG9zdCBpbnNpZGUgPQp0aGUgVk0uPGJyPjxicj5jb2x2 bTUzOn4gIyBjYXQgPQovc3lzL2RldmljZXMvc3lzdGVtL2Nsb2Nrc291cmNlL2Nsb2Nrc291cmNl MC9hdmFpbGFibGVfY2xvY2tzb3VyY2U8YnI+a3ZtLT0KY2xvY2sgaHBldCBhY3BpX3BtPGJyPjxi cj5BbnkgPQppZGVhcz88YnI+PGJyPk1hcmt1czxicj48YnI+PGJyPjwvZGl2PjxzcGFuPiZsdDtJ bnRlclNjYW5fRGlzY2xhaW1lci50eHQmZz0KdDs8L3NwYW4+X19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX188YnI+VXNlcnMgPQptYWlsaW5nIGxpc3Q8YnI+PGEg PQpocmVmPTNEIm1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+ PGEgPQpocmVmPTNEImh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vy cyI+aHR0cDovL2xpc3RzLm92aXJ0Lj0Kb3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+PC9k aXY+PC9zcGFuPjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9kaXY+PC89CmJvZHk+PC9odG1sPj0K Ci0tQXBwbGUtTWFpbD1fQkE0M0ZFNDEtNzMzQi00OTdELUJGRTAtRkFEOEYyQ0FFNTdBLS0K --===============8530820087776768257==-- From darrell.budic at zenfire.com Fri Apr 4 19:19:16 2014 Content-Type: multipart/mixed; boundary="===============8031873497385404125==" MIME-Version: 1.0 From: Darrell Budic To: users at ovirt.org Subject: Re: [Users] TSC clocksource gets lost after live migration Date: Fri, 04 Apr 2014 18:19:55 -0500 Message-ID: In-Reply-To: 56307342-39C8-4D67-B92E-E68201B68F89@redhat.com --===============8031873497385404125== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_2FC1045B-BCA5-43A7-B186-961C9DFED565 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dwindows-1252 I see this on some guests as well, possibly relating to moving between =3D hosts with the same family of CPU but different absolute CPU speeds? -Darrell On Apr 4, 2014, at 8:33 AM, Michal Skrivanek =3D wrote: > Hi, > this is more for the KVM folks I suppose=3D85can you get the qemu =3D process cmdline please? >=3D20 > Thanks, > michal >=3D20 > On 3 Apr 2014, at 12:13, Markus Stockhausen wrote: >=3D20 >> Hello, >>=3D20 >> we have an up to date ovirt 3.4 installation. Inside we are running =3D SLES11 SP3 >> VMs (Kernel 3.0.76-0.11). After live migration of these VMs they all =3D of a sudden >> do not react any longer and CPU usage of the VM goes to 100%. >>=3D20 >> We identified kvm-clock source to be the culprit and therefore =3D switched to another >> clocksource. We ended with hpet but are not happy with that as our =3D inital goal >> was to use the more simple designed TSC clocksoure.=3D20 >>=3D20 >> The reason behind that is the question I have for you experts. >>=3D20 >> Our hosts all have the constant_tsc CPU flag available. Just to =3D mention these >> are not identical hosts. We have a mix of Xeon 5500 and 5600 =3D machines. E.G. >> [root(a)colovn01 ~]# cat /proc/cpuinfo | grep constant_tsc | wc -l >> 8 >>=3D20 >> When we start the VM the client sees TSC as available clocksource: >>=3D20 >> colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource >> kvm-clock tsc hpet acpi_pm >>=3D20 >> After the first live migration to another host that also has =3D constant_tsc (see above) >> that flag is lost inside the VM. >>=3D20 >> colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource >> kvm-clock hpet acpi_pm >>=3D20 >> Any ideas? >>=3D20 >> Markus >>=3D20 >>=3D20 >> =3D _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >=3D20 > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --Apple-Mail=3D_2FC1045B-BCA5-43A7-B186-961C9DFED565 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dwindows-1252 I see =3D this on some guests as well, possibly relating to moving between hosts =3D with the same family of CPU but different absolute CPU =3D speeds?

  =3D -Darrell

On Apr 4, 2014, at 8:33 AM, =3D Michal Skrivanek <michal.skrivanek(a)redhat.c= om> wrote:

Hi,
this is more for the KVM folks I =3D suppose=3D85can you get the qemu process cmdline =3D please?

Thanks,
michal

On 3 Apr 2014, at 12:13, Markus Stockhausen wrote:

Hello,

we have an up to date ovirt 3.4 installation. =3D Inside we are running SLES11 SP3
VMs (Kernel 3.0.76-0.11). After live = =3D migration of these VMs they all of a sudden
do not react any longer =3D and CPU usage of the VM goes to 100%.

We identified kvm-clock =3D source to be the culprit and therefore switched to =3D another
clocksource. We ended with hpet but are not happy with that =3D as our inital goal
was to use the more simple designed TSC =3D clocksoure. 

The= =3D reason behind that is the question I have for you experts.

Our =3D hosts all have the constant_tsc CPU flag available. Just to mention =3D these
are not identical hosts. We have a mix of Xeon 5500 and 5600 =3D machines. E.G.
[root(a)colovn01 ~]# cat /proc/cpuinfo | grep =3D constant_tsc | wc -l
8

When we start the VM the client sees =3D TSC as available clocksource:

colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-= =3D clock tsc hpet acpi_pm

After the first live migration to another =3D host that also has constant_tsc (see above)
that flag is lost inside =3D the VM.

colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-= =3D clock hpet acpi_pm

Any =3D ideas?

Markus


<InterScan_Disclaimer.txt>____= =3D ___________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt= .=3D org/mailman/listinfo/users

__= =3D _____________________________________________
Users mailing =3D list
Users(a)ovirt.org
http://lists.o= virt.=3D org/mailman/listinfo/users

= =3D --Apple-Mail=3D_2FC1045B-BCA5-43A7-B186-961C9DFED565-- --===============8031873497385404125== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fMkZDMTA0NUItQkNBNS00M0E3LUIxODYtOTYxQzlERkVENTY1CkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD13aW5kb3dzLTEyNTIKCkkgc2VlIHRoaXMgb24gc29tZSBndWVzdHMg YXMgd2VsbCwgcG9zc2libHkgcmVsYXRpbmcgdG8gbW92aW5nIGJldHdlZW4gPQpob3N0cyB3aXRo IHRoZSBzYW1lIGZhbWlseSBvZiBDUFUgYnV0IGRpZmZlcmVudCBhYnNvbHV0ZSBDUFUgc3BlZWRz PwoKICAtRGFycmVsbAoKT24gQXByIDQsIDIwMTQsIGF0IDg6MzMgQU0sIE1pY2hhbCBTa3JpdmFu ZWsgPQo8bWljaGFsLnNrcml2YW5la0ByZWRoYXQuY29tPiB3cm90ZToKCj4gSGksCj4gdGhpcyBp cyBtb3JlIGZvciB0aGUgS1ZNIGZvbGtzIEkgc3VwcG9zZT04NWNhbiB5b3UgZ2V0IHRoZSBxZW11 ID0KcHJvY2VzcyBjbWRsaW5lIHBsZWFzZT8KPj0yMAo+IFRoYW5rcywKPiBtaWNoYWwKPj0yMAo+ IE9uIDMgQXByIDIwMTQsIGF0IDEyOjEzLCBNYXJrdXMgU3RvY2toYXVzZW4gd3JvdGU6Cj49MjAK Pj4gSGVsbG8sCj4+PTIwCj4+IHdlIGhhdmUgYW4gdXAgdG8gZGF0ZSBvdmlydCAzLjQgaW5zdGFs bGF0aW9uLiBJbnNpZGUgd2UgYXJlIHJ1bm5pbmcgPQpTTEVTMTEgU1AzCj4+IFZNcyAoS2VybmVs IDMuMC43Ni0wLjExKS4gQWZ0ZXIgbGl2ZSBtaWdyYXRpb24gb2YgdGhlc2UgVk1zIHRoZXkgYWxs ID0Kb2YgYSBzdWRkZW4KPj4gZG8gbm90IHJlYWN0IGFueSBsb25nZXIgYW5kIENQVSB1c2FnZSBv ZiB0aGUgVk0gZ29lcyB0byAxMDAlLgo+Pj0yMAo+PiBXZSBpZGVudGlmaWVkIGt2bS1jbG9jayBz b3VyY2UgdG8gYmUgdGhlIGN1bHByaXQgYW5kIHRoZXJlZm9yZSA9CnN3aXRjaGVkIHRvIGFub3Ro ZXIKPj4gY2xvY2tzb3VyY2UuIFdlIGVuZGVkIHdpdGggaHBldCBidXQgYXJlIG5vdCBoYXBweSB3 aXRoIHRoYXQgYXMgb3VyID0KaW5pdGFsIGdvYWwKPj4gd2FzIHRvIHVzZSB0aGUgbW9yZSBzaW1w bGUgZGVzaWduZWQgVFNDIGNsb2Nrc291cmUuPTIwCj4+PTIwCj4+IFRoZSByZWFzb24gYmVoaW5k IHRoYXQgaXMgdGhlIHF1ZXN0aW9uIEkgaGF2ZSBmb3IgeW91IGV4cGVydHMuCj4+PTIwCj4+IE91 ciBob3N0cyBhbGwgaGF2ZSB0aGUgY29uc3RhbnRfdHNjIENQVSBmbGFnIGF2YWlsYWJsZS4gSnVz dCB0byA9Cm1lbnRpb24gdGhlc2UKPj4gYXJlIG5vdCBpZGVudGljYWwgaG9zdHMuIFdlIGhhdmUg YSBtaXggb2YgWGVvbiA1NTAwIGFuZCA1NjAwID0KbWFjaGluZXMuIEUuRy4KPj4gW3Jvb3RAY29s b3ZuMDEgfl0jIGNhdCAvcHJvYy9jcHVpbmZvIHwgZ3JlcCBjb25zdGFudF90c2MgfCB3YyAtbAo+ PiA4Cj4+PTIwCj4+IFdoZW4gd2Ugc3RhcnQgdGhlIFZNIHRoZSBjbGllbnQgc2VlcyBUU0MgYXMg YXZhaWxhYmxlIGNsb2Nrc291cmNlOgo+Pj0yMAo+PiBjb2x2bTUzOn4gIyBjYXQgPQovc3lzL2Rl dmljZXMvc3lzdGVtL2Nsb2Nrc291cmNlL2Nsb2Nrc291cmNlMC9hdmFpbGFibGVfY2xvY2tzb3Vy Y2UKPj4ga3ZtLWNsb2NrIHRzYyBocGV0IGFjcGlfcG0KPj49MjAKPj4gQWZ0ZXIgdGhlIGZpcnN0 IGxpdmUgbWlncmF0aW9uIHRvIGFub3RoZXIgaG9zdCB0aGF0IGFsc28gaGFzID0KY29uc3RhbnRf dHNjIChzZWUgYWJvdmUpCj4+IHRoYXQgZmxhZyBpcyBsb3N0IGluc2lkZSB0aGUgVk0uCj4+PTIw Cj4+IGNvbHZtNTM6fiAjIGNhdCA9Ci9zeXMvZGV2aWNlcy9zeXN0ZW0vY2xvY2tzb3VyY2UvY2xv Y2tzb3VyY2UwL2F2YWlsYWJsZV9jbG9ja3NvdXJjZQo+PiBrdm0tY2xvY2sgaHBldCBhY3BpX3Bt Cj4+PTIwCj4+IEFueSBpZGVhcz8KPj49MjAKPj4gTWFya3VzCj4+PTIwCj4+PTIwCj4+ID0KPElu dGVyU2Nhbl9EaXNjbGFpbWVyLnR4dD5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXwo+PiBVc2VycyBtYWlsaW5nIGxpc3QKPj4gVXNlcnNAb3ZpcnQub3JnCj4+ IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycwo+PTIwCj4gX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPiBVc2VycyBtYWls aW5nIGxpc3QKPiBVc2Vyc0BvdmlydC5vcmcKPiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxt YW4vbGlzdGluZm8vdXNlcnMKCgotLUFwcGxlLU1haWw9XzJGQzEwNDVCLUJDQTUtNDNBNy1CMTg2 LTk2MUM5REZFRDU2NQpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxl CkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOwoJY2hhcnNldD13aW5kb3dzLTEyNTIKCjxodG1sPjxo ZWFkPjxtZXRhIGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBlIiBjb250ZW50PTNEInRleHQvaHRt bCA9CmNoYXJzZXQ9M0R3aW5kb3dzLTEyNTIiPjwvaGVhZD48Ym9keSBzdHlsZT0zRCJ3b3JkLXdy YXA6IGJyZWFrLXdvcmQ7ID0KLXdlYmtpdC1uYnNwLW1vZGU6IHNwYWNlOyAtd2Via2l0LWxpbmUt YnJlYWs6IGFmdGVyLXdoaXRlLXNwYWNlOyI+SSBzZWUgPQp0aGlzIG9uIHNvbWUgZ3Vlc3RzIGFz IHdlbGwsIHBvc3NpYmx5IHJlbGF0aW5nIHRvIG1vdmluZyBiZXR3ZWVuIGhvc3RzID0Kd2l0aCB0 aGUgc2FtZSBmYW1pbHkgb2YgQ1BVIGJ1dCBkaWZmZXJlbnQgYWJzb2x1dGUgQ1BVID0Kc3BlZWRz PzxkaXY+PGJyPjwvZGl2PjxkaXY+Jm5ic3A7ID0KLURhcnJlbGw8L2Rpdj48ZGl2Pjxicj48L2Rp dj48ZGl2PjxkaXY+PGRpdj5PbiBBcHIgNCwgMjAxNCwgYXQgODozMyBBTSwgPQpNaWNoYWwgU2ty aXZhbmVrICZsdDs8YSA9CmhyZWY9M0QibWFpbHRvOm1pY2hhbC5za3JpdmFuZWtAcmVkaGF0LmNv bSI+bWljaGFsLnNrcml2YW5la0ByZWRoYXQuY29tPC9hPQo+Jmd0OyB3cm90ZTo8L2Rpdj48YnIg Y2xhc3M9M0QiQXBwbGUtaW50ZXJjaGFuZ2UtbmV3bGluZSI+PGJsb2NrcXVvdGUgPQp0eXBlPTNE ImNpdGUiPjxiYXNlIGhyZWY9M0QieC1tc2c6Ly8xNTc1LyI+PGRpdiBzdHlsZT0zRCJ3b3JkLXdy YXA6ID0KYnJlYWstd29yZDsgLXdlYmtpdC1uYnNwLW1vZGU6IHNwYWNlOyAtd2Via2l0LWxpbmUt YnJlYWs6ID0KYWZ0ZXItd2hpdGUtc3BhY2U7ICI+SGksPGRpdj50aGlzIGlzIG1vcmUgZm9yIHRo ZSBLVk0gZm9sa3MgSSA9CnN1cHBvc2U9ODVjYW4geW91IGdldCB0aGUgcWVtdSBwcm9jZXNzIGNt ZGxpbmUgPQpwbGVhc2U/PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5UaGFua3MsPC9kaXY+PGRp dj5taWNoYWw8L2Rpdj48ZGl2Pjxicj48ZD0KaXY+PGRpdj5PbiAzIEFwciAyMDE0LCBhdCAxMjox MywgTWFya3VzIFN0b2NraGF1c2VuIHdyb3RlOjwvZGl2PjxiciA9CmNsYXNzPTNEIkFwcGxlLWlu dGVyY2hhbmdlLW5ld2xpbmUiPjxibG9ja3F1b3RlIHR5cGU9M0QiY2l0ZSI+PHNwYW4gPQpjbGFz cz0zRCJBcHBsZS1zdHlsZS1zcGFuIiBzdHlsZT0zRCJib3JkZXItY29sbGFwc2U6IHNlcGFyYXRl OyA9CmZvbnQtZmFtaWx5OiBIZWx2ZXRpY2E7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJp YW50OiBub3JtYWw7ID0KZm9udC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1h bDsgbGluZS1oZWlnaHQ6IG5vcm1hbDsgPQpvcnBoYW5zOiAyOyB0ZXh0LWFsaWduOiAtd2Via2l0 LWF1dG87IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiA9Cm5vbmU7IHdoaXRlLXNw YWNlOiBub3JtYWw7IHdpZG93czogMjsgd29yZC1zcGFjaW5nOiAwcHg7IGJvcmRlci1zcGFjaW5n OiA9CjBweDsgLXdlYmtpdC10ZXh0LWRlY29yYXRpb25zLWluLWVmZmVjdDogbm9uZTsgPQotd2Vi a2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IGZvbnQtc2l6ZTogaW5oZXJpdDsiPjxkaXYgPQpz dHlsZT0zRCJkaXJlY3Rpb246IGx0cjsgZm9udC1mYW1pbHk6IFRhaG9tYTsgZm9udC1zaXplOiA9 CjEwcHQ7Ij5IZWxsbyw8YnI+PGJyPndlIGhhdmUgYW4gdXAgdG8gZGF0ZSBvdmlydCAzLjQgaW5z dGFsbGF0aW9uLiA9Ckluc2lkZSB3ZSBhcmUgcnVubmluZyBTTEVTMTEgU1AzPGJyPlZNcyAoS2Vy bmVsIDMuMC43Ni0wLjExKS4gQWZ0ZXIgbGl2ZSA9Cm1pZ3JhdGlvbiBvZiB0aGVzZSBWTXMgdGhl eSBhbGwgb2YgYSBzdWRkZW48YnI+ZG8gbm90IHJlYWN0IGFueSBsb25nZXIgPQphbmQgQ1BVIHVz YWdlIG9mIHRoZSBWTSBnb2VzIHRvIDEwMCUuPGJyPjxicj5XZSBpZGVudGlmaWVkIGt2bS1jbG9j ayA9CnNvdXJjZSB0byBiZSB0aGUgY3VscHJpdCBhbmQgdGhlcmVmb3JlIHN3aXRjaGVkIHRvID0K YW5vdGhlcjxicj5jbG9ja3NvdXJjZS4gV2UgZW5kZWQgd2l0aCBocGV0IGJ1dCBhcmUgbm90IGhh cHB5IHdpdGggdGhhdCA9CmFzIG91ciBpbml0YWwgZ29hbDxicj53YXMgdG8gdXNlIHRoZSBtb3Jl IHNpbXBsZSBkZXNpZ25lZCBUU0MgPQpjbG9ja3NvdXJlLjxzcGFuIGNsYXNzPTNEIkFwcGxlLWNv bnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxicj48YnI+VGhlID0KcmVhc29uIGJlaGluZCB0 aGF0IGlzIHRoZSBxdWVzdGlvbiBJIGhhdmUgZm9yIHlvdSBleHBlcnRzLjxicj48YnI+T3VyID0K aG9zdHMgYWxsIGhhdmUgdGhlIGNvbnN0YW50X3RzYyBDUFUgZmxhZyBhdmFpbGFibGUuIEp1c3Qg dG8gbWVudGlvbiA9CnRoZXNlPGJyPmFyZSBub3QgaWRlbnRpY2FsIGhvc3RzLiBXZSBoYXZlIGEg bWl4IG9mIFhlb24gNTUwMCBhbmQgNTYwMCA9Cm1hY2hpbmVzLiBFLkcuPGJyPltyb290QGNvbG92 bjAxIH5dIyBjYXQgL3Byb2MvY3B1aW5mbyB8IGdyZXAgPQpjb25zdGFudF90c2MgfCB3YyAtbDxi cj44PGJyPjxicj5XaGVuIHdlIHN0YXJ0IHRoZSBWTSB0aGUgY2xpZW50IHNlZXMgPQpUU0MgYXMg YXZhaWxhYmxlIGNsb2Nrc291cmNlOjxicj48YnI+Y29sdm01Mzp+ICMgY2F0ID0KL3N5cy9kZXZp Y2VzL3N5c3RlbS9jbG9ja3NvdXJjZS9jbG9ja3NvdXJjZTAvYXZhaWxhYmxlX2Nsb2Nrc291cmNl PGJyPmt2bS09CmNsb2NrIHRzYyBocGV0IGFjcGlfcG08YnI+PGJyPkFmdGVyIHRoZSBmaXJzdCBs aXZlIG1pZ3JhdGlvbiB0byBhbm90aGVyID0KaG9zdCB0aGF0IGFsc28gaGFzIGNvbnN0YW50X3Rz YyAoc2VlIGFib3ZlKTxicj50aGF0IGZsYWcgaXMgbG9zdCBpbnNpZGUgPQp0aGUgVk0uPGJyPjxi cj5jb2x2bTUzOn4gIyBjYXQgPQovc3lzL2RldmljZXMvc3lzdGVtL2Nsb2Nrc291cmNlL2Nsb2Nr c291cmNlMC9hdmFpbGFibGVfY2xvY2tzb3VyY2U8YnI+a3ZtLT0KY2xvY2sgaHBldCBhY3BpX3Bt PGJyPjxicj5BbnkgPQppZGVhcz88YnI+PGJyPk1hcmt1czxicj48YnI+PGJyPjwvZGl2PiZsdDtJ bnRlclNjYW5fRGlzY2xhaW1lci50eHQmZ3Q7X19fXz0KX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXzxicj5Vc2VycyBtYWlsaW5nIGxpc3Q8YnI+PGEgPQpocmVmPTNE Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+PGEgPQpocmVm PTNEImh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyI+aHR0cDov L2xpc3RzLm92aXJ0Lj0Kb3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+PC9zcGFuPjwvYmxv Y2txdW90ZT48L2Rpdj48YnI+PC9kaXY+PC9kaXY+X189Cl9fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fXzxicj5Vc2VycyBtYWlsaW5nID0KbGlzdDxicj48YSA9Cmhy ZWY9M0QibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyI+VXNlcnNAb3ZpcnQub3JnPC9hPjxicj5odHRw Oi8vbGlzdHMub3ZpcnQuPQpvcmcvbWFpbG1hbi9saXN0aW5mby91c2Vyczxicj48L2Jsb2NrcXVv dGU+PC9kaXY+PGJyPjwvZGl2PjwvYm9keT48L2h0bWw+PQoKLS1BcHBsZS1NYWlsPV8yRkMxMDQ1 Qi1CQ0E1LTQzQTctQjE4Ni05NjFDOURGRUQ1NjUtLQo= --===============8031873497385404125==-- From stockhausen at collogia.de Sat Apr 5 13:38:50 2014 Content-Type: multipart/mixed; boundary="===============7506407795493750969==" MIME-Version: 1.0 From: Markus Stockhausen To: users at ovirt.org Subject: Re: [Users] TSC clocksource gets lost after live migration Date: Sat, 05 Apr 2014 17:38:47 +0000 Message-ID: <12EF8D94C6F8734FB2FF37B9FBEDD173586165C3@EXCHANGE.collogia.de> In-Reply-To: 56307342-39C8-4D67-B92E-E68201B68F89@redhat.com --===============7506407795493750969== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. ------=3D_NextPartTM-000-65b5fc42-4c2e-4b1f-9849-7feb5e008797 Content-Type: text/plain; charset=3D"Windows-1252" Content-Transfer-Encoding: quoted-printable > Von: Michal Skrivanek [michal.skrivanek(a)redhat.com]=3D0A=3D > Gesendet: Freitag, 4. April 2014 15:33=3D0A=3D > An: Markus Stockhausen=3D0A=3D > Cc: ovirt-users=3D0A=3D > =3D0A=3D > Betreff: Re: [Users] TSC clocksource gets lost after live migration=3D0A= =3D > =3D0A=3D > Hi,=3D0A=3D > this is more for the KVM folks I suppose=3D85can you get the qemu process= c=3D mdline please?=3D0A=3D > =3D0A=3D > Thanks,=3D0A=3D > michal=3D0A=3D =3D0A=3D Hello,=3D0A=3D =3D0A=3D the bug exists even with two absolutely identical hosts. With others=3D0A= =3D reporting the same issue I opened BZ 1084700. QEMU command line=3D0A=3D posted over there.=3D0A=3D =3D0A=3D Markus=3D ------=3D_NextPartTM-000-65b5fc42-4c2e-4b1f-9849-7feb5e008797 Content-Type: text/plain; name=3D"InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=3D"InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmlich erhalten haben, informieren Sie bitte sofort den Absender u= nd vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. =C3=9Cber das Internet versandte E-Mails k=C3=B6nnen unter fremden Namen er= stellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserkl=C3=A4rung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln Vorstand: Kadir Akin Dr. Michael H=C3=B6hnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht K=C3=B6ln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln executive board: Kadir Akin Dr. Michael H=C3=B6hnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=3D_NextPartTM-000-65b5fc42-4c2e-4b1f-9849-7feb5e008797-- --===============7506407795493750969== Content-Type: multipart/mixed MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" ClRoaXMgaXMgYSBtdWx0aS1wYXJ0IG1lc3NhZ2UgaW4gTUlNRSBmb3JtYXQuCgotLS0tLS09X05l eHRQYXJ0VE0tMDAwLTY1YjVmYzQyLTRjMmUtNGIxZi05ODQ5LTdmZWI1ZTAwODc5NwpDb250ZW50 LVR5cGU6IHRleHQvcGxhaW47IGNoYXJzZXQ9IldpbmRvd3MtMTI1MiIKQ29udGVudC1UcmFuc2Zl ci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKPiBWb246IE1pY2hhbCBTa3JpdmFuZWsgW21p Y2hhbC5za3JpdmFuZWtAcmVkaGF0LmNvbV09MEE9Cj4gR2VzZW5kZXQ6IEZyZWl0YWcsIDQuIEFw cmlsIDIwMTQgMTU6MzM9MEE9Cj4gQW46IE1hcmt1cyBTdG9ja2hhdXNlbj0wQT0KPiBDYzogb3Zp cnQtdXNlcnM9MEE9Cj4gPTBBPQo+IEJldHJlZmY6IFJlOiBbVXNlcnNdIFRTQyBjbG9ja3NvdXJj ZSBnZXRzIGxvc3QgYWZ0ZXIgbGl2ZSBtaWdyYXRpb249MEE9Cj4gPTBBPQo+IEhpLD0wQT0KPiB0 aGlzIGlzIG1vcmUgZm9yIHRoZSBLVk0gZm9sa3MgSSBzdXBwb3NlPTg1Y2FuIHlvdSBnZXQgdGhl IHFlbXUgcHJvY2VzcyBjPQptZGxpbmUgcGxlYXNlPz0wQT0KPiA9MEE9Cj4gVGhhbmtzLD0wQT0K PiBtaWNoYWw9MEE9Cj0wQT0KSGVsbG8sPTBBPQo9MEE9CnRoZSBidWcgZXhpc3RzIGV2ZW4gd2l0 aCB0d28gYWJzb2x1dGVseSBpZGVudGljYWwgaG9zdHMuIFdpdGggb3RoZXJzPTBBPQpyZXBvcnRp bmcgdGhlIHNhbWUgaXNzdWUgSSBvcGVuZWQgQlogMTA4NDcwMC4gUUVNVSBjb21tYW5kIGxpbmU9 MEE9CnBvc3RlZCBvdmVyIHRoZXJlLj0wQT0KPTBBPQpNYXJrdXM9Ci0tLS0tLT1fTmV4dFBhcnRU TS0wMDAtNjViNWZjNDItNGMyZS00YjFmLTk4NDktN2ZlYjVlMDA4Nzk3CkNvbnRlbnQtVHlwZTog dGV4dC9wbGFpbjsKCW5hbWU9IkludGVyU2Nhbl9EaXNjbGFpbWVyLnR4dCIKQ29udGVudC1UcmFu c2Zlci1FbmNvZGluZzogN2JpdApDb250ZW50LURpc3Bvc2l0aW9uOiBhdHRhY2htZW50OwoJZmls ZW5hbWU9IkludGVyU2Nhbl9EaXNjbGFpbWVyLnR4dCIKCioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioKRGll c2UgRS1NYWlsIGVudGjDpGx0IHZlcnRyYXVsaWNoZSB1bmQvb2RlciByZWNodGxpY2ggZ2VzY2jD vHR6dGUKSW5mb3JtYXRpb25lbi4gV2VubiBTaWUgbmljaHQgZGVyIHJpY2h0aWdlIEFkcmVzc2F0 IHNpbmQgb2RlciBkaWVzZSBFLU1haWwKaXJydMO8bWxpY2ggZXJoYWx0ZW4gaGFiZW4sIGluZm9y bWllcmVuIFNpZSBiaXR0ZSBzb2ZvcnQgZGVuIEFic2VuZGVyIHVuZAp2ZXJuaWNodGVuIFNpZSBk aWVzZSBNYWlsLiBEYXMgdW5lcmxhdWJ0ZSBLb3BpZXJlbiBzb3dpZSBkaWUgdW5iZWZ1Z3RlCldl aXRlcmdhYmUgZGllc2VyIE1haWwgaXN0IG5pY2h0IGdlc3RhdHRldC4KCsOcYmVyIGRhcyBJbnRl cm5ldCB2ZXJzYW5kdGUgRS1NYWlscyBrw7ZubmVuIHVudGVyIGZyZW1kZW4gTmFtZW4gZXJzdGVs bHQgb2RlcgptYW5pcHVsaWVydCB3ZXJkZW4uIERlc2hhbGIgaXN0IGRpZXNlIGFscyBFLU1haWwg dmVyc2NoaWNrdGUgTmFjaHJpY2h0IGtlaW5lCnJlY2h0c3ZlcmJpbmRsaWNoZSBXaWxsZW5zZXJr bMOkcnVuZy4KCkNvbGxvZ2lhClVudGVybmVobWVuc2JlcmF0dW5nIEFHClViaWVycmluZyAxMQpE LTUwNjc4IEvDtmxuCgpWb3JzdGFuZDoKS2FkaXIgQWtpbgpEci4gTWljaGFlbCBIw7ZobmVyYmFj aAoKVm9yc2l0emVuZGVyIGRlcyBBdWZzaWNodHNyYXRlczoKSGFucyBLcmlzdGlhbiBMYW5ndmEK ClJlZ2lzdGVyZ2VyaWNodDogQW10c2dlcmljaHQgS8O2bG4KUmVnaXN0ZXJudW1tZXI6IEhSQiA1 MiA0OTcKClRoaXMgZS1tYWlsIG1heSBjb250YWluIGNvbmZpZGVudGlhbCBhbmQvb3IgcHJpdmls ZWdlZCBpbmZvcm1hdGlvbi4gSWYgeW91CmFyZSBub3QgdGhlIGludGVuZGVkIHJlY2lwaWVudCAo b3IgaGF2ZSByZWNlaXZlZCB0aGlzIGUtbWFpbCBpbiBlcnJvcikKcGxlYXNlIG5vdGlmeSB0aGUg c2VuZGVyIGltbWVkaWF0ZWx5IGFuZCBkZXN0cm95IHRoaXMgZS1tYWlsLiBBbnkKdW5hdXRob3Jp emVkIGNvcHlpbmcsIGRpc2Nsb3N1cmUgb3IgZGlzdHJpYnV0aW9uIG9mIHRoZSBtYXRlcmlhbCBp biB0aGlzCmUtbWFpbCBpcyBzdHJpY3RseSBmb3JiaWRkZW4uCgplLW1haWxzIHNlbnQgb3ZlciB0 aGUgaW50ZXJuZXQgbWF5IGhhdmUgYmVlbiB3cml0dGVuIHVuZGVyIGEgd3JvbmcgbmFtZSBvcgpi ZWVuIG1hbmlwdWxhdGVkLiBUaGF0IGlzIHdoeSB0aGlzIG1lc3NhZ2Ugc2VudCBhcyBhbiBlLW1h aWwgaXMgbm90IGEKbGVnYWxseSBiaW5kaW5nIGRlY2xhcmF0aW9uIG9mIGludGVudGlvbi4KCkNv bGxvZ2lhClVudGVybmVobWVuc2JlcmF0dW5nIEFHClViaWVycmluZyAxMQpELTUwNjc4IEvDtmxu CgpleGVjdXRpdmUgYm9hcmQ6CkthZGlyIEFraW4KRHIuIE1pY2hhZWwgSMO2aG5lcmJhY2gKClBy ZXNpZGVudCBvZiB0aGUgc3VwZXJ2aXNvcnkgYm9hcmQ6CkhhbnMgS3Jpc3RpYW4gTGFuZ3ZhCgpS ZWdpc3RyeSBvZmZpY2U6IGRpc3RyaWN0IGNvdXJ0IENvbG9nbmUKUmVnaXN0ZXIgbnVtYmVyOiBI UkIgNTIgNDk3CgoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqCgotLS0tLS09X05leHRQYXJ0VE0tMDAwLTY1 YjVmYzQyLTRjMmUtNGIxZi05ODQ5LTdmZWI1ZTAwODc5Ny0tCg== --===============7506407795493750969==-- From darrell.budic at zenfire.com Sat Apr 5 21:46:30 2014 Content-Type: multipart/mixed; boundary="===============0680058089245764235==" MIME-Version: 1.0 From: Darrell Budic To: users at ovirt.org Subject: Re: [Users] TSC clocksource gets lost after live migration Date: Sat, 05 Apr 2014 12:09:11 -0500 Message-ID: <26FC5FA9-AF55-43BB-B3EB-41FA872A983F@zenfire.com> In-Reply-To: C6B57894-BB8B-4D3C-AB8D-766BB03AA8F7@zenfire.com --===============0680058089245764235== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_426EFAFD-B3CC-41E6-85E6-EAE4A82FC441 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dwindows-1252 Whups, should note that I don=3D92t see the CPU utilization issues, just = =3D the clock source errors. This is on a 3.3.4 install with centos 6.4 =3D guests. On Apr 4, 2014, at 6:19 PM, Darrell Budic =3D wrote: > I see this on some guests as well, possibly relating to moving between = =3D hosts with the same family of CPU but different absolute CPU speeds? >=3D20 > -Darrell >=3D20 > On Apr 4, 2014, at 8:33 AM, Michal Skrivanek =3D wrote: >=3D20 >> Hi, >> this is more for the KVM folks I suppose=3D85can you get the qemu =3D process cmdline please? >>=3D20 >> Thanks, >> michal >>=3D20 >> On 3 Apr 2014, at 12:13, Markus Stockhausen wrote: >>=3D20 >>> Hello, >>>=3D20 >>> we have an up to date ovirt 3.4 installation. Inside we are running =3D SLES11 SP3 >>> VMs (Kernel 3.0.76-0.11). After live migration of these VMs they all = =3D of a sudden >>> do not react any longer and CPU usage of the VM goes to 100%. >>>=3D20 >>> We identified kvm-clock source to be the culprit and therefore =3D switched to another >>> clocksource. We ended with hpet but are not happy with that as our =3D inital goal >>> was to use the more simple designed TSC clocksoure.=3D20 >>>=3D20 >>> The reason behind that is the question I have for you experts. >>>=3D20 >>> Our hosts all have the constant_tsc CPU flag available. Just to =3D mention these >>> are not identical hosts. We have a mix of Xeon 5500 and 5600 =3D machines. E.G. >>> [root(a)colovn01 ~]# cat /proc/cpuinfo | grep constant_tsc | wc -l >>> 8 >>>=3D20 >>> When we start the VM the client sees TSC as available clocksource: >>>=3D20 >>> colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource >>> kvm-clock tsc hpet acpi_pm >>>=3D20 >>> After the first live migration to another host that also has =3D constant_tsc (see above) >>> that flag is lost inside the VM. >>>=3D20 >>> colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource >>> kvm-clock hpet acpi_pm >>>=3D20 >>> Any ideas? >>>=3D20 >>> Markus >>>=3D20 >>>=3D20 >>> =3D _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>=3D20 >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >=3D20 > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --Apple-Mail=3D_426EFAFD-B3CC-41E6-85E6-EAE4A82FC441 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dwindows-1252 Whups, = =3D should note that I don=3D92t see the CPU utilization issues, just the =3D clock source errors. This is on a 3.3.4 install with centos 6.4 =3D guests.

On Apr 4, 2014, at 6:19 PM, Darrell = =3D Budic <darrell.budic(a)zenfire.com>=3D ; wrote:

I see =3D this on some guests as well, possibly relating to moving between hosts =3D with the same family of CPU but different absolute CPU =3D speeds?

  =3D -Darrell


Hi,
this is more for the KVM folks I =3D suppose=3D85can you get the qemu process cmdline =3D please?

Thanks,
michal

On 3 Apr 2014, at 12:13, Markus Stockhausen wrote:

Hello,

we have an up to date ovirt 3.4 installation. =3D Inside we are running SLES11 SP3
VMs (Kernel 3.0.76-0.11). After live = =3D migration of these VMs they all of a sudden
do not react any longer =3D and CPU usage of the VM goes to 100%.

We identified kvm-clock =3D source to be the culprit and therefore switched to =3D another
clocksource. We ended with hpet but are not happy with that =3D as our inital goal
was to use the more simple designed TSC =3D clocksoure. 

The= =3D reason behind that is the question I have for you experts.

Our =3D hosts all have the constant_tsc CPU flag available. Just to mention =3D these
are not identical hosts. We have a mix of Xeon 5500 and 5600 =3D machines. E.G.
[root(a)colovn01 ~]# cat /proc/cpuinfo | grep =3D constant_tsc | wc -l
8

When we start the VM the client sees =3D TSC as available clocksource:

colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-= =3D clock tsc hpet acpi_pm

After the first live migration to another =3D host that also has constant_tsc (see above)
that flag is lost inside =3D the VM.

colvm53:~ # cat =3D /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-= =3D clock hpet acpi_pm

Any =3D ideas?

Markus


<InterScan_Disclaimer.txt>____= =3D ___________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt= .=3D org/mailman/listinfo/users

__= =3D _____________________________________________
Users mailing =3D list
Users(a)ovirt.org
http://lists.ovirt= .=3D org/mailman/listinfo/users

_____= =3D __________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.o= virt.=3D org/mailman/listinfo/users

= =3D --Apple-Mail=3D_426EFAFD-B3CC-41E6-85E6-EAE4A82FC441-- --===============0680058089245764235== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fNDI2RUZBRkQtQjNDQy00MUU2LTg1RTYtRUFFNEE4MkZDNDQxCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD13aW5kb3dzLTEyNTIKCldodXBzLCBzaG91bGQgbm90ZSB0aGF0IEkg ZG9uPTkydCBzZWUgdGhlIENQVSB1dGlsaXphdGlvbiBpc3N1ZXMsIGp1c3QgPQp0aGUgY2xvY2sg c291cmNlIGVycm9ycy4gVGhpcyBpcyBvbiBhIDMuMy40IGluc3RhbGwgd2l0aCBjZW50b3MgNi40 ID0KZ3Vlc3RzLgoKT24gQXByIDQsIDIwMTQsIGF0IDY6MTkgUE0sIERhcnJlbGwgQnVkaWMgPGRh cnJlbGwuYnVkaWNAemVuZmlyZS5jb20+ID0Kd3JvdGU6Cgo+IEkgc2VlIHRoaXMgb24gc29tZSBn dWVzdHMgYXMgd2VsbCwgcG9zc2libHkgcmVsYXRpbmcgdG8gbW92aW5nIGJldHdlZW4gPQpob3N0 cyB3aXRoIHRoZSBzYW1lIGZhbWlseSBvZiBDUFUgYnV0IGRpZmZlcmVudCBhYnNvbHV0ZSBDUFUg c3BlZWRzPwo+PTIwCj4gICAtRGFycmVsbAo+PTIwCj4gT24gQXByIDQsIDIwMTQsIGF0IDg6MzMg QU0sIE1pY2hhbCBTa3JpdmFuZWsgPQo8bWljaGFsLnNrcml2YW5la0ByZWRoYXQuY29tPiB3cm90 ZToKPj0yMAo+PiBIaSwKPj4gdGhpcyBpcyBtb3JlIGZvciB0aGUgS1ZNIGZvbGtzIEkgc3VwcG9z ZT04NWNhbiB5b3UgZ2V0IHRoZSBxZW11ID0KcHJvY2VzcyBjbWRsaW5lIHBsZWFzZT8KPj49MjAK Pj4gVGhhbmtzLAo+PiBtaWNoYWwKPj49MjAKPj4gT24gMyBBcHIgMjAxNCwgYXQgMTI6MTMsIE1h cmt1cyBTdG9ja2hhdXNlbiB3cm90ZToKPj49MjAKPj4+IEhlbGxvLAo+Pj49MjAKPj4+IHdlIGhh dmUgYW4gdXAgdG8gZGF0ZSBvdmlydCAzLjQgaW5zdGFsbGF0aW9uLiBJbnNpZGUgd2UgYXJlIHJ1 bm5pbmcgPQpTTEVTMTEgU1AzCj4+PiBWTXMgKEtlcm5lbCAzLjAuNzYtMC4xMSkuIEFmdGVyIGxp dmUgbWlncmF0aW9uIG9mIHRoZXNlIFZNcyB0aGV5IGFsbCA9Cm9mIGEgc3VkZGVuCj4+PiBkbyBu b3QgcmVhY3QgYW55IGxvbmdlciBhbmQgQ1BVIHVzYWdlIG9mIHRoZSBWTSBnb2VzIHRvIDEwMCUu Cj4+Pj0yMAo+Pj4gV2UgaWRlbnRpZmllZCBrdm0tY2xvY2sgc291cmNlIHRvIGJlIHRoZSBjdWxw cml0IGFuZCB0aGVyZWZvcmUgPQpzd2l0Y2hlZCB0byBhbm90aGVyCj4+PiBjbG9ja3NvdXJjZS4g V2UgZW5kZWQgd2l0aCBocGV0IGJ1dCBhcmUgbm90IGhhcHB5IHdpdGggdGhhdCBhcyBvdXIgPQpp bml0YWwgZ29hbAo+Pj4gd2FzIHRvIHVzZSB0aGUgbW9yZSBzaW1wbGUgZGVzaWduZWQgVFNDIGNs b2Nrc291cmUuPTIwCj4+Pj0yMAo+Pj4gVGhlIHJlYXNvbiBiZWhpbmQgdGhhdCBpcyB0aGUgcXVl c3Rpb24gSSBoYXZlIGZvciB5b3UgZXhwZXJ0cy4KPj4+PTIwCj4+PiBPdXIgaG9zdHMgYWxsIGhh dmUgdGhlIGNvbnN0YW50X3RzYyBDUFUgZmxhZyBhdmFpbGFibGUuIEp1c3QgdG8gPQptZW50aW9u IHRoZXNlCj4+PiBhcmUgbm90IGlkZW50aWNhbCBob3N0cy4gV2UgaGF2ZSBhIG1peCBvZiBYZW9u IDU1MDAgYW5kIDU2MDAgPQptYWNoaW5lcy4gRS5HLgo+Pj4gW3Jvb3RAY29sb3ZuMDEgfl0jIGNh dCAvcHJvYy9jcHVpbmZvIHwgZ3JlcCBjb25zdGFudF90c2MgfCB3YyAtbAo+Pj4gOAo+Pj49MjAK Pj4+IFdoZW4gd2Ugc3RhcnQgdGhlIFZNIHRoZSBjbGllbnQgc2VlcyBUU0MgYXMgYXZhaWxhYmxl IGNsb2Nrc291cmNlOgo+Pj49MjAKPj4+IGNvbHZtNTM6fiAjIGNhdCA9Ci9zeXMvZGV2aWNlcy9z eXN0ZW0vY2xvY2tzb3VyY2UvY2xvY2tzb3VyY2UwL2F2YWlsYWJsZV9jbG9ja3NvdXJjZQo+Pj4g a3ZtLWNsb2NrIHRzYyBocGV0IGFjcGlfcG0KPj4+PTIwCj4+PiBBZnRlciB0aGUgZmlyc3QgbGl2 ZSBtaWdyYXRpb24gdG8gYW5vdGhlciBob3N0IHRoYXQgYWxzbyBoYXMgPQpjb25zdGFudF90c2Mg KHNlZSBhYm92ZSkKPj4+IHRoYXQgZmxhZyBpcyBsb3N0IGluc2lkZSB0aGUgVk0uCj4+Pj0yMAo+ Pj4gY29sdm01Mzp+ICMgY2F0ID0KL3N5cy9kZXZpY2VzL3N5c3RlbS9jbG9ja3NvdXJjZS9jbG9j a3NvdXJjZTAvYXZhaWxhYmxlX2Nsb2Nrc291cmNlCj4+PiBrdm0tY2xvY2sgaHBldCBhY3BpX3Bt Cj4+Pj0yMAo+Pj4gQW55IGlkZWFzPwo+Pj49MjAKPj4+IE1hcmt1cwo+Pj49MjAKPj4+PTIwCj4+ PiA9CjxJbnRlclNjYW5fRGlzY2xhaW1lci50eHQ+X19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX18KPj4+IFVzZXJzIG1haWxpbmcgbGlzdAo+Pj4gVXNlcnNAb3Zp cnQub3JnCj4+PiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMK Pj49MjAKPj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K Pj4gVXNlcnMgbWFpbGluZyBsaXN0Cj4+IFVzZXJzQG92aXJ0Lm9yZwo+PiBodHRwOi8vbGlzdHMu b3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKPj0yMAo+IF9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4gVXNlcnMgbWFpbGluZyBsaXN0Cj4gVXNl cnNAb3ZpcnQub3JnCj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3Vz ZXJzCgoKLS1BcHBsZS1NYWlsPV80MjZFRkFGRC1CM0NDLTQxRTYtODVFNi1FQUU0QTgyRkM0NDEK Q29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQpDb250ZW50LVR5cGU6 IHRleHQvaHRtbDsKCWNoYXJzZXQ9d2luZG93cy0xMjUyCgo8aHRtbD48aGVhZD48bWV0YSBodHRw LWVxdWl2PTNEIkNvbnRlbnQtVHlwZSIgY29udGVudD0zRCJ0ZXh0L2h0bWwgPQpjaGFyc2V0PTNE d2luZG93cy0xMjUyIj48L2hlYWQ+PGJvZHkgc3R5bGU9M0Qid29yZC13cmFwOiBicmVhay13b3Jk OyA9Ci13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJyZWFrOiBhZnRlci13 aGl0ZS1zcGFjZTsiPldodXBzLCA9CnNob3VsZCBub3RlIHRoYXQgSSBkb249OTJ0IHNlZSB0aGUg Q1BVIHV0aWxpemF0aW9uIGlzc3VlcywganVzdCB0aGUgPQpjbG9jayBzb3VyY2UgZXJyb3JzLiBU aGlzIGlzIG9uIGEgMy4zLjQgaW5zdGFsbCB3aXRoIGNlbnRvcyA2LjQgPQpndWVzdHMuPGRpdj48 YnI+PC9kaXY+PGRpdj48ZGl2PjxkaXY+T24gQXByIDQsIDIwMTQsIGF0IDY6MTkgUE0sIERhcnJl bGwgPQpCdWRpYyAmbHQ7PGEgPQpocmVmPTNEIm1haWx0bzpkYXJyZWxsLmJ1ZGljQHplbmZpcmUu Y29tIj5kYXJyZWxsLmJ1ZGljQHplbmZpcmUuY29tPC9hPiZndD0KOyB3cm90ZTo8L2Rpdj48YnIg Y2xhc3M9M0QiQXBwbGUtaW50ZXJjaGFuZ2UtbmV3bGluZSI+PGJsb2NrcXVvdGUgPQp0eXBlPTNE ImNpdGUiPjxtZXRhIGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBlIiBjb250ZW50PTNEInRleHQv aHRtbCA9CmNoYXJzZXQ9M0R3aW5kb3dzLTEyNTIiPjxkaXYgc3R5bGU9M0Qid29yZC13cmFwOiBi cmVhay13b3JkOyA9Ci13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJyZWFr OiBhZnRlci13aGl0ZS1zcGFjZTsiPkkgc2VlID0KdGhpcyBvbiBzb21lIGd1ZXN0cyBhcyB3ZWxs LCBwb3NzaWJseSByZWxhdGluZyB0byBtb3ZpbmcgYmV0d2VlbiBob3N0cyA9CndpdGggdGhlIHNh bWUgZmFtaWx5IG9mIENQVSBidXQgZGlmZmVyZW50IGFic29sdXRlIENQVSA9CnNwZWVkcz88ZGl2 Pjxicj48L2Rpdj48ZGl2PiZuYnNwOyA9Ci1EYXJyZWxsPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRp dj48ZGl2PjxkaXY+T24gQXByIDQsIDIwMTQsIGF0IDg6MzMgQU0sID0KTWljaGFsIFNrcml2YW5l ayAmbHQ7PGEgPQpocmVmPTNEIm1haWx0bzptaWNoYWwuc2tyaXZhbmVrQHJlZGhhdC5jb20iPm1p Y2hhbC5za3JpdmFuZWtAcmVkaGF0LmNvbTwvYT0KPiZndDsgd3JvdGU6PC9kaXY+PGJyIGNsYXNz PTNEIkFwcGxlLWludGVyY2hhbmdlLW5ld2xpbmUiPjxibG9ja3F1b3RlID0KdHlwZT0zRCJjaXRl Ij48YmFzZSBocmVmPTNEIngtbXNnOi8vMTU3NS8iPjxkaXYgc3R5bGU9M0Qid29yZC13cmFwOiA9 CmJyZWFrLXdvcmQ7IC13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJyZWFr OiA9CmFmdGVyLXdoaXRlLXNwYWNlOyAiPkhpLDxkaXY+dGhpcyBpcyBtb3JlIGZvciB0aGUgS1ZN IGZvbGtzIEkgPQpzdXBwb3NlPTg1Y2FuIHlvdSBnZXQgdGhlIHFlbXUgcHJvY2VzcyBjbWRsaW5l ID0KcGxlYXNlPzwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+VGhhbmtzLDwvZGl2PjxkaXY+bWlj aGFsPC9kaXY+PGRpdj48YnI+PGQ9Cml2PjxkaXY+T24gMyBBcHIgMjAxNCwgYXQgMTI6MTMsIE1h cmt1cyBTdG9ja2hhdXNlbiB3cm90ZTo8L2Rpdj48YnIgPQpjbGFzcz0zRCJBcHBsZS1pbnRlcmNo YW5nZS1uZXdsaW5lIj48YmxvY2txdW90ZSB0eXBlPTNEImNpdGUiPjxzcGFuID0KY2xhc3M9M0Qi QXBwbGUtc3R5bGUtc3BhbiIgc3R5bGU9M0QiYm9yZGVyLWNvbGxhcHNlOiBzZXBhcmF0ZTsgPQpm b250LWZhbWlseTogSGVsdmV0aWNhOyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudDog bm9ybWFsOyA9CmZvbnQtd2VpZ2h0OiBub3JtYWw7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IGxp bmUtaGVpZ2h0OiBub3JtYWw7ID0Kb3JwaGFuczogMjsgdGV4dC1hbGlnbjogLXdlYmtpdC1hdXRv OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogPQpub25lOyB3aGl0ZS1zcGFjZTog bm9ybWFsOyB3aWRvd3M6IDI7IHdvcmQtc3BhY2luZzogMHB4OyBib3JkZXItc3BhY2luZzogPQow cHg7IC13ZWJraXQtdGV4dC1kZWNvcmF0aW9ucy1pbi1lZmZlY3Q6IG5vbmU7ID0KLXdlYmtpdC10 ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBmb250LXNpemU6IGluaGVyaXQ7Ij48ZGl2ID0Kc3R5bGU9 M0QiZGlyZWN0aW9uOiBsdHI7IGZvbnQtZmFtaWx5OiBUYWhvbWE7IGZvbnQtc2l6ZTogPQoxMHB0 OyI+SGVsbG8sPGJyPjxicj53ZSBoYXZlIGFuIHVwIHRvIGRhdGUgb3ZpcnQgMy40IGluc3RhbGxh dGlvbi4gPQpJbnNpZGUgd2UgYXJlIHJ1bm5pbmcgU0xFUzExIFNQMzxicj5WTXMgKEtlcm5lbCAz LjAuNzYtMC4xMSkuIEFmdGVyIGxpdmUgPQptaWdyYXRpb24gb2YgdGhlc2UgVk1zIHRoZXkgYWxs IG9mIGEgc3VkZGVuPGJyPmRvIG5vdCByZWFjdCBhbnkgbG9uZ2VyID0KYW5kIENQVSB1c2FnZSBv ZiB0aGUgVk0gZ29lcyB0byAxMDAlLjxicj48YnI+V2UgaWRlbnRpZmllZCBrdm0tY2xvY2sgPQpz b3VyY2UgdG8gYmUgdGhlIGN1bHByaXQgYW5kIHRoZXJlZm9yZSBzd2l0Y2hlZCB0byA9CmFub3Ro ZXI8YnI+Y2xvY2tzb3VyY2UuIFdlIGVuZGVkIHdpdGggaHBldCBidXQgYXJlIG5vdCBoYXBweSB3 aXRoIHRoYXQgPQphcyBvdXIgaW5pdGFsIGdvYWw8YnI+d2FzIHRvIHVzZSB0aGUgbW9yZSBzaW1w bGUgZGVzaWduZWQgVFNDID0KY2xvY2tzb3VyZS48c3BhbiBjbGFzcz0zRCJBcHBsZS1jb252ZXJ0 ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnI+PGJyPlRoZSA9CnJlYXNvbiBiZWhpbmQgdGhhdCBp cyB0aGUgcXVlc3Rpb24gSSBoYXZlIGZvciB5b3UgZXhwZXJ0cy48YnI+PGJyPk91ciA9Cmhvc3Rz IGFsbCBoYXZlIHRoZSBjb25zdGFudF90c2MgQ1BVIGZsYWcgYXZhaWxhYmxlLiBKdXN0IHRvIG1l bnRpb24gPQp0aGVzZTxicj5hcmUgbm90IGlkZW50aWNhbCBob3N0cy4gV2UgaGF2ZSBhIG1peCBv ZiBYZW9uIDU1MDAgYW5kIDU2MDAgPQptYWNoaW5lcy4gRS5HLjxicj5bcm9vdEBjb2xvdm4wMSB+ XSMgY2F0IC9wcm9jL2NwdWluZm8gfCBncmVwID0KY29uc3RhbnRfdHNjIHwgd2MgLWw8YnI+ODxi cj48YnI+V2hlbiB3ZSBzdGFydCB0aGUgVk0gdGhlIGNsaWVudCBzZWVzID0KVFNDIGFzIGF2YWls YWJsZSBjbG9ja3NvdXJjZTo8YnI+PGJyPmNvbHZtNTM6fiAjIGNhdCA9Ci9zeXMvZGV2aWNlcy9z eXN0ZW0vY2xvY2tzb3VyY2UvY2xvY2tzb3VyY2UwL2F2YWlsYWJsZV9jbG9ja3NvdXJjZTxicj5r dm0tPQpjbG9jayB0c2MgaHBldCBhY3BpX3BtPGJyPjxicj5BZnRlciB0aGUgZmlyc3QgbGl2ZSBt aWdyYXRpb24gdG8gYW5vdGhlciA9Cmhvc3QgdGhhdCBhbHNvIGhhcyBjb25zdGFudF90c2MgKHNl ZSBhYm92ZSk8YnI+dGhhdCBmbGFnIGlzIGxvc3QgaW5zaWRlID0KdGhlIFZNLjxicj48YnI+Y29s dm01Mzp+ICMgY2F0ID0KL3N5cy9kZXZpY2VzL3N5c3RlbS9jbG9ja3NvdXJjZS9jbG9ja3NvdXJj ZTAvYXZhaWxhYmxlX2Nsb2Nrc291cmNlPGJyPmt2bS09CmNsb2NrIGhwZXQgYWNwaV9wbTxicj48 YnI+QW55ID0KaWRlYXM/PGJyPjxicj5NYXJrdXM8YnI+PGJyPjxicj48L2Rpdj4mbHQ7SW50ZXJT Y2FuX0Rpc2NsYWltZXIudHh0Jmd0O19fX189Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX188YnI+VXNlcnMgbWFpbGluZyBsaXN0PGJyPjxhID0KaHJlZj0zRCJtYWls dG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPjxhID0KaHJlZj0zRCJo dHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiPmh0dHA6Ly9saXN0 cy5vdmlydC49Cm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjwvc3Bhbj48L2Jsb2NrcXVv dGU+PC9kaXY+PGJyPjwvZGl2PjwvZGl2Pl9fPQpfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX188YnI+VXNlcnMgbWFpbGluZyA9Cmxpc3Q8YnI+PGEgaHJlZj0zRCJt YWlsdG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPjxhID0KaHJlZj0z RCJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiPmh0dHA6Ly9s aXN0cy5vdmlydC49Cm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj48L2Jsb2NrcXVv dGU+PC9kaXY+PGJyPjwvZGl2PjwvZGl2Pl9fX19fPQpfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX188YnI+VXNlcnMgbWFpbGluZyBsaXN0PGJyPjxhID0KaHJlZj0zRCJt YWlsdG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPmh0dHA6Ly9saXN0 cy5vdmlydC49Cm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPGJyPjwvYmxvY2txdW90ZT48L2Rp dj48YnI+PC9kaXY+PC9ib2R5PjwvaHRtbD49CgotLUFwcGxlLU1haWw9XzQyNkVGQUZELUIzQ0Mt NDFFNi04NUU2LUVBRTRBODJGQzQ0MS0tCg== --===============0680058089245764235==--