<div dir="ltr">Thank you very much for your reply.<div><br></div><div style>I see that the problem appears in vdsm.log.44.xz but doesn't appear in vdsm.log.45.xz</div><div style><br></div><div style><div><i><b>[wil@bufferoverflow vdsm]$</b> xzcat vdsm.log.45.xz | grep StoragePoolMasterNotFound | wc -l</i></div>
<div><i>0</i></div><div><i><b>[wil@bufferoverflow vdsm]$ </b>xzcat vdsm.log.44.xz | grep StoragePoolMasterNotFound | wc -l</i></div><div><i>52</i></div></div><div style><br></div><div style>so I hope the source of the problem is in one of them (attached).</div>
<div style><div><br></div><div><div><i><b>[wil@bufferoverflow vdsm]$</b> ls -la vdsm.log.44.xz</i></div><div><i>-rw-r--r-- 1 vdsm kvm 763808 Mar 24 20:00 vdsm.log.44.xz</i></div><div><i><b>[wil@bufferoverflow vdsm]$</b> ls -la vdsm.log.45.xz</i></div>
<div><i>-rw-r--r-- 1 vdsm kvm 706212 Mar 22 11:00 vdsm.log.45.xz</i></div></div><div><br></div><div style>Unfortunately, I do not have any engine logs from that time (between Mar 22 11:00 and Mar 24 20:00)</div><div style>
<br></div><div style><div><i><b>[wil@bufferoverflow ovirt-engine]$</b> ls -la</i></div><div><i>total 148720</i></div><div><i>drwxrwxr-x 2 wil wil 4096 Apr 17 09:07 .</i></div><div><i>drwxrwxr-x 3 wil wil 4096 Mar 26 20:13 ..</i></div>
<div><i>-rw-rw-r-- 1 wil wil 304 Apr 17 16:31 boot.log</i></div><div><i>-rw-rw---- 1 wil wil 510 Apr 17 16:31 console.log</i></div><div><i>-rw-rw-r-- 1 wil wil 7398188 Apr 17 21:35 engine.log</i></div><div><i>-rw-rw-r-- 1 wil wil 10485813 Apr 13 09:20 engine.log.1</i></div>
<div><i>-rw-rw-r-- 1 wil wil 10485766 Apr 11 13:19 engine.log.2</i></div><div><i>-rw-rw-r-- 1 wil wil 10486016 Apr 11 08:14 engine.log.3</i></div><div><i>-rw-rw-r-- 1 wil wil 10485972 Apr 11 03:06 engine.log.4</i></div><div>
<i>-rw-rw-r-- 1 wil wil 10486208 Apr 10 22:01 engine.log.5</i></div><div><i>-rw-rw-r-- 1 wil wil 8439424 Apr 17 16:31 server.log</i></div><div><i>-rw-rw-r-- 1 wil wil 10485867 Apr 17 09:07 server.log.1</i></div><div><i>-rw-rw-r-- 1 wil wil 10485943 Apr 17 02:40 server.log.2</i></div>
<div><i>-rw-rw-r-- 1 wil wil 10485867 Apr 16 20:15 server.log.3</i></div><div><i>-rw-rw-r-- 1 wil wil 10485943 Apr 16 13:54 server.log.4</i></div><div><i>-rw-rw-r-- 1 wil wil 10485867 Apr 16 07:32 server.log.5</i></div><div>
<i>-rw-rw-r-- 1 wil wil 10485943 Apr 16 01:05 server.log.6</i></div><div><i>-rw-rw-r-- 1 wil wil 10485867 Apr 15 18:46 server.log.7</i></div><div><i>-rw-rw-r-- 1 wil wil 10485781 Apr 15 12:28 server.log.8</i></div><div><i><b>[wil@bufferoverflow ovirt-engine]$</b> pwd</i></div>
<div><i>/home/wil/ovirt-engine/installation/var/log/ovirt-engine</i></div><div><br></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 17, 2013 at 6:54 PM, Yeela Kaplan <span dir="ltr"><<a href="mailto:ykaplan@redhat.com" target="_blank">ykaplan@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It looks like the link to the master domain is not in the tree.<br>
I need to see the full logs and understand what happened. Including the engine log.<br>
Are you sure you don't have them? even if they were rotated they should be kept as a vdsm.log.*.xz under /var/log/vdsm/<br>
<div class="HOEnZb"><div class="h5"><br>
----- Original Message -----<br>
> From: "Yuval M" <<a href="mailto:yuvalme@gmail.com">yuvalme@gmail.com</a>><br>
> To: "Yeela Kaplan" <<a href="mailto:ykaplan@redhat.com">ykaplan@redhat.com</a>><br>
> Cc: "Limor Gavish" <<a href="mailto:lgavish@gmail.com">lgavish@gmail.com</a>>, <a href="mailto:users@ovirt.org">users@ovirt.org</a>, "Nezer Zaidenberg" <<a href="mailto:nzaidenberg@mac.com">nzaidenberg@mac.com</a>><br>
> Sent: Wednesday, April 17, <a href="tel:2013" value="+9722013">2013</a> 4:56:55 PM<br>
> Subject: Re: [Users] oVirt storage is down and doesn't come up<br>
><br>
> 1. we do not have the logs from before the problem.<br>
> 2.<br>
> --------<br>
> $ tree /rhev/data-center/<br>
> /rhev/data-center/<br>
> âââ hsm-tasks<br>
> âââ mnt<br>
> âââ bufferoverflow.home:_home_BO__ISO__Domain<br>
> â  âââ 45d24e2a-705e-440f-954c-fda3cab61298<br>
> â  â  âââ dom_md<br>
> â  â  â  âââ ids<br>
> â  â  â  âââ inbox<br>
> â  â  â  âââ leases<br>
> â  â  â  âââ metadata<br>
> â  â  â  âââ outbox<br>
> â  â  âââ images<br>
> â  â  âââ 11111111-1111-1111-1111-111111111111<br>
> â  â  âââ Fedora-18-x86_64-DVD.iso<br>
> â  â  âââ Fedora-18-x86_64-Live-Desktop.iso<br>
> â  âââ __DIRECT_IO_TEST__<br>
> âââ bufferoverflow.home:_home_BO__Ovirt__Storage<br>
> âââ kernelpanic.home:_home_KP__Data__Domain<br>
> âââ a8286508-db45-40d7-8645-e573f6bacdc7<br>
> â  âââ dom_md<br>
> â  â  âââ ids<br>
> â  â  âââ inbox<br>
> â  â  âââ leases<br>
> â  â  âââ metadata<br>
> â  â  âââ outbox<br>
> â  âââ images<br>
> â  âââ 0df45336-de35-4dc0-9958-95b27d5d4701<br>
> â  â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d<br>
> â  â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d.lease<br>
> â  â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d.meta<br>
> â  â  âââ b245184f-f8e3-479b-8559-8b6af2473b7c<br>
> â  â  âââ b245184f-f8e3-479b-8559-8b6af2473b7c.lease<br>
> â  â  âââ b245184f-f8e3-479b-8559-8b6af2473b7c.meta<br>
> â  âââ 0e1ebaf7-3909-44cd-8560-d05a63eb4c4e<br>
> â  â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d<br>
> â  â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d.lease<br>
> â  â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d.meta<br>
> â  â  âââ 562b9043-bde8-4595-bbea-fa8871f0e19e<br>
> â  â  âââ 562b9043-bde8-4595-bbea-fa8871f0e19e.lease<br>
> â  â  âââ 562b9043-bde8-4595-bbea-fa8871f0e19e.meta<br>
> â  âââ 32ebb85a-0dde-47fe-90c7-7f4fb2c0f1e5<br>
> â  â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d<br>
> â  â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d.lease<br>
> â  â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d.meta<br>
> â  â  âââ 4774095e-db3d-4561-8284-53eabfd28f66<br>
> â  â  âââ 4774095e-db3d-4561-8284-53eabfd28f66.lease<br>
> â  â  âââ 4774095e-db3d-4561-8284-53eabfd28f66.meta<br>
> â  âââ a7e13a25-1694-4509-9e6b-e88583a4d970<br>
> â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d<br>
> â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d.lease<br>
> â  âââ 0d33efc8-a608-439f-abe2-43884c1ce72d.meta<br>
> âââ __DIRECT_IO_TEST__<br>
><br>
> 16 directories, 35 files<br>
><br>
> --------------------<br>
> 3. We have 3 domains:<br>
> BO_Ovirt_Storage (data domain, on the same machine as engine and vdsm, via<br>
> NFS)<br>
> BO_ISO_Domain (ISO domain, same machine via NFS)<br>
> KP_Data_Domain (data domain on an NFS mount on a different machine)<br>
><br>
> Yuval<br>
><br>
><br>
><br>
> On Wed, Apr 17, 2013 at 4:28 PM, Yeela Kaplan <<a href="mailto:ykaplan@redhat.com">ykaplan@redhat.com</a>> wrote:<br>
><br>
> > Hi Limor,<br>
> > 1) Your log starts exactly after the vdsm restart. I need to see the full<br>
> > vdsm log from before the domains went down in order to understand the<br>
> > problem. Can you attach them?<br>
> > 2) can you send the printout of 'tree /rhev/data-center/'<br>
> > 3) how many domains are attached to your DC, and what type are they(ISO,<br>
> > export,data) and (The DC is nfs right)?<br>
> ><br>
> > Thanks,<br>
> > Yeela<br>
> ><br>
> > ----- Original Message -----<br>
> > > From: "Limor Gavish" <<a href="mailto:lgavish@gmail.com">lgavish@gmail.com</a>><br>
> > > To: "Tal Nisan" <<a href="mailto:tnisan@redhat.com">tnisan@redhat.com</a>><br>
> > > Cc: "Yuval M" <<a href="mailto:yuvalme@gmail.com">yuvalme@gmail.com</a>>, <a href="mailto:users@ovirt.org">users@ovirt.org</a>, "Nezer Zaidenberg" <<br>
> > <a href="mailto:nzaidenberg@mac.com">nzaidenberg@mac.com</a>><br>
> > > Sent: Monday, April 15, <a href="tel:2013" value="+9722013">2013</a> 5:10:16 PM<br>
> > > Subject: Re: [Users] oVirt storage is down and doesn't come up<br>
> > ><br>
> > > Thank you very much for your reply.<br>
> > > I ran the commands you asked (see below) but a directory named as the<br>
> > uuid of<br>
> > > the master domain is not mounted. We tried to restart the VDSM and the<br>
> > > entire machine it didn't help.<br>
> > > We succeeded to manually mount " /home/BO_Ovirt_Storage" to a temporary<br>
> > > directory.<br>
> > ><br>
> > > postgres=# \connect engine;<br>
> > > You are now connected to database "engine" as user "postgres".<br>
> > > engine=# select current_database();<br>
> > > current_database<br>
> > > ------------------<br>
> > > engine<br>
> > > (1 row)<br>
> > > engine=# select <a href="http://sds.id" target="_blank">sds.id</a> , ssc.connection from storage_domain_static sds<br>
> > join<br>
> > > storage_server_connections ssc on sds.storage= <a href="http://ssc.id" target="_blank">ssc.id</a> where <a href="http://sds.id" target="_blank">sds.id</a><br>
> > > ='1083422e-a5db-41b6-b667-b9ef1ef244f0';<br>
> > > id | connection<br>
> > ><br>
> > --------------------------------------+--------------------------------------------<br>
> > > 1083422e-a5db-41b6-b667-b9ef1ef244f0 |<br>
> > > bufferoverflow.home:/home/BO_Ovirt_Storage<br>
> > > (1 row)<br>
> > ><br>
> > > [wil@bufferoverflow ~] $ mount<br>
> > > proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)<br>
> > > sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)<br>
> > > devtmpfs on /dev type devtmpfs<br>
> > > (rw,nosuid,size=8131256k,nr_inodes=2032814,mode=755)<br>
> > > securityfs on /sys/kernel/security type securityfs<br>
> > > (rw,nosuid,nodev,noexec,relatime)<br>
> > > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)<br>
> > > devpts on /dev/pts type devpts<br>
> > > (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)<br>
> > > tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)<br>
> > > tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)<br>
> > > cgroup on /sys/fs/cgroup/systemd type cgroup<br>
> > ><br>
> > (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)<br>
> > > cgroup on /sys/fs/cgroup/cpuset type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,cpuset)<br>
> > > cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)<br>
> > > cgroup on /sys/fs/cgroup/memory type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,memory)<br>
> > > cgroup on /sys/fs/cgroup/devices type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,devices)<br>
> > > cgroup on /sys/fs/cgroup/freezer type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,freezer)<br>
> > > cgroup on /sys/fs/cgroup/net_cls type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,net_cls)<br>
> > > cgroup on /sys/fs/cgroup/blkio type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,blkio)<br>
> > > cgroup on /sys/fs/cgroup/perf_event type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,perf_event)<br>
> > > /dev/sda3 on / type ext4 (rw,relatime,data=ordered)<br>
> > > rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)<br>
> > > debugfs on /sys/kernel/debug type debugfs (rw,relatime)<br>
> > > sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)<br>
> > > hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)<br>
> > > systemd-1 on /proc/sys/fs/binfmt_misc type autofs<br>
> > > (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)<br>
> > > mqueue on /dev/mqueue type mqueue (rw,relatime)<br>
> > > tmpfs on /tmp type tmpfs (rw)<br>
> > > configfs on /sys/kernel/config type configfs (rw,relatime)<br>
> > > binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)<br>
> > > /dev/sda5 on /home type ext4 (rw,relatime,data=ordered)<br>
> > > /dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)<br>
> > > kernelpanic.home:/home/KP_Data_Domain on<br>
> > > /rhev/data-center/mnt/kernelpanic.home:_home_KP__Data__Domain type nfs<br>
> > ><br>
> > (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.100.101.100,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.100.101.100)<br>
> > > bufferoverflow.home:/home/BO_ISO_Domain on<br>
> > > /rhev/data-center/mnt/bufferoverflow.home:_home_BO__ISO__Domain type nfs<br>
> > ><br>
> > (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.100.101.108,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.100.101.108)<br>
> > ><br>
> > > [wil@bufferoverflow ~]$ ls -la /home/<br>
> > > total 36<br>
> > > drwxr-xr-x. 6 root root <a href="tel:4096" value="+9724096">4096</a> Mar <a href="tel:22%2011" value="+9722211">22 11</a>:25 .<br>
> > > dr-xr-xr-x. 19 root root <a href="tel:4096" value="+9724096">4096</a> Apr 12 18:53 ..<br>
> > > drwxr-xr-x. 3 vdsm kvm <a href="tel:4096" value="+9724096">4096</a> Mar <a href="tel:27%2017" value="+9722717">27 17</a>:33 BO_ISO_Domain<br>
> > > drwxr-xr-x. 3 vdsm kvm 4096 Mar 27 17:33 BO_Ovirt_Storage<br>
> > > drwx------. 2 root root 16384 Mar 6 09:11 lost+found<br>
> > > drwx------. 27 wil wil 4096 Apr 15 01:50 wil<br>
> > > [wil@bufferoverflow ~]$ cd /home/BO_Ovirt_Storage/<br>
> > > [wil@bufferoverflow BO_Ovirt_Storage]$ ls -la<br>
> > > total 12<br>
> > > drwxr-xr-x. 3 vdsm kvm <a href="tel:4096" value="+9724096">4096</a> Mar <a href="tel:27%2017" value="+9722717">27 17</a>:33 .<br>
> > > drwxr-xr-x. 6 root root <a href="tel:4096" value="+9724096">4096</a> Mar <a href="tel:22%2011" value="+9722211">22 11</a>:25 ..<br>
> > > drwxr-xr-x 5 vdsm kvm <a href="tel:4096" value="+9724096">4096</a> Mar 20 23:06<br>
> > 1083422e-a5db-41b6-b667-b9ef1ef244f0<br>
> > > -rwxr-xr-x 1 vdsm kvm 0 Mar 27 17:33 __DIRECT_IO_TEST__<br>
> > ><br>
> > > Thanks,<br>
> > > Limor<br>
> > ><br>
> > ><br>
> > > On Mon, Apr 15, 2013 at 4:02 PM, Tal Nisan < <a href="mailto:tnisan@redhat.com">tnisan@redhat.com</a> > wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > Hi Limor,<br>
> > > First we should probably start with checking which mount is the master<br>
> > > storage domain that appears as not found, this should be checked against<br>
> > the<br>
> > > oVirt server database, please run<br>
> > ><br>
> > > select <a href="http://sds.id" target="_blank">sds.id</a> , ssc.connection from storage_domain_static sds join<br>
> > > storage_server_connections ssc on sds.storage= <a href="http://ssc.id" target="_blank">ssc.id</a><br>
> > > where <a href="http://sds.id" target="_blank">sds.id</a> ='1083422e-a5db-41b6-b667-b9ef1ef244f0';<br>
> > ><br>
> > > You can run this via psql or a Postgres ui if you have one.<br>
> > > In the results you will see the storage connection in the format of<br>
> > > %hostname%:/%mountName%, then in the VDSM server check in the mount list<br>
> > > that you see that it is mounted, the mount itself should contain a<br>
> > directory<br>
> > > named as the uuid of the master domain, let me know the result.<br>
> > ><br>
> > > Tal.<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On 04/12/2013 07:29 PM, Limor Gavish wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > For some reason, without doing anything, all the storage domains became<br>
> > down<br>
> > > and restarting VDSM or the entire machine do not bring it up.<br>
> > > I am not using lvm<br>
> > > The following errors appear several times in vdsm.log (full logs are<br>
> > > attached):<br>
> > ><br>
> > > Thread-22::WARNING::2013-04-12<br>
> > > 19:00:08,597::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['<br>
> > > Volume group "1083422e-a5db-41b6-b667-b9ef1ef244f0" not found']<br>
> > > Thread-22::DEBUG::2013-04-12<br>
> > > 19:00:08,598::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm<br>
> > reload<br>
> > > operation' released the operation mutex<br>
> > > Thread-22::DEBUG::2013-04-12<br>
> > > 19:00:08,681::resourceManager::615::ResourceManager::(releaseResource)<br>
> > > Trying to release resource 'Storage.5849b030-626e-47cb-ad90-3ce782d831b3'<br>
> > > Thread-22::DEBUG::2013-04-12<br>
> > > 19:00:08,681::resourceManager::634::ResourceManager::(releaseResource)<br>
> > > Released resource 'Storage.5849b030-626e-47cb-ad90-3ce782d831b3' (0<br>
> > active<br>
> > > users)<br>
> > > Thread-22::DEBUG::2013-04-12<br>
> > > 19:00:08,681::resourceManager::640::ResourceManager::(releaseResource)<br>
> > > Resource 'Storage.5849b030-626e-47cb-ad90-3ce782d831b3' is free, finding<br>
> > out<br>
> > > if anyone is waiting for it.<br>
> > > Thread-22::DEBUG::2013-04-12<br>
> > > 19:00:08,682::resourceManager::648::ResourceManager::(releaseResource) No<br>
> > > one is waiting for resource<br>
> > 'Storage.5849b030-626e-47cb-ad90-3ce782d831b3',<br>
> > > Clearing records.<br>
> > > Thread-22::ERROR::2013-04-12<br>
> > > 19:00:08,682::task::850::TaskManager.Task::(_setError)<br>
> > > Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::Unexpected error<br>
> > > Traceback (most recent call last):<br>
> > > File "/usr/share/vdsm/storage/task.py", line 857, in _run<br>
> > > return fn(*args, **kargs)<br>
> > > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper<br>
> > > res = f(*args, **kwargs)<br>
> > > File "/usr/share/vdsm/storage/hsm.py", line 939, in connectStoragePool<br>
> > > masterVersion, options)<br>
> > > File "/usr/share/vdsm/storage/hsm.py", line 986, in _connectStoragePool<br>
> > > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)<br>
> > > File "/usr/share/vdsm/storage/sp.py", line 695, in connect<br>
> > > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)<br>
> > > File "/usr/share/vdsm/storage/sp.py", line 1232, in __rebuild<br>
> > > masterVersion=masterVersion)<br>
> > > File "/usr/share/vdsm/storage/sp.py", line 1576, in getMasterDomain<br>
> > > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)<br>
> > > StoragePoolMasterNotFound: Cannot find master domain:<br>
> > > 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,<br>
> > > msdUUID=1083422e-a5db-41b6-b667-b9ef1ef244f0'<br>
> > > Thread-22::DEBUG::2013-04-12<br>
> > > 19:00:08,685::task::869::TaskManager.Task::(_run)<br>
> > > Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::Task._run:<br>
> > > e35a22ac-771a-4916-851f-2fe9d60a0ae6<br>
> > > ('5849b030-626e-47cb-ad90-3ce782d831b3', 1,<br>
> > > '5849b030-626e-47cb-ad90-3ce782d831b3',<br>
> > > '1083422e-a5db-41b6-b667-b9ef1ef244f0', 3942) {} failed - stopping task<br>
> > > Thread-22::DEBUG::2013-04-12<br>
> > > 19:00:08,685::task::1194::TaskManager.Task::(stop)<br>
> > > Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::stopping in state preparing<br>
> > > (force False)<br>
> > > Thread-22::DEBUG::2013-04-12<br>
> > > 19:00:08,685::task::974::TaskManager.Task::(_decref)<br>
> > > Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::ref 1 aborting True<br>
> > > Thread-22::INFO::2013-04-12<br>
> > > 19:00:08,686::task::1151::TaskManager.Task::(prepare)<br>
> > > Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::aborting: Task is aborted:<br>
> > > 'Cannot find master domain' - code 304<br>
> > ><br>
> > > [wil@bufferoverflow ~]$ sudo vgs --noheadings --units b --nosuffix<br>
> > > --separator \| -o<br>
> > ><br>
> > uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free<br>
> > > No volume groups found<br>
> > ><br>
> > > [wil@bufferoverflow ~]$ mount<br>
> > > proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)<br>
> > > sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)<br>
> > > devtmpfs on /dev type devtmpfs<br>
> > > (rw,nosuid,size=8131256k,nr_inodes=2032814,mode=755)<br>
> > > securityfs on /sys/kernel/security type securityfs<br>
> > > (rw,nosuid,nodev,noexec,relatime)<br>
> > > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)<br>
> > > devpts on /dev/pts type devpts<br>
> > > (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)<br>
> > > tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)<br>
> > > tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)<br>
> > > cgroup on /sys/fs/cgroup/systemd type cgroup<br>
> > ><br>
> > (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)<br>
> > > cgroup on /sys/fs/cgroup/cpuset type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,cpuset)<br>
> > > cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)<br>
> > > cgroup on /sys/fs/cgroup/memory type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,memory)<br>
> > > cgroup on /sys/fs/cgroup/devices type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,devices)<br>
> > > cgroup on /sys/fs/cgroup/freezer type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,freezer)<br>
> > > cgroup on /sys/fs/cgroup/net_cls type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,net_cls)<br>
> > > cgroup on /sys/fs/cgroup/blkio type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,blkio)<br>
> > > cgroup on /sys/fs/cgroup/perf_event type cgroup<br>
> > > (rw,nosuid,nodev,noexec,relatime,perf_event)<br>
> > > /dev/sda3 on / type ext4 (rw,relatime,data=ordered)<br>
> > > rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)<br>
> > > debugfs on /sys/kernel/debug type debugfs (rw,relatime)<br>
> > > sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)<br>
> > > hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)<br>
> > > systemd-1 on /proc/sys/fs/binfmt_misc type autofs<br>
> > > (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)<br>
> > > mqueue on /dev/mqueue type mqueue (rw,relatime)<br>
> > > tmpfs on /tmp type tmpfs (rw)<br>
> > > configfs on /sys/kernel/config type configfs (rw,relatime)<br>
> > > binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)<br>
> > > /dev/sda5 on /home type ext4 (rw,relatime,data=ordered)<br>
> > > /dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)<br>
> > > kernelpanic.home:/home/KP_Data_Domain on<br>
> > > /rhev/data-center/mnt/kernelpanic.home:_home_KP__Data__Domain type nfs<br>
> > ><br>
> > (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.100.101.100,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.100.101.100)<br>
> > > bufferoverflow.home:/home/BO_ISO_Domain on<br>
> > > /rhev/data-center/mnt/bufferoverflow.home:_home_BO__ISO__Domain type nfs<br>
> > ><br>
> > (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.100.101.108,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.100.101.108)<br>
> > ><br>
> > > [wil@bufferoverflow ~]$ sudo find / -name<br>
> > > 5849b030-626e-47cb-ad90-3ce782d831b3<br>
> > > /run/vdsm/pools/5849b030-626e-47cb-ad90-3ce782d831b3<br>
> > ><br>
> > > [wil@bufferoverflow ~]$ sudo find / -name<br>
> > > 1083422e-a5db-41b6-b667-b9ef1ef244f0<br>
> > > /home/BO_Ovirt_Storage/1083422e-a5db-41b6-b667-b9ef1ef244f0<br>
> > ><br>
> > > I will extremely appreciate any help,<br>
> > > Limor Gavish<br>
> > > _______________________________________________<br>
> > > Users mailing list <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> > > <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > Users mailing list<br>
> > > <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> > > <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
> > ><br>
> ><br>
><br>
</div></div></blockquote></div><br></div>