<div dir="ltr"><div><div style>Thank you very much for your reply.</div><div style>I ran the commands you asked (see below) but <span style="font-family:arial,sans-serif;font-size:13px">a directory named as the uuid of the master domain is not mounted. We tried to restart the VDSM and the entire machine it didn't help.</span></div>
<div style><span style="font-family:arial,sans-serif;font-size:13px">We succeeded to manually mount "</span>/home/BO_Ovirt_Storage" to a temporary directory.</div><div><br></div><div><b>postgres=#</b> \connect engine;</div>
<div>You are now connected to database "engine" as user "postgres".</div><div><b>engine=#</b> select current_database();</div><div> current_database</div><div>------------------</div><div> engine</div>
<div>(1 row)</div></div><div><b>engine=#</b> select <a href="http://sds.id">sds.id</a>, ssc.connection from storage_domain_static sds join storage_server_connections ssc on sds.storage=<a href="http://ssc.id">ssc.id</a> where <a href="http://sds.id">sds.id</a>='1083422e-a5db-41b6-b667-b9ef1ef244f0';</div>
<div> id | connection</div><div>--------------------------------------+--------------------------------------------</div><div> 1083422e-a5db-41b6-b667-b9ef1ef244f0 | bufferoverflow.home:/home/BO_Ovirt_Storage</div>
<div>(1 row)</div><div><br></div><div><div><b>[wil@bufferoverflow ~]</b><b>$ mount</b></div><div>proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)</div><div>sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)</div>
<div>devtmpfs on /dev type devtmpfs (rw,nosuid,size=8131256k,nr_inodes=2032814,mode=755)</div><div>securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)</div><div>tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)</div>
<div>devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)</div><div>tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)</div><div>tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)</div>
<div>cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)</div><div>cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)</div>
<div>cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)</div><div>cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)</div><div>cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)</div>
<div>cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)</div><div>cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)</div><div>cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)</div>
<div>cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)</div><div>/dev/sda3 on / type ext4 (rw,relatime,data=ordered)</div><div>rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)</div>
<div>debugfs on /sys/kernel/debug type debugfs (rw,relatime)</div><div>sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)</div><div>hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)</div><div>systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)</div>
<div>mqueue on /dev/mqueue type mqueue (rw,relatime)</div><div>tmpfs on /tmp type tmpfs (rw)</div><div>configfs on /sys/kernel/config type configfs (rw,relatime)</div><div>binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)</div>
<div>/dev/sda5 on /home type ext4 (rw,relatime,data=ordered)</div><div>/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)</div><div>kernelpanic.home:/home/KP_Data_Domain on /rhev/data-center/mnt/kernelpanic.home:_home_KP__Data__Domain type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.100.101.100,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.100.101.100)</div>
<div>bufferoverflow.home:/home/BO_ISO_Domain on /rhev/data-center/mnt/bufferoverflow.home:_home_BO__ISO__Domain type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.100.101.108,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.100.101.108)</div>
</div><div><br></div><div><div><b>[wil@bufferoverflow ~]$</b> ls -la /home/</div><div>total 36</div><div>drwxr-xr-x. 6 root root 4096 Mar 22 11:25 .</div><div>dr-xr-xr-x. 19 root root 4096 Apr 12 18:53 ..</div><div>drwxr-xr-x. 3 vdsm kvm 4096 Mar 27 17:33 BO_ISO_Domain</div>
<div>drwxr-xr-x. 3 vdsm kvm 4096 Mar 27 17:33 BO_Ovirt_Storage</div><div>drwx------. 2 root root 16384 Mar 6 09:11 lost+found</div><div>drwx------. 27 wil wil 4096 Apr 15 01:50 wil</div></div><div><div><b>[wil@bufferoverflow ~]$</b> cd /home/BO_Ovirt_Storage/</div>
<div><b>[wil@bufferoverflow BO_Ovirt_Storage]$ </b>ls -la</div><div>total 12</div><div>drwxr-xr-x. 3 vdsm kvm 4096 Mar 27 17:33 .</div><div>drwxr-xr-x. 6 root root 4096 Mar 22 11:25 ..</div><div>drwxr-xr-x 5 vdsm kvm 4096 Mar 20 23:06 1083422e-a5db-41b6-b667-b9ef1ef244f0</div>
<div>-rwxr-xr-x 1 vdsm kvm 0 Mar 27 17:33 __DIRECT_IO_TEST__</div></div><div><br></div><div style>Thanks,</div><div style>Limor</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Apr 15, 2013 at 4:02 PM, Tal Nisan <span dir="ltr"><<a href="mailto:tnisan@redhat.com" target="_blank">tnisan@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div bgcolor="#ffffff" text="#000000">
Hi Limor,<br>
First we should probably start with checking which mount is the
master storage domain that appears as not found, this should be
checked against the oVirt server database, please run <br>
<br>
select <a href="http://sds.id" target="_blank">sds.id</a>, ssc.connection from storage_domain_static sds join
storage_server_connections ssc on sds.storage=<a href="http://ssc.id" target="_blank">ssc.id</a><br>
where <a href="http://sds.id" target="_blank">sds.id</a>='1083422e-a5db-41b6-b667-b9ef1ef244f0';<br>
<br>
You can run this via psql or a Postgres ui if you have one. <br>
In the results you will see the storage connection in the format of
%hostname%:/%mountName%, then in the VDSM server check in the mount
list that you see that it is mounted, the mount itself should
contain a directory named as the uuid of the master domain, let me
know the result.<br>
<br>
Tal.<div><div class="h5"><br>
<br>
<br>
<br>
On 04/12/2013 07:29 PM, Limor Gavish wrote:
</div></div><blockquote type="cite"><div><div class="h5">
<div dir="ltr">Hi,
<div><br>
</div>
<div>For some reason, without doing anything, all the
storage domains became down and restarting VDSM or the entire
machine do not bring it up.</div>
<div>I am not using lvm<br>
The following errors appear several times in vdsm.log (full
logs are attached):</div>
<div><br>
</div>
<div>
<div>Thread-22::WARNING::2013-04-12
19:00:08,597::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs
failed: 5 [] [' Volume group
"1083422e-a5db-41b6-b667-b9ef1ef244f0" not found']</div>
<div>Thread-22::DEBUG::2013-04-12
19:00:08,598::lvm::402::OperationMutex::(_reloadvgs)
Operation 'lvm reload operation' released the operation
mutex</div>
<div>Thread-22::DEBUG::2013-04-12
19:00:08,681::resourceManager::615::ResourceManager::(releaseResource)
Trying to release resource
'Storage.5849b030-626e-47cb-ad90-3ce782d831b3'</div>
<div>Thread-22::DEBUG::2013-04-12
19:00:08,681::resourceManager::634::ResourceManager::(releaseResource)
Released resource
'Storage.5849b030-626e-47cb-ad90-3ce782d831b3' (0 active
users)</div>
<div>Thread-22::DEBUG::2013-04-12
19:00:08,681::resourceManager::640::ResourceManager::(releaseResource)
Resource 'Storage.5849b030-626e-47cb-ad90-3ce782d831b3' is
free, finding out if anyone is waiting for it.</div>
<div>Thread-22::DEBUG::2013-04-12
19:00:08,682::resourceManager::648::ResourceManager::(releaseResource)
No one is waiting for resource
'Storage.5849b030-626e-47cb-ad90-3ce782d831b3', Clearing
records.</div>
<div>
Thread-22::ERROR::2013-04-12
19:00:08,682::task::850::TaskManager.Task::(_setError)
Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::Unexpected
error</div>
<div>Traceback (most recent call last):</div>
<div> File "/usr/share/vdsm/storage/task.py", line 857, in
_run</div>
<div> return fn(*args, **kargs)</div>
<div> File "/usr/share/vdsm/logUtils.py", line 45, in wrapper</div>
<div> res = f(*args, **kwargs)</div>
<div> File "/usr/share/vdsm/storage/hsm.py", line 939, in
connectStoragePool</div>
<div> masterVersion, options)</div>
<div> File "/usr/share/vdsm/storage/hsm.py", line 986, in
_connectStoragePool</div>
<div> res = pool.connect(hostID, scsiKey, msdUUID,
masterVersion)</div>
<div> File "/usr/share/vdsm/storage/sp.py", line 695, in
connect</div>
<div> self.__rebuild(msdUUID=msdUUID,
masterVersion=masterVersion)</div>
<div> File "/usr/share/vdsm/storage/sp.py", line 1232, in
__rebuild</div>
<div> masterVersion=masterVersion)</div>
<div> File "/usr/share/vdsm/storage/sp.py", line 1576, in
getMasterDomain</div>
<div>
raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)</div>
<div>StoragePoolMasterNotFound: Cannot find master domain:
'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,
msdUUID=1083422e-a5db-41b6-b667-b9ef1ef244f0'</div>
<div>Thread-22::DEBUG::2013-04-12
19:00:08,685::task::869::TaskManager.Task::(_run)
Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::Task._run:
e35a22ac-771a-4916-851f-2fe9d60a0ae6
('5849b030-626e-47cb-ad90-3ce782d831b3', 1,
'5849b030-626e-47cb-ad90-3ce782d831b3',
'1083422e-a5db-41b6-b667-b9ef1ef244f0', 3942) {} failed -
stopping task</div>
<div>Thread-22::DEBUG::2013-04-12
19:00:08,685::task::1194::TaskManager.Task::(stop)
Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::stopping in
state preparing (force False)</div>
<div>Thread-22::DEBUG::2013-04-12
19:00:08,685::task::974::TaskManager.Task::(_decref)
Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::ref 1 aborting
True</div>
<div>Thread-22::INFO::2013-04-12
19:00:08,686::task::1151::TaskManager.Task::(prepare)
Task=`e35a22ac-771a-4916-851f-2fe9d60a0ae6`::aborting: Task
is aborted: 'Cannot find master domain' - code 304</div>
<div>
<br>
</div>
<div>
<div><b>[wil@bufferoverflow ~]$ </b><font color="#666666"><i>sudo
vgs --noheadings --units b --nosuffix --separator \|
-o
uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free</i></font></div>
<div> No volume groups found</div>
</div>
<div><br>
</div>
<div>
<div><b>[wil@bufferoverflow ~]$ </b><i><font color="#444444">mount</font></i></div>
<div>proc on /proc type proc
(rw,nosuid,nodev,noexec,relatime)</div>
<div>sysfs on /sys type sysfs
(rw,nosuid,nodev,noexec,relatime)</div>
<div>devtmpfs on /dev type devtmpfs
(rw,nosuid,size=8131256k,nr_inodes=2032814,mode=755)</div>
<div>securityfs on /sys/kernel/security type securityfs
(rw,nosuid,nodev,noexec,relatime)</div>
<div>tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)</div>
<div>devpts on /dev/pts type devpts
(rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)</div>
<div>tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)</div>
<div>tmpfs on /sys/fs/cgroup type tmpfs
(rw,nosuid,nodev,noexec,mode=755)</div>
<div>cgroup on /sys/fs/cgroup/systemd type cgroup
(rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)</div>
<div>cgroup on /sys/fs/cgroup/cpuset type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset)</div>
<div>cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)</div>
<div>cgroup on /sys/fs/cgroup/memory type cgroup
(rw,nosuid,nodev,noexec,relatime,memory)</div>
<div>cgroup on /sys/fs/cgroup/devices type cgroup
(rw,nosuid,nodev,noexec,relatime,devices)</div>
<div>cgroup on /sys/fs/cgroup/freezer type cgroup
(rw,nosuid,nodev,noexec,relatime,freezer)</div>
<div>cgroup on /sys/fs/cgroup/net_cls type cgroup
(rw,nosuid,nodev,noexec,relatime,net_cls)</div>
<div>cgroup on /sys/fs/cgroup/blkio type cgroup
(rw,nosuid,nodev,noexec,relatime,blkio)</div>
<div>cgroup on /sys/fs/cgroup/perf_event type cgroup
(rw,nosuid,nodev,noexec,relatime,perf_event)</div>
<div>/dev/sda3 on / type ext4 (rw,relatime,data=ordered)</div>
<div>rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs
(rw,relatime)</div>
<div>debugfs on /sys/kernel/debug type debugfs (rw,relatime)</div>
<div>sunrpc on /proc/fs/nfsd type nfsd (rw,relatime)</div>
<div>hugetlbfs on /dev/hugepages type hugetlbfs
(rw,relatime)</div>
<div>systemd-1 on /proc/sys/fs/binfmt_misc type autofs
(rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)</div>
<div>mqueue on /dev/mqueue type mqueue (rw,relatime)</div>
<div>tmpfs on /tmp type tmpfs (rw)</div>
<div>configfs on /sys/kernel/config type configfs
(rw,relatime)</div>
<div>binfmt_misc on /proc/sys/fs/binfmt_misc type
binfmt_misc (rw,relatime)</div>
<div>/dev/sda5 on /home type ext4 (rw,relatime,data=ordered)</div>
<div>/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)</div>
<div>kernelpanic.home:/home/KP_Data_Domain on
/rhev/data-center/mnt/kernelpanic.home:_home_KP__Data__Domain
type nfs
(rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.100.101.100,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.100.101.100)</div>
<div>bufferoverflow.home:/home/BO_ISO_Domain on
/rhev/data-center/mnt/bufferoverflow.home:_home_BO__ISO__Domain
type nfs
(rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.100.101.108,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.100.101.108)</div>
</div>
<div><br>
</div>
<div>
<div><b>[wil@bufferoverflow ~]$ </b><font color="#444444"><i>sudo
find / -name 5849b030-626e-47cb-ad90-3ce782d831b3</i></font></div>
<div>/run/vdsm/pools/5849b030-626e-47cb-ad90-3ce782d831b3</div>
<div><br>
</div>
<div><b>[wil@bufferoverflow ~]$</b> <font color="#444444"><i>sudo
find / -name 1083422e-a5db-41b6-b667-b9ef1ef244f0</i></font></div>
<div>/home/BO_Ovirt_Storage/1083422e-a5db-41b6-b667-b9ef1ef244f0</div>
</div>
<div><br>
</div>
<div>I will extremely appreciate any help,</div>
<div>Limor Gavish</div>
</div>
</div>
</div></div><pre><fieldset></fieldset>
_______________________________________________
Users mailing list
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
</blockquote></div><br></div>