[ovirt-users] Fwd: Re: HA agent fails to start
Richard Neuboeck
hawk at tbi.univie.ac.at
Fri Apr 15 06:45:35 UTC 2016
On 04/14/2016 11:03 PM, Simone Tiraboschi wrote:
> On Thu, Apr 14, 2016 at 10:38 PM, Simone Tiraboschi <stirabos at redhat.com> wrote:
>> On Thu, Apr 14, 2016 at 6:53 PM, Richard Neuboeck <hawk at tbi.univie.ac.at> wrote:
>>> On 14.04.16 18:46, Simone Tiraboschi wrote:
>>>> On Thu, Apr 14, 2016 at 4:04 PM, Richard Neuboeck <hawk at tbi.univie.ac.at> wrote:
>>>>> On 04/14/2016 02:14 PM, Simone Tiraboschi wrote:
>>>>>> On Thu, Apr 14, 2016 at 12:51 PM, Richard Neuboeck
>>>>>> <hawk at tbi.univie.ac.at> wrote:
>>>>>>> On 04/13/2016 10:00 AM, Simone Tiraboschi wrote:
>>>>>>>> On Wed, Apr 13, 2016 at 9:38 AM, Richard Neuboeck <hawk at tbi.univie.ac.at> wrote:
>>>>>>>>> The answers file shows the setup time of both machines.
>>>>>>>>>
>>>>>>>>> On both machines hosted-engine.conf got rotated right before I wrote
>>>>>>>>> this mail. Is it possible that I managed to interrupt the rotation with
>>>>>>>>> the reboot so the backup was accurate but the update not yet written to
>>>>>>>>> hosted-engine.conf?
>>>>>>>>
>>>>>>>> AFAIK we don't have any rotation mechanism for that file; something
>>>>>>>> else you have in place on that host?
>>>>>>>
>>>>>>> Those machines are all CentOS 7.2 minimal installs. The only
>>>>>>> adaptation I do is installing vim, removing postfix and installing
>>>>>>> exim, removing firewalld and installing iptables-service. Then I add
>>>>>>> the oVirt repos (3.6 and 3.6-snapshot) and deploy the host.
>>>>>>>
>>>>>>> But checking lsof shows that 'ovirt-ha-agent --no-daemon' has access
>>>>>>> to the config file (and the one ending with ~):
>>>>>>>
>>>>>>> # lsof | grep 'hosted-engine.conf~'
>>>>>>> ovirt-ha- 193446 vdsm 351u REG
>>>>>>> 253,0 1021 135070683
>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf~
>>>>>>
>>>>>> This is not that much relevant if the file was renamed after
>>>>>> ovirt-ha-agent opened it.
>>>>>> Try this:
>>>>>>
>>>>>> [root at c72he20160405h1 ovirt-hosted-engine-setup]# tail -n1 -f
>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf &
>>>>>> [1] 28866
>>>>>> [root at c72he20160405h1 ovirt-hosted-engine-setup]# port=
>>>>>>
>>>>>> [root at c72he20160405h1 ovirt-hosted-engine-setup]# lsof | grep hosted-engine.conf
>>>>>> tail 28866 root 3r REG
>>>>>> 253,0 1014 1595898 /etc/ovirt-hosted-engine/hosted-engine.conf
>>>>>> [root at c72he20160405h1 ovirt-hosted-engine-setup]# mv
>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf
>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf_123
>>>>>> [root at c72he20160405h1 ovirt-hosted-engine-setup]# lsof | grep hosted-engine.conf
>>>>>> tail 28866 root 3r REG
>>>>>> 253,0 1014 1595898
>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf_123
>>>>>> [root at c72he20160405h1 ovirt-hosted-engine-setup]#
>>>>>>
>>>>>
>>>>> I've issued the commands you suggested but I don't know how that
>>>>> helps to find the process accessing the config files.
>>>>>
>>>>> After moving the hosted-engine.conf file the HA agent crashed
>>>>> logging the information that the config file is not available.
>>>>>
>>>>> Here is the output from every command:
>>>>>
>>>>> # tail -n1 -f /etc/ovirt-hosted-engine/hosted-engine.conf &
>>>>> [1] 167865
>>>>> [root at cube-two ~]# port=
>>>>> # lsof | grep hosted-engine.conf
>>>>> ovirt-ha- 166609 vdsm 5u REG
>>>>> 253,0 1021 134433491
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf (deleted)
>>>>> ovirt-ha- 166609 vdsm 7u REG
>>>>> 253,0 1021 134433453
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf (deleted)
>>>>> ovirt-ha- 166609 vdsm 8u REG
>>>>> 253,0 1021 134433489
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf (deleted)
>>>>> ovirt-ha- 166609 vdsm 9u REG
>>>>> 253,0 1021 134433493
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf~
>>>>> ovirt-ha- 166609 vdsm 10u REG
>>>>> 253,0 1021 134433495
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf
>>>>> tail 167865 root 3r REG
>>>>> 253,0 1021 134433493
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf~
>>>>> # mv /etc/ovirt-hosted-engine/hosted-engine.conf
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf_123
>>>>> # lsof | grep hosted-engine.conf
>>>>> ovirt-ha- 166609 vdsm 5u REG
>>>>> 253,0 1021 134433491
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf (deleted)
>>>>> ovirt-ha- 166609 vdsm 7u REG
>>>>> 253,0 1021 134433453
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf (deleted)
>>>>> ovirt-ha- 166609 vdsm 8u REG
>>>>> 253,0 1021 134433489
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf (deleted)
>>>>> ovirt-ha- 166609 vdsm 9u REG
>>>>> 253,0 1021 134433493
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf (deleted)
>>>>> ovirt-ha- 166609 vdsm 10u REG
>>>>> 253,0 1021 134433495
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf (deleted)
>>>>> ovirt-ha- 166609 vdsm 12u REG
>>>>> 253,0 1021 134433498
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf~
>>>>> ovirt-ha- 166609 vdsm 13u REG
>>>>> 253,0 1021 134433499
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf_123
>>>>> tail 167865 root 3r REG
>>>>> 253,0 1021 134433493
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf (deleted)
>>>>>
>>>>>
>>>>>> The issue is understanding who renames that file on your host.
>>>>>
>>>>> From what I've seen so far it looks like a child of vdsm accesses
>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf periodically but is not
>>>>> responsible for the ~ file.
>>>>>
>>>>> # auditctl -w /etc/ovirt-hosted-engine/hosted-engine.conf
>>>>> and
>>>>> # auditctl -w /etc/ovirt-hosted-engine/hosted-engine.conf~
>>>>>
>>>>> auditd.log shows this:
>>>>>
>>>>> type=SYSCALL msg=audit(1460639783.613:482590): arch=c000003e
>>>>> syscall=2 success=yes exit=75 a0=7f29b400f0b0 a1=0 a2=1b6 a3=24
>>>>> items=1 ppid=1 pid=3701 auid=4294967295 uid=36 gid=36 euid=36
>>>>> suid=36 fsuid=36 egid=36 sgid=36 fsgid=36 tty=(none) ses=4294967295
>>>>> comm="jsonrpc.Executo" exe="/usr/bin/python2.7"
>>>>> subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 key=(null)
>>>>> type=CWD msg=audit(1460639783.613:482590): cwd="/"
>>>>> type=PATH msg=audit(1460639783.613:482590): item=0
>>>>> name="/etc/ovirt-hosted-engine/hosted-engine.conf" inode=134433499
>>>>> dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00
>>>>> obj=system_u:object_r:etc_t:s0 objtype=NORMAL
>>>>>
>>>>>
>>>>> Now that the HA agent is dead I'm removing the ~ file and starting
>>>>> the HA agent again. The ~ file immediately appears again.
>>>>>
>>>>> # rm hosted-engine.conf~
>>>>> rm: remove regular file ‘hosted-engine.conf~’? y
>>>>> [root at cube-two ovirt-hosted-engine]# ls -l
>>>>> total 6800
>>>>> -rw-r--r--. 1 root root 3252 Apr 8 10:35 answers.conf
>>>>> -rw-r--r--. 1 root root 6948582 Apr 14 14:48 ha-trace.log
>>>>> -rw-r--r--. 1 root root 1021 Apr 14 15:07 hosted-engine.conf
>>>>> -rw-r--r--. 1 root root 413 Apr 8 10:35 iptables.example
>>>>> [root at cube-two ovirt-hosted-engine]# systemctl start ovirt-ha-agent
>>>>> [root at cube-two ovirt-hosted-engine]# ls -l
>>>>> total 6804
>>>>> -rw-r--r--. 1 root root 3252 Apr 8 10:35 answers.conf
>>>>> -rw-r--r--. 1 root root 6948582 Apr 14 14:48 ha-trace.log
>>>>> -rw-r--r--. 1 root root 1021 Apr 14 15:18 hosted-engine.conf
>>>>> -rw-r--r--. 1 root root 1021 Apr 14 15:07 hosted-engine.conf~
>>>>> -rw-r--r--. 1 root root 413 Apr 8 10:35 iptables.example
>>>>>
>>>>> The auditd.log shows that ~ file is moved into place but not what
>>>>> issued the mv:
>>>>>
>>>>> type=CONFIG_CHANGE msg=audit(1460639919.277:482750): auid=4294967295
>>>>> ses=4294967295 op="updated_rules"
>>>>> path="/etc/ovirt-hosted-engine/hosted-engine.conf~" key=(null)
>>>>> list=4 res=1
>>>>> type=SYSCALL msg=audit(1460639919.277:482751): arch=c000003e
>>>>> syscall=82 success=yes exit=0 a0=7ffe4b3c0e90 a1=7ffe4b3bf920
>>>>> a2=7f68083a2778 a3=7ffe4b3bf680 items=5 ppid=170233 pid=170234
>>>>> auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 eg
>>>>> id=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="mv"
>>>>> exe="/usr/bin/mv" subj=system_u:system_r:unconfined_service_t:s0
>>>>> key=(null)
>>>>> type=CWD msg=audit(1460639919.277:482751): cwd="/"
>>>>> type=PATH msg=audit(1460639919.277:482751): item=0
>>>>> name="/etc/ovirt-hosted-engine/" inode=69555 dev=fd:00 mode=040755
>>>>> ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0 objtype=PARENT
>>>>> type=PATH msg=audit(1460639919.277:482751): item=1
>>>>> name="/etc/ovirt-hosted-engine/" inode=69555 dev=fd:00 mode=040755
>>>>> ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0 objtype=PARENT
>>>>> type=PATH msg=audit(1460639919.277:482751): item=2
>>>>> name="/etc/ovirt-hosted-engine/hosted-engine.conf" inode=134433453
>>>>> dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00
>>>>> obj=system_u:object_r:etc_t:s0 objtype=DELETE
>>>>> type=PATH msg=audit(1460639919.277:482751): item=3
>>>>> name="/etc/ovirt-hosted-engine/hosted-engine.conf~" inode=134433499
>>>>> dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00
>>>>> obj=system_u:object_r:etc_t:s0 objtype=DELETE
>>>>> type=PATH msg=audit(1460639919.277:482751): item=4
>>>>> name="/etc/ovirt-hosted-engine/hosted-engine.conf~" inode=134433453
>>>>> dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00
>>>>> obj=system_u:object_r:etc_t:s0 objtype=CREATE
>>>>>
>>>>>
>>>>>> As a thumb rule, if a file name is appended with a tilde~, it only
>>>>>> means that it is a backup created by a text editor or similar program.
>>>>>
>>>>> If anyone except myself would have access to these systems I would
>>>>> guess the same. But since I'm not editing anything in
>>>>> /etc/ovirt-hosted-engine there must be another reason. And there is.
>>>>>
>>>>> Aside from auditd I tried to strace the whole thing just to make
>>>>> sure it comes from the HA agent.
>>>>>
>>>>> [root at cube-two ~]# strace -o ha-trace.log -f
>>>>> /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon
>>>>>
>>>>> Looking at the trace log I found this:
>>>>>
>>>>> 183409 statfs("/etc/ovirt-hosted-engine/.", {f_type=0x58465342,
>>>>> f_bsize=4096, f_blocks=13100800, f_bfree=12523576,
>>>>> f_bavail=12523576, f_files=52428800, f_ffree=52379892,
>>>>> f_fsid={64768, 0}, f_namelen=255, f_frsize=4096}) = 0
>>>>> 183409 rename("/etc/ovirt-hosted-engine/hosted-engine.conf",
>>>>> "/etc/ovirt-hosted-engine/hosted-engine.conf~") = 0
>>>>> 183409 rename("/var/lib/ovirt-hosted-engine-ha/tmpNjTElr",
>>>>> "/etc/ovirt-hosted-engine/hosted-engine.conf") = 0
>>>>> 183409 newfstatat(AT_FDCWD,
>>>>> "/etc/ovirt-hosted-engine/hosted-engine.conf",
>>>>> {st_mode=S_IFREG|0600, st_size=1021, ...}, AT_SYMLINK_NOFOLLOW) = 0
>>>>> 183409 open("/etc/ovirt-hosted-engine/hosted-engine.conf",
>>>>> O_RDONLY|O_NOFOLLOW) = 3
>>>>>
>>>>>
>>>>> Putting it all together I started reading the HA agent sources and
>>>>> found the function _wrote_updated_conf_file in
>>>>> /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/upgrade.py
>>>>> which issues a mv -b which creates the ~ file.
>>>>
>>>> This should just trigger during 3.5 to 3.6 upgrade but your host are new.
>>>> Can you please attach /var/log/ovirt-hosted-engine-ha/agent.log from
>>>> one of them?
>>>
>>> The agent.log of host cube-two is attached to this mail.
>>
>> Yes, you are right:
>> it's looping trying to fix a path in the config file (on 3.5 we didn't
>> check if an NFS path was ending with a '/' while for other reasons it
>> wasn't working on 3.6 and so we need to fix it) but its doesn't seams
>> you case and so the strange loop.
>>
>> Now I need to understand why it enters there.
>> Can you please execute
>> tree /rhev/data-center/
>> and post me the output?
>>
>> Thanks again
>
> OK, I think it's just a side effect if this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=1317699
I stumbled upon this bug and the mount problem. I thought this bug
was caused by the missing vfs_type for the engine gluster storage
volume.
> The local mount point of NFS and GlusterFS storage domain are slightly
> different.
> The hosted-engine storage domain autoimport procedure didn't probably
> recognized the gluster storage domain as a gluster one and so marked
> it as nfs in the engine and so it got mounted twice on different paths
> (by ovirt-ha-agent at boot time and by engine further on);
> ovirt-ha-agent notices it and think that's due to the old trailing
> slash issue on NFS paths and so it tries to fix but the there is
> nothing to fix in the config file and so the infinite loop.
Thanks for the explanation.
> I'll try to push a patch ASAP.
Thank you!
> Can you please still provide the output of
> tree /rhev/data-center/
Sure. It's attached to this mail (tree ran on host cube-two).
Anything else I can do to help?
>
>>>>> The question now is why is this done so frequently. Especially
>>>>> considering since there are no modifications to the file. Is this
>>>>> behavior normal?
>>>>>
>>>>> [root at cube-two ~]# diff /etc/ovirt-hosted-engine/hosted-engine.conf*
>>>>> [root at cube-two ~]#
>>>>>
>>>>>
>>>>>>>>> [root at cube-two ~]# ls -l /etc/ovirt-hosted-engine
>>>>>>>>> total 16
>>>>>>>>> -rw-r--r--. 1 root root 3252 Apr 8 10:35 answers.conf
>>>>>>>>> -rw-r--r--. 1 root root 1021 Apr 13 09:31 hosted-engine.conf
>>>>>>>>> -rw-r--r--. 1 root root 1021 Apr 13 09:30 hosted-engine.conf~
>>>>>>>>>
>>>>>>>>> [root at cube-three ~]# ls -l /etc/ovirt-hosted-engine
>>>>>>>>> total 16
>>>>>>>>> -rw-r--r--. 1 root root 3233 Apr 11 08:02 answers.conf
>>>>>>>>> -rw-r--r--. 1 root root 1002 Apr 13 09:31 hosted-engine.conf
>>>>>>>>> -rw-r--r--. 1 root root 1002 Apr 13 09:31 hosted-engine.conf~
>>>>>>>>>
>>>>>>>>> On 12.04.16 16:01, Simone Tiraboschi wrote:
>>>>>>>>>> Everything seams fine here,
>>>>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf seams to be correctly
>>>>>>>>>> created with the right name.
>>>>>>>>>> Can you please check the latest modification time of your
>>>>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf~ and compare it with the
>>>>>>>>>> setup time?
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 12, 2016 at 2:34 PM, Richard Neuboeck <hawk at tbi.univie.ac.at> wrote:
>>>>>>>>>>> On 04/12/2016 11:32 AM, Simone Tiraboschi wrote:
>>>>>>>>>>>> On Mon, Apr 11, 2016 at 8:11 AM, Richard Neuboeck <hawk at tbi.univie.ac.at> wrote:
>>>>>>>>>>>>> Hi oVirt Group,
>>>>>>>>>>>>>
>>>>>>>>>>>>> in my attempts to get all aspects of oVirt 3.6 up and running I
>>>>>>>>>>>>> stumbled upon something I'm not sure how to fix:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Initially I installed a hosted engine setup. After that I added
>>>>>>>>>>>>> another HA host (with hosted-engine --deploy). The host was
>>>>>>>>>>>>> registered in the Engine correctly and HA agent came up as expected.
>>>>>>>>>>>>>
>>>>>>>>>>>>> However if I reboot the second host (through the Engine UI or
>>>>>>>>>>>>> manually) HA agent fails to start. The reason seems to be that
>>>>>>>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf is empty. The backup
>>>>>>>>>>>>> file ending with ~ exists though.
>>>>>>>>>>>>
>>>>>>>>>>>> Can you please attach hosted-engine-setup logs from your additional hosts?
>>>>>>>>>>>> AFAIK our code will never take a ~ ending backup of that file.
>>>>>>>>>>>
>>>>>>>>>>> ovirt-hosted-engine-setup logs from both additional hosts are
>>>>>>>>>>> attached to this mail.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Here are the log messages from the journal:
>>>>>>>>>>>>> Apr 11 07:29:39 cube-two.tbi.univie.ac.at systemd[1]: Starting oVirt
>>>>>>>>>>>>> Hosted Engine High Availability Monitoring Agent...
>>>>>>>>>>>>> Apr 11 07:29:39 cube-two.tbi.univie.ac.at ovirt-ha-agent[3747]:
>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.agent.Agent:ovirt-hosted-engine-ha
>>>>>>>>>>>>> agent 1.3.5.3-0.0.master started
>>>>>>>>>>>>> Apr 11 07:29:39 cube-two.tbi.univie.ac.at ovirt-ha-agent[3747]:
>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Found
>>>>>>>>>>>>> certificate common name: cube-two.tbi.univie.ac.at
>>>>>>>>>>>>> Apr 11 07:29:39 cube-two.tbi.univie.ac.at ovirt-ha-agent[3747]:
>>>>>>>>>>>>> ovirt-ha-agent
>>>>>>>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Hosted
>>>>>>>>>>>>> Engine is not configured. Shutting down.
>>>>>>>>>>>>> Apr 11 07:29:39 cube-two.tbi.univie.ac.at ovirt-ha-agent[3747]:
>>>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Hosted
>>>>>>>>>>>>> Engine is not configured. Shutting down.
>>>>>>>>>>>>> Apr 11 07:29:39 cube-two.tbi.univie.ac.at ovirt-ha-agent[3747]:
>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.agent.Agent:Agent shutting down
>>>>>>>>>>>>> Apr 11 07:29:39 cube-two.tbi.univie.ac.at systemd[1]:
>>>>>>>>>>>>> ovirt-ha-agent.service: main process exited, code=exited, status=255/n/a
>>>>>>>>>>>>>
>>>>>>>>>>>>> If I restore the configuration from the backup file and manually
>>>>>>>>>>>>> restart the HA agent it's working properly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For testing purposes I added a third HA host which turn out to
>>>>>>>>>>>>> behave exactly the same.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any help would be appreciated!
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Richard
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> /dev/null
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> /dev/null
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> Users at ovirt.org
>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> /dev/null
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> /dev/null
>>>>>
>>>
--
/dev/null
-------------- next part --------------
/rhev/data-center/
├── 00000001-0001-0001-0001-0000000003ce
│ ├── 2732cda7-41c3-4b6f-87cc-14cddc6c0fe4 -> /rhev/data-center/mnt/glusterSD/borg-sphere-one:_export/2732cda7-41c3-4b6f-87cc-14cddc6c0fe4
│ ├── 516a2d77-edcc-46cf-9860-5fb44112f4d0 -> /rhev/data-center/mnt/vinculum.tbi.univie.ac.at:_var_lib_exports_iso/516a2d77-edcc-46cf-9860-5fb44112f4d0
│ ├── 81b59b08-dc02-414b-8c01-43b7ded14de3 -> /rhev/data-center/mnt/glusterSD/borg-sphere-one:_engine/81b59b08-dc02-414b-8c01-43b7ded14de3
│ ├── de6281b6-f843-48cc-b7f6-0ad467760e0d -> /rhev/data-center/mnt/glusterSD/borg-sphere-one:_plexus/de6281b6-f843-48cc-b7f6-0ad467760e0d
│ └── mastersd -> /rhev/data-center/mnt/glusterSD/borg-sphere-one:_plexus/de6281b6-f843-48cc-b7f6-0ad467760e0d
└── mnt
├── glusterSD
│ ├── borg-sphere-one:_engine
│ │ ├── 81b59b08-dc02-414b-8c01-43b7ded14de3
│ │ │ ├── dom_md
│ │ │ │ ├── ids
│ │ │ │ ├── inbox
│ │ │ │ ├── leases
│ │ │ │ ├── metadata
│ │ │ │ └── outbox
│ │ │ ├── ha_agent
│ │ │ │ ├── hosted-engine.lockspace -> /var/run/vdsm/storage/81b59b08-dc02-414b-8c01-43b7ded14de3/6c127f4f-8ea7-4356-99ad-c1182fb560f5/1fcb4b0b-c8d3-4bd6-b604-ed7648d0d52e
│ │ │ │ └── hosted-engine.metadata -> /var/run/vdsm/storage/81b59b08-dc02-414b-8c01-43b7ded14de3/b03a7f5f-a64e-403d-aac2-5f177c2a2305/c2362a91-475c-4617-a848-32725af1eec4
│ │ │ └── images
│ │ │ ├── 01822279-6fad-4b2c-9fb7-5e21d71574fe
│ │ │ │ ├── 179da82e-aa55-4d59-aef2-36dd1564d026
│ │ │ │ ├── 179da82e-aa55-4d59-aef2-36dd1564d026.lease
│ │ │ │ └── 179da82e-aa55-4d59-aef2-36dd1564d026.meta
│ │ │ ├── 6c127f4f-8ea7-4356-99ad-c1182fb560f5
│ │ │ │ ├── 1fcb4b0b-c8d3-4bd6-b604-ed7648d0d52e
│ │ │ │ ├── 1fcb4b0b-c8d3-4bd6-b604-ed7648d0d52e.lease
│ │ │ │ └── 1fcb4b0b-c8d3-4bd6-b604-ed7648d0d52e.meta
│ │ │ ├── 8a49e0fd-9d46-46ce-a040-aadb25c0a129
│ │ │ │ ├── 44008af0-53ad-405f-8be3-c37c3a44dc28
│ │ │ │ ├── 44008af0-53ad-405f-8be3-c37c3a44dc28.lease
│ │ │ │ └── 44008af0-53ad-405f-8be3-c37c3a44dc28.meta
│ │ │ ├── a4800611-eb6a-46ab-b518-3d7940a029d7
│ │ │ │ ├── 04e10a84-e9f8-4794-8548-9a24461ccb3f
│ │ │ │ ├── 04e10a84-e9f8-4794-8548-9a24461ccb3f.lease
│ │ │ │ └── 04e10a84-e9f8-4794-8548-9a24461ccb3f.meta
│ │ │ ├── b03a7f5f-a64e-403d-aac2-5f177c2a2305
│ │ │ │ ├── c2362a91-475c-4617-a848-32725af1eec4
│ │ │ │ ├── c2362a91-475c-4617-a848-32725af1eec4.lease
│ │ │ │ └── c2362a91-475c-4617-a848-32725af1eec4.meta
│ │ │ └── ddedbd89-93e3-4e75-9e94-32eadd6f9221
│ │ │ ├── eecacd50-c19f-435a-bbd3-5972b9232fba
│ │ │ ├── eecacd50-c19f-435a-bbd3-5972b9232fba.lease
│ │ │ └── eecacd50-c19f-435a-bbd3-5972b9232fba.meta
│ │ └── __DIRECT_IO_TEST__
│ ├── borg-sphere-one:_export
│ │ ├── 2732cda7-41c3-4b6f-87cc-14cddc6c0fe4
│ │ │ ├── dom_md
│ │ │ │ ├── ids
│ │ │ │ ├── inbox
│ │ │ │ ├── leases
│ │ │ │ ├── metadata
│ │ │ │ └── outbox
│ │ │ ├── images
│ │ │ └── master
│ │ │ ├── tasks
│ │ │ └── vms
│ │ └── __DIRECT_IO_TEST__
│ └── borg-sphere-one:_plexus
│ ├── de6281b6-f843-48cc-b7f6-0ad467760e0d
│ │ ├── dom_md
│ │ │ ├── ids
│ │ │ ├── inbox
│ │ │ ├── leases
│ │ │ ├── metadata
│ │ │ └── outbox
│ │ ├── images
│ │ │ ├── 4083884e-517e-449d-8f5c-69f21b44f5f5
│ │ │ │ ├── e2ead827-231b-4adf-a122-921ee3febbc9
│ │ │ │ ├── e2ead827-231b-4adf-a122-921ee3febbc9.lease
│ │ │ │ └── e2ead827-231b-4adf-a122-921ee3febbc9.meta
│ │ │ ├── 937e350f-b8b9-4a12-ab2b-c7d1c9386468
│ │ │ │ ├── a1f1dfc3-a59e-4b31-bb86-6682e4c8e217
│ │ │ │ ├── a1f1dfc3-a59e-4b31-bb86-6682e4c8e217.lease
│ │ │ │ └── a1f1dfc3-a59e-4b31-bb86-6682e4c8e217.meta
│ │ │ ├── a4646fc4-d71d-4bce-92a6-97747694b9fa
│ │ │ │ ├── 5f7fa95d-bccb-4ce7-86b1-d7bb8f8ebbdd
│ │ │ │ ├── 5f7fa95d-bccb-4ce7-86b1-d7bb8f8ebbdd.lease
│ │ │ │ └── 5f7fa95d-bccb-4ce7-86b1-d7bb8f8ebbdd.meta
│ │ │ ├── bd96ed15-42d1-4f91-82cc-008e671a45f3
│ │ │ │ ├── 35dd8fa3-3678-425a-ac2b-12409f11a50e
│ │ │ │ ├── 35dd8fa3-3678-425a-ac2b-12409f11a50e.lease
│ │ │ │ └── 35dd8fa3-3678-425a-ac2b-12409f11a50e.meta
│ │ │ └── cc14ba05-5ae4-4c6f-8c8a-42ad88410636
│ │ │ ├── 618f41b1-e26e-453e-8bbd-50cf6ee375d0
│ │ │ ├── 618f41b1-e26e-453e-8bbd-50cf6ee375d0.lease
│ │ │ ├── 618f41b1-e26e-453e-8bbd-50cf6ee375d0.meta
│ │ │ ├── 9b6df3d4-6983-44c1-bda6-e6d2d69157c8
│ │ │ ├── 9b6df3d4-6983-44c1-bda6-e6d2d69157c8.lease
│ │ │ └── 9b6df3d4-6983-44c1-bda6-e6d2d69157c8.meta
│ │ └── master
│ │ ├── tasks
│ │ └── vms
│ └── __DIRECT_IO_TEST__
├── _var_lib_ovirt-hosted-engine-setup_tmpICcmDW
└── vinculum.tbi.univie.ac.at:_var_lib_exports_iso
├── 516a2d77-edcc-46cf-9860-5fb44112f4d0
│ ├── dom_md
│ │ ├── ids
│ │ ├── inbox
│ │ ├── leases
│ │ ├── metadata
│ │ └── outbox
│ └── images
│ └── 11111111-1111-1111-1111-111111111111
│ ├── CentOS-7-x86_64-NetInstall-1511.iso
│ └── Fedora-Workstation-Live-x86_64-24_Alpha-7.iso
└── __DIRECT_IO_TEST__
44 directories, 64 files
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160415/e06137fc/attachment-0001.sig>
More information about the Users
mailing list