[JIRA] (OVIRT-1421) Docker storage issues on FC24 nodes

[Barak Korren (oVirt JIRA)]

    [ https://ovirt-jira.atlassian.net/browse/OVIRT-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=31701#comment-31701 ] 

Barak Korren commented on OVIRT-1421:
-------------------------------------

More information:

The original issue was found on "{{vm0064.workers-phx.ovirt.org}}" which is an FC24 VM.

This is what you get when you try to run "{{docker-storage-setup}}" manually on it:

{code}
# docker-storage-setup 
INFO: Volume group backing root filesystem could not be determined
ERROR: No valid volume group found. Exiting.
{code}

We seem to be seeing a very similar issue on EL7. the following is from "{{vm0024.workers-phx.ovirt.org}}".

{code}
[root at vm0024 ~]# systemctl status docker-storage-setup
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2017-05-31 03:00:42 UTC; 3h 53min ago
  Process: 16223 ExecStart=/usr/bin/docker-storage-setup (code=exited, status=1/FAILURE)
 Main PID: 16223 (code=exited, status=1/FAILURE)

May 31 03:00:42 vm0024.workers-phx.ovirt.org systemd[1]: Starting Docker Storage Setup...
May 31 03:00:42 vm0024.workers-phx.ovirt.org docker-storage-setup[16223]: INFO: Volume group backing root filesystem could not be determined
May 31 03:00:42 vm0024.workers-phx.ovirt.org docker-storage-setup[16223]: ERROR: No valid volume group found. Exiting.
May 31 03:00:42 vm0024.workers-phx.ovirt.org systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
May 31 03:00:42 vm0024.workers-phx.ovirt.org systemd[1]: Failed to start Docker Storage Setup.
May 31 03:00:42 vm0024.workers-phx.ovirt.org systemd[1]: Unit docker-storage-setup.service entered failed state.
May 31 03:00:42 vm0024.workers-phx.ovirt.org systemd[1]: docker-storage-setup.service failed.
{code}

This is the output when running the command manually:
{code}
[root at vm0024 ~]# docker-storage-setup 
INFO: Volume group backing root filesystem could not be determined
ERROR: No valid volume group found. Exiting.
{code}

This is, strangely enough, not causing processes to get stuck on EL7 though:
{code}
[root at vm0024 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
{code}

In fact, things seem to be working fine despite the failure:
{code}
[root at vm0024 ~]# docker run --rm -it centos hostname
62d53a5ab5a7
{code}

I suspect we're seeing a recent regression in docker, and previously the setup script knew how to handle nodes without LVM. I'm not finding evidence of an upgrade in yum/dnf logs though.

> Docker storage issues on FC24 nodes
> -----------------------------------
>
>                 Key: OVIRT-1421
>                 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1421
>             Project: oVirt - virtualization made easy
>          Issue Type: Bug
>          Components: oVirt CI
>            Reporter: Barak Korren
>            Assignee: infra
>            Priority: High
>              Labels: containers
>
> docekr startup seems to be failing on FC24 nodes and making other dokcer commands get stuck forever.
> The core reason behind the failure seems to be that the Docker storage setup failed:
> {code}
> [root at vm0064 ~]# systemctl status docker-storage-setup
> ● docker-storage-setup.service - Docker Storage Setup
>    Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
>    Active: failed (Result: exit-code) since Tue 2017-05-30 03:14:49 UTC; 1 day 2h ago
>   Process: 1489 ExecStart=/usr/bin/docker-storage-setup (code=exited, status=1/FAILURE)
>  Main PID: 1489 (code=exited, status=1/FAILURE)
> May 30 03:14:49 vm0064.workers-phx.ovirt.org systemd[1]: Starting Docker Storage Setup...
> May 30 03:14:49 vm0064.workers-phx.ovirt.org docker-storage-setup[1489]: INFO: Volume group backing root filesystem could not be determined
> May 30 03:14:49 vm0064.workers-phx.ovirt.org docker-storage-setup[1489]: ERROR: No valid volume group found. Exiting.
> May 30 03:14:49 vm0064.workers-phx.ovirt.org systemd[1]: docker-storage-setup.service: Main process exited, code=exited, status=1/FAILURE
> May 30 03:14:49 vm0064.workers-phx.ovirt.org systemd[1]: Failed to start Docker Storage Setup.
> May 30 03:14:49 vm0064.workers-phx.ovirt.org systemd[1]: docker-storage-setup.service: Unit entered failed state.
> May 30 03:14:49 vm0064.workers-phx.ovirt.org systemd[1]: docker-storage-setup.service: Failed with result 'exit-code'.
> {code}
> This seem to be does to the fact that the system's root filesystem is on a plain disk rather then LVM. This is although the docker default DM storage driver is supposed to be able to use a plaing file on the filesystem. Need to check if this is an issue specific to this slave of something that has to do with all FC24 slaves.



--
This message was sent by Atlassian JIRA
(v1000.1010.2#100044)