DRBD Integration with oVirt
Hello all, A DRBD integration into oVirt is currently being planned. DRBD provides replicated, highly available block storage. For those unfamiliar with DRBD more information can be found here: https://www.drbd.org/en/comp/drbd-linux-driver DRBD 9 is the latest version of DRBD and interacts with DRBD Manage which is a daemon exposed via a D-Bus api. More information can be found here: https://www.drbd.org/en/comp/drbdmanage DRBD Manage handles the creation, assignment, etc. of DRBD volumes and is the interface through which Cinder manages DRBD. We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options: 1) Use Cinder. Although we are integrated into Cinder, there are concerns about the need to set up a complete Cinder instance to provide access to DRBD. 2) The alternative to Cinder would be to have the oVirt Engine communicate directly to DRBD Manage via D-Bus. Any advice on the best way to achieve this would be much appreciated. -- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212
We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options:
My vote would be for native support rather then a Cinder based one. I don't have any hard numbers, but it seems to me the adoption of OpenStack technologies for use with oVirt is not as widespread as one may hope. But what sort of user experience and system architecture is intended? Is configuration of volumes planned to be done directly from the oVirt UI? While oVirt can do a little Gluster volume management for now, it seems it was done because it some point it was thought that oVirt would be the primary management UI for Gluster. AFAIK Gluster is now getting its own UI. As far as using DRBD for backing oVirt VMs, this seems to already be working. The oVirt CI system (which runs on top of oVirt) had been using a DRBD-backed NFS storage for years... -- Barak Korren bkorren@redhat.com RHEV-CI Team
On 06/04/2016 10:40 PM, Barak Korren wrote:
We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options:
My vote would be for native support rather then a Cinder based one. I don't have any hard numbers, but it seems to me the adoption of OpenStack technologies for use with oVirt is not as widespread as one may hope.
But what sort of user experience and system architecture is intended? Is configuration of volumes planned to be done directly from the oVirt UI?
Once a DRBD Manage storage cluster is set up, oVirt would handle the management of DRBD volumes.
While oVirt can do a little Gluster volume management for now, it seems it was done because it some point it was thought that oVirt would be the primary management UI for Gluster. AFAIK Gluster is now getting its own UI.
The DRBD Manage daemon ships with a command line utility which is the UI for interacting with DRBD 9. I do not believe that there is a stand alone DRBD GUI planned at this time. At least for now, the user most likely will prepare a DRBD Manage cluster up to the point where it's ready to start creating/assigning volumes and then oVirt will take over from there.
As far as using DRBD for backing oVirt VMs, this seems to already be working. The oVirt CI system (which runs on top of oVirt) had been using a DRBD-backed NFS storage for years...
The most recent release of DRBD contains new features that allow it to integrate more directly with virtualization/cloud platforms. So VMs could have DRBD-backed storage without additional overhead from NFS. -- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212
On Mon, Jun 6, 2016 at 7:23 PM, Hayley Swimelar <hayley@linbit.com> wrote:
On 06/04/2016 10:40 PM, Barak Korren wrote:
We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options:
My vote would be for native support rather then a Cinder based one. I don't have any hard numbers, but it seems to me the adoption of OpenStack technologies for use with oVirt is not as widespread as one may hope.
But what sort of user experience and system architecture is intended? Is configuration of volumes planned to be done directly from the oVirt UI?
Once a DRBD Manage storage cluster is set up, oVirt would handle the management of DRBD volumes.
While oVirt can do a little Gluster volume management for now, it
seems it was done because it some point it was thought that oVirt would be the primary management UI for Gluster. AFAIK Gluster is now getting its own UI.
The DRBD Manage daemon ships with a command line utility which is the UI for interacting with DRBD 9. I do not believe that there is a stand alone DRBD GUI planned at this time.
Is using the CLI considered a 'stable API' ? (or perhaps it can output a JSON or something that can be considered as such)?
At least for now, the user most likely will prepare a DRBD Manage cluster up to the point where it's ready to start creating/assigning volumes and then oVirt will take over from there.
As far as using DRBD for backing oVirt VMs, this seems to already be
working. The oVirt CI system (which runs on top of oVirt) had been using a DRBD-backed NFS storage for years...
The most recent release of DRBD contains new features that allow it to integrate more directly with virtualization/cloud platforms. So VMs could have DRBD-backed storage without additional overhead from NFS.
Via RADOS pools? Y.
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On 06/06/2016 11:02 AM, Yaniv Kaul wrote:
On Mon, Jun 6, 2016 at 7:23 PM, Hayley Swimelar <hayley@linbit.com> wrote:
On 06/04/2016 10:40 PM, Barak Korren wrote:
We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options:
My vote would be for native support rather then a Cinder based one. I don't have any hard numbers, but it seems to me the adoption of OpenStack technologies for use with oVirt is not as widespread as one may hope.
But what sort of user experience and system architecture is intended? Is configuration of volumes planned to be done directly from the oVirt UI?
Once a DRBD Manage storage cluster is set up, oVirt would handle the management of DRBD volumes.
While oVirt can do a little Gluster volume management for now, it
seems it was done because it some point it was thought that oVirt would be the primary management UI for Gluster. AFAIK Gluster is now getting its own UI.
The DRBD Manage daemon ships with a command line utility which is the UI for interacting with DRBD 9. I do not believe that there is a stand alone DRBD GUI planned at this time.
Is using the CLI considered a 'stable API' ? (or perhaps it can output a JSON or something that can be considered as such)?
oVirt wouldn't use the CLI, it would talk directly to the underlying DRBD Manage process (drbdmanaged) via D-Bus. As far as setting up the cluster initially, the DRBD Manage client (CLI) is designed to be used by the end user, so it's pretty user-friendly.
At least for now, the user most likely will prepare a DRBD Manage cluster up to the point where it's ready to start creating/assigning volumes and then oVirt will take over from there.
As far as using DRBD for backing oVirt VMs, this seems to already be
working. The oVirt CI system (which runs on top of oVirt) had been using a DRBD-backed NFS storage for years...
The most recent release of DRBD contains new features that allow it to integrate more directly with virtualization/cloud platforms. So VMs could have DRBD-backed storage without additional overhead from NFS.
Via RADOS pools?
The DRBD devices would be attached over the network, similar to the way they are in Cinder: https://www.drbd.org/en/doc/users-guide-90/s-openstack-transport-protocol
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212
But what sort of user experience and system architecture is intended? Is configuration of volumes planned to be done directly from the oVirt UI?
Once a DRBD Manage storage cluster is set up, oVirt would handle the management of DRBD volumes.
So does this mean there would be some kind of a "DRBD" cluster, like the "Gluseter cluster" oVirt has right now?
As far as using DRBD for backing oVirt VMs, this seems to already be working. The oVirt CI system (which runs on top of oVirt) had been using a DRBD-backed NFS storage for years...
The most recent release of DRBD contains new features that allow it to integrate more directly with virtualization/cloud platforms. So VMs could have DRBD-backed storage without additional overhead from NFS.
Could you elaborate a little on how this works? Would this means the block devices would be configured on the hypervisors themselves? Is this intended to be some kind of a hyper-converged architecture? -- Barak Korren bkorren@redhat.com RHEV-CI Team
On 06/06/2016 10:38 PM, Barak Korren wrote:
But what sort of user experience and system architecture is intended? Is configuration of volumes planned to be done directly from the oVirt UI?
Once a DRBD Manage storage cluster is set up, oVirt would handle the management of DRBD volumes.
So does this mean there would be some kind of a "DRBD" cluster, like the "Gluseter cluster" oVirt has right now?
Correct, there would be a cluster made up of nodes that contribute to the storage pool that is made available through DRBD. Host nodes would have to be in this cluster as well, but they would not necessarily have to have a large pool of their own storage.
As far as using DRBD for backing oVirt VMs, this seems to already be working. The oVirt CI system (which runs on top of oVirt) had been using a DRBD-backed NFS storage for years...
The most recent release of DRBD contains new features that allow it to integrate more directly with virtualization/cloud platforms. So VMs could have DRBD-backed storage without additional overhead from NFS.
Could you elaborate a little on how this works? Would this means the block devices would be configured on the hypervisors themselves? Is this intended to be some kind of a hyper-converged architecture?
The DRBD devices will appear on the hypervisors (/dev/drbd100/dev/drbd101, etc.) but writes and reads on those devices will be completed over the DRBD replication link. Unless that node has that volume locally, in which case it would use it's local storage like a standard DRBD device. -- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212
Maybe I'm being daft, but I'm missing something basic here. What's the end goal? A DRBD Storage Domain in oVirt? On Fri, Jun 3, 2016 at 9:13 PM, Hayley Swimelar <hayley@linbit.com> wrote:
Hello all,
A DRBD integration into oVirt is currently being planned.
DRBD provides replicated, highly available block storage. For those unfamiliar with DRBD more information can be found here: https://www.drbd.org/en/comp/drbd-linux-driver
DRBD 9 is the latest version of DRBD and interacts with DRBD Manage which is a daemon exposed via a D-Bus api. More information can be found here: https://www.drbd.org/en/comp/drbdmanage
DRBD Manage handles the creation, assignment, etc. of DRBD volumes and is the interface through which Cinder manages DRBD.
We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options:
1) Use Cinder. Although we are integrated into Cinder, there are concerns about the need to set up a complete Cinder instance to provide access to DRBD.
2) The alternative to Cinder would be to have the oVirt Engine communicate directly to DRBD Manage via D-Bus.
Any advice on the best way to achieve this would be much appreciated.
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On 06/13/2016 09:51 AM, Allon Mureinik wrote:
Maybe I'm being daft, but I'm missing something basic here. What's the end goal? A DRBD Storage Domain in oVirt?
Hi Allon, A DRBD Storage Domain in oVirt is the goal, yes.
On Fri, Jun 3, 2016 at 9:13 PM, Hayley Swimelar <hayley@linbit.com> wrote:
Hello all,
A DRBD integration into oVirt is currently being planned.
DRBD provides replicated, highly available block storage. For those unfamiliar with DRBD more information can be found here: https://www.drbd.org/en/comp/drbd-linux-driver
DRBD 9 is the latest version of DRBD and interacts with DRBD Manage which is a daemon exposed via a D-Bus api. More information can be found here: https://www.drbd.org/en/comp/drbdmanage
DRBD Manage handles the creation, assignment, etc. of DRBD volumes and is the interface through which Cinder manages DRBD.
We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options:
1) Use Cinder. Although we are integrated into Cinder, there are concerns about the need to set up a complete Cinder instance to provide access to DRBD.
2) The alternative to Cinder would be to have the oVirt Engine communicate directly to DRBD Manage via D-Bus.
Any advice on the best way to achieve this would be much appreciated.
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212
Then definitely don't go the Cinder route. We make WAY too many Ceph-centric assumptions there. I wonder how much effort it would be to implement a DRBD domain in a similar fashion to the existing block domains. On Mon, Jun 13, 2016 at 7:54 PM, Hayley Swimelar <hayley@linbit.com> wrote:
On 06/13/2016 09:51 AM, Allon Mureinik wrote:
Maybe I'm being daft, but I'm missing something basic here. What's the end goal? A DRBD Storage Domain in oVirt?
Hi Allon,
A DRBD Storage Domain in oVirt is the goal, yes.
On Fri, Jun 3, 2016 at 9:13 PM, Hayley Swimelar <hayley@linbit.com> wrote:
Hello all,
A DRBD integration into oVirt is currently being planned.
DRBD provides replicated, highly available block storage. For those unfamiliar with DRBD more information can be found here: https://www.drbd.org/en/comp/drbd-linux-driver
DRBD 9 is the latest version of DRBD and interacts with DRBD Manage which is a daemon exposed via a D-Bus api. More information can be found here: https://www.drbd.org/en/comp/drbdmanage
DRBD Manage handles the creation, assignment, etc. of DRBD volumes and is the interface through which Cinder manages DRBD.
We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options:
1) Use Cinder. Although we are integrated into Cinder, there are concerns about the need to set up a complete Cinder instance to provide access to DRBD.
2) The alternative to Cinder would be to have the oVirt Engine communicate directly to DRBD Manage via D-Bus.
Any advice on the best way to achieve this would be much appreciated.
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212
On 06/13/2016 10:00 AM, Allon Mureinik wrote:
Then definitely don't go the Cinder route. We make WAY too many Ceph-centric assumptions there.
I wonder how much effort it would be to implement a DRBD domain in a similar fashion to the existing block domains.
That was the original plan. The Cinder integration came up in an thread concerning interacting with VDSM directly. Here's the thread: http://lists.ovirt.org/pipermail/devel/2016-May/013062.html
On Mon, Jun 13, 2016 at 7:54 PM, Hayley Swimelar <hayley@linbit.com> wrote:
On 06/13/2016 09:51 AM, Allon Mureinik wrote:
Maybe I'm being daft, but I'm missing something basic here. What's the end goal? A DRBD Storage Domain in oVirt?
Hi Allon,
A DRBD Storage Domain in oVirt is the goal, yes.
On Fri, Jun 3, 2016 at 9:13 PM, Hayley Swimelar <hayley@linbit.com> wrote:
Hello all,
A DRBD integration into oVirt is currently being planned.
DRBD provides replicated, highly available block storage. For those unfamiliar with DRBD more information can be found here: https://www.drbd.org/en/comp/drbd-linux-driver
DRBD 9 is the latest version of DRBD and interacts with DRBD Manage which is a daemon exposed via a D-Bus api. More information can be found here: https://www.drbd.org/en/comp/drbdmanage
DRBD Manage handles the creation, assignment, etc. of DRBD volumes and is the interface through which Cinder manages DRBD.
We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options:
1) Use Cinder. Although we are integrated into Cinder, there are concerns about the need to set up a complete Cinder instance to provide access to DRBD.
2) The alternative to Cinder would be to have the oVirt Engine communicate directly to DRBD Manage via D-Bus.
Any advice on the best way to achieve this would be much appreciated.
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212
Hi Hayley. In the past I developed a POC for DRBD in oVirt. https://gerrit.ovirt.org/#/c/13585/ It's quite old and things have changed so for sure it's not up to date. It also took some assumptions that may different from what you want to achieve. In fact it assumed to be hyper-converged which, at that time, it meant that only data-centers of two nodes were supported (maybe DRBD changed in this respect now, I remember there were some plans around this circa 2014). The idea was to achieve HA in small deployment of few VMs (e.g. ROBO). On the other hand in case you want to separate compute and storage then you could deploy DRBD on two hosts (outside of the oVirt clusters) and expose the block device using iscsi (active/passive or active/active with multipath). I haven't checked what Cinder does, maybe it's something similar to what I described here above. Anyway that would work today out-of-the-box with no modifications to oVirt, but of course has some complexities in deployment, testing and maintenance (and hopefully a negligible performance penalty). I don't know what are the general directions nowadays but in the past (regardless of the architecture) I would have suggested to use VDSM as agent to setup the DRBD storage. (Anyway provisioning would come after you picked the architecture). Thanks, On Fri, Jun 3, 2016 at 8:13 PM, Hayley Swimelar <hayley@linbit.com> wrote:
Hello all,
A DRBD integration into oVirt is currently being planned.
DRBD provides replicated, highly available block storage. For those unfamiliar with DRBD more information can be found here: https://www.drbd.org/en/comp/drbd-linux-driver
DRBD 9 is the latest version of DRBD and interacts with DRBD Manage which is a daemon exposed via a D-Bus api. More information can be found here: https://www.drbd.org/en/comp/drbdmanage
DRBD Manage handles the creation, assignment, etc. of DRBD volumes and is the interface through which Cinder manages DRBD.
We are looking for advice on how best to have oVirt communicate with DRBD Manage and we have come to two options:
1) Use Cinder. Although we are integrated into Cinder, there are concerns about the need to set up a complete Cinder instance to provide access to DRBD.
2) The alternative to Cinder would be to have the oVirt Engine communicate directly to DRBD Manage via D-Bus.
Any advice on the best way to achieve this would be much appreciated.
-- Hayley Swimelar LINBIT | Keeping the Digital World Running DRBD — Corosync — Pacemaker +1-503-573-1262 x212 _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Federico
participants (5)
-
Allon Mureinik -
Barak Korren -
Federico Simoncelli -
Hayley Swimelar -
Yaniv Kaul