[Kimchi-devel] [PATCH 3/4] Host's software update: Update backend.
Paulo Ricardo Paz Vital
pvital at linux.vnet.ibm.com
Mon Feb 10 19:40:40 UTC 2014
Sheldon.
Please, do not consider my answer about your suggestion to use
utils.run_command().
I did some misunderstand with utils.parse_cmd_output() suggestion from
Aline. I'm so sorry!
V3 will use this!!!
--
Paulo Ricardo Paz Vital <pvital at linux.vnet.ibm.com>
IBM Linux Technology Center
On Mon, 2014-02-10 at 17:32 -0200, Paulo Ricardo Paz Vital wrote:
> Hello Sheldon,
>
> My answers about your comments inline:
>
> On Mon, 2014-02-10 at 16:57 +0800, Sheldon wrote:
> > Just a minor comment below
> >
> > On 02/10/2014 10:50 AM, Paulo Vital wrote:
> > > Update model and mockmodel to support backend opertions.
> > > Add new file implementing backend operations, with four new classes:
> > >
> > > 1) SoftwareUpdate (object): Class to represent and operate with OS software
> > > update system in Kimchi's perspective. It's agnostic to host's package management
> > > system, and can execute all operations necessary: get all packages to update,
> > > get information about one package and execute the update. This class will load
> > > in runtime the necessary classes to work with the host's package management:
> > > YumUpdate for YUM systems based, AptUpdate for APT systems based and ZypperUpdate
> > > for Zypper systems based.
> > >
> > > 2) YumUpdate (object): Class to represent and operate with YUM. Loaded only on
> > > those systems that supports YUM, it's responsible to connect and collect
> > > information of the packages to be updated. Also it's responsible to execute the
> > > update of the system.
> > >
> > > 3) AptUpdate (object): Class to represent and operate with APT. Loaded only on
> > > those systems that supports APT, it's responsible to connect and collect
> > > information of the packages to be updated. Also it's responsible to execute the
> > > update of the system.
> > >
> > > 4) ZypperUpdate (object): Class to represent and operate with Zypper. Loaded only
> > > on those systems that supports Zypper, it's responsible to connect and collect
> > > information of the packages to be updated. Also it's responsible to execute the
> > > update of the system.
> > >
> > > Signed-off-by: Paulo Vital <pvital at linux.vnet.ibm.com>
> > > Signed-off-by: Ramon Medeiros <ramonn at linux.vnet.ibm.com>
> > > ---
> > > src/kimchi/mockmodel.py | 46 +++++++++
> > > src/kimchi/model/host.py | 20 +++-
> > > src/kimchi/swupdate.py | 264 +++++++++++++++++++++++++++++++++++++++++++++++
> > > 3 files changed, 329 insertions(+), 1 deletion(-)
> > > create mode 100644 src/kimchi/swupdate.py
> > >
> > > diff --git a/src/kimchi/mockmodel.py b/src/kimchi/mockmodel.py
> > > index 4e276eb..235c72f 100644
> > > --- a/src/kimchi/mockmodel.py
> > > +++ b/src/kimchi/mockmodel.py
> > > @@ -75,6 +75,7 @@ class MockModel(object):
> > > self._mock_storagepools = {'default': MockStoragePool('default')}
> > > self._mock_networks = {'default': MockNetwork('default')}
> > > self._mock_interfaces = self.dummy_interfaces()
> > > + self._mock_swupdate = MockPackageUpdate()
> > > self.next_taskid = 1
> > > self.storagepool_activate('default')
> > >
> > > @@ -665,6 +666,16 @@ class MockModel(object):
> > > 'display_proxy_port':
> > > kconfig.get('display', 'display_proxy_port')}
> > >
> > > + def packageupdate_get_list(self):
> > > + return self._mock_swupdate.getUpdates()
> > > +
> > > + def packageupdate_lookup(self, pkg_name):
> > > + return self._mock_swupdate.getUpdate(pkg_name)
> > > +
> > > + def packageupdate_update(self, args=None):
> > > + kimchi_log.info('Host is going to be updated.')
> > > + self._mock_swupdate.doUpdate()
> > > +
> > >
> > > class MockVMTemplate(VMTemplate):
> > > def __init__(self, args, mockmodel_inst=None):
> > > @@ -825,6 +836,41 @@ class MockVMScreenshot(VMScreenshot):
> > > image.save(thumbnail)
> > >
> > >
> > > +class MockPackageUpdate(object):
> > > + def __init__(self):
> > > + self._packages = {'udevmountd': {'repo': 'openSUSE-13.1-Update',
> > > + 'version': '0.81.5-14.1',
> > > + 'arch': 'x86_64',
> > > + 'package_name': 'udevmountd'},
> > > + 'sysconfig-network': {'repo': 'openSUSE-13.1-Extras',
> > > + 'version': '0.81.5-14.1',
> > > + 'arch': 'x86_64',
> > > + 'package_name': 'sysconfig-network'},
> > > + 'libzypp': {'repo': 'openSUSE-13.1-Update',
> > > + 'version': '13.9.0-10.1',
> > > + 'arch': 'noarch',
> > > + 'package_name': 'libzypp'}}
> > > + self._num2update = 3
> > > +
> > > + def getUpdates(self):
> > > + return self._packages
> > > +
> > > + def getUpdate(self, name):
> > > + if not name in self._packages.keys():
> > > + raise NotFoundError("Package %s is not marked to be updated." %
> > > + name)
> > > + return self._packages[name]
> > > +
> > > + def getNumOfUpdates(self):
> > > + return self._num2update
> > > +
> > > + def doUpdate(self):
> > > + if self._num2update == 0:
> > > + kimchi_log.info("No packages marked for update")
> > > + raise OperationFailed("No packages marked for update")
> > > + sleep(60)
> > > +
> > > +
> > > def get_mock_environment():
> > > model = MockModel()
> > > for i in xrange(5):
> > > diff --git a/src/kimchi/model/host.py b/src/kimchi/model/host.py
> > > index a3d9e38..1bccd53 100644
> > > --- a/src/kimchi/model/host.py
> > > +++ b/src/kimchi/model/host.py
> > > @@ -30,10 +30,12 @@ from cherrypy.process.plugins import BackgroundTask
> > >
> > > from kimchi import disks
> > > from kimchi import netinfo
> > > +from kimchi import swupdate
> > > from kimchi.basemodel import Singleton
> > > from kimchi.exception import NotFoundError, OperationFailed
> > > +from kimchi.model.tasks import TaskModel
> > > from kimchi.model.vms import DOM_STATE_MAP
> > > -from kimchi.utils import kimchi_log
> > > +from kimchi.utils import add_task, kimchi_log
> > >
> > >
> > > HOST_STATS_INTERVAL = 1
> > > @@ -199,3 +201,19 @@ class PartitionModel(object):
> > > raise NotFoundError("Partition %s not found in the host"
> > > % name)
> > > return disks.get_partition_details(name)
> > > +
> > > +
> > > +class PackageUpdateModel(object):
> > > + def __init__(self, **kargs):
> > > + self.host_swupdate = swupdate.SoftwareUpdate()
> > > + self.objstore = kargs['objstore']
> > > + self.task = TaskModel(**kargs)
> > > +
> > > + def lookup(self, *name):
> > > + return self.host_swupdate.getUpdates()
> > > +
> > > + def update(self, **kargs):
> > > + kimchi_log.info('Host is going to be updated.')
> > > + gen_cmd = self.host_swupdate.doUpdate()
> > > + taskid = add_task('', gen_cmd, self.objstore, 'packageupdate')
> > > + return self.task.lookup(taskid)
> > > diff --git a/src/kimchi/swupdate.py b/src/kimchi/swupdate.py
> > > new file mode 100644
> > > index 0000000..5cda9c1
> > > --- /dev/null
> > > +++ b/src/kimchi/swupdate.py
> > > @@ -0,0 +1,264 @@
> > > +#
> > > +# Project Kimchi
> > > +#
> > > +# Copyright IBM, Corp. 2014
> > > +#
> > > +# Authors:
> > > +# Paulo Vital <pvital at linux.vnet.ibm.com>
> > > +# Ramon Medeiros <ramonn at linux.vnet.ibm.com>
> > > +#
> > > +# This library is free software; you can redistribute it and/or
> > > +# modify it under the terms of the GNU Lesser General Public
> > > +# License as published by the Free Software Foundation; either
> > > +# version 2.1 of the License, or (at your option) any later version.
> > > +#
> > > +# This library is distributed in the hope that it will be useful,
> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > > +# Lesser General Public License for more details.
> > > +#
> > > +# You should have received a copy of the GNU Lesser General Public
> > > +# License along with this library; if not, write to the Free Software
> > > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> > > +
> > > +import platform
> > > +import subprocess
> > > +
> > > +from kimchi.exception import NotFoundError, OperationFailed
> > > +from kimchi.utils import kimchi_log, run_command
> > > +
> > > +YUM_DISTROS = ['fedora', 'red hat enterprise linux',
> > > + 'red hat enterprise linux server']
> > > +APT_DISTROS = ['debian', 'ubuntu']
> > > +ZYPPER_DISTROS = ['opensuse ', 'suse linux enterprise server ']
> > > +
> > > +
> > > +class SoftwareUpdate(object):
> > > + """
> > > + Class to represent and operate with OS software update.
> > > + """
> > > + def __init__(self):
> > > + # This stores all packages to be updated for Kimchi perspective. It's a
> > > + # dictionary of dictionaries, in the format {'package_name': package},
> > > + # where:
> > > + # package = {'package_name': <string>, 'version': <string>,
> > > + # 'arch': <string>, 'repo': <string>
> > > + # }
> > > + self._packages = {}
> > > +
> > > + # This stores the number of packages to update
> > > + self._num2update = 0
> > > +
> > > + # Get the distro of host machine and creates an object related to
> > > + # correct package management system
> > > + self._distro = platform.linux_distribution()[0].lower()
> > > + if (self._distro in YUM_DISTROS):
> > > + kimchi_log.info("Loading YumUpdate features.")
> > > + self._pkg_mnger = YumUpdate()
> > > + elif (self._distro in APT_DISTROS):
> > > + kimchi_log.info("Loading AptUpdate features.")
> > > + self._pkg_mnger = AptUpdate()
> > > + elif (self._distro in ZYPPER_DISTROS):
> > > + kimchi_log.info("Loading ZypperUpdate features.")
> > > + self._pkg_mnger = ZypperUpdate()
> > > + else:
> > > + self._pkg_mnger = None
> > > +
> > > + if not self._pkg_mnger:
> > > + kimchi_log.info("There is no compatible package manager for \
> > > + this system.")
> > > +
> > > + def _scanUpdates(self):
> > > + """
> > > + Update self._packages with packages to be updated.
> > > + """
> > > + self._packages = {}
> > > + self._num2update = 0
> > > +
> > > + # Call system pkg_mnger to get the packages as list of dictionaries.
> > > + for pkg in self._pkg_mnger.getPackagesList():
> > > +
> > > + # Check if already exist a package in self._packages
> > > + pkg_id = pkg.get('package_name')
> > > + if pkg_id in self._packages.keys():
> > > + # package already listed to update. do nothing
> > > + continue
> > > +
> > > + # Update the self._packages and self._num2update
> > > + self._packages[pkg_id] = pkg
> > > + self._num2update = self._num2update + 1
> > > + print self._num2update
> > > + print self._packages
> > log?
>
> LOL! Actually it was a little debug print :-D
> Fix on V3.
>
> > > +
> > > + def getUpdates(self):
> > > + """
> > > + Return the self._packages.
> > > + """
> > > + self._scanUpdates()
> > > + return self._packages
> > > +
> > > + def getNumOfUpdates(self):
> > > + """
> > > + Return the number of packages to be updated.
> > > + """
> > > + self._scanUpdates()
> > > + return self._num2update
> > > +
> > > + def doUpdate(self):
> > > + """
> > > + Execute the update
> > > + """
> > > + if self._num2update == 0:
> > > + kimchi_log.info("No packages marked for update")
> > > + raise OperationFailed("No packages marked for update")
> > > + return self._pkg_mnger.update()
> > > +
> > > +
> > > +class YumUpdate(object):
> > > + """
> > > + Class to represent and operate with YUM software update system.
> > > + It's loaded only on those systems listed at YUM_DISTROS and loads necessary
> > > + modules in runtime.
> > > + """
> > > + def __init__(self):
> > > + self._pkgs = {}
> > > + self._yb = getattr(__import__('yum'), 'YumBase')()
> > > +
> > > + def _refreshUpdateList(self):
> > > + """
> > > + Update the list of packages to be updated in the system.
> > > + """
> > > + self._pkgs = self._yb.doPackageLists('updates')
> > > +
> > > + def getPackagesList(self):
> > > + """
> > > + Return a list of package's dictionaries. Each dictionary contains the
> > > + information about a package, in the format
> > > + package = {'package_name': <string>, 'version': <string>,
> > > + 'arch': <string>, 'repo': <string>
> > > + }
> > indent?
> >
>
> Thanks! Fix on V3.
>
> > > + """
> > > + self._refreshUpdateList()
> > > + pkg_list = []
> > > + for pkg in self._pkgs:
> > > + package = {'package_name': pkg.name,
> > > + 'version': "%s-%s" % (pkg.version, pkg.release),
> > > + 'arch': pkg.arch, 'repo': pkg.ui_from_repo}
> > > + pkg_list.append(package)
> > > + return pkg_list
> > > +
> > > + def update(self):
> > > + """
> > > + Execute the update of all packages marked to be update.
> > > + """
> > > + # FIXME: Due to incompatabilities between cherrypy and yum/sqlite3
> > > + # threading, we need execute the YUM command line to execute the update
> > what incompatabilities?
>
> The builtin yum module executes some sqlite queries to search RPM's
> database in a different thread, and at the moment to execute one of
> builtin yum module methods to execute the update, it breaks complaining
> about the thread which is calling this method is not the correct.
>
> Looks like the class responsible to execute this queries prevents that a
> different thread than thread running/importing the YumBase class execute
> the sqlite queries. Doing a basic debug, I could check that the CherryPy
> thread is the main thread responsible to execute this queries - instead
> of the thread running kimchi.server module.
>
> So, once the logic to update the packages using the builtin yum methods
> is the same of shell command "yum -y -d 0 -e 0 update", I decided to use
> this command until I debug better the problem and find an "elegant"
> solution.
>
> > > + cmd = ["yum", "-y", "-d", "0", "-e", "0", "update"]
> > > + (stdout, stderr, returncode) = run_command(cmd)
> > > +
> > > + if len(stderr) > 0:
> > > + raise OperationFailed("ERROR when executing command: %s" % stderr)
> > > +
> > > +
> > > +class AptUpdate(object):
> > > + """
> > > + Class to represent and operate with APT software update system.
> > > + It's loaded only on those systems listed at APT_DISTROS and loads necessary
> > > + modules in runtime.
> > > + """
> > > + def __init__(self):
> > > + self._pkgs = {}
> > > + self._apt_cache = getattr(__import__('apt'), 'Cache')()
> > > +
> > > + def _refreshUpdateList(self):
> > > + """
> > > + Update the list of packages to be updated in the system.
> > > + """
> > > + self._apt_cache.update()
> > > + self._apt_cache.upgrade()
> > > + self._pkgs = self._apt_cache.get_changes()
> > > +
> > > + def getPackagesList(self):
> > > + """
> > > + Return a list of package's dictionaries. Each dictionary contains the
> > > + information about a package, in the format
> > > + package = {'package_name': <string>, 'version': <string>,
> > > + 'arch': <string>, 'repo': <string>
> > > + }
> > indent?
>
> Thanks! Fix on V3.
>
> > > + """
> > > + self._refreshUpdateList()
> > > + pkg_list = []
> > > + for pkg in self._pkgs:
> > > + package = {'package_name': pkg.shortname,
> > > + 'version': pkg.candidate.version,
> > > + 'arch': pkg.architecture(),
> > > + 'repo': pkg.candidate.origins[0].label}
> > > + pkg_list.append(package)
> > > + return pkg_list
> > > +
> > > + def update(self):
> > > + """
> > > + Execute the update of all packages marked to be update.
> > > + """
> > > + try:
> > > + self._apt_cache.update()
> > > + self._apt_cache.open(None)
> > > + self._apt_cache.upgrade()
> > > + self._apt_cache.commit()
> > > + except Exception, e:
> > > + raise OperationFailed("ERROR when executing command: %s" % e)
> > > +
> > > +
> > > +class ZypperUpdate(object):
> > > + """
> > > + Class to represent and operate with Zypper software update system.
> > > + It's loaded only on those systems listed at ZYPPER_DISTROS and loads
> > > + necessary modules in runtime.
> > > + """
> > > + def __init__(self):
> > > + self._pkgs = {}
> > > +
> > > + def _refreshUpdateList(self):
> > > + """
> > > + Update the list of packages to be updated in the system.
> > > + """
> > > + self._pkgs = {}
> > > + cmd = ["zypper", "list-updates"]
> > > + (stdout, stderr, returncode) = run_command(cmd)
> > > +
> > > + if len(stderr) > 0:
> > > + raise OperationFailed("ERROR when executing command: %s" % stderr)
> > > +
> > > + for line in stdout.split('\n'):
> > > + if line.find('v |') >= 0:
> > what does "v" means ?
>
> It's related to the first column of the output. I understood that is
> about the status of the package in the system. All packages with "v" has
> update candidates to be installed. The output of this command, only
> prints packages with update candidates.
>
> > > + info = line.split(' | ')
> > > + package = {'package_name': info[2], 'version': info[4],
> > > + 'arch': info[5], 'repo': info[1]}
> > > + self._pkgs[info[2]] = package
> > > +
> > > + def getPackagesList(self):
> > > + """
> > > + Return a list of package's dictionaries. Each dictionary contains the
> > > + information about a package, in the format
> > > + package = {'package_name': <string>, 'version': <string>,
> > > + 'arch': <string>, 'repo': <string>
> > > + }
> > > + """
> > > + self._refreshUpdateList()
> > > + pkg_list = []
> > > + for pkg in self._pkgs:
> > > + pkg_list.append(pkg)
> > > + return pkg_list
> > > +
> > > + def update(self):
> > > + """
> > > + Execute the update of all packages marked to be update.
> > > + """
> > > + cmd = ["zypper", "--non-interactive", "update",
> > > + "--auto-agree-with-licenses"]
> > > + proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
> > > + stderr=subprocess.PIPE)
> > > + stdout, stderr = proc.communicate()
> > why not?
> > stdout, stderr, returncode = run_command(cmd)
> > if len(stderr) > 0:
>
> I explained this in feedback from Aline's comments in the V1 patch set.
>
> > > +
> > > + if len(stderr) > 0:
> > > + raise OperationFailed("ERROR when executing command: %s" % stderr)
> >
> >
>
> _______________________________________________
> Kimchi-devel mailing list
> Kimchi-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>
More information about the Kimchi-devel
mailing list