[Kimchi-devel] [PATCH 3/4] Host's software update: Update backend.
Paulo Ricardo Paz Vital
pvital at linux.vnet.ibm.com
Thu Jan 30 19:10:10 UTC 2014
Aline,
It's not possible to implement your suggestion to use Royce's
parse_cmd_output() function.
Royce's function returns a list of dictionaries using as key the n-items
of a list expected by me and as values the exactly n-words of each line
of the output, in the correct sequence. For example, if I have a file
with the content:
"bla ble bli blo blu
cla cle cli
dla dlu dlo dle"
and I use Royce's fucntion, like:
output_items = ['a', 'e', 'i'] # expect the items
res = parse_cmd_output(stdout, output_items)
the res will be
[{'a': 'bla', 'i': 'bli', 'e': 'ble'},
{'a': 'cla', 'i': 'cli', 'e': 'cle'},
{'a': 'dla', 'i': 'dlo', 'e': 'dlu'}]
My output is very volatile, I don't know the name of the packages to be
updated, even, I don't know if some package will be updated, so I can't
"search" by a specific field. Also, my output has a lot of "trash" to
execute the function, and by it's design I should pass as argument a hug
list of expected items to use less than 50% of them.
Would be interesting improve parse_cmd_output() by splitting for a
specific delimiter (passed as argument) instead of "only" one space,
and/or, returning the dictionary's values with specific fields instead
of the sequential values by now.
Looking all other comments you did, I'm implementing them and send the
V2 ASAP.
Thanks and best regards,
--
Paulo Ricardo Paz Vital <pvital at linux.vnet.ibm.com>
IBM Linux Technology Center
> On 01/27/2014 02:34 PM, Paulo Vital wrote:
> > Update model and mockmodel to support backend opertions.
> > Add new file implementing backend operations, with four new classes:
> >
> > 1) SoftwareUpdate (object): Class to represent and operate with OS software
> > update system in Kimchi's perspective. It's agnostic to host's package management
> > system, and can execute all operations necessary: get all packages to update,
> > get information about one package and execute the update. This class will load
> > in runtime the necessary classes to work with the host's package management:
> > YumUpdate for YUM systems based, AptUpdate for APT systems based and ZypperUpdate
> > for Zypper systems based.
> >
> > 2) YumUpdate (object): Class to represent and operate with YUM. Loaded only on
> > those systems that supports YUM, it's responsible to connect and collect
> > information of the packages to be updated. Also it's responsible to execute the
> > update of the system.
> >
> > 3) AptUpdate (object): Class to represent and operate with APT. Loaded only on
> > those systems that supports APT, it's responsible to connect and collect
> > information of the packages to be updated. Also it's responsible to execute the
> > update of the system.
> >
> > 4) ZypperUpdate (object): Class to represent and operate with Zypper. Loaded only
> > on those systems that supports Zypper, it's responsible to connect and collect
> > information of the packages to be updated. Also it's responsible to execute the
> > update of the system.
> >
> > Signed-off-by: Paulo Vital <pvital at linux.vnet.ibm.com>
> > Signed-off-by: Ramon Medeiros <ramonn at linux.vnet.ibm.com>
> > ---
> > src/kimchi/mockmodel.py | 12 +++
> > src/kimchi/model.py | 8 ++
> > src/kimchi/swupdate.py | 279 ++++++++++++++++++++++++++++++++++++++++++++++++
> > 3 files changed, 299 insertions(+)
> > create mode 100644 src/kimchi/swupdate.py
> >
> > diff --git a/src/kimchi/mockmodel.py b/src/kimchi/mockmodel.py
> > index 916020a..a276021 100644
> > --- a/src/kimchi/mockmodel.py
> > +++ b/src/kimchi/mockmodel.py
> > @@ -44,6 +44,7 @@ except ImportError:
> > import kimchi.model
> > from kimchi import config
> > from kimchi import network as knetwork
> > +from kimchi import swupdate
> > from kimchi.asynctask import AsyncTask
> > from kimchi.distroloader import DistroLoader
> > from kimchi.exception import InvalidOperation, InvalidParameter
> > @@ -75,6 +76,7 @@ class MockModel(object):
> > self._mock_interfaces = self.dummy_interfaces()
> > self.next_taskid = 1
> > self.storagepool_activate('default')
> > + self.host_swupdate = swupdate.SoftwareUpdate()
> >
> > def _static_vm_update(self, dom, params):
> > state = dom.info['state']
> > @@ -658,6 +660,16 @@ class MockModel(object):
> > % name)
> > return disks.get_partition_details(name)
>
> > + def swupdate_get_list(self):
> > + return [pkg for pkg in self.host_swupdate.getUpdates().keys()]
> > +
> > + def swupdate_lookup(self, name):
> > + return self.host_swupdate.getUpdate(name)
> > +
>
> From previous patch SoftwareUpdate is a Resource (class
> SoftwareUpdate(Resource))
> And there is no Collection for it.
>
> From that you should only implement lookup() function.
>
> Also controller will look for a method named softwareupdate_lookup()
> instead of swupdate_lookup()
> As you named the Resource as SoftwareUpdate
>
> From my understanding, you need to have a Collection and Resource.
> The Collection will return all packages names to be update and have the
> action "update"
>
> And the Resource will get the package information (name, arch and all
> else you added in API.md)
>
> I'd suggest to change the uri from SoftwareUpdate to PackageUpdate as it
> is all related to packages.
>
> So we will have:
>
> GET /packageupdate
> [{name: pkg1, arch: ...}, {name: pkg2, arch:...}, {name: pkg3, arch: ...}]
>
> GET /packageupdate/pkg1
> {name: pkg1, arch:...}
>
> POST /packageupdate/update
> # do the update in background and return a Task element as the update
> can take long time to be done
>
> The Task element is needed in order to UI know what is happening in
> background and show it to the user
> You can check the DebugReports implementation which uses the same mechanism.
>
>
> Use curl command to test it
> curl -u <user:password> -H "Content-Type: application/json" -H "Accept:
> application/json" http://localhost:8000/<uri>
>
> That way you can make sure the uris are working as expected
>
> > + def swupdate_update(self):
> > + kimchi_log.info('Host is going to be updated.')
> > + self.host_swupdate.doUpdate()
> > +
> > class MockVMTemplate(VMTemplate):
> > def __init__(self, args, mockmodel_inst=None):
> > VMTemplate.__init__(self, args)
> > diff --git a/src/kimchi/model.py b/src/kimchi/model.py
> > index 81c1507..d8baa8c 100644
> > --- a/src/kimchi/model.py
> > +++ b/src/kimchi/model.py
> > @@ -63,6 +63,7 @@ from kimchi import config
> > from kimchi import netinfo
> > from kimchi import network as knetwork
> > from kimchi import networkxml
> > +from kimchi import swupdate
> > from kimchi import vnc
> > from kimchi import xmlutils
> > from kimchi.asynctask import AsyncTask
> > @@ -140,6 +141,7 @@ class Model(object):
> > self.stats = {}
> > self.host_stats = defaultdict(int)
> > self.host_info = {}
>
> > + self.host_swupdate = swupdate.SoftwareUpdate()
> > self.qemu_stream = False
> > self.qemu_stream_dns = False
> > self.libvirt_stream_protocols = []
> > @@ -1583,6 +1585,12 @@ class Model(object):
> > kimchi_log.info('Host is going to reboot.')
> > os.system('reboot')
> >
> > + def softwareupdate_lookup(self, *name):
> > + return self.host_swupdate.getUpdates()
> > +
> > + def softwareupdate_update(self, args=None):
> > + kimchi_log.info('Host is going to be updated.')
> > + self.host_swupdate.doUpdate()
>
> The mockmodel is a fake model.
> Which means we should not touch the system.
> You need to return default values there.
>
> > class LibvirtVMTemplate(VMTemplate):
> > def __init__(self, args, scan=False, conn=None):
> > diff --git a/src/kimchi/swupdate.py b/src/kimchi/swupdate.py
> > new file mode 100644
> > index 0000000..5f0cf84
> > --- /dev/null
> > +++ b/src/kimchi/swupdate.py
> > @@ -0,0 +1,279 @@
> > +#
> > +# Project Kimchi
> > +#
> > +# Copyright IBM, Corp. 2014
> > +#
> > +# Authors:
> > +# Paulo Vital <pvital at linux.vnet.ibm.com>
> > +# Ramon Medeiros <ramonn at linux.vnet.ibm.com>
> > +#
> > +# This library is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU Lesser General Public
> > +# License as published by the Free Software Foundation; either
> > +# version 2.1 of the License, or (at your option) any later version.
> > +#
> > +# This library is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > +# Lesser General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU Lesser General Public
> > +# License along with this library; if not, write to the Free Software
> > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> > +
> > +import platform
> > +import subprocess
> > +
> > +from kimchi.exception import *
>
> Don't use * in imports
> Explicit add only what you will use from the module
>
> > +from kimchi.utils import kimchi_log
> > +
> > +YUM_DISTROS = [ 'fedora', 'red hat enterprise linux',
> > + 'red hat enterprise linux server']
> > +APT_DISTROS = [ 'debian', 'ubuntu']
> > +ZYPPER_DISTROS = [ 'opensuse ' ]
> > +
> > +class SoftwareUpdate(object):
> > + """
> > + Class to represent and operate with OS software update.
> > + """
> > + def __init__(self):
> > + # This stores all packages to be updated for Kimchi perspective. It's a
> > + # dictionary of dictionaries, in the format {'package_name': package},
> > + # where:
> > + # package = {'package_name': <string>, 'version': <string>,
> > + # 'arch': <string>, 'repo': <string>
> > + # }
> > + self._packages = {}
> > +
> > + # This stores the number of packages to update
> > + self._num2update = 0
> > +
> > + # Get the distro of host machine and creates an object related to
> > + # correct package management system
> > + self._distro = platform.linux_distribution()[0].lower()
> > + if (self._distro in YUM_DISTROS):
> > + kimchi_log.info("Loading YumUpdate features.")
> > + self._pkg_mnger = YumUpdate()
> > + elif (self._distro in APT_DISTROS):
> > + kimchi_log.info("Loading AptUpdate features.")
> > + self._pkg_mnger = AptUpdate()
> > + elif (self._distro in ZYPPER_DISTROS):
> > + kimchi_log.info("Loading ZypperUpdate features.")
> > + self._pkg_mnger = ZypperUpdate()
> > + else:
> > + self._pkg_mnger = None
> > +
> > + if not self._pkg_mnger:
> > + kimchi_log.info("There is no compatible package manager for \
> > + this system.")
>
> Just a log?
> What happen when I try to update the packages with a self._pkg_mnger ==
> None ?
>
> We need to expose that info to UI to disable the "Update" button when no
> package manager is recognized.
> And also check that info before doing the update to avoid bad REST
> requesters
>
>
> > +
> > + def _scanUpdates(self):
> > + """
> > + Update self._packages with packages to be updated.
> > + """
> > + self._packages = {}
> > + self._num2update = 0
> > +
> > + # Call system pkg_mnger to get the packages as list of dictionaries.
> > + for pkg in self._pkg_mnger.getPackagesList():
> > +
> > + # Check if already exist a package in self._packages
> > + pkg_id = pkg.get('package_name')
> > + if pkg_id in self._packages.keys():
> > + # package already listed to update. do nothing
> > + continue
> > +
> > + # Update the self._packages and self._num2update
> > + self._packages[pkg_id] = pkg
> > + self._num2update = self._num2update + 1
> > +
> > + def getUpdates(self):
> > + """
> > + Return the self._packages.
> > + """
> > + self._scanUpdates()
> > + return self._packages
> > +
> > + def getUpdate(self, name):
> > + """
> > + Return a dictionary with all info from a given package name.
> > + """
> > + if not name in self._packages.keys():
> > + raise NotFoundError("Package %s is not marked to be updated." % name)
> > +
> > + return self._packages[name]
> > +
> > +
> > + def getNumOfUpdates(self):
> > + """
> > + Return the number of packages to be updated.
> > + """
> > + self._scanUpdates()
> > + return self._num2update
> > +
> > + def doUpdate(self):
> > + """
> > + Execute the update
> > + """
> > + if self._num2update == 0:
> > + kimchi_log.info("No packages marked for update")
> > + raise OperationFailed("No packages marked for update")
> > + return self._pkg_mnger.update()
> > +
> > +class YumUpdate(object):
> > + """
> > + Class to represent and operate with YUM software update system.
> > + It's loaded only on those systems listed at YUM_DISTROS and loads necessary
> > + modules in runtime.
> > + """
> > + def __init__(self):
> > + self._pkgs = {}
> > + self._yb = getattr(__import__('yum'),'YumBase')()
> > +
> > + def _refreshUpdateList(self):
> > + """
> > + Update the list of packages to be updated in the system.
> > + """
> > + self._pkgs = self._yb.doPackageLists('updates')
> > +
> > + def getPackagesList(self):
> > + """
> > + Return a list of package's dictionaries. Each dictionary contains the
> > + information about a package, in the format
> > + package = {'package_name': <string>, 'version': <string>,
> > + 'arch': <string>, 'repo': <string>
> > + }
> > + """
> > + self._refreshUpdateList()
> > + pkg_list = []
> > + for pkg in self._pkgs:
> > + package = { 'package_name': pkg.name,
> > + 'version': "%s-%s" % (pkg.version, pkg.release),
> > + 'arch': pkg.arch, 'repo': pkg.ui_from_repo
> > + }
> > + pkg_list.append(package)
> > + return pkg_list
> > +
> > + def update(self):
> > + """
> > + Execute the update of all packages marked to be update.
> > + """
> > + # FIXME: Due to incompatabilities between cherrypy and yum/sqlite3
> > + # threading, we need execute the YUM command line to execute the update
>
> What is the error when using the yum python binding here?
> Did you open an issue for it?
>
> > + cmd = [ "yum", "-y", "-d", "0", "-e", "0", "update" ]
> > + proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
> > + stderr=subprocess.PIPE)
> > + stdout, stderr = proc.communicate()
> > +
> > + if len(stderr) > 0:
> > + raise OperationFailed("ERROR when executing command: %s" % stderr)
> > +
> > +class AptUpdate(object):
> > + """
> > + Class to represent and operate with APT software update system.
> > + It's loaded only on those systems listed at APT_DISTROS and loads necessary
> > + modules in runtime.
> > + """
> > + def __init__(self):
> > + self._pkgs = {}
> > + self._apt_cache = getattr(__import__('apt'),'Cache')()
> > +
> > + def _refreshUpdateList(self):
> > + """
> > + Update the list of packages to be updated in the system.
> > + """
> > + self._apt_cache.update()
> > + self._apt_cache.upgrade()
> > + self._pkgs = self._apt_cache.get_changes()
> > +
> > + def getPackagesList(self):
> > + """
> > + Return a list of package's dictionaries. Each dictionary contains the
> > + information about a package, in the format
> > + package = {'package_name': <string>, 'version': <string>,
> > + 'arch': <string>, 'repo': <string>
> > + }
> > + """
> > + self._refreshUpdateList()
> > + pkg_list = []
> > + for pkg in self._pkgs:
> > + package = { 'package_name': pkg.shortname,
> > + 'version': pkg.candidate.version,
> > + 'arch': pkg.architecture(),
> > + 'repo': pkg.candidate.origins[0].label
> > + }
> > + pkg_list.append(package)
> > + return pkg_list
> > +
> > + def update(self):
> > + """
> > + Execute the update of all packages marked to be update.
> > + """
> > + try:
> > + self._apt_cache.update()
> > + self._apt_cache.open(None)
> > + self._apt_cache.upgrade()
> > + self._apt_cache.commit()
> > + except Exception, e:
> > + raise OperationFailed("ERROR when executing command: %s" % e)
> > +
> > +class ZypperUpdate(object):
> > + """
> > + Class to represent and operate with Zypper software update system.
> > + It's loaded only on those systems listed at ZYPPER_DISTROS and loads
> > + necessary modules in runtime.
> > + """
> > + def __init__(self):
> > + self._pkgs = {}
> > +
> > + def _refreshUpdateList(self):
> > + """
> > + Update the list of packages to be updated in the system.
> > + """
> > + self._pkgs = {}
> > + cmd = ["zypper", "list-updates"]
>
> > + proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
> > + stderr=subprocess.PIPE)
> > + stdout, stderr = proc.communicate()
> > +
> > + if len(stderr) > 0:
> > + raise OperationFailed("ERROR when executing command: %s" % stderr)
> > +
> > + lines = stdout.split('\n')
> > + for line in lines:
>
> Royce, add a utility to parse command output.
>
> parse_cmd_output() in src/kimchi/utils.py
> It will return a dict for you with the elements you need.
>
>
> > + if line.find('v |') >= 0:
> > + info = line.split(' | ')
> > + package = { 'package_name': info[2], 'version': info[4],
> > + 'arch': info[5], 'repo': info[1]
> > + }
> > + self._pkgs[info[2]] = package
> > +
> > + def getPackagesList(self):
> > + """
> > + Return a list of package's dictionaries. Each dictionary contains the
> > + information about a package, in the format
> > + package = {'package_name': <string>, 'version': <string>,
> > + 'arch': <string>, 'repo': <string>
> > + }
> > + """
> > + self._refreshUpdateList()
> > + pkg_list = []
> > + for pkg in self._pkgs:
> > + pkg_list.append(pkg)
> > + return pkg_list
> > +
> > + def update(self):
> > + """
> > + Execute the update of all packages marked to be update.
> > + """
> > + cmd = [ "zypper", "--non-interactive", "update",
> > + "--auto-agree-with-licenses"
> > + ]
> > + proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
> > + stderr=subprocess.PIPE)
> > + stdout, stderr = proc.communicate()
> > +
> > + if len(stderr) > 0:
> > + raise OperationFailed("ERROR when executing command: %s" % stderr)
> > +
> > +
>
More information about the Kimchi-devel
mailing list