Planet GLLUG

Syndicate content
Planet GLLUG - http://planet.gllug.org.uk
Updated: 1 year 13 weeks ago

Dean Wilson: Facter 1.7+ and External facts

Wed, 15/05/2013 - 21:46
While Puppet may get all the glory, Facter, the hard working information gathering library that can, seldom gets much exciting new functionality. However with the release of Facter 1.7 Puppetlabs have standardised and included a couple of useful facter enhancements that make it easier than ever to add custom facts to your puppet runs.

These two improvements come under the banner of 'External Facts'. The first allows you to surface your own facts from a static file, either plain text key value pairs or a specific YAML / JSON format. These static files should be placed under /etc/facter/facts.d

$ sudo mkdir -p /etc/facter/facts.d # note - the .txt file extension $ echo 'external_fact=yes' | sudo tee /etc/facter/facts.d/external_test.txt external_fact=worked $ facter external_fact worked

At its simplest this is a way to surface basic, static, details from system provisioning and other similar large events but it's also an easy way to include details from other daemon and cronjobs. One of my first use cases for this was to create 'last_backup_time' and 'last_backup_status' facts that are written at the conclusion of my backup cronjob. Having the values inserted from out of band is a much nicer prospect that writing a custom fact that parses the cron logs.

If that's a little too static for you then the second usage might be what you're looking for. Any executable scripts dropped in the same directory that produce the same output formats as allowed above will be executed by facter when it's invoked.

# scripts must be executable! $ sudo chmod a+rx /etc/facter/facts.d/process_count $ cat /etc/facter/facts.d/process_count #!/bin/bash count=$(ps -efwww | wc -l | tr -s ' ') echo "process_count=$count" $ facter process_count 209

The ability to run scripts that provide facts and values makes customisation easier in situations where ruby isn't the best language for the job. It's also a nice way to reuse existing tools or for including information from further afield - such as the current binary log in use by MySQL or Postgres or the hosts current state in the load balancer.

While there have been third party extensions that provided this functionality for a while it's great to see these enhancements get included in core facter.

Categories: LUG Community Blogs

Richard WM Jones: rich

Mon, 13/05/2013 - 11:53

You can use qcow2 backing files as a convenient way to test what happens when you try to create exabyte-sized filesystems. Just to remind you, 1 exabyte is a million terabytes, or a pile of ordinary hard disks stacked 8 miles high.

There is a bug in qemu that prevents you from creating very large disks unless you adjust the cluster_size option (thanks Kevin Wolf):

$ qemu-img create -f qcow2 huge.qcow2 \ $((1024*1024))T -o cluster_size=2M Formatting 'huge.qcow2', fmt=qcow2 size=1152921504606846976 encryption=off cluster_size=2097152 lazy_refcounts=off

After that you can just attach the disk to guestfish and start playing with huge filesystems.

[I should note that virt-rescue is probably a better choice of tool here, especially for people who need to experiment with unusual filesystem or LVM options]

$ guestfish -a huge.qcow2 Welcome to guestfish, the guest filesystem shell for editing virtual machine filesystems and disk images. Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell ><fs> run ><fs> blockdev-getsize64 /dev/sda 1152921504606846976 ><fs> part-disk /dev/sda gpt

Ext4 (according to Wikipedia) is supposed to support 1 exabyte disks, but I couldn’t get that to work, possibly because there was not enough RAM:

><fs> mkfs ext4 /dev/sda1 libguestfs: error: mkfs: ext4: /dev/sda1: mke2fs 1.42.5 (29-Jul-2012) /dev/sda1: Not enough space to build proposed filesystem while setting up superblock

XFS could create a filesystem, but I didn’t let it run to completion because it would need about 5 petabytes to store the filesystem metadata:

><fs> mkfs xfs /dev/sda1 [ disks churn for many minutes while qcow2 file grows and grows and grows ... ]

LVM2 PVs are possible, but creating a VG requires us to adjust the extent size:

><fs> pvcreate /dev/sda1 ><fs> vgcreate VG /dev/sda1 libguestfs: error: vgcreate: PV /dev/sda1 too large for extent size 4.00 MiB. Format-specific setup of physical volume '/dev/sda1' failed. Unable to add physical volume '/dev/sda1' to volume group 'VG'. ><fs> debug sh "vgcreate -s 1G VG /dev/sda1" Volume group "VG" successfully created ><fs> lvcreate LV VG 1000000000 ><fs> lvs-full [0] = { lv_name: LV [...] lv_size: 1048576536870912 }

Previously …


Categories: LUG Community Blogs

Richard WM Jones: rich

Sat, 11/05/2013 - 18:49

New in libguestfs upstream and 1.21.39 is the ability to access disks over FTP, FTPS, HTTP, HTTPS and TFTP (read-only).

You can use it like this:

$ export LIBGUESTFS_BACKEND=direct $ guestfish --ro -a http://x.x.x.x/scratch/winxp.img -i Welcome to guestfish, the guest filesystem shell for editing virtual machine filesystems and disk images. Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell Operating system: Microsoft Windows XP /dev/sda1 mounted on / ><fs> ll / total 1573209 drwxrwxrwx 1 root root 4096 Apr 16 2012 . drwxr-xr-x 23 1000 1000 4096 May 11 18:45 .. -rwxrwxrwx 1 root root 0 Oct 11 2011 AUTOEXEC.BAT -rwxrwxrwx 1 root root 0 Oct 11 2011 CONFIG.SYS drwxrwxrwx 1 root root 4096 Oct 11 2011 Documents and Settings -rwxrwxrwx 1 root root 0 Oct 11 2011 IO.SYS -rwxrwxrwx 1 root root 0 Oct 11 2011 MSDOS.SYS -rwxrwxrwx 1 root root 47564 Apr 14 2008 NTDETECT.COM drwxrwxrwx 1 root root 4096 Oct 11 2011 Program Files drwxrwxrwx 1 root root 4096 Oct 11 2011 System Volume Information drwxrwxrwx 1 root root 28672 Oct 11 2011 WINDOWS -rwxrwxrwx 1 root root 211 Oct 11 2011 boot.ini -rwxrwxrwx 1 root root 250048 Apr 14 2008 ntldr -rwxrwxrwx 1 root root 1610612736 Oct 11 2011 pagefile.sys

Apart from being a tiny bit slower, it just works as if the disk was local.


Categories: LUG Community Blogs

Richard WM Jones: rich

Sat, 11/05/2013 - 17:19

In libguestfs ≥ 1.21.38 you can access at least some iSCSI disks.

On my server (RHEL 6 in this case) I create an iSCSI target backed by a Windows XP disk image:

# service tgtd start Starting SCSI target daemon: [ OK ] # tgtadm --lld iscsi --op new --mode target --tid 1 \ -T iqn.1994-05.com.redhat # chcon system_u:object_r:tgtd_var_lib_t:s0 /tmp/winxp.img # tgtadm --lld iscsi --op new --mode logicalunit --tid 1 \ --lun 1 -b /tmp/winxp.img # tgt-admin -s ...

Previously I opened port 3250 on the server. Because libguestfs doesn’t yet support authentication against the iSCSI server, I had to bypass that:

# tgtadm --lld iscsi --mode target --op bind --tid 1 -I ALL

Now on the client, I can connect to the iSCSI target using libguestfs like this:

$ export LIBGUESTFS_BACKEND=direct $ guestfish --format=raw -a iscsi://x.x.x.x/iqn.1994-05.com.redhat/1 -i Welcome to guestfish, the guest filesystem shell for editing virtual machine filesystems and disk images. Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell Operating system: Microsoft Windows XP /dev/sda1 mounted on / ><fs> ll / total 1573209 drwxrwxrwx 1 root root 4096 Apr 16 2012 . drwxr-xr-x 23 1000 1000 4096 May 11 17:16 .. -rwxrwxrwx 1 root root 0 Oct 11 2011 AUTOEXEC.BAT -rwxrwxrwx 1 root root 0 Oct 11 2011 CONFIG.SYS drwxrwxrwx 1 root root 4096 Oct 11 2011 Documents and Settings -rwxrwxrwx 1 root root 0 Oct 11 2011 IO.SYS -rwxrwxrwx 1 root root 0 Oct 11 2011 MSDOS.SYS -rwxrwxrwx 1 root root 47564 Apr 14 2008 NTDETECT.COM drwxrwxrwx 1 root root 4096 Oct 11 2011 Program Files drwxrwxrwx 1 root root 4096 Oct 11 2011 System Volume Information drwxrwxrwx 1 root root 28672 Oct 11 2011 WINDOWS -rwxrwxrwx 1 root root 211 Oct 11 2011 boot.ini -rwxrwxrwx 1 root root 250048 Apr 14 2008 ntldr -rwxrwxrwx 1 root root 1610612736 Oct 11 2011 pagefile.sys
Categories: LUG Community Blogs

Richard WM Jones: rich

Fri, 10/05/2013 - 16:18

(Thanks to Jeff Cody for both writing the code and helping me to get it to work).

New in qemu 1.5 is read-only support for Hyper-V‘s native disk format, vhdx. You can now open vhdx files in libguestfs just like any other:

$ guestfish --ro --format=vhdx -a /tmp/f18x64.vhdx -i Welcome to guestfish, the guest filesystem shell for editing virtual machine filesystems and disk images. Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell Operating system: Fedora release 18 (Spherical Cow) /dev/fedora/root mounted on / /dev/sda1 mounted on /boot

Notes:

  1. As stated above, you will need the very latest qemu for this to work.
  2. You must specify --format=vhdx at the moment because of a missing feature in libvirt.
  3. You must specify --ro because writing is not supported (by the qemu driver).

Categories: LUG Community Blogs

Richard WM Jones: rich

Thu, 09/05/2013 - 15:11

Situation: You have a Windows DVD (or ISO), but like any sane person in 2013 you don’t have a DVD drive on the computer. You want to convert the Windows DVD into a bootable USB key. There are many recipes for this online, but they all require another Windows machine and of course cannot be automated.

However with guestfish (and the always brilliant SYSLINUX doing most of the heavy lifting), this script will unpack the ISO and turn it into a bootable USB key.

Notes:

  1. I am not going to support this script. You will need to read the script, look up the commands in the guestfish man page, and understand what it does. Any requests for help will be deleted unread.
  2. You need to edit the USB key device before using the script.
  3. You need libguestfs ≥ 1.21 with SYSLINUX support compiled in.
#!/bin/bash - guestfish <<'EOF' trace on add-ro en_microsoft_hyper-v_server_2012_x64_dvd_915600.iso # NB: The next line MUST be changed to your USB drive. # ANYTHING ON THIS DRIVE WILL BE OVERWRITTEN WITHOUT WARNING. add /dev/sdX run # Inside the appliance, /dev/sda = DVD, /dev/sdb = USB. # THESE ARE NOT RELATED TO HOST DISK NAMES. echo "Partitioning the USB disk ..." part-init /dev/sdb mbr part-add /dev/sdb p 63 -1 part-set-mbr-id /dev/sdb 1 0xb part-set-bootable /dev/sdb 1 true mkfs vfat /dev/sdb1 echo "Copying the contents of the DVD to the USB key ..." mkmountpoint /cd mkmountpoint /usb mount /dev/sda /cd mount /dev/sdb1 /usb # XXX We should add cp-r command XXX debug sh "cp -rP /sysroot/cd/* /sysroot/usb" #glob cp-a /cd/* /usb umount /cd umount /usb rmmountpoint /cd rmmountpoint /usb echo "Making the USB key bootable using SYSLINUX ..." syslinux /dev/sdb1 mount /dev/sdb1 / upload /usr/share/syslinux/chain.c32 /chain.c32 write /syslinux.cfg "DEFAULT windows\n\nLABEL windows\nCOM32 chain.c32\nAPPEND fs ntldr=/bootmgr\n" umount /dev/sdb1 upload /usr/share/syslinux/mbr.bin /dev/sdb echo "Finished." EOF
Categories: LUG Community Blogs

Richard WM Jones: rich

Tue, 07/05/2013 - 13:30

libguestfs has high quality Python bindings. Using rpyc you can make a remote libguestfs server with almost no effort at all.

Firstly start an rpyc server:

$ /usr/lib/python2.7/site-packages/rpyc/servers/classic_server.py [SLAVE INFO 13:21:17 tid=140019939981120] server started on 0.0.0.0:18812 [SLAVE INFO 13:21:17 tid=140019784894208] started background auto-register thread (interval = 60) [REGCLNT INFO 13:21:17] registering on 255.255.255.255:18811 [REGCLNT WARNING 13:21:19] no registry acknowledged

Now, possibly from the same machine or some other machine, you can connect to this server and use Python objects remotely as if they were local:

$ python Python 2.7.3 (default, Aug 9 2012, 17:23:57) [GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import rpyc >>> c = rpyc.classic.connect('localhost')

You can now create a libguestfs handle, following the example here.

>>> g = c.modules.guestfs.GuestFS() >>> g.version() {'release': 36L, 'major': 1L, 'minor': 21L, 'extra': 'fedora=20,release=1.fc20,libvirt'} >>> g.add_drive('/dev/fedora/f18x64', readonly=True) >>> g.launch() >>> roots = g.inspect_os() >>> g.inspect_get_product_name(roots[0]) 'Fedora release 18 (Spherical Cow)' >>> g.inspect_get_mountpoints(roots[0]) [('/', '/dev/mapper/fedora-root'), ('/boot', '/dev/sda1')]

As you can see, the g object is transparently remoted without you needing to do anything.


Categories: LUG Community Blogs

Dean Wilson: Deprecation Warnings From Puppet Resources

Sat, 27/04/2013 - 12:53
Over time parts of your puppet manifests will become unneeded. You might move a cronjob or a users in to a package or no longer need a service to be enabled after a given release. I've recently had this use case and had two options - either rely on comments in the Puppet code and write an out of band tool to scan the code base and present a report or add them to the puppet resources themselves. I chose the latter.

Below you'll find a simple metaparameter (a parameter that works with any resource type) that adds this feature to puppet. As this is an early prototype I've hacked it directly in to my local puppet fork. Below you'll see a sample resource that declares a deprecation date and message, the code that implements it and a simple command line test you can run to confirm it works.

# sample puppet resource using :deprecation file { '/ec/cron.d/remove_foos': ensure => 'file', source => 'puppet:///modules/foo/foo.cron', deprecation => '20130425:Release 6 removes the need for the foo cronjob', } $ sudo vi puppet-3.1.1/lib/puppet/type.rb newmetaparam(:deprecation) do desc " Add a deprecation warning to resources. file { '/etc/foo': content => 'Bar', deprecation => '20130425:We no longer need the foo' } The deprecation comes in two parts, separated by a : The date is in format YYYYMMDD and the message is a free form string. " munge do |deprecation| date, message = deprecation.split(':') # YYY MM DD - one true timestamp now = Time.now.strftime('%Y%m%d') if (now >= date) rsrc = "#{@resource.type.capitalize}[#{@resource.name}]" Puppet.warning "#{rsrc} expired on #{date}: #{message}" end end end # command line test $ puppet apply -e 'file { "/tmp/dep": content => "foo\n", deprecation => "20120425:We can remove this file after release 4" }' Warning: File[/tmp/dep] expired on 20120425: We can remove this file after release 4 Notice: Finished catalog run in 0.06 seconds

Using the metaparameter is easy enough, just specify 'deprecation' as a property on a resource and provide a string that contains the date to start flagging the deprecation on (in YYYYMMDD format) and the message puppet should show. I don't currently fail the run on an expired resource but this is an option.

The are some other aspects of this to consider - Richard Clamp raised the idea of having a native type that could indicate this for an entire class (I'd rather use a function, but only because they are much easier to write) and Trevor Vaughan suggested a Puppet face that could present a report of the expired, and soon to be expired, code.

I don't know how widely useful this is but it made a nice change to write some puppet code. The small size of the example will hopefully show how easy it is to extend nearly every part of puppet - including more 'complicated' aspects like metaparameters. Although not the relationship ones, those are horrible ;) I've submitted the idea to the upstream development list so we'll see what happens.

Categories: LUG Community Blogs

Richard WM Jones: rich

Fri, 26/04/2013 - 19:17

Brian is Red Hat’s CTO, and hence my boss’s boss’s boss (or something like that). This is a pretty good (and honest) talk about Red Hat’s plans for OpenStack.

Edit: By the way, the thumbnail (the one I see at any rate) is not Brian.


Categories: LUG Community Blogs

Richard WM Jones: rich

Tue, 23/04/2013 - 20:46

There are some items which are just not possible to order online any more, assuming you want something of minimal quality. These include:

  1. chargers for mobile phones — all now fake as far as I can tell
  2. batteries for older mobile phones — second hand, dead, fake, or simply the wrong battery
  3. consumer cables, like USB to micro USB
  4. convection heaters — didn’t expect this, but I can’t find a reliable one (for the new office) online

That’s not counting the stuff where it’s still just about possible to get non-fake stuff, but it’s a crapshoot, eg. computer memory, hard disks, batteries, flash memory.


Categories: LUG Community Blogs

Richard WM Jones: 20130423_155905.jpg

Tue, 23/04/2013 - 16:00

Mike the electrician is here wiring up the garden office to the mains (I’m doing the networking). The armoured cable (black) and the two cat7 network cables (yellow) will be buried in this trench:

They go in through the wall. This will later be hidden with metal conduit. That wall is 12 inches thick, and close behind it is the gas main so he had to be very careful where he was drilling.

Finally the armoured cable finishes next to the consumer unit (to the right — not shown). Look at the lovely 1950s-era mains supply:


Categories: LUG Community Blogs

Richard WM Jones: rich

Mon, 22/04/2013 - 10:09

Since libguestfs 1.20 it has been possible to use rsync to upload or download files to a disk image incrementally. This is one way to do backups, but note that it won’t work on live guests unless you take a snapshot.

rsync involves using a network connection into or out of the appliance, and is therefore a lot more involved to set up. The script below shows one way to do this, by running an rsync daemon on the host, and letting the libguestfs appliance connect to it.

The script runs rsync inside the appliance, copying /home from the attached disk image out to /tmp/backup on the host. If the operation is repeated, then only incrementally changed files will be copied out. (To incrementally delete files on the target, add the deletedest:true flag).

Note you will need to open port 2999 on your host’s firewall for this to work.

#!/bin/bash - set -x # The target directory. mkdir -p /tmp/backup # Create the daemon. rm -f /tmp/rsyncd.pid cat <<EOF > /tmp/rsyncd.conf port = 2999 pid file = /tmp/rsyncd.pid [backup] path = /tmp/backup use chroot = false read only = false EOF rsync --daemon --config=/tmp/rsyncd.conf # Run guestfish and attach to the guest. guestfish --ro --network -a /dev/fedora/f19rawhidex32 -i <<EOF trace on rsync-out /home rsync://rjones@192.168.122.1:2999/backup archive:true EOF # Kill the rsync daemon. kill `cat /tmp/rsyncd.pid`
Categories: LUG Community Blogs

Richard WM Jones: 20130420_175710.jpg

Sat, 20/04/2013 - 17:59

Inside there’s a ring for sockets and a lighting circuit. The windows and doors are pre-made double-glazed units.


Categories: LUG Community Blogs

Richard WM Jones: image

Fri, 19/04/2013 - 15:15


Categories: LUG Community Blogs

Richard WM Jones: 20130419_094942.jpg

Thu, 18/04/2013 - 18:39

Update: Further progress this morning …


Categories: LUG Community Blogs

Richard WM Jones: rich

Thu, 18/04/2013 - 17:26

Dominic Cleal’s short introduction to the Augeas configuration API.

We use Augeas a lot in libguestfs and virt-v2v, and it’s been very effective for us.

I asked Dominic how he made this video.

He uses gtk-recordmydesktop, max 100/100 audio/video quality, 30fps, 2 channel audio at 48kHz.

Sound and video are recorded at the same time, with a Sennheiser headset.

Editing is done in kdenlive.


Categories: LUG Community Blogs

Richard WM Jones: 20130418_145150.jpg

Thu, 18/04/2013 - 14:55

The cabin has a double wall which will contain insulation. Shown here is the ring main for the four sets of sockets around the room.


Categories: LUG Community Blogs

Richard WM Jones: 20130417_170829.jpg

Thu, 18/04/2013 - 13:17

This trench will take 1 armoured mains cable and 2 cat 7 network cables.


Categories: LUG Community Blogs

Richard WM Jones: rich

Wed, 17/04/2013 - 14:31

Also (not shown) they dug a trench down the side of the garden for electrics and network cables. We’re going to run some hefty armoured cable, plus two or three cat 7 network cables, covered with bricks and warning tape, at a depth of 18″ (mandated by building regulations).


Categories: LUG Community Blogs