UK Linux User Groups

I got a letter from HMRC this morning – to my company, not to me personally. It basically said “we’ve been looking at the PAYE you paid in 2010/11 and it looks like you’ve overpaid by [a surprisingly large number of pounds]“.

Now 2010/11 was the year that I was having some difficulties with my accountants. The difficulties eventually got so bad that I switched to my new accountants (who I’m still very happy with). So it doesn’t really surprise me that something went wrong that year, although the amount (it’s about 25% of the PAYE/NI I paid that year) is impressive.

What really surprises me is the tone of this letter. Having told me that I’ve overpaid (and, helpfully, pointed out the exact extra payment that I made) the letter goes on to say:

Before I can agree to either a refund or a credit, please let me the reason the overpayment has arisen

And:

Please complete the enclosed P35D giving a full explanation as to how the overpayment occurred.

Below are some example of reasons I cannot accept to justify an overpayment

• Duplicate payments with no evidence to explain why
• Statements such as ‘Payment(s) made in error’ with no further explanation
• The overpayment is due to monthly payments which do not match our records
• Any other explanation without evidence to support it

I’m finding it hard to read that in any way other than “we know we’ve got some of your money but you can’t have it back until you’ve explained in detail just how crap your record-keeping is”.

You’ve got my money. You know it’s my money. Either my accountants or I screwed up in some way. There’s no more detailed explanation than that. Just give it back to me, you bastards.

Related Posts:

• Chuggers
• Five Pounds and Twenty Pence
• Homeopathy Petition Response
• See Tickets
• Greens and Science

New in libguestfs ≥ 1.21.15, the virt-df and virt-alignment-scan tools now use parallel appliances when scanning your libvirt guests.

The amount of parallelism is selected heuristically when the tool starts up — by dividing the amount of free memory in MB by 500. You can also override this choice by using the new -P option to both tools, but the default should be fine for everyone. -P 1 disables multiple threads.

Users won’t see much difference, although I found that both tools are noticeably faster.

The implementation of threads in these tools is a little bit interesting. Of course there is a pool of worker threads. These take the libvirt guests from a list sorted in alphabetical order and process them.

However each guest takes a variable amount of time to process, and the trick is that the output from each thread mustn’t overlap or be in non-alphabetical order.

The worker threads do two things to ensure this: Firstly output from each guest scan is saved up in an open_memstream buffer. Secondly, domains are retired in order using a pthread condition variable — each worker waits until the previous domain has been retired, before retiring (ie. printing) its own result.

The outcome is that there should be no difference between what the old tools and the rewritten tools print out.

Chilling in Jervis Bay. Weather has turned out much better than the forecasts. Lovely.

See the full gallery on Posterous

Permalink | Leave a comment  »

[Part 1, part 2, part 3.]

Finally I modified the test to do some representative work: We now load a real Windows XP guest, inspect it (a heavyweight operation), and mount and stat each filesystem. I won’t reproduce the entire test program again because only the test subroutine has changed:

sub test { my $g = Sys::Guestfs->new; $g->add_drive_ro ("/tmp/winxp.img"); $g->launch (); # Inspect the guest (ignore the result). $g->inspect_os (); # Approximate what virt-df does. my %fses = $g->list_filesystems (); foreach (keys %fses) { my $mounted = 0; eval { $g->mount_ro ($_, "/"); $mounted = 1; }; if ($mounted) { $g->statvfs ("/"); $g->umount_all (); } } return $g; }

Even with all that work going on, I was able to inspect more than 1 disk per second on my laptop, and run 60 threads in parallel with good performance and scalability:

A problem encountered in part 2 was that I couldn’t measure the maximum number of parallel libguestfs appliances that can be run at the same time. There are two reasons for that. The simpler one is that libvirt has a limit of 20 connections, which is easily overcome by setting LIBGUESTFS_ATTACH_METHOD=appliance to eliminate libvirt and run qemu directly. The harder one is that by the time the last appliances in the test are starting to launch, earlier ones have already shut down and their threads have exited.

What is needed is for the test to work in two phases: In the first phase we start up all the threads and launch all the appliances. Only when this is complete do we enter the second phase where we shut down all the appliances.

The easiest way to do this is by modifying the test to use a barrier (or in fact to implement a barrier using the condition primitives). See the modified test script below.

With the modified test script I was able to run ≥ 110 and < 120 parallel appliances in ~ 13 GB of free RAM, or around 120 MB / appliance, still with excellent performance and nearly linear scalability:

#!/usr/bin/perl -w use strict; use threads qw(yield); use threads::shared qw(cond_broadcast cond_wait lock); use Sys::Guestfs; use Time::HiRes qw(time); my $nr_threads_launching :shared; sub test { my $g = Sys::Guestfs->new; $g->add_drive_ro ("/dev/null"); $g->launch (); return $g; } # Get everything into cache. test (); test (); test (); sub thread { my $g = test (); { lock ($nr_threads_launching); $nr_threads_launching--; cond_broadcast ($nr_threads_launching); cond_wait ($nr_threads_launching) until $nr_threads_launching == 0; } $g->close (); } # Test increasing numbers of threads until it fails. for (my $nr_threads = 10; $nr_threads < 1000; $nr_threads += 10) { my $start_t = time (); $nr_threads_launching = $nr_threads; my @threads; foreach (1..$nr_threads) { push @threads, threads->create (\&thread) } foreach (@threads) { $_->join (); if (my $err = $_->error ()) { die "launch failed with $nr_threads threads: $err" } } my $end_t = time (); printf ("%d %.2f\n", $nr_threads, $end_t - $start_t); }

One problem with the previous test is that I hit a limit of 20 parallel appliances and mistakenly thought that I’d hit a memory limit. In fact libvirt out of the box limits the number of client connections to 20. You can adjust libvirt’s limit by editing /etc/libvirt/libvirtd.conf, but easier for us is to simply eliminate libvirt from the equation by doing:

export LIBGUESTFS_ATTACH_METHOD=appliance

which causes libguestfs to run qemu directly. In my first test I reached 48 parallel launches before I killed the program (because that’s a lot of parallelism and there seemed no end in sight). Scalability of the libguestfs / qemu combination was excellent again:

But there’s more! (In the next part …)

I wrote the Perl script below to find out how many libguestfs appliances we can start in parallel. The results are surprising (-ly good):

What’s happening here is that we’re booting up a KVM guest with 500 MB of memory, booting the Linux kernel, booting a minimal userspace, then shutting the whole lot down. And then doing that in parallel with 1, 2, .. 20 threads.

[Note: Hardware is my Lenovo x230 laptop with an Intel Core(TM) i7-3520M CPU @ 2.90GHz, 2 cores with 4 threads, 16 GB of RAM with approx. 13 GB free. Software is: Fedora 18 with libguestfs 1.20.2, libvirt 1.0.2 (from Rawhide), qemu 1.4.0 (from Rawhide)]

The test fails at 21 threads because there isn’t enough free memory, so each qemu instance is allocating around 660 MB of RAM. This is wrong: It failed because libvirt out of the box limits the maximum number of clients to 20. See next part in this series.

Up to 4 parallel launches, you can clearly see the effect of better utilization of the parallelism of the CPU — the total elapsed time hardly moves, even though we’re doing up to 4 times more work.

#!/usr/bin/perl -w use strict; use threads; use Sys::Guestfs; use Time::HiRes qw(time); sub test { my $g = Sys::Guestfs->new; $g->add_drive_ro ("/dev/null"); $g->launch (); } # Get everything into cache. test (); test (); test (); # Test increasing numbers of threads until it fails. for my $nr_threads (1..100) { my $start_t = time (); my @threads; foreach (1..$nr_threads) { push @threads, threads->create (\&test) } foreach (@threads) { $_->join (); if (my $err = $_->error ()) { die "launch failed with nr_threads = $nr_threads: $err" } } my $end_t = time (); printf ("%d %.2f\n", $nr_threads, $end_t - $start_t); }

This just popped up in my Google alerts: https://github.com/M2IHP13-admin/JonesForth-arm

My blog has been on Posterous for some years now and it's been awesome. The best thing about it is that you just email a bunch of stuff, with whatever attachments in whatever format are relevant, and they just work.

Sadly they're shutting down following their talent acquisition by Twitter. That's a real shame. Now I have to find an alternative.

Requirements:

• Hosted. I'm not going to maintain a server just for blogging thanks.
• Allows custom JS. I'm always testing out new analytics tools on my own sites.
• Post through email

Squarespace is lovely but pretty expensive for what I need, unless I can consolidate all three sites into one platform while keeping the domains (waiting on their ticket response).

I had high hopes for Markdown-based blog tools like Jekyll, but I find them a bit clunky. Posterous has got me used to a really easy blogging workflow that works well for me.

Any suggestions? I'm happy to pay.

Permalink | Leave a comment  »

It works too …

sd 4:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) sd 4:0:0:0: [sdc] 4096-byte physical blocks sd 4:0:0:0: [sdc] Write Protect is off sd 4:0:0:0: [sdc] Mode Sense: 00 3a 00 00 sd 4:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 5:0:0:0: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) sd 5:0:0:0: [sdd] 4096-byte physical blocks sd 5:0:0:0: [sdd] Write Protect is off sd 5:0:0:0: [sdd] Mode Sense: 00 3a 00 00 sd 5:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA

https://www.sciencedirect.com/science/article/pii/S0168900297002118

Well, that's another FOSDEM over with. In general this year seemed the same as the last couple of years but slightly bigger than usual (although it seems that way every year). The (newish) K building was in full swing with dozens of project stalls and dev rooms. The usual suspects - virtualisation / cloud, configuration management and MySQL rooms had nearly as many people trying to get in to the rooms as they did sitting down.

I think some of the main dev rooms have reached the level of popularity that forces you to either arrive early, get a seat and not move for the rest of the day or accept a very high level of probability that you won't get to see the talks you want. I know a few of us had trouble cherry picking sessions across tracks - which obviously means we have excellent taste in topics. I wonder if having the same talks on both days would make it easier to move around as a visitor - you'd attempt to catch it the first time and if that fails, come back tomorrow. I realise however that this puts even more of a burden on speakers that graciously give their own time in both the preparation and performing of their talks. It does seem that scaling the rooms is the problem of the day once again.

I'd like to say a big thank you to all the organisers, speakers and other attendees for making it another enjoyable couple of days. See you next year.

To clarify, what is the memory overhead, or how many guests can you cram onto a single host, memory being the typical limiting factor when you virtualize.

This was the question someone asked at work today. I don’t know the answer either, but the small program I wrote (below) aims to find out. If you believe the numbers below from qemu 1.2.2 running on Fedora 18, then the overhead is around 150 MB per qemu process that cannot be shared, plus around 200 MB per host (that is, shared between all qemu processes).

guest size 256 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 404.20 MB Shared writable memory: 0.03 MB guest size 512 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 643.76 MB Shared writable memory: 0.03 MB guest size 1024 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 1172.38 MB Shared writable memory: 0.03 MB guest size 2048 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 2237.16 MB Shared writable memory: 0.03 MB guest size 4096 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 4245.13 MB Shared writable memory: 0.03 MB

The number to pay attention to is “Anonymous memory” since that is what cannot be shared between guests (except if you have KSM and your guests are such that KSM can be effective).

There are some known shortcomings with my testing methodology that I summarise below. You may be able to see others.

1. We’re testing a libguestfs appliance. A libguestfs appliance does not have the full range of normal qemu devices that a real guest would have, and so the overhead of a real guest is likely to be higher. The main difference is probably lack of a video device (so no video RAM is evident).
2. This uses virtio-scsi. Real guests use IDE, virtio-blk, etc which may have quite different characteristics.
3. This guest has one user network device (ie. SLIRP) which could be quite different from a real network device.
4. During the test, the guest only runs for a few seconds. A normal, long-running guest would experience qemu memory growth or even memory leaks. You could fix this relatively easily by adding some libguestfs busy-work after the launch.
5. The guest does not do any significant writes, so during the test qemu won’t be storing any cached or in-flight data blocks.
6. It only accounts for memory used by qemu in userspace, not memory used by the host kernel on behalf of qemu.
7. The effectiveness or otherwise of KSM is not tested. It’s likely that KSM depends heavily on your workload, so it wouldn’t be fair to publish any KSM figures.
8. The script uses /proc/PID/maps but it would be better to use smaps so that we can see how much of the file-backed copy-on-write segments have actually been copied. Currently the script overestimates these by assuming that (eg) all the data pages from a library would be dirtied by qemu.

Another interesting question would be whether qemu is getting better or worse over time.

#!/usr/bin/perl -w # Estimate memory usage of qemu-kvm at different guest RAM sizes. # By Richard W.M. Jones <rjones@redhat.com> use strict; use Sys::Guestfs; no warnings "portable"; # 64 bit platform required. # Loop over different guest RAM sizes. my $mbytes; for $mbytes (256, 512, 1024, 2048, 4096) { print "guest size ", $mbytes, " MB:\n"; my $g = Sys::Guestfs->new; # Ensure we're using the direct qemu launch backend, otherwise # libvirt stops us from finding the qemu PID. $g->set_attach_method ("appliance"); # Set guest memory size. $g->set_memsize ($mbytes); # Enable user networking just to be more like a "real" guest. $g->set_network (1); # Launch guest with one dummy disk. $g->add_drive ("/dev/null"); $g->launch (); # Get process ID of qemu. my $pid = $g->get_pid (); die unless $pid > 0; # Read the memory maps of the guest. open MAPS, "/proc/$pid/maps" or die "cannot open memory map of pid $pid"; my @maps = <MAPS>; close MAPS; # Kill qemu. $g->close (); # Parse the memory maps. my $shared_file_backed = 0; my $anonymous = 0; my $shared_writable = 0; my $map; foreach $map (@maps) { chomp $map; if ($map =~ m/ ^([0-9a-f]+)-([0-9a-f]+) \s (....) \s [0-9a-f]+ \s ..:.. \s (\d+) \s+ (\S+)? /x) { my ($start, $end) = (hex $1, hex $2); my $size = $end - $start; my $mode = $3; my $inode = $4; my $filename = $5; # could also be "[heap]", "[vdso]", etc. # Shared file-backed text: r-xp, r--p, etc. with a file backing. if ($inode != 0 && ($mode eq "r-xp" || $mode eq "r--p" || $mode eq "---p")) { $shared_file_backed += $size; } # Anonymous memory: rw-p. elsif ($mode eq "rw-p") { $anonymous += $size; } # Writable and shared. Not sure what this is ... elsif ($mode eq "rw-s") { $shared_writable += $size; } # Ignore [vdso], [vsyscall]. elsif (defined $filename && ($filename eq "[vdso]" || $filename eq "[vsyscall]")) { } # Ignore ---p with no file. What's this? elsif ($inode == 0 && $mode eq "---p") { } # Ignore kvm-vcpu. elsif ($filename eq "anon_inode:kvm-vcpu") { } else { warn "warning: could not parse '$map'\n"; } } else { die "incorrect maps format: '$map'"; } } printf("Shared memory backed by a file: %.2f MB\n", $shared_file_backed / 1024.0 / 1024.0); printf("Anonymous memory (eg. malloc, COW, stack), not shared: %.2f MB\n", $anonymous / 1024.0 / 1024.0); printf("Shared writable memory: %.2f MB\n", $shared_writable / 1024.0 / 1024.0); print "\n"; }

I’m told that Richard Harman (twitter) will mention libguestfs in his talk about malware analysis at ShmooCon next Saturday (16th).

The conference is in Washington DC at the Hyatt Regency, but talks should be available online afterwards (also good because it’s sold out!)

It's been a while since I've attended a Puppet Camp but considering the quality of the last one (organised by Patrick Debois) and the fact it was being held in the lovely city of Ghent again I thought it'd be a wise investment to scrape together the time off.

The quality of the talks seemed quite high and considering the number of newer users present the content level was well pitched. A couple of deeper talks for the more experienced members would have been nice but we mostly made our own in the open sessions. Facter, writing MCollective plugins, off-line and bulk catalogue compilation and the murky corners of our production puppets all came under discussion - in some cases quite fruitfully.

The wireless was a point of annoyance and amusement (depending on the person and the time of day). We had 20 users for an audience of ten times that - the attitudes covered the gamut from "I only need to check my mail once a day" to "I have my own tethering" and all the way to "This is my brute force script I run in a loop". You can tell when most of us lost our access based on the twitter hash tag.

I was a little surprised at the number of Puppet Camps there will be this year - 27 was the number mentioned. I think a lot of the more experienced members of the community value the camps and confs as a chance to catch up with each other and the PuppetLabs people and I'd hate to see us sticking to our own local camps and losing the cross pollination of ideas, plans and pains.

You can also view the Puppet Camp slides for a number of the sessions.

With a title like Resilience and Reliability on AWS I had quite high expectations for this slim book. Unfortunately, they were not met.

The first four chapters provide brief introductions to AWS and some of its more popular services. While these were fine I'd point people looking for this level of information at the Amazon Webservice Advent 2012 instead. Following this are a handful of more cookbook like chapters that each present a small amount of theory and advice about how to run a given applications on AWS - interspaced with multiple pages of python code. The chapters don't go in to enough details to bring much value to their subjects and the code detracts from the narrative without bringing much technical insight. I was particularly irked at the commented out sections - if you're going to publish a lot of code in a small book then at least be conscious that each line should bring something to the table.

It feels like this book should have been a series of blog posts rather than a printed book. Very disappointing and not recommended. Programming Amazon EC2 Programming Amazon EC2 by the same authors is much better.

Creating BERT dicts in Go

I’ve been learning Go recently and have written a program to connect to an existing service (written in Ruby) that sends and receives messages serialised as BERT terms.

I’m posting this partly because I had quite a lot of fun figuring it out and partly to document creating BERT dicts in Go should anyone else need to do this in the future and hit the same issues I did.

Why BERT?

I’m a big fan of BERT. It’s compact, flexible, and there are good libs available for serialisation/de-serialisation. So far I’ve exclusively been using the bert gem (written by Tom Preston-Werner, author of the BERT spec).

Creating BERT dicts

One of the great features of BERT is the complex types it supports, including dicts. The equivalent to a dict in Ruby would be a hash, in Go a map. They are really simple to create in Ruby:

require 'bert' BERT.encode({"key" => "val"}) => "\x83h\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02m\x00\x00\x00\x03keym\x00\x00\x00\x03valj"

We can pull this apart and see exactly what the bert gem did to our data. Let’s dump the string to an array of 8-bit unsigned integers:

BERT.encode({"key" => "val"}).unpack("C*") => [131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 1, 104, 2, 109, 0, 0, 0, 3, 107, 101, 121, 109, 0, 0, 0, 3, 118, 97, 108, 106]

It’s hard to see exactly what happened, but with the BERT docs and the erlang External Term Format docs we can see how the hash got encoded.

magic| tuple | atom | bert | | dict | list 1 elem | list | atom | key | atom | | val | nil | nil 131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 1, 108, 0, 0, 0, 2, 100, 0, 3, 107, 101, 121, 100, 0, 3, 118, 97, 108, 106, 106

If the formatting of that breakdown is messed up here’s a raw gist that may be clearer.

What you can see here are what the bytes represent (you can see the breakdown of each data type on the External Term Format docs). This is great, but why write a blog post just about dicts? Well, they’re easy to create in Ruby:

BERT.encode(:complex => {"key" => [:data, {:structures => "are easy to serialise"}]}) => "\x83h\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02d\x00\acomplexh\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02m\x00\x00\x00\x03keyl\x00\x00\x00\x02d\x00\x04datah\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02d\x00\nstructuresm\x00\x00\x00\x15are easy to serialisejjjj"

but it’s not so obvious in Go, and I hit some issues when trying to create them.

Serialising to BERT in Golang

Serialising data to BERT/BERP in Go is pretty easy for simple cases using the gobert lib:

package main import ( "fmt" "bytes" "github.com/sethwklein/gobert" ) func main() { var buf = new(bytes.Buffer) bert.MarshalResponse(buf, bert.Atom("foo")) for _, b := range(buf.Bytes()) { fmt.Printf("%d ", b) } fmt.Println() }

This gives us:

0 0 0 7 131 100 0 3 102 111 111

If we run that through the Ruby lib decoder we get:

> BERT.decode([131, 100, 0, 3, 102, 111, 111].pack("C*")) => :foo

(The Ruby bert lib decodes atoms to symbols).

Serialising to BERT dicts in Golang

However, there is a little more effort involved serialising more complex data structures, in particular dicts, as I found out.

You might have thought that you could just pass in a map:

package main import ( "fmt" "bytes" "github.com/sethwklein/gobert" ) func main() { message := map[string]string{"key1": "val1", "key2": "val2"} var buf = new(bytes.Buffer) bert.MarshalResponse(buf, message) for _, b := range(buf.Bytes()) { fmt.Printf("%d ", b) } fmt.Println() }

We get the output:

0 0 0 1 131

Well, that doesn’t work. What you end up with is a one byte long BERP. It seems that gobert doesn’t automatically serialise maps. No problem, we’ll build one up manually. A quick look at the BERT documentation shows the format of a dict:

“Dictionaries (hash tables) are expressed via an array of 2-tuples representing the key/value pairs. The KeysAndValues array is mandatory, such that an empty dict is expressed as {bert, dict, []}. Keys and values may be any term. For example, {bert, dict, [{name, <<“Tom”>>}, {age, 30}]}.”

So let’s create this special structure manually.

package main import ( "fmt" "bytes" "github.com/sethwklein/gobert" ) func main() { message1 := []bert.Term{bert.Atom("key1"), bert.Atom("val1")} message2 := []bert.Term{bert.Atom("key2"), bert.Atom("val3")} keys_and_values := []bert.Term{message1, message2} dict := []bert.Term{bert.BertAtom, bert.Atom("dict"), keys_and_values} var buf = new(bytes.Buffer) bert.MarshalResponse(buf, dict) for _, b := range(buf.Bytes()) { fmt.Printf("%d ", b) } fmt.Println() }

The result:

0 0 0 51 131 104 3 100 0 4 98 101 114 116 100 0 4 100 105 99 116 104 2 104 2 100 0 4 107 101 121 49 100 0 4 118 97 108 49 104 2 100 0 4 107 101 121 50 100 0 4 118 97 108 51

It looks better, but it doesn’t decode, using Ruby:

> BERT.decode([131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 104, 2, 104, 2, 100, 0, 4, 107, 101, 121, 49, 100, 0, 4, 118, 97, 108, 49, 104, 2, 100, 0, 4, 107, 101, 121, 50, 100, 0, 4, 118, 97, 108, 51].pack("C*")) TypeError: Invalid dict spec, not an erlang list

We’re still missing something. Let’s compare the output of the Ruby bert lib to the output of gobert for the same data structure:

> BERT.encode({:key1 => :val1, :key2 => :val2}).unpack("C*") => [131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 2, 104, 2, 100, 0, 4, 107, 101, 121, 49, 100, 0, 4, 118, 97, 108, 49, 104, 2, 100, 0, 4, 107, 101, 121, 50, 100, 0, 4, 118, 97, 108, 50, 106]

We’re definitely missing some data in the gobert output.

If you follow along the byte sequences you can see that they start off the same until the 18th byte. In the Ruby output this is ‘108’, or LIST_EXT. In the gobert output it’s 104, a SMALL_TUPLE_EXT. We can see where this difference happens in encode.go in the gobert lib (in the writeTag func):

case reflect.Slice: writeSmallTuple(w, v) case reflect.Array: writeList(w, v)

Let’s decode the BERT data to see where the diversion happens in the underlying data structures:

magic| tuple | atom | bert | atom | dict 131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116

We can see that the “bert” and “dict” atoms are encoded the same, but the keys_and_values array is getting encoded as a SMALL_TUPLE_EXT by gobert when we wanted a LIST_EXT. If we look back at the gobert code we can see that the decision to use SMALL_TUPLE_EXT over LIST_EXT is dependent on a slice or array being present. We can use the go “reflect” package to look at the arrays/slices we are creating and see what they are:

package main import ( "fmt" "reflect" "github.com/sethwklein/gobert" ) func main() { array := [2]bert.Term{} slice := []bert.Term{} array_val := reflect.ValueOf(array) slice_val := reflect.ValueOf(slice) fmt.Printf("array is a: %v\n", array_val.Kind()) fmt.Printf("slice is a: %v\n", slice_val.Kind()) } array is a: array slice is a: slice The fix

So, in order to fix our data structure to get gobert to correctly encode the dict we need to change the keys_and_values slice to an array:

package main import ( "fmt" "bytes" "github.com/sethwklein/gobert" ) func main() { message1 := []bert.Term{bert.Atom("key1"), bert.Atom("val1")} message2 := []bert.Term{bert.Atom("key2"), bert.Atom("val3")} keys_and_values := [2]bert.Term{message1, message2} // Now an array dict := []bert.Term{bert.BertAtom, bert.Atom("dict"), keys_and_values} var buf = new(bytes.Buffer) bert.MarshalResponse(buf, dict) for _, b := range(buf.Bytes()) { fmt.Printf("%d ", b) } fmt.Println() }

The result:

0 0 0 55 131 104 3 100 0 4 98 101 114 116 100 0 4 100 105 99 116 108 0 0 0 2 104 2 100 0 4 107 101 121 49 100 0 4 118 97 108 49 104 2 100 0 4 107 101 121 50 100 0 4 118 97 108 51 106

But more importantly, can we decode the data we encoded?

> BERT.decode([131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 2, 104, 2, 100, 0, 4, 107, 101, 121, 49, 100, 0, 4, 118, 97, 108, 49, 104, 2, 100, 0, 4, 107, 101, 121, 50, 100, 0, 4, 118, 97, 108, 51, 106].pack("C*")) => {:key1=>:val1, :key2=>:val3}

Yes!

Introducing ProjectRaindrops, a service that will build disk images for you ready to be used in your favourite cloud or virtualised environment.

One of the key barriers to entry into a cloud or virtualised environment : setting up and maintaining a piece of infrastructure that builds disk images. Its also a colossal waste of time and involves needing a complete instance of the environment one is going to deploy the image in. Wouldn't it be nice if there was a service that allowed you to drop in a kickstart file, write up a config to go with that kickstart and just build the image for you ?

Now there is. Its called ProjectRaindrops and its live, in beta mode, at http://projectraindrops.net/ ; As an initial kick off we are doing builds for HVM ( ie. any fully virtualised environment, be it KVM, Xen, VirtualBox or VMware ); With more HyperVisors and more disk formats coming soon.

Getting started is easy, goto the Raindrops website; sign in using either github or twitter credentials ( we don't store any personal details, but if your account does not have an email address, we won't be able to send out email notifications ). There are two key components:

• A config file: that contains metadata about what you want the build to do
• A kickstart file: that has the actual details on what the build should contain

Example pre-populated templates are available, just click on the new button and the template will be injected in. And there is some validation rules that track the config and kickstart file, so if you make mistakes or lose format validation, you will find out right away. And we have versioning for each file built in too. Finally, there is no real correlation between a config file and a kickstart file, when you create a new job you can pick an arbitrary kickstart and any config file in your account to match it with.

So, now that you have a config and a kickstart file, click on new job, give it a name, select which config you want to use and what kickstart file, click on 'Submit' and in a few minutes your build should be done. You can even track the job as it works its way through various stages.

Lots of interesting things in planning and development stages, stay tuned for more news in the coming weeks. For now, go ahead and drop in on http://projectraindrops.net/ and give it a shot. Just consider it to be a Beta release, so send us lots of feedback.

- KB

Life on Earth is being repeated on iPlayer. It’s an incredible series.

Years ago I played around with CIL to analyze libvirt. More recently Dan used CIL to analyze libvirt’s locking code.

We didn’t get so far either time, but I’ve been taking a deeper look at CIL in an attempt to verify error handling in libguestfs.

Here is my partly working code so far.

(* * Analyse libguestfs APIs to find error overwriting. * Copyright (C) 2008-2013 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. * * This library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * You should have received a copy of the GNU Lesser General Public * License along with this library. If not, see * <http://www.gnu.org/licenses/>. * * Author: Daniel P. Berrange <berrange@redhat.com> * Author: Richard W.M. Jones <rjones@redhat.com> *) open Unix open Printf open Cil let debug = ref false (* Set of ints. *) module IntSet = Set.Make (struct type t = int let compare = compare end) (* A module for storing any set (unordered list) of functions. *) module FunctionSet = Set.Make ( struct type t = varinfo let compare v1 v2 = compare v1.vid v2.vid end ) (* Directed graph of functions. * * Function = a node in the graph * FunctionDigraph = the directed graph * FunctionPathChecker = path checker module using Dijkstra's algorithm *) module Function = struct type t = varinfo let compare f1 f2 = compare f1.vid f2.vid let hash f = Hashtbl.hash f.vid let equal f1 f2 = f1.vid = f2.vid end module FunctionDigraph = Graph.Imperative.Digraph.Concrete (Function) module FunctionPathChecker = Graph.Path.Check (FunctionDigraph) (* Module used to analyze the paths through each function. *) module ErrorCounter = struct let name = "ErrorCounter" let debug = debug (* Our current state is very simple, just the number of error * function calls did encountered up to this statement. *) type t = int let copy errcalls = errcalls (* Start data for each statement. *) let stmtStartData = Inthash.create 97 let printable errcalls = sprintf "(errcalls=%d)" errcalls let pretty () t = Pretty.text (printable t) let computeFirstPredecessor stmt x = x (* XXX??? *) let combinePredecessors stmt ~old:old_t new_t = if old_t = new_t then None else Some new_t (* This will be initialized after we have calculated the set of all * functions which can call an error function, in main() below. *) let error_functions_set = ref FunctionSet.empty (* Handle a Cil.Instr. *) let doInstr instr _ = match instr with (* A call to an error function. *) | Call (_, Lval (Var callee, _), _, _) when FunctionSet.mem callee !error_functions_set -> Dataflow.Post (fun errcalls -> errcalls+1) | _ -> Dataflow.Default (* Handle a Cil.Stmt. *) let doStmt _ _ = Dataflow.SDefault (* Handle a Cil.Guard. *) let doGuard _ _ = Dataflow.GDefault (* Filter statements we've seen already to prevent loops. *) let filter_set = ref IntSet.empty let filterStmt { sid = sid } = if IntSet.mem sid !filter_set then false else ( filter_set := IntSet.add sid !filter_set; true ) (* Initialize the module before each function that we examine. *) let init stmts = filter_set := IntSet.empty; Inthash.clear stmtStartData; (* Add the initial statement(s) to the hash. *) List.iter (fun stmt -> Inthash.add stmtStartData stmt.sid 0) stmts end module ForwardsErrorCounter = Dataflow.ForwardsDataFlow (ErrorCounter) (* The always useful filter + map function. *) let rec filter_map f = function | [] -> [] | x :: xs -> match f x with | Some y -> y :: filter_map f xs | None -> filter_map f xs let rec main () = (* Read the list of input C files. *) let files = let chan = open_process_in "find src -name '*.i' | sort" in let files = input_chan chan in if close_process_in chan <> WEXITED 0 then failwith "failed to read input list of files"; if files = [] then failwith "no input files; is the program running from the top directory? did you compile with make -C src CFLAGS=\"-save-temps\"?"; files in (* Load and parse each input file. *) let files = List.map ( fun filename -> printf "loading %s\n%!" filename; Frontc.parse filename () ) files in (* Merge the files. *) printf "merging files\n%!"; let sourcecode = Mergecil.merge files "libguestfs" in (* CFG analysis. *) printf "computing control flow\n%!"; Cfg.computeFileCFG sourcecode; let functions = filter_map (function GFun (f, loc) -> Some (f, loc) | _ -> None) sourcecode.globals in (* Examine which functions directly call which other functions. *) printf "computing call graph\n%!"; let call_graph = make_call_graph functions in (* FunctionDigraph.iter_edges ( fun caller callee -> printf "%s calls %s\n" caller.vname callee.vname ) call_graph; *) (* The libguestfs error functions. These are global function names, * but to be any use to us we have to look these up in the list of * all global functions (ie. 'functions') and turn them into the * corresponding varinfo structures. *) let error_function_names = [ "guestfs_error_errno"; "guestfs_perrorf" ] in let find_function name = try List.find (fun ({ svar = { vname = n }}, _) -> n = name) functions with Not_found -> failwith ("function '" ^ name ^ "' does not exist") in let error_function_names = List.map ( fun f -> (fst (find_function f)).svar ) error_function_names in (* Get a list of functions that might (directly or indirectly) call * one of the error functions. *) let error_functions, non_error_functions = functions_which_call call_graph error_function_names functions in (* List.iter ( fun f -> printf "%s can call an error function\n" f.vname ) error_functions; List.iter ( fun f -> printf "%s can NOT call an error function\n" f.vname ) non_error_functions; *) (* Save the list of error functions in a global set for fast lookups. *) let set = List.fold_left ( fun set elt -> FunctionSet.add elt set ) FunctionSet.empty error_functions in ErrorCounter.error_functions_set := set; (* Analyze each top-level function to ensure it calls an error * function exactly once on error paths, and never on normal return * paths. *) printf "analyzing correctness of error paths\n%!"; List.iter compute_error_paths functions; () (* Make a directed graph of which functions directly call which other * functions. *) and make_call_graph functions = let graph = FunctionDigraph.create () in List.iter ( fun ({ svar = caller; sallstmts = sallstmts }, _) -> (* Evaluate which other functions 'caller' calls. First pull * out every 'Call' instruction anywhere in the function. *) let insns = List.concat ( filter_map ( function | { skind = Instr insns } -> Some insns | _ -> None ) sallstmts ) in let calls = List.filter (function Call _ -> true | _ -> false) insns in (* Then examine what function is being called at each place. *) let callees = filter_map ( function | Call (_, Lval (Var callee, _), _, _) -> Some callee | _ -> None ) calls in List.iter ( fun callee -> FunctionDigraph.add_edge graph caller callee ) callees ) functions; graph (* [functions_which_call g endpoints functions] partitions the * [functions] list, returning those functions that call directly or * indirectly one of the functions in [endpoints], and a separate list * of functions which do not. [g] is the direct call graph. *) and functions_which_call g endpoints functions = let functions = List.map (fun ({ svar = svar }, _) -> svar) functions in let checker = FunctionPathChecker.create g in List.partition ( fun f -> (* Does a path exist from f to any of the endpoints? *) List.exists ( fun endpoint -> try FunctionPathChecker.check_path checker f endpoint with (* It appears safe to ignore this exception. It seems to * mean that this function is in a part of the graph which * is completely disconnected from the other part of the graph * that contains the endpoint. *) | Invalid_argument "[ocamlgraph] iter_succ" -> false ) endpoints ) functions and compute_error_paths ({ svar = svar } as f, loc) = (*ErrorCounter.debug := true;*) (* Find the initial statement in this function (assumes that the * function can only be entered in one place, which is normal for C * functions). *) let initial_stmts = match f.sbody.bstmts with | [] -> [] | first::_ -> [first] in (* Initialize ErrorCounter. *) ErrorCounter.init initial_stmts; (* Compute the error counters along paths through the function. *) ForwardsErrorCounter.compute initial_stmts; (* Process all Return statements in this function. *) List.iter ( fun stmt -> try let errcalls = Inthash.find ErrorCounter.stmtStartData stmt.sid in match stmt with (* return -1; *) | { skind = Return (Some i, loc) } when is_literal_minus_one i -> if errcalls = 0 then printf "%s:%d: %s: may return an error code without calling error/perrorf\n" loc.file loc.line svar.vname else if errcalls > 1 then printf "%s:%d: %s: may call error/perrorf %d times (more than once) along an error path\n" loc.file loc.line svar.vname errcalls (* return 0; *) | { skind = Return (Some i, loc) } when is_literal_zero i -> if errcalls >= 1 then printf "%s:%d: %s: may call error/perrorf along a non-error return path\n" loc.file loc.line svar.vname (* return; (void return) *) | { skind = Return (None, loc) } -> if errcalls >= 1 then printf "%s:%d: %s: may call error/perrorf and return void\n" loc.file loc.line svar.vname | _ -> () with Not_found -> printf "%s:%d: %s: may contain unreachable code\n" loc.file loc.line svar.vname ) f.sallstmts (* Some convenience CIL matching functions. *) and is_literal_minus_one = function | Const (CInt64 (-1L, _, _)) -> true | _ -> false and is_literal_zero = function | Const (CInt64 (0L, _, _)) -> true | _ -> false (* Convenient routine to load the contents of a channel into a list of * strings. *) and input_chan chan = let lines = ref [] in try while true; do lines := input_line chan :: !lines done; [] with End_of_file -> List.rev !lines and input_file filename = let chan = open_in filename in let r = input_chan chan in close_in chan; r let () = try main () with exn -> prerr_endline (Printexc.to_string exn); Printexc.print_backtrace Pervasives.stderr; exit 1