UK Linux User Groups

Start: 2013-12-14 11:00 End: 2013-12-14 11:00

We have regular sessions on the second Saturday of each month. Location this month yet to be arranged.

Who

New members are very welcome. We're not a cliquey bunch, so you won't feel out of place! Usually between 10 and 30 people come along. 

Start: 2013-11-09 11:00 End: 2013-11-09 11:00

We have regular sessions on the second Saturday of each month. Location this month yet to be arranged.

Who

New members are very welcome. We're not a cliquey bunch, so you won't feel out of place! Usually between 10 and 30 people come along. 

Start: 2013-10-12 11:00 End: 2013-10-12 11:00

We have regular sessions on the second Saturday of each month. Location this month yet to be arranged.

Who

New members are very welcome. We're not a cliquey bunch, so you won't feel out of place! Usually between 10 and 30 people come along. 

Start: 2013-09-14 11:00 End: 2013-09-14 11:00

We have regular sessions on the second Saturday of each month. Location this month yet to be arranged.

Who

New members are very welcome. We're not a cliquey bunch, so you won't feel out of place! Usually between 10 and 30 people come along. 

Start: 2013-08-10 11:00 End: 2013-08-10 11:00

We have regular sessions on the second Saturday of each month. Location this month yet to be arranged.

Who

New members are very welcome. We're not a cliquey bunch, so you won't feel out of place! Usually between 10 and 30 people come along. 

Start: 2013-07-13 11:00 End: 2013-07-13 11:00

We have regular sessions on the second Saturday of each month. Location this month yet to be arranged.

Who

New members are very welcome. We're not a cliquey bunch, so you won't feel out of place! Usually between 10 and 30 people come along. 

To clarify, what is the memory overhead, or how many guests can you cram onto a single host, memory being the typical limiting factor when you virtualize.

This was the question someone asked at work today. I don’t know the answer either, but the small program I wrote (below) aims to find out. If you believe the numbers below from qemu 1.2.2 running on Fedora 18, then the overhead is around 150 MB per qemu process that cannot be shared, plus around 200 MB per host (that is, shared between all qemu processes).

guest size 256 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 404.20 MB Shared writable memory: 0.03 MB guest size 512 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 643.76 MB Shared writable memory: 0.03 MB guest size 1024 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 1172.38 MB Shared writable memory: 0.03 MB guest size 2048 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 2237.16 MB Shared writable memory: 0.03 MB guest size 4096 MB: Shared memory backed by a file: 201.41 MB Anonymous memory (eg. malloc, COW, stack), not shared: 4245.13 MB Shared writable memory: 0.03 MB

The number to pay attention to is “Anonymous memory” since that is what cannot be shared between guests (except if you have KSM and your guests are such that KSM can be effective).

There are some known shortcomings with my testing methodology that I summarise below. You may be able to see others.

1. We’re testing a libguestfs appliance. A libguestfs appliance does not have the full range of normal qemu devices that a real guest would have, and so the overhead of a real guest is likely to be higher. The main difference is probably lack of a video device (so no video RAM is evident).
2. This uses virtio-scsi. Real guests use IDE, virtio-blk, etc which may have quite different characteristics.
3. This guest has one user network device (ie. SLIRP) which could be quite different from a real network device.
4. During the test, the guest only runs for a few seconds. A normal, long-running guest would experience qemu memory growth or even memory leaks. You could fix this relatively easily by adding some libguestfs busy-work after the launch.
5. The guest does not do any significant writes, so during the test qemu won’t be storing any cached or in-flight data blocks.
6. It only accounts for memory used by qemu in userspace, not memory used by the host kernel on behalf of qemu.
7. The effectiveness or otherwise of KSM is not tested. It’s likely that KSM depends heavily on your workload, so it wouldn’t be fair to publish any KSM figures.
8. The script uses /proc/PID/maps but it would be better to use smaps so that we can see how much of the file-backed copy-on-write segments have actually been copied. Currently the script overestimates these by assuming that (eg) all the data pages from a library would be dirtied by qemu.

Another interesting question would be whether qemu is getting better or worse over time.

#!/usr/bin/perl -w # Estimate memory usage of qemu-kvm at different guest RAM sizes. # By Richard W.M. Jones <rjones@redhat.com> use strict; use Sys::Guestfs; no warnings "portable"; # 64 bit platform required. # Loop over different guest RAM sizes. my $mbytes; for $mbytes (256, 512, 1024, 2048, 4096) { print "guest size ", $mbytes, " MB:\n"; my $g = Sys::Guestfs->new; # Ensure we're using the direct qemu launch backend, otherwise # libvirt stops us from finding the qemu PID. $g->set_attach_method ("appliance"); # Set guest memory size. $g->set_memsize ($mbytes); # Enable user networking just to be more like a "real" guest. $g->set_network (1); # Launch guest with one dummy disk. $g->add_drive ("/dev/null"); $g->launch (); # Get process ID of qemu. my $pid = $g->get_pid (); die unless $pid > 0; # Read the memory maps of the guest. open MAPS, "/proc/$pid/maps" or die "cannot open memory map of pid $pid"; my @maps = <MAPS>; close MAPS; # Kill qemu. $g->close (); # Parse the memory maps. my $shared_file_backed = 0; my $anonymous = 0; my $shared_writable = 0; my $map; foreach $map (@maps) { chomp $map; if ($map =~ m/ ^([0-9a-f]+)-([0-9a-f]+) \s (....) \s [0-9a-f]+ \s ..:.. \s (\d+) \s+ (\S+)? /x) { my ($start, $end) = (hex $1, hex $2); my $size = $end - $start; my $mode = $3; my $inode = $4; my $filename = $5; # could also be "[heap]", "[vdso]", etc. # Shared file-backed text: r-xp, r--p, etc. with a file backing. if ($inode != 0 && ($mode eq "r-xp" || $mode eq "r--p" || $mode eq "---p")) { $shared_file_backed += $size; } # Anonymous memory: rw-p. elsif ($mode eq "rw-p") { $anonymous += $size; } # Writable and shared. Not sure what this is ... elsif ($mode eq "rw-s") { $shared_writable += $size; } # Ignore [vdso], [vsyscall]. elsif (defined $filename && ($filename eq "[vdso]" || $filename eq "[vsyscall]")) { } # Ignore ---p with no file. What's this? elsif ($inode == 0 && $mode eq "---p") { } # Ignore kvm-vcpu. elsif ($filename eq "anon_inode:kvm-vcpu") { } else { warn "warning: could not parse '$map'\n"; } } else { die "incorrect maps format: '$map'"; } } printf("Shared memory backed by a file: %.2f MB\n", $shared_file_backed / 1024.0 / 1024.0); printf("Anonymous memory (eg. malloc, COW, stack), not shared: %.2f MB\n", $anonymous / 1024.0 / 1024.0); printf("Shared writable memory: %.2f MB\n", $shared_writable / 1024.0 / 1024.0); print "\n"; }

OK - so I've just been reading the Gaurdian Article on Terry Deary saying that Libraries are outdated and should be got rid of. I entirely disagree with him, I spent a fair amount of my youth borrowing books from my local library, when I had no income, and so the only alternative to borrowing from the library would have been saving up for second hand books... which he wouldn't have seen a penny from. These days I buy Quite A Few books (OK - so, mostly on the kindle these days, but still), if it hadn't have been for libraries when I was younger and couldn't afford to purchase books, I may never have properly picked up the reading habit.

His claim that they're killing bookshops is also, in my opinion, entirely wrong. Bookshops are closing more because of the ease of ordering books online and getting them delivered to your door, with a huge collection of books available from large warehouses rather than the stock that a book store can sell easily. We've got a vast collection of literature available to us now, and it's only ever going upwards, no book shop or single library is going to be able to cater for the entirely different needs of their customers. Libraries do inter library lending, which means that the collection of rarer, less popular books are still available (potentially with a bit of a wait), and every time the book is lent the author gets some funds. If he seriously thinks that if libraries closed the number of people reading his material would stay the same I think he's mistaken. He also doesn't seem to take in to account at all the second hand book market.

All I have to say is NYARGH.

Start: 2013-02-21 18:30 End: 2013-02-21 18:30

Pub meets have been a rarity over the past year so, following on from last month, we're going to start having them more regularly.

This month will be in Guildford again, due to its location and public transport but that doesn't have to be the case every month ;)

It will be at The White House, which is at the bottom end of town (8 High Street, Guildford, GU2 4AJ) and a short walk from the main station, on Thursday 21st starting from around 6.30 - 7.00pm. Their website: http://whitehouseguildford.co.uk/

BTW You don't 'bring a box' to these meets (though I suppose tablet computers are fine) as it's more of a social gathering over beer and food.

Start: 2013-03-09 11:00 End: 2013-03-09 17:00

This month we are at LiNCORE offices (part of the ReigateHub) in Reigate on 9th March. Our thanks go to Jay Bennie for hosting us.

Bring a 'box', bring a notebook/netbook, bring anything that might run Linux, or just bring yourself and enjoy socialising, learning, teaching or simply chilling out!

New members are very welcome. We're not a cliquey bunch, so you won't feel out of place! Usually between 15 and 30 people come along.

Round number

Tomorrow I'll be celebrating a nice round-numbered birthday. I'll be 25 :) I grew up at just the right time that powers of two have been important my whole life.

I'll be celebrating the milestone but having my brother round for the afternoon and playing on my first 32-bit computer - my A1200.

Git Aux

For a while now, I've been wanting to keep various parts of my home directory in sync.

At first, I created a git repository for storing my dotfiles but I found it a pain to keep the repository up-to-date.

Fairly recently, someone pointed out git-annex to me. After a good read of the documentation, it sounded like it could be useful but probably more than I needed and perhaps not quite what I really wanted. Besides, I couldn't get the bloody thing to install.

So I did what any geek would do, I wrote my own :D

See git-aux (or AUR if you're on Arch Linux).

Basically, I wanted an easy to way to keep a git repository in sync with an directory external to it. With git aux installed, I get pretty much exactly what I wanted with a few simple commands.

After creating a new git repository, I do git aux init ~/ to tell git-aux that I want it to sync this repo with my home directory.

I then do git aux add ~/.vimrc ~/.ssh/config ~/.bashrc and any other files I want from my home directory. This copies those files into the repository and I can then commit them in the usual way.

If I make changes in my home directory, I use git aux sync to update the copies in the repository.

If I've made changes on another machine and want to apply those changes from the repo to the home dir on this machine, I do git aux apply.

And that's it :)

It's unfinished and probably broken in places but mostly does what I was looking for.

Oh, and it's written with node of course ;)

Last month I wrote about the BFI’s screening of “An Unearthly Child”, the first ever Doctor Who story. This month it was the turn of the second Doctor and the story chosen to represent that era of the show was “Tomb of the Cybermen”. Lost from the BBC archives for twenty years and believed wiped, the story has always had a mystique within fandom as a lost classic. When it was rediscovered in 1992 some people were underwhelmed but not me.

James at the Doctor Who Podcast had bagged great seats in the third row, which gave us a great view of Frank Skinner and Steven Moffat as they introduced the screening. Frank revealed his true fan colours and even remembered the first episode being broadcast fifty years ago. Moffat’s words, “everything we do [in modern Doctor Who] today is to try and recapture the feeling you get from watching this story” were perfect. The man should be a writer or something. It was good to see the current production team continuing to support the screenings. As well as SteMo, sitting right behind us were Caroline Skinner, Mark Gatiss, Reece Shearsmith and Edward Russell.

I’m starting to think Doctor Who should always be watched on a massive screen with a few hundred other people. Impractical, yes, but jolly exciting. Whilst the shortcomings of some of the effects, most notably the cybermats, were made very clear, the atmosphere was great. Patrick Troughton’s performance was superb, as were the other regulars and supporting cast.

During a hiatus caused by a failed tape machine, Michael Troughton talked briefly about his father Patrick. After the last episode, there was a discussion panel. Pictured above, L-R, are Michael Ferguson (60′s and 70′s Who director), Anneke Wills (Polly), Bernard Holley (Haydon), Deborah Watling (Victoria), Shirley Cooklin (Kaftan), Michael Kilgarrif (Cybercontroller). Although this panel didn’t have the energy of last month’s, it still threw up a few interesting pieces of information.

Afterwards James, Ian (also from the DWP), Chris (from the Oodcast) and I recorded another special review of the screening, huddled against the cold in one of the more welcoming stairwells on the south bank. And then there was pizza, in the warm. I’ve been lucky enough to secure a ticket for the third screening next month too, thanks to James. So next stop, “The Mind of Evil.”

 

Pin It

I’m told that Richard Harman (twitter) will mention libguestfs in his talk about malware analysis at ShmooCon next Saturday (16th).

The conference is in Washington DC at the Hyatt Regency, but talks should be available online afterwards (also good because it’s sold out!)

Customer: "Is xyz there?"
Me: "I'm afraid he's at lunch."
Customer: "Well he was logged into my machine and I cant close the box."
Me: "Oh, well as I said he's at lunch. If you're desperate, reboot the PC that will sever the link."
Customer: "Cant you close it?"
Me: "I do not have access to his machine, I don't know the password."
Customer: "I need to sort some stuff out on my machine."
Me: "Well this is why I suggested rebooting. If you can hang on xyz will be back in say an hour."
Customer: "Nobody will be here, I need to go out."
Me: "Well reboot then?"
Customer: "But I don't want to do that."
Me: "Then wait for xyz"
Customer: "But I need this stuff now."
Me: "I have no other solutions for you."

/me smacks head off desk!

It's been a while since I've attended a Puppet Camp but considering the quality of the last one (organised by Patrick Debois) and the fact it was being held in the lovely city of Ghent again I thought it'd be a wise investment to scrape together the time off.

The quality of the talks seemed quite high and considering the number of newer users present the content level was well pitched. A couple of deeper talks for the more experienced members would have been nice but we mostly made our own in the open sessions. Facter, writing MCollective plugins, off-line and bulk catalogue compilation and the murky corners of our production puppets all came under discussion - in some cases quite fruitfully.

The wireless was a point of annoyance and amusement (depending on the person and the time of day). We had 20 users for an audience of ten times that - the attitudes covered the gamut from "I only need to check my mail once a day" to "I have my own tethering" and all the way to "This is my brute force script I run in a loop". You can tell when most of us lost our access based on the twitter hash tag.

I was a little surprised at the number of Puppet Camps there will be this year - 27 was the number mentioned. I think a lot of the more experienced members of the community value the camps and confs as a chance to catch up with each other and the PuppetLabs people and I'd hate to see us sticking to our own local camps and losing the cross pollination of ideas, plans and pains.

You can also view the Puppet Camp slides for a number of the sessions.

So did someone discover @Pontifex contains horse meat?