Planet ALUG

Syndicate content
Planet ALUG - http://planet.alug.org.uk/
Updated: 1 hour 24 min ago

Steve Engledow (stilvoid): It turns

Sat, 04/04/2015 - 10:24

Over the past few months, I've mentioned to a few people that I like the idea of Firefox OS and that I've used the emulator and been fairly impressed. Well I've just put my money where my mouth is and ordered a Geeksphone revolution.

I'll admit that the decision to buy the phone wasn't purely to support Firefox OS; my hand was forced and I faced a dilemma:

After dropping my nexus 4 and cracking the screen (leaving the touchscreen only partially working), I found out that a repair with genuine parts would cost me nearly £100. My N4 being quite old now - and fairly crashy with lollipop - I couldn't quite justify the repair cost.

To be honest, I've been enjoying living without a usable phone except for the odd rare moments when I've needed to make a call while out. For that reason I decided that a brand new top of the range phone was out of the question.

I didn't want to go as far as getting a feature phone (I'm so used to having a smart phone now) and the Revolution seems like it offers a good compromise on cost vs. features.

So the intention is to live with Firefox OS but if it turns out to be ghastly, the phone will happily run AOSP and I can switch back and forth at will - yes, it even dual boots.

I'm sure I'll waffle on about my first impressions when it arrives...

Categories: LUG Community Blogs

Jonathan McDowell: Shipping my belongings across the globe

Tue, 31/03/2015 - 15:35

I previously wrote about tracking a ship around the world, but never followed up with the practical details involved with shipping my life from the San Francisco Bay Area back to Belfast. So here they are, in the hope they provide a useful data point for anyone considering a similar move.

Firstly, move out. I was in a one bedroom apartment in Fremont, CA. At the time I was leaving the US I didn’t have anywhere for my belongs to go - the hope was I’d be back in the Bay Area, but there was a reasonable chance I was going to end up in Belfast or somewhere in England. So on January 24th 2014 I had my all of my belongings moved out and put into storage, pending some information about where I might be longer term. When I say all of my belongings I mean that; I took 2 suitcases and everything else went into storage. That means all the furniture for probably a 2 bed apartment (I’d moved out of somewhere a bit larger) - the US doesn’t really seem to go in for the concept of a furnished lease the same way as the UK does.

I had deliberately picked a moving company that could handle the move out, the storage and the (potential) shipping. They handed off to a 3rd party for the far end bit, but that was to be expected. Having only one contact to deal with throughout the process really helped.

Fast forward 8 months and on September 21st I contacted my storage company to ask about getting some sort of rough shipping quote and timescales to Belfast. The estimate came back as around a 4-6 week shipping time, which was a lot faster than I was expecting. However it turned out this was the slow option. On October 27th (delay largely due to waiting for confirmation of when I’d definitely have keys on the new place) I gave the go ahead.

Container pickup (I ended up with exclusive use of a 20ft container - not quite full, but not worth part shipment) from the storage location was originally due on November 7th. Various delays at the Port of Oakland meant this didn’t happen until November 17th. It then sat in Oakland until December 2nd. At that point the ETA into Southampton was January 8th. Various other delays, including a week off the coast of LA (yay West Coast Port Backups) meant that the ship finally arrived in Southampton on January 13th. It then had to get to Belfast and clear customs. On January 22nd 2015, 2 days shy of a year since I’d seen them, my belongings and I were reunited.

So, on the face of it, the actual time on the ship was only slightly over 6 weeks, but all of the extra bits meant that the total time from “Ship it” to “I have it” was nearly 3 months. Which to be honest is more like what I was expecting. The lesson: don’t forget to factor in delays at every stage.

The relocation cost in the region of US$8000. It was more than I’d expected, but far cheaper than the cost of buying all my furniture again (plus the fact there were various things I couldn’t easily replace that were in storage). That cost didn’t cover the initial move into storage or the storage fees - it covered taking things out, packing them up for shipment and everything after that. Including delivery to a (UK) 3rd floor apartment at the far end and insurance. It’s important to note that I’d included this detail before shipment - the quote specifically mentioned it, which was useful when the local end tried to levy an additional charge for the 3rd floor aspect. They were fine once I showed them the quote as including that detail.

Getting an entire apartment worth of things I hadn’t seen in so long really did feel a bit like a second Christmas. I’d forgotten a lot of the things I had, and it was lovely to basically get a “home in a container” delivered.

Categories: LUG Community Blogs

Mick Morgan: the russians are back

Mon, 30/03/2015 - 09:50

About four years ago I was getting a huge volume of backscatter email to the non-existent address info@baldric.net. After a month or so it started to go quiet and eventually I got hardly any hits on that (or any other) address. A couple of weeks or so ago they came back. My logs for weeks ending 15 March, 22 March and 29 March show 92%, 96% and 94% respectively of all email to my main mail server is failed connection attempts from Russian domains to dear old non-existent “info”. Out of curiosity I decided to capture some of the inbound mails. Most were in Russian, but the odd one or two were in (broken) english. Below is a typical example:

From: “Olga”
To: Subject: Are you still looking for love? Look at my photos!
Date: Thu, 12 Mar 2015 15:22:08 +0300
X-Mailer: Microsoft Windows Live Mail 16.4.3528.331

Sunshine!
Are you still looking for love? I will be very pleased to become your half and save you from loneliness. My name is Olga, 25 years old.
For now I live in Russia, but it’s a bad time in my country, and I think about moving to another state.
I need a safer place for life, is your country good for that?
If you are interested and want to get in touch with me, just look at this international dating site.
Hope to see you soon!
Just click here!

Sadly, I believe that many recipients of such emails will indeed, “click here”. Certainly enough to further propagate whatever malware was used to compromise the end system which actually sent the above email.

Categories: LUG Community Blogs

Mick Morgan: kidnapped by aliens

Mon, 30/03/2015 - 09:21

An old friend of mine has expressed some concern at the lack of activity on trivia of late. In his most recent email to me he said:

“You really should revive Baldric you know. Everyone will believe it if you just say you were kidnapped by aliens, and then you can just resume where you left off.”

So Peter, this one is just for you. Oh, and Happy Birthday too.

Mick

Categories: LUG Community Blogs

Steve Engledow (stilvoid): Cleaning out my closet

Thu, 12/03/2015 - 15:59

Or: Finding out what crud you installed that's eating all of your space in Arch Linux

I started running out of space on one of my Arch boxes and wondered (beyond what was in my home directory) what I'd installed that was eating up all the space.

A little bit of bash-fu does the job:

for pkg in $(pacman -Qq); do size=$(pacman -Qi $pkg | grep "Installed Size" | cut -d ":" -f 2) echo "$size | $pkg" done | sed -e 's/ //g' | sort -h

This outputs a list of packages with those using the most disk space at the bottom:

25.99MiB|llvm-libs 31.68MiB|raspberrypi-firmware-examples 32.69MiB|systemd 32.86MiB|glibc 41.88MiB|perl 54.31MiB|gtk2 62.13MiB|python2 73.27MiB|gcc 77.93MiB|python 84.21MiB|linux-firmware

The above is from my pi; not much I can uninstall there ;)

Categories: LUG Community Blogs

Steve Engledow (stilvoid): Devicive

Thu, 12/03/2015 - 00:53

With all the tech world moving towards the idea that you have a single device that does everything, I've found myself suckered in to the convergence ideal in recent months. I was even genuinely excited by the recent video about Unity 8.

I have an Android phone that I use for a lot of purposes (nothing unusual: music, podcasts, messaging, web, phone calls) and a tablet that I use for more or less the same set of things with a bigger screen.

Yesterday, I managed to break my phone's screen leaving me with the horrifying prospect of a ride to work without being able to catch up on Linux Luddites (cha-ching) so I dug out my old mp3 player and it reminded me how clean and efficient the interface was.

Until a lot more work has happened, I've officially woken up from the convergence dream.

  • Reading

    I have a Kindle Paperwhite. I use it to read books. It's fantastic for that.

    The web browser is dire.

  • Web browsing

    I have a Nexus 7. I use it to surf the information superhighway. The web browser is great.

    It's not so good for reading books - the display is too bright (even on a low setting).

  • On-the-move Music / Podcasts

    I've fallen back in love with the MuVo and I think I'm going to stick with it for a while.

  • Development

    There really is no substitute for a screen and a keyboard. The laptop I have for work is perfect.

    It is, however, much to big to use in bed.

  • Gaming / home media

    I don't do much gaming so that doesn't warrant its own hardware but shares a home in the PC behind the telly which runs a Plex server and Kodi and has a big hard drive attached with all of our media on it.

  • Time

    I have a watch. It's a Pebble so it tells me when things are going on but it's mostly there to tell me what time it is (and relatedly, when I'm about to be late for a meeting).

I'm left feeling a little unsure of what I need a phone for now. Breaking it and subsequently not missing it one iota today speaks volumes. I barely use SMS (I mostly stay in contact through various other means) and I don't make or receive enough phone calls for it to seem worth carting an expensive oblong around.

I'm sure I'll change my mind soon.

Categories: LUG Community Blogs

Ben Francis: Pinned Apps – An App Model for the Web

Mon, 09/03/2015 - 16:54

(re-posted from a page I created on the Mozilla wiki on 17th December 2014)

Problem Statement

The per-OS app store model has resulted in a market where a small number of OS companies have a large amount of control, limiting choice for users and app developers. In order to get things done on mobile devices users are restricted to using apps from a single app store which have to be downloaded and installed on a compatible device in order to be useful.

Design Concept Concept Overview

The idea of pinned apps is to turn the apps model on its head by making apps something you discover simply by searching and browsing the web. Web apps do not have to be installed in order to be useful, “pinning” is an optional step where the user can choose to split an app off from the rest of the web to persist it on their device and use it separately from the browser.

”If you think of the current app store experience as consumers going to a grocery store to buy packaged goods off a shelf, the web is more like a hunter-gatherer exploring a forest and discovering new tools and supplies along their journey.”

App Discovery

A Web App Manifest linked from a web page says “I am part of a web app you can use separately from the browser”. Users can discover web apps simply by searching or browsing the web, and use them instantly without needing to install them first.

”App discovery could be less like shopping, and more like discovering a new piece of inventory while exploring a new level in a computer game.”

App Pinning

If the user finds a web app useful they can choose to split it off from the rest of the web to persist it on their device and use it separately from the browser. Pinned apps can provide a more app-like experience for that part of the web with no browser chrome and get their own icon on the homescreen.

”For the user pinning apps becomes like collecting pin badges for all their favourite apps, rather than cluttering their device with apps from an app store that they tried once but turned out not to be useful.”

Deep Linking

Once a pinned app is registered as managing its own part of the web (defined by URL scope), any time the user navigates to a URL within that scope, it will open in the app. This allows deep linking to a particular page inside an app and seamlessly linking from one app to another.

”The browser is like a catch-all app for pages which don’t belong to a particular pinned app.”

Going Offline

Pinning an app could download its contents to the device to make it work offline, by registering a Service Worker for the app’s URL scope.

”Pinned apps take pinned tabs to the next level by actually persisting an app on the device. An app pin is like an anchor point to tether a collection of web pages to a device.”

Multiple Pages

A web app is a collection of web pages dedicated to a particular task. You should be able to have multiple pages of the app open at the same time. Each app could be represented in the task manager as a collection of sheets, pinned together by the app.

”Exploding apps out into multiple sheets could really differentiate the Firefox OS user experience from all other mobile app platforms which are limited to one window per app.”

Travel Guide

Even in a world without app stores there would still be a need for a curated collection of content. The Marketplace could become less of a grocery store, and more of a crowdsourced travel guide for the web.

”If a user discovers an app which isn’t yet included in the guide, they could be given the opportunity to submit it. The guide could be curated by the community with descriptions, ratings and tags.”

3 Questions

What value (the importance, worth or usefulness of something) does your idea deliver?

The pinned apps concept makes web apps instantly useful by making “installation” optional. It frees users from being tied to a single app store and gives them more choice and control. It makes apps searchable and discoverable like the rest of the web and gives developers the freedom of where to host their apps and how to monetise them. It allows Mozilla to grow a catalogue of apps so large and diverse that no walled garden can compete, by leveraging its user base to discover the apps and its community to curate them.

What technological advantage will your idea deliver and why is this important?

Pinned apps would be implemented with emerging web standards like Web App Manifests and Service Workers which add new layers of functionality to the web to make it a compelling platform for mobile apps. Not just for Firefox OS, but for any user agent which implements the standards.

Why would someone invest time or pay money for this idea?

Users would benefit from a unique new web experience whilst also freeing themselves from vendor lock-in. App developers can reduce their development costs by creating one searchable and discoverable web app for multiple platforms. For Mozilla, pinned apps could leverage the unique properties of the web to differentiate Firefox OS in a way that is difficult for incumbents to follow.

UI Mockups App Search

Pin App

Pin Page

Multiple Pages

App Directory

Implementation Web App Manifest

A manifest is linked from a web page with a link relation:

<link rel=”manifest” href=”/manifest.json”>

A manifest can specify an app name, icon, display mode and orientation:

{ "name": "GMail" "icons": {...}, "display": "standalone", "orientation": “portrait”, ... }

There is a proposal for a manifest to be able to specify an app scope:

{ ... "scope": "/" ... }

Service Worker

There is also a proposal to be able to reference a Service Worker from within the manifest:

{ ... service_worker: { src: "app.js", scope: "/" ... }

A Service Worker has an install method which can populate a cache with a web app’s resources when it is registered:

this.addEventListener('install', function(event) { event.waitUntil( caches.create('v1').then(function(cache) { return cache.add( '/index.html', '/style.css', '/script.js', '/favicon.ico' ); }, function(error) { console.error('error populating cache ' + error); }; ); });

So that the app can then respond to requests for resources when offline:

this.addEventListener('fetch', function(event) { event.respondWith( caches.match(event.request).catch(function() { return event.default(); }) ); });

Categories: LUG Community Blogs

MJ Ray: Rebooting democracy? The case for a citizens constitutional convention.

Fri, 20/02/2015 - 05:03

I’m getting increasingly cynical about our largest organisations and their voting-centred approach to democracy. You vote once, for people rather than programmes, then you’re meant to leave them to it for up to three years until they stand for reelection and in most systems, their actions aren’t compared with what they said they’d do in any way.

I have this concern about Cooperatives UK too, but then its CEO publishes http://www.uk.coop/blog/ed-mayo/2015-02-18/rebooting-democracy-case-citizens-constitutional-convention and I think there may be hope for it yet. Well worth a read if you want to organise better groups.

Categories: LUG Community Blogs

Chris Lamb: Calculating the ETA to zero in shell

Fri, 30/01/2015 - 21:49
< Faux> I have a command which emits a number. This number is heading towards zero. I want to know when it will arrive at zero, and how close to zero it has got.

Damn right you can.

eta2zero () { A=$(eval ${@}) while [ ${A} -gt 0 ] do B=$(eval ${@}) printf %$((${A} - ${B}))s A=${B} sleep 1 done | pv -s ${A} >/dev/null }

In action:

$ rm -rf /big/path & [1] 4895 $ eta2zero find /big/path \| wc -l 10 B 0:00:14 [ 0 B/s] [================================> ] 90% ETA 0:00:10

(Sincere apologies for the lack of strace...)

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Caius -- A heirarchical delegable password safe

Thu, 29/01/2015 - 14:08

A long while ago I, Rob Kendrick, Clive Jones (and possibly others) sat down and tried to come up with a way to store passwords a-la Password Safe. However, being us, we wanted to ensure a number of properties which password safes commonly don't have. We wanted to allow the delegation of access to some subset of the passwords. We also wanted for it to be reasonable to deny that there is content which has not been decrypted.

I was reminded of this work when I was discussing the concept of deniable storage of secrets with a colleague (An idea I'll expand upon in another blog post at another time). I am therefore presenting, with little change other than formatting the design from years ago. I would be very interested if anyone knows of software which meets the properties of the Caius system since I would like to have one but simply don't trust myself (see another future posting) to write it right now.

Caius

The following concepts are assumed to be understood:

The 'Caius' system is a password-safe type system sporting hierarchical delegable access to the data it stores. The 'Caius Fob' is the data-store for the system.

The 'Caius Fob' is a file which consists of a header and then three sections. The header identifies it as such a file, the first section lists a number of 'External IDs' which can be used to access portions of the file. The second section lists ACL entries as defined below. The third section of the file is the encrypted data looked after by this file. It is not intended that the holder of a CaiusFob be able to deny it is a CaiusFob, but it is expected that it be possible to deny an ability to decrypt (perhaps by lacking a password) any ACL entries. Given that the structure of the file is known, it is necessary that there be external IDs for which the password or GPG key is not valid or cannot decrypt an ACL entry, and ACL entries which even if decrypted may not be valid, and ACL entries which even if decrypted and valid may not be used to encode any data blocks.

An External ID External ID ::= LENGTH TYPE DATA

Where TYPE is one of: * 0: Unused (ID slot placeholder) * 1: GPG key, where DATA is the keyid of the gpg key * 2: Password, where DATA is some hash of a password which can be used to derive a key for decrypting an ACL entry.

The list of external ids forms a numbered sequence where the index (0-based) into the sequence is the External ID number. (EIDnr)

An ACL Entry ACL Entry ::= LENGTH EIDnr DATA HMAC

The EIDnr is the number of the External ID as explained above. The LENGTH is the length of the DATA section which is a key-pair as explained below, encrypted to the external id. The HMAC uses the authentication key in the key-pair in the DATA section, and authenticates the EIDnr, LENGTH, DATA tuple.

One possibility for increasing deniability is to remove the EIDnr from this part of the file, and assume that for every external ID you try to decrypt all ACLs you've not succeeded in decrypting thus-far. This has the benefit of being able to deny that an ACL entry ought to be decryptable with the credentials you hold, but also an increased inability to know if you have successfully unlocked everything appropriate to being able to fully manipulate a CaiusFob. This tradeoff is currently set in favour of better understanding of the content, but a future design feature might suggest EIDnr should always be -1 to indicate "unknown, try every EID".

A key pair Key Pair ::= ENCRYPTIONKEY AUTHENTICATIONKEY

The ENCRYPTIONKEY is used to initialise the stream cipher for the data section. The AUTHENTICATIONKEY is used to compute HMACs for the appropriate ACL entries or data blocks (as defined below)

The data section

First consider a set of stream ciphers. There exists always one cipher which we will call the NULL cipher. It is defined such that Cipher(Byte, Offset) == Byte and is always available. Then there is a cipher initialised for each key pair we can successfully extract from the ACL entry section of the file.

Each of these ciphers is initialised and made ready such that the data section can be xored with the bytes as they come out of the stream ciphers in an attempt to decrypt the blocks. Essentially this is equivalent to decrypting the entire data section with each cipher in turn to produce N proposed cleartexts which can then be examined to find decrypted blocks.

Whenever a cipher, combined with the data stream in the file, manages to produce a sequence of eight bytes of value zero, we have reached a synchronisation point and what follows is a data block enciphered with which ever cipher managed to reveal the synchronisation sequence.

Since it is vanishingly unlikely that you will find eight zeros in a row when playing about with arbitrary cipher initialisation, we consider this to be an acceptable synchronisation sequence.

Once we have found a sync sequence, we can know the length of this block and thus we do not look for sync markers again until after the block we have just found.

A data block Data block ::= DATALENGTH PADLENGTH TYPE DATA PAD HMAC

Such that each field is the obvious, DATA is DATALENGTH bytes of data, the texture of which is defined by TYPE and PAD is PADLENGTH arbitrary bytes which pad this block. HMAC is keyed using the authentication key associated with the stream cipher managing to decrypt this and is over the DATALENGTH, PADLENGTH, TYPE, DATA, PAD tuple.

If TYPE is zero then this is a ''free-space'' block and will typically contain zero bytes of DATA and some number of padding. This is however arbitrary and not enforced, the free space can be DATA if the implementer prefers and implementations are encouraged if possible to randomise the distribution of the consumed space between the DATA and PAD sections.

A node block TYPE == 1 (Node) DATA ::= MY_ID PARENT_ID NAME NOTES

MY_ID is a unique ID for this node. (generally a random number, probably 64 bits long, perhaps a UUID). PARENT_ID if not all NULLs is the ID for the parent node. If all NULLs then this is the root of a heirarchy. NAME is a NULL terminated byte string of one or more characters which is the name of this node. It may consist only of any characters other than NULL and the forward-slash character. NOTES is a byte string of zero or more characters, NULL terminated. Note that the DATALENGTH of the data block clearly delimits this field but the NULL is present to aid parsing.

A system block TYPE == 2 (System) DATA ::= PARENT_ID USERNAME PASSWORD EXPIRYDATE NOTES

PARENT_ID is the node to which this block belongs. It is required that any system blocks you succeed in decrypting can be placed within a node you succeed in decrypting. If the library encounters a system block which belongs to a node it cannot find then this is considered to be a corrupt system block and will be treated as though it could not be decrypted.

USERNAME is a byte string of one or more characters terminated by a NULL, ditto for PASSWORD and as for a node block, the NOTES are NULL terminated also.

EXPIRYDATE is a byte string of characters in RFC-822 date format.

Implementation notes

It is expected that any implementation of Caius will adhere to the following guidelines in order to enhance the security of content over time.

  1. Any time a block is invalidated (such as by the changing of a password, the obsoleting of an entry, the changing of notes, names, or reparenting a node) anywhere from one to all of the bits in the block can be changed. Trivially, this includes the synchronisation sequence preceeding the block as if the synchronisation sequence isn't present then the block does not exist.

  2. Every time a CaiusFob is altered in any way, some amount of known intra-block padding must be altered in a random way. Ideally this will be done so that it looks like number 1 has happened somewhere else in the file as well. Anywhere from zero to all of the padding can be thusly altered in any given change.

  3. No attempt will be made to write to any part of the file which cannot be positively identified as padding unless the user has explicitly stated that they will accept damage to data they cannot currently decrypt.

  4. No indication will be given to the user if any part of the file was unable to be decrypted or failed an HMAC check. Such data is simply incorrectly decrypted and thus ignored.

  5. Intrablock padding can be positively identified if you have two consecutive blocks in a CaiusFob such that the number of bytes between them could not possibly hold the simplest of free space blocks.

  6. When appending a block to a CaiusFob it is encouraged to place up to 50% of the size of the intrablock spacing before it as random padding, and up to 50% afterwards also. Naturally anywhere between zero and the full amount is acceptable, ideally the implementation would choose at random each time.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): I promise that I...

Tue, 27/01/2015 - 23:27

A friend and ex-colleague Francis Irving (@frabcus on Twitter) has recently been on a bit of an anti C/C++ kick, including tweeting about the problems which happen in software written in so called "insecure" languages, and culminating in his promise website which boldly calls for people to promise to not use C/C++ for new projects.

Recently I've not been programming enough. I'm still a member of the NetSurf browser project, and I'm still (slowly) working on Gitano from time to time. I am still (in theory) an upstream on the Cherokee Webserver project (and I really do need to sit down and fix some bugs in logging) and any number of smaller projects as well. I am still part of Debian and would like to start making positive contributions beyond voting and egging others on, but I have been somewhat burned out by everything going on in my life, including both home and work. While I am hardly in any kind of mental trouble, I've simply not had any tuits of late.

I find it very hard to make public promises which I know I am going to break. Francis suggested that the promise can be broken which while it might not devalue it for him (or you) it does for me. I do however think that public promises are a good thing, especially when they foster useful discussion in the communities I am part of, so from that point of view I very much support Francis in his efforts.

Even given all of the above, I'd like to make a promise statement of my own. I'd like to make it in public and hopefully that'll help me to keep it. I know I could easily fail to live up to this promise, but I'm hoping I'll do well and to some extent I'm relying on all you out there to help me keep it. Given we're almost at the end of the month, I am making the promise now and I want it to take effect starting on the 1st of February 2015.

I hereby promise that I will do better at contributing to all the projects I am nominally part of, making at least one useful material contribution to at least one project every week. I also promise to be more mindful of the choices I make when choosing how to implement solutions to problems, selecting appropriate languages and giving full consideration to how my solution might be attacked if appropriate.

I can't and won't promise not to use C/C++ but if you honestly feel you can make that promise, then I'm certain Francis would love for you to head over to his promise website and pledge. Also regardless of your opinions, please do join in the conversation, particularly regarding being mindful of attack vectors whenever you write something.

Categories: LUG Community Blogs

Chris Lamb: Recent Redis hacking

Sun, 25/01/2015 - 21:52

I've done a bunch of hacking on the Redis key/value database server recently:

  • Lua-based maxmemory eviction scripts. (#2319)

    (This changeset was sponsored by an anonymous client.)

    Redis typically stores the entire data set in memory, using the operating system's virtual memory facilities if required. However, one can use Redis more like a cache or ring buffer by enabling a "maxmemory policy" where a RAM limit is set and then data is evicted when required based on a predefined algorithm.

    This change enables entirely custom control over exactly what data to remove from RAM when this maxmemory limit is reached. This is an advantage over the existing policies of, say, removing entire keys based on the existing TTL, Least Recently Used (LRU) or random eviction strategies as it permits bespoke behaviours based on application-specific requirements, crucially without maintaining a private fork of Redis.

    As an example behaviour of what is possible with this change, to remove the lowest ranked member of an arbitrary sorted set, you could load the following eviction policy script:

    local bestkey = nil local bestval = 0 for s = 1, 5 do local key = redis.call("RANDOMKEY") local type_ = redis.call("TYPE", key) if type_.ok == "zset" then local tail = redis.call("ZRANGE", key, "0", "0", "WITHSCORES") local val = tonumber(tail[2]) if not bestkey or val < bestval then bestkey = key bestval = val end end end if not bestkey then -- We couldn't find anything to remove, so return an error return false end redis.call("ZREMRANGEBYRANK", bestkey, "0", "0") return true
  • TCP_FASTOPEN support. (#2307)

    The aim of TCP_FASTOPEN is to eliminate one roundtrip from a TCP conversation by allowing data to be included as part of the SYN segment that initiates the connection. (More info.)

  • Support infinitely repeating commands in redis-cli. (#2297)

  • Add --failfast option to testsuite runner. (#2290)

  • Add a -q (quiet) argument to redis-cli. (#2305)

  • Making some Redis Sentinel defaults a little saner. (#2292)

I also made the following changes to the Debian packaging:

  • Add run-parts(8) directories to be executed at various points in the daemon's lifecycle. (e427f8)

    This is especially useful for loading Lua scripts as they are not persisted across restarts.

  • Split out Redis Sentinel into its own package. (#775414, 39f642)

    This makes it possible to run Sentinel sanely on Debian systems without bespoke scripts, etc.

  • Ensure /etc/init.d/redis-server start idempotency with --oknodo (60b7dd)

    Idempotency in initscripts is especially important given the rise of configuration managment systems.

  • Uploaded 3.0.0 RC2 to Debian experimental. (37ac55)

  • Re-enabled the testsuite. (7b9ed1)

Categories: LUG Community Blogs

Chris Lamb: Slack integration for Django

Fri, 23/01/2015 - 23:46

I recently started using the Slack group chat tool in a few teams. Wishing to add some vanity notifications such as sales and user growth milestones from some Django-based projects, I put together an easy-to-use integration between the two called django-slack.

Whilst you can use any generic Python-based method of sending messages to Slack, using a Django-specific integration has some advantages:

  • It can use the Django templating system, rather than constructing messages "by hand" in views.py and models.py which violates abstraction layers and often requires unwieldy and ugly string manipulation routines that would be trivial inside a regular template.
  • It can easily enabled and disabled in certain environments, preventing DRY violations by centralising logic to avoid sending messages in development, staging environments, etc.
  • It can use other Django idioms such as a pluggable backend system for greater control over exactly how messages are transmitted to the Slack API (eg. sent asynchronously using your queuing system, avoiding slowing down clients).

Here is an example of how to send a message from a Django view:

from django_slack import slack_message @login_required def view(request, item_id): item = get_object_or_404(Item, pk=item_id) slack_message('items/viewed.slack', { 'item': item, 'user': request.user, }) return render(request, 'items/view.html', { 'item': item, })

Where items/viewed.slack (in your templates directory) might contain:

{% extends django_slack %} {% block text %} {{ user.get_full_name }} just viewed {{ item.title }} ({{ item.content|urlize }}). {% endblock %}

.slack files are regular Django templates — text is automatically escaped as appropriate and that you can use the regular template filters and tags such as urlize, loops, etc.

By default, django-slack posts to the #general channel, but it can be overridden on a per-message basis by specifying a channel block:

{% block channel %} #mychannel {% endblock %}

You can also set the icon, URL and emoji in a similar fashion. You can set global defaults for all of these attributes to avoid DRY violations within .slack templates as well.

For more information please see the project homepage or read the documentation. Patches and other contributions are welcome via the django-slack GitHub project.

Categories: LUG Community Blogs

MJ Ray: Outsourcing email to Google means SPF allows phishing?

Thu, 22/01/2015 - 04:57

I expect this is obvious to many people but bahumbug To Phish, or Not to Phish? just woke me up to the fact that if Google hosts your company email then its Sender Policy Framework might make other Google-sent emails look legitimate for your domain. When combined with the unsupportive support of the big free webmail hosts, is this another black mark against SPF?

Categories: LUG Community Blogs

Chris Lamb: Sprezzatura

Wed, 21/01/2015 - 11:31

Wolf Hall on Twitter et al:

He says, "Majesty, we were talking of Castiglione's book. You have found time to read it?"

"Indeed. He extrolls sprezzatura. The art of doing everything gracefully and well, without the appearance of effort. A quality princes should cultivate."

"Yes. But besides sprezzatura one must exhibit at all times a dignified public restraint..."

Categories: LUG Community Blogs

Jonathan McDowell: Moving to Jekyll

Wed, 21/01/2015 - 11:00

I’ve been meaning to move away from Movable Type for a while; they no longer provide the “Open Source” variant, I’ve had some issues with the commenting side of things (more the fault of spammers than Movable Type itself) and there are a few minor niggles that I wanted to resolve. Nothing has been particularly pressing me to move and I haven’t been blogging as much so while I’ve been keeping an eye open for a replacement I haven’t exerted a lot of energy into the process. I have a little bit of time at present so I asked around on IRC for suggestions. One was ikiwiki, which I use as part of helping maintain the SPI website (and think is fantastic for that), the other was Jekyll. Both are available as part of Debian Jessie.

Jekyll looked a bit fancier out of the box (I’m no web designer so pre-canned themes help me a lot), so I decided to spend some time investigating it a bit more. I’d found a Movable Type to ikiwiki converter which provided a starting point for exporting from the SQLite3 DB I was using for MT. Most of my posts are in markdown, the rest (mostly from my Blosxom days) are plain HTML, so there wasn’t any need to do any conversion on the actual content. A minor amount of poking convinced Jekyll to use the same URL format (permalink: /:year/:month/:title.html in the _config.yml did what I wanted) and I had to do a few bits of fix up for some images that had been uploaded into MT, but overall fairly simple stuff.

Next I had to think about comments. My initial thought was to just ignore them for the moment; they weren’t really working on the MT install that well so it’s not a huge loss. I then decided I should at least see what the options were. Google+ has the ability to embed in your site, so I had a play with that. It worked well enough but I didn’t really want to force commenters into the Google ecosystem. Next up was Disqus, which I’ve seen used in various places. It seems to allow logins via various 3rd parties, can cope with threading and deals with the despamming. It was easy enough to integrate to play with, and while I was doing so I discovered that it could cope with importing comments. So I tweaked my conversion script to generate a WXR based file of the comments. This then imported easily into Disqus (and also I double checked that the export system worked).

I’m sure the use of a third party to handle comments will put some people off, but given the ability to export I’m confident if I really feel like dealing with despamming comments again at some point I can switch to something locally hosted. I do wish it didn’t require Javascript, but again it’s a trade off I’m willing to make at present.

Anyway. Thanks to Tollef for the pointer (and others who made various suggestions). Hopefully I haven’t broken (or produced a slew of “new” posts for) any of the feed readers pointed at my site (but you should update to use feed.xml rather than any of the others - I may remove them in the future once I see usage has died down).

(On the off chance it’s useful to someone else the conversion script I ended up with is available. There’s a built in Jekyll importer that may be a better move, but I liked ending up with a git repository containing a commit for each post.)

Categories: LUG Community Blogs

Chris Lamb: Adjusting a backing track with SoX

Sun, 18/01/2015 - 13:28

Earlier today I came across some classical sheet music that included a "playalong" CD, just like a regular recording except it omits the solo cello part. After a quick listen it became clear there were two problems:

  • The recording was made at A=442, rather than the more standard A=440.
  • The tempi of the movements was not to my taste, either too fast or too slow.

SoX, the "Swiss Army knife of sound processing programs", can easily adjust the latter, but to remedy the former it must be provided with a dimensionless "cent" unit—ie. 1/100th of a semitone—rather than the 442Hz and 440Hz reference frequencies.

First, we calculate the cent difference with:

Next, we rip the material from the CD:

$ sudo apt-get install ripit flac [..] $ ripit --coder 2 --eject --nointeraction [..]

And finally we adjust the tempo and pitch:

$ apt-get install sox libsox-fmt-mp3 [..] $ sox 01.flac 01.mp3 pitch -7.85 tempo 1.00 # (Tuning notes) $ sox 02.flac 02.mp3 pitch -7.85 tempo 0.95 # Too fast! $ sox 03.flac 03.mp3 pitch -7.85 tempo 1.01 # Close.. $ sox 04.flac 04.mp3 pitch -7.85 tempo 1.03 # Too slow!

(I'm converting to MP3 at the same time it'll be more convenient on my phone.)

Categories: LUG Community Blogs

Jonathan McDowell: Tracking a ship around the world

Tue, 13/01/2015 - 17:07

I moved back from the California Bay Area to Belfast a while back and for various reasons it looks like I’m going to be here a while, so it made sense to have my belongings shipped over here. They haven’t quite arrived yet, and I’ll do another post about that process once they have, but I’ve been doing various tweets prefixed with “[shipping]” during the process. Various people I’ve spoken to (some who should know me better) thought this was happening manually. It wasn’t. If you care about how it was done, read on.

I’d been given details of the ship carrying my container, and searching for that turned up the excellent MarineTraffic which let me see the current location of the ship. Turns out ships broadcast their location using AIS and anyone with a receiver can see the info. Very cool, and I spent some time having a look at various bits of shipping around the UK out of interest. I also found the ship’s itinerary which give me some idea of where it would be calling and when. Step one was to start recording this data; it was time sensitive and I wanted to be able to see historical data. I took the easy route and set up a cron job to poll the location and itinerary on an hourly basis, and store the results. That meant I had the data over time, if my parsing turned out to miss something I could easily correct it, and that I wasn’t hammering Marine Traffic while writing the parsing code.

Next I wanted to parse the results, store them in a more manageable format than the HTML, and alert me when the ship docked somewhere or set off again. I’ve been trying to learn more Python rather than doing my default action of turning to Perl for these things, and this seemed like a simple enough project to try out. Beautiful Soup seemed to turn up top for HTML parsing in Python, so that formed the basis. Throwing the info into a database so I could do queries felt like the right move so I used SQLite - if this had been more than a one off I’d have considered looking at PostgreSQL and its GIS support. Finally Tweepy made it very easy to tweet from Python in about 4 lines of code. The whole thing weighed in at only 175 lines of code, mostly around pulling the info out of the HTML and then a little to deal with checking for state changes against the current status and the last piece of info in the database.

The pieces of information I chose to store were the time of the update (i.e. when the ship sent it, not when my script ran), reported area, reported state, the position + course, reported origin, reported destination and eta. The fact this is all in a database makes it very easy to do a few queries on the data.

How fast did the ship go?

sqlite> SELECT MAX(speed) FROM status; MAX(speed) 21.9

What areas did it report?

sqlite> SELECT area FROM status GROUP BY area; area - Atlantic North California Caribbean Sea Celtic Sea English Channel Hudson River Pacific North Panama Canal

What statuses did we see?

sqlite> SELECT status FROM status GROUP BY status; status At Anchor Moored Stopped Underway Underway using Engine

Finally having hourly data lets me draw a map of where the ship went. The data isn’t complete, because the free AIS info depends on the ship being close enough to a receiving station. That means there were a few days in the North Atlantic without updates, for example. However there’s enough to give a good idea of just how well traveled my belongings are, and it gave me an excuse to play with OpenLayers.

(Apologies if the zoom buttons aren’t working for you here; I think I need to kick the CSS in some manner I haven’t quite figured out yet.)

Categories: LUG Community Blogs

Chris Lamb: Giant's Causeway puzzle

Mon, 12/01/2015 - 11:34

Over Christmas I found the above puzzle at my parent's house. I don't know if it has a name but it seemed apt to name it after The Giant's Causeway.

The idea is that you swap and/or rotate the outer tiles until the colours at the edges match. The center tile is fixed, or at least I assume it is as the wooden edges are deliberately less rounded.

Of course, solving it manually would be boring so here's a dumb brute force solution. I actually admire how inefficient and stupid it is...

import itertools class Tile(list): def __getitem__(self, x): return super(Tile, self).__getitem__(x % len(self)) def rotate(self, x): return Tile(self[x:] + self[:x]) COLOURS = ('YELLOW', 'WHITE', 'BLACK', 'RED', 'GREEN', 'BLUE') YELLOW, WHITE, BLACK, RED, GREEN, BLUE = range(len(COLOURS)) TILES = ( Tile((WHITE, YELLOW, RED, BLUE, BLACK, GREEN)), Tile((RED, BLUE, BLACK, YELLOW, GREEN, WHITE)), Tile((WHITE, BLACK, YELLOW, GREEN, BLUE, RED)), Tile((WHITE, BLUE, GREEN, YELLOW, BLACK, RED)), Tile((GREEN, BLUE, BLACK, YELLOW, RED, WHITE)), Tile((RED, YELLOW, GREEN, BLACK, BLUE, WHITE)), ) CENTER = Tile((WHITE, BLACK, RED, GREEN, BLUE, YELLOW)) def pairwise(it): a, b = itertools.tee(it) next(b, None) return itertools.izip(a, b) def validate(xs): for idx, x in enumerate(xs): if x[idx + 3] != CENTER[idx]: raise ValueError("Tile does not match center") for idx, (a, b) in enumerate(pairwise(xs)): if a[idx + 2] != b[idx + 5]: raise ValueError("Tile does not match previous anticlockwise tile") return xs def find_solution(): # For all possible tile permutations.. for xs in itertools.permutations(TILES): # ... try all possible rotations for ys in itertools.product(range(len(COLOURS)), repeat=len(COLOURS)): try: return validate([x.rotate(y) for x, y in zip(xs, ys)]) except ValueError: pass raise ValueError("Could not find a solution.") for x in find_solution(): print ', '.join(COLOURS[y] for y in x)

This prints (after 5 minutes!):

$ python giants.py GREEN, BLUE, RED, WHITE, BLACK, YELLOW WHITE, BLUE, GREEN, YELLOW, BLACK, RED YELLOW, GREEN, BLACK, BLUE, WHITE, RED GREEN, WHITE, YELLOW, RED, BLUE, BLACK RED, BLUE, BLACK, YELLOW, GREEN, WHITE BLUE, BLACK, YELLOW, RED, WHITE, GREEN python giants.py 269.08s user 0.02s system 99% cpu 4:29.36 total

UPDATE: Seems like this puzzle is called "Circus Seven". Now to dust off my Prolog...

Categories: LUG Community Blogs

Chris Lamb: Uploading a large number of files to Amazon S3

Fri, 09/01/2015 - 14:31

I recently had to upload a large number (~1 million) of files to Amazon S3.

My first attempts revolved around s3cmd (and subsequently s4cmd) but both projects seem to based around analysing all the files first, rather than blindly uploading them. This not only requires a large amount of memory, non-trivial experimentation, fiddling and patching is also needed to avoid unnecessary stat(2) calls. I even tried a simple find | xargs -P 5 s3cmd put [..] but I just didn't trust the error handling correctly.

I finally alighted on s3-parallel-put, which worked out well. Here's a brief rundown on how to use it:

  1. First, change to your source directory. This is to ensure that the filenames created in your S3 bucket are not prefixed with the directory structure of your local filesystem — whilst s3-parallel-put has a --prefix option, it is ignored if you pass a fully-qualified source, ie. one starting with a /.
  2. Run with --dry-run --limit=1 and check that the resulting filenames will be correct after all:
$ export AWS_ACCESS_KEY_ID=FIXME $ export AWS_SECRET_ACCESS_KEY=FIXME $ /path/to/bin/s3-parallel-put \ --bucket=my-bucket \ --host=s3.amazonaws.com \ --put=stupid \ --insecure \ --dry-run --limit=1 \ . [..] INFO:s3-parallel-put[putter-21714]:./yadt/profile.Profile/image/circle/807.jpeg -> yadt/profile.Profile/image/circle/807.jpeg [..]
  1. Remove --dry-run --limit=1, and let it roll.
Categories: LUG Community Blogs