Continuing the theme from the last post I made, I've recently started working my way down the list of existing object-storage implementations.
tahoe-LAFS is a well-established project which looked like a good fit for my needs:
Getting the system up and running, on four nodes, was very simple. Setup a single/simple "introducer" which is a well-known node that all hosts can use to find each other, and then setup four deamons for storage.
When files are uploaded they are split into chunks, and these chunks are then distributed amongst the various nodes. There are some configuration settings which determine how many chunks files are split into (10 by default), how many chunks are required to rebuild the file (3 by default) and how many copies of the chunks will be created.
The biggest problem I have with tahoe is that there is no rebalancing support: Setup four nodes, and the space becomes full? You can add more nodes, new uploads go to the new nodes, while old ones stay on the old. Similarly if you change your replication-counts because you're suddenly more/less paranoid this doesn't affect existing nodes.
In my perfect world you'd distribute blocks around pretty optimistically, and I'd probably run more services:
The storage nodes would have the primitives "List all blocks", "Get block", "Put block", and using that you could ensure that each node had sent its data to at least N other nodes. This could be done in the background.
The indexer would be responsible for keeping track of which blocks live where, and which blocks are needed to reassemble upload N. There's probably more that it could do.
Over the bank holiday weekend I made two batches of jam: rhubarb & ginger and rhubarb & orange. I made a small batch last year - which we've not yet eaten - but it's quite a while since I've made so much and for sale.
This year I remembered to grade the rhubarb first, so that each batch was made from stems of similar diameter, which means that they cook evenly and you don't end up with a heterogeneous mixture - which is bad.
If you use a Linux or Unix box with bash or zsh, and you haven’t come across Liquid Prompt, then I suggest you head there right now to install it. I’m loving having more info on the status line, especially near code version control, but even having cpu load and temperature along with battery life right under where I am typing is really useful
As I slowly upgrade all my machines to Debian 8.0 (jessie) they’re all ending up with systemd. That’s fine; my laptop has been running it since it went into testing whenever it was. Mostly I haven’t had to care, but I’m dimly aware that it has a lot of bits I should learn about to make best use of it.
Today I discovered systemctl is-system-running. Which I’m not sure why I’d use it, but when I ran it it responded with degraded. That’s not right, thought I. How do I figure out what’s wrong? systemctl --state=failed turned out to be the answer.# systemctl --state=failed UNIT LOAD ACTIVE SUB DESCRIPTION ● systemd-modules-load.service loaded failed failed Load Kernel Modules LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 1 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'.
Ok, so it’s failed to load some kernel modules. What’s it trying to load? systemctl status -l systemd-modules-load.service led me to /lib/systemd/systemd-modules-load which complained about various printer modules not being able to be loaded. Turned out this was because CUPS had dropped them into /etc/modules-load.d/cups-filters.conf on upgrade, and as I don’t have a parallel printer I hadn’t compiled up those modules. One of my other machines had also had an issue with starting up filesystem quotas (I think because there’d been some filesystems that hadn’t mounted properly on boot - my fault rather than systemd). Fixed that up and then systemctl is-system-running started returning a nice clean running.
Now this is probably something that was silently failing back under sysvinit, but of course nothing was tracking that other than some output on boot up. So I feel that I’ve learnt something minor about systemd that actually helped me cleanup my system, and sets me in better stead for when something important fails.
I was first elected to the Software in the Public Interest board back in 2009. I was re-elected in 2012. This July I am up for re-election again. For a variety of reasons I’ve decided not to stand; mostly a combination of the fact that I think 2 terms (6 years) is enough in a single stretch and an inability to devote as much time to the organization as I’d like. I mentioned this at the May board meeting. I’m planning to stay involved where I can.
My main reason for posting this here is to cause people to think about whether they might want to stand for the board. Nominations open on July 1st and run until July 13th. The main thing you need to absolutely commit to is being able to attend the monthly board meeting, which is held on IRC at 20:30 UTC on the second Thursday of the month. They tend to last at most 30 minutes. Of course there’s a variety of tasks that happen in the background, such as answering queries from prospective associated projects or discussing ongoing matters on the membership or board lists depending on circumstances.
It’s my firm belief that SPI do some very important work for the Free software community. Few people realise the wide variety of associated projects. SPI offload the boring admin bits around accepting donations and managing project assets (be those machines, domains, trademarks or whatever), leaving those projects able to concentrate on the actual technical side of things. Most project members don’t realise the involvement of SPI, and that’s largely a good thing as it indicates the system is working. However it also means that there can sometimes be a lack of people wanting to stand at election time, and an absence of diversity amongst the candidates.
I’m happy to answer questions of anyone who might consider standing for the board; #spi on irc.oftc.net is a good place to ask them - I am there as Noodles.
Time has come once again to announce WYLUG’s monthly meeting. As usual it will be held in the Lord Darcy. As usual it will start at around 7pm. There will be no obligation to book in advance. However, the man with the red hat lanyard will not be there to guide you steps, this month.
It turns out that a raspberry pi does a very good job of being a print server for a google cloud printer. Thanks to https://matthew.mceachen.us/blog/add-google-cloudprint-wifi-access-to-your-older-printer-with-a-raspberry-pi-1342.html I can now print at home directly from my phone!
Update: Replacing the battery and retraining the receiver fixed it. I suppose it must have had enough juice to flash the LED but not transmit.
A few days ago my CurrentCost starting reading just dashes. There’s also no transmitter icon, so I think it’s not receiving anything from the transmitter. It looks like this:
I went and fished the transmitter box out of the meter closet expecting its batteries to be dead, but it still has its red LED flashing periodically, so I don’t think it’s that.
I did the thing where you hold down the button on the transmitter for 9 seconds and also hold down the V button on the display to make them pair. The display showed its “searching” screen for a while but then it went back to how it looks above.
Anyone had that happen before? It’s otherwise worked fine for 4 years or so (batteries replaced once).
You don't need to install anything to run Go binaries.
At first - I'm sure like many others - I felt a little revultion when I heard that Go compiles to statically-linked binaries but after having used and played with Go a bit over the past few weeks, I think it's rather clever and was somewhat ahead of the game. In the current climate where DevOps folks (and developers) are getting excited about containers and componentised services, being able to simply curl a binary and have it usable in your container without needing to install a stack of dependencies is actually pretty powerful. It seems there's a general trend towards preferring readiness of use over efficiency of space used both in RAM and disk space. And it makes sense; storage is cheap these days. A 10MiB binary is no concern - even if you need several of them - when you have a 1TiB drive. The extravagance of large binaries is no longer so relevant when you're comparing it with your collection of 2GiB bluray rips. The days of needing to count the bytes are gone.
Go has the feeling of C but without all that tedious mucking about in hyperspace memory
Sometimes you just feel you need to write something fairly low level and you want more direct control than you have whilst you're working from the comfort blanket of python or ruby. Go gives you the ability to have well-defined data structures and to care about how much memory you're eating when you know your application needs to process tebibytes of data. What Go doesn't give you is the freedom to muck about in memory, fall off the end of arrays, leave pointers dangling around all over the place, and generally make tiny, tiny mistakes that take years for anyone to discover.
The build system is designed around how we (as developers) use code hosting facilities
Go has a fairly impressive set of features built in but if you need something that's not already included, there's a good chance that someone out there has written what you need. Go provides a package search tool that makes it very easy to find what you're looking for. And when you've found it, using it is stupidly simple. You add an import declaration in your code:import "github.com/codegangsta/cli"
which makes it very clear where the code has come from and where you'd need to go to check the source code and/or documentation. Next, pulling the code down and compiling it ready for linking into your own binary takes a simple:go get github.com/codegangsta/cli
Go implicitly understands git and the various methods of retrieving code so you just need to tell it where to look and it'll figure the rest out.
In summary, I'm starting to wonder if Google have a time machine. Go seems to have nicely predicted several worries and trends since its announcement: Docker, Heartbleed, and social coding.
Yesterday the new Government published a press release about the forthcoming first meeting of the new National Security Council (NSC). That meeting was due to discuss the Tory administration’s plans for a new Counter-Extremism Bill. The press release includes the following extraordinary stement which is attributed to the Prime Minister:
“For too long, we have been a passively tolerant society, saying to our citizens: as long as you obey the law, we will leave you alone. “
Forgive me, but what exactly is wrong with that view? Personally I think it admirable that we live in a tolerant society (“passive” or not). Certainly I believe that tolerance of difference, tolerance of free speech, tolerance of the right to hold divergent opinion, and to voice that opinion is to be cherished and lauded. And is it not right and proper that a Government should indeed “leave alone” any and all of its citizens who are obeying the law?
Clearly, however, our Prime Minster disagrees with me and believes that a tolerant society is not what we really need in the UK because the press release continues:
“This government will conclusively turn the page on this failed approach. “
If tolerance is a “failed approach”, what are we likely to see in its place?
A while ago, I switched from tritium to herbstluftwm. In general, it’s been a good move, benefitting from active development and greater stability, even if I do slightly mourn the move from python scripting to a shell client.
One thing that was annoying me was that throwing the pointer into an edge didn’t find anything clickable. Window borders may be pretty, but they’re a pretty poor choice as the thing that you can locate most easily, the thing that is on the screen edge.
It finally annoyed me enough to find the culprit. The .config/herbstluftwm/autostart file said “hc pad 0 26″ (to keep enough space for the panel at the top edge) and changing that to “hc pad 0 -8 -7 26 -7″ and reconfiguring the panel to be on the bottom (where fewer windows have useful controls) means that throwing the pointer at the top or the sides now usually finds something useful like a scrollbar or a menu.
I wonder if this is a useful enough improvement that I should report it as an enhancement bug.
Without going into any of the details, it's a web application with a front end written using Ember and various services that it calls out to, written using whatever seems appropriate per service.
At the outset of the project, we decided we would bite the bullet and build for Docker from the outset. This meant we would get to avoid the usual dependency and developer environment setup nightmares.The problem
What we quickly realised as we started to put the bare bones of a few of the services in place, was that we had three seemingly conflicting goals for each component and for the application as a whole.
Build images that can be deployed in production.
Allow developers to run services locally.
Provide a means for running unit tests (both by developers and our CI server).
So here's what we've ended up with:The solution
Or: docker-compose to the rescueFolder structure
Here's what the project layout looks like:Project | +-docker-compose.yml | +-Service 1 | | | +-Dockerfile | | | +-docker.compose.yml | | | +-<other files> | +-Service 2 | | +-Dockerfile | +-docker.compose.yml | +-<other files> Building for production
This is the easy bit and is where we started first. The Dockerfile for each service was designed to run everything with the defaults. Usually, this is something simple like:FROM python:3-onbuild CMD ["python", "main.py"]
Our CI server can easily take these, produce images, and push them to the registry.Allowing developers to run services locally
This is slightly harder. In general, each service wants to do something slightly different when being run for development; e.g. automatically restarting when code changes. Additionally, we don't want to have to rebuild an image every time we make a code change. This is where docker-compose comes in handy.
The docker-compose.yml at the root of the project folder looks like this:service1: build: Service 1 environment: ENV: dev volumes: - Service 1:/usr/src/app links: - service2 - db ports: - 8001:8000 service2: build: Service2 environment: ENV: dev volumes: - Service 2:/usr/src/app links: - service1 - db ports: - 8002:8000 db: image: mongo
This gives us several features right away:
We can locally run all of the services together with docker-compose up
The ENV environment variable is set to dev in each service so that the service can configure itself when it starts to run things in "dev" mode where needed.
The source folder for each service is mounted inside the container. This means you don't need to rebuild the image to try out new code.
Each service is bound to a different port so you can connect to each part directly where needed.
Each service defines links to the other services it needs.
This was the trickiest part to get right. Some services have dependencies on other things even just to get unit tests running. For example, Eve is a huge pain to get running with a fake database so it's much easier to just link it to a temporary "real" database.
Additionally, we didn't want to mess with the idea that the images should run production services by default but also didn't want to require folks to need to churn out complicated docker invocations like docker run --rm -v $(pwd):/usr/src/app --link db:db service1 python -m unittest just to run the test suite after coding up some new features.
So, it was docker-compose to the rescue again :)
Each service has a docker-compose.yml that looks something like:tests: build: . command: python -m unittest volumes: - .:/usr/src/app links: - db db: image: mongo
Which sets up any dependencies needed just for the tests, mounts the local source in the container, and runs the desired command for running the tests.
So, a developer (or the CI box) can run the unit tests with:docker-compose run tests Summary
Each Dockerfile builds an image that can go straight into production without further configuration required.
Each image runs in "developer mode" if the ENV environment variable is set.
Running docker-compose up from the root of the project gets you a full stack running locally in developer mode.
Running docker-compose run tests in each service's own folder will run the unit tests for that service - starting any dependencies as needed.
I've been meaning to blog about the podcasts I listen to and the setup I use for consuming them as both have evolved a little over the past few months.The podcasts
I use syncthing to have those replicated to my laptop and home media server.
When I'm cycling to work or in the car, I use the mp3 player to listen to them. (No, when I'm in the car, I plug it in to the stereo, I don't drive with headphones on :P)
When I'm sitting at a computer or at home, I use Plex to serve up podcasts from my home media box.
I keep on top of everything by making sure that I move (rather than copy) when putting things on the mp3 player and rely on Syncthing to remove listened-to podcasts from everywhere else.
It's not the most elegant setup I've heard of but it's simple and works for me :)What next?
I find I have a lot of things I want to listen to and not really enough time to listen to them in. I've heard that some people speed podcasts up (I've heard as much as 50%). Does anyone do this? Does it make things any less enjoyable to listen to? I really enjoy the quality of what I listen to; I don't want to feel like I'm just consuming information for the sake of it.
The Debian Ruby Ruby team had a first sprint in 2014. The experience was very positive, and it was decided to do it again in 2015. Last April, the team once more met at the IRILL offices, in Paris, France.
The participants worked to improve the quality Ruby packages in Debian, including fixing release critical and security bugs, improving metadata and packaging code, and triaging test failures on the Debian Continuous Integration service.
The sprint also served to prepare the team infrastructure for the future Debian 9 release:
the gem2deb packaging helper to improve the semi-automated generation of Debian source packages from existing standard-compliant Ruby packages from Rubygems.
there was also an effort to prepare the switch to Ruby 2.2, the latest stable release of the Ruby language which was released after the Debian testing suite was already frozen for the Debian 8 release.
Left to right: Christian Hofstaedtler, Tomasz Nitecki, Sebastien Badia and Antonio Terceiro.
A full report with technical details has been posted to the relevant Debian mailing lists.
The UK has just had it's General Election. Labour failed miserably to increase their vote. The SNP picked uploads of votes and seats - mostly as they felt betrayed by the failure of delivery of anything after they agreed to remain in the union. The Liberal Democrats lost votes and seats a plenty as expected. The result is now we have a weak Conservative government with a slim majority - that will no doubt destroy it's self as the swivel-eyed loons on the far right of the party start to make increasingly unrealistic demands on the rest of the party.
The nutters in the home office, with the Liberal Democrat "sanity" checks removed will now demand ever increasing powers to snoop on everything we do, so that they can protect us from what ever problem they have invented to scare us with next...
I now feel compelled to support the Open Rights Group with my money as well as my moral support. If the lunatics aren't stopped then we'll have no civil liberties left.
Earlier this week I allowed some of my colleagues at work to try the Kimchi and one of them (James) liked it enough to ask for a tub of it just for himself. Being a lovely person I obliged.
My darling Rob decided that what he really wanted was Kimchiguk or Kimchijijae. Being both wonderful and lacking in ingredients, I set to do achieving something.
I read a few recipes, worked out that I lacked critical ingredients and so set about simulating Kimchiguk. I grabbed two and a bit cupfuls of kimchi from the box in the fridge, chopped it up more finely, popped it into a pot along with about a pound of cubed pork belly, two teaspoons of hot pepper flakes, two teaspoons of sugar, and two teaspoons of cornflour. I then topped that off with five cups of water, mixed it up, brought it to the boil and simmered it for 30 minutes. Then I cubed some tofu, popped that in and simmered it for another 10 minutes while some rice cooked. Served in a bowl over rice, my approximation of Kimchiguk turned out pretty well. (Note it fed both of us and there was a serving left over.)
With a do-over, I'd try harder to get pork shoulder (belly is quite fatty) and given the option I'd wait until I had hot pepper paste because the malty fermented sweetness of hot pepper paste is pretty much impossible to emulate otherwise.
Last night we tried the small amount of 'fresh kimchi' from my epic kimchi making trial.
I marinaded some beef strips in a blended paste of onion, garlic, ginger, and sesame oil for a few hours before frying that up, and serving on rice with stir-fried carrots, leek and spring onion. A little fresh kimchi on the side for flavour.
I think the Kimchi was a success. Dearest Rob said that it tasted of Kimchi. The flavours were not very well combined and quite clearly underdeveloped, and the pepper flakes were still a little hard. I expect this to resolve over the next few days as it begins to ferment.
All in all, experiment going well, expect part 3 when I know how it tastes after fermentation begins.
Theresa May hasn’t wasted any time. The Independent reports today that Ms May (Home Secretary in the coalition administration) has said that the new Tory administration will bring the Draft Communications Data Bill, previously blocked by the Liberal Democrats, back to the House of Commons with the intention of getting it passed into law. As the Independent also reports, dear Dave, who is let us say, technically challenged, has in the past expressed the view that no form of communication should be unreadable by the Goverment. This implies severe restrictions on all forms of encryption.
Given that the Tories now have the majority they lacked in the last administration, it is clear that they will see themselves free to attack the kind of liberties I, and millions like me enjoy and cherish. The Open Rights Group maintain a wiki devoted to the relevant points of each political party’s manifesto devoted to surveillance or other possible attacks on privacy. As they point out, the Tory party is committed to:
There are, of course, huge practical and technical difficulties in implementing much of what the Tories wish to do (consider for example the idiocy of attempting to outlaw VPN technology) but that won’t stop them trying. Indeed, some of the technical difficulties may cause the new administration to bring in mechanisms to get around those problems. An obvious example would be the requirement for key escrow for anyone wishing to use encryption.
Excuse me if I find that unacceptable. Time to encrypt much, much more of my everyday activity from now on.
The early results of yesterday’s poll are depressing beyond belief. It looks almost certain that the Tory party will have sufficient seats to form the next government.
I don’t often make party political points here (though my political leanings may sometimes be obvious) but I was reminded today of Neil Kinnock’s heart rending speech in Bridgend, Glamorgan, on Tuesday 7 June 1983, two days before the election in which Margaret Thatcher was returned as Prime Minister.
“If Margaret Thatcher wins on Thursday, I warn you not to be ordinary. I warn you not to be young. I warn you not to fall ill. I warn you not to get old.”
Those words resonate even more today than they did 32 years ago. I fear for the old, the poor, the disposessed, the weak, the young, the sick and yes, indeed, the ordinary people of this country. David Cameron and his cronies both inside and outside Government will now return to the task of dismantling all that is good and admirable about our society. A society should be judged on the way it treats its weakest and less able members. Cameron’s Tories are, at heart, brutal and uncaring. That frightens me.