News aggregator

Jonathan McDowell: Experiments with 1-Wire

Planet ALUG - Tue, 24/01/2017 - 21:49

As previously mentioned, at the end of last year I got involved with a project involving the use of 1-Wire. In particular a DS28E15 device, intended to be used as a royalty tracker for a licensed piece of hardware IP. I’d no previous experience with 1-Wire (other than knowing it’s commonly used for driving temperature sensors), so I took it as an opportunity to learn a bit more about it.

The primary goal was to program a suitable shared key into the DS28E15 device that would also be present in the corresponding hardware device. A Maxim programmer had been ordered, but wasn’t available in stock so had to be back ordered. Of course I turned to my trusty Bus Pirate, which claimed 1-Wire support. However it failed to recognise the presence of the device at all. After much head scratching I finally listened to a co-worker who had suggested it was a clock speed issue - the absence of any option to select the 1-Wire speed in the Bus Pirate or any mention of different speeds in the documentation I had read had made me doubt it was an issue. Turns out that the Bus Pirate was talking “standard” 1-Wire and the DS28E15 only talks “overdrive” 1-Wire, to the extent that it won’t even announce its presence if the reset pulse conforms to the standard, rather than overdrive, reset time period. Lesson learned: listen to your co-workers sooner.

A brief period of yak shaving led to adding support to the Bus Pirate for the overdrive mode (since landed in upstream), and resulted in a search request via the BP interface correctly finding the device and displaying its ROM ID. This allowed exploration of the various commands the authenticator supports, to verify that the programming sequence operated as expected. These allow for setting the shared secret, performing a SHA256 MAC against this secret and a suitable nonce, and retrieving the result.

Next problem: the retrieved SHA256 MAC did not match the locally computed value. Initially endianness issues were suspected, but trying the relevant permutations did not help. Some searching found an implementation of SHA256 for the DS28E15 that showed differences between a standard SHA256 computation and what the authenticator performs. In particular SHA256 normally adds the current working state (a-g) to the current hash value (h0-h7) at the end of every block. The authenticator does this for all but the final block, where instead the hash value is set to the working state. I haven’t been able to find any documentation from Maxim that this is how things are calculated, nor have I seen any generic implementation of SHA256 which supports this mode. However rolling my own C implementation based on the code I found and using it to compare the results retrieved from the device confirms that this is what’s happening.

So at this point we’re done, right? Wait for the proper programming hardware to turn up, write the key to the devices, profit? Well, no. There was a bit of a saga involving the programmer (actually programmers, one with at least some documentation that allowed the creation of a Python tool to allow setting the key and reading + recording the ROM ID for tracking, and one with no programming documentation that came with a fancy GUI for manually doing the programming), but more importantly it was necessary to confirm that the programmed device interacted with the hardware correctly.

Initial testing with the hardware was unsuccessful. Again endianness issues were considered and permutations tried, but without success. A simple key constructed to avoid such issues was tried, and things worked fine. There was a hardware simulation of both components available, so it was decided to run that and obtain a capture of the traffic between them. As the secret key was known this would then allow the random nonce to be captured, and the corresponding (correct) hash value. Tests could then be performed in software to determine what the issue was & how to generate the same hash for verification.

Two sets of analyzer software were tried, OpenBench LogicSniffer (OLS) and sigrok. As it happened both failed to correctly decode the bitstream detected as 1-Wire, but were able to show the captured data graphically, allowing for decoding by eye. A slight patch to OLS to relax the timing constraints allowed it to successfully decode the full capture and provided the appropriate data for software reproduction. The end issue? A 256 bit number (as defined in VHDL) is not the same as 32 element byte array… Obvious when you know what the issue is!

So? What did I learn, other than a lot about 1-Wire? Firstly, don’t offhandedly discount suggestions that you don’t think make sense. Secondly, having a tool (in this case the Bus Pirate) that lets you easily play with a protocol via a simple interface is invaluable in understanding it. Thirdly, don’t trust manufacturers to be doing something in a normal fashion when they claim to be using a well defined technology. Fourthly, be conscious about all of the different ways bitstreams can be actually processed in memory. It’s not just endianness. Finally, spending the time to actually understand what’s going on up front can really help when things don’t work as you’d expect later on - without the yak shaving to support Overdrive on the BP I wouldn’t have been able to so quickly use the simulation capture to help diagnose the issue.

Categories: LUG Community Blogs

YES FOLKS ITS TODAY

West Yorkshire LUG News - Mon, 23/01/2017 - 13:29

This month the monthly meeting is on TUESDAY of THIS WEEK. Come and join us in the Lord Darcy! be surprised about Linux ! Run a web server on you laptop ! The future here today!

Steve Engledow (stilvoid): Angst

Planet ALUG - Sun, 22/01/2017 - 03:06

I had planned to spend this evening playing games; something I really enjoy doing but rarely set aside any time for. However, while we were eating dinner, I put some music on and it got me in the mood for playing some guitar. Over the course of dinner and playing with my son afterwards, that developed into wanting to write and record some music. I used to write electronic nonsense sometimes but this evening, I fancied trying my hand at some metal.

The first 90 minutes was - as almost every time I get the rare combination of an urge to do something musical and time to do it in - spent trying to remember how my setup worked, which bits of software I needed to install, and how to get the right combination of inputs and outputs I want. I eventually got it sussed and decided I'd better write it down for my own future reference.

Hardware
  1. Plug the USB audio interface from the V-Amp3 into the laptop.
  2. Plug external audio sources into the audio interface's input. (e.g. the V-Amp or a synth).
  3. Plug some headphones into the headphone socket of the audio interface.
  4. Switch on the audio interface's monitoring mode ;) (this kept me going for a little while; it's a small switch)
Software
  1. The following packages need to be installed at a minimum:

    • qjackctl
    • qsynth
    • soundfont-fluidsynth
    • vkeybd
    • ardour
    • hydrogen
  2. Use pavucontrol or similar to disable the normal audio system and just use the USB audio interface.

  3. Qjackctl needs the following snippets in its config for when jack comes up and goes down, respectively:

    • pacmd suspend true

      This halts pulseaudio so that jack can take over

    • pacmd suspend false

      This starts puseaudio back up again

  4. Use the connection tool in Jack to hook hydrogen's and qsynth's outputs to ardour's input. Use the ALSA tab to connect vkeybd to qsynth.

  5. When starting Ardour and Hydrogen, make sure they're both configured to use Jack for MIDI. Switch Ardour's clock from Internal to JACK.

For posterity, here's this evening's output.

Categories: LUG Community Blogs

Monthly Meeting Tuesday 24th Jan 2017

West Yorkshire LUG News - Wed, 18/01/2017 - 17:02

This month’s meeting is on Tuesday 24th Jan, not the usual Thursday. The location is still the same, The Lord Darcy on Harrogate Road. If you have never used linux before, please come and try it out on one of our laptops. If you want to discuss things before hand, or between meetings, try our meetups pages, or the email mailing lists at the right hand side of this page. See you next week!

Jonathan McDowell: Cloning a USB LED device

Planet ALUG - Sat, 14/01/2017 - 12:53

A month or so ago I got involved in a discussion on IRC about notification methods for a headless NAS. One of the options considered was some sort of USB attached LED. DealExtreme had a cheap “Webmail notifier”, which was already supported by mainline kernels as a “Riso Kagaku” device but it had been sold out for some time.

This seemed like a fun problem to solve with a tinyAVR and V-USB. I had my USB relay board so I figured I could use that to at least get some code to the point that the kernel detected it as the right device, and the relay output could be configured as one of the colours to ensure it was being driven in roughly the right manner. The lack of a full lsusb dump (at least when I started out) made things a bit harder, plus the fact that the Riso uses an output report unlike the relay code, which uses a control message. However I had the kernel source for the driver and with a little bit of experimentation had something which would cause the driver to be loaded and the appropriate files in /sys/class/leds/ to be created. The relay was then successfully activated when the red LED was supposed to be on.

hid-led 0003:1294:1320.0001: hidraw0: USB HID v1.01 Device [MAIL MAIL ] on usb-0000:00:14.0-6.2/input0 hid-led 0003:1294:1320.0001: Riso Kagaku Webmail Notifier initialized

I subsequently ordered some Digispark clones and modified the code to reflect the pins there (my relay board used pins 1+2 for USB, the Digispark uses pins 3+4). I then soldered a tricolour LED to the board, plugged it in and had a clone of the Riso Kaguku device for about £1.50 in parts (no doubt much cheaper in bulk). Very chuffed.

In case it’s useful to someone, the code is released under GPLv3+ and is available at https://the.earth.li/gitweb/?p=riso-kagaku-clone.git;a=summary or on GitHub at https://github.com/u1f35c/riso-kagaku-clone. I’m seeing occasional issues on an older Dell machine that only does USB2 with enumeration, but it generally is fine once it gets over that.

(FWIW, Jon, who started the original discussion, ended up with a BlinkStick Nano which is a neater device with 2 LEDs but still based on an Tiny85.)

Categories: LUG Community Blogs

Andy Smith: XFS, Reflinks and Deduplication

Planet HantsLUG - Tue, 10/01/2017 - 21:45
btrfs Past

This post is about XFS but it’s about features that first hit Linux in btrfs, so we need to talk about btrfs for a bit first.

For a long time now, btrfs has had a useful feature called reflinks. Basically this is exposed as cp --reflink=always and takes advantage of extents and copy-on-write in order to do a quick copy of data by merely adding another reference to the extents that the data is currently using, rather than having to read all the data and write it out again, as would be the case in other filesystems.

Here’s an excerpt from the man page for cp:

When –reflink[=always] is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails, or if –reflink=auto is specified, fall back to a standard copy.

Without reflinks a common technique for making a quick copy of a file is the hardlink. Hardlinks have a number of disadvantages though, mainly due to the fact that since there is only one inode all hardlinked copies must have the same metadata (owner, group, permissions, etc.). Software that might modify the files also needs to be aware of hardlinks: naive modification of a hardlinked file modifies all copies of the file.

With reflinks, life becomes much easier:

  • Each copy has its own inode so can have different metadata. Only the data extents are shared.
  • The filesystem ensures that any write causes a copy-on-write, so applications don’t need to do anything special.
  • Space is saved on a per-extent basis so changing one extent still allows all the other extents to remain shared. A change to a hardlinked file requires a new copy of the whole file.

Another feature that extents and copy-on-write allow is block-level out-of-band deduplication.

  • Deduplication – the technique of finding and removing duplicate copies of data.
  • Block-level – operating on the blocks of data on storage, not just whole files.
  • Out-of-band – something that happens only when triggered or scheduled, not automatically as part of the normal operation of the filesystem.

btrfs has an ioctl that a userspace program can use—presumably after finding a sequence of blocks that are identical—to tell the kernel to turn one into a reference to the other, thus saving some space.

It’s necessary that the kernel does it so that any IO that may be going on at the same time that may modify the data can be dealt with. Modifications after the data is reflinked will just case a copy-on-write. If you tried to do it all in a userspace app then you’d risk something else modifying the files at the same time, but by having the kernel do it then in theory it becomes completely safe to do it at any time. The kernel also checks that the sequence of extents really are identical.

In-band deduplication is a feature that’s being worked on in btrfs. It already exists in ZFS though, and there is it rarely recommended for use as it requires a huge amount of memory for keeping hashes of data that has been written. It’s going to be the same story with btrfs, so out-of-band deduplication is still something that will remain useful. And it exists as a feature right now, which is always a bonus.

XFS Future

So what has all this got to do with XFS?

Well, in recognition that there might be more than one Linux filesystem with extents and so that reflinks might be more generally useful, the extent-same ioctl got lifted up to be in the VFS layer of the kernel instead of just in btrfs. And the good news is that XFS recently became able to make use of it.

When I say “recently” I do mean really recently. I mean like kernel release 4.9.1 which came out on 2017-01-04. At the moment it comes with massive EXPERIMENTAL warnings, requires a new filesystem to be created with a special format option, and will need an xfsprogs compiled from recent git in order to have a mkfs.xfs that can create such a filesystem.

So before going further, I’m going to assume you’ve compiled a new enough kernel and booted into it, then compiled up a new enough xfsprogs. Both of these are quite simple things to do, for example the Debian documentation for building kernel packages from upstream code works fine.

XFS Reflink Demo

Make yourself a new filesystem, with the reflink=1 format option.

# mkfs.xfs -L reflinkdemo -m reflink=1 /dev/xvdc meta-data=/dev/xvdc isize=512 agcount=4, agsize=3276800 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=1 data = bsize=4096 blocks=13107200, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=6400, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0

Put it in /etc/fstab for convenience, and mount it somewhere.

# echo "LABEL=reflinkdemo /mnt/xfs xfs relatime 0 2" >> /etc/fstab # mkdir -vp /mnt/xfs mkdir: created directory ‘/mnt/xfs’ # mount /mnt/xfs # df -h /mnt/xfs Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 339M 50G 1% /mnt/xfs

Create a few files with random data.

# mkdir -vp /mnt/xfs/reflink mkdir: created directory ‘/mnt/xfs/reflink’ # chown -c andy: /mnt/xfs/reflink changed ownership of ‘/mnt/xfs/reflink’ from root:root to andy:andy # exit $ for i in {1..5}; do > echo "Writing $i…"; dd if=/dev/urandom of=/mnt/xfs/reflink/$i bs=1M count=1024; > done Writing 1… 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 4.34193 s, 247 MB/s Writing 2… 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 4.33207 s, 248 MB/s Writing 3… 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 4.33527 s, 248 MB/s Writing 4… 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 4.33362 s, 248 MB/s Writing 5… 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 4.32859 s, 248 MB/s $ df -h /mnt/xfs Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 5.4G 45G 11% /mnt/xfs $ du -csh /mnt/xfs 5.0G /mnt/xfs 5.0G total

Copy a file and as expected usage will go up by 1GiB. And it will take a little while, even on my nice fast SSDs.

$ time cp -v /mnt/xfs/reflink/{,copy_}1 ‘/mnt/xfs/reflink/1’ -> ‘/mnt/xfs/reflink/copy_1’   real 0m3.420s user 0m0.008s sys 0m0.676s $ df -h /mnt/xfs; du -csh /mnt/xfs/reflink Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 6.4G 44G 13% /mnt/xfs 6.0G /mnt/xfs/reflink 6.0G total

So what about a reflink copy?

$ time cp -v --reflink=always /mnt/xfs/reflink/{,reflink_}1 ‘/mnt/xfs/reflink/1’ -> ‘/mnt/xfs/reflink/reflink_1’   real 0m0.003s user 0m0.000s sys 0m0.004s $ df -h /mnt/xfs; du -csh /mnt/xfs/reflink Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 6.4G 44G 13% /mnt/xfs 7.0G /mnt/xfs/reflink 7.0G total

The apparent usage went up by 1GiB but the amount of free space as shown by df stayed the same. No more actual storage was used because the new copy is a reflink. And the copy got done in 4ms as opposed to 3,420ms.

Can we tell more about how these files are laid out? Yes, we can use the filefrag -v command to tell us more.

$ filefrag -v /mnt/xfs/reflink/{,copy_,reflink_}1 Filesystem type is: 58465342 File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: last,shared,eof /mnt/xfs/reflink/1: 1 extent found File size of /mnt/xfs/reflink/copy_1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 917508.. 1179651: 262144: last,eof /mnt/xfs/reflink/copy_1: 1 extent found File size of /mnt/xfs/reflink/reflink_1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: last,shared,eof /mnt/xfs/reflink/reflink_1: 1 extent found

What we can see here is that all three files are composed of a single extent which is 262,144 4KiB blocks in size, but it also tells us that /mnt/xfs/reflink/1 and /mnt/xfs/reflink/reflink_1 are using the same range of physical blocks: 1572884..1835027.

XFS Deduplication Demo

We’ve demonstrated that you can use cp --reflink=always to take a cheap copy of your data, but what about data that may already be duplicates without your knowledge? Is there any way to take advantage of the extent-same ioctl for deduplication?

There’s a couple of software solutions for out-of-band deduplication in btrfs, but one I know that works also in XFS is duperemove. You will need to use a git checkout of duperemove for this to work.

A quick reminder of the storage use before we start.

$ df -h /mnt/xfs; du -csh /mnt/xfs/reflink Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 6.4G 44G 13% /mnt/xfs 7.0G /mnt/xfs/reflink 7.0G total $ filefrag -v /mnt/xfs/reflink/{,copy_,reflink_}1 Filesystem type is: 58465342 File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: last,shared,eof /mnt/xfs/reflink/1: 1 extent found File size of /mnt/xfs/reflink/copy_1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 917508.. 1179651: 262144: last,eof /mnt/xfs/reflink/copy_1: 1 extent found File size of /mnt/xfs/reflink/reflink_1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: last,shared,eof /mnt/xfs/reflink/reflink_1: 1 extent found

Run duperemove.

# duperemove -hdr --hashfile=/var/tmp/dr.hash /mnt/xfs/reflink Using 128K blocks Using hash: murmur3 Gathering file list... Adding files from database for hashing. Loading only duplicated hashes from hashfile. Using 2 threads for dedupe phase Kernel processed data (excludes target files): 4.0G Comparison of extent info shows a net change in shared extents of: 1.0G $ df -h /mnt/xfs; du -csh /mnt/xfs/reflink Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 5.4G 45G 11% /mnt/xfs 7.0G /mnt/xfs/reflink 7.0G total $ filefrag -v /mnt/xfs/reflink/{,copy_,reflink_}1 Filesystem type is: 58465342 File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: last,shared,eof /mnt/xfs/reflink/1: 1 extent found File size of /mnt/xfs/reflink/copy_1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: last,shared,eof /mnt/xfs/reflink/copy_1: 1 extent found File size of /mnt/xfs/reflink/reflink_1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: last,shared,eof /mnt/xfs/reflink/reflink_1: 1 extent found

The output of du remained the same, but df says that there’s now 1GiB more free space, and filefrag confirms that what’s changed is that copy_1 now uses the same extents as 1 and reflink_1. The duplicate data in copy_1 that in theory we did not know was there, has been discovered and safely reference-linked to the extent from 1, saving us 1GiB of storage.

By the way, I told duperemove to use a hash file because otherwise it will keep that in RAM. For the sake of 7 files that won’t matter but it will if I have millions of files so it’s a habit I get into. It uses that hash file to avoid having to repeatedly re-hash files that haven’t changed.

All that has been demonstrated so far though is whole-file deduplication, as copy_1 was just a regular copy of 1. What about when a file is only partially composed of duplicate data? Well okay.

$ cat /mnt/xfs/reflink/{1,2} > /mnt/xfs/reflink/1_2 $ ls -lah /mnt/xfs/reflink/{1,2,1_2} -rw-r--r-- 1 andy andy 1.0G Jan 10 15:41 /mnt/xfs/reflink/1 -rw-r--r-- 1 andy andy 2.0G Jan 10 16:55 /mnt/xfs/reflink/1_2 -rw-r--r-- 1 andy andy 1.0G Jan 10 15:41 /mnt/xfs/reflink/2 $ df -h /mnt/xfs; du -csh /mnt/xfs/reflink Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 7.4G 43G 15% /mnt/xfs 9.0G /mnt/xfs/reflink 9.0G total $ filefrag -v /mnt/xfs/reflink/{1,2,1_2} Filesystem type is: 58465342 File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: last,shared,eof /mnt/xfs/reflink/1: 1 extent found File size of /mnt/xfs/reflink/2 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262127: 20.. 262147: 262128: 1: 262128.. 262143: 2129908.. 2129923: 16: 262148: last,eof /mnt/xfs/reflink/2: 2 extents found File size of /mnt/xfs/reflink/1_2 is 2147483648 (524288 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262127: 262164.. 524291: 262128: 1: 262128.. 524287: 655380.. 917539: 262160: 524292: last,eof /mnt/xfs/reflink/1_2: 2 extents found

I’ve concatenated 1 and 2 together into a file called 1_2 and as expected, usage goes up by 2GiB. filefrag confirms that the physical extents in 1_2 are new. We should be able to do better because this 1_2 file does not contain any new unique data.

$ duperemove -hdr --hashfile=/var/tmp/dr.hash /mnt/xfs/reflink Using 128K blocks Using hash: murmur3 Gathering file list... Adding files from database for hashing. Using 2 threads for file hashing phase Kernel processed data (excludes target files): 4.0G Comparison of extent info shows a net change in shared extents of: 3.0G $ df -h /mnt/xfs; du -csh /mnt/xfs/reflink Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 5.4G 45G 11% /mnt/xfs 9.0G /mnt/xfs/reflink 9.0G total

We can. Apparent usage stays at 9GiB but real usage went back to 5.4GiB which is where we were before we created 1_2.

And the physical layout of the files?

$ filefrag -v /mnt/xfs/reflink/{1,2,1_2} Filesystem type is: 58465342 File size of /mnt/xfs/reflink/1 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: last,shared,eof /mnt/xfs/reflink/1: 1 extent found File size of /mnt/xfs/reflink/2 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262127: 20.. 262147: 262128: shared 1: 262128.. 262143: 2129908.. 2129923: 16: 262148: last,shared,eof /mnt/xfs/reflink/2: 2 extents found File size of /mnt/xfs/reflink/1_2 is 2147483648 (524288 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 262143: 1572884.. 1835027: 262144: shared 1: 262144.. 524271: 20.. 262147: 262128: 1835028: shared 2: 524272.. 524287: 2129908.. 2129923: 16: 262148: last,shared,eof /mnt/xfs/reflink/1_2: 3 extents found

It shows that 1_2 is now made up from the same extents as 1 and 2 combined, as expected.

Less of the urandom

These synthetic demonstrations using a handful of 1GiB blobs of data from /dev/urandom are all very well, but what about something a little more like the real world?

Okay well let’s see what happens when I take ~30GiB of backup data created by rsnapshot on another host.

rsnapshot is a backup program which makes heavy use of hardlinks. It runs periodically and compares the previous backup data with the new. If they are identical then instead of storing an identical copy it makes a hardlink. This saves a lot of space but does have a lot of limitations as discussed previously.

This won’t be the best example because in some ways there is expected to be more duplication; this data is composed of multiple backups of the same file trees. But on the other hand there shouldn’t be as much because any truly identical files have already been hardlinked together by rsnapshot. But it is a convenient source of real-world data.

So, starting state:

(I deleted all the reflink files)

$ df -h /mnt/xfs; sudo du -csh /mnt/xfs/rsnapshot Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 30G 21G 59% /mnt/xfs 29G /mnt/xfs/rsnapshot 29G total

A small diversion about how rsnapshot lays out its backups may be useful here. They are stored like this:

  • rsnapshot_root / [iteration a] / [client foo] / [directory structure from client foo]
  • rsnapshot_root / [iteration a] / [client bar] / [directory structure from client bar]
  • rsnapshot_root / [iteration b] / [client foo] / [directory structure from client foo]
  • rsnapshot_root / [iteration b] / [client bar] / [directory structure from client bar]

The iterations are commonly things like daily.0, daily.1daily.6. As a consequence, the paths:

rsnapshot/daily.*/client_foo

would be backups only from host foo, and:

rsnapshot/daily.0/*

would be backups from all hosts but only the most recent daily sync.

Let’s first see what the savings would be like in looking for duplicates in just one client’s backups.

Here’s the backups I have in this blob of data. The names of the clients are completely made up, though they are real backups.

Client Size (MiB) darbee 14,504 achorn 11,297 spader 2,612 reilly 2,276 chino 2,203 audun 2,184

So let’s try deduplicating all of the biggest one’s—darbee‘s—backups:

$ df -h /mnt/xfs Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 30G 21G 59% /mnt/xfs # time duperemove -hdr --hashfile=/var/tmp/dr.hash /mnt/xfs/rsnapshot/*/darbee Using 128K blocks Using hash: murmur3 Gathering file list... Kernel processed data (excludes target files): 8.8G Comparison of extent info shows a net change in shared extents of: 6.8G 9.85user 78.70system 3:27.23elapsed 42%CPU (0avgtext+0avgdata 23384maxresident)k 50703656inputs+790184outputs (15major+20912minor)pagefaults 0swaps $ df -h /mnt/xfs Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 25G 26G 50% /mnt/xfs

3m27s of run time, somewhere between 5 and 6.8GiB saved. That’s 35%!

Now to deduplicate the lot.

# time duperemove -hdr --hashfile=/var/tmp/dr.hash /mnt/xfs/rsnapshot Using 128K blocks Using hash: murmur3 Gathering file list... Kernel processed data (excludes target files): 5.4G Comparison of extent info shows a net change in shared extents of: 3.4G 29.12user 188.08system 5:02.31elapsed 71%CPU (0avgtext+0avgdata 34040maxresident)k 34978360inputs+572128outputs (18major+45094minor)pagefaults 0swaps $ df -h /mnt/xfs Filesystem Size Used Avail Use% Mounted on /dev/xvdc 50G 23G 28G 45% /mnt/xfs

5m02 used this time, and another 2–3.4G saved.

Since the actual deduplication does take some time (the kernel having to read the extents, mainly), and most of it was already done in the first pass, a full pass would more likely take the sum of the times, i.e. more like 8m29s.

Still, a total of about 7GiB was saved which is 23%.

It would be very interesting to try this on one of my much larger backup stores.

Why Not Just Use btrfs?

Using a filesystem that already has all of these features would certainly seem easier, but I personally don’t think btrfs is stable enough yet. I use it at home in a relatively unexciting setup (8 devices, raid1 for data and metadata, no compression or deduplication) and I wish I didn’t. I wouldn’t dream of using it in a production environment yet.

I’m on the btrfs mailing list and there are way too many posts regarding filesystems that give ENOSPC and become unavailable for writes, or systems that were unexpectedly powered off and when powered back on the btrfs filesystem is completely lost.

I expect the reflink feature in XFS to become non-experimental before btrfs is stable enough for production use.

ZFS?

ZFS is great. It doesn’t have out-of-band deduplication or reflinks though, and they don’t plan to any time soon.

Categories: LUG Community Blogs

Viva Amiga: The Story of a Beautiful Machine | Documentary

Planet SurreyLUG - Mon, 09/01/2017 - 13:08

Acclaimed documentary film Viva Amiga is a retro love letter to the freaks, geeks, and geniuses behind the best damned computer ever made: the Amiga.

Source: Viva Amiga: The Story of a Beautiful Machine | Documentary

The post Viva Amiga: The Story of a Beautiful Machine | Documentary appeared first on dowe.io.

Categories: LUG Community Blogs

Debian Bits: New Debian Developers and Maintainers (November and December 2016)

Planet HantsLUG - Mon, 09/01/2017 - 00:30

The following contributors got their Debian Developer accounts in the last two months:

  • Karen M Sandler (karen)
  • Sebastien Badia (sbadia)
  • Christos Trochalakis (ctrochalakis)
  • Adrian Bunk (bunk)
  • Michael Lustfield (mtecknology)
  • James Clarke (jrtc27)
  • Sean Whitton (spwhitton)
  • Jerome Georges Benoit (calculus)
  • Daniel Lange (dlange)
  • Christoph Biedl (cbiedl)
  • Gustavo Panizzo (gefa)
  • Gert Wollny (gewo)
  • Benjamin Barenblat (bbaren)
  • Giovani Augusto Ferreira (giovani)
  • Mechtilde Stehmann (mechtilde)
  • Christopher Stuart Hoskin (mans0954)

The following contributors were added as Debian Maintainers in the last two months:

  • Dmitry Bogatov
  • Dominik George
  • Gordon Ball
  • Sruthi Chandran
  • Michael Shuler
  • Filip Pytloun
  • Mario Anthony Limonciello
  • Julien Puydt
  • Nicholas D Steeves
  • Raoul Snyman

Congratulations!

Categories: LUG Community Blogs

Steve Kemp: Patching scp and other updates.

Planet HantsLUG - Sun, 08/01/2017 - 17:39

I use openssh every day, be it the ssh command for connecting to remote hosts, or the scp command for uploading/downloading files.

Once a day, or more, I forget that scp uses the non-obvious -P flag for specifying the port, not the -p flag that ssh uses.

Enough is enough. I shall not file a bug report against the Debian openssh-client page, because no doubt compatibility with both upstream, and other distributions, is important. But damnit I've had enough.

apt-get source openssh-client shows the appropriate code:

fflag = tflag = 0; while ((ch = getopt(argc, argv, "dfl:prtvBCc:i:P:q12346S:o:F:")) != -1) switch (ch) { .. .. case 'P': addargs(&remote_remote_args, "-p"); addargs(&remote_remote_args, "%s", optarg); addargs(&args, "-p"); addargs(&args, "%s", optarg); break; .. .. case 'p': pflag = 1; break; .. .. ..

Swapping those two flags around, and updating the format string appropriately, was sufficient to do the necessary.

In other news I've done some hardware development, using both Arduino boards and the WeMos D1-mini. I'm still at the stage where I'm flashing lights, and doing similarly trivial things:

I have more complex projects planned for the future, but these are on-hold until the appropriate parts are delivered:

  • MP3 playback.
  • Bluetooth-speakers.
  • Washing machine alarm.
  • LCD clock, with time set by NTP, and relay control.

Even with a few LEDs though I've had fun, for example writing a trivial binary display.

Categories: LUG Community Blogs

Jonathan McDowell: 2016 in 50 Words

Planet ALUG - Fri, 06/01/2017 - 09:03

Idea via Roger. Roughly chronological order. Some things were obvious inclusions but it was interesting to go back and look at the year to get to the full 50 words.

Speaking at BelFOSS. Earthlings birthday. ATtiny hacking. Speaking at ISCTSJ. Dublin Anomaly. Co-habiting. DebConf. Peak Lion. Laura’s wedding. Christmas + picnic. Engagement. Car accident. Car write off. Tennent’s Vital. Dissertation. OMGWTFBBQ. BSides. New job. Rachel’s wedding. Digital Privacy talk. Graduation. All The Christmas Dinners. IMDB Top 250. Shay leaving drinks.

(This also serves as a test to see if I’ve correctly updated Planet Debian to use https and my new Hackergotchi that at least looks a bit more like I currently do.)

Categories: LUG Community Blogs

Steve Kemp: So I'm gonna start doing arduino-things

Planet HantsLUG - Sat, 31/12/2016 - 20:11

Since I've got a few weeks off I've decided I need to find a project, or two, to occupy me. Happily the baby is settling in well, mostly he sleeps for 4-5 hours, then eats, before the cycle repeats. It could have been so much worse.

My plan is to start exploring Arduino-related projects. It has been years since I touched hardware, with the exception of building a new PC for myself every 12-48 months.

There are a few "starter kits" you can buy, consisting of a board, and some discrete components such as a bunch of buttons, an LCD-output screen, some sensors (pressure, water, tilt), etc.

There are also some nifty little pre-cooked components you can buy such as:

The appeal of the former is that I can get the hang of marrying hardware with software, and the appeal of the latter is that the whole thing is pre-built, so I don't need to worry about anything complex. Looking over similar builds people have made, the process is more akin to building with Lego than real hardware-assembling.

So, for the next few weeks my plan is to :

  • Explore the various sensors, and tutorials, via the starter-kit.
  • Wire the MP3-playback device to a wireless D1-mini-board.
    • Which will allow me to listen to (static) music stored on an SD-card.
    • And sending "next", "previous", "play", "volume-up", etc, via a mobile.

The end result should be that I will be able to listen to music in my living room. Albeit in a constrained fashion (if I want to change the music I'll have to swap out the files on the SD-card). But it's something that's vaguely useful, and something that I think is within my capability, even as a beginner.

I'm actually not sure what else I could usefully do, but I figured I could probably wire up a vibration sensor to another wireless board. The device can sit on the top of my washing machine:

  • If vibration is sensed move into the "washing is on" state.
    • If vibration stops after a few minutes move into the "washing machine done" state.
      • Send a HTTP GET-request, which will trigger an SMS/similar.

There's probably more to it than that, but I expect that a simple vibration sensor will be sufficient to allow me to get an alert of some kind when the washing machine is ready to be emptied - and I don't need to poke inside the guts of the washing machine, nor hang reed-switches off the door, etc.

Anyway the only downside to my plan is that no doubt shipping the toys from AliExpress will take 2-4 weeks. Oops.

Categories: LUG Community Blogs

Jonathan McDowell: IMDB Top 250: Complete. Sort of.

Planet ALUG - Sat, 31/12/2016 - 17:01

Back in 2010, inspired by Juliet, I set about doing 101 things in 1001 days. I had various levels of success, but one of the things I did complete was the aim of watching half of the IMDB Top 250. I didn’t stop at that point, but continued to work through it at a much slower pace until I realised that through the Queen’s library I had access to quite a few DVDs of things I was missing, and that it was perfectly possible to complete the list by the end of 2016. So I did.

I should point out that I didn’t set out to watch the list because I’m some massive film buff. It was more a mixture of watching things that I wouldn’t otherwise choose to, and also watching things I knew were providing cultural underpinnings to films I had already watched and enjoyed. That said, people have asked for some sort of write up when I was done. So here are some random observations, which are almost certainly not what they were looking for.

My favourite film is not in the Top 250

First question anyone asks is “What’s your favourite film?”. That depends a lot on what I’m in the mood for really, but fairly consistently my answer is The Hunt for Red October. This has never been in the Top 250 that I’ve noticed. Which either says a lot about my taste in films, or the Top 250, or both. Das Boot was in the list and I would highly recommend it (but then I like all submarine movies it seems).

The Shawshank Redemption is overrated

I can’t recall a time when The Shawshank Redemption was not top of the list. It’s a good film, and I’ve watched it many times, but I don’t think it’s good enough to justify its seemingly unbroken run. I don’t have a suggestion for a replacement, however.

The list is constantly changing

I say I’ve completed the Top 250, but that’s working from a snapshot I took back in 2010. Today the site is telling me I’ve watched 215 of the current list. Last night it was 214 and I haven’t watched anything in between. Some of those are films released since 2010 (in particular new releases often enter high and then fall out of the list over a month or two), but the current list has films as old as 1928 (The Passion of Joan of Arc) that weren’t there back in 2010. So keeping up to date is not simply a matter of watching new releases.

The best way to watch the list is terrestrial TV

There were various methods I used to watch the list. Some I’d seen in the cinema when they came out (or was able to catch that way anyway - the QFT showed Duck Soup, for example). Netflix and Amazon Video had some films, but overall a very disappointing percentage. The QUB Library, as previously mentioned, had a good number of DVDs on the list (especially the older things). I ended up buying a few (Dial M for Murder on 3D Bluray was well worth it; it’s beautifully shot and unobtrusively 3D), borrowed a few from friends and ended up finishing off the list by a Lovefilm one month free trial. The single best source, however, was UK terrestrial TV. Over the past 6 years Freeview (the free-to-air service here) had the highest percentage of the list available. Of course this requires some degree of organisation to make sure you don’t miss things.

Films I enjoyed

Not necessarily my favourite, but things I wouldn’t have necessarily watched and was pleasantly surprised by. No particular order, and I’m leaving out a lot of films I really enjoyed but would have got around to watching anyway.

  • Clint Eastwood films - Gran Torino and Million Dollar Baby were both excellent but neither would have appealed to me at first glance. I hated Unforgiven though.
  • Jimmy Stewart. I’m not a fan of It’s a Wonderful Life (which I’d already watched because it’s Lister’s favourite film), but Harvey is obviously the basis of lots of imaginary friend movies and Rear Window explained a Simpsons episode (there were a lot of Simpsons episodes explained by watching the list).
  • Spaghetti Westerns. I wouldn’t have thought they were my thing, but I really enjoyed the Sergio Leone films (A Fistful of Dollars etc.). You can see where Tarantino gets a lot of his inspiration.
  • Foreign language films. I wouldn’t normally seek these out. And in general it seems I cannot get on with Italian films (except Life is Beautiful), but Amores Perros, Amelie and Ikiru were all better than expected.
  • Kind Hearts and Coronets. For some reason I didn’t watch this until almost the end; I think the title always put me off. Turned out to be very enjoyable.
Films I didn’t enjoy

I’m sure these mark me out as not being a film buff, but there are various things I would have turned off if I’d caught them by accident rather than setting out to watch them.

I’ve kept the full list available, if you’re curious.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in December 2016

Planet ALUG - Sat, 31/12/2016 - 11:40

Here is my monthly update covering what I have been doing in the free software world (previous month):

  • Celebrated my 10-year anniversary of contributing to Debian. An excerpt of this post was quoted on LWN.
  • Made a number of improvements to AptFS, my FUSE-based filesystem that provides a view on unpacked Debian source packages as regular folders, including move from the popen2 Python module to subprocess and correcting the parsing of package lists.
  • Corrected an UnboundLocalError exception in the Finnish social security number generator in faker, a tool to generate test data in Python applications. (#441)
  • Made a small change to travis.debian.net (my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every code change) to fix an issue with malformed YAML.
  • Added the ability to specify the clone target to gbp-import-dsc etc. in git-buildpackage, a tool to build Debian packages using Git. (commit)
  • Filed three issues against the Redis key-value database:
    • Tests fail on the alpha architecture due to "memory efficiency". (#3666)
    • Please update hiredis (#3687)
    • Correct "whenever" typo. (#3652)
Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to permit verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

This month:

I also made the following changes to our tooling:

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

  • Optimisations:
    • Avoid unnecessary string manipulation writing --text output (~20x speedup).
    • Avoid n iterations over archive files (~8x speedup).
    • Don't analyse .deb s twice when comparing .changes files (2x speedup).
    • Avoid shelling out to colordiff by implementing color support directly.
    • Memoize calls to distutils.spawn.find_executable to avoid excessive stat(1) syscalls.
  • Progress bar:
    • Show current file / ELF section under analysis etc. in progress bar.
    • Move the --status-fd output to use JSON and to include the current filename.
  • Code tidying:
    • Split out the try.diffoscope.org client so that it can be released separately on PyPI.
    • Completely rework the diffoscope and diffoscope.comparators modules, grouping similar utilities into their own modules, etc.
  • Miscellaneous:
    • Update dex_expected_diffs test to ensure compatibility with enjarify ≥ 1.0.3.
    • Ensure that running from Git will always use that checkout's Python modules.
    • Add a simple profiling framework.

strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.

  • Makefile.PL: Change NAME argument to a Perl package name.
  • Ensure our binaries are available in autopkgtest tests.

try.diffoscope.org

trydiffoscope is a web-based version of the diffoscope in-depth and content-aware diff utility. Continued thanks to Bytemark for sponsoring the hardware.

  • Show progress bar and position in queue, etc. (#25 & #26)
  • Promote command-line client with PyPI instructions.
  • Increase comparison time limit to 90 seconds.

buildinfo.debian.net

buildinfo.debian.net is my experiment into how to process, store and distribute .buildinfo files after the Debian archive software has processed them.

  • Added support for version 0.2 .buildinfo files. (#15)

Debian Debian LTS

This month I have been paid to work 13½ hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 733-1 for openafs, fixing an information leak vulnerability. Due to incomplete initialization or clearing of reused memory, directory objects could contain 'dead' directory entry information.
  • Issued DLA 734-1 for mapserver closing an information leakage vulnerability.
  • Issued DLA 737-1 for roundcube preventing arbitrary remote code execution by sending a specially crafted email.
  • Issued DLA 738-1 for spip patching a cross-site scripting (XSS) vulnerability.
  • Issued DLA 740-1 for libgsf fixing a null pointer deference exploit via a crafted .tar file.
Debian Uploads
  • redis:
    • 3.2.5-5 — Add RunTimeDirectory=redis to systemd .service files.
    • 3.2.5-6 — Add missing Depends on lsb-base for /lib/lsb/init-functions usage in redis-sentinel's initscript.
    • 3.2.6-1 — New upstream release.
    • 4.0-1 & 4.0-rc2-1 — New upstream experimental releases.
  • aptfs: 0.9-1 & 0.10-1 — New upstream releases.
Debian bugs filed

I filed 29 FTBFS bugs against a7xpg, conntrack-tools, factory-boy, faker, glimpse, gunroar, hexchat-otr, jackson-datatype-guava, jalview, jquery, kodi-pvr-mythtv, leap-cli, libbio-graphics-perl, libparanoid-perl, libsass-python, metastudent-data, node-temporary, node-yargs, python-requests-unixsocket, python-restless, ruby-bunny, ruby-github-markup, ruby-rabl, sagenb-export, seaborn, soapdenovo2, titanion, ufw & vagrant-cachier.

I additionally filed 2 bugs for packages that access the internet during build against fence-agents & lua-geoip.

Debian FTP Team

As a Debian FTP assistant I ACCEPTed 107 packages: android-platform-libcore, compiz, debian-edu, dehydrated, dh-cargo, gnome-shell-extension-pixelsaver, golang-1.8, golang-github-btcsuite-btcd-btcec, golang-github-elithrar-simple-scrypt, golang-github-pelletier-go-toml, golang-github-restic-chunker, golang-github-weaveworks-mesh, golang-google-genproto, igmpproxy, jimfs, kpmcore, libbio-coordinate-perl, libdata-treedumper-oo-perl, libdate-holidays-de-perl, libpgobject-type-bytestring-perl, libspecio-library-path-tiny-perl, libterm-table-perl, libtext-hogan-perl, lighttpd, linux, linux-signed, llmnrd, lua-geoip, lua-sandbox-extensions, lua-systemd, node-cli-cursor, node-command-join, node-death, node-detect-indent, node-domhandler, node-duplexify, node-end-of-stream, node-first-chunk-stream, node-from2, node-glob-stream, node-has-binary, node-inquirer, node-interpret, node-is-negated-glob, node-is-unc-path, node-lazy-debug-legacy, node-lazystream, node-load-grunt-tasks, node-merge-stream, node-object-assign-sorted, node-orchestrator, node-pkg-up, node-resolve-from, node-resolve-pkg, node-rx, node-sorted-object, node-stream-shift, node-streamtest, node-string.prototype.codepointat, node-strip-bom-stream, node-through2-filter, node-to-absolute-glob, node-unc-path-regex, node-vinyl, openzwave, openzwave-controlpanel, pcb-rnd, pd-upp, pg-partman, postgresql-common, pybigwig, python-acora, python-cartopy, python-codegen, python-efilter, python-flask-sockets, python-intervaltree, python-jsbeautifier, python-portpicker, python-pretty-yaml, python-protobix, python-sigmavirus24-urltemplate, python-sqlsoup, python-tinycss, python-watson-developer-cloud, python-zc.customdoctests, python-zeep, r-cran-dbitest, r-cran-dynlm, r-cran-mcmcpack, r-cran-memoise, r-cran-modelmetrics, r-cran-plogr, r-cran-prettyunits, r-cran-progress, r-cran-withr, ruby-clean-test, ruby-gli, ruby-json-pure, ruby-parallel, rustc, sagemath, sbuild, scram, sidedoor, toolz & yabasic.

I additionally filed 4 RC bugs against packages that had incomplete debian/copyright files against jimfs, compiz, python-efilter & ruby-json-pure.

Categories: LUG Community Blogs

Chris Lamb: My favourite books of 2016

Planet ALUG - Fri, 30/12/2016 - 19:51

Whilst I managed to read almost sixty books in 2016 here are ten of my favourites in no particular order.

Disappointments this year include Stewart Lee's Content Provider (nothing like his stand-up), Christopher Hitchens' And Yet (his best essays are already published) and Heinlein's Stranger in a Strange Land (great exposition, bizarre conclusion).

The worst book I finished, by far, was Mark Edward's Follow You Home.






Animal QC

Gary Bell, QC

Subtitled My Preposterous Life, this rags-to-riches story about a working-class boy turned eminent lawyer would be highly readable as a dry and factual account but I am compelled to include it here for its extremely entertaining style of writing.

Full of unsurprising quotes that take one unaware: would you really expect a now-Queen's Counsel to "heartily suggest that if you find yourself suffering from dysentery in foreign climes you do not medicate it with lobster thermidor and a bottle of Ecuadorian red?"

A real good yarn.

So You've Been Publically Shamed

Jon Ronson

The author was initially recommended to me by Brad but I believe I started out with the wrong book. In fact, I even had my doubts about this one, prematurely judging from the title that it was merely cashing-in on a fairly recent internet phenomenon — like his more recent shallow take on Trump and the alt-Right — but in the end I read Publically Shamed thrice in quick succession.

I would particularly endorse the audiobook version: Ronson's deadpan drawl suits his writing perfectly.

The Obstacle is the Way

Ryan Holiday

Whilst everyone else appears to be obligated to include Ryan's recent Ego is the Enemy in their Best of 2016 lists I was actually taken by his earlier "introduction by stealth" to stoic philosophy.

Certainly not your typical self-help book, this is "a manual to turn to in troubling times".

Returning to this work at least three times over the year — even splashing out on the audiobook at some point — I feel like I learned a great deal, although it is now difficult to pinpoint exactly what. Perhaps another read in 2017 is thus in order…

Layer Cake

J.J. Connolly

To judge a book in comparison to the film is to do both a disservice, but reading the book of Layer Cake really underscored just how well the film played to the strengths of that medium.

All of the aspects that would not have worked had been carefully excised from the screenplay, ironically leaving more rewarding "layers" for readers attempting the book. A parallel adaption here might be No Country for Old Men - I would love to read (or write) a comparative essay between these two adaptions although McCarthy's novel is certainly the superior source material.

Lying

Sam Harris

I've absorbed a lot of Sam Harris's œuvre this year in the form of his books but moreover via his compelling podcast. I'm especially fond of Waking Up on spirituality without religion and would rank that as my favourite work of his.

Lying is a comparatively short read, more of a long essay in fact, where he argues that we can radically simplify our lives by merely telling the truth in situations where others invariably lie. Whilst it would take a brave soul to adopt his approach his case is superlatively well-argued and a delight to read.

Letters from a Stoic

Seneca

Great pleasure is to be found not only in keeping up an old and established friendship but also in beginning and building up a new one.

Reading this in a beautifully svelte hardback, I tackled a randomly-chosen letter per day rather than attempting to read it cover-to-cover. Breaking with a life-long tradition, I even decided to highlight sections in pen so I could return to them at ease.

I hope it's not too hackneyed to claim I gained a lot from "building up" a relationship with this book. Alas, it is one of those books that is too easy to recommend given that it might make one appear wise and learned, but if you find yourself in a slump, either in life or in your reading habits, it certainly has my approval.

Solo: A James Bond Novel

William Boyd

I must have read all of the canonical Fleming novels as a teenager and Solo really rewards anyone who has done so. It would certainly punish anyone expecting a Goldeneye or at least be a little too foreign to be enjoyed.

Indeed, its really a pastiche of these originals, both in terms of the time period, general tone (Bond is more somber; more vulnerable) and in various obsessions of Fleming's writing, such as the overly-detailed description of the gambling and dining tables. In this universe, 007's restaurant expenses probably contributed signifcantly to the downfall of the British Empire, let alone his waistline.

Bond flicking through a ornithological book at one point was a cute touch…

The Subtle Art of Not Giving A F*ck

Mark Manson

Certainly a wildcard to include here and not without its problems, The Subtle Art… is a curious manifesto on how to approach life. Whilst Manson expouses an age-old philosophy of grounding yourself and ignoring the accumulation of flatscreen TVs, etc. he manages to do so in a fresh and provocative "21st-centry gonzo" style.

Highly entertaining, at one point the author posits an alternative superhero ("Disappointment Panda") that dishes out unsolicited and uncomfortable truths to strangers before simply walking away: "You know, if you make more money, that’s not going to make your kids love you," or: "What you consider friendship is really just your constant attempts to impress people."

Ouch.

The Fourth Protocol

Frederick Forsyth

I have a crystal-clear memory from my childhood of watching a single scene from a film in the dead of night: Pierce Brosnan sets a nuclear device to detonate after he can get away but a double-crossing accomplice surreptitiously brings the timetable forward in order that the bomb also disposes of him…

Anyway, at some point whilst reading The Fourth Protocol it dawned on me that this was that book. I might thus be giving the book more credit due to this highly satisfying connection but I think it stands alone as a superlative political page-turner and is still approachable outside the machinations of the Cold War.

The Partner

John Grisham

After indulging in a bit too much non-fiction and an aborted attempt at The Ministry of Fear, I turned to a few so-called lower-brow writers such as Jeffrey Archer, etc.

However, it was The Partner that turned out to be a real page-turner for somewhat undefinable reasons. Alas, it appears the rest of the author's output is unfortunately in the same vein (laywers, etc.) so I am hesitant to immediately begin others but judging from various lists online I am glad I approached this one first.

Shogun: The First Novel of the Asian saga

James Clavell

Despite its length, I simply couldn't resist returning to Shogun this year although it did fatigue me to the point that I have still yet to commence on its sequel, Tai-Pan.

Like any good musical composition, one is always rewarded by returning to a book and I took great delight in uncovering more symbolism throughout (such as noticing that one of the first words Blackthorne learns in Japanese is "truth") but also really savouring the tragic arcs that run throughout the novel, some beautiful phrases ("The day seemed to lose its warmth…") and its wistful themes of inevitability and karma.

Categories: LUG Community Blogs

Steve Kemp: I finally made something worthwhile.

Planet HantsLUG - Mon, 26/12/2016 - 10:34

So for once I made something useful.

Oiva Adam Kemp.

Happy Christmas, if you believe in that kind of thing.

Categories: LUG Community Blogs

Jonathan McDowell: The terrible PIC ecosystem

Planet ALUG - Sun, 25/12/2016 - 01:51

I recently had call to play with some 1-Wire devices at work (more of which in a future post). It was taking a while for the appropriate programmer to turn up, so of course I pulled out my trusty BusPirate. It turned out the devices in question would only talk in overdrive mode, while the Bus Pirate could only offer standard mode. So I set about trying to figure out how to add the appropriate support.

This is is a huge endorsement for test equipment with Free Software firmware. Rather than giving up I was able to go and grab the current firmware, which has been adopted by the community since Dangerous Prototypes have discontinued development. What let me down was the ecosystem around the PIC24FJ64GA002.

My previous recent experience with microcontrollers has been with the ATTiny range and the STM32. Getting up and running with both of these was fairly easy - the tool chains necessary were already present in Debian, so all it took was a simple apt invocation to install everything I needed to compile code and program it to the devices.

Not so with the PIC series, which surprised me. There seems to be some basic support for the earlier PIC16 range, but for later chips there’s nothing that works out of the box with Debian. Investigation revealed that this was because there’s nothing maintained that enabled Free development for the PIC range. The accepted solution is the closed MPLAB X. Now, in one sense fair play to Microchip for making this available. But in another, shame on you. I can’t imagine ever choosing to build something based on a chip that only had a closed source tool chain available. I want things I can use in Makefiles and properly script, that are available in my distro of choice and that generally work in the same fashion as the tool chains I’m used to. I understand there might be some benefit in a closed compiler in terms of performance (and have HPC friends who would never trust a benchmark provided using GCC), but in general that’s not the space I move in. Nor does it seem to be the sort of attitude you should be taking if you are trying to attract the hobbyist and small production run market.

Any yet this seems common amongst hardware manufacturers. People whose core business is selling physical items, where the software is only relevant in terms of being able to use those items, seem to consider the software to be precious. Instead of opening up programming specifications and allowing a more widespread use of the hardware, increasing sales. I understand there are some cases where this isn’t practical, but the default attitude is definitely one of being closed rather than open, which is a terrible shame.

Anyway. I do have some Bus Pirate 1-Wire overdrive support now working (pending some testing to ensure standard mode still works), but I am glad I never spent a lot of time getting involved with PICs now.

Categories: LUG Community Blogs

Mick Morgan: Merry Christmas 2016

Planet ALUG - Sat, 24/12/2016 - 20:08

As is now traditional :-) I post today to wish everyone a very merry christmas.

Today is trivia’s birthday – indeed it is trivia’s 10th birthday so I have been writing here for a decade. Good grief. If I had known then what I know now trivia might have been still born. As it is we are both still here – more importantly so is everyone else I really care about.

Here’s to the next 10 years. And I might actually write some more next year.

Best Wishes

Mick

Categories: LUG Community Blogs

Debian Bits: Free FPGA programming with Debian

Planet HantsLUG - Thu, 22/12/2016 - 18:15

FPGA (Field Programmable Gate Array) are increasingly popular for data acquisition, device control and application acceleration. Debian now features a completely Free set of tools to program FPGA in Verilog, prepare the binary and have it executed on an affordable device.

See http://wiki.debian.org/FPGA/Lattice for details. Readers familiar with the technology may rightly guess that this refers to the yosys package together with berkeley-abc, arachne-"Place-and-Route" and the icestorm tools to communicate with the device.

The packages have been contributed by the Debian Science team.

We hope this effort to support the FPGA community to collect an increasing number of skills to further smoothen the Open Source experience and lower the entry barriers for this tantalising technology.

Categories: LUG Community Blogs
Syndicate content