Planet ALUG

Syndicate content
Planet ALUG - http://planet.alug.org.uk/
Updated: 8 min 20 sec ago

Mick Morgan: raid performance

Mon, 02/05/2016 - 16:46

I have recently been building a new NAS box (of which, possibly, more later). In fact the build is really a rebuild because I initially built the server about three years ago in order to consolidate a bunch of services I was running on assorted separate servers into one place. That first build was a RAID 1 array of two 2 TB disks (to give me a mirrored setup with a total of 2 TB store). At the time that was sufficient to hold all my important data (backed up to both other networked devices and standalone USB disks for safety). But I have just upgraded my main desktop machine to a nice shiny new core i7 Skylake box with 16 GB of DDR4 and a 3 TB disk. That disk is already two thirds full (my old machine had a rather full 2 TB disk). This meant that my NAS backup storage requirements exceeded the capacity of my RAID 1 setup. Adding disks wouldn’t help of course because all that would do is add mirror capability rather than capacity. So I decided to upgrade the NAS and bought a bunch of new 2 TB disks with the intention of setting up a RAID 5 array of 4 disks, thus giving a total storage capacity of 6 TB (8 TB minus 2 TB for parity). Furthermore I initially looked at using FreeNAS rather than my usual debian or ubuntu server with software RAID simply because it looked interesting and, with plugins, could probably meet most of my requirements. But I could not get the software to install properly and after three abortive attempts I gave up and deecided that I didn’t really like freeBSD anyway….

So I decided to go back to mdadm on linux – at least I know that works. Better still I would be able to retain all my old setup from the old RAID 1 system without having to worry about finding plugins to handle my media streaming requirements, or owncloud installation, for example.

My previous build was on debian (which is by far my preferred server OS) but ubuntu server has recently been released in a LTS version at 16.04 and I thought it might be fun to try that instead. So I did. (For any readers who have not tried installing linux on a RAID system there are plenty of sites offering advice, but the official ubuntu pages are pretty good). During the build I hit what I initially thought was a snag because the installation seemed to get stuck at around the 83% level when it was apparently installing the linux kernel image and headers. Indeed I confess that on the first such installation I pulled the plug after about three hours of no apparent activity because I was beginning to think that there might be something wrong with my hardware (the earlier FreeNAS failures worried me). My on-line searches for assistance were initially not particularly helpful since none of the huge number of sites advising on software RAID installation bothered to mention that initial RAID 5 build (or rebuild) using large capacity disks takes a very long time because of the need to calculate the parity data. Incidentally, it is this parity data and its layout that gives RAID 5 its write performance penalty.

One useful outcome of my research about RAID 5 build times (which in my case took just over 6 hours) was my discovery of the wintelguy’s site providing an on-line calculator (and much more besides) for RAID performance and capacity. There is even a very useful page allowing you to compare two separate configurations side by side – thoroughly recommended. More worrying, and thought provoking, is the reclaime.com calculator for RAID failure. That site suggests that the probability of successfully rebuilding a RAID 5 array of 4 * 2 TB disks after a failure is only 52.8%.

That is why you need to keep backups…….

Categories: LUG Community Blogs

Chris Lamb: Free software activities in April 2016

Sat, 30/04/2016 - 23:20

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

  • Added Python 3 support to django-template-tests, a tool to perform simple static analysis on Django templates. (#1)
  • Corrected my Chrome extension for the FastMail web interface to not disable the CTRL+Enter keyboard shortcut when authoring emails. (#3)
  • Corrected a subtle bug in my django-staticfiles-dotd "staticfiles" library where the Content-Length HTTP header was calculated incorrectly in the presence of Unicode characters resulting in truncated output. (#2)
  • Various fixes to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
    • Don't require an explicit backend import when using the Celery task queue backend. (#41)
    • Actually generate and send messages asynchronously when using the Celery backend. (#44)
  • Fixed an issue with my local-debian-mirror tool where the option to disable DEP-11 mirroring wasn't working. (#1)
  • Fixed an issue in django-hipchat, a library to easily post messages to the Hipchat group-messaging utility from projects using the Django web development framework where the templates were not includes when installing via PyPI. (#1)
  • Created a quick-and-dirty tool to scrape a Squarespace blog and convert it to a PDF so I can read them on my Kindle e-reader. (tree)
  • Updated django-keyerror — a library to post exceptions to the KeyError.com error tracking service — to silence an AttributeError exception in some error-reporting edge-cases. (commit)
  • Suggested an improvement to the documentation for the upcoming Twitter Bootstrap version for the deprecated .hidden and .show CSS classes. (#19789)
  • Submitted a documentation update to the Ansible sever configuration tool's ufw firewall module. (commit)
  • I also blogged about parsing Jenkins CI output to determine job success or failure.
Debian

My work in the Reproducible Builds project was covered in our weekly reports. (#48, #49, #50, #51 & #52)

Uploads
  • redis (2:3.0.7-3) — Adding, amongst some other changes, systemd LimitNOFILE support to allow a higher number of open file descriptors.
RC bugs

I filed 58 FTBFS bugs against agg, basex, c++-annotations, camera.app, cl-babel, cl-lparallel, collab-qa-tools, diagnostics, enjarify, enscript, felix-main, girara, gnome-shell-pomodoro, golang-github-spf13-viper, gst-plugins-base0.10, gstreamer0.10, guessnet, htslib, ifrit, indicator-session, jackson-module-afterburner, kamera, lgogdownloader, libbde, libmlx4, libsqlite3-0:, libykneomgr, nifti2dicom, node-starttls, nuitka, oath-toolkit, pdf-presenter-console, perlbal, poker-engine, pycountry, pysycache, python-oslo.privsep, python-shade, r-cran-tgp, raincat, rapache, resteasy, ruby-crb-blast, ruby-email-reply-parser, ruby-gollum-lib, samtools, sipwitch, sooperlooper, tomcat-maven-plugin, transmission, trivial-features, tuskar-ui, twinkle, twisted-web2, uclmmbase, workrave, xlog & yafc.

FTP Team

As a Debian FTP assistant I ACCEPTed 135 packages: aptitude, asm, beagle, blends, btrfs-progs, camitk, cegui-mk2, cmor-tables, containerd, debian-science, debops, debops-playbooks, designate-dashboard, efitools, facedetect, flask-testing, fstl, ganeti-os-noop, gnupg, golang-fsnotify, golang-github-appc-goaci, golang-github-benbjohnson-tmpl, golang-github-dchest-safefile, golang-github-docker-go, golang-github-dylanmei-winrmtest, golang-github-hawkular-hawkular-client-go, golang-github-hlandau-degoutils, golang-github-hpcloud-tail, golang-github-klauspost-pgzip, golang-github-kyokomi-emoji, golang-github-masterminds-semver-dev, golang-github-masterminds-vcs-dev, golang-github-masterzen-xmlpath, golang-github-mitchellh-ioprogress, golang-github-smartystreets-assertions, golang-gopkg-hlandau-configurable.v1, golang-gopkg-hlandau-easyconfig.v1, golang-gopkg-hlandau-service.v2, golang-objx, golang-pty, golang-text, gpaste, gradle-plugin-protobuf, grip, haskell-brick, haskell-hledger-ui, haskell-lambdabot-haskell-plugins, haskell-text-zipper, haskell-werewolf, hkgerman, howdoi, jupyter-client, jupyter-core, letsencrypt.sh, libbpp-phyl, libbpp-raa, libbpp-seq, libbpp-seq-omics, libcbor-xs-perl, libdancer-plugin-email-perl, libdata-page-pageset-perl, libevt, libevtx, libgit-version-compare-perl, libgovirt, libmsiecf, libnet-ldap-server-test-perl, libpgobject-type-datetime-perl, libpgobject-type-json-perl, libpng1.6, librest-client-perl, libsecp256k1, libsmali-java, libtemplates-parser, libtest-requires-git-perl, libtext-xslate-perl, linux, linux-signed, mandelbulber2, netlib-java, nginx, node-rc, node-utml, nvidia-cuda-toolkit, openfst, openjdk-9, openssl, php-cache-integration-tests, pulseaudio, pyfr, pygccxml, pytest-runner, python-adventure, python-arrayfire, python-django-feincms, python-fastimport, python-fitsio, python-imagesize, python-lib389, python-libtrace, python-neovim-gui, python3-proselint, pythonpy, pyzo, r-cran-ca, r-cran-fitbitscraper, r-cran-goftest, r-cran-rnexml, r-cran-rprotobuf, rrdtool, ruby-proxifier, ruby-seamless-database-pool, ruby-syslog-logger, rustc, s5, sahara-dashboard, salt-formula-ceilometer, salt-formula-cinder, salt-formula-glance, salt-formula-heat, salt-formula-horizon, salt-formula-keystone, salt-formula-neutron, salt-formula-nova, seer, simplejson, smrtanalysis, tiles-autotag, tqdm, tran, trove-dashboard, vim, vulkan, xapian-bindings & xapian-core.

Categories: LUG Community Blogs

Jonathan McDowell: Notes on Kodi + IR remotes

Tue, 26/04/2016 - 21:32

This post is largely to remind myself of the details next time I hit something similar; I found bits of relevant information all over the place, but not in one single location.

I love Kodi. These days the Debian packages give me a nice out of the box experience that is easy to use. The problem comes in dealing with remote controls and making best use of the available buttons. In particular I want to upgrade the VDR setup my parents have to a more modern machine that’s capable of running Kodi. In this instance an AMD E350 nettop, which isn’t recent but does have sufficient hardware acceleration of video decoding to do the job. Plus it has a built in fintek CIR setup.

First step was finding a decent remote. The fintek is a proper IR receiver supported by the in-kernel decoding options, so I had a lot of flexibility. As it happened I ended up with a surplus to requirements Virgin V Box HD remote (URC174000-04R01). This has the advantage of looking exactly like a STB remote, because it is one.

Pointed it at the box, saw that the fintek_cir module was already installed and fired up irrecord. Failed to get it to actually record properly. Googled lots. Found ir-keytable. Fired up ir-keytable -t and managed to get sensible output with the RC-5 decoder. Used irrecord -l to get a list of valid button names and proceed to construct a vboxhd file which I dropped in /etc/rc_keymaps/. I then added a

fintek-cir * vboxhd

line to /etc/rc_maps.cfg to force my new keymap to be loaded on boot.

That got my remote working, but then came the issue of dealing with the fact that some keys worked fine in Kodi and others didn’t. This seems to be an issue with scancodes above 0xff. I could have remapped the remote not to use any of these, but instead I went down the inputlirc approach (which is already in use on the existing VDR box).

For this I needed a stable device file to point it at; the /dev/input/eventN file wasn’t stable and as a platform device it didn’t end up with a useful entry in /dev/input/by-id. A ‘quick’

udevadm info -a -p $(udevadm info -q path -n /dev/input/eventN)

provided me with the PNP id (FIT0002) allowing me to create /etc/udev/rules.d/70-remote-control.rules containing

KERNEL=="event*",ATTRS{id}=="FIT0002",SYMLINK="input/remote"

Bingo, a /dev/input/remote symlink. /etc/defaults/inputlirc ended up containing:

EVENTS="/dev/input/remote" OPTIONS="-g -m 0"

The options tell it to grab the device for its own exclusive use, and to take all scancodes rather than letting the keyboard ones through to the normal keyboard layer. I didn’t want anything other than things specifically configured to use the remote to get the key presses.

At this point Kodi refused to actually do anything with the key presses. Looking at ~kodi/.kodi/temp/kodi.log I could see them getting seen, but not understood. Further searching led me to construct an Lircmap.xml - in particular the piece I needed was the <remote device="/dev/input/remote"> bit. The existing /usr/share/kodi/system/Lircmap.xml provided a good starting point for what I wanted and I dropped my generated file in ~kodi/.kodi/userdata/.

(Sadly it turns out I got lucky with the remote; it seems to be using the RC-5x variant which was broken in 3.17; works fine with the 3.16 kernel in Debian 8 (jessie) but nothing later. I’ve narrowed down the offending commit and raised #117221.)

Helpful pages included:

Categories: LUG Community Blogs

Jonathan McDowell: Going to DebConf 16

Mon, 18/04/2016 - 14:12

Whoop! Looking forward to it already (though will probably spend it feeling I should be finishing my dissertation).

Outbound:

2016-07-01 15:20 DUB -> 16:45 LHR BA0837 2016-07-01 21:35 LHR -> 10:00 CPT BA0059

Inbound:

2016-07-10 19:20 CPT -> 06:15 LHR BA0058 2016-07-11 09:20 LHR -> 10:45 DUB BA0828

(image stolen from Gunnar)

Categories: LUG Community Blogs

Jonathan McDowell: Going to DebConf 16

Mon, 18/04/2016 - 14:12

Whoop! Looking forward to it already (though will probably spend it feeling I should be finishing my dissertation).

Outbound:

2016-07-01 15:20 DUB -> 16:45 LHR BA0837 2016-07-01 21:35 LHR -> 10:00 CPT BA0059

Inbound:

2016-07-10 19:20 CPT -> 06:15 LHR BA0058 2016-07-11 09:20 LHR -> 10:45 DUB BA0828

(image stolen from Gunnar)

Categories: LUG Community Blogs

Jonathan McDowell: Software in the Public Interest contributing members: Check your activity status!

Wed, 13/04/2016 - 13:04

That’s a longer title than I’d like, but I want to try and catch the attention of anyone who might have missed more directed notifications about this. If you’re not an SPI contributing member there’s probably nothing to see here…

Although I decided not to stand for re-election at the Software in the Public Interest (SPI) board elections last July, I haven’t stopped my involvement with the organisation. In particular I’ve spent some time working on an overhaul of the members website and rolling it out. One of the things this has enabled is implementation of 2009-11-04.jmd.1: Contributing membership expiry, by tracking activity in elections and providing an easy way for a member to indicate they consider themselves active even if they haven’t voted.

The plan is that this will run at some point after the completion of every board election. A first pass of cleanups was completed nearly a month ago, contacting all contributing members who’d never been seen to vote and asking them to update their status if they were still active. A second round, of people who didn’t vote in the last board election (in 2014), is currently under way. Affected members will have been emailed directly and there was a mail to spi-announce, but I’m aware people often overlook these things or filter mail off somewhere that doesn’t get read often.

If you are an SPI Contributing member who considers themselves an active member I strongly recommend you login to the SPI Members Website and check the “Last active” date displayed is after 2014-07-14 (i.e. post the start of the last board election). If it’s not, click on the “Update” link beside the date. The updated date will be shown once you’ve done so.

Why does pruning inactive members matter? The 2015 X.Org election results provide at least one indication of why ensuring you have an engaged membership is important - they failed to make a by-laws change that a vast majority of votes were in favour of, due to failing to make quorum. (If you’re an X.org member, go vote!)

Categories: LUG Community Blogs

Jonathan McDowell: Software in the Public Interest contributing members: Check your activity status!

Wed, 13/04/2016 - 13:04

That’s a longer title than I’d like, but I want to try and catch the attention of anyone who might have missed more directed notifications about this. If you’re not an SPI contributing member there’s probably nothing to see here…

Although I decided not to stand for re-election at the Software in the Public Interest (SPI) board elections last July, I haven’t stopped my involvement with the organisation. In particular I’ve spent some time working on an overhaul of the members website and rolling it out. One of the things this has enabled is implementation of 2009-11-04.jmd.1: Contributing membership expiry, by tracking activity in elections and providing an easy way for a member to indicate they consider themselves active even if they haven’t voted.

The plan is that this will run at some point after the completion of every board election. A first pass of cleanups was completed nearly a month ago, contacting all contributing members who’d never been seen to vote and asking them to update their status if they were still active. A second round, of people who didn’t vote in the last board election (in 2014), is currently under way. Affected members will have been emailed directly and there was a mail to spi-announce, but I’m aware people often overlook these things or filter mail off somewhere that doesn’t get read often.

If you are an SPI Contributing member who considers themselves an active member I strongly recommend you login to the SPI Members Website and check the “Last active” date displayed is after 2014-07-14 (i.e. post the start of the last board election). If it’s not, click on the “Update” link beside the date. The updated date will be shown once you’ve done so.

Why does pruning inactive members matter? The 2015 X.Org election results provide at least one indication of why ensuring you have an engaged membership is important - they failed to make a by-laws change that a vast majority of votes were in favour of, due to failing to make quorum. (If you’re an X.org member, go vote!)

Categories: LUG Community Blogs

Chris Lamb: Parsing Jenkins log output to determine job status

Mon, 11/04/2016 - 18:28

I recently made the same mistake a number of times when adding new hosts to my Ansible configuration and decided to ensure it couldn't happen again. The specifics of this particular issue were that whilst I had added the hostname to inventory file, I had neglected to add the host to the relevant group in my playbook:

oldhost newhost [mygroup] oldhost # missing newhost here

ansible-playbook would output no hosts matched but crucially return with an successful exit code. My continuous integration system (Jenkins) would infer that the task was successful and not notify me that anything was wrong:

$ ansible-playbook deploy-mygroup.yml --limit-hosts=newhost <snip> PLAY [deploy] ************************************************* skipping: no hosts matched PLAY RECAP **************************************************** [ERROR]: No plays were matched by any host. $ echo $? 0

This seemed to violate a few principles to me (at the very least due of the "loud" use of ERROR without the corresponding return code) so I filed a pull request against Ansible that added an optional --error-if-no-plays-matched switch:

$ ansible-playbook [..] --error-if-no-plays-matched <snip> [ERROR]: No plays were matched by any host. $ echo $? 1

In the end, upstream decided to pass on it as it could be implemented via a plugin system and desiring an immediate and potentially more-general solution I briefly looked into parsing the ansible-playbook output before moving onto parsing the Jenkins log itself.

This turned out to be straightforward; using the Text-Finder plugin, I configured my Jenkins job to simply error if the log contained the string skipping: no hosts matched:

I am using the Job DSL plugin so that my configuration is backed onto revision control (highly recommended) so I actually used its textFinder publisher rather than the interface above:

publishers { textFinder(/^skipping: no hosts matched$/, '', true, false, false) }

This results in the job "correctly" failing and alerting me:

+ ansible-playbook deploy-mygroup.yml --limit-hosts=newhost <snip> PLAY [deploy] ************************************************* skipping: no hosts matched PLAY RECAP **************************************************** [ERROR]: No plays were matched by any host. Checking console output /var/lib/jenkins/jobs/deploy-mygroup/builds/126/log: skipping: no hosts matched Build step 'Jenkins Text Finder' changed build result to FAILURE Finished: FAILURE
Categories: LUG Community Blogs

Steve Engledow (stilvoid): Disco

Fri, 08/04/2016 - 17:09

Things I discovered this week:

  • I really, really don't like sesame oil.

    I spent all day yesterday complaining that there was a smell of burnt plastic in the house. Today I shared a sandwich with my wife and it tasted like burnt plastic smells. Turns out this was the sesame oil she just bought and loves. Yuck.

  • I've become quite unfit over the past 3 months.

    Since starting my new job, I haven't had the daily cycle ride into the office (I work from home or get the train to wherever I need to be) and it's really beginning to show. I decided to go for a bike ride on my lunch break today. 30 minutes and a handful of miles later, I was a sweaty, panting mass.

  • Making smoothies using only fruit is a bad idea.

    Since buying a smoothie maker (a blender, a blender, a blender) I've been trying various strategies depending on how guilty I'm feeling. Today I decided to try a smoothie with just three ingredients: grapes (lots of them), a pear, and a dash of cranberry juice. It was, without a doubt, the sweetest thing I have ever tasted. I ended up necking some and then watering it down.

  • I should have bought a new monitor ages ago.

    I've been using a TV that I bought years ago as my main monitor for ages and recently decided I wanted something a bit better (the monitor only went up to 1680x1050) so I bought a monitor. What an improvement!

Categories: LUG Community Blogs

Wayne Stallwood (DrJeep): Simple USB2 Host Switch

Sat, 02/04/2016 - 20:38
Initially created for the BigBox 3D printer to allow use of both the Internal Raspberry Pi running Octoprint and the rear mounted USB port for diagnostic access. The Rumba has only one USB port and can only be attached to one of these at a time.

However this circuit will work in any other scenario where you want to be able to switch between USB Hosts.

Plug a Host PC or other host device into port X1 and the device you want to control into Port X3, everything should work as normal.

Plug an additional powered Host PC or other host device into Port X2 and and the host plugged into Port X1 should be disconnected in preference to this device which should now be connected to the device plugged into port X3.

Please note, in many cases, particularly with devices that are bus powered like memory sticks, the device will not function if there is no powered host PC plugged into port X1

Categories: LUG Community Blogs

Chris Lamb: Free software activities in March 2016

Thu, 31/03/2016 - 22:12

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

  • Submitted a number of pull requests to the Ansible server configuration tool:
    • Add the ability to specify the --allow-unauthenticated to APT. (#2023)
    • Ignore EPIPE when flushing standard input and output to avoid unnecessary and ugly tracebacks. (#14774)
    • Add an option for ansible-playbook to exit with an error if no plays were matched. (#14742)
  • Various improvements to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
    • Merged a patch from Joshua Blum to update the logging integration documentation. (#39)
    • Fixed an issue when rendering various exceptions would in themselves raise an exception. (#40)
    • Improved and merged a patch from Patrick Cloke to actually allow settings to be overriden using the @override_settings decorator. (#37)
  • Updated local-debian-mirror, a package that attempts to make it easy to maintain a partial local mirror of the Debian archive:
    • Added an option to exclude DEP-11 files. (ee03b21)
    • Added an option to exclude Translation-* files. (28bbf34)
    • Corrected the phrasing of the debconf prompt to not ask a direct question to appease Lintian. (13fccc7b)
  • Updated django-keyerror — a library to post exceptions to the KeyError.com error tracking service — to support pushing tracebacks from the Celery queue processor using the task_failure signal. (#1)
  • Submitted a number of pull requests for django-zebra, a collection of utilities that make it easier to integrate the Stripe payment processor and the Django web development framework:
    • Add the ability to test an incoming Stripe webhook. (#44)
    • Prefer @require_POST over an explicit check in the view. (#43)
    • Move away from the deprecated django.conf.urls.patterns method. (#45)
  • Updated django-template-tests, a tool to perform simple static analysis on Django templates prior to production to improve the robustness of dynamic test generation. (e98f9fc)
  • Fixed django-force-logout — a library to allow administrators to log user's out of Django projects — to not require a custom JSON serialiser. (c0e5d64)
  • Corrected the documentation for travis.debian.net — my hosted script to easily test and build Debian packages on the Travis CI continuous integration platform — to reflect the actual default mirror. (4c862f0)
  • Blogged about generating dynamic Python tests using metaclasses.
Debian
  • Presented Reproducible Builds - fulfilling the original promise of free software at FOSSASIA '16.
  • Uploaded libfiu (0.94-4), adding a patch from Logan Rose to fix a FTBFS with ld --as-needed.

My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#44, #45, #46, #47).

LTS

This month I have been paid to work 7 hours on Debian Long Term Support (LTS). Whilst the LTS team will take over support from the Security Team on April 26, 2016, in the meantime I did the following:

  • Archived the squeeze distribution (via the FTPteam).
  • Assisted in preparing updates for python-django.
  • Helping end-users migrate to wheezy now that squeeze LTS has reached end-of-life.
Bugs filed

I also filed 71 FTBFS bugs against ada-reference-manual, arachne-pnr, cdrkit, civicrm, cura-engine, dopewars, eclipse-cdt, fail2ban, fcitx-table-other, fleet, flint, flycheck, git-build-recipe, gitmagic, gnome-audio, golang-ginkgo, golang-github-armon-gomdb, golang-github-aws-aws-sdk-go, golang-github-gorilla-handlers, golang-github-hashicorp-raft-mdb, golang-github-jinzhu-gorm, golang-github-miekg-pkcs11, golang-goyaml, gstreamer0.10, gstreamer1.0, haskell-http2, hidrd, hplip, httpfs2, jedit, kodi-pvr-dvbviewer, kodi-pvr-mythtv, kodi-pvr-vuplus, kodi-pvr-wmc, libraw1394, libtest-file-contents-perl, metview, mpi4py, ndoutils, opensp, phpunit, python-glanceclient, python-oslo.messaging, python-phabricator, rrdtool, ruby-actionpack-xml-parser, ruby-acts-as-list, ruby-acts-as-list, ruby-acts-as-list, ruby-acts-as-list, ruby-amqp, ruby-buff-config, ruby-buff-extensions, ruby-clockwork, ruby-compass, ruby-em-synchrony, ruby-numru-misc, ruby-rabl, ruby-rails-deprecated-sanitizer, ruby-riddle, ruby-safe-yaml, ruby-solve, ruby-toml, ruby-twitter, ruby-typhoeus, ruby-varia-model, shared-mime-info, siproxd, stylish, sup-mail & uima-as.

FTP Team

As a Debian FTP assistant I ACCEPTed 143 packages: acme-tiny, berkshelf-api, circlator, cloud-utils, corsix-th, cronic, diaspora-installer, dub, dumb-init, firehol, firetools, flask-bcrypt, flask-oldsessions, flycheck, ganeti, geany-plugins, git-build-recipe, git-phab, gnome-shell-extension-caffeine, gnome-shell-extension-mediaplayer, golang-github-cheggaaa-pb, golang-github-coreos-ioprogress, golang-github-cyberdelia-go-metrics-graphite, golang-github-cznic-ql, golang-github-elazarl-goproxy, golang-github-hashicorp-hil, golang-github-mitchellh-go-wordwrap, golang-github-mvdan-xurls, golang-github-paulrosania-go-charset, golang-github-xeipuuv-gojsonreference, golang-github-xeipuuv-gojsonschema, grilo-plugins, gtk3-nocsd, herisvm, identity4c, lemonldap-ng, libisal, libmath-gsl-perl, libmemcached-libmemcached-perl, libplack-middleware-logany-perl, libplack-middleware-logwarn-perl, libpng1.6, libqmi, librdf-generator-http-perl, libtime-moment-perl, libvirt-php, libxml-compile-soap-perl, libxml-compile-wsdl11-perl, linux, linux-tools, mdk-doc, mesa, mpdecimal, msi-keyboard, nauty, node-addressparser, node-ansi-regex, node-argparse, node-array-find-index, node-base62, node-co, node-component-consoler, node-crypto-cacerts, node-decamelize, node-delve, node-for-in, node-function-bind, node-generator-supported, node-invert-kv, node-json-localizer, node-normalize-git-url, node-nth-check, node-obj-util, node-read-file, node-require-dir, node-require-main-filename, node-seq, node-starttls, node-through, node-uid-number, node-uri-path, node-url-join, node-xmlhttprequest-ssl, ocrmypdf, octave-netcdf, open-infrastructure-container-tools, osmose-emulator, pdal, pep8, pg-backup-ctl, php-guzzle, printrun, pydocstyle, pysynphot, python-antlr3, python-biom-format, python-brainstorm, python-django-adminsortable, python-feather-format, python-gevent, python-lxc, python-mongoengine, python-nameparser, python-pdal, python-pefile, python-phabricator, python-pika-pool, python-pynlpl, python-qtawesome, python-requests-unixsocket, python-saharaclient, python-stringtemplate3, r-cran-adegraphics, r-cran-assertthat, r-cran-bold, r-cran-curl, r-cran-data.table, r-cran-htmltools, r-cran-httr, r-cran-lazyeval, r-cran-mcmc, r-cran-openssl, r-cran-pbdzmq, r-cran-rncl, r-cran-uuid, rawtran, reel, ruby-certificate-authority, ruby-rspec-pending-for, ruby-ruby-engine, ruby-ruby-version, scribus-ng, specutils, symfony, tandem-mass, tdb, thrift, udfclient, vala, why3, wmaker, xdg-app & xiccd.

Categories: LUG Community Blogs

Mick Morgan: jibber jabber

Wed, 30/03/2016 - 19:52

For some time mow I have been increasingly fed up with the poor service offered by SMS on my mobile phone. I’m not a hugely prolific sender of text messages, and those I do send are primarily aimed at my wife and kids, but when I do send them, I expect them to get there, on time and reliably. I also expect to be able to send and receive images (by MMS) because that is what I have paid for in my contract. Oh, and I would also like to do this securely, and in privacy.

Guess what? I can’t.

My wife and kids want me to use Facebook’s messenger because that is what they use and they are happy with it. I have signally failed to convince them /not/ to use that application, largely because they already have it installed and use it to send messages to everyone else they know. But then I have also failed to get my kids to understand my concerns about wider Facebook usage. (“There Dad goes again, off on one about Facebook.”).

So, what to do? Answer, set up my own XMPP server and use that. XMPP is an open standard, there are plenty of good FOSS XMPP servers about (jabberd, ejabberd, prosody etc.) and there are also plenty of reasonable looking XMPP clients for both android and linux (the OS’s I care about). And better yet, it is perfectly possible (and relatively easy) to set up the server to accept only TLS encrypted connections so that conversations take place in private. Many clients also now support OTR or OMEMO encryption so the conversations can be made completely secure through end-to-end encryption. Yes, I am aware that OTR is a bit of a kludge, but it is infinitely preferable to clear text SMS over the mobile network. And I like my privacy. Better yet, unlike GPG which I use for email, OTR also provides forward secrecy, so even if my keys should be compromised, my conversations won’t be.

And yes I also know that Facebook messenger itself offers “security and pivacy”, but it also used to be capable of interacting with the open XMPP standard before Zuckerberg made it proprietary a few years ago. And I just don’t trust Facebook. For anything.

So for some time now I have been using my own XMPP server alongside my mail server. It works just fine, and I have even convinced my wife and kids to use it when they converse with me.

I may now move away from using GPG to using OTR in preference. Anyone wishing to contact me can now do so at my XMPP address and may also encrypt messages to me using my OTR key. My fingerprints are published here.

Categories: LUG Community Blogs

Chris Lamb: Generating dynamic Python tests using metaclasses

Sun, 27/03/2016 - 16:09

One common testing anti-pattern is a single testcase that loops over multiple yet independent inputs:

from django.test import TestCase class MyTests(TestCase): def test_all(self): for x in ( 'foo', 'bar', 'baz', # etc. ): self.assertEqual(test(x), None)

Whilst this code style typically occurs in an honourable attempt to avoid a DRY violation, these tests:

  1. Can only report the first failure
  2. Prevent individual inputs from being tested independently
  3. Are typically slower than their neighbours, or a performance hotspot generally
  4. Do not allow for parallel computation, a feature recently added to Django 1.9

(Note that whilst foo, bar, etc. are defined statically above for simplicity, the values could be determined dynamically by, for example, iterating over the filesystem.)

If you have such tests, consider splitting them out using a metaclass like so:

class MyTestsMeta(type): def __new__(cls, name, bases, attrs): for x in ( 'foo', 'bar', 'baz', ): attrs['test_%s' % x] = cls.gen(x) return super(MyTestsMeta, cls).__new__(cls, name, bases, attrs) @classmethod def gen(cls, x): # Return a testcase that tests ``x``. def fn(self): self.assertEqual(test(x), None) return fn class MyTests(TestCase): __metaclass__ = MyTestsMeta

This has the effect of replacing the single testcase with individual test_foo, test_bar & test_baz testcases. Each test can then be run separately:

$ ./manage.py test myproject.myapp.tests.MyTests.test_baz Creating test database for alias 'default'... . ---------------------------------------------------------------------- Ran 1 test in 0.039s OK Destroying test database for alias 'default'...

... or we can test them all in parallel:

$ ./manage.py test myproject.myapp.tests.MyTests --parallel Creating test database for alias 'default'... . ---------------------------------------------------------------------- Ran 3 tests in 0.065s OK Destroying test database for alias 'default'...

You must ensure that the tests have unique names to avoid cases masking each other. In the above example we could simply use the input string itself, but if you have no obvious candidate you could try using Python's enumerate method to generate a unique, if somewhat opaque, suffix:

for idx, x in enumerate(( 'foo', 'bar', 'baz', )): attrs['test_%d' % idx] = cls.gen(x)

One alternative approach to a metaclass is to generate the test methods and use setattr, to bind them to the TestCase class. However, using a metaclass:

  1. Is cleaner and/or more "Pythonic"
  2. Avoids a number of subtle pitfalls with parameter binding
  3. Prevents your TestCase class from being polluted with loop variables, etc.
  4. Can be composed or abstracted into reusable testing components


Note that you can still use setUp and all the other unittest.TestCase and django.test.TestCase methods as before:

from .utils import MyBaseTestCase # custom superclass class MyTestsMeta(type): # <snip> @classmethod def gen(cls, x): def fn(self): self.assertRedirects(...) return fn class MyTests(MyBaseTestCase): __metaclass__ = MyTestsMeta def setUp(self): super(MyTests, self).setUp() # Code here is run before every test def test_other(self): # Test some other functionality here

UPDATE: Stu Cox asked: Do you even need the metaclass, or could you do this with __new__ straight on the TestCase? Curiously, unittest does not initialise classes in the typical way if you do not explicitly define at least one test_ method.

Categories: LUG Community Blogs

Jonathan McDowell: Dr Stoll: Or how I learned to stop worrying and love the GPL

Sat, 26/03/2016 - 17:28

[I wrote this as part of BelFOSS but I think it’s worth posting here.]

My Free Software journey starts with The Cuckoo’s Egg. Back in the early 90s a family friend suggested I might enjoy reading it. He was right; I was fascinated by the world of interconnected machines it introduced me to. That helped start my involvement in FidoNet, but it also got me interested in Unix. So when I saw a Linux book at the Queen’s University bookshop (sadly no longer with us) with a Slackware CD in the back I had to have it.

The motivation at this point was to have a low cost version of Unix I could run on the PC hardware I already owned. I had no knowledge of the GNU Project before this point, and as I wasn’t a C programmer I had no interest in looking at the source code. I spent some time futzing around with it and that partition (I was dual booting with DOS 6.22) fell into disuse. It wasn’t until I’d learnt some C and turned up to university, which provided me with an internet connection and others who were either already using Linux or interested in doing so, that I started running a Linux box full time.

Once I was doing that I became a lot more interested in the Open Source side of the equation. Rather than running a closed operating system that even the API for wasn’t properly specified (or I wouldn’t have needed my copy of Undocumented DOS) I had the complete source to both the underlying OS and all the utilities that it was using. For someone doing a computer science degree this was invaluable. Minix may have been the OS discussed in the OS Design module I studied, but Linux was a much more feature complete option that I was running on my desktop and could also peer under the hood of.

In my professional career I’ve always welcomed the opportunities to work with Open Source. A long time ago I experienced a particularly annoying issue when writing a device driver under QNX. The documentation didn’t seem to match the observed behaviour of the subsystem I was interfacing with. However due to licensing issues only a small number of people in the organisation were able to actually look at the QNX source. So I ended up wasting a much more senior engineer’s time with queries like “I think it’s actually doing x, y and z instead of a, b and c; can you confirm?”. Instances where I can look directly at the source code myself make me much more productive.

Commercial development also started to make me more understanding of the Free Software nature of the code I was running. It wasn’t just the ability to look at the code which was useful, but also the fact there was no need to reinvent the wheel. Need a base OS to build an appliance on? Debian ensures that the main component is Free for all usage. No need to worry about rolling your own compilers, base libraries etc. From a commercial perspective that allows you to concentrate on the actual product. And when you hit problems, the source is available and you can potentially fix it yourself or at least more easily find out if there’s been a fix for that issue released (being able to see code development in version control systems rather than getting a new upstream release with a whole heap on unrelated fixes in it really helps with that).

I had thus progressed from using FLOSS because it was free-as-in-beer, to appreciating the benefits of Open Source in my own learning and employment experiences, to a deeper understanding of the free-as-in-speech benefits that could be gained. However at this point I was still thinking very much from a developer mindset. Even my thoughts about how users can benefit from Free Software were in the context of businesses being able to easily switch suppliers or continue to maintain legacy software because they had the source to their systems available.

One of the major factors that has helped me to see beyond this is the expansion of the Internet of Things (IoT). With desktop or server software there is by and large a choice about what to use. This is not the case with appliances. While manufacturers will often produce a few revisions of software for their devices, usually eventually there is a newer and shiny model and the old one is abandoned. This is problematic for many reasons. For example, historically TVs have been long lived devices (I had one I bought second hand that happily lasted me 7+ years). However the “smart” capabilities of the TV I purchased in 2012 are already of limited usefulness, and LG have moved on to their current models. I have no intention of replacing the device any time soon, so have had to accept it is largely acting as a dumb display. More serious is the lack of security updates. For a TV that doesn’t require a network connection to function this is not as important, but the IoT is a trickier proposition. For example Matthew Garrett had an awful experience with some ‘intelligent’ light bulbs, which effectively circumvented any home network security you might have set up. The manufacturer’s defence? No longer manufactured or supported.

It’s cases like these that have slowly led me to a more complete understanding of the freedom that Free Software truly offers to users. It’s not just about cost free/low cost software. It’s not just about being able to learn from looking at the source to the programs you are running. It’s not even about the freedom to be able to modify the programs that we use. It’s about giving users true Freedom to use and modify their devices as they see fit. From this viewpoint it is much easier to understand the protections against Tivoization that were introduced with GPLv3, and better appreciate the argument sometimes made that the GPL offers more freedom than BSD style licenses.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in February 2016

Mon, 29/02/2016 - 23:53

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

  • Various improvements to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
    • Added explicit support to send messages asynchronously via the Celery distributed task queue. (#29)
    • Worked with Patrick Clope to add an escapeslack template filter to ensure the correct characters for Slack's API are escaped — using Django's built-in safe is too invasive. (#36)
    • Corrected an issue where custom endpoints and channel names were incompatible. (#27)
    • Overhaul of the field/option/block parsing. (49ff3c4)
    • Moved away from using a django.template.Context instance, preventing a deprecation warning. (#31)
  • Added a create-folders subcommand to my tickle-me-email Getting Things Done (GTD) email toolbox to create numbered folders for the rotate "tickler" functionality. (#3)
  • Wrote and released a Django template tag to collapse multiple whitespace/newline characters in order to correctly format, for example, plaintext emails that make extensive use of for-loops. These would normally require careful and fragile placement of the {% for .. %} and {% endfor %} tags. (Repo)
  • Pushed a number of updates to my Strava Enhancement Suite, a Chrome extension that improves and fixes annoyances in the web interface of the Strava cycling and running tracker:
    • Fix unit conversion where the element was wrapped in an another HTML element. (#48)
    • Hide premium indicator from top-level navigation. (#45)
    • Correct matching of the "shop" link in top-level navigation and footer module. (#47 & #46)
    • Remove "premium" pages on Club pages. (#44)
  • Added a security-oriented warning to Ansible's documentation regarding the behaviour if an UFW firewall application profile is added and subsequently removed. (#1740)
  • Added support to my django-staticfiles-dotd "staticfiles" library — which concatenates Javascript and CSS files from .d-style directories — to support an arbitrary file rendering method. This allows the use of media pre-processors such as SASS within such directories. (#1)
  • Corrected my Chrome extension for the FastMail web interface to correctly hook into the internal "send" mechanism. (#2)
  • Moved my stravabot IRC bot to use dh-virtualenv and systemd, improving security and reliability. (1c48709)
  • Updated django-pedantic-http-methods — a tool to raise an exception during development when attempting to perform side effects in GET and HEAD HTTP methods — to support the latest version of Django. (#1)
Debian
  • Updated travis.debian.net — a hosted script to easily test and build Debian packages on the Travis CI continuous integration platform — to support:
    • Automatic bumping of the version number in debian/changelog based on TRAVIS_BUILD_NUMBER. (#14)
    • Security repositories. Thanks to Stefan Jenkner for the initial pull request. These are additionally now enabled by default. (#15)
    • The backports repositories. (#13)
  • Applied #812830 and #812830 from James Clark to the Debian Archive Kit to improve the interface of various webpages it generates.
  • Updated the SSL certificate for try.diffoscope.org, a hosted version of the diffoscope in-depth and content-aware diff utility. Thanks to Bytemark for sponsoring the hardware.
  • Worked on my slides for Reproducible Builds - fulfilling the original promise of free software, to be presented at FOSSASIA '16.

My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#40, #41, #42, #43)

LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duty for the week of 22nd—28th, triaging CVEs, etc.
  • Proofread announcements, etc. for the upcoming migration to wheezy-lts.
  • Issued DLA 417-1 for xdelta3 to fix a buffer overflow that allowed arbitrary code execution from input files.
  • Issued DLA 420-1 for libmatroska, correcting a heap information leak.
  • Issued DLA 428-1 for websvn fixing a cross-site scripting vulnerability.
  • Issued DLA 429-1 for pixman fixing a buffer overflow issue.
  • Issued DLA 430-1 & DLA 431-1 for libfcgi and libfcgi-perl respectfully, fixing a remote denial-of-service (DoS) vulnerability.
Uploads
  • redis (2:3.0.7-2) — Correcting my SOURCE_DATE_EPOCH reproducibility patch as the conditional was accidentally inverted. Thanks to Reiner Herrmann (deki).
  • disque (1.0~rc1-5) — Making the parallel SOURCE_DATE_EPOCH patch change and additionally tidying the packaging after introducing procps as a build-dependency.
Patches/bugs contributed RC bugs

I also filed 137 FTBFS bugs against aac-tactics, angular.js, astyle, bcftools, blacs-mpi, bogofilter, boxes, caldav-tester, ccdproc, ckeditor, coq-float, cqrlog, dasher, django-recurrence, dspdfviewer, eclipse-egit, ess, etcd, felix-latin, fio, flexml, funny-manpages, gap-atlasrep, garmin-plugin, gitlab, gnome-mines, graphicsmagick, haskell-nettle, healpy, hg-git, hunspell, hwloc, ijs, ipset, janest-core-extended, jpathwatch, kcompletion, kcompletion, keyrings.alt, kodi-pvr-hts, kodi-pvr-vdr-vnsi, libcommons-compress-java, libgnome2-wnck-perl, libkate, liblrdf, libm4ri, libnet-server-mail-perl, libsis-jhdf5-java, libspectre, libteam, libwnck, libwnckmm, libxkbcommon, lombok, lombok-patcher, mako, maven-dependency-analyzer, mopidy-mpris, mricron, multcomp, netty-3.9, numexpr, ocaml-textutils, openimageio, openttd-openmsx, osmcoastline, osmium-tool, php-guzzle, php-net-smartirc, plexus-component-metadata, polari, profitbricks-client, pyentropy, pynn, pyorbital, pypuppetdb, python-aioeventlet, python-certifi, python-hglib, python-kdcproxy, python-matplotlib-venn, python-mne, python-mpop, python-multipletau, python-pbh5tools, python-positional, python-pydot-ng, python-pysam, python-snuggs, python-tasklib, r-cran-arm, r-cran-httpuv, r-cran-tm, rjava, ros-geometry-experimental, ros-image-common, ros-pluginlib, ros-ros-comm, rows, rr, ruby-albino, ruby-awesome-print, ruby-default-value-for, ruby-fast-gettext, ruby-github-linguist, ruby-gruff, ruby-hipchat, ruby-omniauth-crowd, ruby-packetfu, ruby-termios, ruby-thinking-sphinx, ruby-tinder, ruby-versionomy, ruby-zentest, sbsigntool, scikit-learn, scolasync, sdl-image1.2, signon-ui, sisu-guice, sofa-framework, spykeutils, ssreflect, sunpy, tomcat-maven-plugin, topmenu-gtk, trocla, trocla, tzdata, verbiste, wcsaxes, whitedune, wikidiff2, wmaker, xmlbeans, xserver-xorg-input-aiptek & zeroc-icee-java.

FTP Team

As a Debian FTP assistant I ACCEPTed 107 packages: androguard, android-platform-dalvik, android-platform-development, android-platform-frameworks-base, android-platform-frameworks-native, android-platform-libnativehelper, android-platform-system-core, android-platform-system-extras, android-platform-tools-base, android-sdk-meta, apktool, armci-mpi, assertj-core, bart, bind9, caja, caldav-tester, clamav, class.js, diamond, diffoscope, django-webpack-loader, djangocms-admin-style, dnsvi, esptool, fuel-astute, gcc-6-cross, gcc-6-cross-ports, gdal, giella-core, gnupg, golang-github-go-ini-ini, golang-github-tarm-serial, gplaycli, gradle-jflex-plugin, haskell-mountpoints, haskell-simple, hurd, iceweasel, insubstantial, intellij-annotations, jetty9, juce, keyrings.alt, leptonlib, libclamunrar, libdate-pregnancy-perl, libgpg-error, libhtml5parser-java, libica, libvoikko, linux, llvm-toolchain-3.8, lombok-patcher, mate-dock-applet, mate-polkit, mono-reference-assemblies, mxt-app, node-abab, node-array-equal, node-array-flatten, node-array-unique, node-bufferjs, node-cors, node-deep-extend, node-original, node-setimmediate, node-simplesmtp, node-uglify-save-license, node-unpipe, oar, openjdk-8, openjdk-9, pg8000, phantomjs, php-defaults, php-random-compat, php-symfony-polyfill, pnetcdf, postgresql-debversion, pulseaudio-dlna, pyconfigure, pyomo, pysatellites, python-fuelclient, python-m3u8, python-pbh5tools, python-qtpy, python-shellescape, python-tunigo, pyutilib, qhull, r-cran-rjsonio, r-cran-tm, reapr, ruby-fog-dynect, scummvm-tools, symfony, talloc, tesseract, twextpy, unattended-upgrades, uwsgi, vim-command-t, win-iconv, xkcdpass & xserver-xorg-video-ast.

I additionally REJECTed 4 packages.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Kicad hacking - Intra-sheet links and ERC

Mon, 29/02/2016 - 12:19

This is a bit of an odd posting since it's about something I've done but is also here to help me explain why I did it and thus perhaps encourage some discussion around the topic within the Kicad community...

Recently (as you will know if you follow this blog anywhere it is syndicated) I have started playing with Kicad for the development of some hardware projects I've had a desire for. In addition, some of you may be aware that I used to work for a hardware/software consultancy called Simtec, and there I got to play for a while with an EDA tool called Mentor Designview. Mentor was an expensive, slow, clunky, old-school EDA tool, but I grew to understand and like the workflow.

I spent time looking at gEDA and Eagle when I wanted to get back into hardware hacking for my own ends; but neither did I really click with. On the other hand, a mere 10 minutes with Kicad and I knew I had found the tool I wanted to work with long-term.

I designed the beer'o'meter project (a flow meter for the pub we are somehow intimately involved with) and then started on my first personal surface-mount project -- SamDAC which is a DAC designed to work with our HiFi in our study at home.

As I worked on the SamDAC project, I realised that I was missing a very particular thing from Mentor, something which I had low-level been annoyed by while looking at other EDA tools -- Kicad lacks a mechanism to mark a wire as being linked to somewhere else on the same sheet. Almost all of the EDA tools I've looked at seem to lack this nicety, and honestly I miss it greatly, so I figured it was time to see if I could successfully hack on Kicad.

Kicad is written in C++, and it has been mumble mumble years since I last did any C++, either for personal hacking or professionally, so it took a little while for that part of my brain to kick back in enough for me to grok the codebase. Kicad is not a small project, taking around ten minutes to build on my not-inconsiderable computer. And while it beavered away building, I spent time looking around the source code, particularly the schematic editor eeschema.

To skip ahead a bit, after a couple of days of hacking around, I had a proof-of-concept for the intra-sheet links which I had been missing from my days with Mentor, and some ERC (electrical rules checking) to go alongside that to help produce schematics without unwanted "sharp corners".

In total, I added:

  • an "intra-sheet link" schematic element, including:
    • the schematic element itself
    • loading and saving of the element (which required a bump in schematic file format versioning)
    • UI for adding intra-sheet links to your schematic
    • netlist build support to ensure that the links found their way into the internal netlist content.
  • an ERC tickybox "Strict wiring checks" and behind that I hid:
    • checks for wires with dangling ends (dangles are bad)
    • junctions which have no point in being present (pointless junctions are confusing to readers)
    • checks that intra-sheet links exist in groups of two or more on any given net, since individual links are pointless.

I forked the Kicad mirror on Github and pushed my own branch with this work to my Kicad fork.

All of this is meant to allow schematic capture engineers to more clearly state their intentions regarding what they are drawing. The intra-sheet link could be thought of like a no-connect element, except instead of saying "this explicitly goes nowhere" we're saying "this explicitly goes somewhere else on this sheet, you can go look for it".

Obviously, people who dislike (or simply don't want to use) such intra-sheet link elements can just disable that ERC tickybox and not be bothered by them in the least (well except for the toolbar button and menu item I suppose).

Whether this work gets accepted into Kicad, or festers and dies on the vine, it was good fun developing it and I'd like to illustrate how it could help you, and why I wrote it in the first place:

A contrived story

Note, while this story is meant to be taken seriously, it is somewhat contrived, the examples are likely electrical madness, but please just think about the purpose of the checks etc.

To help to illustrate the feature and why it exists, I'd like to tell you a somewhat contrived story about Fred. Fred is a schematic capture engineer and his main job is to review schematics generated by his colleagues. Fred and his colleagues work with Kicad (hurrah) but of late they've been having a few issues with being able to cleanly review schematics.

Fred's colleagues are not the neatest of engineers. In particular they tend to be quite lazy when it comes to running busses, which are not (for example) address and data busses, around their designs and they tend to simply have wires which end in mid-space and pick up somewhere else on the sheet. All this is perfectly reasonable of course, and Kicad handles it with aplomb. Sadly it seems quite error prone for Fred's workplace.

As an example, Fred's colleague Ben has been designing the power supply for a particular board. As with most power supplies, plenty of capacitors are needed to stabilise the regulators and smooth the output. In the example below, the intent is that all of the capacitors are on the FOO net.

Sadly there's a missing junction and/or slightly misplaced label in the upper section which means that C2 and C3 simply don't join to the FOO net. This could easily be missed, but the ERC can't spot it at all since there's more than one thing on each net, so the pins of the capacitors are connected to something.

Fred is very sad that this kind of problem can sometimes escape notice by the schematic designer Ben, Fred himself, and the layout engineer, resulting in boards which simply do not work. Fred takes it upon himself to request that the strict wiring checks ERC is made mandatory for all designs, and that the design engineers be required to use intra-sheet link symbols when they have signals which wander off to other parts of the sheet like FOO does in the example. Without any further schematic changes, strict wiring checks enabled gives the following points of ERC concern for Ben to think about:

As you can see, the ERC is pointing at the wire ends and the warnings are simply that the wires are dangling and that this is not acceptable. This warning is very like the pin-not-connected warnings which can be silenced with an explicit no-connect schematic element. Ben, being a well behaved and gentle soul, obeys the design edicts from Fred and seeks out the intra-sheet link symbols, clearing off the ERC markers and then adding intra-sheet links to his design:

This silences the dangling end ERC check, which is good, however it results in another ERC warning:

This time, the warning for Ben to deal with is that the intra-sheet links are pointless. Each exists without a companion to link to because of the net name hiccough in the top section. It takes Ben a moment to realise that the mistake which has been made is that a junction is missing in the top section. He adds the junction and bingo the ERC is clean once more:

Now, this might not seem like much gain for so much effort, but Ben can now be more confident that the FOO net is properly linked across his design and Fred can know, when he looks at the top part of the design, that Ben intended for the FOO net to go somewhere else on the sheet and he can look for it.

Why do this at all?

Okay, dropping out of our story now, let's discuss why these ERC checks are worthwhile and why the intra-sheet link schematic element is needed.

Note: This bit is here to remind me of why I did the work, and to hopefully explain a little more about why I think it's worth adding to Kicad...

Designers are (one assumes) human beings. As humans we (and I count myself here too) are prone to mistakes. Sadly mistakes are often subtle and could easily be thought of as deliberate if the right thought processes are not followed carefully when reviewing. Anyone who has ever done code review, proofread a document, or performed any such activity, will be quite familiar with the problems which can be introduced by a syntactically and semantically valid construct which simply turns out to be wrong in the greater context.

When drawing designs, I often end up with bits of wire sticking out of schematic sections which are not yet complete. Sadly if I sleep between design sessions, I often lose track of whether such a dangling wire is meant to be attached to more stuff, or is simply left because the net is picked up elsewhere on the sheet. With intra-sheet link elements available, if I had intended the latter, I'd have just dropped such an element on the end of the wire before I stopped for the day.

Also, when drawing designs, I sometimes forget to label a wire, especially if it has just passed through a filter or current-limiting resistor or similar. As such, even with intra-sheet link elements to show me when I mean for a net to go bimbling off across the sheet, I can sometimes end up with unnamed nets whose capacitors end up not used for anything useful. This is where the ERC comes in.

By having the ERC complain if a wire dangles -- the design engineer won't forget to add links (or check more explicitly if the wire is meant to be attached to something else). By having junctions which don't actually link anything warned about, the engineer can't just slap a junction blob down on the end of a wire to silence that warning, since that doesn't mean anything to a reviewer later down the line. By having the ERC warn if a net has exactly one intra-sheet link attached to it, missing net names and errors such as that shown in my contrived example above can be spotted and corrected.

Ultimately this entire piece of work is about ensuring that the intent of the design engineer is captured clearly in the schematic. If the design engineer meant to leave that wire dangling because it's joining to another bit of wire elsewhere on the sheet, they can put the intra-sheet links in to show this. The associated ERC checks are there purely to ensure that the validation of this intent is not bypassed accidentally, or deliberately, in order to make the use of this more worthwhile and to increase the usefulness of the ERC on designs where signals jump around on sheets where wiring them up directly would just create a mess.

Categories: LUG Community Blogs

Steve Engledow (stilvoid): Digital Subscriber

Thu, 25/02/2016 - 22:37

Maybe it's just me, but I reckon DSLs are the next (ok ok, they've been around for ages) big (ok, hipster) thing. I know I'm by no means the first to say so it's just that I'm increasingly bemused at seeing things squeezed into data structures they've outgrown.

In general, as everyone's finally warming to the idea that you can use code to describe not just your application but also how it's deployed, we're reaching a state where that code needs to be newbie-friendly - by which I mean that it ought to be easily understandable by humans. If it isn't, it's prone to mistakes.

A few months ago, I experimented with creating a DSL for writing web pages and I was fairly happy with the result (though there's lots more work to be done). I'm thinking of applying the same ideas to CloudFormation.

resources: db: type: rds engine: mysql size: c3.xlarge app: type: ec2 ami: my-app-image size: t2.micro scale: min: 1 max: 10 expose: 80 security: db: app app: 0.0.0.0:80

Obviously I've put little to no thought into the above but it shouldn't be too hard to come up with something useful.

Maybe some day soon ;)

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): The Beer'o'Meter project

Sun, 31/01/2016 - 18:51

As some of you may know, I have been working on a small hardware project called the Beer'o'Meter whose purpose is to allow us to extend Ye Olde Vic's beer board to indicate the approximate fullness of each cask. For some time now, we've been operating an electronic beer board at the Vic which you may see tweeted out from time to time. The pumpotron has become very popular with the visitors to the pub, especially that it can be viewed online in a basic textual form.

Of course, as many of you who visit pubs know only too well. That a beer is "on" is no indication of whether or not you need to get there sharpish to have a pint, or if you can take your time and have a curry first. As a result, some of us have noticed a particular beer on, come to the pub after dinner, and then been very sad that if only we'd come 30 minutes previously, we'd have had a chance at the very beer we were excited about.

Combine this kind of sadness with a two week break at Christmas, and I started to develop a Beer'o'Meter to extend the pumpotron with an indication of how much of a given beer had already been served. Recently my boards came back from Elecrow along with various bits and bobs, and I have spent some time today building one up for test purposes.

As always, it's important to start with some prep work to collect all the necessary components. I like to use cake cases as you may have noticed on the posting yesterday about the oscilloscope I built.

Naturally, after prep comes the various stages of assembly. You start with the lowest-height components, so here's the board after I fitted the ceramic capacitors:

And here's after I fitted the lying-down electrolytic decoupling capacitor for the 3.3 volt line:

Next I should have fitted the six transitors from the middle cake case, but I discovered that I'd used the wrong pinout for them. Even after weeks of verification by myself and others, I'd made a mistake. My good friend Vincent Sanders recently posted about how creativity is allowing yourself to make mistakes and here I had made a doozy I hadn't spotted until I tried to assemble the board. Fortunately TO-92 transistors have nice long legs and I have a pair of tweezers and some electrical tape. As such I soon had six transistors doing the river dance:

With that done, I noticed that the transistors now stood taller than the pins (previously I had been intending to fit the transistors before the pins) so I had to shuffle things around and fit all my 0.1" pins and sockets next:

Then I could fit my dancing transistors:

We're almost finished now, just one more capacitor to provide some input decoupling on the 9v power supply:

Of course, it wouldn't be complete without the ESP8266Huzzah I acquired from AdaFruit though I have to say that I'm unlikely to use these again, but rather I might design in the surface-mount version of the module instead.

And since this is the very first Beer'o'Meter to be made, I had to go and put a 1 on the serial-number space on the back of the board. I then tried to sign my name in the box, made a hash of it, so scribbled in the gap

Finally I got to fit all six of my flow meters ready for some testing. I may post again about testing the unit, but for now, here's a big spider of a flow meter for beer:

This has been quite a learning experience for me, and I hope in the future to be able to share more of my hardware projects, perhaps from an earlier stage.

I have plans for a DAC board, and perhaps some other things.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Building an Oscilloscope

Sun, 31/01/2016 - 18:51

I recently ordered some PCBs from Elecrow for the Vic's beer-measurement system I've been designing with Rob. While on the site, I noticed that they have a single-channel digital oscilloscope kit based on an STM32. This is a JYE Tech DSO138 which arrives as a PCB whose surface-mount stuff has been fitted, along with a whole bunch of pin-through components for you to solder up the scope yourself. There's a non-trivial number of kinds of components, so first you should prep by splitting them all up and double-checking them all.

Once you've done that, the instructions start you off fitting a whole bunch of resistors...

Then some diodes, RF chokes, and the 8MHz crystal for the STM32.

The single most-difficult bit for me to solder was the USB socket. Fine pitch leads, coupled with high-thermal-density socket.

There is a veritable mountain of ceramic capacitors to fit...

And then buttons, inductors, trimming capacitors and much more...

THe switches were the next hardest things to solder, after the USB socket...

Finally you have to solder a test loop and close some jumpers before you power-test the board.

The last bit of soldering is to solder pins to the LCD panel board...

Before you finally have a working oscilloscope

I followed the included instructions to trim the scope using the test point and the trimming capacitors, before having a break to write this up for you all. I'd say that it was a fun day because I enjoyed getting a lot of soldering practice (before I have to solder up the beer'o'meter for the pub) and at the end of it I got a working oscilloscope. For 40 USD, I'd recommend this to anyone who fancies a go.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in January 2016

Sun, 31/01/2016 - 09:16

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

  • Changed Django's project/app templates to use a py-tpl suffix to workaround the 1.9 release series shipping invalid .py files that are used as templates. (#5735)
  • Pushed a number of updates to my Strava Enhancement Suite Chrome extension:
    • Added the ability to sort starred segments first. (#42)
    • Added an option to display hidden segments by default. (#34)
    • Also hide "Yearly Goals" as part of hiding upcoming data. (#41)
    • Removed more "premium" badges. (#43)
    • Fixed an issue where removing feed entries didn't correctly cleanup subsequently-empty date headers. (commit)
  • Released a work-in-progress django.contrib.staticfiles library to recursively transparently concatenate Javascript and CSS files using "Debian-style" .d directories. I had previously been doing this manually via the various release processes — and bespoke views during development — as the idea pre-dated the existence of the staticfiles framework. (Repo)
  • Updated travis.debian.net, a hosted script to easily test and build Debian packages on Travis CI, to support the wheezy distribution and to improve error-handling in general. (Repo)
  • Ensured that try.diffoscope.org, a hosted version of the diffoscope in-depth and content aware diff utility, periodically cleans up containers.
  • Moved my personal music library to use dh-virtualenv and to deploy to its own server via Ansible. I also updated the codebase to the latest version of Django at the same time, taking advantage of a large number of new features and recommendations.
  • Fixed a strange issue with empty IMAP messages in my tickle-me-email GTD email toolbox.
  • It was also a month of significant bug reports and feature requests being sent to my private email.
Debian
  • Had a talk proposal accepted (Reproducible Builds - fulfilling the original promise of free software) at FOSSASIA 16.

My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#35, #36, #37, #38, #39)

LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:

  • Sevend days of "frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 386-1 for cacti to patch an SQL injection vulnerability.
  • Issued DLA 388-1 for dwarfutils fixing a NULL deference issue.
  • Issued DLA 391-1 for prosody correcting the use of a weak pseudo-random number generator.
  • Issued DLA 404-1 for nginx to prevent against an invalid pointer deference.
Uploads
  • redis (2:3.0.7-1) — New upstream stable release, also ensure that test processes are cleaned up and replacing an existing reproducibility patch with a SOURCE_DATE_EPOCH solution.
  • python-django (1.9.1-1) — New upstream release.
  • disque (1.0~rc1-4) — Make the build reproducible via SOURCE_DATE_EPOCH, ensure that test processes are cleaned up and that the nocheck flag is correctly honoured.
  • gunicorn (19.4.5-1) — New upstream release.
  • redis (2:3.2~rc3-1) — New upstream RC release (to experimental).
Bugs filed Patches contributed RC bugs

I also filed 100 FTBFS bugs against apache-log4j2, awscli, binutils, brian, ccbuild, coala, commons-beanutils, commons-vfs, composer, cyrus-sasl2, debiandoc-sgml-doc-pt-br, dfvfs, dillo, django-compat, dulwich, git-annex, grpc, hdf-eos5, hovercraft, ideviceinstaller, ircp-tray, isomd5sum, javamail, jhdf, jsonpickle, kivy, klog, libcloud, libcommons-jexl2-java, libdata-objectdriver-perl, libdbd-sqlite3-perl, libpam-krb5, libproc-waitstat-perl, libslf4j-java, libvmime, linuxdcpp, lsh-utils, mailutils, mdp, menulibre, mercurial, mimeo, molds, mugshot, nose, obex-data-server, obexfs, obexftp, orafce, p4vasp, pa-test, pgespresso, pgpool2, pgsql-asn1oid, php-doctrine-cache-bundle, php-net-ldap2, plv8, pngtools, postgresql-mysql-fdw, pyfftw, pylint-common, pylint-django, pylint-django, python-ase, python-axiom, python-biopython, python-dcos, python-falcon, python-instagram, python-markdown, python-pysam, python-requests-toolbelt, python-ruffus, pytsk, pyviennacl, ros-class-loader, ros-ros-comm, ros-roscpp-core, roxterm, ruby-celluloid-extras, ruby-celluloid-fsm, ruby-celluloid-supervision, ruby-eye, ruby-net-scp, ruby-net-ssh, ruby-sidekiq, ruby-sidekiq-cron, ruby-sinatra-contrib, seaview, smc, spatial4j-0.4, swift-plugin-s3, tilecache, typecatcher, ucommon, undertaker, urdfdom, ussp-push, xserver-xorg-video-intel & yt.

FTP Team

As a Debian FTP assistant I ACCEPTed 201 packages: abi-tracker, android-platform-build, android-platform-frameworks-native, android-platform-libcore, android-platform-system-core, animate.css, apitrace, argon2, autosize.js, bagel, betamax, bittorrent, bls-standalone, btfs, caja-dropbox, cegui-mk2, complexity, corebird, courier-authlib, cpopen, ctop, dh-haskell, django-python3-ldap, e2fsprogs1.41, emacs-async, epl, fast5, fastkml, flask-restful, flask-silk, gcc-6, gitlab, golang-github-kolo-xmlrpc, golang-github-kr-fs, golang-github-pkg-sftp, golang-github-prometheus-common, google-auth-library-php, h5py, haskell-aeson-compat, haskell-userid, heroes, hugo, ioprocess, iptables, ivy-debian-helper, ivyplusplus, jquery-timer.js, klaus, kpatch, lazarus, libatteanx-store-sparql-perl, libbrowserlauncher-java, libcgi-test-perl, libdata-sah-normalize-perl, libfsntfs, libjs-fuzzaldrin-plus, libjung-free-java, libmongoc, libmygpo-qt, libnet-nessus-rest-perl, liborcus, libperinci-sub-util-propertymodule-perl, libpodofo, librep, libsodium, libx11-xcb-perl, linux, linux-grsec-base, list.js, lombok, lua-mediator, luajit, maven-script-interpreter, midicsv, mimeo, miniasm, mlpack, mom, mosquitto-auth-plugin, moxie.js, msgpuck, nanopolish, neovim, netcdf, network-manager-applet, network-manager-ssh, node-esprima-fb, node-mocks-http, node-schlock, nomacs, ns3, openalpr, openimageio, openmpi, openms, orafce, pbsim, pd-iemutils, pd-nusmuk, pd-puremapping, pd-purest-json, pg-partman, pg-rage-terminator, pgfincore, pgmemcache, pgsql-asn1oid, php-defaults, php-jwt, php-mf2, php-redis, pkg-info-el, plr, pnmixer, postgresql-multicorn, postgresql-mysql-fdw, powa-archivist, previsat, pylint-flask, pyotherside, python-caldav, python-cookies, python-dcos, python-flaky, python-flickrapi, python-frozendict, python-genty, python-git, python-greenlet, python-instagram, python-ironic-inspector-client, python-manilaclient, python-neutronclient, python-openstackclient, python-openstackdocstheme, python-prometheus-client, python-pymzml, python-pysolr, python-reno, python-requests-toolbelt, python-scales, python-socketio-client, qdox2, qgis, r-cran-biasedurn, rebar.js, repmgr, rfcdiff, rhythmbox-plugin-alternative-toolbar, ripe-atlas-cousteau, ripe-atlas-sagan, ripe-atlas-tools, ros-image-common, ruby-acts-as-list, ruby-allocations, ruby-appraiser, ruby-appraiser-reek, ruby-appraiser-rubocop, ruby-babosa, ruby-combustion, ruby-did-you-mean, ruby-fixwhich, ruby-fog-xenserver, ruby-hamster, ruby-jeweler, ruby-mime-types-data, ruby-monkey-lib, ruby-net-telnet, ruby-omniauth-azure-oauth2, ruby-omniauth-cas3, ruby-puppet-forge, ruby-racc, ruby-reek, ruby-rubinius-debugger, ruby-rubysl, ruby-rubysl-test-unit, ruby-sidekiq-cron, ruby-threach, ruby-wavefile, ruby-websocket-driver, ruby-xmlhash, rustc, s-nail, scrm, select2.js, senlin, skytools3, slurm-llnl, sphinx-argparse, sptk, sunpy, swauth, swift, tdiary, three.js, tiny-initramfs, tlsh, ublock-origin, vagrant-cachier, xapian-core, xmltooling, & yp-tools.

I additionally REJECTed 29 packages.

Categories: LUG Community Blogs