Planet ALUG

Syndicate content
Planet ALUG -
Updated: 21 min 15 sec ago

Mick Morgan: multilingual chat

Sat, 14/10/2017 - 16:29

I use email fairlly extensively for my public communication but I use XMPP (with suitable end-to-end encryption) for my private, personal communication. And I use my own XMPP server to facilitate this. But as I have mentioned in previous posts my family and many of my friends insist on using proprietary variants of this open standard (facebook, whatsapp etc. ad nauseam). I was thus amused to note that I am not alone in having difficulty in keeping track of “which of my contacts use which chat systems“.

(My thanks, as ever, to Randall Munroe over at XKCD.)

I must find a client which can handle all of my messaging systems. Better yet, I’d like one which worked, and seamlessly synchronised, across my mobile devices and my linux desktop. Even better again, such a client should offer simple (i.e. easy to use) e-to-e crypto and use an open server platform which I can manage myself.

Proprietary systems suck.

Categories: LUG Community Blogs

Chris Lamb: python-gfshare: Secret sharing in Python

Sat, 07/10/2017 - 11:12

I've just released python-gfshare, a Python library that implements Shamir’s method for secret sharing, a technique to split a "secret" into multiple parts.

An arbitrary number of those parts are then needed to recover the original file but any smaller combination of parts are useless to an attacker.

For instance, you might split a GPG key into a “3-of-5” share, putting one share on each of three computers and two shares on a USB memory stick. You can then use the GPG key on any of those three computers using the memory stick.

If the memory stick is lost you can ultimately recover the key by bringing the three computers back together again.

For example:

$ pip install gfshare >>> import gfshare >>> shares = gfshare.split(3, 5, b"secret") >>> shares {104: b'1\x9cQ\xd8\xd3\xaf', 164: b'\x15\xa4\xcf7R\xd2', 171: b'>\xf5*\xce\xa2\xe2', 173: b'd\xd1\xaaR\xa5\x1d', 183: b'\x0c\xb4Y\x8apC'} >>> gfshare.combine(shares) b"secret"

After removing two "shares" we can still reconstruct the secret as we have 3 out of the 5 originals:

>>> del shares['104'] >>> del shares['171'] >>> gfshare.combine(shares) b"secret"

Under the hood it uses Daniel Silverstone’s libgfshare library. The source code is available on GitHub as is the documentation.

Patches welcome.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): F/LOSS (in)activity, September 2017

Wed, 04/10/2017 - 13:53

In the interests of keeping myself "honest" regarding F/LOSS activity, here's a report, sadly it's not very good.

Unfortunately, September was a poor month for me in terms of motivation and energy for F/LOSS work. I did some amount of Gitano work, merging a patch from Richard Ipsum for help text of the config command. I also submitted another patch to the STM32F103xx Rust repository, though it wasn't a particularly big thing. Otherwise I've been relatively quiet on the Rust/USB stuff and have otherwise kept away from projects.

Sometimes one needs to take a step away from things in order to recuperate and care for oneself rather than the various demands on ones time. This is something I had been feeling I needed for a while, and with a lack of motivation toward the start of the month I gave myself permission to take a short break.

Next weekend is the next Gitano developer day and I hope to pick up my activity again then, so I should have more to report for October.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in September 2017

Sat, 30/09/2017 - 19:31

Here is my monthly update covering what I have been doing in the free software world in September 2017 (previous month):

  • Submitted a pull request to Quadrapassel (the Gnome version of Tetris) to start a new game when the pause button is pressed outside of a game. This means you would no longer have to use the mouse to start a new game. [...]
  • Made a large number of improvements to AptFS — my FUSE-based filesystem that provides a view on unpacked Debian source packages as regular folders — including moving away from manual parsing of package lists [...] and numerous code tidying/refactoring changes.
  • Sent a small patch to django-sitetree, a Django library for menu and breadcrumb navigation elements to not mask test exit codes from the surrounding shell. [...]
  • Updated, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds:
    • Add support for "sloppy" backports. Thanks to Bernd Zeimetz for the idea and ongoing testing. [...]
    • Merged a pull request from James McCoy to pass DEB_BUILD_PROFILES through to the build. [...]
    • Workaround Travis CI's HTTP proxy which does not appear to support SRV records. [...]
    • Run debc from devscripts if the build was successful [...] and output the .buildinfo file if it exists [...].
  • Fixed a few issues in local-debian-mirror, my package to easily maintain and customise a local Debian mirror via the DebConf configuration tool:
    • Fix an issue where file permissions from the remote could result in a local archive that was impossible to access. [...]
    • Clear out empty directories on the local repository. [...]
  • Updated django-staticfiles-dotd, my Django staticfiles adaptor to concatentate static media in .d-style directories to support Python 3.x by using bytes objects (commit) and move away from monkeypatch as it does not have a Python 3.x port yet (commit).
  • I also posted a short essay to my blog entitled "Ask the Dumb Questions" as well as provided an update on the latest Lintian release.
Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

I have generously been awarded a grant from the Core Infrastructure Initiative to fund my work in this area.

This month I:

  • Published a short blog post about how to determine which packages on your system are reproducible. [...]
  • Submitted a pull request for Numpy to make the generated files reproducible. [...]
  • Provided a patch to GTK upstream to ensure the immodules.cache files are reproducible. [...]
  • Within Debian:
    • Updated, moving it to HTTPS, adding cachebusting as well as keeping the number up-to-date.
    • Submitted the following patches to fix reproducibility-related toolchain issues:
      • gdk-pixbuf: Make the output of gdk-pixbuf-query-loaders reproducible. (#875704)
      • texlive-bin: Make PDF IDs reproducible. (#874102)
    • Submitted a patch to fix a reproducibility issue in doit.
  • Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.
  • Chaired our monthly IRC meeting. [...]
  • Worked on publishing our weekly reports. (#123, #124, #125, #126 & #127)

I also made the following changes to our tooling:


reproducible-check is our script to determine which packages actually installed on your system are reproducible or not.

  • Handle multi-architecture systems correctly. (#875887)
  • Use the "restricted" data file to mask transient issues. (#875861)
  • Expire the cache file after one day and base the local cache filename on the remote name. [...] [...]

I also blogged about this utility. [...]


diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

  • Filed an issue attempting to identify the causes behind an increased number of timeouts visible in our CI infrastructure, including running a number of benchmarks of recent versions. (#875324)
  • New features:
    • Add "binwalking" support to analyse concatenated CPIO archives such as initramfs images. (#820631).
    • Print a message if we are reading data from standard input. [...]
  • Bug fixes:
    • Loosen matching of file(1)'s output to ensure we correctly also match TTF files under file version 5.32. [...]
    • Correct references to path_apparent_size in comparators.utils.file and self.buf in diffoscope.diff. [...] [...]
  • Testing:
    • Make failing some critical flake8 tests result in a failed build. [...]
    • Check we identify all CPIO fixtures. [...]
  • Misc:
    • No need for try-assert-except block in [...]
    • Compare types with identity not equality. [...] [...]
    • Use's lazy argument interpolation. [...]
    • Remove unused imports. [...]
    • Numerous PEP8, flake8, whitespace, other cosmetic tidy-ups.


strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.

  • Log which handler processed a file. (#876140). [...]


disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues.


My activities as the current Debian Project Leader are covered in my monthly "Bits from the DPL" email to the debian-devel-announce mailing list.


I made a large number of changes to Lintian, the static analysis tool for Debian packages. It reports on various errors, omissions and general quality-assurance issues to maintainers:

I also blogged specifically about the Lintian 2.5.54 release.

Patches contributed
  • debconf: Please add a context manager to (#877096)
  • Add pronouns to ALL_STATUS_DESC. (#875128)
  • user-setup: Please drop set_special_users hack added for "the convenience of heavy testers". (#875909)
  • postgresql-common: Please update README.Debian for PostgreSQL 10. (#876438)
  • django-sitetree: Should not mask test failures. (#877321)
  • charmtimetracker:
    • Missing binary dependency on libqt5sql5-sqlite. (#873918)
    • Please drop "Cross-Platform" from package description. (#873917)

I also submitted 5 patches for packages with incorrect calls to find(1) in debian/rules against hamster-applet, libkml, pyferret, python-gssapi & roundcube.

Debian LTS

This month I have been paid to work 15¾ hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Documented an example usage of autopkgtests to test security changes.
  • Issued DLA 1084-1 and DLA 1085-1 for libidn and libidn2-0 to fix an integer overflow vulnerabilities in Punycode handling.
  • Issued DLA 1091-1 for unrar-free to prevent a directory traversal vulnerability from a specially-crafted .rar archive. This update introduces an regression test.
  • Issued DLA 1092-1 for libarchive to prevent malicious .xar archives causing a denial of service via a heap-based buffer over-read.
  • Issued DLA 1096-1 for wordpress-shibboleth, correcting an cross-site scripting vulnerability in the Shibboleth identity provider module.
  • python-django:
    • 1.11.5-1 — New upstream security release. (#874415)
    • 1.11.5-2 — Apply upstream patch to fix QuerySet.defer() with "super" and "subclass" fields. (#876816)
    • 2.0~alpha1-2 — New upstream alpha release of Django 2.0, dropping support for Python 2.x.
  • redis:
    • 4.0.2-1 — New upstream release.
    • 4.0.2-2 — Update 0004-redis-check-rdb autopkgtest test to ensure that the redis.rdb file exists before testing against it.
    • 4.0.2-2~bpo9+1 — Upload to stretch-backports.
  • aptfs (0.11.0-1) — New upstream release, moving away from using /var/lib/apt/lists internals. Thanks to Julian Andres Klode for a helpful bug report. (#874765)
  • lintian (2.5.53, 2.5.54) — New upstream releases. (Documented in more detail above.)
  • bfs (1.1.2-1) — New upstream release.
  • docbook-to-man (1:2.0.0-39) — Tighten autopkgtests and enable testing via
  • python-daiquiri (1.3.0-1) — New upstream release.

I also made the following non-maintainer uploads (NMUs):

  • vimoutliner (0.3.4+pristine-9.3):
    • Make the build reproducible. (#776369)
    • Expand placeholders in Debian.README. (#575142, #725634)
    • Recommend that the ftplugin is enabled. (#603115)
    • Correct "is not enable" typo.
  • bittornado (0.3.18-10.3):
    • Make the build reproducible. (#796212).
    • Add missing Build-Depends on dh-python.
  • dtc-xen (0.5.17-1.1):
    • Make the build reproducible. (#777322)
    • Add missing Build-Depends on dh-python.
  • dict-gazetteer2k (1.0.0-5.4):
    • Make the build reproducible. (#776376).
    • Override empty-binary-packagea Lintian warning to avoid dak autoreject.
  • cgilib (0.6-1.1) — Make the build reproducible. (#776935)
  • dhcping (1.2-4.2) — Make the build reproducible. (#777320)
  • dict-moby-thesaurus (1.0-6.4) — Make the build reproducible. (#776375)
  • dtaus (0.9-1.1) — Make the build reproducible. (#777321)
  • fastforward (1:0.51-3.2) — Make the build reproducible. (#776972)
  • wily (0.13.41-7.3) — Make the build reproducible. (#777360)
Debian bugs filed
  • clipit: Please choose a sensible startup default in "live" mode. (#875903)
  • git-buildpackage: Please add a --reset option to gbp pull. (#875852)
  • bluez: Please default Device "friendly name" to hostname without domain. (#874094)
  • Please explicitly link to {packages,tracker} (#876746)
  • Requests for packaging:
    • selfspy — log everything you do on the computer. (#873955)
    • shoogle — use the Google API from the shell. (#873916)
FTP Team

As a Debian FTP assistant I ACCEPTed 86 packages: bgw-replstatus, build-essential, caja-admin, caja-rename, calamares, cdiff, cockpit, colorized-logs, comptext, comptty, copyq, django-allauth, django-paintstore, django-q, django-test-without-migrations, docker-runc, emacs-db, emacs-uuid, esxml, fast5, flake8-docstrings, gcc-6-doc, gcc-7-doc, gcc-8, golang-github-go-logfmt-logfmt, golang-github-google-go-cmp, golang-github-nightlyone-lockfile, golang-github-oklog-ulid, golang-pault-go-macchanger, h2o, inhomog, ip4r, ldc, libayatana-appindicator, libbson-perl, libencoding-fixlatin-perl, libfile-monitor-lite-perl, libhtml-restrict-perl, libmojo-rabbitmq-client-perl, libmoosex-types-laxnum-perl, libparse-mime-perl, libplack-test-agent-perl, libpod-projectdocs-perl, libregexp-pattern-license-perl, libstring-trim-perl, libtext-simpletable-autowidth-perl, libvirt, linux, mac-fdisk, myspell-sq, node-coveralls, node-module-deps, nov-el, owncloud-client, pantomime-clojure, pg-dirtyread, pgfincore, pgpool2, pgsql-asn1oid, phpliteadmin, powerlevel9k, pyjokes, python-evdev, python-oslo.db, python-pygal, python-wsaccel, python3.7, r-cran-bindrcpp, r-cran-dotcall64, r-cran-glue, r-cran-gtable, r-cran-pkgconfig, r-cran-rlang, r-cran-spatstat.utils, resolvconf-admin, retro-gtk, ring-ssl-clojure, robot-detection, rpy2-2.8, ruby-hocon, sass-stylesheets-compass, selinux-dbus, selinux-python, statsmodels, webkit2-sharp & weston.

I additionally filed 4 RC bugs against packages that had incomplete debian/copyright files against: comptext, comptext, ldc & python-oslo.concurrency.

Categories: LUG Community Blogs

Mick Morgan: geeks rule

Sat, 30/09/2017 - 15:28

Well, sliderule, actually.

The ‘net is a truly wondrous space. I can’t recall exactly how I stumbled across the “International Sliderule Museum” but it is such a wonderful resource devoted to a tool which most people under the age of 40 will never have used that I just had to post a link to it.


Categories: LUG Community Blogs

Chris Lamb: Lintian: We are all Perl developers now

Mon, 25/09/2017 - 17:26

Lintian is a static analysis tool for Debian packages, reporting on various errors, omissions and general quality-assurance issues to maintainers.

I've previously written about my exploits with Lintian as well as authoring a short tutorial on how to write your own Lintian check.

Anyway, I recently uploaded version 2.5.53 about two months since previous release. The biggest changes you may notice are supporting the latest version of the Debian Policy as well the addition of checks to encourage the migration to Python 3.

Thanks to all who contributed patches, code review and bug reports to this release. The full changelog is as follows:

lintian (2.5.53) unstable; urgency=medium The "we are all Perl developers now" release. * Summary of tag changes: + Added: - alternatively-build-depends-on-python-sphinx-and-python3-sphinx - build-depends-on-python-sphinx-only - dependency-on-python-version-marked-for-end-of-life - maintainer-script-interpreter - missing-call-to-dpkg-maintscript-helper - node-package-install-in-nodejs-rootdir - override-file-in-wrong-package - package-installs-java-bytecode - python-foo-but-no-python3-foo - script-needs-depends-on-sensible-utils - script-uses-deprecated-nodejs-location - transitional-package-should-be-oldlibs-optional - unnecessary-testsuite-autopkgtest-header - vcs-browser-links-to-empty-view + Removed: - debug-package-should-be-priority-extra - missing-classpath - transitional-package-should-be-oldlibs-extra * checks/ + [CL] Fix an apache2-unparsable-dependency false positive by allowing periods (".") in dependency names. (Closes: #873701) * checks/ + [CL] Apply patches from Guillem Jover & Boud Roukema to improve the description of the binary-file-built-without-LFS-support tag. (Closes: #874078) * checks/changes.{pm,desc}: + [CL] Ignore DFSG-repacked packages when checking for upstream source tarball signatures as they will never match by definition. (Closes: #871957) + [CL] Downgrade severity of orig-tarball-missing-upstream-signature from "E:" to "W:" as many common tools do not make including the signatures easy enough right now. (Closes: #870722, #870069) + [CL] Expand the explanation of the orig-tarball-missing-upstream-signature tag to include the location of where dpkg-source will look. Thanks to Theodore Ts'o for the suggestion. * checks/ + [CL] Address a number of issues in copyright-year-in-future: - Prevent false positives in port numbers, email addresses, ISO standard numbers and matching specific and general street addresses. (Closes: #869788) - Match all violating years in a line, not just the first (eg. "2000-2107"). - Ignore meta copyright statements such as "Original Author". Thanks to Thorsten Alteholz for the bug report. (Closes: #873323) - Expand testsuite. * checks/cruft.{pm,desc}: + [CL] Downgrade severity of file-contains-fixme-placeholder tag from "important" (ie. "E:") to "wishlist" (ie. "I:"). Thanks to Gregor Herrmann for the suggestion. + [CL] Apply patch from Alex Muntada (alexm) to use "substr" instead of "substring" in mentions-deprecated-usr-lib-perl5-directory's description. (Closes: #871767) + [CL] Don't check copyright_hints file for FIXME placeholders. (Closes: #872843) + [CL] Don't match quoted "FIXME" variants as they are almost always deliberate. Thanks to Adrian Bunk for the report. (Closes: #870199) + [CL] Avoid false positives in missing source checks for "CSS Browser Selector". (Closes: #874381) * checks/ + [CL] Prevent a false positive of missing-build-dependency-for-dh_-command that can be exposed by following the advice for the recently added useless-autoreconf-build-depends tag. (Closes: #869541) * checks/debian-readme.{pm,desc}: + [CL] Ensure readme-debian-contains-debmake-template also checks for templates "Automatically generated by debmake". * checks/description.{desc,pm}: + [CL] Clarify explanation of description-starts-with-leading-spaces tag. Thanks to Taylor Kline for the report and patch. (Closes: #849622) + [NT] Skip capitalization-error-in-description-synopsis for auto-generated packages (such as dbgsym packages). * checks/fields.{desc,pm}: + [CL] Ensure that python3-foo packages have "Section: python", not just python2-foo. (Closes: #870272) + [RG] Do no longer require debug packages to be priority extra. + [BR] Use Lintian::Data for name/section mapping + [CL] Check for packages including "?rev=0&sc=0" in Vcs-Browser. (Closes: #681713) + [NT] Transitional packages should now be "oldlibs/optional" rather than "oldlibs/extra". The related tag has been renamed accordingly. * checks/ + [NT] Skip the check on auto-generated binary packages (such as dbgsym packages). * checks/files.{pm,desc}: + [BR] Avoid privacy-breach-generic false positives for legal.xml. + [BR] Detect install of node package under /usr/lib/nodejs/[^/]*$ + [CL] Check for packages shipping compiled Java class files. Thanks Carnë Draug . (Closes: #873211) + [BR] Privacy breach is no longer experimental. * checks/init.d.desc: + [RG] Do not recommend a versioned dependency on lsb-base in init.d-script-needs-depends-on-lsb-base. (Closes: #847144) * checks/ + [CL] Additionally consider .cljc files as code to avoid false- positive codeless-jar warnings. (Closes: #870649) + [CL] Drop problematic missing-classpath check. (Closes: #857123) * checks/menu-format.desc: + [CL] Prevent false positives in desktop-entry-lacks-keywords-entry for "Link" and "Directory" .desktop files. (Closes: #873702) * checks/python.{pm,desc}: + [CL] Split out Python checks from "scripts" check to a new, source check of type "source". + [CL] Check for python-foo without corresponding python3-foo packages to assist in Python 2.x deprecation. (Closes: #870681) + [CL] Check for packages that Build-Depend on python-sphinx only. (Closes: #870730) + [CL] Check for packages that alternatively Build-Depend on the Python 2 and Python 3 versions of Sphinx. (Closes: #870758) + [CL] Check for binary packages that depend on Python 2.x. (Closes: #870822) * checks/ + [CL] Correct false positives in unconditional-use-of-dpkg-statoverride by detecting "if !" as a valid shell prefix. (Closes: #869587) + [CL] Check for missing calls to dpkg-maintscript-helper(1) in maintainer scripts. (Closes: #872042) + [CL] Check for packages using sensible-utils without declaring a dependency after its split from debianutils. (Closes: #872611) + [CL] Warn about scripts using "nodejs" as an interpreter now that nodejs provides /usr/bin/node. (Closes: #873096) + [BR] Add a statistic tag giving interpreter. * checks/testsuite.{desc,pm}: + [CL] Remove recommendations to add a "Testsuite: autopkgtest" field to debian/control as it is added when needed by dpkg-source(1) since dpkg 1.17.1. (Closes: #865531) + [CL] Warn if we see an unnecessary "Testsuite: autopkgtest" header in debian/control. + [NT] Recognise "autopkgtest-pkg-go" as a valid test suite. + [CL] Recognise "autopkgtest-pkg-elpa" as a valid test suite. (Closes: #873458) + [CL] Recognise "autopkgtest-pkg-octave" as a valid test suite. (Closes: #875985) + [CL] Update the description of unknown-testsuite to reflect that "autopkgtest" is not the only valid value; the referenced URL is out-of-date (filed as #876008). (Closes: #876003) * data/binaries/embedded-libs: + [RG] Detect embedded copies of heimdal, libgxps, libquicktime, libsass, libytnef, and taglib. + [RG] Use an additional string to detect embedded copies of openjpeg2. (Closes: #762956) * data/fields/name_section_mappings: + [BR] node- package section is javascript. + [CL] Apply patch from Guillem Jover to add more section mappings. (Closes: #874121) * data/fields/obsolete-packages: + [MR] Add dh-systemd. (Closes: #872076) * data/fields/perl-provides: + [CL] Refresh perl provides. * data/fields/virtual-packages: + [CL] Update data file from archive. This fixes a false positive for "bacula-director". (Closes: #835120) * data/files/obsolete-paths: + [CL] Add note to /etc/bash_completion.d entry regarding stricter filename requirements. (Closes: #814599) * data/files/privacy-breaker-websites: + [BR] Detect custom donation logos like apache. + [BR] Detect generic counter website. * data/standards-version/release-dates: + [CL] Add 4.0.1 and 4.1.0 as known standards versions. (Closes: #875509) * debian/control: + [CL] Mention Debian Policy v4.1.0 in the description. + [CL] Add myself to Uploaders. + [CL] Drop unnecessary "Testsuite: autopkgtest"; this is implied from debian/tests/control existing. * commands/ + [CL] Add a --list-tags option to print all tags Lintian knows about. Thanks to Rajendra Gokhale for the suggestion. (Closes: #779675) * commands/ + [CL] Apply patch from Maia Everett to avoid British spelling when using en_US locale. (Closes: #868897) * lib/Lintian/ + [CL] Stop emitting {maintainer,uploader}-address-causes-mail-loops for addresses. (Closes: #871575) * lib/Lintian/Collect/ + [NT] Introduce an "auto-generated" argument for "is_pkg_class". * lib/Lintian/ + [CL] Modify Lintian::Data's "all" to always return keys in insertion order, dropping dependency on libtie-ixhash-perl. * helpers/coll/objdump-info-helper: + [CL] Apply patch from Steve Langasek to accommodate binutils 2.29 outputting symbols in a different format on ppc64el. (Closes: #869750) * t/tests/fields-perl-provides/tags: + [CL] Update expected output to match new Perl provides. * t/tests/files-privacybreach/*: + [CL] Add explicit test for packages including external fonts via the Google Font API. Thanks to Ian Jackson for the report. (Closes: #873434) + [CL] Add explicit test for packages including external fonts via the Typekit API via <script/> HTML tags. * t/tests/*/desc: + [CL] Add missing entries in "Test-For" fields to make development/testing workflow less error-prone. * private/generate-tag-summary: + [CL] git-describe(1) will usually emit 7 hexadecimal digits as the abbreviated object name, However, as this can be user-dependent, pass --abbrev=0 to ensure it does not vary between systems. This also means we do not need to strip it ourselves. * private/refresh-*: + [CL] Use as the default mirror. + [CL] Update locations of Contents-<arch> files; they are now namespaced by distribution (eg. "main"). -- Chris Lamb <> Wed, 20 Sep 2017 09:25:06 +0100

Categories: LUG Community Blogs

Chris Lamb: Which packages on my system are reproducible?

Fri, 15/09/2017 - 09:29

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users. The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process.

As part of this project I wrote a script to determine which packages installed on your system are "reproducible" or not:

$ apt install devscripts […] $ reproducible-check […] W: subversion (1.9.7-2) is unreproducible (libsvn-perl, libsvn1, subversion) <> W: taglib (1.11.1+dfsg.1-0.1) is unreproducible (libtag1v5, libtag1v5-vanilla) <> W: tcltk-defaults (8.6.0+9) is unreproducible (tcl, tk) <> W: tk8.6 (8.6.7-1) is unreproducible (libtk8.6, tk8.6) <> W: valgrind (1:3.13.0-1) is unreproducible <> W: wavpack (5.1.0-2) is unreproducible (libwavpack1) <> W: x265 (2.5-2) is unreproducible (libx265-130) <> W: xen (4.8.1-1+deb9u1) is unreproducible (libxen-4.8, libxenstore3.0) <> W: xmlstarlet (1.6.1-2) is unreproducible <> W: xorg-server (2:1.19.3-2) is unreproducible (xserver-xephyr, xserver-xorg-core) <> 282/4494 (6.28%) of installed binary packages are unreproducible.

Whether a package is "reproducible" or not is determined by querying the Debian Reproducible Builds testing framework.

The --raw command-line argument lets you play with the data in more detail. For example, you can see who maintains your unreproducible packages:

$ reproducible-check --raw | dd-list --stdin Alec Leamas <> lirc (U) Alessandro Ghedini <> valgrind Alessio Treglia <> fluidsynth (U) libsoxr (U) […]

reproducible-check is available in devscripts since version 2.17.10, which landed in Debian unstable on 14th September 2017.

Categories: LUG Community Blogs

Chris Lamb: Ask the dumb questions

Tue, 05/09/2017 - 12:51

In the same way it vital to ask the "smart questions", it is equally important to ask the dumb ones.

Whilst your milieu might be—say—comparing and contrasting the finer details of commission structures between bond brokers, if you aren't quite sure of the topic learn to be bold and confident enough to boldly ask: I'm sorry, but what actually is a bond?

Don't consider this to be an all-or-nothing affair. After all, you might have at least some idea about what a bond is. Rather, adjust your tolerance to also ask for clarification when you are merely slightly unsure or merely slightly uncertain about a concept, term or reference.

So why do this? Most obviously, you are learning something and expanding your knowledge about the world, but a clarification can avoid problems later if you were mistaken in your assumptions.

Not only that, asking "can you explain that?" or admitting "I don't follow…" is not only being honest with yourself, the vulnerability you show when admitting one's ignorance opens yourself to others leading to closer friendships and working relationships.

We clearly have a tendency to want to come across as knowledgable or―perhaps more honestly―we don't want to appear dumb or uninformed as it will bruise our ego. But the precise opposite is true: nodding and muddling your way through conversations you only partly understand is unlikely to cultivate true feelings of self-respect and a healthy self-esteem.

Since adopting this approach I have found I've rarely derailed the conversation. In fact, speaking up not only encourages and flatters others that you care about their subject, it has invariably lead to related matters which are not only more inclusive but actually novel and interesting to all present.

So push through the voice in your head and be that elephant in the room. After all, you might not the only person thinking it. If it helps, try reframing it to yourself as helping others…

You'll be finding it effortless soon enough. Indeed, asking the dumb question is actually a positive feedback loop where each question you pose helps you make others in the future. Excellence is not an act, but a habit.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): F/LOSS activity, August 2017

Sat, 02/09/2017 - 11:00

Shockingly enough, my focus started out on Gitano once more. We managed a 1.1 release of Gitano during the Debian conference's "camp" which occurs in the week before the conference. This was a joint effort of myself, Richard Maw, and Richard Ipsum. I have to take my hat off to Richard Maw, because without his dedication to features, 1.1 would lack some stuff which Richard Ipsum proposed around ruleset support for basic readers/writers and frankly 1.1 would be a weaker release without it.

Because of the debconf situation, we didn't have a Gitano developer day which, while sad, didn't slow us down much...

  • Once again, we reviewed our current task state
  • I submitted a series which fixed our test suite for Git 2.13 which was an FTBFS bug submitted against the Debian package for Gitano. Richard Maw reviewed and merged it.
  • Richard Maw supplied a series to add testing for dangling HEAD syndrome. I reviewed and merged that.
  • Richard Maw submitted a patch to improve the auditability of the 'as' command and I reviewed and merged that.
  • Richard Ipsum submitted a patch to add reader/writer configs to ease simple project management in Gitano. I'm not proud to say that I was too busy to look at this and ended up saying it was unlikely it'd get in. Richard Maw, quite rightly, took umbrage at that and worked on the patch, eventually submitting a new series with tests which I then felt obliged to review and I merged the series eventually.

    This is an excellent example of where just because one person is too busy doesn't mean that a good idea should be dropped, and I am grateful to Richard Maw for getting this work mergeable and effectively guilt-tripping me into reviewing/merging. This is a learnable moment for me and I hope to do better into the future.

  • During all that time, I was working on a plugin to support in Gitano. This work ranged across hooks and caused me to spend a long time thinking about the semantics of configuration overriding etc. Fortunately I got there in the end, and with a massive review effort from Richard Maw, we got it merged into Gitano.
  • Finally I submitted a patch which caused the tests we run in Gitano to run from an 'install' directory which ensures that we catch bugs such as those which happened in earlier series where we missed out rules files for installation etc. Richard Maw reviewed and merged that.
  • And then we released the new version of Gitano and subsidiary libraries.

    There was Luxio version 13 which switched us to readdir() from readdir_r() thanks to Richard Ipsum; Gall 1.3 which contained a bunch of build cleanups, and also a revparse_single() implementation in the C code to speed things up thanks to Richard Maw; Supple 1.0.8 which improved wrapper environment cleanups thanks to Richard Ipsum, allowed baking of paths in which means Nix is easier to support (again thanks to Richard Ipsum), fixed setuid handling so that Nix is easier to support (guess what? Richard Ipsum again); Lace 1.4 which now verifies definition names in allow/deny/anyof/allof and also produces better error messages from nested includes.

    And, of course, Gitano 1.1 whose changes were somewhat numerous and so you are invited to read them in the Gitano NEWS file for the release.

Not Gitano

Of course, not everything I did in August was Gitano related. In fact once I had completed the 1.1 release and uploaded everything to Debian I decided that I was going to take a break from Gitano until the next developer day. (In fact there's even some patch series still unread on the mailing list which I will get to when I start the developer day.)

I have long been interested in STM32 microcontrollers, using them in a variety of projects including the Entropy Key which some of you may remember. Jorge Aparicio was working on Cortex-M3 support (among other microcontrollers) in Rust and he then extended that to include a realtime framework called RTFM and from there I got interested in what I might be able to do with Rust on STM32. I noticed that there weren't any pure Rust implementations of the USB device stack which would be necessary in order to make a device, programmed in Rust, appear on a USB port for a computer to control/use. This tweaked my interest.

As many of my readers are aware, I am very bad at doing things without some external motivation. As such, I then immediately offered to give a talk at a conference which should be happening in November, just so that I'd be forced to get on with learning and implementing the stack. I have been chronicling my work in this blog, and you're encouraged to go back and read them if you have similar interests. I'm sure that as my work progresses, I'll be doing more and more of that and less of Gitano, for at least the next two months.

To bring that into context as F/LOSS work, I did end up submitting some patches to Jorge's STM32F103xx repository to support a couple more clock configuration entries so that USB and ADCs can be set up cleanly. So at least there's that.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in August 2017

Thu, 31/08/2017 - 21:39

Here is my monthly update covering what I have been doing in the free software world in August 2017 (previous month):

  • Created ZeroCoolOS, a live operating system that plays the film Hackers (1995) on a continuous loop.
  • Sent a patch for pristine-tar to allow storage of detached upstream signatures. (#871809)
  • Worked more on Lintian, a static analysis tool for Debian packages, reporting on various errors, omissions and quality-assurance issues to the maintainer (previous changes):
    • Fix an apache2-unparsable-dependency false positive by allowing periods in dependency names. (#873701)
    • Ignore "repacked" packages when checking for upstream source tarball signatures as they will never match.
    • Downgrade the severity of orig-tarball-missing-upstream-signature. (#870722)
    • From a suggestion by Theodore Ts'o, expand the explanation of orig-tarball-missing-upstream-signature to include the location of where dpkg-source looks.
    • Address a number of issues in the copyright-year-in-future tag including preventing false positives in port numbers, email addresses, ISO standard numbers and street addresses (#869788), as well as "meta" or testing statements (#873323). In addition, report all violating years in a line and expand the testsuite.
    • Don't match quoted "FIXME" variants of file-contains-fixme-placeholder (#870199), avoid checking copyright_hints files (#872843) and downgrade the tag's severity.
    • Apply a patch from Alex Muntada to recommend "substr" over of "substring" in mentions-deprecated-usr-lib-perl5-directory. (#871767)
    • Prevent missing-build-dependency-for-dh_-command false positives exposed by following the advice in useless-autoreconf-build-depends. (#869541)
    • Ensure readme-debian-contains-debmake-template also checks for files containing "Automatically generated by debmake".
    • Check python3-foo packages have a Section: python, not just python2-foo. (#870272)
    • Check for packages shipping compiled Java class files. (#873211)
    • Additionally consider .cljc files to avoid codeless-jar warnings. (#870649)
    • Prevent desktop-entry-lacks-keywords-entry false positives for Link and Directory-style .desktop files. (#873702)
    • Split out Python checks from checks/ check to a new, source check of type source.
    • Check for python-foo without a corresponding python3-foo package. (#870681)
    • Complain about packages that Build-Depend on python-sphinx only. (#870730)
    • Warn about packages that alternatively Build-Depend on the Python 2 and Python 3 versions of Sphinx. (#870758)
    • Check for packages that depend on Python 2.x. (#870822)
    • Correct false positives in unconditional-use-of-dpkg-statoverride by detecting "if !" as a shell prefix. (#869587)
    • Alert on for missing calls to dpkg-maintscript-helper(1) in maintainer scripts. (#872042)
    • Check for packages using sensible-utils without declaring a dependency after splitting from debianutils. (#872611)
    • Warn about scripts using nodejs as an interpreter now that the nodejs script provides /usr/bin/node. (#873096)
    • Remove recommendations to add a Testsuite: autopkgtest field to debian/control and emit a new tag the package if it does so. (#865531)
    • Recognise autopkgtest-pkg-elpa as a valid test suite. (#873458)
    • Add note to /etc/bash_completion.d's obsolete path warning output regarding stricter filename requirements. (#814599)
    • Add 4.0.1 and 4.1.0 as known Policy standards versions.
    • Apply a patch from Maia Everett to avoid British spellings under the en_US locale. (#868897)
    • Stop emitting {maintainer,uploader}-address-causes-mail-loops for addresses. (#871575)
    • Modify Lintian::Data's all subroutine to always return keys in insertion order.
    • Apply a patch from Steve Langasek to accomodate binutils outputting symbols in a different format on the ppc64el architecture. (#869750)
    • Add an explicit test for packages including external fonts via the Google Font and TypeKit APIs. (#873434)
    • Add missing entries in internal Test-For fields to make development/testing workflow less error-prone.
  • Sent three pull requests to git-buildpackage, a tool to assist in Debian packaging from Git repositories:
    • Make pq --abbrev= configurable. (#872351)
    • Use build profiles to avoid installation of test dependencies. (#31)
    • Correct "allow to" grammar. (#30)
  • Updated (my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform for testing):
    • Move away from; Travis appears to be using a HTTP proxy that strips SRV records. (commit)
    • Highlight double quotes are required for TRAVIS_DEBIAN_EXTRA_REPOSITORY. (commit)
    • Use force-unsafe-io. (commit)
    • Clarify docs when upstream already has a travis.yml file. (#46)
    • Make documentation easier to copy-paste. (commit)
  • Merged a pull request in django-slack, my library to easily post messages to the Slack group-messaging utility, where instantiation of a SlackException was failing. (#71)
  • Assigned two pull requests to the Redis key-value database store to correct "did not received" and "faield" typos. (#4216 & #4215).
Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

I have generously been awarded a grant from the Core Infrastructure Initiative to fund my work in this area.

This month I:

  • Presented a status update at Debconf17 in Montréal, Canada alongside Holger Levsen, Maria Glukhova, Steven Chamberlain, Vagrant Cascadian, Valerie Young and Ximin Luo.
  • I worked on the following issues upstream:
    • glib2.0: Please make the output of gio-querymodules reproducible. (...)
    • gcab: Please make the output reproducible. (...)
    • gtk+2.0: Please make the immodules.cache files reproducible. (...)
    • desktop-file-utils: Please make the output reproducible. (...)
  • Within Debian:
  • Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.
  • Worked on publishing our weekly reports. (#118, #119, #120, #121 & #122)

I also made the following changes to our tooling:


diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

  • Use name attribute over path to avoid leaking comparison full path in output. (commit)
  • Add missing skip_unless_module_exists import. (commit)
  • Tidy diffoscope.progress and the XML comparator (commit, commit)


disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues.

  • Add a simple autopkgtest smoke test. (commit)

Debian Patches contributed
  • openssh: Quote the IP address in ssh-keygen -f suggestions. (#872643)
  • libgfshare:
    • SIGSEGV if /dev/urandom is not accessible. (#873047)
    • Add bindnow hardening. (#872740)
    • Support nodoc build profile. (#872739)
  • devscripts:
  • memcached: Add hardening to systemd .service file. (#871610)
  • googler: Tidy long and short package descriptions. (#872461)
  • gnome-split: Homepage points to domain-parked website. (#873037)
  • python-django 1:1.11.4-1 — New upstream release.
  • redis:
    • 4:4.0.1-3 — Drop yet more non-deterministic tests.
    • 4:4.0.1-4 — Tighten systemd/seccomp hardening.
    • 4:4.0.1-5 — Drop even more tests with timing issues.
    • 4:4.0.1-6 — Don't install completions to /usr/share/bash-completion/completions/debian/bash_completion/.
    • 4:4.0.1-7 — Don't let sentinel integration tests fail the build as they use too many timers to be meaningful. (#872075)
  • python-gflags 1.5.1-3 — If SOURCE_DATE_EPOCH is set, either use that as a source of current dates or the UTC-version of the file's modification time (#836004), don't call update-alternatives --remove in postrm. update debian/watch/Homepage & refresh/tidy the packaging.
  • bfs 1.1.1-1 — New upstream release, tidy autopkgtest & patches, organising the latter with Pq-Topic.
  • python-daiquiri 1.2.2-1 — New upstream release, tidy autopkgtests & update travis.yml from
  • aptfs 2:0.10-2 — Add upstream signing key, refer to /usr/share/common-licenses/GPL-3 in debian/copyright & tidy autopkgtests.
  • adminer 4.3.1-2 — Add a simple autopkgtest & don't install the Selenium-based tests in the binary package.
  • zoneminder (1.30.4+dfsg-2) — Prevent build failures with GCC 7 (#853717) & correct example /etc/fstab entries in README.Debian (#858673).

Finally, I reviewed and sponsored uploads of astral, inflection, more-itertools, trollius-redis & wolfssl.

Debian LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 1049-1 for libsndfile preventing a remote denial of service attack.
  • Issued DLA 1052-1 against subversion to correct an arbitrary code execution vulnerability.
  • Issued DLA 1054-1 for the libgxps XML Paper Specification library to prevent a remote denial of service attack.
  • Issued DLA 1056-1 for cvs to prevent a command injection vulnerability.
  • Issued DLA 1059-1 for the strongswan VPN software to close a denial of service attack.
Debian bugs filed
  • wget: Please hash the hostname in ~/.wget-hsts files. (#870813)
  • debian-policy: Clarify whether mailing lists in Maintainers/Uploaders may be moderated. (#871534)
  • git-buildpackage: "pq export" discards text within square brackets. (#872354)
  • Escape HTML in debcheck before outputting. (#872646)
  • pristine-tar: Enable multithreaded compression in pristine-xz. (#873229)
  • tryton-meta: Please combine tryton-modules-* into a single source package with multiple binaries. (#873042)
  • azure-cli:
  • fwupd-tests: Don't ship test files to generic /usr/share/installed-tests dir. (#872458)
  • libvorbis: Maintainer fields points to a moderated mailing list. (#871258)
  • rmlint-gui: Ship a rmlint-gui binary. (#872162)
  • template-glib: debian/copyright references online source without quotation. (#873619)
FTP Team

As a Debian FTP assistant I ACCEPTed 147 packages: abiword, adacgi, adasockets, ahven, animal-sniffer, astral, astroidmail, at-at-clojure, audacious, backdoor-factory, bdfproxy, binutils, blag-fortune, bluez-qt, cheshire-clojure, core-match-clojure, core-memoize-clojure, cypari2, data-priority-map-clojure, debian-edu, debian-multimedia, deepin-gettext-tools, dehydrated-hook-ddns-tsig, diceware, dtksettings, emacs-ivy, farbfeld, gcc-7-cross-ports, git-lfs, glewlwyd, gnome-recipes, gnome-shell-extension-tilix-dropdown, gnupg2, golang-github-aliyun-aliyun-oss-go-sdk, golang-github-approvals-go-approval-tests, golang-github-cheekybits-is, golang-github-chzyer-readline, golang-github-denverdino-aliyungo, golang-github-glendc-gopher-json, golang-github-gophercloud-gophercloud, golang-github-hashicorp-go-rootcerts, golang-github-matryer-try, golang-github-opentracing-contrib-go-stdlib, golang-github-opentracing-opentracing-go, golang-github-tdewolff-buffer, golang-github-tdewolff-minify, golang-github-tdewolff-parse, golang-github-tdewolff-strconv, golang-github-tdewolff-test, golang-gopkg-go-playground-validator.v8, gprbuild, gsl, gtts, hunspell-dz, hyperlink, importmagic, inflection, insighttoolkit4, isa-support, jaraco.itertools, java-classpath-clojure, java-jmx-clojure, jellyfish1, lazymap-clojure, libblockdev, libbytesize, libconfig-zomg-perl, libdazzle, libglvnd, libjs-emojify, libjwt, libmysofa, libundead, linux, lua-mode, math-combinatorics-clojure, math-numeric-tower-clojure, mediagoblin, medley-clojure, more-itertools, mozjs52, openssh-ssh1, org-mode, oysttyer, pcscada, pgsphere, poppler, puppetdb, py3status, pycryptodome, pysha3, python-cliapp, python-coloredlogs, python-consul, python-deprecation, python-django-celery-results, python-dropbox, python-fswrap, python-hbmqtt, python-intbitset, python-meshio, python-parameterized, python-pgpy, python-py-zipkin, python-pymeasure, python-thriftpy, python-tinyrpc, python-udatetime, python-wither, python-xapp, pythonqt, r-cran-bit, r-cran-bit64, r-cran-blob, r-cran-lmertest, r-cran-quantmod, r-cran-ttr, racket-mode, restorecond, rss-bridge, ruby-declarative, ruby-declarative-option, ruby-errbase, ruby-google-api-client, ruby-rash-alt, ruby-representable, ruby-test-xml, ruby-uber, sambamba, semodule-utils, shimdandy, sjacket-clojure, soapysdr, stencil-clojure, swath, template-glib, tools-analyzer-jvm-clojure, tools-namespace-clojure, uim, util-linux, vim-airline, vim-airline-themes, volume-key, wget2, xchat, xfce4-eyes-plugin & xorg-gtest.

I additionally filed 6 RC bugs against packages that had incomplete debian/copyright files against: gnome-recipes, golang-1.9, libdazzle, poppler, python-py-zipkin & template-glib.

Categories: LUG Community Blogs

Jonathan McDowell: C, floating point, and help!

Thu, 31/08/2017 - 17:58

Floating point is a pain. I know this. But I recently took over the sigrok packages in Debian and as part of updating to the latest libsigkrok4 library enabled the post compilation tests. Which promptly failed on i386. Some narrowing down of the problem leads to the following test case (which fails on both gcc-6 under Debian/Stretch and gcc-7 on Debian/Testing):

#include <inttypes.h> #include <stdio.h> #include <stdint.h> int main(int argc, char *argv[]) { printf("%" PRIu64 "\n", (uint64_t)((1.034567) * (uint64_t)(1000000ULL))); }

We expect to see 1.034567 printed out. On x86_64 we do:

$ arch x86_64 $ gcc -Wall t.c -o t ; ./t 1034567

If we compile for 32-bit the result is also as expected:

$ gcc -Wall -m32 t.c -o t ; ./t 1034567

Where things get interesting is when we enable --std=c99:

$ gcc -Wall --std=c99 t.c -o t ; ./t 1034567 $ gcc -Wall -m32 --std=c99 t.c -o t ; ./t 1034566

What? It turns out all the cXX standards result in the last digit incorrectly being 6, while the gnuXX standards (gnu11 is apparently the default) result in the correct trailing 7. Is there some postfix I can add to the value to prevent the floating point truncation taking place? Or do I just have to accept this? It works fine on armel, so it’s not a simple 32/64 bit issue.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): STM32 USB and Rust - Packet Memory Area

Wed, 30/08/2017 - 17:15

In this, our next exciting installment of STM32 and Rust for USB device drivers, we're going to look at what the STM32 calls the 'packet memory area'. If you've been reading along with the course, including reading up on the datasheet content then you'll be aware that as well as the STM32's normal SRAM, there's a 512 byte SRAM dedicated to the USB peripheral. This SRAM is called the 'packet memory area' and is shared between the main bus and the USB peripheral core. Its purpose is, simply, to store packets in transit. Both those IN to the host (so stored queued for transmission) or OUT from the host (so stored, queued for the application to extract and consume).

It's time to actually put hand to keyboard on some Rust code, and the PMA is the perfect starting point, since it involves two basic structures. Packets are the obvious first structure, and they are contiguous sets of bytes which for the purpose of our work we shall assume are one to sixty-four bytes long. The second is what the STM32 datasheet refers to as the BTABLE or Buffer Descriptor Table. Let's consider the BTABLE first.

The Buffer Descriptor Table

The BTABLE is arranged in quads of 16bit words. For "normal" endpoints this is a pair of descriptors, each consisting of two words, one for transmission, and one for reception. The STM32 also has a concept of double buffered endpoints, but we're not going to consider those in our proof-of-concept work. The STM32 allows for up to eight endpoints (EP0 through EP7) in internal register naming, though they support endpoints numbered from zero to fifteen in the sense of the endpoint address numbering. As such there're eight descriptors each four 16bit words long (eight bytes) making for a buffer descriptor table which is 64 bytes in size at most.

Buffer Descriptor Table Byte offset in PMA Field name Description (EPn * 8) + 0 USB_ADDRn_TX The address (inside the PMA) of the TX buffer for EPn (EPn * 8) + 2 USB_COUNTn_TX The number of bytes present in the TX buffer for EPn (EPn * 8) + 4 USB_ADDRn_RX The address (inside the PMA) of the RX buffer for EPn (EPn * 8) + 6 USB_COUNTn_RX The number of bytes of space available for the RX buffer for EPn (and once received, the number of bytes received)

The TX entries are trivial to comprehend. To transmit a packet, part of the process involves writing the packet into the PMA, putting the address into the appropriate USB_ADDRn_TX entry, and the length into the corresponding USB_COUNTn_TX entry, before marking the endpoint as ready to transmit.

To receive a packet though is slightly more complex. The application must allocate some space in the PMA, setting the address into the USB_ADDRn_RX entry of the BTABLE before filling out the top half of the USB_COUNTn_RX entry. For ease of bit sizing, the STM32 only supports space allocations of two to sixty-two bytes in steps of two bytes at a time, or thirty-two to five-hundred-twelve bytes in steps of thirty-two bytes at a time. Once the packet is received, the USB peripheral will fill out the lower bits of the USB_COUNTn_RX entry with the actual number of bytes filled out in the buffer.

Packets themselves

Since packets are, typically, a maximum of 64 bytes long (for USB 2.0) and are simply sequences of bytes with no useful structure to them (as far as the USB peripheral itself is concerned) the PMA simply requires that they be present and contiguous in PMA memory space. Addresses of packets are relative to the base of the PMA and are byte-addressed, however they cannot start on an odd byte, so essentially they are 16bit addressed. Since the BTABLE can be anywhere within the PMA, as can the packets, the application will have to do some memory management (either statically, or dynamically) to manage the packets in the PMA.

Accessing the PMA

The PMA is accessed in 16bit word sections. It's not possible to access single bytes of the PMA, nor is it conveniently structured as far as the CPU is concerned. Instead the PMA's 16bit words are spread on 32bit word boundaries as far as the CPU knows. This is done for convenience and simplicity of hardware, but it means that we need to ensure our library code knows how to deal with this.

First up, to convert an address in the PMA into something which the CPU can use we need to know where in the CPU's address space the PMA is. Fortunately this is fixed at 0x4000_6000. Secondly we need to know what address in the PMA we wish to access, so we can determine which 16bit word that is, and thus what the address is as far as the CPU is concerned. If we assume we only ever want to access 16bit entries, we can just multiply the PMA offset by two before adding it to the PMA base address. So, to access the 16bit word at byte-offset 8 in the PMA, we'd look for the 16bit word at 0x4000_6000 + (0x08 * 2) => 0x4000_6010.

Bundling the PMA into something we can use

I said we'd do some Rust, and so we shall…

// Thanks to the work by Jorge Aparicio, we have a convenient wrapper // for peripherals which means we can declare a PMA peripheral: pub const PMA: Peripheral<PMA> = unsafe { Peripheral::new(0x4000_6000) }; // The PMA struct type which the peripheral will return a ref to pub struct PMA { pma_area: PMA_Area, } // And the way we turn that ref into something we can put a useful impl on impl Deref for PMA { type Target = PMA_Area; fn deref(&self) -> &PMA_Area { &self.pma_area } } // This is the actual representation of the peripheral, we use the C repr // in order to ensure it ends up packed nicely together #[repr(C)] pub struct PMA_Area { // The PMA consists of 256 u16 words separated by u16 gaps, so lets // represent that as 512 u16 words which we'll only use every other of. words: [VolatileCell<u16>; 512], }

That block of code gives us three important things. Firstly a peripheral object which we will be able to (later) manage nicely as part of the set of peripherals which RTFM will look after for us. Secondly we get a convenient packed array of u16s which will be considered volatile (the compiler won't optimise around the ordering of writes etc). Finally we get a struct on which we can hang an implementation to give our PMA more complex functionality.

A useful first pair of functions would be to simply let us get and put u16s in and out of that word array, since we're only using every other word…

impl PMA_Area { pub fn get_u16(&self, offset: usize) -> u16 { assert!((offset & 0x01) == 0); self.words[offset].get() } pub fn set_u16(&self, offset: usize, val: u16) { assert!((offset & 0x01) == 0); self.words[offset].set(val); } }

These two functions take an offset in the PMA and return the u16 word at that offset. They only work on u16 boundaries and as such they assert that the bottom bit of the offset is unset. In a release build, that will go away, but during debugging this might be essential. Since we're only using 16bit boundaries, this means that the first word in the PMA will be at offset zero, and the second at offset two, then four, then six, etc. Since we allocated our words array to expect to use every other entry, this automatically converts into the addresses we desire.

If we pop (and please don't worry about the unsafe{} stuff for now):

unsafe { (&*usb::pma::PMA.get()).set_u16(4, 64); }

into our main function somewhere, and then build and objdump our test binary we can see the following set of instructions added:

80001e4: f246 0008 movw r0, #24584 ; 0x6008 80001e8: 2140 movs r1, #64 ; 0x40 80001ea: f2c4 0000 movt r0, #16384 ; 0x4000 80001ee: 8001 strh r1, [r0, #0]

This boils down to a u16 write of 0x0040 (64) to the address 0x4006008 which is the third 32 bit word in the CPU's view of the PMA memory space (where offset 4 is the third 16bit word) which is exactly what we'd expect to see.

We can, from here, build up some functions for manipulating a BTABLE, though the most useful ones for us to take a look at are the RX counter functions:

pub fn get_rxcount(&self, ep: usize) -> u16 { self.get_u16(BTABLE + (ep * 8) + 6) & 0x3ff } pub fn set_rxcount(&self, ep: usize, val: u16) { assert!(val <= 1024); let rval: u16 = { if val > 62 { assert!((val & 0x1f) == 0); (((val >> 5) - 1) << 10) | 0x8000 } else { assert!((val & 1) == 0); (val >> 1) << 10 } }; self.set_u16(BTABLE + (ep * 8) + 6, rval) }

The getter is fairly clean and clear, we need the BTABLE base in the PMA, add the address of the USB_COUNTn_RX entry to that, retrieve the u16 and then mask off the bottom ten bits since that's the size of the relevant field.

The setter is a little more complex, since it has to deal with the two possible cases, this isn't pretty and we might be able to write some better peripheral structs in the future, but for now, if the length we're setting is 62 or less, and is divisible by two, then we put a zero in the top bit, and the number of 2-byte lumps in at bits 14:10, and if it's 64 or more, we mask off the bottom to check it's divisible by 32, and then put the count (minus one) of those blocks in, instead, and set the top bit to mark it as such.

Fortunately, when we set constants, Rust's compiler manages to optimise all this very quickly. For a BTABLE at the bottom of the PMA, and an initialisation statement of:

unsafe { (&*usb::pma::PMA.get()).set_rxcount(1, 64); }

then we end up with the simple instruction sequence:

80001e4: f246 001c movw r0, #24604 ; 0x601c 80001e8: f44f 4104 mov.w r1, #33792 ; 0x8400 80001ec: f2c4 0000 movt r0, #16384 ; 0x4000 80001f0: 8001 strh r1, [r0, #0]

We can decompose that into a C like *((u16*)0x4000601c) = 0x8400 and from there we can see that it's writing to the u16 at 0x1c bytes into the CPU's view of the PMA, which is 14 bytes into the PMA itself. Since we know we set the BTABLE at the start of the PMA, it's 14 bytes into the BTABLE which is firmly in the EP1 entries. Specifically it's USB_COUNT1_RX which is what we were hoping for. To confirm this, check out page 651 of the datasheet. The value set was 0x8400 which we can decompose into 0x8000 and 0x0400. The first is the top bit and tells us that BL_SIZE is one, and thus the blocks are 32 bytes long. Next the 0x4000 if we shift it right ten places, we get the value 2 for the field NUM_BLOCK and multiplying 2 by 32 we get the 64 bytes we asked it to set as the size of the RX buffer. It has done exactly what we hoped it would, but the compiler managed to optimise it into a single 16 bit store of a constant value to a constant location. Nice and efficient.

Finally, let's look at what happens if we want to write a packet into the PMA. For now, let's assume packets come as slices of u16s because that'll make our life a little simpler:

pub fn write_buffer(&self, base: usize, buf: &[u16]) { for (ofs, v) in buf.iter().enumerate() { self.set_u16(base + (ofs * 2), *v); } }

Yes, even though we're deep in no_std territory, we can still get an iterator over the slice, and enumerate it, getting a nice iterator of (index, value) though in this case, the value is a ref to the content of the slice, so we end up with *v to deref it. I am sure I could get that automatically happening but for now it's there.

Amazingly, despite using iterators, enumerators, high level for loops, function calls, etc, if we pop:

unsafe { (&*usb::pma::PMA.get()).write_buffer(0, &[0x1000, 0x2000, 0x3000]); }

into our main function and compile it, we end up with the instruction sequence:

80001e4: f246 0000 movw r0, #24576 ; 0x6000 80001e8: f44f 5180 mov.w r1, #4096 ; 0x1000 80001ec: f2c4 0000 movt r0, #16384 ; 0x4000 80001f0: 8001 strh r1, [r0, #0] 80001f2: f44f 5100 mov.w r1, #8192 ; 0x2000 80001f6: 8081 strh r1, [r0, #4] 80001f8: f44f 5140 mov.w r1, #12288 ; 0x3000 80001fc: 8101 strh r1, [r0, #8]

which, as you can see, ends up being three sequential halfword stores directly to the right locations in the CPU's view of the PMA. You have to love seriously aggressive compile-time optimisation

Hopefully, by next time, we'll have layered some more pleasant routines on our PMA code, and begun a foray into the setup necessary before we can begin handling interrupts and start turning up on a USB port.

Categories: LUG Community Blogs

Jonathan McDowell: On my way home from OMGWTFBBQ

Sun, 27/08/2017 - 19:42

I started writing this while sitting in Stansted on my way home from the annual UK Debian BBQ. I’m finally home now, after a great weekend catching up with folk. It’s a good social event for a bunch of Debian folk, and I’m very grateful that Steve and Jo continue to make it happen. These days there are also a number of generous companies chipping in towards the cost of food and drink, so thanks also to Codethink and QvarnLabs AB for the food, Collabora and Mythic Beasts for the beer and Chris for the coffee. And Rob for chasing us all for contributions to cover the rest.

I was trying to remember when the first one of these I attended was; trawling through mail logs there was a Cambridge meetup that ended up at Steve’s old place in April 2001, and we’ve consistently had the summer BBQ since 2004, but I’m not clear on what happened in between. Nonetheless it’s become a fixture in the calendar for those of us in the UK (and a number of people from further afield who regularly turn up). We’ve become a bit more sedate, but it’s good to always see a few new faces, drink some good beer (yay Milton), eat a lot and have some good conversations. This year also managed to get me a SheevaPlug so I could investigate #837989 - a bug with OpenOCD not being able to talk to the device. Turned out to be a channel configuration error in the move to new style FTDI support, so I’ve got that fixed locally and pushed the one line fix upstream as well.

Categories: LUG Community Blogs

Jonathan McDowell: Notes on upgrading from Jessie to Stretch

Tue, 22/08/2017 - 21:29

I upgraded my last major machine from Jessie to Stretch last week. That machine was the one running the most services, but I’d made notes while updating various others to ensure it went smoothly. Below are the things I noted along the way, both for my own reference and in case they are of use to anyone else.

  • Roundcube with the sqlite3 backend stopped working after the upgrade; fix was to edit /etc/roundcube/debian-db-roundcube.php and change sqlite3:// to sqlite:// in the $config['db_dsnw'] line.
  • Dovecot no longer supports SSLv2 so had to remove !SSLv2 from the ssl_protocols list in /etc/dovecot/conf.d/10-ssl.conf
  • Duplicity now tries to do a mkdir so I had to change from the scp:// backend to the sftp:// backend in my backup scripts.
  • Needed to add needs_root_rights=yes to /etc/X11/Xwrapper.config so Kodi systemd unit could still start it on a new VT. Need to figure out how to get this working without the need for root.
  • Upgrading fail2ban would have been easier if I’d dropped my additions in /etc/fail2ban/jail.d/ rather than the master config. Fixed for next time.
  • ejabberd continues to be a pain; I do wonder if it’s worth running an XMPP server these days. I certainly don’t end up using it to talk to people myself.
  • Upgrading 1200+ packages takes a long time, even when the majority of them don’t have any questions to ask during the process.
  • PostgreSQL upgrades have got so much easier. pg_upgradecluster 9.4 main chugged away but did exactly what I needed.

Other than those points things were pretty smooth. Nice work by all those involved!

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Building a USB descriptor table set

Tue, 22/08/2017 - 11:58

In order to proceed further on our USB/STM32 oddessy, we need to start to build a USB descriptor set for our first prototype piece of code. For this piece, we're going to put together a USB device which is vendor-specific class-wise and has a single configuration with a interface with a single endpoint which we're not going to actually implement anything of. What we're after is just to get the information presented to the computer so that lsusb can see it.

To get these built, let's refer to information we discovered and recorded in a previous post about how descriptors go together.

Device descriptor

Remembering that values which are > 1 byte in length are always stored little-endian, we can construct our device descriptor as:

Our device descriptor Field Name Value Bytes bLength 18 0x12 bDescriptorType DEVICE 0x01 bcdUSB USB 2.0 0x00 0x02 bDeviceClass 0 0x00 bDeviceSubClass 0 0x00 bDeviceProtocol 0 0x00 bMaxPacketSize 64 0x40 idVendor TEST 0xff 0xff idProduct TEST 0xff 0xff bcdDevice 0.0.1 0x01 0x00 iManufacturer 1 0x01 iProduct 2 0x02 iSerialNumber 3 0x03 bNumConfigurations 1 0x01

We're using the vendor ID and product id 0xffff because at this point we don't have any useful values for this (it costs $5,000 to register a vendor ID).

This gives us a final byte array of:

0x12 0x01 0x00 0x02 0x00 0x00 0x00 0x40 (Early descriptor)

0xff 0xff 0xff 0xff 0x01 0x00 0x01 0x02 0x03 0x01 (and the rest)

We're reserving string ids 1, 2, and 3, for the manufacturer string, product name string, and serial number string respectively. I'm deliberately including them all so that we can see it all come out later in lsusb.

If you feed the above hex sequence into a USB descriptor decoder then you can check my working.

Endpoint Descriptor

We want a single configuration, which covers our one interface, with one endpoint in it. Let's start with the endpoint...

Our bulk IN endpoint Field Name Value Bytes bLength 7 0x07 bDescriptorType ENDPOINT 0x05 bEndpointAddress EP2IN 0x82 bmAttributes BULK 0x02 wMaxPacketSize 64 0x40 0x00 bInterval IGNORED 0x00

We're giving a single bulk IN endpoint, since that's the simplest thing to describe at this time. This endpoint will never be ready and so nothing will ever be read into the host.

All that gives us:

0x07 0x05 0x82 0x02 0x40 0x00 0x00

Interface Descriptor

The interface descriptor prefaces the endpoint set, and thanks to our simple single endpoint, and no plans for alternate interfaces, we can construct the interface simply as:

Our single simple interface Field Name Value Bytes bLength 9 0x09 bDescriptorType INTERFACE 0x04 bInterfaceNumber 1 0x01 bAlternateSetting 1 0x01 bNumEndpoints 1 0x01 bInterfaceClass 0 0x00 bInterfaceSubClass 0 0x00 bInterfaceProtocol 0 0x00 iInterface 5 0x05

All that gives us:

0x09 0x04 0x01 0x01 0x01 0x00 0x00 0x00 0x05

Configuration descriptor

Finally we can put it all together and get the configuration descriptor...

Our sole configuration, encapsulating the interface and endpoint above Field Name Value Bytes bLength 9 0x09 bDescriptorType CONFIG 0x02 wTotalLength 9+9+7 0x19 0x00 bNumInterfaces 1 0x01 bConfigurationValue 1 0x01 iConfiguration 4 0x04 bmAttributes Bus powered, no wake 0x80 bMaxPower 500mA 0xfa

The wTotalLength field is interesting. It contains the configuration length, the interface length, and the endpoint length, hence 9 plus 9 plus 7 is 25.

This gives:

0x09 0x02 0x19 0x00 0x01 0x01 0x04 0x80 0xfa

String descriptors

We allowed ourselves a total of five strings, they were iManufacturer, iProduct, iSerial (from the device descriptor), iConfiguration (from the configuration descriptor), and iInterface (from the interface descriptor) respectively.

Our string descriptors will therefore be:

String descriptor zero, en_GB only Field Name Value Bytes bLength 4 0x04 bDescriptorType STRING 0x03 wLangID[0] en_GB 0x09 0x08

0x04 0x03 0x09 0x08


String descriptor one, iManufacturer Field Name Value Bytes bLength 38 0x26 bDescriptorType STRING 0x03 bString "Rusty Manufacturer" ...

0x26 0x03 0x52 0x00 0x75 0x00 0x73 0x00

0x74 0x00 0x79 0x00 0x20 0x00 0x4d 0x00

0x61 0x00 0x6e 0x00 0x75 0x00 0x66 0x00

0x61 0x00 0x63 0x00 0x74 0x00 0x75 0x00

0x72 0x00 0x65 0x00 0x72 0x00

(You get the idea, there's no point me breaking down the rest of the string descriptors here, suffice it to say that the other strings are appropriate for the values they represent - namely product, serial, configuration, and interface.)

Putting it all together

Given all the above, we have a device descriptor which is standalone, then a configuration descriptor which encompasses the interface and endpoint descriptors too. Finally we have a string descriptor table with six entries, the first is the language sets available, and the rest are our strings. In total we have:

// Device descriptor const DEV_DESC: [u8; 18] = { 0x12, 0x01, 0x00, 0x02, 0x00, 0x00, 0x00, 0x40, 0xff, 0xff, 0xff, 0xff, 0x01, 0x00, 0x01, 0x02, 0x03, 0x01 }; // Configuration descriptor const CONF_DESC: [u8; 25] = { 0x09, 0x02, 0x19, 0x00, 0x01, 0x01, 0x04, 0x80, 0xfa, 0x09, 0x04, 0x01, 0x01, 0x01, 0x00, 0x00, 0x00, 0x05, 0x07, 0x05, 0x82, 0x02, 0x40, 0x00, 0x00 }; // String Descriptor zero const STR_DESC_0: [u8; 4] = {0x04, 0x03, 0x09, 0x08}; // String Descriptor 1, "Rusty Manufacturer" const STR_DESC_1: [u8; 38] = { 0x26, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00, 0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x4d, 0x00, 0x61, 0x00, 0x6e, 0x00, 0x75, 0x00, 0x66, 0x00, 0x61, 0x00, 0x63, 0x00, 0x74, 0x00, 0x75, 0x00, 0x72, 0x00, 0x65, 0x00, 0x72, 0x00 }; // String Descriptor 2, "Rusty Product" const STR_DESC_2: [u8; 28] = { 0x1c, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00, 0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x50, 0x00, 0x72, 0x00, 0x6f, 0x00, 0x64, 0x00, 0x75, 0x00, 0x63, 0x00, 0x74, 0x00 }; // String Descriptor 3, "123ABC" const STR_DESC_3: [u8; 14] = { 0x0e, 0x03, 0x31, 0x00, 0x32, 0x00, 0x33, 0x00, 0x41, 0x00, 0x42, 0x00, 0x43, 0x00 }; // String Descriptor 4, "Rusty Configuration" const STR_DESC_4: [u8; 40] = { 0x28, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00, 0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x43, 0x00, 0x6f, 0x00, 0x6e, 0x00, 0x66, 0x00, 0x69, 0x00, 0x67, 0x00, 0x75, 0x00, 0x72, 0x00, 0x61, 0x00, 0x74, 0x00, 0x69, 0x00, 0x6f, 0x00, 0x6e, 0x00 }; // String Descriptor 5, "Rusty Interface" const STR_DESC_5: [u8; 32] = { 0x20, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00, 0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x49, 0x00, 0x6e, 0x00, 0x74, 0x00, 0x65, 0x00, 0x72, 0x00, 0x66, 0x00, 0x61, 0x00, 0x63, 0x00, 0x65, 0x00 };

With the above, we're a step closer to our first prototype which will hopefully be enumerable. Next time we'll look at beginning our prototype low level USB device stack mock-up.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): STM32 and RTFM

Sun, 06/08/2017 - 17:23

I have been working with STM32 chips on-and-off for at least eight, possibly closer to nine years. About as long as ST have been touting them around. I love the STM32, and have done much with them in C. But, as my previous two posts may have hinted, I would like to start working with Rust instead of C. To that end, I have been looking with great joy at the work which Jorge Aparicio has been doing around Cortex-M3 and Rust. I've had many comments in person at Debconf, and also several people mention on Twitter, that they're glad more people are looking at this. But before I can get too much deeper into trying to write my USB stack, I need to sort a few things from what Jorge has done as demonstration work.

Okay, this is fast, but we need Ludicrous speed

All of Jorge's examples seem to leave the system clocks in a fairly default state, excepting turning on the clocks to the peripherals needed during the initialisation phase. Sadly, if we're going to be running the USB at all, we need the clocks to run a tad faster. Since my goal is to run something moderately CPU intensive on the end of the USB too, it makes sense to try and get our STM32 running at maximum clock speed. For the one I have, that's 72MHz rather than the 8MHz it starts out with. Nine times more cycles to do computing in makes a lot of sense.

As I said above, I've been doing STM32 in C a lot for many years; and fortunately I have built systems with the exact chip that's on the blue-pill before. As such, if I rummage, I can find some old C code which does what we need...

/* Enable HSE */ RCC_HSEConfig(RCC_HSE_ON); /* Wait till HSE is ready */ HSEStartUpStatus = RCC_WaitForHSEStartUp(); if (HSEStartUpStatus == SUCCESS) { /* Enable Prefetch Buffer */ FLASH_PrefetchBufferCmd(FLASH_PrefetchBuffer_Enable); /* Flash 2 wait state */ FLASH_SetLatency(FLASH_Latency_2); /* HCLK = SYSCLK */ RCC_HCLKConfig(RCC_SYSCLK_Div1); /* PCLK2 = HCLK */ RCC_PCLK2Config(RCC_HCLK_Div1); /* PCLK1 = HCLK/2 */ RCC_PCLK1Config(RCC_HCLK_Div2); /* ADCCLK = PCLK2/6 */ RCC_ADCCLKConfig(RCC_PCLK2_Div6); /* PLLCLK = 8MHz * 9 = 72 MHz */ RCC_PLLConfig(RCC_PLLSource_HSE_Div1, RCC_PLLMul_9); /* Enable PLL */ RCC_PLLCmd(ENABLE); /* Wait till PLL is ready */ while (RCC_GetFlagStatus(RCC_FLAG_PLLRDY) == RESET) {} /* Select PLL as system clock source */ RCC_SYSCLKConfig(RCC_SYSCLKSource_PLLCLK); /* Wait till PLL is used as system clock source */ while (RCC_GetSYSCLKSource() != 0x08) {} }

This code, rather conveniently, uses an 8MHz external crystal so we can almost direct-port it to the blue-pill Rust code and see how we go. If you're used to the CMSIS libraries for STM32, then you won't completely recognise the above since it uses the pre-CMSIS core libraries to do its thing. Library code from 2008 and it's still good on today's STM32s providing they're in the right family :-)

A direct conversion to Rust, using Jorge's beautifully easy to work with crates made from svd2rust results in:

fn make_go_faster(rcc: &RCC, flash: &FLASH) {|_, w| w.hseon().enabled()); while ! {} flash.acr.modify(|_, w| w.prftbe().enabled()); flash.acr.modify(|_, w| w.latency().two()); rcc.cfgr.modify(|_, w| w .hpre().div1() .ppre2().div1() .ppre1().div2() // .adcpre().bits(8) .pllsrc().external() .pllxtpre().div1() .pllmul().mul9() );|_, w| w.pllon().enabled()); while {} rcc.cfgr.modify(|_,w| w.sw().pll()); while ! {} }

Now, I've not put the comments in which were in the C code, because I'm being very lazy right now, but if you follow the two together you should be able to work it through. I don't have timeouts for the waits, and you'll notice a single comment there (I cannot set up the ADC prescaler because for some reason the SVD is missing any useful information and so the generated crate only carries an unsafe function (bits()) and I'm trying to steer clear of unsafe for now. Still, I don't need the ADC immediately, so I'm okay with this.

By using this function in the beginning of the init() function of the blinky example, I can easily demonstrate the clock is going faster since the LED blinks more quickly.

This function demonstrates just how simple it is to take bit-manipulation from the C code and turn it into (admittedly bad looking) Rust with relative ease and without any of the actual bit-twiddling. I love it.

Mess with time, and you get unexpected consequences

Sadly, when you mess with the clock tree on a microcontroller, you throw a lot of things out of whack. Not least, by adjusting the clock frequency up we end up adjusting the AHB, APB1, and APB2 clock frequencies. This has direct consequences for peripherals floating around on those busses. Fortunately Jorge thought of this and while the blue-pill crate hard-wires those frequencies to 8MHz, they are, at least, configurable in code in some sense.

If we apply the make_go_faster() function to the serial loopback example, it simply fails to work because now the bus which the USART1 peripheral is connected to (APB2) is going at a different speed from the expected power-on default of 8MHz. If you remember from the function, we did .hpre().div1() which set HCLK to 72MHz, then .ppre1().div2() which sets the APB1 bus clock to be HCLK divided by 2, and .ppre2().div1() which sets APB2 bus clock to be HCLK. This means that we'd need to alter src/ to reflect these changes in the clock frequences and in theory loopback would start working once more.

It'd be awkward to try and demonstrate all that to you since I only have a phone camera to hand, but if you own a blue-pill then you can clone Jorge's repo and have a go yourself and see that I'm not bluffing you.

With all this done, it'll be time to see if we can bring the USB peripheral in the STM32 online, and that will be the topic of my next post in this discovery series.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): USB Device Stacks, on RTFM, part 2

Sat, 05/08/2017 - 17:08

Previously we talked about all the different kinds of descriptors which USB devices use to communicate their capability. This is important stuff because to write any useful USB device firmware we need to be able to determine how to populate our descriptors. However, having that data on the device is entirely worthless without an understanding of how it gets from the device to the host so that it can be acted upon. To understand that, let's look at the USB wire protocol.

Note, I'll again be talking mostly about USB2.0 low- and full-speed. I believe that high speed is approximately the same but with faster wires, except not quite that simple.

Down to the wire

I don't intend to talk about the actual electrical signalling, though it's not un-reasonable for you to know that USB is a pair of wires forming a differentially signalled bidirectional serial communications link. The host is responsible for managing all the framing and timing on the link, and for formatting the communications into packets.

There are a number of packet types which can appear on the USB link:

Packet type Purpose Token Packet When the host wishes to send a message to the Control endpoint to configure the device, read data IN, or write data OUT, it uses this to start the transaction. Data(0/1) Packet Following a Setup, In, or Out token, a Data packet is a transfer of data (in either direction). The 0 and 1 alternate to provide a measure of confidence against lost packets. Handshake Packet Following a data packet of some kind, the other end may ACK the packet (all was well), NAK the packet (report that the device cannot, temporarily, send/receive data, or that an interrupt endpoint isn't triggered), or STALL the bus in which case the host needs to intervene. Start of Frame Every 1ms (full-speed) the host will send a SOF packet which carries a frame number. This can be used to help keep time on very simple devices. It also divides the bus into frames within which bandwidth is allocated.

As an example, when the host wishes to perform a control transfer, the following packets are transacted in turn:

  1. Setup Token - The host addresses the device and endpoint (OUT0)
  2. Data0 Packet - The host transmits a GET_DESCRIPTOR for the device descriptor
  3. Ack Packet - The device acknowledges receipt of the request

This marks the end of the first transaction. The device decodes the GET_DESCRIPTOR request and prepares the device descriptor for transmission. The transmission occurs as the next transaction on the bus. In this example, we're assuming 8 byte maximum transmission sizes, for illustrative purposes.

  1. In Token - The host addresses the device and endpoint (IN0)
  2. Data1 Packet - The device transmits the first 8 bytes of the descriptor
  3. Ack Packet - The host acknowledges the data packet
  4. In Token - The host addresses the device and endpoint (IN0)
  5. Data0 Packet - The device transmits the remaining 4 bytes of the descriptor (padded)
  6. Ack Packet - The host acknowledges the data packet

The second transaction is now complete, and the host has all the data it needs to proceed. Finally a status transaction occurs in which:

  1. Out Token - The host addresses the device and endpoint (OUT0)
  2. Data1 Packet - The host transmits a 0 byte data packet to indicate successful completion
  3. Ack Packet - The device acknowledges the completion, indicating its own satisfaction

And thus ends the full control transaction in which the host retrieves the device descriptor.

From a high level, we need only consider the activity which occurs at the point of the acknowledgement packets. In the above example:

  1. On the first ACK the device prepares IN0 to transmit the descriptor, readying whatever low level device stack there is with a pointer to the descriptor and its length in bytes.
  2. On the second ACK the low levels are still thinking.
  3. On the third ACK the transmission from IN0 is complete and the endpoint no longer expects to transfer data.
  4. On the fourth ACK the control transaction is entirely complete.
Thinking at the low levels of the control interface

Before we can build a high level USB stack, we need to consider the activity which might occur at the lower levels. At the low levels, particularly of the device control interface, work has to be done at each and every packet. The hardware likely deals with the token packet for us, leaving the data packets for us to process, and the resultant handshake packets will be likely handled by the hardware in response to our processing the data packets.

Since every control transaction is initiated by a setup token, let's look at the setup requests which can come our way...

Setup Packet (Data) Format Field Name Byte start Byte length Encoding Meaning bmRequestType 0 1 Bitmap Describes the kind of request, and the target of it. See below. bRequest 1 1 Code The request code itself, meanings of the rest of the fields vary by bRequest wValue 2 2 Number A 16 bit value whose meaning varies by request type wIndex 4 2 Number A 16 bit value whose meaning varies by request type but typically encodes an interface number or endpoint. wLength 6 2 Number A 16 bit value indicating the length of the transfer to come.

Since bRequest is essentially a switch against which multiple kinds of setup packet are selected between, here's the meanings of a few...

GET_DESCRIPTOR (Device) setup packet Field Name Value Meaning bmRequestType 0x08 Data direction is IN (from device to host), recipient is the device bRequest 0x06 GET_DESCRIPTOR (in this instance, the device descriptor is requested) wValue 0x0001 This means the device descriptor wIndex 0x0000 Irrelevant, there's only 1 device descriptor anyway wLength 12 This is the length of a device descriptor (12 bytes) SET_ADDRESS to set a device's USB address Field Name Value Meaning bmRequestType 0x00 Data direction is OUT (from host to device), recipient is the device bRequest 0x05 SET_ADDRESS (Set the device's USB address) wValue 0x00nn The address for the device to adopt (max 127) wIndex 0x0000 Irrelevant for address setting wLength 0 There's no data transfer expected for this setup operation

Most hardware blocks will implement an interrupt at the point that the Data packet following the Setup packet has been receive. This is typically called receiving a 'Setup' packet and then it's up to the device stack low levels to determine what to do and dispatch a handler. Otherwise an interrupt will fire for the IN or OUT tokens and if the endpoint is zero, the low level stack will handle it once more.

One final thing worth noting about SET_ADDRESS is that it doesn't take effect until the completion of the zero-length "status" transaction following the setup transaction. As such, the status request from the host will still be sent to address zero (the default for new devices).

A very basic early "packet trace"

This is an example, and is not guaranteed to be the packet sequence in all cases. It's a good indication of the relative complexity involved in getting a fresh USB device onto the bus though...

When a device first attaches to the bus, the bus is in RESET state and so the first event a device sees is a RESET which causes it to set its address to zero, clear any endpoints, clear the configuration, and become ready for control transfers. Shortly after this, the device will become suspended.

Next, the host kicks in and sends a port reset of around 30ms. After this, the host is ready to interrogate the device.

The host sends a GET_DESCRIPTOR to the device, whose address at this point is zero. Using the information it receives from this, it can set up the host-side memory buffers since the device descriptor contains the maximum transfer size which the device supports.

The host is now ready to actually 'address' the device, and so it sends another reset to the device, again around 30ms in length.

The host sends a SET_ADDRESS control request to the device, telling it that its new address is nn. Once the acknowledgement has been sent from the host for the zero-data status update from the device, the device sets its internal address to the value supplied in the request. From now on, the device shall respond only to requests to nn rather than to zero.

At this point, the host will begin interrogating further descriptors, looking at the configuration descriptors and the strings, to build its host-side representation of the device. These will be GET_DESCRIPTOR and GET_STRING_DESCRIPTOR requests and may continue for some time.

Once the host has satisfied itself that it knows everything it needs to about the device, it will issue a SET_CONFIGURATION request which basically starts everything up in the device. Once the configuration is set, interrupt endpoints will be polled, bulk traffic will be transferred, Isochronous streams begin to run, etc.

Okay, but how do we make this concrete?

So far, everything we've spoken about has been fairly abstract, or at least "soft". But to transfer data over USB does require some hardware. (Okay, okay, we could do it all virtualised, but there's no fun in that). The hardware I'm going to be using for the duration of this series is the STM32 on the blue-pill development board. This is a very simple development board which does (in theory at least) support USB device mode.

If we view the schematic for the blue-pill, we can see a very "lightweight" USB interface which has a pullup resistor for D+. This is the way that a device signals to the host that it is present, and that it wants to speak at full-speed. If the pullup were on D- then it would be a low-speed device. High speed devices need a little more complexity which I'm not going to go into for today.

The USB lines connect to pins PA11 and PA12 which are the USB pins on the STM32 on the board. Since USB is quite finicky, the STM32 doesn't let you remap that function elsewhere, so this is all looking quite good for us so far.

The specific STM32 on the blue-pill is the STM32F103C8T6. By viewing its product page on ST's website we can find the reference manual for the part. Jumping to section 23 we learn that this STM32 supports full-speed USB2.0 which is convenient given the past article and a half. We also learn it supports up to eight endpoints active at any one time, and offers double-buffering for our bulk and isochronous transfers. It has some internal memory for packet buffering, so it won't use our RAM bandwidth while performing transfers, which is lovely.

I'm not going to distill the rest of that section here, because there's a large amount of data which explains how the USB macrocell operates. However useful things to note are:

  • How IN OUT and SETUP transfers work.
  • How the endpoint buffer memory is configured.
  • That all bus-powered devices MUST respond to suspend/resume properly
  • That the hardware will prioritise endpoint interrupts for us so that we only need deal with the most pressing item at any given time.
  • There is an 'Enable Function' bit in the address register which must be set or we won't see any transactions at all.
  • How the endpoint registers signal events to the device firmware.

Next time, we're going to begin the process of writing a very hacky setup routine to try and initialise the USB device macrocell so that we can see incoming transactions through the ITM. It should be quite exciting, but given how complex this will be for me to learn, it might be a little while before it comes through.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): USB Device Stacks, on RTFM

Fri, 04/08/2017 - 17:05

I have been spending time with Jorge Aparicio's RTFM for Cortex M3 framework for writing Rust to target Cortex-M3 devices from Arm (and particularly the STM32F103 from ST Microelectronics). Jorge's work in this area has been of interest to me ever since I discovered him working on this stuff a while ago. I am very tempted by the idea of being able to implement code for the STM32 with the guarantees of Rust and the language features which I have come to love such as the trait system.

I have been thinking to myself that, while I admire and appreciate the work done on the GNUK, I would like to, personally, have a go at implementing some kind of security token on an STM32 as a USB device. And with the advent of the RTFM for M3 work, and Jorge's magical tooling to make it easier to access and control the registers on an M3 microcontroller, I figured it'd be super-nice to do this in Rust, with all the advantages that entails in terms of isolating unsafe behaviour and generally having the potential to be more easily verified as not misbehaving.

To do this though, means that I need a USB device stack which will work in the RTFM framework. Sadly it seems that, thus-far, only Jorge has been working on drivers for any of the M3 devices his framework supports. And one person can only do so much. So, in my infinite madness, I decided I should investigate the complexity of writing a USB device stack in Rust for the RTFM/M3 framework. (Why I thought this was a good idea is lost to the mists of late night Googling, but hey, it might make a good talk at the next conference I go to). As such, this blog post, and further ones along these lines, will serve as a partial tour of what I'm up to, and a partial aide-memoir for me about learning USB. If I get something horribly wrong, please DO contact me to correct me, otherwise I'll just continue to be wrong. If I've simplified something but it's still strictly correct, just let me know if it's an oversimplification since in a lot of cases there's no point in me putting the full details into a blog posting. I will mostly be considering USB2.0 protocol details but only really for low and full speed devices. (The hardware I'm targetting does low-speed and full-speed, but not high-speed. Though some similar HW does high-speed too, I don't have any to hand right now)

A brief introduction to USB

In order to go much further, I needed a grounding in USB. It's a multi-layer protocol as you might expect, though we can probably ignore the actual electrical layer since any device we might hope to support will have to have a hardware block to deal with that. We will however need to consider the packet layer (since that will inform how the hardware block is implemented and thus its interface) and then the higher level protocols on top.

USB is a deliberately asymmetric protocol. Devices are meant to be significantly easier to implement, both in terms of hardware and software, as compared with hosts. As such, despite some STM32s having OTG ports, I have no intention of supporting host mode at this time.

USB is arranged into a set of busses which are, at least in the USB1.1 case, broadcast domains. As such, each device has an address assigned to it by the host during an early phase called 'configuration'. Once the address is assigned, the device is expected to only ever respond to messages addressed to it. Note that since everything is asymmetric in USB, the device can't send messages on its own, but has to be asked for them by the host, and as such the addressing is always from host toward device.

USB devices then expose a number of endpoints through which communication can flow IN to the host or OUT to the device. Endpoints are not bidirectional, but the in and out endpoints do overlap in numbering. There is a special pair of endpoints, IN0 and OUT0 which, between them, form what I will call the device control endpoints. The device control endpoints are important since every USB device MUST implement them, and there are a number of well defined messages which pass over them to control the USB device. In theory a bare minimum USB device would implement only the device control endpoints.

Configurations, and Classes, and Interfaces, Oh My!

In order for the host to understand what the USB device is, and what it is capable of, part of the device control endpoints' responsibility is to provide a set of descriptors which describe the device. These descriptors form a heirarchy and are then glommed together into a big lump of data which the host can download from the device in order to decide what it is and how to use it. Because of various historical reasons, where a multi-byte value is used, they are defined to be little-endian, though there are some BCD fields. Descriptors always start with a length byte and a type byte because that way the host can parse/skip as necessary, with ease.

The first descriptor is the device descriptor, is a big one, and looks like this:

Device Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (18) bDescriptorType 1 1 Constant Device Descriptor (0x01) bcdUSB 2 2 BCD USB spec version compiled with bDeviceClass 4 1 Class Code, assigned by USB org (0 means "Look at interface descriptors", common value is 2 for CDC) bDeviceSubClass 5 1 SubClass Code, assigned by USB org (usually 0) bDeviceProtocol 6 1 Protocol Code, assigned by USB org (usually 0) bMaxPacketSize 7 1 Number Max packet size for IN0/OUT0 (Valid are 8, 16, 32, 64) idVendor 8 2 ID 16bit Vendor ID (Assigned by USB org) idProduct 10 2 ID 16bit Product ID (Assigned by manufacturer) bcdDevice 12 2 BCD Device version number (same encoding as bcdUSB) iManufacturer 14 1 Index String index of manufacturer name (0 if unavailable) iProduct 15 1 Index String index of product name (0 if unavailable) iSerialNumber 16 1 Index String index of device serial number (0 if unavailable) bNumConfigurations 17 1 Number Count of configurations the device has.

This looks quite complex, but breaks down into a relatively simple two halves. The first eight bytes carries everything necessary for the host to be able to configure itself and the device control endpoints properly in order to communicate effectively. Since eight bytes is the bare minimum a device must be able to transmit in one go, the host can guarantee to get those, and they tell it what kind of device it is, what USB protocol it supports, and what the maximum transfer size is for its device control endpoints.

The encoding of the bcdUSB and bcdDevice fields is interesting too. It is of the form 0xMMmm where MM is the major number, mm the minor. So USB2.0 is encoded as 0x0200, USB1.1 as 0x0110 etc. If the device version is 17.36 then that'd be 0x1736.

Other fields of note are bDeviceClass which can be 0 meaning that interfaces will specify their classes, and idVendor/idProduct which between them form the primary way for the specific USB device to be identified. The Index fields are indices into a string table which we'll look at later. For now it's enough to know that wherever a string index is needed, 0 can be provided to mean "no string here".

The last field is bNumConfigurations and this indicates the number of ways in which this device might function. A USB device can provide any number of these configurations, though typically only one is provided. If the host wishes to switch between configurations then it will have to effectively entirely quiesce and reset the device.

The next kind of descriptor is the configuration descriptor. This one is much shorter, but starts with the same two fields:

Configuration Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (9) bDescriptorType 1 1 Constant Configuration Descriptor (0x02) wTotalLength 2 2 Number Size of the configuration in bytes, in total bNumInterfaces 4 1 Number The number of interfaces in this configuration bConfigurationValue 5 1 Number The value to use to select this configuration iConfiguration 6 1 Index The name of this configuration (0 for unavailable) bmAttributes 7 1 Bitmap Attributes field (see below) bMaxPower 8 1 Number Maximum bus power this configuration will draw (in 2mA increments)

An important field to consider here is the bmAttributes field which tells the host some useful information. Bit 7 must be set, bit 6 is set if the device would be self-powered in this configuration, bit 5 indicates that the device would like to be able to wake the host from sleep mode, and bits 4 to 0 must be unset.

The bMaxPower field is interesting because it encodes the power draw of the device (when set to this configuration). USB allows for up to 100mA of draw per device when it isn't yet configured, and up to 500mA when configured. The value may be used to decide if it's sensible to configure a device if the host is in a low power situation. Typically this field will be set to 50 to indicate the nominal 100mA is fine, or 250 to request the full 500mA.

Finally, the wTotalLength field is interesting because it tells the host the total length of this configuration, including all the interface and endpoint descriptors which make it up. With this field, the host can allocate enough RAM to fetch the entire configuration descriptor block at once, simplifying matters dramatically for it.

Each configuration has one or more interfaces. The interfaces group some endpoints together into a logical function. For example a configuration for a multifunction scanner/fax/printer might have an interface for the scanner function, one for the fax, and one for the printer. Endpoints are not shared among interfaces, so when building this table, be careful.

Next, logically, come the interface descriptors:

Interface Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (9) bDescriptorType 1 1 Constant Interface Descriptor (0x04) bInterfaceNumber 2 1 Number The number of the interface bAlternateSetting 3 1 Number The interface alternate index bNumEndpoints 4 1 Number The number of endpoints in this interface bInterfaceClass 5 1 Class The interface class (USB Org defined) bInterfaceSubClass 6 1 SubClass The interface subclass (USB Org defined) bInterfaceProtocol 7 1 Protocol The interface protocol (USB Org defined) iInterface 8 1 Index The name of the interface (or 0 if not provided)

The important values here are the class/subclass/protocol fields which provide a lot of information to the host about what the interface is. If the class is a USB Org defined one (e.g. 0x02 for Communications Device Class) then the host may already have drivers designed to work with the interface meaning that the device manufacturer doesn't have to provide host drivers.

The bInterfaceNumber is used by the host to indicate this interface when sending messages, and the bAlternateSetting is a way to vary interfaces. Two interfaces with the came bInterfaceNumber but different bAlternateSettings can be switched between (like configurations, but) without resetting the device.

Hopefully the rest of this descriptor is self-evident by now.

The next descriptor kind is endpoint descriptors:

Endpoint Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (7) bDescriptorType 1 1 Constant Endpoint Descriptor (0x05) bEndpointAddress 2 1 Endpoint Endpoint address (see below) bmAttributes 3 1 Bitmap Endpoint attributes (see below) wMaxPacketSize 4 2 Number Maximum packet size this endpoint can send/receive bInterval 6 1 Number Interval for polling endpoint (in frames)

The bEndpointAddress is a 4 bit endpoint number (so there're 16 endpoint indices) and a bit to indicate IN vs. OUT. Bit 7 is the direction marker and bits 3 to 0 are the endpoint number. This means there are 32 endpoints in total, 16 in each direction, 2 of which are reserved (IN0 and OUT0) giving 30 endpoints available for interfaces to use in any given configuration. The bmAttributes bitmap covers the transfer type of the endpoint (more below), and the bInterval is an interval measured in frames (1ms for low or full speed, 125µs in high speed). bInterval is only valid for some endpoint types.

The final descriptor kind is for the strings which we've seen indices for throughout the above. String descriptors have two forms:

String Descriptor (index zero) Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (variable) bDescriptorType 1 1 Constant String Descriptor (0x03) wLangID[0] 2 2 Number Language code zero (e.g. 0x0409 for en_US) wLangID[n] 4.. 2 Number Language code n ...

This form (for descriptor 0) is that of a series of language IDs supported by the device. The device may support any number of languages. When the host requests a string descriptor, it will supply both the index of the string and also the language id it desires (from the list available in string descriptor zero). The host can tell how many language IDs are available simply by dividing bLength by 2 and subtracting 1 for the two header bytes.

And for string descriptors of an index greater than zero:

String Descriptor (index greater than zero) Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (variable) bDescriptorType 1 1 Constant String Descriptor (0x03) bString 2.. .. Unicode The string, in "unicode" format

This second form of the string descriptor is simply the the string is in what the USB spec calls 'Unicode' format which is, as of 2005, defined to be UTF16-LE without a BOM or terminator.

Since string descriptors are of a variable length, the host must request strings in two transactions. First a request for 2 bytes is sent, retrieving the bLength and bDescriptorType fields which can be checked and memory allocated. Then a request for bLength bytes can be sent to retrieve the entire string descriptor.

Putting that all together

Phew, this is getting to be quite a long posting, so I'm going to leave this here and in my next post I'll talk about how the host and device pass packets to get all that information to the host, and how it gets used.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Gitano 1.1

Thu, 03/08/2017 - 17:34

Today marks the release of Gitano 1.1. Richard(s) and I have spent quite a lot of time and effort on this release, and there's plenty of good stuff in it. We also released new versions of Lace, Supple, Luxio, and Gall to go alongside it, with bugfixes and improvements.

At this point, I intend to take a short break from Gitano to investigate some Rust-on-STM32 stuff, and then perhaps do some NetSurf work too.

Categories: LUG Community Blogs

Mick Morgan: a letter to our dear home secretary

Wed, 02/08/2017 - 15:56

Dear Amber

So,”real people” don’t care about privacy? All they really want is ease of use and a pretty GUI so that they can chat to all their friends on-line? Only “the enemy” (who is that exactly anyway?) needs encryption? Excuse me for asking, but what have you been smoking? Does the Home Office know about that?

I’m a real person. And I care deeply about privacy. I care enough to fund both my own Tor node and various openVPN servers dotted around the world just to get past your ludicrous attempts at gratuitous surveillance of my (and my family’s) routine use of the ‘net. I care about the security and privacy of my transactions with various commercial enterprises, including my bank (which is why I expect them to use TLS on their website). I care about privacy when I correspond with my Doctor and other professionals. I care about privacy when I use an on-line search engine (which, incidentally, is not Google). I care about privacy because privacy matters. I have the right to freedom of thought and expression. I have the right to discuss those thoughts with others of my choice – when I choose and how I choose. You may not like that, but it’s a fact of life. That doesn’t make me “the enemy”. Get over it.

Love and Kisses


(Note to readers: Aral Balkan has deconstructed Rudd’s ramblings. I commend the article to you.)

Categories: LUG Community Blogs