Planet ALUG

Syndicate content
Planet ALUG - http://planet.alug.org.uk/
Updated: 1 hour 29 min ago

Chris Lamb: How to write your first Lintian check

Mon, 05/09/2016 - 16:33

Lintian's humble description of "Debian package checker" belies its importance within the Debian GNU/Linux project. An extensive static analysis tool, it's not only used by the vast majority of developers, falling foul of some of its checks even cause uploads to be automatically rejected by the archive maintenance software.

As you may have read in my recent monthly report, I've recently been hacking on Lintian itself. In particular:

  • #798983: Check for libjs-* binary package name outside of the web section
  • #814326: Warn if filenames contain wildcard characters
  • #829744: Add new-package-should-not-package-python2-module tag
  • #831864: Warn about Python packages that ship Coverage.py information
  • #832096 Check for common typos in debian/rules target names
  • #832099: Check for unnecessary SOURCE_DATE_EPOCH assignments
  • #832771: Warn about systemd .service files with a missing Install key

However, this rest of this post will go through the steps needed to start contributing yourself.

To demonstrate this I will be walking through submitting a patch for bug #831864 which warns about Python packages that ship .coverage files generated by Coverage.py.

Getting started

First, let's obtain the Lintian sources and create a branch for our work:

$ git clone https://anonscm.debian.org/git/lintian/lintian.git […] $ cd lintian $ git checkout -b warn-about-dotcoverage-files Switched to a new branch 'warn-about-dotcoverage-files'

The most interesting files are under checks/*:

$ ls -l checks/ | head -n 9 total 1356 -rw-r--r-- 1 lamby lamby 6393 Jul 29 14:19 apache2.desc -rw-r--r-- 1 lamby lamby 8619 Jul 29 14:19 apache2.pm -rw-r--r-- 1 lamby lamby 1956 Jul 29 14:19 application-not-library.desc -rw-r--r-- 1 lamby lamby 3285 Jul 29 14:19 application-not-library.pm -rw-r--r-- 1 lamby lamby 544 Jul 29 14:19 automake.desc -rw-r--r-- 1 lamby lamby 1354 Jul 29 14:19 automake.pm -rw-r--r-- 1 lamby lamby 19506 Jul 29 14:19 binaries.desc -rw-r--r-- 1 lamby lamby 25204 Jul 29 14:19 binaries.pm -rw-r--r-- 1 lamby lamby 15641 Aug 24 21:42 changelog-file.desc -rw-r--r-- 1 lamby lamby 19606 Jul 29 14:19 changelog-file.pm

Note that the files are in pairs; a foo.desc file that contains description of the tags and a sibling foo.pm Perl module that actually performs the checks.


Let's add our new tag before we go any further. After poking around, it looks like files.{pm,desc} would be most appropriate, so we'll add our new tag definition to files.desc:

Tag: package-contains-python-coverage-file Severity: normal Certainty: certain Info: The package contains a file that looks like output from the Python coverage.py tool. These are generated by python{,3}-coverage during a test run, noting which parts of the code have been executed. They can then be subsequently analyzed to identify code that could have been executed but was not. . As they are are unlikely to be of utility to end-users, these files should be removed from the package.

The Severity and Certainty fields are documented in the manual. Note the convention of using double spaces after full stops in the Info section.

Extending the testsuite

Lintian has many moving parts based on regular expressions and other subtle logic, so it's especially important to provide tests in order to handle edge cases and to catch any regressions in the future.

We create tests by combining a tiny Debian package that will deliberately violate our check, along with some metadata and the expected output of running Lintian against this package.

The tests themselves are stored under t/tests. There may be an existing test that it would be more appropriate to extend, but I've gone with creating a new directory called files-python-coverage:

$ mkdir -p t/tests/files-python-coverage $ cd t/tests/files-python-coverage

First, we create a simple package, installing dummy file to trigger the check:

$ mkdir -p debian/debian $ touch debian/.coverage $ echo ".coverage /usr/share/files-python-coverage" > debian/debian/install

Note that we do not need a debian/rules file as long as we do not deviate from a "skeleton" debhelper style. We then add the aforementioned metadata to t/tests/files-python-coverage/desc:

Testname: files-python-coverage Sequence: 6000 Version: 1.0 Description: Check for Python .coverage files Test-For: package-contains-python-coverage-file

… and the expected warning to t/tests/files-python-coverage/tags:

$ echo "W: files-python-coverage: package-contains-python-coverage-file" \ "usr/share/files-python-coverage/.coverage" > tags

When we run the testsuite, it should fail because we don't emit the check yet:

$ cd $(git rev-parse --show-toplevel) $ debian/rules runtests onlyrun=tag:package-contains-python-coverage-file […] --- t/tests/files-python-coverage/tags +++ debian/test-out/tests/files-python-coverage/tags.files-python-coverage @@ -1 +0,0 @@ -W: files-python-coverage: package-contains-python-coverage-file usr/share/files-python-coverage/.coverage fail tests::files-python-coverage: output differs! Failed tests (1) tests::files-python-coverage debian/rules:48: recipe for target 'runtests' failed make: *** [runtests] Error 1 $ echo $? 1

Specifying onlyrun= means we only run the tests that are designed to trigger this tag rather than the whole testsuite. This is controlled by the Test-For key in our desc file, not by scanning the tags files.

This recipe for creating a testcase could be used when submitting a regular bug against Lintian — providing a failing testcase not only clarifies misunderstandings resulting from the use of natural language, it also makes it easier, quicker and safer to correct the offending code itself.

Emitting the tag

Now, let's actually implement the check:

tag 'package-installs-python-egg', $file; } + # ---------------- .coverage (coverage.py output) + if ($file->basename eq ".coverage") { + tag 'package-contains-python-coverage-file', $file; + } # ---------------- /usr/lib/site-python

Our testsuite now passes:

$ debian/rules runtests onlyrun=tag:package-contains-python-coverage-file private/generate-profiles.pl .... running tests .... mkdir -p "debian/test-out" t/runtests -k -j 9 t "debian/test-out" tag:package-contains-python-coverage-file ENV[PATH]=[..] pass tests::files-python-coverage if [ "tag:package-contains-python-coverage-file" = "" ]; then touch runtests; fi $ echo $? 0 Submitting the patch

Lastly, we create a patch for submission to the bug tracking system:

$ git commit -a -m "c/files: Warn about Python packages which ship" \ "coverage.py information. (Closes: #831864)" $ git format-patch HEAD~ 0001-c-files-Warn-about-Python-packages-which-ship-covera.patch

… and we finally attach it to the existing bug:

To: 831864@bugs.debian.org Cc: 831864-submitter@bugs.debian.org Bcc: control@bugs.debian.org tags 831864 + patch thanks Patch attached. /lamby
Summary

I hope this post will encourage at some extra contributions towards this important tool.

(Be aware that I'm not a Lintian maintainer, so not only should you not treat anything here as gospel and expect this post may be edited over time if clarifications arise.)

Categories: LUG Community Blogs

Chris Lamb: Free software activities in August 2016

Wed, 31/08/2016 - 22:48

Here is my monthly update covering what I have been doing in the free software world (previously):

  • Worked on nsntrace, a userspace tool to perform network traces on processes using kernel namespaces:
    • Overhauled error handling to ensure the return code of the wrapped process is returned to the surrounding environment. (#10).
    • Permit the -u argument to also accept uids as well as usernames. (#16).
    • Always kill the (hard-looping) udp_send utility, even on test failures. (#13).
    • Updated configure.ac to look for iptables in /sbin & /usr/sbin (#11) and to raise an error if pcap.h is missing (#15).
    • Drop bashisms in #!/bin/sh script (#14) and ignore the generated manpage in the Git repository (#12).
  • Independently discovered an regression in the Django web development framework where field__isnull=False filters were not working with some foreign keys, resulting in extending the testsuite and release documentation. (#7104).
  • Proposed a change to django-enumfield (a custom field for type-safe constants) to ensure passing a string type to Enum.get returned None on error to match the documentation. (#36).
  • Fixed an issue in the Mopidy music player's podcast extension where the testsuite was failing tests in extreme timezones. (#40).
  • Proposed changes to make various upstream's reproducible:
    • botan, a crypto/TLS library for C++11. (#587).
    • cookiecutter, a project template generator, removing nondeterministic keyword arguments from appearing in the documentation. (#800).
    • pyicu, a Python wraper for the IBM Unicode library. (#27).
  • Integrated a number of issues raised by @piotr1212 to python-fadvise, my Python interface to posix_fadvise(2), where the API was not being applied to open file descriptors (#1) and moving the .so to a module directory (#2).
  • Various improvements to try.diffoscope.org, a hosted version of the diffoscope in-depth and content-aware diff utility, including introducing an HTTP API (#21), updating the SSL certificate and correcting a logic issue where errors in diffoscope itself were not being detected correctly (b0ff49). Continued thanks to Bytemark for sponsoring the hardware.
  • Fixed a bug in django-slack, my library to easily post messages to the Slack group-messaging utility, correcting an EncodeError exception under Python 3 (#53) and updated the minimum required version of Django to 1.7 (#54).
  • Various updates to tickle-me-email, my Getting Things Done-inspired email toolbox, to also match / in IMAP's LIST separators (#6) and to encode the folder list as UTF-7 (#7). Thanks to @resiak.
  • Clarified the documentation for travis.debian.net — my hosted script to easily test and build Debian packages on the Travis CI continuous integration platform — regarding how to integrate with Github (#20).

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most Linux distributions provide binary (or "compiled") packages to end users.

The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced — either maliciously and accidentally — during this compilation process by promising identical binary packages are always generated from a given source.


Toolchain issues

I submitted the following patches to fix reproducibility-related toolchain issues:


My work in the Reproducible Builds project was also covered in our weekly reports. (#67, #68, #69, #70).


Diffoscope

diffoscope is our "diff on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them:

  • Added a command-line interface to the try.diffoscope.org web service.
  • Added a JSON comparator.
  • In the HTML output, highlight lines when hovering to make it easier to visually track.
  • Ensure that we pass str types to our Difference class, otherwise we can't be sure we can render them later.
  • Testsuite improvements:
    • Generate test coverage reports.
    • Add tests for Haskell and GitIndex comparators.
    • Completely refactored all of the comparator tests, extracting out commonly-used routines.
    • Confirm rendering of text and HTML presenters when checking non-existing files.
    • Dropped a squashfs test as it was simply too unreliable and/or has too many requirements to satisfy.
  • A large number of miscellaneous cleanups, including:
    • Reworking the comparator setup/preference internals by dynamically importing classes via a single list.
    • Split exceptions out into dedicated diffoscope.exc module.
    • Tidying the PROVIDERS dict in diffoscope/__init__.py.
    • Use html.escape over xml.sax.saxutils.escape, cgi.escape, etc.
    • Removing hard-coding of manual page targets names in debian/rules.
    • Specify all string format arguments as logging function parameters, not using interpolation.
    • Tidying imports, correcting indentation levels and drop unnecessary whitespace.

disorderfs

disorderfs is our FUSE filesystem that deliberately introduces nondeterminism in system calls such as readdir(3).

  • Added a testsuite to prevent regressions. (f124965)
  • Added a --sort-dirents=yes|no option for forcing deterministic ordering. (2aae325)

Other
  • Improved strip-nondeterminism, our tool to remove specific nondeterministic information after a build:
    • Match more styles of Java .properties files.
    • Remove hyphen from "non-determinism" and "non-deterministic" throughout package for consistency.
  • Improvements to our testing infrastucture:
    • Improve the top-level navigation so that we can always get back to "home" of a package.
    • Give expandable elements cursor: pointer CSS styling to highlight they are clickable.
    • Drop various trailing underlined whitespaces after links.
    • Explicitly log that build was successful or not.
    • Various code-quality improvements, including prefering str.format over concatentation.
  • Miscellaneous updates to our filter-packages internal tool:
    • Add --random=N and --url options.
    • Add support for --show=comments.
    • Correct ordering so that --show-version runs after --filter-ftbfs.
    • Rename --show-ftbfs to --filter-ftbfs and --show-version to --show=version.
  • Created a proof-of-concept reproducible-utils package to contain commonly-used snippets aimed at developers wishing to make their packages reproducible.


I also submitted 92 patches to fix specific reproducibility issues in advi, amora-server, apt-cacher-ng, ara, argyll, audiotools, bam, bedtools, binutils-m68hc1x, botan1.10, broccoli, congress, cookiecutter, dacs, dapl, dateutils, ddd, dicom3tools, dispcalgui, dnssec-trigger, echoping, eekboek, emacspeak, eyed3, fdroidserver, flashrom, fntsample, forkstat, gkrellm, gkrellm, gnunet-gtk, handbrake, hardinfo, ircd-irc2, ircd-ircu, jack-audio-connection-kit, jpy, kxmlgui, libbson, libdc0, libdevel-cover-perl, libfm, libpam-ldap, libquvi, librep, lilyterm, mozvoikko, mp4h, mp4v2, myghty, n2n, nagios-nrpe, nikwi, nmh, nsnake, openhackware, pd-pdstring, phpab, phpdox, phpldapadmin, pixelmed-codec, pleiades, pybit, pygtksourceview, pyicu, python-attrs, python-gflags, quvi, radare2, rc, rest2web, roaraudio, rt-extension-customfieldsonupdate, ruby-compass, ruby-pg, sheepdog, tf5, ttf-tiresias, ttf-tiresias, tuxpaint, tuxpaint-config, twitter-bootstrap3, udpcast, uhub, valknut, varnish, vips, vit, wims, winswitch, wmweather+ & xshisen.


Debian GNU/Linux Patches contributed

I also submitted 22 patches to fix typos in debian/rules files against ctsim, f2c, fonts-elusive-icons, ifrit, ldapscripts, libss7, libvmime, link-grammar, menulibre, mit-scheme, mugshot, nlopt, nunit, proftpd-mod-autohost, proftpd-mod-clamav, rabbyt, radvd, ruby-image-science, snmpsim, speech-tools, varscan & whatmaps.

Debian LTS

This month I have been paid to work 15 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Authored the patch & issued DLA 596-1 for extplorer, a web-based file manager, fixing an archive traversal exploit.
  • Issued DLA 598-1 for suckless-tools, fixing a segmentation fault in the slock screen locking tool.
  • Issued DLA 599-1 for cracklib2, a pro-active password checker library, fixing a stack-based buffer overflow when parsing large GECOS fields.
  • Improved the find-work internal tool adding optional colour highlighting and migrating it to Python 3.
  • Wrote an lts-missing-uploads tool to find mistakes where there was no correponding package in the archive after an announcement.
  • Added optional colour highlighting to the lts-cve-triage tool.
Uploads
  • redis 2:3.2.3-1 — New upstream release, move to the DEP-5 debian/copyright format, ensure that we are running as root in LSB initscripts and add a README.Source regarding our local copies of redis.conf and sentinel.conf.
  • python-django:
    • 1:1.10-1 — New upstream release.
    • 1:1.10-2 — Fix test failures due to mishandled upstream translation updates.

  • gunicorn:
    • 19.6.0-2 — Reload logrotate in the postrotate action to avoid processes writing to the old files and move to DEP-5 debian/copyright format.
    • 19.6.0-3 — Drop our /usr/sbin/gunicorn{,3}-debian and related Debian-specific machinery to be more like upstream.
    • 19.6.0-4 — Drop "template" systemd .service files and point towards examples and documentation instead.

  • adminer:
    • 4.2.5-1 — Take over package maintenance, completely overhauling the packaging with a new upstream version, move to virtual-mysql-server to support MariaDB, updating package names of dependencies and fix the outdated Apache configuration.
    • 4.2.5-2 — Correct the php5 package names.

Bugs filed (without patches) RC bugs

I filed 3 RC bugs with patches:



I additionally filed 8 RC bugs for packages that access the internet during build against autopkgtest, golang-github-xenolf-lego, pam-python, pexpect, python-certbot, python-glanceclient, python-pykka & python-tornado.



I also filed 74 FTBFS bugs against airlift-airline, airlift-slice, alter-sequence-alignment, apktool, atril, auto-apt-proxy, bookkeeper, bristol, btfs, caja-extensions, ccbuild, cinder, clustalo, colorhug-client, cpp-netlib, dimbl, edk2, elasticsearch, ganv, git-remote-hg, golang-codegangsta-cli, golang-goyaml, gr-radar, imagevis3d, jacktrip, jalv, kdepim, kiriki, konversation, libabw, libcereal, libdancer-plugin-database-perl, libdist-zilla-plugins-cjm-perl, libfreemarker-java, libgraph-writer-dsm-perl, libmail-gnupg-perl, libminc, libsmi, linthesia, lv2-c++-tools, lvtk, mate-power-manager, mcmcpack, mopidy-podcast, nageru, nfstrace, nova, nurpawiki, open-gram, php-crypt-gpg, picmi, projectl, pygpgme, python-apt, python-django-bootstrap-form, python-django-navtag, python-oslo.config, qmmp, qsapecng, r-cran-sem, rocs, ruby-mini-magick, seahorse-nautilus, shiro, snap, tcpcopy, tiledarray, triggerhappy, ucto, urdfdom, vmmlib, yara-python, yi & z3.


FTP Team

As a Debian FTP assistant I ACCEPTed 90 packages: android-platform-external-jsilver, android-platform-frameworks-data-binding, camlpdf, consolation, dfwinreg, diffoscope, django-restricted-resource, django-testproject, django-testscenarios, gitlab-ci-multi-runner, gnome-shell-extension-taskbar, golang-github-flynn-archive-go-shlex, golang-github-jamesclonk-vultr, golang-github-weppos-dnsimple-go, golang-golang-x-time, google-android-ndk-installer, haskell-expiring-cache-map, haskell-hclip, haskell-hdbc-session, haskell-microlens-ghc, haskell-names-th, haskell-persistable-record, haskell-should-not-typecheck, haskell-soap, haskell-soap-tls, haskell-th-reify-compat, haskell-with-location, haskell-wreq, kbtin, libclipboard-perl, libgtk3-simplelist-perl, libjs-jquery-selectize.js, liblemon, libplack-middleware-header-perl, libreoffice, libreswan, libtest-deep-json-perl, libtest-timer-perl, linux, linux-signed, live-tasks, llvm-toolchain-3.8, llvm-toolchain-snapshot, lua-luv, lua-torch-image, lua-torch-nn, magic-wormhole, mini-buildd, ncbi-vdb, node-ast-util, node-es6-module-transpiler, node-es6-promise, node-inline-source-map, node-number-is-nan, node-object-assign, nvidia-graphics-drivers, openhft-chronicle-bytes, openhft-chronicle-core, openhft-chronicle-network, openhft-chronicle-threads, openhft-chronicle-wire, pycodestyle, python-aptly, python-atomicwrites, python-click-log, python-django-casclient, python-git-os-job, python-hypothesis, python-nosehtmloutput, python-overpy, python-parsel, python-prov, python-py, python-schema, python-tackerclient, python-tornado, pyvo, r-cran-cairo, r-cran-mi, r-cran-rcppgsl, r-cran-sem, ruby-curses, ruby-fog-rackspace, ruby-mixlib-archive, ruby-tzinfo-data, salt-formula-swift, scapy3k, self-destructing-cookies, trollius-redis & websploit.

Categories: LUG Community Blogs

Chris Lamb: try.diffoscope.org CLI client

Sun, 14/08/2016 - 19:43

One criminally-unknown new UNIX tool is diffoscope, a diff "on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them instead of simply showing the raw difference in hexadecimal.

In an attempt to remedy its underuse, in December 2015 I created the try.diffoscope.org service so that I—and hopefully others—could use diffoscope without necessarily installing the multitude of third-party tools that using it can require. It also enables trivial sharing of the HTML reports in bugs or on IRC.

To make this even easier, I've now introduced a command-line client to the web service:

$ apt-get install trydiffoscope [..] Setting up trydiffoscope (57) ... $ trydiffoscope /etc/hosts.allow /etc/hosts.deny --- a/hosts.allow +++ b/hosts.deny │ @@ -1,10 +1,17 @@ │ -# /etc/hosts.allow: list of hosts that are allowed to access the system. │ -# See the manual pages hosts_access(5) and hosts_options(5). │ +# /etc/hosts.deny: list of hosts that are _not_ allowed to access the system. │ +# See the manual pages hosts_access(5) and hosts_options(5).

You can also install it from PyPI with:

$ pip install trydiffoscope

Mirroring the original diffoscope command, you can save the output locally in an even more-readable HTML report format by appending "--html output.html".

In addition, if you specify the --webbrowser (or -w) argument:

$ trydiffoscope -w /etc/hosts.allow /etc/hosts.deny https://try.diffoscope.org/gaauupyapzkb

... this will automatically open your default browser to view the results.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in July 2016

Mon, 01/08/2016 - 05:20

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):



  • Ensured that the Webconverger web kiosk operating system builds reproducibly. I may rework some of the patches to libisoburn and libisofs before sending them upstream. This work was sponsored by Webconverger.
  • Proposed a pull request for Regex Replace (a Chrome extension to automatically replace text on webpages) to ensure that the rules were correctly HTML encoded on the options page. (#3)
  • Proposed a change to ronn, a documentation generator that "is the opposite of roff", to make the output reproducible. (#98)
  • Fixed an issue in django-enumfield, a custom Django web development field for type-safe named constants, to make the Enum.get interface more consistent. (#36)
  • Proposed a change to txt2tags to make the output use SOURCE_DATE_EPOCH and non-timezone timestamps. (#204).
Debian
  • Created a proof-of-concept wrapper for pymysql to reduce the diff between Ubuntu and Debian's packaging of python-django. (tree)
  • Improved the NEW queue HTML report to display absolute timestamps when placing the cursor over relative times as well as to tidy the underlying HTML generation.
  • Tidied and pushed for the adoption of a patch against dak to also send mails to the signer of an uploaded package on security-master. (#796784)
LTS

This month I have been paid to work 14 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Improved the bin/lts-cve-triage.py script to ignore packages that have been marked as unsupported.
  • Improved the bin/contact-maintainers script to print a nicer error message if you mistype the package name.
  • Issued the following advisories:
    • DLA 541-1 for libvirt making the password policy consistent across the QEMU and VNC backends with respect to empty passwords.
    • DLA 574-1 for graphicsmagick fixing two denial-of-service vulnerabilities.
    • DLA 548-1 and DLA 550-1 for drupal7 fixing an open HTTP redirect vulnerability and a privilege escalation issue respectfully.
    • DLA 557-1 for dietlibc removing the current directory from the current path.
    • DLA 577-1 for redis preventing the redis-cli tool creating world-readable history files.
Uploads
  • redis:
    • 3.2.1-2 — Avoiding race conditions in upstream test suite.
    • 3.2.1-3 — Correcting world_readable ~/.rediscli_history files.
    • 3.2.1-4 — Preventing a race condition in the previous upload's patch.
    • 3.2.2-1 — New upstream release.
    • 3.2.1-4~bpo8+1 — Backport to jessie-backports.
  • strip-nondeterminism:
    • 0.020-1 — Improved the PNG handler to not blindly trust chunk sizes, rewriting most of the existing code.
    • 0.021-1 — Correcting a regression in the PNG handler where it would leave temporary files in the generated binaries.
    • 0.022-1 — Correcting a further regression in the PNG handler with respect to IEND chunk detection.
  • python-redis (2.10.5-1~bpo8+1) — Backport to jessie-backports.
  • reprotest (0.2) — Sponsored upload.
Patches contributed

I submitted patches to fix faulty initscripts in lm-sensors, rsync, sane-backends & vsftpd.


In addition, I submitted 7 patches to fix typos in debian/rules against cme:, gnugk: `incorrect reference to dh_install_init, php-sql-formatter, python-django-crispy-forms, libhook-lexwrap-perl, mknbi & ruby-unf-ext.


I also submitted 6 patches to fix reproducible toolchain issues (ie. ensuring the output is reproducible rather than the package itself) against libextutils-parsexs-perl: `Please make the output reproducible, perl, naturaldocs, python-docutils, ruby-ronn & txt2tags.


Lastly, I submitted 65 patches to fix specific reproducibility issues in amanda, boolector, borgbackup, cc1111, cfingerd, check-all-the-things, cobbler, ctop, cvs2svn, eb, eurephia, ezstream, feh, fonts-noto, fspy, ftplib, fvwm, gearmand, gngb, golang-github-miekg-pkcs11, gpick, gretl, hibernate, hmmer, hocr, idjc, ifmail, ironic, irsim, lacheck, libmemcached-libmemcached-perl, libmongoc, libwebsockets, minidlna, mknbi, nbc, neat, nfstrace, nmh, ntopng, pagekite, pavuk, proftpd-dfsg, pxlib, pysal, python-kinterbasdb, python-mkdocs, sa-exim, speech-tools, stressapptest, tcpflow, tcpreen, ui-auto, uisp, uswsusp, vtun, vtwm, why3, wit, wordgrinder, xloadimage, xmlcopyeditor, xorp, xserver-xorg-video-openchrome & yersinia.


Bugs filed without patches
RC bugs

I also filed 68 RC bugs for packages that access the internet during build against betamax, curl, django-localflavor, django-polymorphic, dnspython, docker-registry, elasticsearch-curator, elib.intl, elib.intl, elib.intl, fabulous, flask-restful, flask-restful, flask-restful, foolscap, gnucash-docs, golang-github-azure-go-autorest, golang-github-fluent-fluent-logger-golang, golang-github-franela-goreq, golang-github-mesos-mesos-go, golang-github-shopify-sarama, golang-github-unknwon-com, golang-github-xeipuuv-gojsonschema, htsjdk, lemonldap-ng, libanyevent-http-perl, libcommons-codec-java, libfurl-perl, libgravatar-url-perl, libgravatar-url-perl, libgravatar-url-perl, libgravatar-url-perl, libgravatar-url-perl, libhttp-async-perl, libhttp-oai-perl, libhttp-proxy-perl, libpoe-component-client-http-perl, libuv, libuv1, licenseutils, licenseutils, licenseutils, musicbrainzngs, node-oauth, node-redis, nodejs, pycurl, pytest, python-aiohttp, python-asyncssh, python-future, python-guacamole, python-latexcodec, python-pysnmp4, python-qtawesome, python-simpy, python-social-auth, python-structlog, python-sunlight, python-webob, python-werkzeug, python-ws4py, testpath, traitlets, urlgrabber, varnish-modules, webtest & zurl.



Finally, I filed 100 FTBFS bugs against abind, backup-manager, boot, bzr-git, cfengine3, chron, cloud-sptheme, cookiecutter, date, django-uwsgi, djangorestframework, docker-swarm, ekg2, evil-el, fasianoptions, fassets, fastinfoset, fest-assert, fimport, ftrading, gdnsd, ghc-testsuite, golang-github-magiconair-properties, golang-github-mattn-go-shellwords, golang-github-mitchellh-go-homedir, gplots, gregmisc, highlight.js, influxdb, jersey1, jflex, jhdf, kimwitu, libapache-htpasswd-perl, libconfig-model-itself-perl, libhtml-tidy-perl, liblinux-prctl-perl, libmoox-options-perl, libmousex-getopt-perl, libparanamer-java, librevenge, libvirt-python, license-reconcile, louie, mako, mate-indicator-applet, maven-compiler-plugin, mgt, mgt, mgt, misc3d, mnormt, nbd, ngetty, node-xmpp, nomad, perforate, pyoperators, pyqi, python-activipy, python-bioblend, python-cement, python-gevent, python-pydot-ng, python-requests-toolbelt, python-ruffus, python-scrapy, r-cran-digest, r-cran-getopt, r-cran-lpsolve, r-cran-rms, r-cran-timedate, resteasy, ruby-berkshelf-api-client, ruby-fog-libvirt, ruby-grape-msgpack, ruby-jquery-rails, ruby-kramdown-rfc2629, ruby-moneta, ruby-parser, ruby-puppet-forge, ruby-rbvmomi, ruby-redis-actionpack, ruby-unindent, ruby-web-console, scalapack-doc, scannotation, snow, sorl-thumbnail, svgwrite, systemd-docker, tiles-request, torcs, utf8proc, vagrant-libvirt, voms-api-java, wcwidth, xdffileio, xmlgraphics-commons & yorick.

FTP Team

As a Debian FTP assistant I ACCEPTed 114 packages: apertium-isl-eng, apertium-mk-bg, apertium-urd-hin, apprecommender, auto-apt-proxy, beast-mcmc, caffe, caffe-contrib, debian-edu, dh-make-perl, django-notification, dpkg-cross, elisp-slime-nav, evil-el, fig2dev, file, flightgear-phi, friendly-recovery, fwupd, gcc-5-cross, gdbm, gnustep-gui, golang-github-cznic-lldb, golang-github-dghubble-sling, golang-github-docker-leadership, golang-github-rogpeppe-fastuuid, golang-github-skarademir-naturalsort, golang-glide, gtk+2.0, gtranscribe, kdepim4, kitchen, lepton, libcgi-github-webhook-perl, libcypher-parser, libimporter-perl, liblist-someutils-perl, liblouis, liblouisutdml, libneo4j-client, libosinfo, libsys-cpuaffinity-perl, libtest2-suite-perl, linux, linux-grsec, lua-basexx, lua-compat53, lua-fifo, lua-http, lua-lpeg-patterns, lua-mmdb, lua-openssl, mash, mysql-5.7, node-quickselect, nsntrace, nvidia-graphics-drivers, nvidia-graphics-drivers-legacy-304xx, nvidia-graphics-drivers-legacy-340xx, openorienteering-mapper, oslo-sphinx, p4est, patator, petsc, php-mailparse, php-yaml, pykdtree, pypass, python-bioblend, python-cotyledon, python-jack-client, python-mido, python-openid-cla, python-os-api-ref, python-pydotplus, python-qtconsole, python-repoze.sphinx.autointerface, python-vispy, python-zenoss, r-cran-bbmle, r-cran-corpcor, r-cran-ellipse, r-cran-minpack.lm, r-cran-rglwidget, r-cran-rngtools, r-cran-scatterd3, r-cran-shinybs, r-cran-tibble, reproject, retext, ring, ruby-github-api, ruby-rails-assets-jquery-ui, ruby-swd, ruby-url-safe-base64, ruby-vmstat, ruby-webfinger, rustc, shadowsocks-libev, slepc, staticsite, steam, straight.plugin, svgwrite, tasksh, u-msgpack-python, ufo2otf, user-mode-linux, utf8proc, vizigrep, volk, wchartype, websockify & wireguard.

Categories: LUG Community Blogs

Chris Lamb: Python quirk: Signatures are evaluated at import time

Thu, 21/07/2016 - 12:07

Every Python programmer knows to avoid mutable default arguments:

def fn(mutable=[]): mutable.append('elem') print mutable fn() fn() $ python test.py ['elem'] ['elem', 'elem']

However, many are not clear that this is due to arguments being evaluated at import time, rather than the first time the function is evaluated.

This results in related quirks such as:

def never_called(error=1/0): pass $ python test.py Traceback (most recent call last): File "test.py", line 1, in <module> ZeroDivisionError: integer division or modulo by zero

... and an—implementation-specific—quirk caused by naive constant folding:

def never_called(): 99999999 ** 9999999 $ python test.py [hangs]

I suspect that this can be used as denial-of-service vector.

Categories: LUG Community Blogs

Chris Lamb: Python quirk: os.stat's return type

Tue, 19/07/2016 - 11:20
import os import stat st = os.stat('/etc/fstab') # __getitem__ x = st[stat.ST_MTIME] print((x, type(x))) # __getattr__ x = st.st_mtime print((x, type(x))) (1441565864, <class 'int'>) (1441565864.3485234, <class 'float'>)
Categories: LUG Community Blogs

Mick Morgan: show me yours

Wed, 13/07/2016 - 17:30

As Theresa May moves from the Home Office to Number 10, it is perhaps timely to reflect on public attitudes to surveillance as evidenced in Liberty’s campaign film “Show me yours” in April of this year. In the film (shown below), comedian Olivia Lee pursues members of the public with the intention of taking details from their mobile phones of all their recent communications or browsing activity. The reactions of the people approached speak for themselves. Unfortunately, Liberty research suggests that 75% of adults in the UK had never heard of the impending legislation laid out in the Investigatory Powers Bill.

Categories: LUG Community Blogs

Jonathan McDowell: Confirming all use of an SSH agent

Sun, 03/07/2016 - 16:55

For a long time I’ve wanted an ssh-agent setup that would ask me before every use, so I could slightly more comfortably forward authentication over SSH without worrying that my session might get hijacked somewhere at the remote end (I often find myself wanting to pull authenticated git repos on remote hosts). I’m at DebConf this week, which is an ideal time to dig further into these things, so I did so today. As is often the case it turns out this is already possible, if you know how.

I began with a setup that was using GNOME Keyring to manage my SSH keys. This isn’t quite what I want (eventually I want to get to the point that I can sometimes forward a GPG agent to remote hosts for signing purposes as well), so I set about setting up gpg-agent. I used Chris’ excellent guide to GnuPG/SSH Agent setup as a starting point and ended up doing the following:

$ echo use-agent >> ~/.gnupg/options $ echo enable-ssh-support >> ~/.gnupg/gpg-agent.conf $ sudo sed -i.bak "s/^use-ssh-agent/# use-ssh-agent/" /etc/X11/Xsession.options $ sudo rm /etc/xdg/autostart/gnome-keyring-ssh.desktop

The first 2 commands setup my local agent, and told it to do SSH agent foo. The next stopped X from firing up ssh-agent, and the final one prevents GNOME Keyring from being configured to be the SSH agent, without having to remove libpam-gnome-keyring as Chris did. After the above I logged out of and into X again, and could see ~/.gnupg/S.gpg-agent.ssh getting created and env | grep SSH showing SSH_AUTH_SOCK pointing to it (if GNOME Keyring is still handling things it ends up pointing to something like /run/user/1000/keyring/ssh).

[Update: Luca Capello emailed to point out this was a bad approach; there’s thankfully no need to do the last 2 commands that require root. #767341 removed the need to edit Xsession.options and you can prevent GNOME Keyring starting on a per user basis with:

(cat /etc/xdg/autostart/gnome-keyring-ssh.desktop ; echo 'X-GNOME-Autostart-enabled=false') > \ ~/.config/autostart/gnome-keyring-ssh.desktop

]

After this it turned out all I need to do was ssh-add -c <ssh keyfile>. The -c says “confirm use” and results in the confirm flag being appended to the end of ~/.gnupg/sshcontrol (so if you’ve already done the ssh-add you can go and add the confirm if that’s the behaviour you’d like).

Simple when you know how, but I’ve had conversations with several people in the past who wanted the same thing and hadn’t figured out how, so hopefully this is helpful to others.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in June 2016

Thu, 30/06/2016 - 21:32

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

Debian

My work in the Reproducible Builds project was covered in our weekly reports. (#58, #59 & #60)

Debian LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Extended the lts-cve-triage.py script to ignore packages that are not subject to Long Term Support.

  • Issued DLA 512-1 for mantis fixing an XSS vulnerability.
  • Issued DLA 513-1 for nspr correcting a buffer overflow in a sprintf utility.
  • Issued DLA 515-1 for libav patching a memory corruption issue.
  • Issued DLA 524-1 for squidguard fixing a reflected cross-site scripting vulnerability.
  • Issued DLA 525-1 for gimp correcting a use-after-free vulnerability in the channel and layer properties parsing process.
Uploads
  • redis (2:3.2.1-1) — New upstream bugfix release, plus subsequent upload to the backports repository.
  • python-django (1.10~beta1-1) — New upstream experimental release.
  • libfiu (0.94-5) — Misc packaging updates.
Patches contributed
RC bugs

I also filed 170 FTBFS bugs against a7xpg, acepack, android-platform-dalvik, android-platform-frameworks-base, android-platform-system-extras, android-platform-tools-base, apache-directory-api, aplpy, appstream-generator, arc-gui-clients, assertj-core, astroml, bamf, breathe, buildbot, cached-property, calf, celery-haystack, charmtimetracker, clapack, cmake, commons-javaflow, dataquay, dbi, django-celery, django-celery-transactions, django-classy-tags, django-compat, django-countries, django-floppyforms, django-hijack, django-localflavor, django-markupfield, django-model-utils, django-nose, django-pipeline, django-polymorphic, django-recurrence, django-sekizai, django-sitetree, django-stronghold, django-taggit, dune-functions, elementtidy, epic4-help, fcopulae, fextremes, fnonlinear, foreign, fort77, fregression, gap-alnuth, gcin, gdb-avr, ggcov, git-repair, glance, gnome-twitch, gnustep-gui, golang-github-audriusbutkevicius-go-nat-pmp, golang-github-gosimple-slug, gprbuild, grafana, grantlee5, graphite-api, guacamole-server, ido, jless, jodreports, jreen, kdeedu-data, kdewebdev, kwalify, libarray-refelem-perl, libdbusmenu, libdebian-package-html-perl, libdevice-modem-perl, libindicator, liblrdf, libmail-milter-perl, libopenraw, libvisca, linuxdcpp, lme4, marble, mgcv, mini-buildd, mu-cade, mvtnorm, nose, octave-epstk, onioncircuits, opencolorio, parsec47, phantomjs, php-guzzlehttp-ringphp, pjproject, pokerth, prayer, pyevolve, pyinfra, python-asdf, python-ceilometermiddleware, python-django-bootstrap-form, python-django-compressor, python-django-contact-form, python-django-debug-toolbar, python-django-extensions, python-django-feincms, python-django-formtools, python-django-jsonfield, python-django-mptt, python-django-openstack-auth, python-django-pyscss, python-django-registration, python-django-tagging, python-django-treebeard, python-geopandas, python-hdf5storage, python-hypothesis, python-jingo, python-libarchive-c, python-mhash, python-oauth2client, python-proliantutils, python-pytc, python-restless, python-tidylib, python-websockets, pyvows, qct, qgo, qmidinet, quodlibet, r-cran-gss, r-cran-runit, r-cran-sn, r-cran-stabledist, r-cran-xml, rgl, rglpk, rkt, rodbc, ruby-devise-two-factor, ruby-json-schema, ruby-puppet-syntax, ruby-rspec-puppet, ruby-state-machine, ruby-xmlparser, ryu, sbd, scanlogd, signond, slpvm, sogo, sphinx-argparse, squirrel3, sugar-jukebox-activity, sugar-log-activity, systemd, tiles, tkrplot, twill, ucommon, urca, v4l-utils, view3dscene, xqilla, youtube-dl & zope.interface.

FTP Team

As a Debian FTP assistant I ACCEPTed 186 packages: akonadi4, alljoyn-core-1509, alljoyn-core-1604, alljoyn-gateway-1504, alljoyn-services-1504, alljoyn-services-1509, alljoyn-thin-client-1504, alljoyn-thin-client-1509, alljoyn-thin-client-1604, apertium-arg, apertium-arg-cat, apertium-eo-fr, apertium-es-it, apertium-eu-en, apertium-hbs, apertium-hin, apertium-isl, apertium-kaz, apertium-spa, apertium-spa-arg, apertium-tat, apertium-urd, arc-theme, argus-clients, ariba, beast-mcmc, binwalk, bottleneck, colorfultabs, dh-runit, django-modeltranslation, dq, dublin-traceroute, duktape, edk2, emacs-pdf-tools, eris, erlang-p1-oauth2, erlang-p1-sqlite3, erlang-p1-xmlrpc, faba-icon-theme, firefox-branding-iceweasel, golang-1.6, golang-defaults, golang-github-aelsabbahy-gonetstat, golang-github-howeyc-gopass, golang-github-oleiade-reflections, golang-websocket, google-android-m2repository-installer, googler, goto-chg-el, gr-radar, growl-for-linux, guvcview, haskell-open-browser, ipe, labplot, libalt-alien-ffi-system-perl, libanyevent-fcgi-perl, libcds-savot-java, libclass-ehierarchy-perl, libconfig-properties-perl, libffi-checklib-perl, libffi-platypus-perl, libhtml-element-library-perl, liblwp-authen-oauth2-perl, libmediawiki-dumpfile-perl, libmessage-passing-zeromq-perl, libmoosex-types-portnumber-perl, libmpack, libnet-ip-xs-perl, libperl-osnames-perl, libpodofo, libprogress-any-perl, libqtpas, librdkafka, libreoffice, libretro-beetle-pce-fast, libretro-beetle-psx, libretro-beetle-vb, libretro-beetle-wswan, libretro-bsnes-mercury, libretro-mupen64plus, libservicelog, libtemplate-plugin-datetime-perl, libtext-metaphone-perl, libtins, libzmq-ffi-perl, licensecheck, link-grammar, linux, linux-signed, lua-busted, magics++, mkalias, moka-icon-theme, neutron-vpnaas, newlisp, node-absolute-path, node-ejs, node-errs, node-has-flag, node-lodash-compat, node-strip-ansi, numba, numix-icon-theme, nvidia-graphics-drivers, nvidia-graphics-drivers-legacy-304xx, nvidia-graphics-drivers-legacy-340xx, obs-studio, opencv, pacapt, pgbackrest, postgis, powermock, primer3, profile-sync-daemon, pyeapi, pypandoc, pyssim, python-cutadapt, python-cymruwhois, python-fisx, python-formencode, python-hkdf, python-model-mommy, python-nanomsg, python-offtrac, python-social-auth, python-twiggy, python-vagrant, python-watcherclient, python-xkcd, pywps, r-bioc-deseq2, r-bioc-dnacopy, r-bioc-ensembldb, r-bioc-geneplotter, r-cran-adegenet, r-cran-adephylo, r-cran-distory, r-cran-fields, r-cran-future, r-cran-globals, r-cran-htmlwidgets, r-cran-listenv, r-cran-mlbench, r-cran-mlmrev, r-cran-pheatmap, r-cran-pscbs, r-cran-r.cache, refind, relatorio, reprotest, ring, ros-ros-comm, ruby-acts-as-tree, ruby-chronic-duration, ruby-flot-rails, ruby-numerizer, ruby-u2f, selenium-firefoxdriver, simgrid, skiboot, smtpping, snap-confine, snapd, sniffles, sollya, spin, subuser, superlu, swauth, swift-plugin-s3, syncthing, systemd-bootchart, tdiary-theme, texttable, tidy-html5, toxiproxy, twinkle, vmtk, wait-for-it, watcher, wcslib & xapian-core.

Categories: LUG Community Blogs

Jonathan McDowell: Hire me!

Mon, 27/06/2016 - 23:21

It’s rare to be in a position to be able to publicly announce you’re looking for a new job, but as the opportunity is currently available to me I feel I should take advantage of it. That’s especially true given the fact I’ll be at DebConf 16 next week and hope to be able to talk to various people who might be hiring (and will, of course, be attending the job fair).

I’m coming to the end of my Masters in Legal Science and although it’s been fascinating I’ve made the decision that I want to return to the world of tech. I like building things too much it seems. There are various people I’ve already reached out to, and more that are on my list to contact, but I figure making it more widely known that I’m in the market can’t hurt with finding the right fit.

  • Availability: August 2016 onwards. I can wait for the right opportunity, but I’ve got a dissertation to write up so can’t start any sooner.
  • Location: Preferably Belfast, Northern Ireland. I know that’s a tricky one, but I’ve done my share of moving around for the moment (note I’ve no problem with having to do travel as part of my job). While I prefer an office environment I’m perfectly able to work from home, as long as it’s as part of a team that is tooled up for disperse workers - in my experience being the only remote person rarely works well. There’s a chance I could be persuaded to move to Dublin for the right role.
  • Type of role: I sit somewhere on the software developer/technical lead/architect spectrum. I expect to get my hands dirty (it’s the only way to learn a system properly), but equally if I’m not able to be involved in making high level technical decisions then I’ll find myself frustrated.
  • Technology preferences: Flexible. My background is backend systems programming (primarily C in the storage and networking spaces), but like most developers these days I’ve had exposure to a bunch of different things and enjoy the opportunity to learn new things.

I’m on LinkedIn and OpenHUB, which should give a bit more info on my previous experience and skill set. I know I’m light on details here, so feel free to email me to talk about what I might be able to specifically bring to your organisation.

Categories: LUG Community Blogs

Steve Engledow (stilvoid): Brugger Off

Fri, 24/06/2016 - 13:32

I'm putting this here and then I'm going to try not to say anything else on the subject for a while.

I'm disappointed and upset by result of the referendum. Not because we're (probably) leaving the EU. Us leaving may be the beginning of the fall of the EU and I can't tell one way or another how that will affect anyone in the world.

I'm hurt and ashamed because it's a measure of the sentiments of the people who live in the UK. 52% of you are leaning in a direction that I want no part of and don't want my son to be surrounded by as he grows up. I grew up in the tail of end of Thatcher's Britain and the UK today has the same oppressive feeling that you can sense when you watch the Young Ones.

I have some very good friends who voted out and they are good people so I'm certainly not tarring everyone with the racist brush but I've seen much fear and hate generally and I'm just saddened that this country is following the international trend and moving to the far right.

It's not an exaggeration to say that I'm pretty damn scared of the future with the US possibly about to vote in a right wing leadership too.

Don't tell me "it'll be alright" because it's not the fact of the decision that has me upset; it's what it tells me about the country I love. Or used to love. I don't know.

Categories: LUG Community Blogs

Jonathan McDowell: Fixing missing text in Firefox

Thu, 23/06/2016 - 15:23

Every now and again I get this problem where Firefox won’t render text correctly (on a Debian/stretch system). Most websites are fine, but the odd site just shows up with blanks where the text should be. Initially I thought it was NoScript, but turning that off didn’t help. Daniel Silverstone gave me a pointer today that the pages in question were using webfonts, and that provided enough information to dig deeper. The sites in question were using Cantarell, via:

src: local('Cantarell Regular'), local('Cantarell-Regular'), url(cantarell.woff2) format('woff2'), url(cantarell.woff) format('woff');

The Firefox web dev inspector didn’t show it trying to fetch the font remotely, so I removed the local() elements from the CSS. That fixed the page, letting me pinpoint the problem as a local font issue. I have fonts-cantarell installed so at first I tried to remove it, but that breaks gnome-core. So instead I did an fc-list | grep -i cant to ask fontconfig what it thought was happening. That gave:

/usr/share/fonts/opentype/cantarell/Cantarell-Regular.otf.dpkg-tmp: Cantarell:style=Regular /usr/share/fonts/opentype/cantarell/Cantarell-Bold.otf.dpkg-tmp: Cantarell:style=Bold /usr/share/fonts/opentype/cantarell/Cantarell-Bold.otf: Cantarell:style=Bold /usr/share/fonts/opentype/cantarell/Cantarell-Oblique.otf: Cantarell:style=Oblique /usr/share/fonts/opentype/cantarell/Cantarell-Regular.otf: Cantarell:style=Regular /usr/share/fonts/opentype/cantarell/Cantarell-Bold-Oblique.otf: Cantarell:style=Bold-Oblique /usr/share/fonts/opentype/cantarell/Cantarell-Oblique.otf.dpkg-tmp: Cantarell:style=Oblique /usr/share/fonts/opentype/cantarell/Cantarell-BoldOblique.otf: Cantarell:style=BoldOblique

Hmmm. Those .dpkg-tmp files looked odd, and sure enough they didn’t actually exist. So I did a sudo fc-cache -f -v to force a rebuild of the font cache and restarted Firefox (it didn’t seem to work before doing so) and everything works fine now.

It seems that fc-cache must have been run at some point when dpkg had not yet completed installing an update to the fonts-cantarell package. That seems like a bug - fontconfig should probably ignore .dpkg* files, but equally I wouldn’t expect it to be run before dpkg had finished its unpacking stage fully.

Categories: LUG Community Blogs