Planet HantsLUG

Syndicate content
Planet HantsLUG - http://hantslug.org.uk/planet/
Updated: 31 min 55 sec ago

Andy Smith: Supermicro SATA DOM flash devices don’t report lifetime writes correctly

Sat, 26/11/2016 - 16:43

I’m playing around with a pair of Supermicro SATA DOM flash devices at the moment, evaluating them for use as the operating system storage for servers (as opposed to where customer data goes).

They’re flash devices with a limited write endurance. The smallest model (16GB), for example, is good for 17TB of writes. Therefore it’s important to know how much you’ve actually written to it.

Many SSDs and other flash devices expose the total amount written through the SMART attribute 241, Total_LBAs_Written. The SATA DOM devices do seem to expose this attribute, but right now they say this:

$ for dom in $(sudo lsblk --paths -d -o NAME,MODEL --noheadings | awk '/SATA SSD/ { print $1 }') do echo -n "$dom: " sudo smartctl -A "$dom" | awk '/^241/ { print $10 * 512 * 1.0e-9, "GB" }' done /dev/sda: 0.00856934 GB /dev/sdb: 0.00881715 GB

This being after install and (as of now) more than a week of uptime, ~9MB of lifetime writes isn’t credible.

Another place we can look for amount of bytes written is /proc/diskstats. The 10th column is the number of (512-byte) sectors written, so:

$ for dom in $(sudo lsblk -d -o NAME,MODEL --noheadings | awk '/SATA SSD/ { print $1 }') do awk "/$dom / { print \$3, \$10 / 2 * 1.0e-6, \"GB\" }" /proc/diskstats done sda 3.93009 GB sdb 3.93009 GB

Almost 4GB is a lot more believable, so can we just use /proc/diskstats? Well, the problem there is that those figures are only since boot. That won’t include, for example, all the data written during install.

Okay, so, are these figures even consistent? Let’s write 100MB and see what changes.

Since the figure provided by SMART attribute 241 apparently isn’t actually 512-byte blocks we’ll just print the raw value there.

Before:

$ for dom in $(sudo lsblk -d -o NAME,MODEL --noheadings | awk '/SATA SSD/ { print $1 }') do awk "/$dom / { print \$3, \$10 / 2 * 1.0e-6, \"GB\" }" /proc/diskstats done sda 4.03076 GB sdb 4.03076 GB $ for dom in $(sudo lsblk --paths -d -o NAME,MODEL --noheadings | awk '/SATA SSD/ { print $1 }') do echo -n "$dom: " sudo smartctl -A "$dom" | awk '/^241/ { print $10 }' done /dev/sda: 16835 /dev/sdb: 17318

Write 100MB:

$ dd if=/dev/urandom bs=1MB count=100 > /var/tmp/one_hundred_megabytes 100+0 records in 100+0 records out 100000000 bytes (100 MB) copied, 7.40454 s, 13.5 MB/s

(I used /dev/urandom just in case some compression might take place or something)

After:

$ for dom in $(sudo lsblk -d -o NAME,MODEL --noheadings | awk '/SATA SSD/ { print $1 }') do awk "/$dom / { print \$3, \$10 / 2 * 1.0e-6, \"GB\" }" /proc/diskstats done sda 4.13046 GB sdb 4.13046 GB $ for dom in $(sudo lsblk --paths -d -o NAME,MODEL --noheadings | awk '/SATA SSD/ { print $1 }') do echo -n "$dom: " sudo smartctl -A "$dom" | awk '/^241/ { print $10 }' done /dev/sda: 16932 /dev/sdb: 17416

Well, alright, all is apparently not lost: SMART attribute 241 went up by ~100 and diskstats agrees that ~100MB was written too, so it looks like it does actually report lifetime writes, but it’s reporting them as megabytes (109 bytes), not 512-byte sectors.

Every reference I can find says that Total_LBAs_Written is the number of 512-byte sectors, though, so in reporting units of 1MB I feel that these devices are doing the wrong thing.

Anyway, I’m a little alarmed that ~0.1% of the lifetime has gone already, although a lot of that would have been the install. I probably should take this opportunity to get rid of a lot of writes by tracking down logging of mundane garbage. Also this is the smallest model; the devices are rated for 1 DWPD so just over-provisioning by using a larger model than necessary will help.

Categories: LUG Community Blogs

Steve Kemp: Detecting fraudulent signups?

Mon, 21/11/2016 - 05:37

I run a couple of different sites that allow users to sign-up and use various services. In each of these sites I have some minimal rules in place to detect bad signups, but these are a little ad hoc, because the nature of "badness" varies on a per-site basis.

I've worked in a couple of places where there are in-house tests of bad signups, and these usually boil down to some naive, and overly-broad, rules:

  • Does the phone numbers' (international) prefix match the country of the user?
  • Does the postal address supplied even exist?

Some places penalise users based upon location too:

  • Does the IP address the user submitted from come from TOR?
  • Does the geo-IP country match the users' stated location?
  • Is the email address provided by a "free" provider?

At the moment I've got a simple HTTP-server which receives a JSON post of a new users' details, and returns "200 OK" or "403 Forbidden" based on some very very simple critereon. This is modeled on the spam detection service for blog-comments server I use - something that is itself becoming less useful over time. (Perhaps time to kill that? A decision for another day.)

Unfortunately this whole approach is very reactive, as it takes human eyeballs to detect new classes of problems. Code can't guess in advance that it should block usernames which could collide with official ones, for example allowing a username of "admin", "help", or "support".

I'm certain that these systems have been written a thousand times, as I've seen at least five such systems, and they're all very similar. The biggest flaw in all these systems is that they try to classify users in advance of them doing anything. We're trying to say "Block users who will use stolen credit cards", or "Block users who'll submit spam", by correlating that behaviour with other things. In an ideal world you'd judge users only by the actions they take, not how they signed up. And yet .. it is better than nothing.

For the moment I'm continuing to try to make the best of things, at least by centralising the rules for myself I cut down on duplicate code. I'll pretend I'm being cool, modern, and sexy, and call this a micro-service! (Ignore the lack of containers for the moment!)

Categories: LUG Community Blogs

Debian Bits: Debian Contributors Survey 2016

Wed, 16/11/2016 - 14:45

The Debian Contributor Survey launched last week!

In order to better understand and document who contributes to Debian, we (Mathieu ONeil, Molly de Blanc, and Stefano Zacchiroli) have created this survey to capture the current state of participation in the Debian Project through the lense of common demographics. We hope a general survey will become an annual effort, and that each year there will also be a focus on a specific aspect of the project or community. The 2016 edition contains sections concerning work, employment, and labour issues in order to learn about who is getting paid to work on and with Debian, and how those relationships affect contributions.

We want to hear from as many Debian contributors as possible—whether you've submitted a bug report, attended a DebConf, reviewed translations, maintain packages, participated in Debian teams, or are a Debian Developer. Completing the survey should take 10-30 minutes, depending on your current involvement with the project and employment status.

In an effort to reflect our own ideals as well as those of the Debian project, we are using LimeSurvey, an entirely free software survey tool, in an instance of it hosted by the LimeSurvey developers.

Survey responses are anonymous, IP and HTTP information are not logged, and all questions are optional. As it is still likely possible to determine who a respondent is based on their answers, results will only be distributed in aggregate form, in a way that does not allow deanonymization. The results of the survey will be analyzed as part of ongoing research work by the organizers. A report discussing the results will be published under a DFSG-free license and distributed to the Debian community as soon as it's ready. The raw, disaggregated answers will not be distributed and will be kept under the responsibility of the organizers.

We hope you will fill out the Debian Contributor Survey. The deadline for participation is: 4 December 2016, at 23:59 UTC.

If you have any questions, don't hesitate to contact us via email at:

Categories: LUG Community Blogs

Debian Bits: New Debian Developers and Maintainers (September and October 2016)

Thu, 03/11/2016 - 11:00

The following contributors got their Debian Developer accounts in the last two months:

  • Adriano Rafael Gomes (adrianorg)
  • Arturo Borrero González (arturo)
  • Sandro Knauß (hefee)

The following contributors were added as Debian Maintainers in the last two months:

  • Abhijith PA
  • Mo Zhou
  • Víctor Cuadrado Juan
  • Zygmunt Bazyli Krynicki
  • Robert Haist
  • Sunil Mohan Adapa
  • Elena Grandi
  • Eric Heintzmann
  • Dylan Aïssi
  • Daniel Shahaf
  • Samuel Henrique
  • Kai-Chung Yan
  • Tino Mettler

Congratulations!

Categories: LUG Community Blogs

Debian Bits: "softWaves" will be the default theme for Debian 9

Tue, 25/10/2016 - 17:50

The theme "softWaves" by Juliette Taka Belin has been selected as default theme for Debian 9 'stretch'.

After the Debian Desktop Team made the call for proposing themes, a total of twelve choices have been submitted, and any Debian contributor has received the opportunity to vote on them in a survey. We received 3,479 responses ranking the different choices, and softWaves has been the winner among them.

We'd like to thank all the designers that have participated providing nice wallpapers and artwork for Debian 9, and encourage everybody interested in this area of Debian, to join the Design Team. It is being considered to package all of them so they are easily available in Debian. If you want to help in this effort, or package any other artwork (for example, particularly designed to be accessibility-friendly), please contact the Debian Desktop Team, but hurry up, because the freeze for new packages in the next release of Debian starts on January 5th, 2017.

This is the second time that Debian ships a theme by Juliette Belin, who also created the theme "Lines" that enhances our actual stable release, Debian 8. Congratulations, Juliette, and thank you very much for your continued commitment to Debian!

Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Mon, 17/10/2016 - 10:40
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Sun, 16/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Sat, 15/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Fri, 14/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Thu, 13/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Wed, 12/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Tue, 11/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Mon, 10/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Sun, 09/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Debian Bits: Debian is participating in the next round of Outreachy!

Sun, 09/10/2016 - 17:50

Following the success of the last round of Outreachy, we are glad to announce that Debian will take part in the program for the next round, with internships lasting from the 6th of December 2016 to the 6th of March 2017.

From the official website: Outreachy helps people from groups underrepresented in free and open source software get involved. We provide a supportive community for beginning to contribute any time throughout the year and offer focused internship opportunities twice a year with a number of free software organizations.

Currently, internships are open internationally to women (cis and trans), trans men, and genderqueer people. Additionally, they are open to residents and nationals of the United States of any gender who are Black/African American, Hispanic/Latin@, American Indian, Alaska Native, Native Hawaiian, or Pacific Islander.

If you want to apply to an internship in Debian, you should take a look at the wiki page, and contact the mentors for the projects listed, or seek more information on the (public) debian-outreach mailing-list. You can also contact the Outreach Team directly. If you have a project idea and are willing to mentor an intern, you can submit a project idea on the Outreachy wiki page.

Here's a few words on what the interns for the last round achieved within Outreachy:

  • Tatiana Malygina worked on Continuous Integration for Bioinformatics applications; She has pushed more than a hundred commits to the Debian Med SVN repository over the last months, and has been sponsored for more than 20 package uploads.

  • Valerie Young worked on Reproducible Builds infrastructure, driving a complete overhaul of the database and software behind the tests.reproducible-builds.org website. Her blog contains regular updates throughout the program.

  • ceridwen worked on creating reprotest, an all-in-one tool allowing anyone to check whether a build is reproducible or not, replacing the string of ad-hoc scripts the reproducible builds team used so far. She posted regular updates on the Reproducible Builds team blog.

  • While Scarlett Clark did not complete the internship (as she found a full-time job by the mid-term evaluation!), she spent the four weeks she participated in the program providing patches for reproducible builds in Debian and KDE upstream.

Debian would not be able to participate in Outreachy without the help of the Software Freedom Conservancy, who provides administrative support for Outreachy, as well as the continued support of Debian's donors, who provide funding for the internships. If you want to donate, please get in touch with one of our trusted organizations.

Debian is looking forward to welcoming new interns for the next few months, come join us!

Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Sat, 08/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Fri, 07/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Thu, 06/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Wed, 05/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs

Steve Kemp: This blog has moved

Tue, 04/10/2016 - 18:30
This blog has moved to https://blog.steve.fi/. Please update to use the new feed location.
Categories: LUG Community Blogs