LUG Community Blogs

Daniel Silverstone (Kinnison): USB Device Stacks, on RTFM, part 2

Planet ALUG - Sat, 05/08/2017 - 17:08

Previously we talked about all the different kinds of descriptors which USB devices use to communicate their capability. This is important stuff because to write any useful USB device firmware we need to be able to determine how to populate our descriptors. However, having that data on the device is entirely worthless without an understanding of how it gets from the device to the host so that it can be acted upon. To understand that, let's look at the USB wire protocol.

Note, I'll again be talking mostly about USB2.0 low- and full-speed. I believe that high speed is approximately the same but with faster wires, except not quite that simple.

Down to the wire

I don't intend to talk about the actual electrical signalling, though it's not un-reasonable for you to know that USB is a pair of wires forming a differentially signalled bidirectional serial communications link. The host is responsible for managing all the framing and timing on the link, and for formatting the communications into packets.

There are a number of packet types which can appear on the USB link:

Packet type Purpose Token Packet When the host wishes to send a message to the Control endpoint to configure the device, read data IN, or write data OUT, it uses this to start the transaction. Data(0/1) Packet Following a Setup, In, or Out token, a Data packet is a transfer of data (in either direction). The 0 and 1 alternate to provide a measure of confidence against lost packets. Handshake Packet Following a data packet of some kind, the other end may ACK the packet (all was well), NAK the packet (report that the device cannot, temporarily, send/receive data, or that an interrupt endpoint isn't triggered), or STALL the bus in which case the host needs to intervene. Start of Frame Every 1ms (full-speed) the host will send a SOF packet which carries a frame number. This can be used to help keep time on very simple devices. It also divides the bus into frames within which bandwidth is allocated.

As an example, when the host wishes to perform a control transfer, the following packets are transacted in turn:

  1. Setup Token - The host addresses the device and endpoint (OUT0)
  2. Data0 Packet - The host transmits a GET_DESCRIPTOR for the device descriptor
  3. Ack Packet - The device acknowledges receipt of the request

This marks the end of the first transaction. The device decodes the GET_DESCRIPTOR request and prepares the device descriptor for transmission. The transmission occurs as the next transaction on the bus. In this example, we're assuming 8 byte maximum transmission sizes, for illustrative purposes.

  1. In Token - The host addresses the device and endpoint (IN0)
  2. Data1 Packet - The device transmits the first 8 bytes of the descriptor
  3. Ack Packet - The host acknowledges the data packet
  4. In Token - The host addresses the device and endpoint (IN0)
  5. Data0 Packet - The device transmits the remaining 4 bytes of the descriptor (padded)
  6. Ack Packet - The host acknowledges the data packet

The second transaction is now complete, and the host has all the data it needs to proceed. Finally a status transaction occurs in which:

  1. Out Token - The host addresses the device and endpoint (OUT0)
  2. Data1 Packet - The host transmits a 0 byte data packet to indicate successful completion
  3. Ack Packet - The device acknowledges the completion, indicating its own satisfaction

And thus ends the full control transaction in which the host retrieves the device descriptor.

From a high level, we need only consider the activity which occurs at the point of the acknowledgement packets. In the above example:

  1. On the first ACK the device prepares IN0 to transmit the descriptor, readying whatever low level device stack there is with a pointer to the descriptor and its length in bytes.
  2. On the second ACK the low levels are still thinking.
  3. On the third ACK the transmission from IN0 is complete and the endpoint no longer expects to transfer data.
  4. On the fourth ACK the control transaction is entirely complete.
Thinking at the low levels of the control interface

Before we can build a high level USB stack, we need to consider the activity which might occur at the lower levels. At the low levels, particularly of the device control interface, work has to be done at each and every packet. The hardware likely deals with the token packet for us, leaving the data packets for us to process, and the resultant handshake packets will be likely handled by the hardware in response to our processing the data packets.

Since every control transaction is initiated by a setup token, let's look at the setup requests which can come our way...

Setup Packet (Data) Format Field Name Byte start Byte length Encoding Meaning bmRequestType 0 1 Bitmap Describes the kind of request, and the target of it. See below. bRequest 1 1 Code The request code itself, meanings of the rest of the fields vary by bRequest wValue 2 2 Number A 16 bit value whose meaning varies by request type wIndex 4 2 Number A 16 bit value whose meaning varies by request type but typically encodes an interface number or endpoint. wLength 6 2 Number A 16 bit value indicating the length of the transfer to come.

Since bRequest is essentially a switch against which multiple kinds of setup packet are selected between, here's the meanings of a few...

GET_DESCRIPTOR (Device) setup packet Field Name Value Meaning bmRequestType 0x08 Data direction is IN (from device to host), recipient is the device bRequest 0x06 GET_DESCRIPTOR (in this instance, the device descriptor is requested) wValue 0x0001 This means the device descriptor wIndex 0x0000 Irrelevant, there's only 1 device descriptor anyway wLength 12 This is the length of a device descriptor (12 bytes) SET_ADDRESS to set a device's USB address Field Name Value Meaning bmRequestType 0x00 Data direction is OUT (from host to device), recipient is the device bRequest 0x05 SET_ADDRESS (Set the device's USB address) wValue 0x00nn The address for the device to adopt (max 127) wIndex 0x0000 Irrelevant for address setting wLength 0 There's no data transfer expected for this setup operation

Most hardware blocks will implement an interrupt at the point that the Data packet following the Setup packet has been receive. This is typically called receiving a 'Setup' packet and then it's up to the device stack low levels to determine what to do and dispatch a handler. Otherwise an interrupt will fire for the IN or OUT tokens and if the endpoint is zero, the low level stack will handle it once more.

One final thing worth noting about SET_ADDRESS is that it doesn't take effect until the completion of the zero-length "status" transaction following the setup transaction. As such, the status request from the host will still be sent to address zero (the default for new devices).

A very basic early "packet trace"

This is an example, and is not guaranteed to be the packet sequence in all cases. It's a good indication of the relative complexity involved in getting a fresh USB device onto the bus though...

When a device first attaches to the bus, the bus is in RESET state and so the first event a device sees is a RESET which causes it to set its address to zero, clear any endpoints, clear the configuration, and become ready for control transfers. Shortly after this, the device will become suspended.

Next, the host kicks in and sends a port reset of around 30ms. After this, the host is ready to interrogate the device.

The host sends a GET_DESCRIPTOR to the device, whose address at this point is zero. Using the information it receives from this, it can set up the host-side memory buffers since the device descriptor contains the maximum transfer size which the device supports.

The host is now ready to actually 'address' the device, and so it sends another reset to the device, again around 30ms in length.

The host sends a SET_ADDRESS control request to the device, telling it that its new address is nn. Once the acknowledgement has been sent from the host for the zero-data status update from the device, the device sets its internal address to the value supplied in the request. From now on, the device shall respond only to requests to nn rather than to zero.

At this point, the host will begin interrogating further descriptors, looking at the configuration descriptors and the strings, to build its host-side representation of the device. These will be GET_DESCRIPTOR and GET_STRING_DESCRIPTOR requests and may continue for some time.

Once the host has satisfied itself that it knows everything it needs to about the device, it will issue a SET_CONFIGURATION request which basically starts everything up in the device. Once the configuration is set, interrupt endpoints will be polled, bulk traffic will be transferred, Isochronous streams begin to run, etc.

Okay, but how do we make this concrete?

So far, everything we've spoken about has been fairly abstract, or at least "soft". But to transfer data over USB does require some hardware. (Okay, okay, we could do it all virtualised, but there's no fun in that). The hardware I'm going to be using for the duration of this series is the STM32 on the blue-pill development board. This is a very simple development board which does (in theory at least) support USB device mode.

If we view the schematic for the blue-pill, we can see a very "lightweight" USB interface which has a pullup resistor for D+. This is the way that a device signals to the host that it is present, and that it wants to speak at full-speed. If the pullup were on D- then it would be a low-speed device. High speed devices need a little more complexity which I'm not going to go into for today.

The USB lines connect to pins PA11 and PA12 which are the USB pins on the STM32 on the board. Since USB is quite finicky, the STM32 doesn't let you remap that function elsewhere, so this is all looking quite good for us so far.

The specific STM32 on the blue-pill is the STM32F103C8T6. By viewing its product page on ST's website we can find the reference manual for the part. Jumping to section 23 we learn that this STM32 supports full-speed USB2.0 which is convenient given the past article and a half. We also learn it supports up to eight endpoints active at any one time, and offers double-buffering for our bulk and isochronous transfers. It has some internal memory for packet buffering, so it won't use our RAM bandwidth while performing transfers, which is lovely.

I'm not going to distill the rest of that section here, because there's a large amount of data which explains how the USB macrocell operates. However useful things to note are:

  • How IN OUT and SETUP transfers work.
  • How the endpoint buffer memory is configured.
  • That all bus-powered devices MUST respond to suspend/resume properly
  • That the hardware will prioritise endpoint interrupts for us so that we only need deal with the most pressing item at any given time.
  • There is an 'Enable Function' bit in the address register which must be set or we won't see any transactions at all.
  • How the endpoint registers signal events to the device firmware.

Next time, we're going to begin the process of writing a very hacky setup routine to try and initialise the USB device macrocell so that we can see incoming transactions through the ITM. It should be quite exciting, but given how complex this will be for me to learn, it might be a little while before it comes through.

Categories: LUG Community Blogs

Debian Bits: DebConf17 Open Day

Planet HantsLUG - Sat, 05/08/2017 - 15:00

Today, the day preceeding the official start of the annual Debian Conference, is the Open Day at DebConf17, at Collège Maisonneuve in Montreal (Canada).

This day is open to the public with events of interest to a wide audience.

The schedule of today's events include, among others:

  • A Newbie's Newbie Guide to Debian
  • Ask Anything About Debian
  • Debian Packaging 101
  • Debian InstallFest
  • Presentations or workshops related to free software projects and local organizations.

Everyone is welcome to attend! It is a great possibility for interested users to meet our community and for Debian to widen our community.

See the full schedule for today's events at https://debconf17.debconf.org/schedule/open-day/.

If you want to engage remotely, you can watch the video streaming of the Open Day events happening in the "Rex" room, or join the conversation in the channels #debconf17-rex, #debconf17-potato and #debconf17-woody in the OFTC IRC network.

DebConf is committed to a safe and welcome environment for all participants. See the DebConf Code of Conduct and the Debian Code of Conduct for more details on this.

Debian thanks the commitment of numerous sponsors to support DebConf17, particularly our Platinum Sponsors Savoir-Faire Linux, Hewlett Packard Enterprise, and Google.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): USB Device Stacks, on RTFM

Planet ALUG - Fri, 04/08/2017 - 17:05

I have been spending time with Jorge Aparicio's RTFM for Cortex M3 framework for writing Rust to target Cortex-M3 devices from Arm (and particularly the STM32F103 from ST Microelectronics). Jorge's work in this area has been of interest to me ever since I discovered him working on this stuff a while ago. I am very tempted by the idea of being able to implement code for the STM32 with the guarantees of Rust and the language features which I have come to love such as the trait system.

I have been thinking to myself that, while I admire and appreciate the work done on the GNUK, I would like to, personally, have a go at implementing some kind of security token on an STM32 as a USB device. And with the advent of the RTFM for M3 work, and Jorge's magical tooling to make it easier to access and control the registers on an M3 microcontroller, I figured it'd be super-nice to do this in Rust, with all the advantages that entails in terms of isolating unsafe behaviour and generally having the potential to be more easily verified as not misbehaving.

To do this though, means that I need a USB device stack which will work in the RTFM framework. Sadly it seems that, thus-far, only Jorge has been working on drivers for any of the M3 devices his framework supports. And one person can only do so much. So, in my infinite madness, I decided I should investigate the complexity of writing a USB device stack in Rust for the RTFM/M3 framework. (Why I thought this was a good idea is lost to the mists of late night Googling, but hey, it might make a good talk at the next conference I go to). As such, this blog post, and further ones along these lines, will serve as a partial tour of what I'm up to, and a partial aide-memoir for me about learning USB. If I get something horribly wrong, please DO contact me to correct me, otherwise I'll just continue to be wrong. If I've simplified something but it's still strictly correct, just let me know if it's an oversimplification since in a lot of cases there's no point in me putting the full details into a blog posting. I will mostly be considering USB2.0 protocol details but only really for low and full speed devices. (The hardware I'm targetting does low-speed and full-speed, but not high-speed. Though some similar HW does high-speed too, I don't have any to hand right now)

A brief introduction to USB

In order to go much further, I needed a grounding in USB. It's a multi-layer protocol as you might expect, though we can probably ignore the actual electrical layer since any device we might hope to support will have to have a hardware block to deal with that. We will however need to consider the packet layer (since that will inform how the hardware block is implemented and thus its interface) and then the higher level protocols on top.

USB is a deliberately asymmetric protocol. Devices are meant to be significantly easier to implement, both in terms of hardware and software, as compared with hosts. As such, despite some STM32s having OTG ports, I have no intention of supporting host mode at this time.

USB is arranged into a set of busses which are, at least in the USB1.1 case, broadcast domains. As such, each device has an address assigned to it by the host during an early phase called 'configuration'. Once the address is assigned, the device is expected to only ever respond to messages addressed to it. Note that since everything is asymmetric in USB, the device can't send messages on its own, but has to be asked for them by the host, and as such the addressing is always from host toward device.

USB devices then expose a number of endpoints through which communication can flow IN to the host or OUT to the device. Endpoints are not bidirectional, but the in and out endpoints do overlap in numbering. There is a special pair of endpoints, IN0 and OUT0 which, between them, form what I will call the device control endpoints. The device control endpoints are important since every USB device MUST implement them, and there are a number of well defined messages which pass over them to control the USB device. In theory a bare minimum USB device would implement only the device control endpoints.

Configurations, and Classes, and Interfaces, Oh My!

In order for the host to understand what the USB device is, and what it is capable of, part of the device control endpoints' responsibility is to provide a set of descriptors which describe the device. These descriptors form a heirarchy and are then glommed together into a big lump of data which the host can download from the device in order to decide what it is and how to use it. Because of various historical reasons, where a multi-byte value is used, they are defined to be little-endian, though there are some BCD fields. Descriptors always start with a length byte and a type byte because that way the host can parse/skip as necessary, with ease.

The first descriptor is the device descriptor, is a big one, and looks like this:

Device Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (18) bDescriptorType 1 1 Constant Device Descriptor (0x01) bcdUSB 2 2 BCD USB spec version compiled with bDeviceClass 4 1 Class Code, assigned by USB org (0 means "Look at interface descriptors", common value is 2 for CDC) bDeviceSubClass 5 1 SubClass Code, assigned by USB org (usually 0) bDeviceProtocol 6 1 Protocol Code, assigned by USB org (usually 0) bMaxPacketSize 7 1 Number Max packet size for IN0/OUT0 (Valid are 8, 16, 32, 64) idVendor 8 2 ID 16bit Vendor ID (Assigned by USB org) idProduct 10 2 ID 16bit Product ID (Assigned by manufacturer) bcdDevice 12 2 BCD Device version number (same encoding as bcdUSB) iManufacturer 14 1 Index String index of manufacturer name (0 if unavailable) iProduct 15 1 Index String index of product name (0 if unavailable) iSerialNumber 16 1 Index String index of device serial number (0 if unavailable) bNumConfigurations 17 1 Number Count of configurations the device has.

This looks quite complex, but breaks down into a relatively simple two halves. The first eight bytes carries everything necessary for the host to be able to configure itself and the device control endpoints properly in order to communicate effectively. Since eight bytes is the bare minimum a device must be able to transmit in one go, the host can guarantee to get those, and they tell it what kind of device it is, what USB protocol it supports, and what the maximum transfer size is for its device control endpoints.

The encoding of the bcdUSB and bcdDevice fields is interesting too. It is of the form 0xMMmm where MM is the major number, mm the minor. So USB2.0 is encoded as 0x0200, USB1.1 as 0x0110 etc. If the device version is 17.36 then that'd be 0x1736.

Other fields of note are bDeviceClass which can be 0 meaning that interfaces will specify their classes, and idVendor/idProduct which between them form the primary way for the specific USB device to be identified. The Index fields are indices into a string table which we'll look at later. For now it's enough to know that wherever a string index is needed, 0 can be provided to mean "no string here".

The last field is bNumConfigurations and this indicates the number of ways in which this device might function. A USB device can provide any number of these configurations, though typically only one is provided. If the host wishes to switch between configurations then it will have to effectively entirely quiesce and reset the device.

The next kind of descriptor is the configuration descriptor. This one is much shorter, but starts with the same two fields:

Configuration Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (9) bDescriptorType 1 1 Constant Configuration Descriptor (0x02) wTotalLength 2 2 Number Size of the configuration in bytes, in total bNumInterfaces 4 1 Number The number of interfaces in this configuration bConfigurationValue 5 1 Number The value to use to select this configuration iConfiguration 6 1 Index The name of this configuration (0 for unavailable) bmAttributes 7 1 Bitmap Attributes field (see below) bMaxPower 8 1 Number Maximum bus power this configuration will draw (in 2mA increments)

An important field to consider here is the bmAttributes field which tells the host some useful information. Bit 7 must be set, bit 6 is set if the device would be self-powered in this configuration, bit 5 indicates that the device would like to be able to wake the host from sleep mode, and bits 4 to 0 must be unset.

The bMaxPower field is interesting because it encodes the power draw of the device (when set to this configuration). USB allows for up to 100mA of draw per device when it isn't yet configured, and up to 500mA when configured. The value may be used to decide if it's sensible to configure a device if the host is in a low power situation. Typically this field will be set to 50 to indicate the nominal 100mA is fine, or 250 to request the full 500mA.

Finally, the wTotalLength field is interesting because it tells the host the total length of this configuration, including all the interface and endpoint descriptors which make it up. With this field, the host can allocate enough RAM to fetch the entire configuration descriptor block at once, simplifying matters dramatically for it.

Each configuration has one or more interfaces. The interfaces group some endpoints together into a logical function. For example a configuration for a multifunction scanner/fax/printer might have an interface for the scanner function, one for the fax, and one for the printer. Endpoints are not shared among interfaces, so when building this table, be careful.

Next, logically, come the interface descriptors:

Interface Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (9) bDescriptorType 1 1 Constant Interface Descriptor (0x04) bInterfaceNumber 2 1 Number The number of the interface bAlternateSetting 3 1 Number The interface alternate index bNumEndpoints 4 1 Number The number of endpoints in this interface bInterfaceClass 5 1 Class The interface class (USB Org defined) bInterfaceSubClass 6 1 SubClass The interface subclass (USB Org defined) bInterfaceProtocol 7 1 Protocol The interface protocol (USB Org defined) iInterface 8 1 Index The name of the interface (or 0 if not provided)

The important values here are the class/subclass/protocol fields which provide a lot of information to the host about what the interface is. If the class is a USB Org defined one (e.g. 0x02 for Communications Device Class) then the host may already have drivers designed to work with the interface meaning that the device manufacturer doesn't have to provide host drivers.

The bInterfaceNumber is used by the host to indicate this interface when sending messages, and the bAlternateSetting is a way to vary interfaces. Two interfaces with the came bInterfaceNumber but different bAlternateSettings can be switched between (like configurations, but) without resetting the device.

Hopefully the rest of this descriptor is self-evident by now.

The next descriptor kind is endpoint descriptors:

Endpoint Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (7) bDescriptorType 1 1 Constant Endpoint Descriptor (0x05) bEndpointAddress 2 1 Endpoint Endpoint address (see below) bmAttributes 3 1 Bitmap Endpoint attributes (see below) wMaxPacketSize 4 2 Number Maximum packet size this endpoint can send/receive bInterval 6 1 Number Interval for polling endpoint (in frames)

The bEndpointAddress is a 4 bit endpoint number (so there're 16 endpoint indices) and a bit to indicate IN vs. OUT. Bit 7 is the direction marker and bits 3 to 0 are the endpoint number. This means there are 32 endpoints in total, 16 in each direction, 2 of which are reserved (IN0 and OUT0) giving 30 endpoints available for interfaces to use in any given configuration. The bmAttributes bitmap covers the transfer type of the endpoint (more below), and the bInterval is an interval measured in frames (1ms for low or full speed, 125µs in high speed). bInterval is only valid for some endpoint types.

The final descriptor kind is for the strings which we've seen indices for throughout the above. String descriptors have two forms:

String Descriptor (index zero) Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (variable) bDescriptorType 1 1 Constant String Descriptor (0x03) wLangID[0] 2 2 Number Language code zero (e.g. 0x0409 for en_US) wLangID[n] 4.. 2 Number Language code n ...

This form (for descriptor 0) is that of a series of language IDs supported by the device. The device may support any number of languages. When the host requests a string descriptor, it will supply both the index of the string and also the language id it desires (from the list available in string descriptor zero). The host can tell how many language IDs are available simply by dividing bLength by 2 and subtracting 1 for the two header bytes.

And for string descriptors of an index greater than zero:

String Descriptor (index greater than zero) Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (variable) bDescriptorType 1 1 Constant String Descriptor (0x03) bString 2.. .. Unicode The string, in "unicode" format

This second form of the string descriptor is simply the the string is in what the USB spec calls 'Unicode' format which is, as of 2005, defined to be UTF16-LE without a BOM or terminator.

Since string descriptors are of a variable length, the host must request strings in two transactions. First a request for 2 bytes is sent, retrieving the bLength and bDescriptorType fields which can be checked and memory allocated. Then a request for bLength bytes can be sent to retrieve the entire string descriptor.

Putting that all together

Phew, this is getting to be quite a long posting, so I'm going to leave this here and in my next post I'll talk about how the host and device pass packets to get all that information to the host, and how it gets used.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Gitano 1.1

Planet ALUG - Thu, 03/08/2017 - 17:34

Today marks the release of Gitano 1.1. Richard(s) and I have spent quite a lot of time and effort on this release, and there's plenty of good stuff in it. We also released new versions of Lace, Supple, Luxio, and Gall to go alongside it, with bugfixes and improvements.

At this point, I intend to take a short break from Gitano to investigate some Rust-on-STM32 stuff, and then perhaps do some NetSurf work too.

Categories: LUG Community Blogs

Andy Smith: A slightly more realistic look at lvmcache

Planet HantsLUG - Thu, 03/08/2017 - 14:59
Recap And then…

I decided to perform some slightly more realistic benchmarks against lvmcache.

The problem with the initial benchmark was that it only covered 4GiB of data with a 4GiB cache device. Naturally once lvmcache was working correctly its performance was awesome – the entire dataset was in the cache. But clearly if you have enough fast block device available to fit all your data then you don’t need to cache it at all and may as well just use the fast device directly.

I decided to perform some fio tests with varying data sizes, some of which were larger than the cache device.

Test methodology

Once again I used a Zipf distribution with a factor of 1.2, which should have caused about 90% of the hits to come from just 10% of the data. I kept the cache device at 4GiB but varied the data size. The following data sizes were tested:

  • 1GiB
  • 2GiB
  • 4GiB
  • 8GiB
  • 16GiB
  • 32GiB
  • 48GiB

With the 48GiB test I expected to see lvmcache struggling, as the hot 10% (~4.8GiB) would no longer fit within the 4GiB cache device.

A similar fio job spec to those from the earlier articles was used:

[cachesize-1g] size=512m ioengine=libaio direct=1 iodepth=8 numjobs=2 readwrite=randread random_distribution=zipf:1.2 bs=4k unlink=1 runtime=30m time_based=1 per_job_logs=1 log_avg_msec=500 write_iops_log=/var/tmp/fio-${FIOTEST}

…the only difference being that several different job files were used each with a different size= directive. Note that as there are two jobs, the size= is half the desired total data size: each job lays out a data file of the specified size.

For each data size I took care to fill the cache with data first before doing a test run, as unreproducible performance is still seen against a completely empty cache device. This produced IOPS logs and a completion latency histogram. Test were also run against SSD and HDD to provide baseline figures.

Results IOPS graphs All-in-one

Immediately we can see that for data sizes 4GiB and below performance converges quite quickly to near-SSD levels. That is very much what we would expect when the cache device is 4GiB, so big enough to completely cache everything.

Let’s just have a look at the lower-performing configurations.

Low-end performers

For 8, 16 and 32GiB data sizes performance clearly gets progressively worse, but it is still much better than baseline HDD. The 10% of hot data still fits within the cache device, so plenty of acceleration is still happening.

For the 48GiB data size it is a little bit of a different story. Performance is still better (on average) than baseline HDD, but there are periodic dips back down to roughly HDD figures. This is because not all of the 10% hot data fits into the cache device any more. Cache misses cause reads from HDD and consequently end up with HDD levels of performance for those reads.

The results no longer look quite so impressive, with even the 8GiB data set achieving only a few thousand IOPS on average. Are things as bad as they seem? Well no, I don’t think they are, and to see why we will have to look at the completion latency histograms.

Completion latency histograms

The above graphs are generated by fitting a Bezier curve to a scatter of data points each of which represents a 500ms average of IOPS achieved. The problem there is the word average.

It’s important to understand what effect averaging the figures gives. We’ve already seen that HDDs are really slow. Even if only a few percent of IOs end up missing cache and going to HDD, the massive latency of those requests will pull the average for the whole 500ms window way down.

Presumably we have a cache because we suspect we have hot spots of data, and we’ve been trying to evaluate that by doing most of the reads from only 10% of the data. Do we care what the average performance is then? Well it’s a useful metric but it’s not going to say much about the performance of reads from the hot data.

The histogram of completion latencies can be more useful. This shows how long it took between issuing the IO and completing the read for a certain percentage of issued IOs. Below I have focused on the 50% to 99% latency buckets, with the times for each bucket averaged between the two jobs. In the interests of being able to see anything at all I’ve had to double the height of the graph and still cut off the y axis for the three worst performers.

A couple of observations:

  • Somewhere between 70% and 80% of IOs complete with a latency that’s so close to SSD performance as to be near-indistinguishable, no matter what the data size. So what I think I am proving is that:

    you can cache a 48GiB slow backing device with 4GiB of fast SSD and if you have 10% hot data then you can expect it to be served up at near-SSD latencies 70%–80% of the time. If your hot spots are larger (not so hot) then you won’t achieve that. If your fast device is larger than 1/12th the backing device then you should do better than 70%–80%.

  • If the cache were perfect then we should expect the 90th percentile to be near SSD performance even for the 32GiB data set, as the 10% hot spot of ~3.2GiB fits inside the 4GiB cache. For whatever reason this is not achieved, but for that data size the 90th percentile latency is still about half that of HDD.
  • When the backing device is many times larger (32GiB+) than the cache device, the 99th percentile latencies can be slightly worse than for baseline HDD.

    I hesitate to suggest there is a problem here as there are going to be very few samples in the top 1%, so it could just be showing close to HDD performance.

Conclusion

Assuming you are okay with using a 4.12..x kernel, and assuming you are already comfortable using LVM, then at the moment it looks fairly harmless to deploy lvmcache.

Getting a decent performance boost out of it though will require you to check that your data really does have hot spots and size your cache appropriately.

Measuring your existing workload with something like blktrace is probably advisable, and these days you can feed the output of blktrace back into fio to see what performance might be like in a difference configuration.

Full test output

You probably want to stop reading here unless the complete output of all the fio runs is of interest to you.

Baseline SSD $ cd /srv/ssd/fio && FIOTEST=ssd-1g fio ~/cachesize-1g.fio cachesize-1g: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes cachesize-1g: Laying out IO file(s) (1 file(s) / 512MB) cachesize-1g: Laying out IO file(s) (1 file(s) / 512MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [385.7MB/0KB/0KB /s] [98.8K/0/0 iops] [eta 00m:00s] cachesize-1g: (groupid=0, jobs=1): err= 0: pid=15206: Thu Aug 3 03:59:51 2017 read : io=346130MB, bw=196910KB/s, iops=49227, runt=1800001msec slat (usec): min=0, max=4188, avg= 4.76, stdev= 4.10 clat (usec): min=17, max=7093, avg=155.55, stdev=59.28 lat (usec): min=91, max=7098, avg=160.77, stdev=59.32 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 108], 10.00th=[ 112], 20.00th=[ 119], | 30.00th=[ 124], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 147], | 70.00th=[ 159], 80.00th=[ 181], 90.00th=[ 225], 95.00th=[ 274], | 99.00th=[ 366], 99.50th=[ 402], 99.90th=[ 474], 99.95th=[ 506], | 99.99th=[ 924] lat (usec) : 20=0.01%, 50=0.01%, 100=0.57%, 250=92.21%, 500=7.16% lat (usec) : 750=0.04%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=14.54%, sys=42.95%, ctx=50708834, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=88609406/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 cachesize-1g: (groupid=0, jobs=1): err= 0: pid=15207: Thu Aug 3 03:59:51 2017 read : io=346960MB, bw=197381KB/s, iops=49345, runt=1800001msec slat (usec): min=0, max=4180, avg= 4.76, stdev= 3.93 clat (usec): min=16, max=7097, avg=155.16, stdev=59.32 lat (usec): min=60, max=7101, avg=160.38, stdev=59.36 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 108], 10.00th=[ 113], 20.00th=[ 119], | 30.00th=[ 124], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 145], | 70.00th=[ 157], 80.00th=[ 179], 90.00th=[ 225], 95.00th=[ 274], | 99.00th=[ 370], 99.50th=[ 406], 99.90th=[ 474], 99.95th=[ 506], | 99.99th=[ 892] lat (usec) : 20=0.01%, 50=0.01%, 100=0.59%, 250=92.13%, 500=7.22% lat (usec) : 750=0.05%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=14.75%, sys=43.14%, ctx=50816422, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=88821698/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=693090MB, aggrb=394291KB/s, minb=196909KB/s, maxb=197381KB/s, mint=1800001msec, maxt=1800001msec Disk stats (read/write): xvdf: ios=177420039/9, merge=0/6, ticks=27109524/0, in_queue=27169920, util=100.00% Baseline HDD $ cd /srv/slow/fio && FIOTEST=hdd-1g fio ~/cachesize-1g.fio cachesize-1g: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes cachesize-1g: Laying out IO file(s) (1 file(s) / 512MB) cachesize-1g: Laying out IO file(s) (1 file(s) / 512MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [2650KB/0KB/0KB /s] [662/0/0 iops] [eta 00m:00s] cachesize-1g: (groupid=0, jobs=1): err= 0: pid=18951: Thu Aug 3 04:30:08 2017 read : io=2307.8MB, bw=1312.5KB/s, iops=328, runt=1800099msec slat (usec): min=0, max=101, avg=14.31, stdev= 9.09 clat (usec): min=58, max=943511, avg=24362.98, stdev=47600.55 lat (usec): min=63, max=943517, avg=24378.15, stdev=47600.62 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 82], 10.00th=[ 96], 20.00th=[ 127], | 30.00th=[ 181], 40.00th=[ 596], 50.00th=[ 1432], 60.00th=[ 9152], | 70.00th=[21632], 80.00th=[38656], 90.00th=[73216], 95.00th=[115200], | 99.00th=[228352], 99.50th=[280576], 99.90th=[415744], 99.95th=[468992], | 99.99th=[593920] lat (usec) : 100=11.17%, 250=23.28%, 500=4.34%, 750=2.55%, 1000=2.09% lat (msec) : 2=8.86%, 4=1.99%, 10=6.48%, 20=8.26%, 50=15.33% lat (msec) : 100=9.26%, 250=5.65%, 500=0.72%, 750=0.03%, 1000=0.01% cpu : usr=0.30%, sys=0.85%, ctx=572788, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=590611/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 cachesize-1g: (groupid=0, jobs=1): err= 0: pid=18952: Thu Aug 3 04:30:08 2017 read : io=2233.4MB, bw=1270.5KB/s, iops=317, runt=1800071msec slat (usec): min=0, max=113, avg=14.63, stdev= 9.18 clat (usec): min=56, max=1153.7K, avg=25167.62, stdev=48500.18 lat (usec): min=64, max=1153.8K, avg=25183.12, stdev=48500.29 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 84], 10.00th=[ 99], 20.00th=[ 135], | 30.00th=[ 199], 40.00th=[ 892], 50.00th=[ 2640], 60.00th=[12224], | 70.00th=[21888], 80.00th=[39168], 90.00th=[74240], 95.00th=[117248], | 99.00th=[234496], 99.50th=[288768], 99.90th=[428032], 99.95th=[485376], | 99.99th=[602112] lat (usec) : 100=10.22%, 250=22.53%, 500=4.15%, 750=2.20%, 1000=1.83% lat (msec) : 2=8.43%, 4=1.66%, 10=6.40%, 20=11.41%, 50=15.37% lat (msec) : 100=9.26%, 250=5.69%, 500=0.79%, 750=0.04%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.32%, sys=0.82%, ctx=551097, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=571725/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=4540.4MB, aggrb=2582KB/s, minb=1270KB/s, maxb=1312KB/s, mint=1800071msec, maxt=1800099msec Disk stats (read/write): xvde: ios=1162334/4, merge=0/1, ticks=28774452/692, in_queue=28775512, util=100.00% 1GiB data $ (cd /srv/cache/fio && FIOTEST=cachesize-1g-full fio ~/cachesize-1g.fio) cachesize-1g: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes cachesize-1g: Laying out IO file(s) (1 file(s) / 512MB) cachesize-1g: Laying out IO file(s) (1 file(s) / 512MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [381.4MB/0KB/0KB /s] [97.7K/0/0 iops] [eta 00m:00s] cachesize-1g: (groupid=0, jobs=1): err= 0: pid=31905: Thu Aug 3 01:50:57 2017 read : io=340715MB, bw=193829KB/s, iops=48457, runt=1800001msec slat (usec): min=0, max=4500, avg= 7.09, stdev= 4.48 clat (usec): min=27, max=7408, avg=155.45, stdev=48.00 lat (usec): min=94, max=7415, avg=162.99, stdev=48.04 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 112], 10.00th=[ 118], 20.00th=[ 124], | 30.00th=[ 131], 40.00th=[ 137], 50.00th=[ 143], 60.00th=[ 151], | 70.00th=[ 163], 80.00th=[ 179], 90.00th=[ 211], 95.00th=[ 245], | 99.00th=[ 322], 99.50th=[ 350], 99.90th=[ 414], 99.95th=[ 450], | 99.99th=[ 644] lat (usec) : 50=0.01%, 100=0.21%, 250=95.28%, 500=4.49%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=14.89%, sys=52.95%, ctx=39879588, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=87222941/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 cachesize-1g: (groupid=0, jobs=1): err= 0: pid=31906: Thu Aug 3 01:50:57 2017 read : io=347850MB, bw=197888KB/s, iops=49472, runt=1800001msec slat (usec): min=0, max=4535, avg= 7.06, stdev= 4.60 clat (usec): min=25, max=7437, avg=152.11, stdev=47.15 lat (usec): min=62, max=7443, avg=159.62, stdev=47.23 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 111], 10.00th=[ 116], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 147], | 70.00th=[ 157], 80.00th=[ 175], 90.00th=[ 203], 95.00th=[ 239], | 99.00th=[ 314], 99.50th=[ 342], 99.90th=[ 410], 99.95th=[ 446], | 99.99th=[ 724] lat (usec) : 50=0.01%, 100=0.26%, 250=95.76%, 500=3.97%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=15.41%, sys=53.33%, ctx=39452819, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=89049700/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=688565MB, aggrb=391716KB/s, minb=193828KB/s, maxb=197888KB/s, mint=1800001msec, maxt=1800001msec Disk stats (read/write): dm-2: ios=176255221/16, merge=0/0, ticks=26526400/152, in_queue=26612320, util=100.00%, aggrios=58763012/5471, aggrmerge=0/0, aggrticks=8781482/98230, aggrin_queue=8904092, aggrutil=100.00% dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=176289036/11, aggrmerge=0/5, aggrticks=26394628/0, aggrin_queue=26430140, aggrutil=100.00% xvdc: ios=176289036/11, merge=0/5, ticks=26394628/0, in_queue=26430140, util=100.00% dm-0: ios=176289036/16, merge=0/0, ticks=26344448/0, in_queue=26417584, util=100.00% dm-3: ios=0/16397, merge=0/0, ticks=0/294692, in_queue=294692, util=1.07%, aggrios=0/16397, aggrmerge=0/0, aggrticks=0/294580, aggrin_queue=294580, aggrutil=1.07% xvdd: ios=0/16397, merge=0/0, ticks=0/294580, in_queue=294580, util=1.07% 2GiB data $ (cd /srv/cache/fio && FIOTEST=cachesize-2g-full fio ~/cachesize-2g.fio) cachesize-2g: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes cachesize-2g: Laying out IO file(s) (1 file(s) / 1024MB) cachesize-2g: Laying out IO file(s) (1 file(s) / 1024MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [382.1MB/0KB/0KB /s] [98.4K/0/0 iops] [eta 00m:00s] cachesize-2g: (groupid=0, jobs=1): err= 0: pid=3134: Wed Aug 2 17:32:04 2017 read : io=343242MB, bw=195266KB/s, iops=48816, runt=1800001msec slat (usec): min=0, max=4825, avg= 6.98, stdev= 4.42 clat (usec): min=26, max=6629, avg=154.48, stdev=47.36 lat (usec): min=93, max=6634, avg=161.93, stdev=47.41 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 110], 10.00th=[ 115], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 137], 50.00th=[ 143], 60.00th=[ 151], | 70.00th=[ 163], 80.00th=[ 179], 90.00th=[ 207], 95.00th=[ 237], | 99.00th=[ 314], 99.50th=[ 342], 99.90th=[ 418], 99.95th=[ 458], | 99.99th=[ 1020] lat (usec) : 50=0.01%, 100=0.32%, 250=95.80%, 500=3.84%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=14.32%, sys=52.25%, ctx=39483832, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=87869839/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 cachesize-2g: (groupid=0, jobs=1): err= 0: pid=3135: Wed Aug 2 17:32:04 2017 read : io=338592MB, bw=192621KB/s, iops=48155, runt=1800001msec slat (usec): min=0, max=4485, avg= 7.00, stdev= 4.45 clat (usec): min=16, max=365626, avg=156.69, stdev=493.84 lat (usec): min=66, max=365646, avg=164.17, stdev=493.95 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 109], 10.00th=[ 114], 20.00th=[ 122], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 141], 60.00th=[ 149], | 70.00th=[ 161], 80.00th=[ 177], 90.00th=[ 205], 95.00th=[ 241], | 99.00th=[ 318], 99.50th=[ 346], 99.90th=[ 426], 99.95th=[ 474], | 99.99th=[12352] lat (usec) : 20=0.01%, 50=0.01%, 100=0.40%, 250=95.46%, 500=4.10% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=14.08%, sys=51.79%, ctx=38995321, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=86679580/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=681834MB, aggrb=387887KB/s, minb=192621KB/s, maxb=195266KB/s, mint=1800001msec, maxt=1800001msec Disk stats (read/write): dm-2: ios=174542882/20, merge=0/0, ticks=26569204/1076, in_queue=26658352, util=100.00%, aggrios=58194070/26591, aggrmerge=0/0, aggrticks=8834468/168569, aggrin_queue=9026514, aggrutil=100.00% dm-1: ios=15/46976, merge=0/0, ticks=0/8080, in_queue=8084, util=0.35%, aggrios=174561550/46008, aggrmerge=1/6963, aggrticks=26076884/8636, aggrin_queue=26119484, aggrutil=100.00% xvdc: ios=174561550/46008, merge=1/6963, ticks=26076884/8636, in_queue=26119484, util=100.00% dm-0: ios=174561536/6745, merge=0/0, ticks=26039312/1316, in_queue=26111052, util=100.00% dm-3: ios=20660/26054, merge=0/0, ticks=464092/496312, in_queue=960408, util=5.44%, aggrios=20660/26051, aggrmerge=0/3, aggrticks=464080/495444, aggrin_queue=959516, aggrutil=5.44% xvdd: ios=20660/26051, merge=0/3, ticks=464080/495444, in_queue=959516, util=5.44% 4GiB data $ (cd /srv/cache/fio && FIOTEST=cachesize-4g-full fio ~/cachesize-4g.fio) cachesize-4g: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes cachesize-4g: Laying out IO file(s) (1 file(s) / 2048MB) cachesize-4g: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [375.8MB/0KB/0KB /s] [96.2K/0/0 iops] [eta 00m:00s] cachesize-4g: (groupid=0, jobs=1): err= 0: pid=26795: Wed Aug 2 16:22:51 2017 read : io=249614MB, bw=142002KB/s, iops=35500, runt=1800001msec slat (usec): min=0, max=4469, avg= 7.00, stdev= 4.50 clat (usec): min=9, max=1363.9K, avg=215.84, stdev=3099.68 lat (usec): min=65, max=1363.9K, avg=223.29, stdev=3099.98 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 109], 10.00th=[ 114], 20.00th=[ 121], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 147], | 70.00th=[ 159], 80.00th=[ 177], 90.00th=[ 211], 95.00th=[ 255], | 99.00th=[ 354], 99.50th=[ 402], 99.90th=[ 4384], 99.95th=[35072], | 99.99th=[142336] lat (usec) : 10=0.01%, 50=0.01%, 100=0.30%, 250=94.32%, 500=5.21% lat (usec) : 750=0.06%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.03% lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=10.92%, sys=38.72%, ctx=29995970, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=63901149/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 cachesize-4g: (groupid=0, jobs=1): err= 0: pid=26796: Wed Aug 2 16:22:51 2017 read : io=239913MB, bw=136484KB/s, iops=34120, runt=1800001msec slat (usec): min=0, max=4613, avg= 7.08, stdev= 4.64 clat (usec): min=20, max=1315.4K, avg=224.86, stdev=3340.78 lat (usec): min=67, max=1315.4K, avg=232.39, stdev=3341.07 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 110], 10.00th=[ 116], 20.00th=[ 122], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 147], | 70.00th=[ 157], 80.00th=[ 177], 90.00th=[ 219], 95.00th=[ 274], | 99.00th=[ 390], 99.50th=[ 442], 99.90th=[10304], 99.95th=[39680], | 99.99th=[154624] lat (usec) : 50=0.01%, 100=0.27%, 250=92.97%, 500=6.52%, 750=0.12% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.04% lat (msec) : 100=0.02%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=10.28%, sys=37.64%, ctx=29036049, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=61417783/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=489527MB, aggrb=278486KB/s, minb=136483KB/s, maxb=142002KB/s, mint=1800001msec, maxt=1800001msec Disk stats (read/write): dm-2: ios=125296525/29, merge=0/0, ticks=27228000/6004, in_queue=27295900, util=100.00%, aggrios=41794883/193226, aggrmerge=0/0, aggrticks=9825693/38738, aggrin_queue=9881421, aggrutil=84.96% dm-1: ios=86/514016, merge=0/0, ticks=16/61332, in_queue=61376, util=2.08%, aggrios=125186978/561183, aggrmerge=0/17273, aggrticks=18938212/71188, aggrin_queue=19034916, aggrutil=85.91% xvdc: ios=125186978/561183, merge=0/17273, ticks=18938212/71188, in_queue=19034916, util=85.91% dm-0: ios=125186892/64571, merge=0/0, ticks=18904024/11972, in_queue=18966848, util=84.96% dm-3: ios=197671/1091, merge=0/0, ticks=10573040/42912, in_queue=10616040, util=33.53%, aggrios=197671/1084, aggrmerge=0/7, aggrticks=10573020/39132, aggrin_queue=10612148, aggrutil=33.53% xvdd: ios=197671/1084, merge=0/7, ticks=10573020/39132, in_queue=10612148, util=33.53% 8GiB data $ (cd /srv/cache/fio && FIOTEST=cachesize-8g-full fio ~/cachesize-8g.fio) cachesize-8g: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes cachesize-8g: Laying out IO file(s) (1 file(s) / 4096MB) cachesize-8g: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [18048KB/0KB/0KB /s] [4512/0/0 iops] [eta 00m:00s] cachesize-8g: (groupid=0, jobs=1): err= 0: pid=15599: Wed Aug 2 14:53:25 2017 read : io=13260MB, bw=7542.7KB/s, iops=1885, runt=1800146msec slat (usec): min=0, max=385, avg= 8.25, stdev= 5.69 clat (usec): min=9, max=1516.6K, avg=4231.66, stdev=26655.31 lat (usec): min=69, max=1516.6K, avg=4240.38, stdev=26657.41 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 105], 20.00th=[ 107], | 30.00th=[ 109], 40.00th=[ 116], 50.00th=[ 119], 60.00th=[ 121], | 70.00th=[ 122], 80.00th=[ 124], 90.00th=[ 159], 95.00th=[16064], | 99.00th=[113152], 99.50th=[175104], 99.90th=[358400], 99.95th=[456704], | 99.99th=[700416] lat (usec) : 10=0.01%, 100=0.45%, 250=92.98%, 500=0.41%, 750=0.05% lat (usec) : 1000=0.04% lat (msec) : 2=0.08%, 4=0.01%, 10=0.22%, 20=1.29%, 50=2.04% lat (msec) : 100=1.25%, 250=0.93%, 500=0.21%, 750=0.03%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.95%, sys=2.94%, ctx=3521460, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=3394445/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 cachesize-8g: (groupid=0, jobs=1): err= 0: pid=15600: Wed Aug 2 14:53:25 2017 read : io=13886MB, bw=7899.2KB/s, iops=1974, runt=1800112msec slat (usec): min=0, max=2743, avg= 8.26, stdev= 5.84 clat (usec): min=2, max=1515.1K, avg=4040.21, stdev=24950.67 lat (usec): min=69, max=1515.1K, avg=4048.94, stdev=24952.83 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 103], 10.00th=[ 105], 20.00th=[ 107], | 30.00th=[ 108], 40.00th=[ 110], 50.00th=[ 118], 60.00th=[ 120], | 70.00th=[ 121], 80.00th=[ 123], 90.00th=[ 157], 95.00th=[15552], | 99.00th=[109056], 99.50th=[166912], 99.90th=[333824], 99.95th=[419840], | 99.99th=[634880] lat (usec) : 4=0.01%, 100=0.56%, 250=92.98%, 500=0.31%, 750=0.05% lat (usec) : 1000=0.05% lat (msec) : 2=0.07%, 4=0.01%, 10=0.24%, 20=1.33%, 50=2.04% lat (msec) : 100=1.25%, 250=0.91%, 500=0.19%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.94%, sys=3.09%, ctx=3684550, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=3554777/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=27145MB, aggrb=15441KB/s, minb=7542KB/s, maxb=7899KB/s, mint=1800112msec, maxt=1800146msec Disk stats (read/write): dm-2: ios=6948764/45, merge=0/0, ticks=28718292/11380, in_queue=28731160, util=100.00%, aggrios=2383761/603042, aggrmerge=0/0, aggrticks=12202621/76005, aggrin_queue=12278754, aggrutil=100.00% dm-1: ios=48/1607066, merge=0/0, ticks=4/178612, in_queue=178652, util=5.79%, aggrios=6527499/1756743, aggrmerge=0/51937, aggrticks=781204/210664, aggrin_queue=991672, aggrutil=41.93% xvdc: ios=6527499/1756743, merge=0/51937, ticks=781204/210664, in_queue=991672, util=41.93% dm-0: ios=6527451/202019, merge=0/0, ticks=780348/38028, in_queue=818476, util=38.13% dm-3: ios=623784/41, merge=0/0, ticks=35827512/11376, in_queue=35839136, util=100.00%, aggrios=623784/10, aggrmerge=0/31, aggrticks=35827612/2744, aggrin_queue=35830320, aggrutil=100.00% xvdd: ios=623784/10, merge=0/31, ticks=35827612/2744, in_queue=35830320, util=100.00% 16GiB data $ (cd /srv/cache/fio && FIOTEST=cachesize-16g-full fio ~/cachesize-16g.fio) cachesize-16g: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes cachesize-16g: Laying out IO file(s) (1 file(s) / 8192MB) cachesize-16g: Laying out IO file(s) (1 file(s) / 8192MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [8288KB/0KB/0KB /s] [2072/0/0 iops] [eta 00m:00s] cachesize-16g: (groupid=0, jobs=1): err= 0: pid=2957: Wed Aug 2 13:13:01 2017 read : io=5417.5MB, bw=3081.9KB/s, iops=770, runt=1800060msec slat (usec): min=0, max=264, avg=10.14, stdev= 6.63 clat (usec): min=64, max=1972.9K, avg=10370.01, stdev=35718.97 lat (usec): min=71, max=1972.9K, avg=10380.72, stdev=35719.88 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 105], 10.00th=[ 107], 20.00th=[ 116], | 30.00th=[ 119], 40.00th=[ 121], 50.00th=[ 122], 60.00th=[ 131], | 70.00th=[ 143], 80.00th=[ 165], 90.00th=[31104], 95.00th=[65280], | 99.00th=[173056], 99.50th=[228352], 99.90th=[374784], 99.95th=[440320], | 99.99th=[618496] lat (usec) : 100=0.28%, 250=82.96%, 500=0.10%, 750=0.04%, 1000=0.03% lat (msec) : 2=0.06%, 4=0.02%, 10=0.38%, 20=3.05%, 50=6.44% lat (msec) : 100=3.82%, 250=2.43%, 500=0.36%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.40%, sys=1.39%, ctx=1393813, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1386875/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 cachesize-16g: (groupid=0, jobs=1): err= 0: pid=2958: Wed Aug 2 13:13:01 2017 read : io=6208.7MB, bw=3531.9KB/s, iops=882, runt=1800089msec slat (usec): min=0, max=437, avg=10.21, stdev= 6.74 clat (usec): min=1, max=1988.1K, avg=9046.94, stdev=31703.98 lat (usec): min=71, max=1989.2K, avg=9057.72, stdev=31704.93 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 105], 20.00th=[ 107], | 30.00th=[ 112], 40.00th=[ 119], 50.00th=[ 121], 60.00th=[ 125], | 70.00th=[ 137], 80.00th=[ 157], 90.00th=[27776], 95.00th=[57088], | 99.00th=[152576], 99.50th=[201728], 99.90th=[329728], 99.95th=[395264], | 99.99th=[577536] lat (usec) : 2=0.01%, 50=0.01%, 100=0.41%, 250=83.62%, 500=0.09% lat (usec) : 750=0.04%, 1000=0.03% lat (msec) : 2=0.05%, 4=0.01%, 10=0.39%, 20=3.06%, 50=6.51% lat (msec) : 100=3.52%, 250=2.00%, 500=0.25%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.53%, sys=1.56%, ctx=1597691, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1589415/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=11626MB, aggrb=6613KB/s, minb=3081KB/s, maxb=3531KB/s, mint=1800060msec, maxt=1800089msec Disk stats (read/write): dm-2: ios=2976172/77, merge=0/0, ticks=28759760/45044, in_queue=28805924, util=100.00%, aggrios=998518/56115, aggrmerge=0/0, aggrticks=9860773/22645, aggrin_queue=9883437, aggrutil=100.00% dm-1: ios=88/149089, merge=0/0, ticks=4/18444, in_queue=18452, util=0.58%, aggrios=2489913/161345, aggrmerge=0/6821, aggrticks=314936/21908, aggrin_queue=336800, aggrutil=17.38% xvdc: ios=2489913/161345, merge=0/6821, ticks=314936/21908, in_queue=336800, util=17.38% dm-0: ios=2489825/19183, merge=0/0, ticks=314816/4448, in_queue=319308, util=16.88% dm-3: ios=505643/74, merge=0/0, ticks=29267500/45044, in_queue=29312552, util=100.00%, aggrios=505643/12, aggrmerge=0/62, aggrticks=29267500/10580, aggrin_queue=29278060, aggrutil=100.00% xvdd: ios=505643/12, merge=0/62, ticks=29267500/10580, in_queue=29278060, util=100.00% 32GiB data $ (cd /srv/cache/fio && FIOTEST=cachesize-32g-full fio ~/cachesize-32g.fio) cachesize-32g: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes cachesize-32g: Laying out IO file(s) (1 file(s) / 16384MB) cachesize-32g: Laying out IO file(s) (1 file(s) / 16384MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [4624KB/0KB/0KB /s] [1156/0/0 iops] [eta 00m:00s] cachesize-32g: (groupid=0, jobs=1): err= 0: pid=23915: Wed Aug 2 11:42:12 2017 read : io=3378.9MB, bw=1922.5KB/s, iops=480, runt=1800108msec slat (usec): min=0, max=152, avg=11.84, stdev= 9.33 clat (usec): min=61, max=1523.8K, avg=16633.48, stdev=52554.00 lat (usec): min=68, max=1523.8K, avg=16645.89, stdev=52556.11 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108], | 30.00th=[ 117], 40.00th=[ 121], 50.00th=[ 123], 60.00th=[ 133], | 70.00th=[ 147], 80.00th=[ 8768], 90.00th=[49920], 95.00th=[104960], | 99.00th=[257024], 99.50th=[333824], 99.90th=[536576], 99.95th=[626688], | 99.99th=[880640] lat (usec) : 100=0.57%, 250=78.79%, 500=0.19%, 750=0.06%, 1000=0.05% lat (msec) : 2=0.09%, 4=0.02%, 10=0.47%, 20=2.96%, 50=6.82% lat (msec) : 100=4.70%, 250=4.22%, 500=0.94%, 750=0.11%, 1000=0.02% lat (msec) : 2000=0.01% cpu : usr=0.34%, sys=0.87%, ctx=916902, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=864974/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 cachesize-32g: (groupid=0, jobs=1): err= 0: pid=23916: Wed Aug 2 11:42:12 2017 read : io=5022.5MB, bw=2857.5KB/s, iops=714, runt=1800090msec slat (usec): min=0, max=173, avg=11.35, stdev= 9.15 clat (usec): min=62, max=1768.2K, avg=11185.81, stdev=48075.30 lat (usec): min=70, max=1768.2K, avg=11197.69, stdev=48077.75 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 120], 50.00th=[ 122], 60.00th=[ 125], | 70.00th=[ 137], 80.00th=[ 153], 90.00th=[26240], 95.00th=[63744], | 99.00th=[220160], 99.50th=[309248], 99.90th=[585728], 99.95th=[733184], | 99.99th=[1122304] lat (usec) : 100=0.46%, 250=85.40%, 500=0.22%, 750=0.06%, 1000=0.04% lat (msec) : 2=0.07%, 4=0.02%, 10=0.25%, 20=2.04%, 50=5.27% lat (msec) : 100=2.99%, 250=2.39%, 500=0.62%, 750=0.12%, 1000=0.03% lat (msec) : 2000=0.02% cpu : usr=0.50%, sys=1.27%, ctx=1372044, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1285733/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=8401.3MB, aggrb=4779KB/s, minb=1922KB/s, maxb=2857KB/s, mint=1800090msec, maxt=1800108msec Disk stats (read/write): dm-2: ios=2150707/143, merge=0/0, ticks=28767744/34836, in_queue=28804592, util=100.00%, aggrios=751272/286183, aggrmerge=0/0, aggrticks=11166794/101613, aggrin_queue=11268450, aggrutil=100.00% dm-1: ios=62/755357, merge=0/0, ticks=12/85084, in_queue=85108, util=2.67%, aggrios=1798588/823298, aggrmerge=494/28038, aggrticks=232996/102144, aggrin_queue=335068, aggrutil=15.62% xvdc: ios=1798588/823298, merge=494/28038, ticks=232996/102144, in_queue=335068, util=15.62% dm-0: ios=1799020/96389, merge=0/0, ticks=234124/20548, in_queue=254704, util=13.23% dm-3: ios=454734/6803, merge=0/0, ticks=33266248/199208, in_queue=33465540, util=100.00%, aggrios=454734/6184, aggrmerge=0/619, aggrticks=33266152/97928, aggrin_queue=33364124, aggrutil=100.00% xvdd: ios=454734/6184, merge=0/619, ticks=33266152/97928, in_queue=33364124, util=100.00% 48GiB data $ (cd /srv/cache/fio && FIOTEST=cachesize-48g-full fio ~/cachesize-48g.fio) cachesize-48g: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes cachesize-48g: Laying out IO file(s) (1 file(s) / 24576MB) cachesize-48g: Laying out IO file(s) (1 file(s) / 24576MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [4084KB/0KB/0KB /s] [1021/0/0 iops] [eta 00m:00s] cachesize-48g: (groupid=0, jobs=1): err= 0: pid=11765: Tue Aug 1 12:02:44 2017 read : io=3108.7MB, bw=1768.4KB/s, iops=442, runt=1800134msec slat (usec): min=0, max=75057, avg=12.49, stdev=100.40 clat (usec): min=61, max=7717.4K, avg=18080.36, stdev=79780.10 lat (usec): min=70, max=7717.4K, avg=18093.41, stdev=79783.33 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 120], 50.00th=[ 124], 60.00th=[ 133], | 70.00th=[ 145], 80.00th=[ 197], 90.00th=[44288], 95.00th=[103936], | 99.00th=[296960], 99.50th=[419840], 99.90th=[1028096], 99.95th=[1269760], | 99.99th=[2179072] lat (usec) : 100=0.49%, 250=81.20%, 500=0.22%, 750=0.05%, 1000=0.04% lat (msec) : 2=0.07%, 4=0.03%, 10=0.34%, 20=2.26%, 50=6.08% lat (msec) : 100=4.04%, 250=3.79%, 500=1.03%, 750=0.19%, 1000=0.07% lat (msec) : 2000=0.09%, >=2000=0.01% cpu : usr=0.37%, sys=0.78%, ctx=846462, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=795801/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 cachesize-48g: (groupid=0, jobs=1): err= 0: pid=11766: Tue Aug 1 12:02:44 2017 read : io=3181.9MB, bw=1809.1KB/s, iops=452, runt=1800123msec slat (usec): min=0, max=177, avg=12.61, stdev=10.17 clat (usec): min=51, max=4963.9K, avg=17663.67, stdev=73795.84 lat (usec): min=68, max=4963.1K, avg=17676.84, stdev=73797.60 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 105], 10.00th=[ 107], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 137], | 70.00th=[ 151], 80.00th=[ 9664], 90.00th=[48384], 95.00th=[99840], | 99.00th=[259072], 99.50th=[354304], 99.90th=[978944], 99.95th=[1269760], | 99.99th=[2179072] lat (usec) : 100=0.47%, 250=78.72%, 500=0.21%, 750=0.06%, 1000=0.05% lat (msec) : 2=0.09%, 4=0.03%, 10=0.43%, 20=2.92%, 50=7.31% lat (msec) : 100=4.70%, 250=3.92%, 500=0.82%, 750=0.12%, 1000=0.05% lat (msec) : 2000=0.08%, >=2000=0.02% cpu : usr=0.40%, sys=0.80%, ctx=867597, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=814542/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=6290.5MB, aggrb=3578KB/s, minb=1768KB/s, maxb=1809KB/s, mint=1800123msec, maxt=1800134msec Disk stats (read/write): dm-2: ios=1610343/207, merge=0/0, ticks=28775436/83448, in_queue=28860328, util=100.00%, aggrios=562278/204739, aggrmerge=0/0, aggrticks=11044750/380021, aggrin_queue=11424792, aggrutil=100.00% dm-1: ios=53/537570, merge=0/0, ticks=4/63980, in_queue=64000, util=1.97%, aggrios=1303106/589473, aggrmerge=993/17280, aggrticks=184484/78092, aggrin_queue=262524, aggrutil=11.76% xvdc: ios=1303106/589473, merge=993/17280, ticks=184484/78092, in_queue=262524, util=11.76% dm-0: ios=1304046/69309, merge=0/0, ticks=188420/16684, in_queue=205128, util=9.97% dm-3: ios=382737/7340, merge=0/0, ticks=32945828/1059400, in_queue=34005248, util=100.00%, aggrios=382737/6163, aggrmerge=0/1177, aggrticks=32945132/641880, aggrin_queue=33587056, aggrutil=100.00% xvdd: ios=382737/6163, merge=0/1177, ticks=32945132/641880, in_queue=33587056, util=100.00%
Categories: LUG Community Blogs

Steve Kemp: So I did a thing, then another thing.

Planet HantsLUG - Wed, 02/08/2017 - 22:00

So I did start a project, to write a puppet-dashboard, it is functionally complete, but the next step is to allow me to raise alerts based on failing runs of puppet - in real-time.

(i.e. Now that I have a dashboard I wish to not use it. I want to be alerted to failures, without having to remember to go look for them. Something puppet-dashboard can't do ..)

In other news a while back I slipped in a casual note about having a brain scan done, here in sunny Helsinki.

One of the cool things about that experience, in addition to being told I wasn't going to drop dead that particular day, was that the radiologist told me that I could pay €25 to get a copy of my brain data in DICOM format.

I've not yet played with this very much, but I couldn't resist a brief animation:

  • See my brain.
    • Not the best quality, or the best detail, but damn. It is my brain.
    • I shall do better with more experimentation I think.
    • After I posted it my wife, a doctor, corrected me: That wasn't a gif of my brain, instead it was a gif of my skull. D'oh!
Categories: LUG Community Blogs

Mick Morgan: a letter to our dear home secretary

Planet ALUG - Wed, 02/08/2017 - 15:56

Dear Amber

So,”real people” don’t care about privacy? All they really want is ease of use and a pretty GUI so that they can chat to all their friends on-line? Only “the enemy” (who is that exactly anyway?) needs encryption? Excuse me for asking, but what have you been smoking? Does the Home Office know about that?

I’m a real person. And I care deeply about privacy. I care enough to fund both my own Tor node and various openVPN servers dotted around the world just to get past your ludicrous attempts at gratuitous surveillance of my (and my family’s) routine use of the ‘net. I care about the security and privacy of my transactions with various commercial enterprises, including my bank (which is why I expect them to use TLS on their website). I care about privacy when I correspond with my Doctor and other professionals. I care about privacy when I use an on-line search engine (which, incidentally, is not Google). I care about privacy because privacy matters. I have the right to freedom of thought and expression. I have the right to discuss those thoughts with others of my choice – when I choose and how I choose. You may not like that, but it’s a fact of life. That doesn’t make me “the enemy”. Get over it.

Love and Kisses

Mick

(Note to readers: Aral Balkan has deconstructed Rudd’s ramblings. I commend the article to you.)

Categories: LUG Community Blogs

Jonathan McDowell: How to make a keyring

Planet ALUG - Mon, 31/07/2017 - 21:17

Every month or two keyring-maint gets a comment about how a key update we say we’ve performed hasn’t actually made it to the active keyring, or a query about why the keyring is so out of date, or told that although a key has been sent to the HKP interface and that is showing the update as received it isn’t working when trying to upload to the Debian archive. It’s frustrating to have to deal with these queries, but the confusion is understandable. There are multiple public interfaces to the Debian keyrings and they’re not all equal. This post attempts to explain the interactions between them, and how I go about working with them as part of the keyring-maint team.

First, a diagram to show the different interfaces to the keyring and how they connect to each other:

Public interfaces rsync: keyring.debian.org::keyrings

This is the most important public interface; it’s the one that the Debian infrastructure uses. It’s the canonical location of the active set of Debian keyrings and is what you should be using if you want the most up to date copy. The validity of the keyrings can be checked using the included sha512sums.txt file, which will be signed by whoever in keyring-maint did the last keyring update.

HKP interface: hkp://keyring.debian.org/

What you talk to with gpg --keyserver keyring.debian.org. Serves out the current keyrings, and accepts updates to any key it already knows about (allowing, for example, expiry updates, new subkeys + uids or new signatures without the need to file a ticket in RT or otherwise explicitly request it). Updates sent to this interface will be available via it within a few hours, but must be manually folded into the active keyring. This in general happens about once a month when preparing for a general update of the keyring; for example b490c1d5f075951e80b22641b2a133c725adaab8.

Why not do this automatically? Even though the site uses GnuPG to verify incoming updates there are still occasions we’ve seen bugs (such as #787046, where GnuPG would always import subkeys it didn’t understand, even when that subkey was already present). Also we don’t want to allow just any UID to be part of the keyring. It is thus useful to retain a final set of human based sanity checking for any update before it becomes part of the keyring proper.

Alioth/anonscm: https://anonscm.debian.org/git/keyring/keyring.git/

A public mirror of the git repository the keyring-maint team use to maintain the keyring. Every action is recorded here, and in general each commit should be a single action (such as adding a new key, doing a key replacement or moving a key between keyrings). Note that pulling in the updates sent via HKP count as a single action, rather than having a commit per key updated. This mirror is updated whenever a new keyring is made active (i.e. made available via the rsync interface). Until that point pending changes are kept private; we sometimes deal with information such as the fact someone has potentially had a key compromised that we don’t want to be public until we’ve actually disabled it. Every “keyring push” (as we refer to the process of making a new keyring active) is tagged with the date it was performed. Releases are also tagged with their codenames, to make it easy to do comparisons over time.

Debian archive

This is actually the least important public interface to the keyring, at least from the perspective of the keyring-maint team. No infrastructure makes use of it and while it’s mostly updated when a new keyring is made active we only make a concerted effort to do so when it is coming up to release. It’s provided as a convenience package rather than something which should be utilised for active verification of which keys are and aren’t currently part of the keyring.

Team interface Master repository: kaufmann.debian.org:/srv/keyring.debian.org/master-keyring.git

The master git repository for keyring maintenance is stored on kaufmann.debian.org AKA keyring.debian.org. This system is centrally managed by DSA, with only DSA and keyring-maint having login rights to it. None of the actual maintenance work takes place here; it is a bare repo providing a central point for the members of keyring-maint to collaborate around.

Private interface Private working clone

This is where all of the actual keyring work happens. I have a local clone of the repository from kaufmann on a personal machine. The key additions / changes I perform all happen here, and are then pushed to the master repository so that they’re visible to the rest of the team. When preparing to make a new keyring active the changes that have been sent to the HKP interface are copied from kaufmann via scp and folded in using the pull-updates script. The tree is assembled into keyrings with a simple make and some sanity tests performed using make test. If these are successful the sha512sums.txt file is signed using gpg --clearsign and the output copied over to kaufmann. update-keyrings is then called to update the active keyrings (both rsync + HKP). A git push public pushes the changes to the public repository on anonscm. Finally gbp buildpackage --git-builder='sbuild -d sid' tells git-buildpackage to use sbuild to build a package ready to be uploaded to the archive.

Hopefully that helps explain the different stages and outputs of keyring maintenance; I’m aware that it would be a good idea for this to exist somewhere on keyring.debian.org as well and will look at doing so.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): F/LOSS activity, July 2017

Planet ALUG - Mon, 31/07/2017 - 20:10

Once again, my focus was on Gitano, which we're working toward a 1.1 for. We had another one of our Gitano developer days which was attended by Richard maw and myself. You are invited to read the wiki page but a summary of what happened, which directly involved me, is:

  • Once again, we reviewed our current task state
  • We had a good discussion about our code of conduct including adopting a small change from upstream to improve matters
  • I worked on, and submitted a patch for, improving nested error message reports in Lace.
  • I reviewed and merged some work from Richard about pattern centralisation
  • I responded to comments on a number of in-flight series Richard had reviewed for me.
  • We discussed our plans for 1.1 and agreed that we'll be skipping a developer day in August because so much of it is consumed by DebConf and so on.

Other than that, related to Gitano during July I:

  • Submitted some code series before the developer day covering Gall cleanups and hook support in Gitano.
  • Reviewed and merged some more Makefile updates from Richard Ipsum
  • Reviewed and merged a Supple fix for environment cleardown from Richard Ipsum
  • Fixed an issue in one of the Makefiles which made it harder to build with dh-lua
  • I began work in earnest on Gitano CI, preparing a lot of scripts and support to sit around Jenkins (for now) for CIing packaging etc for Gitano and Debian
  • I began work on a system branch concept for Gitano CI which will let us handle the CI of branches in the repos, even if they cross repos.

I don't think I've done much non-Gitano F/LOSS work in July, but I am now in Montréal for debconf 2017 so hopefully more to say next month.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in July 2017

Planet ALUG - Mon, 31/07/2017 - 18:35

Here is my monthly update covering what I have been doing in the free software world during July 2017 (previous month):

  • Updated travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds:
    • Moved the default mirror from ftp.de.debian.org to deb.debian.org. []
    • Create a sensible debian/changelog file if one does not exist. []
  • Updated django-slack, my library to easily post messages to the Slack group-messaging utility:
    • Merged a PR to clarify the error message when a channel could not be found. []
    • Reviewed and merged a suggestion to add a TestBackend. []
  • Added Pascal support to Louis Taylor's anyprint hack to add support for "print" statements from other languages into Python. []
  • Filed a PR against Julien Danjou's daiquiri Python logging helper to clarify an issue in the documentation. []
  • Merged a PR to Strava Enhancement Suite — my Chrome extension that improves and fixes annoyances in the web interface of the Strava cycling and running tracker — to remove Zwift activities with maps. []
  • Submitted a pull request for Redis key-value database store to fix a spelling mistake in a binary. []
  • Sent patches upstream to the authors of the OpenSVC cloud engine and the Argyll Color Management System to fix some "1204" typos.
  • Fixed a number of Python and deployment issues in my stravabot IRC bot. []
  • Correct a "1204" typo in Facebook's RocksDB key-value store. []
  • Corrected =+ typos in the Calibre e-book reader software. []
  • Filed a PR against the diaspy Python interface to the DIASPORA social network to correct the number of seconds in a day. []
  • Sent a pull request to remedy a =+ typo in sparqlwrapper, a SPARQL endpoint interface for Python. []
  • Filed a PR against Postfix Admin to fix some =+ typos. []
  • Fixed a "1042" typo in ImageJ, a Java image processing library. []
  • On a less-serious note, I filed an issue for Brad Abraham's bot for the Reddit sub-reddit to add some missing "hit the gym" advice. []

I also blogged about my recent lintian hacking and installation-birthday package.

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to permit verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

(I have generously been awarded a grant from the Core Infrastructure Initiative to fund my work in this area.)

This month I:

  • Assisted Mattia with a draft of an extensive status update to the debian-devel-announce mailing list. There were interesting follow-up discussions on Hacker News and Reddit.
  • Submitted the following patches to fix reproducibility-related toolchain issues within Debian:
  • I also submitted 5 patches to fix specific reproducibility issues in autopep8, castle-game-engine, grep, libcdio & tinymux.
  • Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.
  • Worked on publishing our weekly reports. (#114 #115, #116 & #117)

I also made the following changes to our tooling:

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

  • comparators.xml:
    • Fix EPUB "missing file" tests; they ship a META-INF/container.xml file. []
    • Misc style fixups. []
  • APK files can also be identified as "DOS/MBR boot sector". (#868486)
  • comparators.sqlite: Simplify file detection by rewriting manual recognizes call with a Sqlite3Database.RE_FILE_TYPE definition. []
  • comparators.directory:
    • Revert the removal of a try-except. (#868534)
    • Tidy module. []

strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.

  • Add missing File::Temp imports in the JAR and PNG handlers. This appears to have been exposed by lazily-loading handlers in #867982. (#868077)

buildinfo.debian.net

buildinfo.debian.net is my experiment into how to process, store and distribute .buildinfo files after the Debian archive software has processed them.

  • Avoid a race condition between check-and-creation of Buildinfo instances. []


Debian

My activities as the current Debian Project Leader are covered in my "Bits from the DPL emails to the debian-devel-announce mailing list.

Patches contributed
  • obs-studio: Remove annoying "click wrapper" on first startup. (#867756)
  • vim: Syntax highlighting for debian/copyright files. (#869965)
  • moin: Incorrect timezone offset applied due to "84600" typo. (#868463)
  • ssss: Add a simple autopkgtest. (#869645)
  • dch: Please bump $latest_bpo_dist to current stable release. (#867662)
  • python-kaitaistruct: Remove Markdown and homepage references from package long descriptions. (#869265)
  • album-data: Correct invalid Vcs-Git URI. (#869822)
  • pytest-sourceorder: Update Homepage field. (#869125)

I also made a very large number of contributions to the Lintian static analysis tool. To avoid duplication here, I have outlined them in a separate post.

Debian LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 1014-1 for libclamunrar, a library to add unrar support to the Clam anti-virus software to fix an arbitrary code execution vulnerability.
  • Issued DLA 1015-1 for the libgcrypt11 crypto library to fix a "sliding windows" information leak.
  • Issued DLA 1016-1 for radare2 (a reverse-engineering framework) to prevent a remote denial-of-service attack.
  • Issued DLA 1017-1 to fix a heap-based buffer over-read in the mpg123 audio library.
  • Issued DLA 1018-1 for the sqlite3 database engine to prevent a vulnerability that could be exploited via a specially-crafted database file.
  • Issued DLA 1019-1 to patch a cross-site scripting (XSS) exploit in phpldapadmin, a web-based interface for administering LDAP servers.
  • Issued DLA 1024-1 to prevent an information leak in nginx via a specially-crafted HTTP range.
  • Issued DLA 1028-1 for apache2 to prevent the leakage of potentially confidential information via providing Authorization Digest headers.
  • Issued DLA 1033-1 for the memcached in-memory object caching server to prevent a remote denial-of-service attack.
Uploads
  • redis:
    • 4:4.0.0-1 — Upload new major upstream release to unstable.
    • 4:4.0.0-2 — Make /usr/bin/redis-server in the primary package a symlink to /usr/bin/redis-check-rdb in the redis-tools package to prevent duplicate debug symbols that result in a package file collision. (#868551)
    • 4:4.0.0-3 — Add -latomic to LDFLAGS to avoid a FTBFS on the mips & mipsel architectures.
    • 4:4.0.1-1 — New upstream version. Install 00-RELEASENOTES as the upstream changelog.
    • 4:4.0.1-2 — Skip non-deterministic tests that rely on timing. (#857855)
  • python-django:
    • 1:1.11.3-1 — New upstream bugfix release. Check DEB_BUILD_PROFILES consistently, not DEB_BUILD_OPTIONS.
  • bfs:
    • 1.0.2-2 & 1.0.2-3 — Use help2man to generate a manpage.
    • 1.0.2-4 — Set hardening=+all for bindnow, etc.
    • 1.0.2-5 & 1.0.2-6 — Don't use upstream's release target as it overrides our CFLAGS & install RELEASES.md as the upstream changelog.
    • 1.1-1 — New upstream release.
  • libfiu:
    • 0.95-4 — Apply patch from Steve Langasek to fix autopkgtests. (#869709)
  • python-daiquiri:
    • 1.0.1-1 — Initial upload. (ITP)
    • 1.1.0-1 — New upstream release.
    • 1.1.0-2 — Tidy package long description.
    • 1.2.1-1 — New upstream release.

I also reviewed and sponsored the uploads of gtts-token 1.1.1-1 and nlopt 2.4.2+dfsg-3.

Debian bugs filed
  • ITP: python-daiquiri — Python library to easily setup basic logging functionality. (#867322)
  • twittering-mode: Correct incorrect time formatting due to "84600" typo. (#868479)
FTP Team

As a Debian FTP assistant I ACCEPTed 45 packages: 2ping, behave, cmake-extras, cockpit, cppunit1.13, curvedns, flask-mongoengine, fparser, gnome-shell-extension-dash-to-panel, graphene, gtts-token, hamlib, hashcat-meta, haskell-alsa-mixer, haskell-floatinghex, haskell-hashable-time, haskell-integer-logarithms, haskell-murmur-hash, haskell-quickcheck-text, haskell-th-abstraction, haskell-uri-bytestring, highlight.js, hoel, libdrm, libhtp, libpgplot-perl, linux, magithub, meson-mode, orcania, pg-dirtyread, prometheus-apache-exporter, pyee, pytest-pep8, python-coverage-test-runner, python-digitalocean, python-django-imagekit, python-rtmidi, python-transitions, qdirstat, redtick, ulfius, weresync, yder & zktop.

I additionally filed 5 RC bugs against packages that had incomplete debian/copyright files against: cockpit, cppunit, cppunit1.13, curvedns & highlight.js.

Categories: LUG Community Blogs

Andy Smith: Tracking down the lvmcache fix

Planet HantsLUG - Sun, 30/07/2017 - 07:00
Background

In the previous article I covered how, in order to get decent performance out of lvmcache with a packaged Debian kernel, you’d have to use the 4.12.2-1~exp1 kernel from experimental. The kernels packaged in sid, testing (buster) and stable (stretch) aren’t new enough.

I decided to bisect the Linux kernel upstream git repository to find out exactly which commit(s) fixed things.

Results

Here’s a graph showing the IOPS over time for baseline SSD and lvmcache with a full cache under several different kernel versions. As in previous articles, the lines are actually Bezier curves fitted to the data which is scattered all over the place from 500ms averages.

What we can see here is that performance starts to improve with commit 4d44ec5ab751 authored by Joe Thornber:

dm cache policy smq: put newly promoted entries at the top of the multiqueue

This stops entries bouncing in and out of the cache quickly.

This is part of a set of commits authored by Joe Thornber on the drivers/md/dm-cache-policy-smq.c

file and committed on 2017-05-14. By the time we reach commit 6cf4cc8f8b3b we have the long-term good performance that we were looking for.

The first of Joe Thornber’s commits on that day in the dm-cache area was 072792dcdfc8 and stepping back to the commit immediately prior to that one (2ea659a9ef48) we get a kernel representing the moment that Linus designated the v4.12-rc1 tag. Joe’s commits went into -rc1, and without them the performance of lvmcache under these test conditions isn’t much better than baseline HDD.

It seems like some of Joe’s changes helped a lot and then the last one really provided the long term performance.

git bisect procedure

Normally when you do a git bisect you’re starting with something that works and you’re looking for the commit that introduced a bug. In this case I was starting off with a known-good state and was interested in which commit(s) got me there. The normal bisect key words of “good” and “bad” in this case would be backwards to what I wanted. Dominic gave me the tip that I could alias the terms in order to reduce my confusion:

$ git bisect start --term-old broken --term-new fixed

From here on, when I encountered a test run that produced poor results I would issue:

$ git bisect broken

and when I had a test run with good results I would issue:

$ git bisect fixed

As I knew that the tag v4.13-rc1 produced a good run and v4.11 was bad, I could start off with:

$ git bisect reset v4.13-rc1 $ git bisect fixed $ git bisect broken v4.11

git would then keep bisecting the search space of commits until I would find the one(s) that resulted in the high performance I was looking for.

Good and bad?

As before I’m using fio to conduct the testing, with the same job specification:

ioengine=libaio direct=1 iodepth=8 numjobs=2 readwrite=randread random_distribution=zipf:1.2 bs=4k size=2g unlink=1 runtime=15m time_based=1 per_job_logs=1 log_avg_msec=500 write_iops_log=/var/tmp/fio-${FIOTEST}

The only difference from the other articles was that the run time was reduced to 15 minutes as all of the interesting behaviour happened within the first 11 minutes.

To recap, this fio job specification lays out two 2GiB files of random data and then starts two processes that perform 4kiB-sized reads against the files. Direct IO is used, in order to bypass the page cache.

A Zipfian distribution with a factor of 1.2 is used; this gives a 90/10 split where about 90% of the reads should come from about 10% of the data. The purpose of this is to simulate the hot spots of popular data that occur in real life. If the access pattern were to be perfectly and uniformly random then caching would not be effective.

In previous tests we had observed that dramatically different performance would be seen on the first run against an empty cache device compared to all other subsequent runs against what would be a full cache device. In the tests using kernels with the fix present the IOPS achieved would converge towards baseline SSD performance, whereas in kernels without the fix the performance would remain down near the level of baseline HDD. Therefore the fio tests were carried out twice.

Where to next?

I think I am going to see what happens when the cache device is pretty small in comparison to the working data.

All of the tests so far have used a 4GiB cache with 4GiB of data, so if everything got promoted it would entirely fit in cache. Not only that but the Zipf distribution makes most of the hits come from 10% of the data, so it’s actually just ~400MiB of hot data. I think it would be interesting to see what happens when the hot 10% is bigger than the cache device.

git bisect progress and test output

Unless you are particularly interested in the fio output and why I considered each one to be either fixed or broken, you probably want to stop reading now.

4.12.0-rc1+

I forgot to record the commit revision for this test so all I know is that it’s from somewhere in the first v4.12 release candidate.

The second job against an empty cache shows poor performance (8,210 average IOPS), and both jobs against a full cache show poor performance (2,141, 730 average IOPS), so this was marked broken.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [206.7MB/0KB/0KB /s] [52.9K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1275: Fri Jul 28 18:23:26 2017 read : io=126101MB, bw=143465KB/s, iops=35866, runt=900065msec slat (usec): min=3, max=25001, avg= 6.12, stdev= 4.73 clat (usec): min=19, max=808324, avg=214.83, stdev=2941.63 lat (usec): min=68, max=808334, avg=221.36, stdev=2942.04 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 105], 10.00th=[ 106], 20.00th=[ 109], | 30.00th=[ 112], 40.00th=[ 116], 50.00th=[ 120], 60.00th=[ 123], | 70.00th=[ 129], 80.00th=[ 137], 90.00th=[ 161], 95.00th=[ 207], | 99.00th=[ 362], 99.50th=[ 422], 99.90th=[27008], 99.95th=[50944], | 99.99th=[138240] lat (usec) : 20=0.01%, 50=0.01%, 100=0.23%, 250=96.86%, 500=2.52% lat (usec) : 750=0.07%, 1000=0.05% lat (msec) : 2=0.11%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.08% lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=9.28%, sys=32.19%, ctx=19968665, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=32281968/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1276: Fri Jul 28 18:23:26 2017 read : io=28868MB, bw=32843KB/s, iops=8210, runt=900062msec slat (usec): min=3, max=238, avg= 7.37, stdev= 3.18 clat (usec): min=1, max=718073, avg=964.43, stdev=8565.89 lat (usec): min=67, max=718080, avg=972.27, stdev=8566.73 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 109], | 30.00th=[ 113], 40.00th=[ 117], 50.00th=[ 120], 60.00th=[ 122], | 70.00th=[ 126], 80.00th=[ 133], 90.00th=[ 157], 95.00th=[ 237], | 99.00th=[27264], 99.50th=[51456], 99.90th=[127488], 99.95th=[164864], | 99.99th=[261120] lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.75% lat (usec) : 250=94.77%, 500=2.20%, 750=0.11%, 1000=0.08% lat (msec) : 2=0.15%, 4=0.02%, 10=0.13%, 20=0.52%, 50=0.75% lat (msec) : 100=0.35%, 250=0.16%, 500=0.01%, 750=0.01% cpu : usr=2.57%, sys=8.79%, ctx=6706909, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=7390152/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=154969MB, aggrb=176307KB/s, minb=32842KB/s, maxb=143465KB/s, mint=900062msec, maxt=900065msec Disk stats (read/write): dm-2: ios=39667388/28, merge=0/0, ticks=14142912/5308, in_queue=14158728, util=100.00%, aggrios=13263605/322030, aggrmerge=0/0, aggrticks=5479672/69964, aggrin_queue=5551697, aggrutil=100.00% dm-1: ios=2/847365, merge=0/0, ticks=0/131084, in_queue=131108, util=10.32%, aggrios=39457066/908046, aggrmerge=1462/48403, aggrticks=5249084/146536, aggrin_queue=5395788, aggrutil=99.89% xvdc: ios=39457066/908046, merge=1462/48403, ticks=5249084/146536, in_queue=5395788, util=99.89% dm-0: ios=39458525/110207, merge=0/0, ticks=5253332/20832, in_queue=5280728, util=99.89% dm-3: ios=332289/8519, merge=0/0, ticks=11185684/57976, in_queue=11243256, util=100.00%, aggrios=332289/6962, aggrmerge=0/1563, aggrticks=11185604/38544, aggrin_queue=11223832, aggrutil=100.00% xvdd: ios=332289/6962, merge=0/1563, ticks=11185604/38544, in_queue=11223832, util=100.00% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [14150KB/0KB/0KB /s] [3537/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1674: Fri Jul 28 18:51:41 2017 read : io=7529.9MB, bw=8566.5KB/s, iops=2141, runt=900093msec slat (usec): min=3, max=190, avg= 7.84, stdev= 5.41 clat (usec): min=34, max=1514.5K, avg=3725.26, stdev=24447.20 lat (usec): min=72, max=1514.6K, avg=3733.56, stdev=24449.04 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 103], 10.00th=[ 104], 20.00th=[ 106], | 30.00th=[ 107], 40.00th=[ 108], 50.00th=[ 109], 60.00th=[ 118], | 70.00th=[ 121], 80.00th=[ 123], 90.00th=[ 155], 95.00th=[12992], | 99.00th=[100864], 99.50th=[158720], 99.90th=[325632], 99.95th=[419840], | 99.99th=[675840] lat (usec) : 50=0.01%, 100=0.39%, 250=93.41%, 500=0.29%, 750=0.08% lat (usec) : 1000=0.06% lat (msec) : 2=0.12%, 4=0.01%, 10=0.29%, 20=1.36%, 50=1.88% lat (msec) : 100=1.12%, 250=0.82%, 500=0.17%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.73%, sys=2.52%, ctx=1993978, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1927642/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1675: Fri Jul 28 18:51:41 2017 read : io=2568.1MB, bw=2921.6KB/s, iops=730, runt=900113msec slat (usec): min=3, max=146, avg= 8.82, stdev= 5.83 clat (usec): min=64, max=1522.3K, avg=10941.33, stdev=44467.18 lat (usec): min=70, max=1522.3K, avg=10950.66, stdev=44468.17 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108], | 30.00th=[ 117], 40.00th=[ 119], 50.00th=[ 121], 60.00th=[ 123], | 70.00th=[ 135], 80.00th=[ 161], 90.00th=[26240], 95.00th=[62720], | 99.00th=[205824], 99.50th=[288768], 99.90th=[528384], 99.95th=[659456], | 99.99th=[1036288] lat (usec) : 100=0.30%, 250=83.22%, 500=0.41%, 750=0.11%, 1000=0.08% lat (msec) : 2=0.15%, 4=0.02%, 10=0.62%, 20=3.47%, 50=5.45% lat (msec) : 100=3.16%, 250=2.31%, 500=0.57%, 750=0.08%, 1000=0.02% lat (msec) : 2000=0.01% cpu : usr=0.35%, sys=0.94%, ctx=673091, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=657432/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=10098MB, aggrb=11487KB/s, minb=2921KB/s, maxb=8566KB/s, mint=900093msec, maxt=900113msec Disk stats (read/write): dm-2: ios=2585032/29, merge=0/0, ticks=14380160/2084, in_queue=14383828, util=100.00%, aggrios=897708/321791, aggrmerge=0/0, aggrticks=6187020/40796, aggrin_queue=6227841, aggrutil=100.00% dm-1: ios=34/857325, merge=0/0, ticks=4/100092, in_queue=100120, util=6.30%, aggrios=2366278/926181, aggrmerge=0/38684, aggrticks=290092/115464, aggrin_queue=405484, aggrutil=35.21% xvdc: ios=2366278/926181, merge=0/38684, ticks=290092/115464, in_queue=405484, util=35.21% dm-0: ios=2366244/108039, merge=0/0, ticks=290336/20212, in_queue=310600, util=30.59% dm-3: ios=326848/10, merge=0/0, ticks=18270720/2084, in_queue=18272804, util=100.00%, aggrios=326848/6, aggrmerge=0/4, aggrticks=18270688/1332, aggrin_queue=18272160, aggrutil=100.00% xvdd: ios=326848/6, merge=0/4, ticks=18270688/1332, in_queue=18272160, util=100.00% 4.12.0+5518b69b7668 (2017-07-05 12:31:59 -0700)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [401.3MB/0KB/0KB /s] [103K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1126: Fri Jul 28 21:06:01 2017 read : io=178623MB, bw=203233KB/s, iops=50808, runt=900001msec slat (usec): min=1, max=3059, avg= 7.04, stdev= 3.00 clat (usec): min=37, max=1175.7K, avg=147.92, stdev=585.27 lat (usec): min=66, max=1175.7K, avg=155.43, stdev=585.32 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 110], 10.00th=[ 114], 20.00th=[ 121], | 30.00th=[ 126], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 185], 95.00th=[ 207], | 99.00th=[ 274], 99.50th=[ 314], 99.90th=[ 434], 99.95th=[ 498], | 99.99th=[ 1400] lat (usec) : 50=0.01%, 100=0.42%, 250=97.80%, 500=1.72%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=14.76%, sys=51.47%, ctx=19663307, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=45727501/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1127: Fri Jul 28 21:06:01 2017 read : io=170314MB, bw=193779KB/s, iops=48444, runt=900001msec slat (usec): min=0, max=4199, avg= 7.13, stdev= 3.11 clat (usec): min=29, max=493434, avg=155.48, stdev=688.81 lat (usec): min=64, max=493446, avg=163.07, stdev=688.95 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 111], 10.00th=[ 116], 20.00th=[ 122], | 30.00th=[ 129], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 153], 80.00th=[ 167], 90.00th=[ 189], 95.00th=[ 215], | 99.00th=[ 290], 99.50th=[ 326], 99.90th=[ 446], 99.95th=[ 588], | 99.99th=[27008] lat (usec) : 50=0.01%, 100=0.32%, 250=97.35%, 500=2.27%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=13.65%, sys=50.13%, ctx=18808830, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=43600368/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=348937MB, aggrb=397012KB/s, minb=193779KB/s, maxb=203233KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=89308932/28, merge=0/0, ticks=13329748/144, in_queue=13365712, util=100.00%, aggrios=29798461/41330, aggrmerge=0/0, aggrticks=4447461/538024, aggrin_queue=4995029, aggrutil=100.00% dm-1: ios=54/56500, merge=0/0, ticks=4/10576, in_queue=10580, util=0.86%, aggrios=89368372/54448, aggrmerge=3/8823, aggrticks=12815140/10948, aggrin_queue=12836708, aggrutil=100.00% xvdc: ios=89368372/54448, merge=3/8823, ticks=12815140/10948, in_queue=12836708, util=100.00% dm-0: ios=89368321/7982, merge=0/0, ticks=12772360/1672, in_queue=12802664, util=100.00% dm-3: ios=27010/59510, merge=0/0, ticks=570020/1601824, in_queue=2171844, util=22.47%, aggrios=27010/59506, aggrmerge=0/4, aggrticks=570012/1588432, aggrin_queue=2158444, aggrutil=22.47% xvdd: ios=27010/59506, merge=0/4, ticks=570012/1588432, in_queue=2158444, util=22.47% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=0): [f(2)] [100.0% done] [398.2MB/0KB/0KB /s] [102K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1701: Fri Jul 28 21:31:14 2017 read : io=81664MB, bw=92916KB/s, iops=23228, runt=900001msec slat (usec): min=1, max=236, avg= 7.03, stdev= 2.96 clat (usec): min=20, max=1436.8K, avg=334.90, stdev=5455.47 lat (usec): min=65, max=1436.8K, avg=342.37, stdev=5455.94 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 106], 10.00th=[ 108], 20.00th=[ 116], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 135], 60.00th=[ 143], | 70.00th=[ 155], 80.00th=[ 171], 90.00th=[ 197], 95.00th=[ 225], | 99.00th=[ 322], 99.50th=[ 406], 99.90th=[57600], 99.95th=[102912], | 99.99th=[244736] lat (usec) : 50=0.01%, 100=1.15%, 250=95.96%, 500=2.51%, 750=0.04% lat (usec) : 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.07%, 50=0.11% lat (msec) : 100=0.06%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=6.80%, sys=24.11%, ctx=10402905, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=20906091/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1702: Fri Jul 28 21:31:14 2017 read : io=70584MB, bw=80309KB/s, iops=20077, runt=900001msec slat (usec): min=0, max=1925, avg= 7.09, stdev= 3.08 clat (usec): min=28, max=1511.1K, avg=388.89, stdev=6127.50 lat (usec): min=66, max=1511.1K, avg=396.42, stdev=6127.96 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 108], 10.00th=[ 114], 20.00th=[ 121], | 30.00th=[ 127], 40.00th=[ 135], 50.00th=[ 145], 60.00th=[ 155], | 70.00th=[ 169], 80.00th=[ 187], 90.00th=[ 223], 95.00th=[ 266], | 99.00th=[ 382], 99.50th=[ 470], 99.90th=[69120], 99.95th=[120320], | 99.99th=[272384] lat (usec) : 50=0.01%, 100=0.87%, 250=92.83%, 500=5.85%, 750=0.06% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.70%, sys=21.56%, ctx=9637603, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=18069579/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=152249MB, aggrb=173225KB/s, minb=80309KB/s, maxb=92915KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=38952573/29, merge=0/0, ticks=13950768/2216, in_queue=13969108, util=100.00%, aggrios=13013773/192960, aggrmerge=0/0, aggrticks=5424846/37388, aggrin_queue=5466722, aggrutil=70.38% dm-1: ios=50/513251, merge=0/0, ticks=0/63836, in_queue=63844, util=4.22%, aggrios=38843781/560322, aggrmerge=0/17355, aggrticks=5786824/73580, aggrin_queue=5863840, aggrutil=72.24% xvdc: ios=38843781/560322, merge=0/17355, ticks=5786824/73580, in_queue=5863840, util=72.24% dm-0: ios=38843731/64559, merge=0/0, ticks=5772804/12088, in_queue=5798340, util=70.38% dm-3: ios=197538/1071, merge=0/0, ticks=10501736/36240, in_queue=10537984, util=66.09%, aggrios=197538/1068, aggrmerge=0/3, aggrticks=10501736/35036, aggrin_queue=10536764, aggrutil=66.09% xvdd: ios=197538/1068, merge=0/3, ticks=10501736/35036, in_queue=10536764, util=66.09% 4.12.0-rc7+458bc30cec26 (2017-07-04 22:35:16 +0100)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc7+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [407.1MB/0KB/0KB /s] [104K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1154: Sat Jul 29 00:38:19 2017 read : io=183121MB, bw=208350KB/s, iops=52087, runt=900001msec slat (usec): min=2, max=7756, avg= 6.96, stdev= 2.90 clat (usec): min=44, max=1200.4K, avg=144.14, stdev=472.71 lat (usec): min=71, max=1200.4K, avg=151.59, stdev=472.73 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 110], 10.00th=[ 114], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 135], 60.00th=[ 143], | 70.00th=[ 151], 80.00th=[ 161], 90.00th=[ 181], 95.00th=[ 201], | 99.00th=[ 262], 99.50th=[ 298], 99.90th=[ 418], 99.95th=[ 466], | 99.99th=[ 1064] lat (usec) : 50=0.01%, 100=0.11%, 250=98.52%, 500=1.34%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=12.82%, sys=49.46%, ctx=19495852, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=46878856/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1155: Sat Jul 29 00:38:19 2017 read : io=173527MB, bw=197435KB/s, iops=49358, runt=900001msec slat (usec): min=2, max=4008, avg= 6.91, stdev= 2.74 clat (usec): min=1, max=1231.4K, avg=152.67, stdev=790.02 lat (usec): min=70, max=1231.4K, avg=160.07, stdev=790.13 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 110], 10.00th=[ 115], 20.00th=[ 121], | 30.00th=[ 126], 40.00th=[ 133], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 183], 95.00th=[ 205], | 99.00th=[ 266], 99.50th=[ 298], 99.90th=[ 430], 99.95th=[ 548], | 99.99th=[24192] lat (usec) : 2=0.01%, 50=0.01%, 100=0.10%, 250=98.42%, 500=1.42% lat (usec) : 750=0.02%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=12.64%, sys=46.41%, ctx=18940551, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=44422812/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=356647MB, aggrb=405784KB/s, minb=197434KB/s, maxb=208350KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=91291408/28, merge=0/0, ticks=13391864/1088, in_queue=13429116, util=100.00%, aggrios=30455976/37704, aggrmerge=0/0, aggrticks=4468269/553084, aggrin_queue=5032300, aggrutil=100.00% dm-1: ios=20/46840, merge=0/0, ticks=4/8444, in_queue=8452, util=0.72%, aggrios=91341450/45704, aggrmerge=35/6871, aggrticks=12941984/9032, aggrin_queue=12962160, aggrutil=100.00% xvdc: ios=91341450/45704, merge=35/6871, ticks=12941984/9032, in_queue=12962160, util=100.00% dm-0: ios=91341465/6637, merge=0/0, ticks=12928860/1388, in_queue=12963072, util=100.00% dm-3: ios=26445/59635, merge=0/0, ticks=475944/1649420, in_queue=2125376, util=22.10%, aggrios=26445/59598, aggrmerge=0/37, aggrticks=475956/1637996, aggrin_queue=2113944, aggrutil=22.10% xvdd: ios=26445/59598, merge=0/37, ticks=475956/1637996, in_queue=2113944, util=22.10% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc7+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=0): [f(2)] [100.0% done] [395.1MB/0KB/0KB /s] [101K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1692: Sat Jul 29 01:03:52 2017 read : io=80360MB, bw=91432KB/s, iops=22857, runt=900001msec slat (usec): min=2, max=2817, avg= 6.86, stdev= 2.92 clat (usec): min=29, max=1517.4K, avg=340.64, stdev=5499.29 lat (usec): min=70, max=1517.4K, avg=347.99, stdev=5499.77 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 108], 10.00th=[ 112], 20.00th=[ 120], | 30.00th=[ 124], 40.00th=[ 129], 50.00th=[ 135], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 171], 90.00th=[ 205], 95.00th=[ 247], | 99.00th=[ 346], 99.50th=[ 414], 99.90th=[57600], 99.95th=[104960], | 99.99th=[248832] lat (usec) : 50=0.01%, 100=0.11%, 250=95.21%, 500=4.30%, 750=0.03% lat (usec) : 1000=0.02% lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.07%, 50=0.11% lat (msec) : 100=0.06%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.73%, sys=21.81%, ctx=10535581, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=20572128/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1693: Sat Jul 29 01:03:52 2017 read : io=78767MB, bw=89620KB/s, iops=22404, runt=900001msec slat (usec): min=2, max=4079, avg= 6.86, stdev= 3.00 clat (usec): min=37, max=1305.7K, avg=347.72, stdev=5796.35 lat (usec): min=70, max=1305.7K, avg=355.07, stdev=5796.74 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 108], 10.00th=[ 112], 20.00th=[ 119], | 30.00th=[ 124], 40.00th=[ 129], 50.00th=[ 135], 60.00th=[ 141], | 70.00th=[ 149], 80.00th=[ 161], 90.00th=[ 181], 95.00th=[ 203], | 99.00th=[ 270], 99.50th=[ 350], 99.90th=[62208], 99.95th=[111104], | 99.99th=[264192] lat (usec) : 50=0.01%, 100=0.14%, 250=98.33%, 500=1.15%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.07%, 50=0.11% lat (msec) : 100=0.06%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.40%, sys=21.50%, ctx=9823341, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=20164463/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=159127MB, aggrb=181051KB/s, minb=89619KB/s, maxb=91431KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=40711294/29, merge=0/0, ticks=13985492/1604, in_queue=14002044, util=100.00%, aggrios=13600731/192513, aggrmerge=0/0, aggrticks=5441234/38712, aggrin_queue=5484525, aggrutil=70.85% dm-1: ios=26/511933, merge=0/0, ticks=4/63176, in_queue=63180, util=4.35%, aggrios=40605636/559333, aggrmerge=1/16864, aggrticks=5853336/73380, aggrin_queue=5930420, aggrutil=72.72% xvdc: ios=40605636/559333, merge=1/16864, ticks=5853336/73380, in_queue=5930420, util=72.72% dm-0: ios=40605611/64403, merge=0/0, ticks=5848084/12288, in_queue=5874032, util=70.85% dm-3: ios=196556/1204, merge=0/0, ticks=10475616/40672, in_queue=10516364, util=65.70%, aggrios=196556/1201, aggrmerge=0/3, aggrticks=10475564/39108, aggrin_queue=10514664, aggrutil=65.70% xvdd: ios=196556/1201, merge=0/3, ticks=10475564/39108, in_queue=10514664, util=65.70% 4.12.0-rc4+458bc30cec26 (2017-07-04 22:35:16 +0100)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc4+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [395.4MB/0KB/0KB /s] [101K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1087: Sat Jul 29 02:19:19 2017 read : io=175192MB, bw=199329KB/s, iops=49832, runt=900001msec slat (usec): min=0, max=3864, avg= 6.57, stdev= 3.22 clat (usec): min=30, max=267928, avg=151.65, stdev=87.18 lat (usec): min=69, max=267942, avg=158.71, stdev=87.17 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 110], 10.00th=[ 114], 20.00th=[ 121], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 155], 80.00th=[ 171], 90.00th=[ 203], 95.00th=[ 245], | 99.00th=[ 334], 99.50th=[ 374], 99.90th=[ 462], 99.95th=[ 516], | 99.99th=[ 1096] lat (usec) : 50=0.01%, 100=0.13%, 250=95.23%, 500=4.58%, 750=0.04% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=11.80%, sys=44.52%, ctx=20627701, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=44849083/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1088: Sat Jul 29 02:19:19 2017 read : io=169484MB, bw=192835KB/s, iops=48208, runt=900001msec slat (usec): min=0, max=920, avg= 6.65, stdev= 3.13 clat (usec): min=27, max=395221, avg=156.93, stdev=712.11 lat (usec): min=64, max=395249, avg=164.07, stdev=712.24 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 108], 10.00th=[ 113], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 167], 90.00th=[ 197], 95.00th=[ 235], | 99.00th=[ 322], 99.50th=[ 358], 99.90th=[ 466], 99.95th=[ 636], | 99.99th=[29056] lat (usec) : 50=0.01%, 100=1.35%, 250=94.81%, 500=3.76%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=11.83%, sys=43.37%, ctx=19677405, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=43387901/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=344676MB, aggrb=392163KB/s, minb=192834KB/s, maxb=199329KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=88216016/26, merge=0/0, ticks=13519572/1384, in_queue=13556832, util=100.00%, aggrios=29434288/40523, aggrmerge=0/0, aggrticks=4526756/395165, aggrin_queue=4932334, aggrutil=100.00% dm-1: ios=16/55677, merge=0/0, ticks=0/10000, in_queue=10004, util=0.85%, aggrios=88278149/52300, aggrmerge=0/9623, aggrticks=13075608/10432, aggrin_queue=13103668, aggrutil=100.00% xvdc: ios=88278149/52300, merge=0/9623, ticks=13075608/10432, in_queue=13103668, util=100.00% dm-0: ios=88278133/7958, merge=0/0, ticks=13062252/1688, in_queue=13095112, util=100.00% dm-3: ios=24715/57934, merge=0/0, ticks=518016/1173808, in_queue=1691888, util=19.21%, aggrios=24715/57931, aggrmerge=0/3, aggrticks=518016/1170636, aggrin_queue=1688652, aggrutil=19.21% xvdd: ios=24715/57931, merge=0/3, ticks=518016/1170636, in_queue=1688652, util=19.21% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc4+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=1): [f(2)] [100.0% done] [397.5MB/0KB/0KB /s] [102K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1441: Sat Jul 29 02:36:06 2017 read : io=87028MB, bw=99018KB/s, iops=24754, runt=900001msec slat (usec): min=0, max=2668, avg= 6.39, stdev= 3.49 clat (usec): min=4, max=1267.9K, avg=314.52, stdev=5200.82 lat (usec): min=70, max=1267.1K, avg=321.40, stdev=5201.22 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 108], 10.00th=[ 112], 20.00th=[ 118], | 30.00th=[ 123], 40.00th=[ 129], 50.00th=[ 133], 60.00th=[ 139], | 70.00th=[ 147], 80.00th=[ 157], 90.00th=[ 179], 95.00th=[ 201], | 99.00th=[ 274], 99.50th=[ 354], 99.90th=[52480], 99.95th=[95744], | 99.99th=[236544] lat (usec) : 10=0.01%, 50=0.01%, 100=0.15%, 250=98.28%, 500=1.22% lat (usec) : 750=0.02%, 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.07%, 50=0.10% lat (msec) : 100=0.06%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.77%, sys=21.69%, ctx=10852657, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=22279060/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1442: Sat Jul 29 02:36:06 2017 read : io=75046MB, bw=85385KB/s, iops=21346, runt=900001msec slat (usec): min=0, max=2520, avg= 6.59, stdev= 3.40 clat (usec): min=18, max=1428.8K, avg=365.77, stdev=5975.46 lat (usec): min=70, max=1428.8K, avg=372.86, stdev=5975.77 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 111], 10.00th=[ 116], 20.00th=[ 122], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 155], 80.00th=[ 173], 90.00th=[ 207], 95.00th=[ 249], | 99.00th=[ 350], 99.50th=[ 426], 99.90th=[63744], 99.95th=[112128], | 99.99th=[264192] lat (usec) : 20=0.01%, 50=0.01%, 100=0.11%, 250=94.94%, 500=4.55% lat (usec) : 750=0.03%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.28%, sys=19.54%, ctx=9973368, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=19211686/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=162073MB, aggrb=184403KB/s, minb=85385KB/s, maxb=99017KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=41467013/28, merge=0/0, ticks=13982352/436, in_queue=14002096, util=100.00%, aggrios=13852123/192595, aggrmerge=0/0, aggrticks=5439580/41926, aggrin_queue=5486213, aggrutil=71.52% dm-1: ios=40/512174, merge=0/0, ticks=16/65512, in_queue=65540, util=4.37%, aggrios=41359847/559061, aggrmerge=1/17352, aggrticks=5953780/75680, aggrin_queue=6035140, aggrutil=73.41% xvdc: ios=41359847/559061, merge=1/17352, ticks=5953780/75680, in_queue=6035140, util=73.41% dm-0: ios=41359808/64396, merge=0/0, ticks=5949272/12428, in_queue=5975644, util=71.52% dm-3: ios=196521/1217, merge=0/0, ticks=10369452/47840, in_queue=10417456, util=65.41%, aggrios=196521/1215, aggrmerge=0/2, aggrticks=10369400/46792, aggrin_queue=10416188, aggrutil=65.41% xvdd: ios=196521/1215, merge=0/2, ticks=10369400/46792, in_queue=10416188, util=65.41% 4.12.0-rc2+b870aa901f4b (2017-05-31 19:29:48 -0400)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc2+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [387.1MB/0KB/0KB /s] [99.3K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1034: Sat Jul 29 03:42:15 2017 read : io=170714MB, bw=194235KB/s, iops=48558, runt=900001msec slat (usec): min=0, max=5188, avg= 7.17, stdev= 3.84 clat (usec): min=33, max=868253, avg=154.79, stdev=246.43 lat (usec): min=74, max=868258, avg=162.45, stdev=246.45 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 112], 10.00th=[ 118], 20.00th=[ 125], | 30.00th=[ 131], 40.00th=[ 137], 50.00th=[ 143], 60.00th=[ 151], | 70.00th=[ 161], 80.00th=[ 175], 90.00th=[ 203], 95.00th=[ 239], | 99.00th=[ 326], 99.50th=[ 362], 99.90th=[ 454], 99.95th=[ 510], | 99.99th=[ 1176] lat (usec) : 50=0.01%, 100=0.07%, 250=95.81%, 500=4.06%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=13.58%, sys=46.69%, ctx=17639239, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=43702872/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1035: Sat Jul 29 03:42:15 2017 read : io=166633MB, bw=189591KB/s, iops=47397, runt=900001msec slat (usec): min=0, max=5427, avg= 7.20, stdev= 3.86 clat (usec): min=25, max=1052.1K, avg=158.78, stdev=971.36 lat (usec): min=67, max=1052.1K, avg=166.47, stdev=971.50 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 111], 10.00th=[ 116], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 135], 50.00th=[ 141], 60.00th=[ 147], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 191], 95.00th=[ 211], | 99.00th=[ 274], 99.50th=[ 310], 99.90th=[ 454], 99.95th=[ 740], | 99.99th=[31104] lat (usec) : 50=0.01%, 100=0.09%, 250=98.15%, 500=1.69%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01% cpu : usr=13.55%, sys=45.80%, ctx=17358595, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=42657995/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=337347MB, aggrb=383825KB/s, minb=189590KB/s, maxb=194234KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=86351783/28, merge=0/0, ticks=13253620/1212, in_queue=13300668, util=100.00%, aggrios=28809668/39785, aggrmerge=0/0, aggrticks=4427297/502156, aggrin_queue=4941461, aggrutil=100.00% dm-1: ios=23/51212, merge=0/0, ticks=0/9628, in_queue=9632, util=0.83%, aggrios=86400786/48163, aggrmerge=36/10034, aggrticks=12739236/9992, aggrin_queue=12763444, aggrutil=100.00% xvdc: ios=86400786/48163, merge=36/10034, ticks=12739236/9992, in_queue=12763444, util=100.00% dm-0: ios=86400799/7291, merge=0/0, ticks=12710904/1508, in_queue=12748372, util=100.00% dm-3: ios=28182/60853, merge=0/0, ticks=570988/1495332, in_queue=2066380, util=21.95%, aggrios=28182/60807, aggrmerge=0/46, aggrticks=570992/1486948, aggrin_queue=2057924, aggrutil=21.95% xvdd: ios=28182/60807, merge=0/46, ticks=570992/1486948, in_queue=2057924, util=21.95% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc2+ fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [373.7MB/0KB/0KB /s] [95.7K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1530: Sat Jul 29 03:59:03 2017 read : io=76723MB, bw=87294KB/s, iops=21823, runt=900001msec slat (usec): min=0, max=2653, avg= 6.62, stdev= 4.06 clat (usec): min=30, max=1178.5K, avg=357.41, stdev=5505.55 lat (usec): min=70, max=1178.5K, avg=364.48, stdev=5505.64 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 112], 10.00th=[ 117], 20.00th=[ 124], | 30.00th=[ 131], 40.00th=[ 135], 50.00th=[ 143], 60.00th=[ 151], | 70.00th=[ 163], 80.00th=[ 183], 90.00th=[ 217], 95.00th=[ 262], | 99.00th=[ 374], 99.50th=[ 458], 99.90th=[61696], 99.95th=[109056], | 99.99th=[246784] lat (usec) : 50=0.01%, 100=0.09%, 250=93.93%, 500=5.55%, 750=0.06% lat (usec) : 1000=0.02% lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.08%, 50=0.11% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.69%, sys=19.74%, ctx=9908053, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=19641135/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1531: Sat Jul 29 03:59:03 2017 read : io=74239MB, bw=84468KB/s, iops=21116, runt=900001msec slat (usec): min=0, max=317, avg= 6.86, stdev= 3.89 clat (usec): min=24, max=1537.3K, avg=369.35, stdev=5950.92 lat (usec): min=71, max=1537.3K, avg=376.68, stdev=5950.98 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 109], 10.00th=[ 113], 20.00th=[ 120], | 30.00th=[ 126], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 145], | 70.00th=[ 157], 80.00th=[ 173], 90.00th=[ 207], 95.00th=[ 247], | 99.00th=[ 350], 99.50th=[ 434], 99.90th=[66048], 99.95th=[116224], | 99.99th=[264192] lat (usec) : 50=0.01%, 100=0.16%, 250=95.11%, 500=4.32%, 750=0.04% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.71%, sys=19.66%, ctx=9087432, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=19005271/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=150963MB, aggrb=171761KB/s, minb=84467KB/s, maxb=87293KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=38629806/29, merge=0/0, ticks=13959752/3564, in_queue=13980884, util=100.00%, aggrios=12904005/192734, aggrmerge=0/0, aggrticks=5435258/41146, aggrin_queue=5481312, aggrutil=71.34% dm-1: ios=57/512619, merge=0/0, ticks=4/65252, in_queue=65272, util=4.40%, aggrios=38515218/559828, aggrmerge=0/17125, aggrticks=5837752/75716, aggrin_queue=5921004, aggrutil=73.26% xvdc: ios=38515218/559828, merge=0/17125, ticks=5837752/75716, in_queue=5921004, util=73.26% dm-0: ios=38515161/64478, merge=0/0, ticks=5828092/12596, in_queue=5855388, util=71.34% dm-3: ios=196799/1107, merge=0/0, ticks=10477680/45592, in_queue=10523276, util=65.98%, aggrios=196799/1100, aggrmerge=0/7, aggrticks=10477632/43520, aggrin_queue=10521220, aggrutil=65.98% xvdd: ios=196799/1100, merge=0/7, ticks=10477632/43520, in_queue=10521220, util=65.98% 4.12.0-rc2+6ce478291137 (2017-05-26 10:51:18 -0700)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc2+6ce478291137 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [398.7MB/0KB/0KB /s] [102K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1324: Sat Jul 29 05:18:27 2017 read : io=181395MB, bw=206387KB/s, iops=51596, runt=900001msec slat (usec): min=0, max=3628, avg= 6.85, stdev= 3.18 clat (usec): min=31, max=650283, avg=145.71, stdev=401.45 lat (usec): min=70, max=650322, avg=153.04, stdev=401.53 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 110], 10.00th=[ 114], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 135], 60.00th=[ 141], | 70.00th=[ 149], 80.00th=[ 161], 90.00th=[ 179], 95.00th=[ 201], | 99.00th=[ 266], 99.50th=[ 302], 99.90th=[ 430], 99.95th=[ 502], | 99.99th=[ 1704] lat (usec) : 50=0.01%, 100=0.17%, 250=98.42%, 500=1.36%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=15.23%, sys=53.43%, ctx=19704592, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=46437113/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1325: Sat Jul 29 05:18:27 2017 read : io=163235MB, bw=185725KB/s, iops=46431, runt=900001msec slat (usec): min=0, max=7249, avg= 7.00, stdev= 3.64 clat (usec): min=29, max=321686, avg=162.76, stdev=759.82 lat (usec): min=69, max=321717, avg=170.23, stdev=759.98 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 112], 10.00th=[ 117], 20.00th=[ 124], | 30.00th=[ 129], 40.00th=[ 135], 50.00th=[ 141], 60.00th=[ 147], | 70.00th=[ 157], 80.00th=[ 173], 90.00th=[ 203], 95.00th=[ 243], | 99.00th=[ 330], 99.50th=[ 370], 99.90th=[ 474], 99.95th=[ 716], | 99.99th=[32128] lat (usec) : 50=0.01%, 100=0.12%, 250=95.44%, 500=4.36%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=13.56%, sys=49.51%, ctx=18629816, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=41788282/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=344630MB, aggrb=392112KB/s, minb=185725KB/s, maxb=206386KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=88203561/28, merge=0/0, ticks=13332148/1696, in_queue=13367996, util=100.00%, aggrios=29431008/44535, aggrmerge=0/0, aggrticks=4467504/392230, aggrin_queue=4869529, aggrutil=100.00% dm-1: ios=13/65959, merge=0/0, ticks=4/11628, in_queue=11636, util=0.98%, aggrios=88262702/62581, aggrmerge=0/10717, aggrticks=12756980/12036, aggrin_queue=12776352, aggrutil=100.00% xvdc: ios=88262702/62581, merge=0/10717, ticks=12756980/12036, in_queue=12776352, util=100.00% dm-0: ios=88262689/9237, merge=0/0, ticks=12731748/1756, in_queue=12762868, util=100.00% dm-3: ios=30323/58410, merge=0/0, ticks=670760/1163308, in_queue=1834084, util=20.15%, aggrios=30323/58407, aggrmerge=0/3, aggrticks=670772/1159656, aggrin_queue=1830424, aggrutil=20.15% xvdd: ios=30323/58407, merge=0/3, ticks=670772/1159656, in_queue=1830424, util=20.15% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc2+6ce478291137 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [364.8MB/0KB/0KB /s] [93.4K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1722: Sat Jul 29 05:41:43 2017 read : io=74359MB, bw=84603KB/s, iops=21150, runt=900001msec slat (usec): min=0, max=2436, avg= 6.76, stdev= 3.33 clat (usec): min=37, max=1305.6K, avg=369.04, stdev=5594.01 lat (usec): min=68, max=1305.9K, avg=376.26, stdev=5594.50 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 109], 10.00th=[ 113], 20.00th=[ 120], | 30.00th=[ 124], 40.00th=[ 129], 50.00th=[ 135], 60.00th=[ 143], | 70.00th=[ 155], 80.00th=[ 179], 90.00th=[ 258], 95.00th=[ 346], | 99.00th=[ 532], 99.50th=[ 636], 99.90th=[63232], 99.95th=[110080], | 99.99th=[250880] lat (usec) : 50=0.01%, 100=0.17%, 250=89.28%, 500=9.28%, 750=0.88% lat (usec) : 1000=0.03% lat (msec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=6.42%, sys=22.97%, ctx=10466417, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=19035803/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1723: Sat Jul 29 05:41:43 2017 read : io=75299MB, bw=85674KB/s, iops=21418, runt=900001msec slat (usec): min=0, max=2420, avg= 6.72, stdev= 3.30 clat (usec): min=32, max=1379.2K, avg=364.38, stdev=5736.94 lat (usec): min=71, max=1379.2K, avg=371.55, stdev=5737.35 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 108], 10.00th=[ 111], 20.00th=[ 117], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 141], | 70.00th=[ 149], 80.00th=[ 167], 90.00th=[ 205], 95.00th=[ 274], | 99.00th=[ 490], 99.50th=[ 612], 99.90th=[64768], 99.95th=[113152], | 99.99th=[259072] lat (usec) : 50=0.01%, 100=0.27%, 250=93.50%, 500=5.31%, 750=0.54% lat (usec) : 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=6.67%, sys=22.54%, ctx=10228527, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=19276639/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=149658MB, aggrb=170277KB/s, minb=84603KB/s, maxb=85673KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=38294873/29, merge=0/0, ticks=13980376/4156, in_queue=13996968, util=100.00%, aggrios=12792699/192601, aggrmerge=0/0, aggrticks=5453128/37806, aggrin_queue=5494990, aggrutil=70.72% dm-1: ios=86/512203, merge=0/0, ticks=16/61356, in_queue=61388, util=4.27%, aggrios=38180502/560167, aggrmerge=1/16429, aggrticks=5907852/71764, aggrin_queue=5984152, aggrutil=72.60% xvdc: ios=38180502/560167, merge=1/16429, ticks=5907852/71764, in_queue=5984152, util=72.60% dm-0: ios=38180417/64538, merge=0/0, ticks=5896096/12280, in_queue=5920524, util=70.72% dm-3: ios=197595/1063, merge=0/0, ticks=10463272/39784, in_queue=10503060, util=65.39%, aggrios=197595/1056, aggrmerge=0/7, aggrticks=10463272/36948, aggrin_queue=10500280, aggrutil=65.39% xvdd: ios=197595/1056, merge=0/7, ticks=10463272/36948, in_queue=10500280, util=65.39% 4.12.0-rc1+331da109ec20 (2017-05-20 08:44:22 -0700)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+331da109ec20 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [378.6MB/0KB/0KB /s] [96.1K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1342: Sat Jul 29 06:54:00 2017 read : io=168144MB, bw=191310KB/s, iops=47827, runt=900001msec slat (usec): min=2, max=2850, avg= 6.93, stdev= 2.60 clat (usec): min=39, max=1150.2K, avg=157.80, stdev=403.97 lat (usec): min=70, max=1150.2K, avg=165.21, stdev=404.00 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 111], 10.00th=[ 116], 20.00th=[ 124], | 30.00th=[ 131], 40.00th=[ 137], 50.00th=[ 145], 60.00th=[ 153], | 70.00th=[ 165], 80.00th=[ 181], 90.00th=[ 211], 95.00th=[ 247], | 99.00th=[ 330], 99.50th=[ 370], 99.90th=[ 466], 99.95th=[ 532], | 99.99th=[ 1336] lat (usec) : 50=0.01%, 100=0.08%, 250=95.27%, 500=4.58%, 750=0.04% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=12.05%, sys=45.56%, ctx=19112237, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=43044758/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1343: Sat Jul 29 06:54:00 2017 read : io=162387MB, bw=184760KB/s, iops=46190, runt=900001msec slat (usec): min=2, max=3771, avg= 6.95, stdev= 2.70 clat (usec): min=18, max=528413, avg=163.71, stdev=689.58 lat (usec): min=70, max=528456, avg=171.14, stdev=689.73 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 110], 10.00th=[ 115], 20.00th=[ 122], | 30.00th=[ 129], 40.00th=[ 135], 50.00th=[ 143], 60.00th=[ 151], | 70.00th=[ 163], 80.00th=[ 181], 90.00th=[ 211], 95.00th=[ 247], | 99.00th=[ 334], 99.50th=[ 374], 99.90th=[ 482], 99.95th=[ 708], | 99.99th=[27520] lat (usec) : 20=0.01%, 50=0.01%, 100=0.11%, 250=95.14%, 500=4.67% lat (usec) : 750=0.04%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=12.07%, sys=43.77%, ctx=18853321, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=41571072/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=330531MB, aggrb=376069KB/s, minb=184760KB/s, maxb=191309KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=84604087/28, merge=0/0, ticks=13465432/868, in_queue=13500164, util=100.00%, aggrios=28227541/40723, aggrmerge=0/0, aggrticks=4500645/520276, aggrin_queue=5032592, aggrutil=100.00% dm-1: ios=25/55370, merge=0/0, ticks=4/10480, in_queue=10484, util=0.88%, aggrios=84654863/53148, aggrmerge=27/9173, aggrticks=12992604/11056, aggrin_queue=13010688, aggrutil=100.00% xvdc: ios=84654863/53148, merge=27/9173, ticks=12992604/11056, in_queue=13010688, util=100.00% dm-0: ios=84654865/7809, merge=0/0, ticks=12987816/1676, in_queue=13024444, util=100.00% dm-3: ios=27734/58990, merge=0/0, ticks=514116/1548672, in_queue=2062848, util=22.31%, aggrios=27734/58961, aggrmerge=0/29, aggrticks=514100/1541080, aggrin_queue=2055176, aggrutil=22.31% xvdd: ios=27734/58961, merge=0/29, ticks=514100/1541080, in_queue=2055176, util=22.31% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+331da109ec20 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [409.7MB/0KB/0KB /s] [105K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1729: Sat Jul 29 07:10:43 2017 read : io=84440MB, bw=96074KB/s, iops=24018, runt=900001msec slat (usec): min=2, max=2584, avg= 6.82, stdev= 2.79 clat (usec): min=16, max=1503.2K, avg=323.80, stdev=5298.35 lat (usec): min=67, max=1503.2K, avg=331.09, stdev=5298.79 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 98], 10.00th=[ 107], 20.00th=[ 115], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 135], 60.00th=[ 141], | 70.00th=[ 151], 80.00th=[ 165], 90.00th=[ 189], 95.00th=[ 211], | 99.00th=[ 282], 99.50th=[ 366], 99.90th=[56064], 99.95th=[100864], | 99.99th=[238592] lat (usec) : 20=0.01%, 50=0.01%, 100=5.36%, 250=92.78%, 500=1.51% lat (usec) : 750=0.03%, 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.07%, 50=0.10% lat (msec) : 100=0.06%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=6.09%, sys=22.54%, ctx=10504259, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=21616707/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1730: Sat Jul 29 07:10:43 2017 read : io=80128MB, bw=91168KB/s, iops=22791, runt=900001msec slat (usec): min=2, max=2258, avg= 6.89, stdev= 2.81 clat (usec): min=21, max=1497.9K, avg=341.59, stdev=5610.12 lat (usec): min=66, max=1497.9K, avg=348.97, stdev=5610.53 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 89], 10.00th=[ 104], 20.00th=[ 113], | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 141], | 70.00th=[ 151], 80.00th=[ 167], 90.00th=[ 191], 95.00th=[ 219], | 99.00th=[ 314], 99.50th=[ 394], 99.90th=[60672], 99.95th=[108032], | 99.99th=[252928] lat (usec) : 50=0.01%, 100=8.41%, 250=88.94%, 500=2.27%, 750=0.03% lat (usec) : 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.07%, 50=0.11% lat (msec) : 100=0.06%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.90%, sys=21.52%, ctx=9740291, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=20512763/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=164568MB, aggrb=187241KB/s, minb=91167KB/s, maxb=96074KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=42117663/29, merge=0/0, ticks=13957468/4044, in_queue=13976576, util=100.00%, aggrios=14065022/191669, aggrmerge=0/0, aggrticks=5437961/43466, aggrin_queue=5485902, aggrutil=71.18% dm-1: ios=23/509401, merge=0/0, ticks=4/64012, in_queue=64032, util=4.37%, aggrios=41998935/556367, aggrmerge=0/16995, aggrticks=5877544/74016, aggrin_queue=5952992, aggrutil=73.08% xvdc: ios=41998935/556367, merge=0/16995, ticks=5877544/74016, in_queue=5952992, util=73.08% dm-0: ios=41998912/64099, merge=0/0, ticks=5872440/12180, in_queue=5897884, util=71.18% dm-3: ios=196133/1507, merge=0/0, ticks=10441440/54208, in_queue=10495792, util=65.28%, aggrios=196133/1501, aggrmerge=0/6, aggrticks=10441400/51888, aggrin_queue=10493276, aggrutil=65.28% xvdd: ios=196133/1501, merge=0/6, ticks=10441400/51888, in_queue=10493276, util=65.28% 4.12.0-rc1+d51aff16e821 (2017-05-20 06:00:49 +1000)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+d51aff16e821 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [406.3MB/0KB/0KB /s] [104K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1248: Sat Jul 29 08:51:19 2017 read : io=177645MB, bw=202121KB/s, iops=50530, runt=900001msec slat (usec): min=0, max=3003, avg= 7.12, stdev= 3.03 clat (usec): min=39, max=971679, avg=148.57, stdev=681.72 lat (usec): min=70, max=971686, avg=156.15, stdev=681.79 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 110], 10.00th=[ 115], 20.00th=[ 122], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 137], 60.00th=[ 145], | 70.00th=[ 151], 80.00th=[ 163], 90.00th=[ 181], 95.00th=[ 201], | 99.00th=[ 262], 99.50th=[ 302], 99.90th=[ 438], 99.95th=[ 556], | 99.99th=[ 7264] lat (usec) : 50=0.01%, 100=0.11%, 250=98.53%, 500=1.30%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=15.07%, sys=52.23%, ctx=18395569, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=45477192/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1249: Sat Jul 29 08:51:19 2017 read : io=169585MB, bw=192950KB/s, iops=48237, runt=900001msec slat (usec): min=0, max=2529, avg= 7.21, stdev= 3.13 clat (usec): min=27, max=1731.8K, avg=156.00, stdev=919.90 lat (usec): min=67, max=1731.8K, avg=163.66, stdev=920.02 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 111], 10.00th=[ 116], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 153], 80.00th=[ 165], 90.00th=[ 185], 95.00th=[ 205], | 99.00th=[ 266], 99.50th=[ 302], 99.90th=[ 450], 99.95th=[ 724], | 99.99th=[31104] lat (usec) : 50=0.01%, 100=0.19%, 250=98.36%, 500=1.38%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=14.06%, sys=50.57%, ctx=17854020, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=43413691/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=347230MB, aggrb=395070KB/s, minb=192949KB/s, maxb=202120KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=88868766/28, merge=0/0, ticks=13255048/544, in_queue=13291156, util=100.00%, aggrios=29652308/43870, aggrmerge=0/0, aggrticks=4437944/678066, aggrin_queue=5124804, aggrutil=100.00% dm-1: ios=40/65580, merge=0/0, ticks=8/11168, in_queue=11184, util=0.96%, aggrios=88925731/62252, aggrmerge=5/10546, aggrticks=12610900/11908, aggrin_queue=12628964, aggrutil=100.00% xvdc: ios=88925731/62252, merge=5/10546, ticks=12610900/11908, in_queue=12628964, util=100.00% dm-0: ios=88925696/9160, merge=0/0, ticks=12587208/1880, in_queue=12615460, util=100.00% dm-3: ios=31188/56871, merge=0/0, ticks=726616/2021152, in_queue=2747768, util=27.31%, aggrios=31188/56865, aggrmerge=0/6, aggrticks=726608/2002592, aggrin_queue=2729192, aggrutil=27.32% xvdd: ios=31188/56865, merge=0/6, ticks=726608/2002592, in_queue=2729192, util=27.32% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+d51aff16e821 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [384.3MB/0KB/0KB /s] [98.4K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1535: Sat Jul 29 09:10:07 2017 read : io=81142MB, bw=92322KB/s, iops=23080, runt=900001msec slat (usec): min=0, max=2500, avg= 6.89, stdev= 3.06 clat (usec): min=27, max=1250.7K, avg=337.19, stdev=5337.52 lat (usec): min=66, max=1250.8K, avg=344.53, stdev=5338.02 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 107], 10.00th=[ 110], 20.00th=[ 118], | 30.00th=[ 123], 40.00th=[ 129], 50.00th=[ 135], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 171], 90.00th=[ 203], 95.00th=[ 245], | 99.00th=[ 358], 99.50th=[ 434], 99.90th=[57600], 99.95th=[103936], | 99.99th=[240640] lat (usec) : 50=0.01%, 100=1.73%, 250=93.69%, 500=4.18%, 750=0.04% lat (usec) : 1000=0.02% lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.07%, 50=0.11% lat (msec) : 100=0.06%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=6.95%, sys=23.93%, ctx=10312344, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=20772473/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1536: Sat Jul 29 09:10:07 2017 read : io=73428MB, bw=83544KB/s, iops=20886, runt=900001msec slat (usec): min=0, max=1453, avg= 6.96, stdev= 2.98 clat (usec): min=15, max=1524.7K, avg=373.51, stdev=6231.54 lat (usec): min=67, max=1524.7K, avg=380.92, stdev=6231.92 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 107], 10.00th=[ 112], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 157], 80.00th=[ 173], 90.00th=[ 203], 95.00th=[ 241], | 99.00th=[ 350], 99.50th=[ 434], 99.90th=[66048], 99.95th=[117248], | 99.99th=[276480] lat (usec) : 20=0.01%, 50=0.01%, 100=1.63%, 250=94.04%, 500=3.91% lat (usec) : 750=0.04%, 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=6.40%, sys=21.56%, ctx=9325698, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=18797466/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=154570MB, aggrb=175866KB/s, minb=83544KB/s, maxb=92321KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=39560967/29, merge=0/0, ticks=13949328/1084, in_queue=13964452, util=100.00%, aggrios=13211863/192921, aggrmerge=0/0, aggrticks=5416640/38728, aggrin_queue=5458597, aggrutil=70.47% dm-1: ios=47/513127, merge=0/0, ticks=0/64104, in_queue=64112, util=4.32%, aggrios=39438165/560401, aggrmerge=0/17140, aggrticks=5784740/74192, aggrin_queue=5861736, aggrutil=72.34% xvdc: ios=39438165/560401, merge=0/17140, ticks=5784740/74192, in_queue=5861736, util=72.34% dm-0: ios=39438118/64534, merge=0/0, ticks=5774760/12144, in_queue=5796584, util=70.47% dm-3: ios=197426/1102, merge=0/0, ticks=10475160/39936, in_queue=10515096, util=66.94%, aggrios=197426/1101, aggrmerge=0/1, aggrticks=10475168/39392, aggrin_queue=10514556, aggrutil=66.94% xvdd: ios=197426/1101, merge=0/1, ticks=10475168/39392, in_queue=10514556, util=66.94% 4.12.0-rc1+a58a260fd96b (2017-05-18 11:21:10 -0700)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+a58a260fd96b fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [403.7MB/0KB/0KB /s] [103K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1469: Sat Jul 29 10:38:52 2017 read : io=178006MB, bw=202531KB/s, iops=50632, runt=900001msec slat (usec): min=0, max=6867, avg= 7.12, stdev= 3.46 clat (usec): min=32, max=1286.2K, avg=148.28, stdev=540.32 lat (usec): min=67, max=1286.2K, avg=155.87, stdev=540.36 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 111], 10.00th=[ 116], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 153], 80.00th=[ 165], 90.00th=[ 185], 95.00th=[ 207], | 99.00th=[ 270], 99.50th=[ 306], 99.90th=[ 430], 99.95th=[ 494], | 99.99th=[ 1352] lat (usec) : 50=0.01%, 100=0.12%, 250=98.28%, 500=1.55%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=15.17%, sys=54.06%, ctx=18845932, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=45569414/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1470: Sat Jul 29 10:38:52 2017 read : io=173743MB, bw=197681KB/s, iops=49420, runt=900001msec slat (usec): min=0, max=3071, avg= 7.07, stdev= 3.36 clat (usec): min=19, max=1304.0K, avg=152.23, stdev=680.26 lat (usec): min=69, max=1304.2K, avg=159.77, stdev=680.38 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 110], 10.00th=[ 115], 20.00th=[ 122], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 185], 95.00th=[ 205], | 99.00th=[ 266], 99.50th=[ 302], 99.90th=[ 438], 99.95th=[ 588], | 99.99th=[22912] lat (usec) : 20=0.01%, 50=0.01%, 100=0.17%, 250=98.29%, 500=1.48% lat (usec) : 750=0.02%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=15.16%, sys=52.26%, ctx=18799543, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=44478196/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=351748MB, aggrb=400211KB/s, minb=197680KB/s, maxb=202530KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=90047147/28, merge=0/0, ticks=13242192/1464, in_queue=13298304, util=100.00%, aggrios=30038236/40110, aggrmerge=0/0, aggrticks=4422056/701994, aggrin_queue=5135569, aggrutil=100.00% dm-1: ios=45/53246, merge=0/0, ticks=8/9560, in_queue=9572, util=0.81%, aggrios=90087618/50539, aggrmerge=21/8883, aggrticks=12812936/9916, aggrin_queue=12830368, aggrutil=100.00% xvdc: ios=90087618/50539, merge=21/8883, ticks=12812936/9916, in_queue=12830368, util=100.00% dm-0: ios=90087594/7476, merge=0/0, ticks=12789572/1472, in_queue=12825580, util=100.00% dm-3: ios=27070/59608, merge=0/0, ticks=476588/2094952, in_queue=2571556, util=26.41%, aggrios=27070/59583, aggrmerge=0/25, aggrticks=476596/2078296, aggrin_queue=2554892, aggrutil=26.41% xvdd: ios=27070/59583, merge=0/25, ticks=476596/2078296, in_queue=2554892, util=26.41% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+a58a260fd96b fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [369.8MB/0KB/0KB /s] [94.7K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1970: Sat Jul 29 11:17:21 2017 read : io=77095MB, bw=87717KB/s, iops=21929, runt=900001msec slat (usec): min=0, max=2471, avg= 6.97, stdev= 3.30 clat (usec): min=4, max=1523.7K, avg=355.35, stdev=5578.26 lat (usec): min=64, max=1523.7K, avg=362.77, stdev=5578.75 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 108], 10.00th=[ 111], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 133], 50.00th=[ 143], 60.00th=[ 153], | 70.00th=[ 167], 80.00th=[ 185], 90.00th=[ 213], 95.00th=[ 247], | 99.00th=[ 346], 99.50th=[ 426], 99.90th=[61184], 99.95th=[108032], | 99.99th=[248832] lat (usec) : 10=0.01%, 50=0.01%, 100=0.83%, 250=94.43%, 500=4.33% lat (usec) : 750=0.04%, 1000=0.02% lat (msec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.11% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=6.49%, sys=24.26%, ctx=10152879, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=19736266/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1971: Sat Jul 29 11:17:21 2017 read : io=70627MB, bw=80357KB/s, iops=20089, runt=900001msec slat (usec): min=0, max=1736, avg= 6.99, stdev= 3.37 clat (usec): min=17, max=1334.4K, avg=388.69, stdev=6080.75 lat (usec): min=62, max=1334.4K, avg=396.14, stdev=6081.19 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 108], 10.00th=[ 114], 20.00th=[ 122], | 30.00th=[ 129], 40.00th=[ 137], 50.00th=[ 147], 60.00th=[ 159], | 70.00th=[ 173], 80.00th=[ 191], 90.00th=[ 225], 95.00th=[ 262], | 99.00th=[ 366], 99.50th=[ 454], 99.90th=[69120], 99.95th=[119296], | 99.99th=[272384] lat (usec) : 20=0.01%, 50=0.01%, 100=0.76%, 250=92.96%, 500=5.83% lat (usec) : 750=0.04%, 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.95%, sys=22.37%, ctx=9542001, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=18080386/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=147721MB, aggrb=168073KB/s, minb=80357KB/s, maxb=87716KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=37811927/29, merge=0/0, ticks=13950924/2420, in_queue=13976400, util=100.00%, aggrios=12627432/192884, aggrmerge=0/0, aggrticks=5423964/40908, aggrin_queue=5469672, aggrutil=70.48% dm-1: ios=41/513019, merge=0/0, ticks=8/61768, in_queue=61776, util=4.19%, aggrios=37684593/560591, aggrmerge=0/16798, aggrticks=5810780/72380, aggrin_queue=5885452, aggrutil=72.25% xvdc: ios=37684593/560591, merge=0/16798, ticks=5810780/72380, in_queue=5885452, util=72.25% dm-0: ios=37684552/64512, merge=0/0, ticks=5808316/12596, in_queue=5835296, util=70.48% dm-3: ios=197703/1122, merge=0/0, ticks=10463568/48360, in_queue=10511944, util=65.97%, aggrios=197703/1119, aggrmerge=0/3, aggrticks=10463584/46584, aggrin_queue=10510148, aggrutil=65.97% xvdd: ios=197703/1119, merge=0/3, ticks=10463584/46584, in_queue=10510148, util=65.97% 4.12.0-rc1+243bfd2c052d (2017-05-17 14:13:44 -0700)

just 2,130 and 738 average IOPS on the run against a full cache means this was marked broken.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+243bfd2c052d fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [223.5MB/0KB/0KB /s] [57.2K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1033: Sat Jul 29 13:21:34 2017 read : io=136455MB, bw=155242KB/s, iops=38810, runt=900078msec slat (usec): min=0, max=26662, avg= 4.82, stdev= 6.30 clat (usec): min=21, max=880182, avg=199.68, stdev=2322.99 lat (usec): min=70, max=880190, avg=204.84, stdev=2323.29 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 106], 10.00th=[ 109], 20.00th=[ 114], | 30.00th=[ 119], 40.00th=[ 122], 50.00th=[ 126], 60.00th=[ 131], | 70.00th=[ 137], 80.00th=[ 149], 90.00th=[ 189], 95.00th=[ 245], | 99.00th=[ 382], 99.50th=[ 446], 99.90th=[17792], 99.95th=[36608], | 99.99th=[108032] lat (usec) : 50=0.01%, 100=0.24%, 250=95.07%, 500=4.31%, 750=0.09% lat (usec) : 1000=0.05% lat (msec) : 2=0.11%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.06% lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=7.38%, sys=26.41%, ctx=19119794, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=34932423/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1034: Sat Jul 29 13:21:34 2017 read : io=42407MB, bw=48247KB/s, iops=12061, runt=900066msec slat (usec): min=0, max=236, avg= 7.12, stdev= 4.37 clat (usec): min=2, max=1038.3K, avg=653.49, stdev=6208.11 lat (usec): min=71, max=1038.3K, avg=661.11, stdev=6209.00 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 109], | 30.00th=[ 113], 40.00th=[ 117], 50.00th=[ 120], 60.00th=[ 124], | 70.00th=[ 129], 80.00th=[ 137], 90.00th=[ 163], 95.00th=[ 237], | 99.00th=[16512], 99.50th=[35072], 99.90th=[93696], 99.95th=[123392], | 99.99th=[195584] lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.64% lat (usec) : 250=94.91%, 500=2.76%, 750=0.11%, 1000=0.07% lat (msec) : 2=0.14%, 4=0.02%, 10=0.10%, 20=0.37%, 50=0.56% lat (msec) : 100=0.23%, 250=0.08%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=3.63%, sys=12.48%, ctx=9079177, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=10856295/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=178862MB, aggrb=203487KB/s, minb=48246KB/s, maxb=155241KB/s, mint=900066msec, maxt=900078msec Disk stats (read/write): dm-2: ios=45783724/28, merge=0/0, ticks=14086884/552, in_queue=14099888, util=100.00%, aggrios=15302554/317325, aggrmerge=0/0, aggrticks=5324778/75793, aggrin_queue=5403120, aggrutil=100.00% dm-1: ios=5/833005, merge=0/0, ticks=0/136424, in_queue=136468, util=10.74%, aggrios=45578059/918360, aggrmerge=965/23157, aggrticks=6367068/155120, aggrin_queue=6522812, aggrutil=100.00% xvdc: ios=45578059/918360, merge=965/23157, ticks=6367068/155120, in_queue=6522812, util=100.00% dm-0: ios=45579019/109294, merge=0/0, ticks=6367368/21496, in_queue=6396436, util=100.00% dm-3: ios=328639/9676, merge=0/0, ticks=9606968/69460, in_queue=9676456, util=100.00%, aggrios=328639/8639, aggrmerge=0/1037, aggrticks=9606840/46124, aggrin_queue=9652956, aggrutil=100.00% xvdd: ios=328639/8639, merge=0/1037, ticks=9606840/46124, in_queue=9652956, util=100.00% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+243bfd2c052d fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [12052KB/0KB/0KB /s] [3013/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1455: Sat Jul 29 13:50:32 2017 read : io=7490.4MB, bw=8521.4KB/s, iops=2130, runt=900105msec slat (usec): min=0, max=128, avg= 1.71, stdev= 4.39 clat (usec): min=64, max=1526.2K, avg=3752.99, stdev=24771.48 lat (usec): min=73, max=1526.2K, avg=3754.82, stdev=24772.16 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 106], 10.00th=[ 111], 20.00th=[ 115], | 30.00th=[ 117], 40.00th=[ 118], 50.00th=[ 119], 60.00th=[ 125], | 70.00th=[ 131], 80.00th=[ 133], 90.00th=[ 171], 95.00th=[12736], | 99.00th=[101888], 99.50th=[160768], 99.90th=[333824], 99.95th=[428032], | 99.99th=[692224] lat (usec) : 100=0.26%, 250=93.35%, 500=0.50%, 750=0.07%, 1000=0.06% lat (msec) : 2=0.11%, 4=0.01%, 10=0.29%, 20=1.36%, 50=1.87% lat (msec) : 100=1.09%, 250=0.82%, 500=0.18%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.19%, sys=0.72%, ctx=1983309, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1917523/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1456: Sat Jul 29 13:50:32 2017 read : io=2596.8MB, bw=2953.6KB/s, iops=738, runt=900054msec slat (usec): min=0, max=154, avg= 3.92, stdev= 5.64 clat (usec): min=66, max=1527.1K, avg=10829.22, stdev=43048.40 lat (usec): min=72, max=1527.1K, avg=10833.40, stdev=43048.88 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 105], 10.00th=[ 106], 20.00th=[ 111], | 30.00th=[ 116], 40.00th=[ 118], 50.00th=[ 120], 60.00th=[ 131], | 70.00th=[ 139], 80.00th=[ 173], 90.00th=[26240], 95.00th=[63232], | 99.00th=[201728], 99.50th=[280576], 99.90th=[501760], 99.95th=[626688], | 99.99th=[937984] lat (usec) : 100=0.47%, 250=83.17%, 500=0.37%, 750=0.09%, 1000=0.07% lat (msec) : 2=0.14%, 4=0.02%, 10=0.60%, 20=3.44%, 50=5.41% lat (msec) : 100=3.19%, 250=2.37%, 500=0.55%, 750=0.08%, 1000=0.02% lat (msec) : 2000=0.01% cpu : usr=0.14%, sys=0.48%, ctx=670686, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=664578/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=10086MB, aggrb=11474KB/s, minb=2953KB/s, maxb=8521KB/s, mint=900054msec, maxt=900105msec Disk stats (read/write): dm-2: ios=2581744/29, merge=0/0, ticks=14383904/4548, in_queue=14390016, util=100.00%, aggrios=896781/322000, aggrmerge=0/0, aggrticks=6180766/42330, aggrin_queue=6223142, aggrutil=100.00% dm-1: ios=5/857732, merge=0/0, ticks=0/101316, in_queue=101320, util=6.42%, aggrios=2363753/933247, aggrmerge=0/32409, aggrticks=294652/118376, aggrin_queue=412980, aggrutil=35.73% xvdc: ios=2363753/933247, merge=0/32409, ticks=294652/118376, in_queue=412980, util=35.73% dm-0: ios=2363748/108251, merge=0/0, ticks=294752/21128, in_queue=315924, util=31.04% dm-3: ios=326590/17, merge=0/0, ticks=18247548/4548, in_queue=18252184, util=100.00%, aggrios=326590/10, aggrmerge=0/7, aggrticks=18247532/2576, aggrin_queue=18250100, aggrutil=100.00% xvdd: ios=326590/10, merge=0/7, ticks=18247532/2576, in_queue=18250100, util=100.00% 4.12.0-rc1+f98e0eb68008 (2017-05-15 15:09:53 -0400)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+f98e0eb68008 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [389.1MB/0KB/0KB /s] [99.9K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1473: Sat Jul 29 15:01:22 2017 read : io=175480MB, bw=199657KB/s, iops=49914, runt=900001msec slat (usec): min=0, max=2865, avg= 7.12, stdev= 3.47 clat (usec): min=32, max=272750, avg=150.49, stdev=61.07 lat (usec): min=82, max=272765, avg=158.13, stdev=61.08 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 111], 10.00th=[ 116], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 197], 95.00th=[ 235], | 99.00th=[ 326], 99.50th=[ 362], 99.90th=[ 454], 99.95th=[ 502], | 99.99th=[ 1020] lat (usec) : 50=0.01%, 100=0.09%, 250=95.99%, 500=3.87%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 500=0.01% cpu : usr=13.53%, sys=47.87%, ctx=18357918, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=44922870/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1474: Sat Jul 29 15:01:22 2017 read : io=166695MB, bw=189662KB/s, iops=47415, runt=900001msec slat (usec): min=0, max=7688, avg= 7.16, stdev= 3.64 clat (usec): min=14, max=600437, avg=158.90, stdev=650.77 lat (usec): min=69, max=600476, avg=166.58, stdev=650.92 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 113], 10.00th=[ 118], 20.00th=[ 124], | 30.00th=[ 129], 40.00th=[ 135], 50.00th=[ 141], 60.00th=[ 147], | 70.00th=[ 155], 80.00th=[ 171], 90.00th=[ 197], 95.00th=[ 231], | 99.00th=[ 318], 99.50th=[ 350], 99.90th=[ 454], 99.95th=[ 564], | 99.99th=[24448] lat (usec) : 20=0.01%, 50=0.01%, 100=0.06%, 250=96.33%, 500=3.54% lat (usec) : 750=0.02%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=13.03%, sys=45.62%, ctx=17895059, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=42673942/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=342175MB, aggrb=389318KB/s, minb=189661KB/s, maxb=199656KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=87581336/28, merge=0/0, ticks=13288760/1272, in_queue=13321684, util=100.00%, aggrios=29220805/38094, aggrmerge=0/0, aggrticks=4442181/395562, aggrin_queue=4846680, aggrutil=100.00% dm-1: ios=13/48662, merge=0/0, ticks=4/8880, in_queue=8892, util=0.76%, aggrios=87640393/44939, aggrmerge=1/9151, aggrticks=12922660/9064, aggrin_queue=12942364, aggrutil=100.00% xvdc: ios=87640393/44939, merge=1/9151, ticks=12922660/9064, in_queue=12942364, util=100.00% dm-0: ios=87640381/6972, merge=0/0, ticks=12891608/1372, in_queue=12919780, util=100.00% dm-3: ios=22021/58648, merge=0/0, ticks=434932/1176436, in_queue=1611368, util=17.77%, aggrios=22021/58631, aggrmerge=0/17, aggrticks=434920/1170812, aggrin_queue=1605732, aggrutil=17.77% xvdd: ios=22021/58631, merge=0/17, ticks=434920/1170812, in_queue=1605732, util=17.77% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+f98e0eb68008 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [378.6MB/0KB/0KB /s] [96.8K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1827: Sat Jul 29 15:19:18 2017 read : io=81213MB, bw=92402KB/s, iops=23100, runt=900001msec slat (usec): min=0, max=653, avg= 6.99, stdev= 3.51 clat (usec): min=25, max=1352.2K, avg=336.83, stdev=5321.69 lat (usec): min=72, max=1352.2K, avg=344.32, stdev=5322.11 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 108], 10.00th=[ 112], 20.00th=[ 119], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 139], 60.00th=[ 149], | 70.00th=[ 159], 80.00th=[ 175], 90.00th=[ 201], 95.00th=[ 229], | 99.00th=[ 314], 99.50th=[ 390], 99.90th=[56576], 99.95th=[101888], | 99.99th=[242688] lat (usec) : 50=0.01%, 100=0.13%, 250=96.71%, 500=2.78%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.07%, 50=0.11% lat (msec) : 100=0.06%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.96%, sys=21.63%, ctx=10178854, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=20790563/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1828: Sat Jul 29 15:19:18 2017 read : io=73647MB, bw=83794KB/s, iops=20948, runt=900001msec slat (usec): min=0, max=2496, avg= 7.18, stdev= 3.62 clat (usec): min=23, max=1510.2K, avg=372.08, stdev=6082.40 lat (usec): min=72, max=1510.2K, avg=379.78, stdev=6082.74 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 110], 10.00th=[ 114], 20.00th=[ 122], | 30.00th=[ 129], 40.00th=[ 135], 50.00th=[ 143], 60.00th=[ 151], | 70.00th=[ 163], 80.00th=[ 179], 90.00th=[ 207], 95.00th=[ 237], | 99.00th=[ 330], 99.50th=[ 414], 99.90th=[65280], 99.95th=[113152], | 99.99th=[272384] lat (usec) : 50=0.01%, 100=0.13%, 250=96.03%, 500=3.44%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.82%, sys=20.07%, ctx=9121390, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=18853758/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=154861MB, aggrb=176196KB/s, minb=83794KB/s, maxb=92402KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=39629245/29, merge=0/0, ticks=13922184/2932, in_queue=13938180, util=100.00%, aggrios=13236641/192742, aggrmerge=0/0, aggrticks=5404900/39769, aggrin_queue=5447944, aggrutil=71.24% dm-1: ios=35/512628, merge=0/0, ticks=0/63156, in_queue=63168, util=4.26%, aggrios=39512858/560041, aggrmerge=1/16890, aggrticks=5866812/73188, aggrin_queue=5946060, aggrutil=73.11% xvdc: ios=39512858/560041, merge=1/16890, ticks=5866812/73188, in_queue=5946060, util=73.11% dm-0: ios=39512824/64453, merge=0/0, ticks=5858616/12180, in_queue=5880596, util=71.24% dm-3: ios=197065/1146, merge=0/0, ticks=10356084/43972, in_queue=10400068, util=65.55%, aggrios=197065/1143, aggrmerge=0/3, aggrticks=10356072/42172, aggrin_queue=10398236, aggrutil=65.55% xvdd: ios=197065/1143, merge=0/3, ticks=10356072/42172, in_queue=10398236, util=65.55% 4.12.0-rc1+701e03e4e180 (2017-05-14 21:54:33 -0400)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+701e03e4e180 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [383.6MB/0KB/0KB /s] [98.2K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1488: Sun Jul 30 00:38:12 2017 read : io=163416MB, bw=185931KB/s, iops=46482, runt=900001msec slat (usec): min=0, max=6338, avg= 6.90, stdev= 4.29 clat (usec): min=2, max=423004, avg=162.74, stdev=659.06 lat (usec): min=70, max=423007, avg=170.12, stdev=659.18 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 112], 10.00th=[ 117], 20.00th=[ 124], | 30.00th=[ 131], 40.00th=[ 137], 50.00th=[ 143], 60.00th=[ 153], | 70.00th=[ 163], 80.00th=[ 179], 90.00th=[ 207], 95.00th=[ 241], | 99.00th=[ 326], 99.50th=[ 362], 99.90th=[ 490], 99.95th=[ 828], | 99.99th=[23680] lat (usec) : 4=0.01%, 20=0.01%, 50=0.01%, 100=0.10%, 250=95.64% lat (usec) : 500=4.17%, 750=0.04%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=11.29%, sys=43.37%, ctx=18655951, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=41834439/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1489: Sun Jul 30 00:38:12 2017 read : io=163729MB, bw=186287KB/s, iops=46571, runt=900001msec slat (usec): min=0, max=3379, avg= 6.91, stdev= 4.05 clat (usec): min=1, max=647713, avg=162.39, stdev=1109.49 lat (usec): min=71, max=647715, avg=169.78, stdev=1109.58 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 112], 10.00th=[ 117], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 135], 50.00th=[ 141], 60.00th=[ 149], | 70.00th=[ 159], 80.00th=[ 173], 90.00th=[ 195], 95.00th=[ 219], | 99.00th=[ 286], 99.50th=[ 322], 99.90th=[ 462], 99.95th=[ 764], | 99.99th=[36096] lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.10%, 250=97.69% lat (usec) : 500=2.13%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=11.62%, sys=43.26%, ctx=18444169, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=41914621/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=327145MB, aggrb=372217KB/s, minb=185930KB/s, maxb=186286KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=83732468/28, merge=0/0, ticks=13437692/4572, in_queue=13474260, util=100.00%, aggrios=27939916/52846, aggrmerge=0/0, aggrticks=4517696/389793, aggrin_queue=4918218, aggrutil=100.00% dm-1: ios=3/87822, merge=0/0, ticks=0/14672, in_queue=14672, util=1.21%, aggrios=83780322/91760, aggrmerge=1107/8163, aggrticks=12562052/16096, aggrin_queue=12583792, aggrutil=100.00% xvdc: ios=83780322/91760, merge=1107/8163, ticks=12562052/16096, in_queue=12583792, util=100.00% dm-0: ios=83781426/12436, merge=0/0, ticks=12554856/2392, in_queue=12589888, util=100.00% dm-3: ios=38320/58280, merge=0/0, ticks=998232/1152316, in_queue=2150096, util=21.10%, aggrios=38320/57106, aggrmerge=0/1182, aggrticks=998232/1128504, aggrin_queue=2126504, aggrutil=21.10% xvdd: ios=38320/57106, merge=0/1182, ticks=998232/1128504, in_queue=2126504, util=21.10% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+701e03e4e180 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [396.7MB/0KB/0KB /s] [102K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=4235: Sun Jul 30 00:57:53 2017 read : io=83984MB, bw=95555KB/s, iops=23888, runt=900001msec slat (usec): min=0, max=2757, avg= 6.65, stdev= 4.15 clat (usec): min=0, max=1331.1K, avg=325.84, stdev=5327.61 lat (usec): min=70, max=1331.1K, avg=333.00, stdev=5327.93 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 108], 10.00th=[ 113], 20.00th=[ 118], | 30.00th=[ 124], 40.00th=[ 129], 50.00th=[ 135], 60.00th=[ 141], | 70.00th=[ 149], 80.00th=[ 161], 90.00th=[ 183], 95.00th=[ 205], | 99.00th=[ 278], 99.50th=[ 362], 99.90th=[56064], 99.95th=[100864], | 99.99th=[242688] lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.14%, 250=98.20% lat (usec) : 500=1.30%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.07%, 50=0.10% lat (msec) : 100=0.06%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.85%, sys=21.52%, ctx=10254989, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=21499909/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=4236: Sun Jul 30 00:57:53 2017 read : io=75994MB, bw=86464KB/s, iops=21616, runt=900001msec slat (usec): min=0, max=3457, avg= 6.91, stdev= 4.10 clat (usec): min=2, max=1517.1K, avg=360.70, stdev=5924.51 lat (usec): min=72, max=1517.1K, avg=368.13, stdev=5924.81 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 110], 10.00th=[ 114], 20.00th=[ 121], | 30.00th=[ 126], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 145], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 195], 95.00th=[ 227], | 99.00th=[ 326], 99.50th=[ 406], 99.90th=[64256], 99.95th=[114176], | 99.99th=[264192] lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.11% lat (usec) : 250=96.53%, 500=2.96%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.07%, 50=0.11% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.46%, sys=20.26%, ctx=9368609, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=19454516/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=159978MB, aggrb=182019KB/s, minb=86464KB/s, maxb=95555KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=40941858/29, merge=0/0, ticks=13939452/2924, in_queue=13957884, util=100.00%, aggrios=13673355/191614, aggrmerge=0/0, aggrticks=5425713/41914, aggrin_queue=5472632, aggrutil=70.78% dm-1: ios=72/509243, merge=0/0, ticks=12/62896, in_queue=62908, util=4.30%, aggrios=40823706/556590, aggrmerge=1/16815, aggrticks=5811880/73008, aggrin_queue=5889632, aggrutil=72.65% xvdc: ios=40823706/556590, merge=1/16815, ticks=5811880/73008, in_queue=5889632, util=72.65% dm-0: ios=40823635/64315, merge=0/0, ticks=5806500/12056, in_queue=5833472, util=70.78% dm-3: ios=196358/1284, merge=0/0, ticks=10470628/50792, in_queue=10521516, util=65.71%, aggrios=196358/1281, aggrmerge=0/3, aggrticks=10470612/48468, aggrin_queue=10519068, aggrutil=65.71% xvdd: ios=196358/1281, merge=0/3, ticks=10470612/48468, in_queue=10519068, util=65.71% 4.12.0-rc1+78c45607b909 (2017-05-14 21:54:32 -0400)

The first job running against a full cache managed 17,398 average IOPS which isn’t terrible, but the second job only managed 694 average IOPS. Although this is a lot better than the worst performance seen, it is far below the best so this must be considered broken.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+78c45607b909 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [127.8MB/0KB/0KB /s] [32.8K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1633: Sun Jul 30 01:32:22 2017 read : io=20052MB, bw=22814KB/s, iops=5703, runt=900027msec slat (usec): min=2, max=79803, avg= 7.19, stdev=36.71 clat (usec): min=13, max=929368, avg=1392.96, stdev=9447.28 lat (usec): min=72, max=929376, avg=1400.61, stdev=9448.72 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 105], 10.00th=[ 108], 20.00th=[ 114], | 30.00th=[ 118], 40.00th=[ 121], 50.00th=[ 124], 60.00th=[ 131], | 70.00th=[ 143], 80.00th=[ 175], 90.00th=[ 247], 95.00th=[ 378], | 99.00th=[39168], 99.50th=[61184], 99.90th=[124416], 99.95th=[158720], | 99.99th=[257024] lat (usec) : 20=0.01%, 50=0.01%, 100=0.37%, 250=89.87%, 500=5.63% lat (usec) : 750=0.28%, 1000=0.19% lat (msec) : 2=0.30%, 4=0.01%, 10=0.16%, 20=1.05%, 50=1.44% lat (msec) : 100=0.52%, 250=0.17%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=1.61%, sys=5.89%, ctx=4272767, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=5133359/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1634: Sun Jul 30 01:32:22 2017 read : io=70562MB, bw=80281KB/s, iops=20070, runt=900023msec slat (usec): min=2, max=63808, avg= 6.52, stdev=17.03 clat (usec): min=12, max=1526.4K, avg=389.81, stdev=4546.59 lat (usec): min=69, max=1526.4K, avg=396.77, stdev=4547.95 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108], | 30.00th=[ 112], 40.00th=[ 118], 50.00th=[ 121], 60.00th=[ 125], | 70.00th=[ 135], 80.00th=[ 159], 90.00th=[ 227], 95.00th=[ 286], | 99.00th=[ 988], 99.50th=[16064], 99.90th=[59136], 99.95th=[84480], | 99.99th=[168960] lat (usec) : 20=0.01%, 50=0.01%, 100=0.34%, 250=92.06%, 500=6.20% lat (usec) : 750=0.23%, 1000=0.18% lat (msec) : 2=0.34%, 4=0.01%, 10=0.03%, 20=0.19%, 50=0.29% lat (msec) : 100=0.10%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=4.86%, sys=17.98%, ctx=12698034, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=18063745/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=90614MB, aggrb=103095KB/s, minb=22814KB/s, maxb=80281KB/s, mint=900023msec, maxt=900027msec Disk stats (read/write): dm-2: ios=23193142/28, merge=0/0, ticks=14243636/2712, in_queue=14251904, util=100.00%, aggrios=7849034/861075, aggrmerge=0/0, aggrticks=5211602/303989, aggrin_queue=5516568, aggrutil=99.10% dm-1: ios=1/2233198, merge=0/0, ticks=0/337548, in_queue=337612, util=25.48%, aggrios=22971489/2441741, aggrmerge=998/75445, aggrticks=3693756/390528, aggrin_queue=4083748, aggrutil=94.88% xvdc: ios=22971489/2441741, merge=998/75445, ticks=3693756/390528, in_queue=4083748, util=94.88% dm-0: ios=22972486/284707, merge=0/0, ticks=3711344/62548, in_queue=3776724, util=94.74% dm-3: ios=574615/65320, merge=0/0, ticks=11923464/511872, in_queue=12435368, util=99.10%, aggrios=574505/64312, aggrmerge=109/1008, aggrticks=11894448/464704, aggrin_queue=12359116, aggrutil=99.10% xvdd: ios=574505/64312, merge=109/1008, ticks=11894448/464704, in_queue=12359116, util=99.10% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+78c45607b909 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [71928KB/0KB/0KB /s] [17.1K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=4136: Sun Jul 30 01:48:29 2017 read : io=61171MB, bw=69594KB/s, iops=17398, runt=900076msec slat (usec): min=0, max=3387, avg= 6.22, stdev= 3.45 clat (usec): min=1, max=1174.6K, avg=451.49, stdev=7201.13 lat (usec): min=68, max=1174.6K, avg=458.13, stdev=7201.29 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108], | 30.00th=[ 110], 40.00th=[ 116], 50.00th=[ 120], 60.00th=[ 121], | 70.00th=[ 123], 80.00th=[ 129], 90.00th=[ 173], 95.00th=[ 239], | 99.00th=[ 548], 99.50th=[ 7008], 99.90th=[91648], 99.95th=[150528], | 99.99th=[313344] lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.30% lat (usec) : 250=95.24%, 500=3.42%, 750=0.18%, 1000=0.15% lat (msec) : 2=0.19%, 4=0.01%, 10=0.04%, 20=0.12%, 50=0.17% lat (msec) : 100=0.09%, 250=0.07%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=3.99%, sys=15.44%, ctx=12579924, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=15659896/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=4137: Sun Jul 30 01:48:29 2017 read : io=2440.7MB, bw=2776.7KB/s, iops=694, runt=900089msec slat (usec): min=2, max=821, avg= 7.64, stdev= 2.96 clat (usec): min=1, max=1511.6K, avg=11514.55, stdev=48644.41 lat (usec): min=69, max=1511.7K, avg=11522.66, stdev=48644.81 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 112], | 30.00th=[ 117], 40.00th=[ 119], 50.00th=[ 121], 60.00th=[ 123], | 70.00th=[ 126], 80.00th=[ 159], 90.00th=[22656], 95.00th=[70144], | 99.00th=[240640], 99.50th=[329728], 99.90th=[569344], 99.95th=[675840], | 99.99th=[921600] lat (usec) : 2=0.01%, 20=0.01%, 100=1.12%, 250=83.31%, 500=2.64% lat (usec) : 750=0.21%, 1000=0.15% lat (msec) : 2=0.26%, 4=0.04%, 10=0.33%, 20=1.50%, 50=4.00% lat (msec) : 100=2.81%, 250=2.69%, 500=0.77%, 750=0.13%, 1000=0.02% lat (msec) : 2000=0.01% cpu : usr=0.25%, sys=0.76%, ctx=624059, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=624801/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=63612MB, aggrb=72369KB/s, minb=2776KB/s, maxb=69593KB/s, mint=900076msec, maxt=900089msec Disk stats (read/write): dm-2: ios=16284697/27, merge=0/0, ticks=14316884/11932, in_queue=14332000, util=100.00%, aggrios=5485619/348652, aggrmerge=0/0, aggrticks=6603850/63144, aggrin_queue=6667324, aggrutil=100.00% dm-1: ios=2/873767, merge=0/0, ticks=0/151744, in_queue=151760, util=8.83%, aggrios=16104758/960703, aggrmerge=0/80804, aggrticks=2186320/160864, aggrin_queue=2346844, aggrutil=91.57% xvdc: ios=16104758/960703, merge=0/80804, ticks=2186320/160864, in_queue=2346844, util=91.57% dm-0: ios=16104756/172087, merge=0/0, ticks=2187352/31620, in_queue=2219940, util=91.28% dm-3: ios=352100/102, merge=0/0, ticks=17624200/6068, in_queue=17630272, util=100.00%, aggrios=352101/91, aggrmerge=0/11, aggrticks=17624100/3584, aggrin_queue=17627592, aggrutil=100.00% xvdd: ios=352101/91, merge=0/11, ticks=17624100/3584, in_queue=17627592, util=100.00% 4.12.0-rc1+6cf4cc8f8b3b (2017-05-14 21:54:33 -0400)

Over 20k average IOPS on the run against a full cache means this was marked fixed.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+6cf4cc8f8b3b fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [399.9MB/0KB/0KB /s] [102K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1314: Sun Jul 30 02:20:12 2017 read : io=132841MB, bw=151143KB/s, iops=37785, runt=900001msec slat (usec): min=2, max=4635, avg= 6.74, stdev= 3.66 clat (usec): min=1, max=1208.1K, avg=202.59, stdev=2398.09 lat (usec): min=68, max=1209.8K, avg=209.79, stdev=2398.38 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 108], 10.00th=[ 111], 20.00th=[ 119], | 30.00th=[ 123], 40.00th=[ 129], 50.00th=[ 135], 60.00th=[ 141], | 70.00th=[ 151], 80.00th=[ 163], 90.00th=[ 189], 95.00th=[ 217], | 99.00th=[ 322], 99.50th=[ 402], 99.90th=[18560], 99.95th=[31360], | 99.99th=[88576] lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.17% lat (usec) : 250=97.09%, 500=2.42%, 750=0.06%, 1000=0.03% lat (msec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.05%, 50=0.07% lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=9.52%, sys=36.77%, ctx=17387398, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=34007324/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1315: Sun Jul 30 02:20:12 2017 read : io=134807MB, bw=153381KB/s, iops=38345, runt=900001msec slat (usec): min=2, max=4192, avg= 6.64, stdev= 3.55 clat (usec): min=1, max=1525.4K, avg=199.62, stdev=3082.99 lat (usec): min=66, max=1525.4K, avg=206.71, stdev=3083.18 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 107], 10.00th=[ 111], 20.00th=[ 118], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 139], | 70.00th=[ 149], 80.00th=[ 161], 90.00th=[ 187], 95.00th=[ 221], | 99.00th=[ 326], 99.50th=[ 398], 99.90th=[13888], 99.95th=[26496], | 99.99th=[99840] lat (usec) : 2=0.01%, 10=0.01%, 50=0.01%, 100=0.19%, 250=96.79% lat (usec) : 500=2.73%, 750=0.05%, 1000=0.03% lat (msec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.05%, 50=0.05% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=9.82%, sys=36.76%, ctx=17659747, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=34510679/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=267648MB, aggrb=304524KB/s, minb=151143KB/s, maxb=153380KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=68501557/28, merge=0/0, ticks=13693912/524, in_queue=13720060, util=100.00%, aggrios=22876855/176918, aggrmerge=0/0, aggrticks=4875376/104950, aggrin_queue=4986021, aggrutil=98.38% dm-1: ios=19/418181, merge=0/0, ticks=4/70080, in_queue=70112, util=5.49%, aggrios=68456360/446285, aggrmerge=2397/24774, aggrticks=9939560/79736, aggrin_queue=10019744, aggrutil=98.40% xvdc: ios=68456360/446285, merge=2397/24774, ticks=9939560/79736, in_queue=10019744, util=98.40% dm-0: ios=68458738/53971, merge=0/0, ticks=9966748/12776, in_queue=9996560, util=98.38% dm-3: ios=171810/58604, merge=0/0, ticks=4659376/231996, in_queue=4891392, util=57.31%, aggrios=171801/56085, aggrmerge=9/2519, aggrticks=4655812/142768, aggrin_queue=4798556, aggrutil=57.31% xvdd: ios=171801/56085, merge=9/2519, ticks=4655812/142768, in_queue=4798556, util=57.31% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+6cf4cc8f8b3b fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [397.6MB/0KB/0KB /s] [102K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=3900: Sun Jul 30 02:38:37 2017 read : io=73397MB, bw=83509KB/s, iops=20877, runt=900001msec slat (usec): min=3, max=3087, avg= 6.85, stdev= 3.37 clat (usec): min=9, max=1501.2K, avg=373.90, stdev=6030.69 lat (usec): min=72, max=1501.2K, avg=381.22, stdev=6031.19 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 109], 10.00th=[ 114], 20.00th=[ 121], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 197], 95.00th=[ 235], | 99.00th=[ 338], 99.50th=[ 418], 99.90th=[69120], 99.95th=[118272], | 99.99th=[268288] lat (usec) : 10=0.01%, 50=0.01%, 100=0.10%, 250=96.05%, 500=3.43% lat (usec) : 750=0.03%, 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.08%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.68%, sys=20.45%, ctx=9729438, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=18789539/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=3901: Sun Jul 30 02:38:37 2017 read : io=76248MB, bw=86753KB/s, iops=21688, runt=900001msec slat (usec): min=3, max=3309, avg= 6.82, stdev= 3.51 clat (usec): min=1, max=1436.1K, avg=359.59, stdev=5873.40 lat (usec): min=70, max=1436.1K, avg=366.87, stdev=5873.83 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 107], 10.00th=[ 110], 20.00th=[ 117], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 139], | 70.00th=[ 147], 80.00th=[ 159], 90.00th=[ 181], 95.00th=[ 205], | 99.00th=[ 282], 99.50th=[ 370], 99.90th=[67072], 99.95th=[116224], | 99.99th=[268288] lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.18%, 250=98.06%, 500=1.36%, 750=0.02%, 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.07%, 50=0.12% lat (msec) : 100=0.07%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=5.57%, sys=21.29%, ctx=9691035, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=19519474/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=149645MB, aggrb=170262KB/s, minb=83508KB/s, maxb=86753KB/s, mint=900001msec, maxt=900001msec Disk stats (read/write): dm-2: ios=38297973/29, merge=0/0, ticks=13982664/7004, in_queue=14003132, util=100.00%, aggrios=12791550/194298, aggrmerge=0/0, aggrticks=5442113/27136, aggrin_queue=5472816, aggrutil=68.60% dm-1: ios=102/517327, merge=0/0, ticks=12/62076, in_queue=62116, util=4.27%, aggrios=38174313/566608, aggrmerge=0/16117, aggrticks=5431508/72500, aggrin_queue=5506176, aggrutil=70.51% xvdc: ios=38174313/566608, merge=0/16117, ticks=5431508/72500, in_queue=5506176, util=70.51% dm-0: ios=38174211/65537, merge=0/0, ticks=5427632/12328, in_queue=5450628, util=68.60% dm-3: ios=200338/30, merge=0/0, ticks=10898696/7004, in_queue=10905704, util=68.37%, aggrios=200338/12, aggrmerge=0/18, aggrticks=10898668/2728, aggrin_queue=10901396, aggrutil=68.37% xvdd: ios=200338/12, merge=0/18, ticks=10898668/2728, in_queue=10901396, util=68.37% 4.12.0-rc1+4d44ec5ab751 (2017-05-14 21:54:33 -0400)

Although this time against a full cache both jobs manage a fairly consistent 6 or 7 thousand IOPS, this is nowhere near the level of performance we had seen before so this must be considered broken.

At this point git bisect was able to report that 6cf4cc8f8b3b was the first fixed commit.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+4d44ec5ab751 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [48908KB/0KB/0KB /s] [12.3K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1745: Sun Jul 30 03:23:34 2017 read : io=24635MB, bw=28027KB/s, iops=7006, runt=900037msec slat (usec): min=1, max=14906, avg= 6.77, stdev= 8.06 clat (usec): min=1, max=1249.6K, avg=1132.59, stdev=9627.84 lat (usec): min=69, max=1249.6K, avg=1139.79, stdev=9628.80 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108], | 30.00th=[ 112], 40.00th=[ 118], 50.00th=[ 121], 60.00th=[ 123], | 70.00th=[ 127], 80.00th=[ 147], 90.00th=[ 213], 95.00th=[ 282], | 99.00th=[32128], 99.50th=[57088], 99.90th=[136192], 99.95th=[177152], | 99.99th=[292864] lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.38%, 250=93.13%, 500=3.55%, 750=0.22%, 1000=0.21% lat (msec) : 2=0.29%, 4=0.01%, 10=0.07%, 20=0.57%, 50=0.98% lat (msec) : 100=0.41%, 250=0.18%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=2.13%, sys=6.56%, ctx=5311621, majf=1, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=6306443/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1746: Sun Jul 30 03:23:34 2017 read : io=35476MB, bw=40361KB/s, iops=10090, runt=900046msec slat (usec): min=1, max=23444, avg= 6.51, stdev=11.66 clat (usec): min=1, max=1501.4K, avg=784.01, stdev=8022.09 lat (usec): min=67, max=1501.4K, avg=790.95, stdev=8022.73 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108], | 30.00th=[ 111], 40.00th=[ 117], 50.00th=[ 121], 60.00th=[ 123], | 70.00th=[ 127], 80.00th=[ 145], 90.00th=[ 211], 95.00th=[ 278], | 99.00th=[19840], 99.50th=[40192], 99.90th=[113152], 99.95th=[152576], | 99.99th=[280576] lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.37%, 250=93.16%, 500=4.24%, 750=0.27%, 1000=0.20% lat (msec) : 2=0.29%, 4=0.02%, 10=0.07%, 20=0.37%, 50=0.62% lat (msec) : 100=0.25%, 250=0.11%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=2.58%, sys=9.44%, ctx=6971695, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=9081785/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=60110MB, aggrb=68388KB/s, minb=28027KB/s, maxb=40361KB/s, mint=900037msec, maxt=900046msec Disk stats (read/write): dm-2: ios=15387198/28, merge=0/0, ticks=14306100/6972, in_queue=14316452, util=100.00%, aggrios=5233578/737062, aggrmerge=0/0, aggrticks=5314757/297484, aggrin_queue=5612629, aggrutil=100.00% dm-1: ios=0/1898650, merge=0/0, ticks=0/275700, in_queue=275740, util=20.28%, aggrios=15166622/2060890, aggrmerge=2630/85657, aggrticks=2377016/315952, aggrin_queue=2692844, aggrutil=88.91% xvdc: ios=15166622/2060890, merge=2630/85657, ticks=2377016/315952, in_queue=2692844, util=88.91% dm-0: ios=15169252/249770, merge=0/0, ticks=2424412/51644, in_queue=2477168, util=88.37% dm-3: ios=531482/62766, merge=0/0, ticks=13519860/565108, in_queue=14084980, util=100.00%, aggrios=531464/60098, aggrmerge=18/2668, aggrticks=13515428/447352, aggrin_queue=13962752, aggrutil=100.00% xvdd: ios=531464/60098, merge=18/2668, ticks=13515428/447352, in_queue=13962752, util=100.00% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+4d44ec5ab751 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [42352KB/0KB/0KB /s] [10.6K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=4248: Sun Jul 30 03:40:34 2017 read : io=27816MB, bw=31644KB/s, iops=7911, runt=900111msec slat (usec): min=1, max=3143, avg= 6.38, stdev= 3.68 clat (usec): min=1, max=1247.3K, avg=1002.62, stdev=10214.33 lat (usec): min=69, max=1247.4K, avg=1009.42, stdev=10214.63 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 103], 10.00th=[ 105], 20.00th=[ 107], | 30.00th=[ 109], 40.00th=[ 114], 50.00th=[ 119], 60.00th=[ 121], | 70.00th=[ 123], 80.00th=[ 135], 90.00th=[ 207], 95.00th=[ 278], | 99.00th=[25472], 99.50th=[53504], 99.90th=[150528], 99.95th=[195584], | 99.99th=[337920] lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.49% lat (usec) : 250=93.10%, 500=4.06%, 750=0.23%, 1000=0.21% lat (msec) : 2=0.29%, 4=0.01%, 10=0.05%, 20=0.36%, 50=0.66% lat (msec) : 100=0.32%, 250=0.20%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=2.18%, sys=7.27%, ctx=6208585, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=7120878/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=4249: Sun Jul 30 03:40:34 2017 read : io=21517MB, bw=24481KB/s, iops=6120, runt=900042msec slat (usec): min=1, max=3946, avg= 6.52, stdev= 4.06 clat (usec): min=1, max=1225.4K, avg=1298.37, stdev=13872.59 lat (usec): min=68, max=1225.4K, avg=1305.32, stdev=13872.88 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 103], 10.00th=[ 105], 20.00th=[ 107], | 30.00th=[ 108], 40.00th=[ 111], 50.00th=[ 118], 60.00th=[ 121], | 70.00th=[ 122], 80.00th=[ 127], 90.00th=[ 195], 95.00th=[ 270], | 99.00th=[29568], 99.50th=[82432], 99.90th=[207872], 99.95th=[268288], | 99.99th=[452608] lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.60% lat (usec) : 250=93.47%, 500=3.81%, 750=0.20%, 1000=0.17% lat (msec) : 2=0.28%, 4=0.01%, 10=0.05%, 20=0.23%, 50=0.44% lat (msec) : 100=0.34%, 250=0.34%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=1.71%, sys=5.81%, ctx=5136977, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=5508414/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=49333MB, aggrb=56123KB/s, minb=24480KB/s, maxb=31644KB/s, mint=900042msec, maxt=900111msec Disk stats (read/write): dm-2: ios=12628996/29, merge=0/0, ticks=14325248/9004, in_queue=14337928, util=100.00%, aggrios=4283504/430396, aggrmerge=0/0, aggrticks=6408228/152828, aggrin_queue=6561282, aggrutil=100.00% dm-1: ios=1/1069937, merge=0/0, ticks=0/186088, in_queue=186096, util=11.15%, aggrios=12454547/1169074, aggrmerge=94/92406, aggrticks=1764348/199032, aggrin_queue=1963216, aggrutil=82.20% xvdc: ios=12454547/1169074, merge=94/92406, ticks=1764348/199032, in_queue=1963216, util=82.20% dm-0: ios=12454640/195514, merge=0/0, ticks=1766212/37200, in_queue=1804008, util=82.11% dm-3: ios=395873/25738, merge=0/0, ticks=17458472/235196, in_queue=17693744, util=100.00%, aggrios=395871/25673, aggrmerge=2/65, aggrticks=17458124/214396, aggrin_queue=17672604, aggrutil=100.00% xvdd: ios=395871/25673, merge=2/65, ticks=17458124/214396, in_queue=17672604, util=100.00% Tests against additional kernel builds 4.12.0-rc1 2ea659a9ef48 (2017-05-13 13:19:49 -0700)

A couple of the release candidates with some of Joe’s fixes in had started to produce approaching decent performance so I wanted to go back to the first tagged kernel for this release candidate to get an overview of where Joe had been starting from.

2,069 and 689 average IOPS against a full cache demonstrate the poor performance.

Empty cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.0-rc1+2ea659a9ef48 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [226.2MB/0KB/0KB /s] [57.9K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1410: Sun Jul 30 05:18:46 2017 read : io=119229MB, bw=135648KB/s, iops=33911, runt=900055msec slat (usec): min=3, max=4243, avg= 6.13, stdev= 2.67 clat (usec): min=12, max=777623, avg=227.66, stdev=3034.59 lat (usec): min=69, max=777634, avg=234.21, stdev=3035.00 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 105], 10.00th=[ 107], 20.00th=[ 111], | 30.00th=[ 117], 40.00th=[ 121], 50.00th=[ 123], 60.00th=[ 126], | 70.00th=[ 131], 80.00th=[ 143], 90.00th=[ 183], 95.00th=[ 237], | 99.00th=[ 370], 99.50th=[ 466], 99.90th=[29568], 99.95th=[52480], | 99.99th=[140288] lat (usec) : 20=0.01%, 50=0.01%, 100=0.22%, 250=95.59%, 500=3.73% lat (usec) : 750=0.10%, 1000=0.07% lat (msec) : 2=0.10%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.09% lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=8.43%, sys=31.48%, ctx=19010691, majf=1, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=30522650/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1411: Sun Jul 30 05:18:46 2017 read : io=38320MB, bw=43598KB/s, iops=10899, runt=900034msec slat (usec): min=3, max=2937, avg= 7.12, stdev= 3.52 clat (usec): min=1, max=664000, avg=724.45, stdev=7322.47 lat (usec): min=69, max=664008, avg=732.02, stdev=7323.28 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 107], 20.00th=[ 111], | 30.00th=[ 117], 40.00th=[ 120], 50.00th=[ 122], 60.00th=[ 125], | 70.00th=[ 129], 80.00th=[ 137], 90.00th=[ 165], 95.00th=[ 235], | 99.00th=[16512], 99.50th=[38656], 99.90th=[111104], 99.95th=[148480], | 99.99th=[246784] lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.59%, 250=95.04%, 500=2.69%, 750=0.11%, 1000=0.08% lat (msec) : 2=0.14%, 4=0.02%, 10=0.10%, 20=0.34%, 50=0.52% lat (msec) : 100=0.25%, 250=0.11%, 500=0.01%, 750=0.01% cpu : usr=3.25%, sys=11.57%, ctx=8505263, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=9809817/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=157549MB, aggrb=179244KB/s, minb=43597KB/s, maxb=135647KB/s, mint=900034msec, maxt=900055msec Disk stats (read/write): dm-2: ios=40332438/28, merge=0/0, ticks=14117864/2196, in_queue=14130156, util=100.00%, aggrios=13483192/313952, aggrmerge=0/0, aggrticks=5420121/72913, aggrin_queue=5494664, aggrutil=100.00% dm-1: ios=2/824719, merge=0/0, ticks=0/128108, in_queue=128144, util=10.06%, aggrios=40125306/891909, aggrmerge=1810/39159, aggrticks=5531356/144120, aggrin_queue=5674304, aggrutil=99.96% xvdc: ios=40125306/891909, merge=1810/39159, ticks=5531356/144120, in_queue=5674304, util=99.96% dm-0: ios=40127114/107088, merge=0/0, ticks=5533456/20608, in_queue=5559116, util=99.96% dm-3: ios=322461/10050, merge=0/0, ticks=10726908/70024, in_queue=10796732, util=100.00%, aggrios=322461/8135, aggrmerge=0/1923, aggrticks=10726860/46396, aggrin_queue=10773180, aggrutil=100.00% xvdd: ios=322461/8135, merge=0/1923, ticks=10726860/46396, in_queue=10773180, util=100.00% Full cache $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.0-rc1+2ea659a9ef48 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [11616KB/0KB/0KB /s] [2904/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=3880: Sun Jul 30 05:36:12 2017 read : io=7278.8MB, bw=8279.1KB/s, iops=2069, runt=900169msec slat (usec): min=3, max=749, avg= 7.94, stdev= 6.36 clat (usec): min=2, max=1369.3K, avg=3854.37, stdev=24787.42 lat (usec): min=72, max=1369.3K, avg=3862.76, stdev=24789.32 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 104], 10.00th=[ 107], 20.00th=[ 108], | 30.00th=[ 116], 40.00th=[ 118], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 122], 80.00th=[ 125], 90.00th=[ 163], 95.00th=[13632], | 99.00th=[104960], 99.50th=[162816], 99.90th=[337920], 99.95th=[423936], | 99.99th=[659456] lat (usec) : 4=0.01%, 50=0.01%, 100=0.15%, 250=93.36%, 500=0.46% lat (usec) : 750=0.09%, 1000=0.07% lat (msec) : 2=0.11%, 4=0.01%, 10=0.29%, 20=1.36%, 50=1.89% lat (msec) : 100=1.15%, 250=0.85%, 500=0.18%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.71%, sys=2.61%, ctx=1927655, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1863344/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=3881: Sun Jul 30 05:36:12 2017 read : io=2423.8MB, bw=2757.4KB/s, iops=689, runt=900123msec slat (usec): min=3, max=702, avg= 8.61, stdev= 5.54 clat (usec): min=64, max=1534.9K, avg=11593.97, stdev=45491.15 lat (usec): min=70, max=1534.9K, avg=11603.08, stdev=45492.10 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 103], 10.00th=[ 104], 20.00th=[ 106], | 30.00th=[ 108], 40.00th=[ 113], 50.00th=[ 119], 60.00th=[ 122], | 70.00th=[ 131], 80.00th=[ 163], 90.00th=[28288], 95.00th=[68096], | 99.00th=[216064], 99.50th=[296960], 99.90th=[544768], 99.95th=[659456], | 99.99th=[929792] lat (usec) : 100=0.68%, 250=82.38%, 500=0.40%, 750=0.11%, 1000=0.08% lat (msec) : 2=0.15%, 4=0.02%, 10=0.62%, 20=3.41%, 50=5.54% lat (msec) : 100=3.34%, 250=2.53%, 500=0.61%, 750=0.10%, 1000=0.02% lat (msec) : 2000=0.01% cpu : usr=0.36%, sys=0.86%, ctx=628886, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=620477/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=9702.5MB, aggrb=11037KB/s, minb=2757KB/s, maxb=8279KB/s, mint=900123msec, maxt=900169msec Disk stats (read/write): dm-2: ios=2483744/29, merge=0/0, ticks=14379036/2900, in_queue=14383400, util=100.00%, aggrios=863141/314477, aggrmerge=0/0, aggrticks=6157485/39758, aggrin_queue=6197288, aggrutil=100.00% dm-1: ios=33/837830, merge=0/0, ticks=0/96648, in_queue=96660, util=6.07%, aggrios=2269963/907940, aggrmerge=0/35094, aggrticks=285192/112048, aggrin_queue=397148, aggrutil=34.50% xvdc: ios=2269963/907940, merge=0/35094, ticks=285192/112048, in_queue=397148, util=34.50% dm-0: ios=2269930/105586, merge=0/0, ticks=285436/19728, in_queue=305184, util=30.03% dm-3: ios=319462/16, merge=0/0, ticks=18187020/2900, in_queue=18190020, util=100.00%, aggrios=319462/8, aggrmerge=0/8, aggrticks=18187012/1068, aggrin_queue=18188076, aggrutil=100.00% xvdd: ios=319462/8, merge=0/8, ticks=18187012/1068, in_queue=18188076, util=100.00%
Categories: LUG Community Blogs

Chris Lamb: More Lintian hacking

Planet ALUG - Sat, 29/07/2017 - 09:31

Lintian is static analysis tool for Debian packages, reporting on various errors, omissions and quality-assurance issues to the maintainer.

I seem to have found myself hacking on it a bit more recently (see my previous installment). In particular, here's the code of mine — which made for a total of 20 bugs closed — that made it into the recent 2.5.52 release:

New tags
  • Check for the presence of an .asc signature in a .changes file if an upstream signing key is present. (#833585, tag)
  • Warn when dpkg-statoverride --add is called without a corresponding --list. (#652963, tag)
  • Check for years in debian/copyright that are later than the top entry in debian/changelog. (#807461, tag)
  • Trigger a warning when DEB_BUILD_OPTIONS is used instead of DEB_BUILD_MAINT_OPTIONS. (#833691, tag)
  • Look for "FIXME" and similar placeholders in various files in the debian directory. (#846009, tag)
  • Check for useless build-dependencies on dh-autoreconf or autotools-dev under Debhelper compatibility levels 10 or higher. (#844191, tag)
  • Emit a warning if GObject Introspection packages are missing dependencies on ${gir:Depends}. (#860801, tag)
  • Check packages do not contain upstart configuration under /etc/init. (#825348, tag)
  • Emit a classification tag if maintainer scripts such as debian/postinst is an ELF binary. (tag)
  • Check for overly-generic manual pages such as README.3pm.gz. (#792846, tag)
  • Ensure that (non-ELF) maintainer scripts begin with #!. (#843428, tag)
Regression fixes
  • Ensure r-data-without-readme-source checks the source package, not the binary; README.source files are not installed in the latter. (#866322, tag)
  • Don't emit source-contains-prebuilt-ms-help-file for files generated by Halibut. (#867673, tag)
  • Add .yml to the list of file extensions to avoid false positives when emitting extra-license-file. (#856137, tag)
  • Append a regression test for enumerated lists in the "a) b) c) …" style, which would previously trigger a "duplicate word" warning if the following paragraph began with an "a." (#844166, tag)
Documentation updates
  • Rename copyright-contains-dh-make-perl-boilerplate to copyright-contains-automatically-extracted-boilerplate as it can be generated by other tools such as dh-make-elpa. (#841832, tag)
  • Changes to new-package-should-not-package-python2-module (tag):
    • Upgrade from I: to W:. (#829744)
    • Clarify wording in description to make the justification clearer.
  • Clarify justification in debian-rules-parses-dpkg-parsechangelog. (#865882, tag)
  • Expand the rationale for the latest-debian-changelog-entry-without-new-date tag to mention possible implications for reproducible builds. (tag)
  • Update the source-contains-prebuilt-ms-help-file description; there exists free software to generate .chm files. (tag)
  • Append an example shell snippet to explain how to prevent init.d-script-sourcing-without-test. (tag)
  • Add a missing "contains" verb to the description of the debhelper-autoscript-in-maintainer-scripts tag. (tag)
  • Consistently use the same "Debian style" RFC822 date format for both "Mirror timestamp" and "Last updated" on the Lintian index page. (#828720)
Misc
  • Allow the use of suppress-tags=<tag>[,<tag>[,<tag>]] in ~/.lintianrc. (#764486)
  • Improve the support for "3.0 (git)" packages. However, they remain marked as unsupported-source-format as they are not accepted by the Debian archive. (#605999)
  • Apply patch from Dylan Aïssi to also check for .RData files (not just .Rdata) when checking for the copyright status of R Project data files. (#868178, tag)
  • Match more Lena Söderberg images. (#827941, tag)
  • Refactor a hard-coded list of possible upstream key locations to the common/signing-key-filenames Lintian::Data resource.
Categories: LUG Community Blogs

Steve Kemp: So I'm considering a new project

Planet HantsLUG - Fri, 28/07/2017 - 22:00

In the past there used to be a puppet-labs project called puppet-dashboard, which would let you see the state of your managed-nodes. Having even a very basic and simple "report user-interface" is pretty neat when you're pushing out a change, and you want to see it be applied across your fleet of hosts.

There are some other neat features, such as allowing you to identify failures easily, and see nodes that haven't reported-in recently.

This was spun out into a community-supported project which is largely stale:

Having a dashboard is nice, but the current state of the software is less good. It turns out that the implementation is pretty simple though:

  • Puppet runs on a node.
  • The node reports back to the puppet-master what happened.
  • The puppet-master can optionally HTTP-post that report to the reporting node.

The reporting node can thus receive real-time updates, and do what it wants with them. You can even sidestep the extra server if you wish:

  • The puppet-master can archive the reports locally.

For example on my puppet-master I have this:

root@master /var/lib/puppet/reports # ls | tail -n4 smaug.dh.bytemark.co.uk ssh.steve.org.uk www.dns-api.com www.steve.org.uk

Inside each directory is a bunch of YAML files which describe the state of the host, and the recipes that were applied. Parsing those is pretty simple, the hardest part would be making a useful/attractive GUI. But happily we have the existing one to "inspire" us.

I think I just need to write down a list of assumptions and see if they make sense. After all the existing installation(s) won't break, it's just a matter of deciding whether it is useful/worthwhile way to spend some time.

  • Assume you have 100+ hosts running puppet 4.x
  • Assume you want a broad overview:
    • All the nodes you're managing.
    • Whether their last run triggered a change, resulted in an error, or logged anything.
    • If so what changed/failed/was output?
  • For each individual run you want to see:
    • Rough overview.
  • Assume you don't want to keep history indefinitely, just the last 50 runs or so of each host.

Beyond that you might want to export data about the managed-nodes themselves. For example you might want a list of all the hosts which have "bash" installed on them. Or "All nodes with local user "steve"." I've written that stuff already, as it is very useful for auditing & etc.

The hard part about that is that to get the extra data you'll need to include a puppet module to collect it. I suspect a new dashboard would be broadly interesting/useful but unless you have that extra detail it might not be so useful. You can't point to a slightly more modern installation and say "Yes this is worth migrating to". But if you have extra meta-data you can say:

  • Give me a list of all hosts running wheezy.
  • Give me a list of all hosts running exim4 version 4.84.2-2+deb8u4.

And that facility is very useful when you have shellshock, or similar knocking at your door.

Anyway as a hacky start I wrote some code to parse reports, avoiding the magic object-fu that the YAML would usually invoke. The end result is this:

root@master ~# dump-run www.steve.org.uk www.steve.org.uk Puppet Version: 4.8.2 /var/lib/puppet/reports/www.steve.org.uk/201707291813.yaml Runtime: 2.16 Status:changed Time:2017-07-29 18:13:04 +0000 Resources total -> 176 skipped -> 2 failed -> 0 changed -> 3 out_of_sync -> 3 scheduled -> 0 corrective_change -> 3 Changed Resources Ssh_authorized_key[skx@shelob-s-fi] /etc/puppet/code/environments/production/modules/ssh_keys/manifests/init.pp:17 Ssh_authorized_key[skx@deagol-s-fi] /etc/puppet/code/environments/production/modules/ssh_keys/manifests/init.pp:22 Ssh_authorized_key[steve@ssh.steve.org.uk-s-fi] /etc/puppet/code/environments/production/modules/ssh_keys/manifests/init.pp:27 Skipped Resources Exec[clone sysadmin utils] Exec[update sysadmin utils]
Categories: LUG Community Blogs

Andy Smith: lvmcache on Debian experimental kernel 4.12.2-1~exp1

Planet HantsLUG - Thu, 27/07/2017 - 15:08

In the previous article I mentioned that it may be worth checking if the kernel version 4.12.2-1~exp1 as (currently) supplied by the linux-image-4.12.0-trunk-amd64 package from Debian experimental was new enough to have the dm-cache performance improvements.

The answer appears to be yes.

Categories: LUG Community Blogs

Andy Smith: lvmcache with a 4.12.3 kernel

Planet HantsLUG - Thu, 27/07/2017 - 10:44
Background

In the previous two articles I had discovered that lvmcache had amazing performance on an empty cache but then on every run after that (i.e. when the cache device was full of junk) went scarcely better than baseline HDD.

A few days ago I happened across an email on the linux-lvm list where Mike Snitzer advised:

the [CentOS] 7.4 dm-cache will be much more performant than the 7.3 cache you appear to be using.

…and…

It could be that your workload isn’t accessing the data enough to warrant promotion to the cache. dm-cache is a “hotspot” cache. If you aren’t accessing the data repeatedly then you won’t see much benefit (particularly with the 7.3 and earlier releases).

Just to get a feel, you could try the latest upstream 4.12 kernel to see how effective the 7.4 dm-cache will be for your setup.

I don’t know what kernel version CentOS 7.3 uses, but the VM I’m testing with is Debian testing (buster), so some version of 4.11.x plus backported patches.

That seemed pretty new, but Mike is suggesting 4.12.x so I thought I’d re-test lvmcache with the latest stable upstream kernel, which at the time of writing is version 4.12.3.

Test methodology

This time around I only focused on fio tests, using the same settings as before:

[partial] ioengine=libaio direct=1 iodepth=8 numjobs=2 readwrite=randread random_distribution=zipf:1.2 bs=4k size=2g unlink=1 runtime=20m time_based=1 per_job_logs=1 log_avg_msec=500 write_iops_log=/var/tmp/fio-${FIOTEST}

The only changes were:

  1. to reduce the run time to 20 minutes from 30 minutes, since all the interesting things happened within the first 20 minutes before.
  2. to write an IOPS log entry every 500ms instead of ever 1000ms, as the log files were quite small and some higher resolution might help smooth graphs out.

Last time there was a dramatic difference between the initial run with an empty cache and subsequent runs with a cache volume full of junk, so I did a test for each of those conditions, as well as tests for the baseline SSD and HDD.

The virtual machine had been upgraded from Debian 9 (stretch) to testing (buster), so it still had packaged kernel versions 4.9.30-2 and 4.11.6-1 laying around to test things with. In addition I compiled up version 4.12.3 by copying the .config from 4.11.6-1 then doing make oldconfig accepting all defaults.

Results

Although the fio job spec was essentially the same as in the previous article, I have since worked out how to merge the IOPS logs from both jobs so the graphs will seem to show about double the IOPS as they did before.

All-in-one

Well that’s an interesting set of graphs but rather hard to distinguish. Let’s try that by kernel version.

Baseline SSD by kernel version

A couple of weird things here:

  1. 4.12.3 and 4.11.6-1 are actually fairly consistent, but 4.9.30-2 varies rather a lot.
  2. All kernels show a sharp dip a few minutes in. I don’t know what that is about.

Although these lines do look quite far apart, bear in mind that this graph’s y axis starts at 92k IOPS. The average IOPS didn’t vary that much:

Average IOPS by kernel version 4.9.30-2 4.11.6-1 4.12.3 102,325 102,742 104,352

So there was actually only a 1.9% difference between the worst performer and the best.

Baseline HDD by kernel version

4.9.30-2 and 4.12.3 are close enough here to probably be within the margin of error, but there is something weird going on with 4.11.6-1.

Its average IOPS across the 20 minute test were only 56% of those for 4.12.3 and 53% of those for 4.9.30-2, which is quite a big disparity. I re-ran these tests 5 times to check it wasn’t some anomaly, but no, it’s reproducible.

Maybe something to look into another day.

lvmcache by kernel version

Dragging things back to the point of this article: previously we discovered that lvmcache worked great the first time through, when its cache volume was completely empty, but then subsequent runs all absolutely sucked. They didn’t perform significantly better than HDD baseline.

Let’s graph all the lvmcache results for each kernel version against the SSD baseline for that kernel to see if things changed at all.

lvmcache 4.9.30-2

This is the similar to what we saw before: an empty cache volume produces decent results of around 47k IOPS. Although it’s interesting that the second job is down around 1k IOPS. Again the results on a full cache are poor. In fact the results for the second job of the empty cache are about the same as the results for both jobs on a full cache.

lvmcache 4.11.6-1

Same story again here, although the performance is a little higher. Again the first job on an empty cache is getting the big results of almost 60k IOPS while the second job—and both jobs on a full cache—get only around 1k IOPS.

lvmcache 4.12.3

Wow. Something dramatic has been fixed. The performance on an empty cache is still better, but crucially the performance on a full cache pretty quickly becomes very close to baseline SSD.

Also the runs against both the empty and full cache device result in both jobs getting roughly the same IOPS performance rather than the first job being great and all others very poor.

What’s next?

It’s really encouraging that the performance is so much better with 4.12.3. It’s changed lvmcache from a “hmm, maybe” option to one that I would strongly consider using anywhere I could.

It’s a shame though that such a new kernel is required. The kernel version in Debian testing (buster) is currently 4.11.6-1. Debian experimental’s linux-image-4.12.0-trunk-amd64 package currently has version 4.12.2-1 so I should test if that is new enough I tested to see if that was new enough.

Failing that I think I should git bisect or similar in order to find out exactly which changeset fixed this, so I could have some chance of knowing when it hits a packaged version.

Test output Kernel 4.12.3 Baseline SSD (4.12.3) $ (cd /srv/ssd/fio && FIOTEST=ssd-4.12.3 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [407.9MB/0KB/0KB /s] [104K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=781: Wed Jul 26 19:03:54 2017 read : io=237280MB, bw=202479KB/s, iops=50619, runt=1200001msec slat (usec): min=0, max=2094, avg= 4.55, stdev= 2.02 clat (usec): min=36, max=8139, avg=151.16, stdev=42.74 lat (usec): min=94, max=8140, avg=156.24, stdev=42.71 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 111], 10.00th=[ 117], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 147], | 70.00th=[ 157], 80.00th=[ 173], 90.00th=[ 201], 95.00th=[ 233], | 99.00th=[ 314], 99.50th=[ 338], 99.90th=[ 406], 99.95th=[ 430], | 99.99th=[ 502] lat (usec) : 50=0.01%, 100=0.10%, 250=96.22%, 500=3.67%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=12.41%, sys=35.81%, ctx=33753770, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=60743781/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=782: Wed Jul 26 19:03:54 2017 read : io=251874MB, bw=214932KB/s, iops=53733, runt=1200001msec slat (usec): min=0, max=1050, avg= 4.58, stdev= 1.99 clat (usec): min=25, max=8148, avg=142.00, stdev=31.19 lat (usec): min=64, max=8151, avg=147.10, stdev=31.17 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 109], 10.00th=[ 114], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 135], 60.00th=[ 141], | 70.00th=[ 149], 80.00th=[ 161], 90.00th=[ 179], 95.00th=[ 199], | 99.00th=[ 243], 99.50th=[ 262], 99.90th=[ 314], 99.95th=[ 334], | 99.99th=[ 414] lat (usec) : 50=0.01%, 100=0.32%, 250=98.86%, 500=0.81%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=12.91%, sys=38.04%, ctx=34484181, majf=1, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=64479700/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=489154MB, aggrb=417411KB/s, minb=202479KB/s, maxb=214932KB/s, mint=1200001msec, maxt=1200001msec Disk stats (read/write): xvda: ios=125216086/42, merge=14/33, ticks=18252156/0, in_queue=18285152, util=100.00%

Combined average IOPS were 104,352 or 52,176 per job.

Baseline HDD (4.12.3) $ (cd /srv/slow/fio && FIOTEST=hdd-4.12.3 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [3711KB/0KB/0KB /s] [927/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=2125: Wed Jul 26 22:41:52 2017 read : io=2194.8MB, bw=1872.8KB/s, iops=468, runt=1200071msec slat (usec): min=1, max=125, avg=14.18, stdev= 8.85 clat (usec): min=54, max=743383, avg=17067.31, stdev=35286.68 lat (usec): min=65, max=743393, avg=17082.36, stdev=35286.75 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 77], 10.00th=[ 87], 20.00th=[ 109], | 30.00th=[ 139], 40.00th=[ 205], 50.00th=[ 1012], 60.00th=[ 1560], | 70.00th=[15424], 80.00th=[26752], 90.00th=[50944], 95.00th=[82432], | 99.00th=[168960], 99.50th=[214016], 99.90th=[313344], 99.95th=[366592], | 99.99th=[481280] lat (usec) : 100=15.90%, 250=25.89%, 500=3.68%, 750=1.82%, 1000=2.48% lat (msec) : 2=10.76%, 4=0.62%, 10=3.27%, 20=10.53%, 50=14.74% lat (msec) : 100=6.77%, 250=3.24%, 500=0.28%, 750=0.01% cpu : usr=0.50%, sys=1.08%, ctx=549488, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=561853/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=2126: Wed Jul 26 22:41:52 2017 read : io=1975.2MB, bw=1685.4KB/s, iops=421, runt=1200063msec slat (usec): min=1, max=169, avg=14.37, stdev= 8.96 clat (usec): min=42, max=894017, avg=18966.69, stdev=38900.32 lat (usec): min=67, max=894053, avg=18981.93, stdev=38900.42 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 77], 10.00th=[ 87], 20.00th=[ 110], | 30.00th=[ 143], 40.00th=[ 213], 50.00th=[ 1064], 60.00th=[ 5984], | 70.00th=[16768], 80.00th=[29056], 90.00th=[57088], 95.00th=[91648], | 99.00th=[189440], 99.50th=[234496], 99.90th=[346112], 99.95th=[399360], | 99.99th=[501760] lat (usec) : 50=0.01%, 100=15.65%, 250=25.63%, 500=3.66%, 750=1.72% lat (usec) : 1000=2.30% lat (msec) : 2=10.10%, 4=0.48%, 10=3.19%, 20=10.79%, 50=14.82% lat (msec) : 100=7.33%, 250=3.93%, 500=0.39%, 750=0.01%, 1000=0.01% cpu : usr=0.48%, sys=0.96%, ctx=494683, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=505636/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=4169.9MB, aggrb=3558KB/s, minb=1685KB/s, maxb=1872KB/s, mint=1200063msec, maxt=1200071msec Disk stats (read/write): xvde: ios=1067472/11, merge=0/17, ticks=19179732/2528, in_queue=19182776, util=100.00%

Average combined IOPS were 889.

lvmcache (4.12.3, empty cache device) $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.12.3 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=0): [f(2)] [100.0% done] [395.1MB/0KB/0KB /s] [101K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1861: Wed Jul 26 21:43:37 2017 read : io=233021MB, bw=198845KB/s, iops=49711, runt=1200001msec slat (usec): min=1, max=696, avg= 6.62, stdev= 2.57 clat (usec): min=32, max=536652, avg=151.86, stdev=102.22 lat (usec): min=71, max=536660, avg=158.96, stdev=102.21 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 111], 10.00th=[ 117], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 147], | 70.00th=[ 157], 80.00th=[ 173], 90.00th=[ 203], 95.00th=[ 239], | 99.00th=[ 326], 99.50th=[ 362], 99.90th=[ 454], 99.95th=[ 510], | 99.99th=[ 1128] lat (usec) : 50=0.01%, 100=0.09%, 250=95.76%, 500=4.10%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 750=0.01% cpu : usr=12.28%, sys=45.34%, ctx=27534900, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=59653437/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1862: Wed Jul 26 21:43:37 2017 read : io=218979MB, bw=186862KB/s, iops=46715, runt=1200001msec slat (usec): min=1, max=2349, avg= 6.67, stdev= 2.64 clat (usec): min=26, max=512486, avg=162.12, stdev=1013.93 lat (usec): min=64, max=512495, avg=169.27, stdev=1014.06 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 109], 10.00th=[ 114], 20.00th=[ 121], | 30.00th=[ 126], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 165], 90.00th=[ 187], 95.00th=[ 209], | 99.00th=[ 274], 99.50th=[ 306], 99.90th=[ 450], 99.95th=[ 9664], | 99.99th=[44800] lat (usec) : 50=0.01%, 100=1.07%, 250=97.19%, 500=1.66%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=11.90%, sys=42.52%, ctx=25462344, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=56058591/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=452000MB, aggrb=385706KB/s, minb=186861KB/s, maxb=198844KB/s, mint=1200001msec, maxt=1200001msec Disk stats (read/write): dm-2: ios=115688209/28, merge=0/0, ticks=18037068/4148, in_queue=18097348, util=100.00%, aggrios=38592557/60889, aggrmerge=0/0, aggrticks=6093288/367342, aggrin_queue=6477849, aggrutil=100.00% dm-1: ios=23/117017, merge=0/0, ticks=4/19880, in_queue=19888, util=1.27%, aggrios=115726125/113023, aggrmerge=2/18688, aggrticks=16993500/21068, aggrin_queue=17022812, aggrutil=100.00% xvdc: ios=115726125/113023, merge=2/18688, ticks=16993500/21068, in_queue=17022812, util=100.00% dm-0: ios=115726104/16785, merge=0/0, ticks=16982820/3328, in_queue=17037740, util=100.00% dm-3: ios=51545/48866, merge=0/0, ticks=1297040/1078820, in_queue=2375920, util=20.61%, aggrios=51545/48849, aggrmerge=0/17, aggrticks=1297044/1075220, aggrin_queue=2372260, aggrutil=20.61% xvdd: ios=51545/48849, merge=0/17, ticks=1297044/1075220, in_queue=2372260, util=20.61% lvmcache (4.12.3, full cache device) $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.12.3 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [391.3MB/0KB/0KB /s] [100K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1988: Wed Jul 26 22:13:33 2017 read : io=142920MB, bw=121958KB/s, iops=30489, runt=1200001msec slat (usec): min=0, max=1489, avg= 6.76, stdev= 2.74 clat (usec): min=27, max=1517.9K, avg=253.15, stdev=3966.27 lat (usec): min=66, max=1517.1K, avg=260.40, stdev=3966.63 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 108], 10.00th=[ 113], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 145], | 70.00th=[ 155], 80.00th=[ 171], 90.00th=[ 203], 95.00th=[ 245], | 99.00th=[ 350], 99.50th=[ 402], 99.90th=[27776], 99.95th=[61696], | 99.99th=[179200] lat (usec) : 50=0.01%, 100=0.86%, 250=94.46%, 500=4.44%, 750=0.04% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.04%, 50=0.06% lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=7.69%, sys=28.25%, ctx=17949976, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=36587548/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1989: Wed Jul 26 22:13:33 2017 read : io=142448MB, bw=121556KB/s, iops=30388, runt=1200001msec slat (usec): min=1, max=2795, avg= 6.79, stdev= 2.76 clat (usec): min=14, max=1271.5K, avg=253.96, stdev=4051.40 lat (usec): min=66, max=1271.5K, avg=261.25, stdev=4051.71 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 108], 10.00th=[ 113], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 197], 95.00th=[ 231], | 99.00th=[ 330], 99.50th=[ 382], 99.90th=[28288], 99.95th=[63232], | 99.99th=[187392] lat (usec) : 20=0.01%, 50=0.01%, 100=1.41%, 250=94.94%, 500=3.42% lat (usec) : 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.04%, 50=0.06% lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=7.79%, sys=28.22%, ctx=17393786, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=36466725/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=285368MB, aggrb=243514KB/s, minb=121555KB/s, maxb=121958KB/s, mint=1200001msec, maxt=1200001msec Disk stats (read/write): dm-2: ios=73045110/29, merge=0/0, ticks=18465212/5508, in_queue=18503620, util=100.00%, aggrios=24373349/191891, aggrmerge=0/0, aggrticks=6909396/40201, aggrin_queue=6960561, aggrutil=80.17% dm-1: ios=78/509945, merge=0/0, ticks=16/63772, in_queue=63792, util=3.22%, aggrios=72922899/556621, aggrmerge=0/17454, aggrticks=10752172/73676, aggrin_queue=10833040, aggrutil=81.60% xvdc: ios=72922899/556621, merge=0/17454, ticks=10752172/73676, in_queue=10833040, util=81.60% dm-0: ios=72922821/64279, merge=0/0, ticks=10747112/12240, in_queue=10792240, util=80.17% dm-3: ios=197149/1449, merge=0/0, ticks=9981060/44592, in_queue=10025652, util=46.75%, aggrios=197149/1433, aggrmerge=0/16, aggrticks=9981044/40808, aggrin_queue=10021852, aggrutil=46.75% xvdd: ios=197149/1433, merge=0/16, ticks=9981044/40808, in_queue=10021852, util=46.75% Kernel 4.11.6-1 Baseline SSD (4.11.6-1) $ (cd /srv/ssd/fio && FIOTEST=ssd-4.11.6-1 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [401.9MB/0KB/0KB /s] [103K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=2028: Wed Jul 26 18:01:52 2017 read : io=240561MB, bw=205278KB/s, iops=51319, runt=1200001msec slat (usec): min=1, max=231, avg= 4.95, stdev= 1.91 clat (usec): min=32, max=7892, avg=148.38, stdev=39.27 lat (usec): min=95, max=7896, avg=153.84, stdev=39.25 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 110], 10.00th=[ 115], 20.00th=[ 122], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 193], 95.00th=[ 223], | 99.00th=[ 294], 99.50th=[ 326], 99.90th=[ 390], 99.95th=[ 410], | 99.99th=[ 474] lat (usec) : 50=0.01%, 100=0.12%, 250=97.04%, 500=2.83%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=13.47%, sys=38.85%, ctx=32087576, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=61583511/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=2029: Wed Jul 26 18:01:52 2017 read : io=241050MB, bw=205696KB/s, iops=51423, runt=1200001msec slat (usec): min=1, max=1065, avg= 4.98, stdev= 1.91 clat (usec): min=38, max=7985, avg=148.06, stdev=41.67 lat (usec): min=63, max=7992, avg=153.54, stdev=41.65 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 109], 10.00th=[ 113], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 197], 95.00th=[ 229], | 99.00th=[ 306], 99.50th=[ 334], 99.90th=[ 398], 99.95th=[ 418], | 99.99th=[ 486] lat (usec) : 50=0.01%, 100=0.14%, 250=96.43%, 500=3.41%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=13.70%, sys=38.78%, ctx=31782354, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=61708761/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=481610MB, aggrb=410973KB/s, minb=205278KB/s, maxb=205695KB/s, mint=1200001msec, maxt=1200001msec Disk stats (read/write): xvda: ios=123286829/13, merge=0/21, ticks=18148640/0, in_queue=18184316, util=100.00%

Combined average IOPS were 102,742 or 51,371 per job.

Baseline HDD (4.11.6-1) $ (cd /srv/slow/fio && FIOTEST=hdd-4.11.6-1 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [2108KB/0KB/0KB /s] [527/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=904: Thu Jul 27 09:17:21 2017 read : io=1218.3MB, bw=1039.6KB/s, iops=259, runt=1200088msec slat (usec): min=0, max=93, avg=14.37, stdev= 8.02 clat (usec): min=62, max=1126.5K, avg=30763.45, stdev=56969.73 lat (usec): min=68, max=1126.5K, avg=30778.70, stdev=56969.84 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 82], 10.00th=[ 96], 20.00th=[ 129], | 30.00th=[ 175], 40.00th=[ 724], 50.00th=[ 4320], 60.00th=[15552], | 70.00th=[29824], 80.00th=[49408], 90.00th=[90624], 95.00th=[140288], | 99.00th=[272384], 99.50th=[333824], 99.90th=[489472], 99.95th=[561152], | 99.99th=[733184] lat (usec) : 100=11.11%, 250=23.85%, 500=3.45%, 750=1.71%, 1000=1.28% lat (msec) : 2=6.65%, 4=1.72%, 10=6.07%, 20=7.46%, 50=16.95% lat (msec) : 100=11.01%, 250=7.46%, 500=1.19%, 750=0.09%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.21%, sys=0.60%, ctx=303130, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=311877/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=905: Thu Jul 27 09:17:21 2017 read : io=1142.7MB, bw=998342B/s, iops=243, runt=1200098msec slat (usec): min=0, max=171, avg=14.44, stdev= 7.98 clat (usec): min=62, max=1288.1K, avg=32802.09, stdev=60366.24 lat (usec): min=65, max=1288.2K, avg=32817.41, stdev=60366.35 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 83], 10.00th=[ 98], 20.00th=[ 133], | 30.00th=[ 183], 40.00th=[ 1064], 50.00th=[ 7840], 60.00th=[16768], | 70.00th=[30592], 80.00th=[51456], 90.00th=[95744], 95.00th=[148480], | 99.00th=[292864], 99.50th=[354304], 99.90th=[522240], 99.95th=[593920], | 99.99th=[749568] lat (usec) : 100=10.59%, 250=22.93%, 500=3.11%, 750=1.64%, 1000=1.14% lat (msec) : 2=6.16%, 4=1.30%, 10=5.73%, 20=10.31%, 50=16.53% lat (msec) : 100=11.09%, 250=7.93%, 500=1.43%, 750=0.11%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.20%, sys=0.57%, ctx=283410, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=292507/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=2360.9MB, aggrb=2014KB/s, minb=974KB/s, maxb=1039KB/s, mint=1200088msec, maxt=1200098msec Disk stats (read/write): xvde: ios=604364/11, merge=0/17, ticks=19188840/3004, in_queue=19192884, util=100.00%

Average combined IOPS were 502.

lvmcache (4.11.6-1, empty cache device) $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.11.6-1 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [229.7MB/0KB/0KB /s] [58.8K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1281: Wed Jul 26 16:36:01 2017 read : io=270469MB, bw=230800KB/s, iops=57700, runt=1200001msec slat (usec): min=1, max=665, avg= 6.06, stdev= 1.39 clat (usec): min=44, max=1129.8K, avg=130.33, stdev=356.51 lat (usec): min=98, max=1129.8K, avg=136.83, stdev=356.51 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 105], 10.00th=[ 107], 20.00th=[ 111], | 30.00th=[ 116], 40.00th=[ 120], 50.00th=[ 123], 60.00th=[ 126], | 70.00th=[ 133], 80.00th=[ 141], 90.00th=[ 161], 95.00th=[ 183], | 99.00th=[ 235], 99.50th=[ 258], 99.90th=[ 310], 99.95th=[ 346], | 99.99th=[ 940] lat (usec) : 50=0.01%, 100=0.32%, 250=99.05%, 500=0.60%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=13.58%, sys=47.73%, ctx=30980580, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=69240158/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1282: Wed Jul 26 16:36:01 2017 read : io=5438.4MB, bw=4640.4KB/s, iops=1160, runt=1200094msec slat (usec): min=3, max=501, avg= 8.90, stdev= 3.01 clat (usec): min=1, max=1109.4K, avg=6884.09, stdev=22209.48 lat (usec): min=66, max=1109.5K, avg=6893.50, stdev=22209.59 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 76], 10.00th=[ 98], 20.00th=[ 111], | 30.00th=[ 118], 40.00th=[ 122], 50.00th=[ 127], 60.00th=[ 135], | 70.00th=[ 149], 80.00th=[ 564], 90.00th=[23680], 95.00th=[43776], | 99.00th=[99840], 99.50th=[125440], 99.90th=[197632], 99.95th=[276480], | 99.99th=[544768] lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=10.37%, 250=68.11%, 500=1.27%, 750=0.77%, 1000=0.64% lat (msec) : 2=1.11%, 4=0.37%, 10=1.68%, 20=4.42%, 50=7.14% lat (msec) : 100=3.13%, 250=0.94%, 500=0.04%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.39%, sys=1.53%, ctx=1364109, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1392218/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=275908MB, aggrb=235422KB/s, minb=4640KB/s, maxb=230800KB/s, mint=1200001msec, maxt=1200094msec Disk stats (read/write): dm-2: ios=70632376/28, merge=0/0, ticks=18853972/3148, in_queue=18878504, util=100.00%, aggrios=23553400/9408, aggrmerge=0/0, aggrticks=6264994/404240, aggrin_queue=6677382, aggrutil=100.00% dm-1: ios=1/372, merge=0/0, ticks=0/120, in_queue=120, util=0.01%, aggrios=70270018/409, aggrmerge=476/2, aggrticks=9314984/128, aggrin_queue=9321056, aggrutil=100.00% xvdc: ios=70270018/409, merge=476/2, ticks=9314984/128, in_queue=9321056, util=100.00% dm-0: ios=70270493/39, merge=0/0, ticks=9339988/4, in_queue=9364408, util=100.00% dm-3: ios=389707/27815, merge=0/0, ticks=9454996/1212596, in_queue=10667620, util=100.00%, aggrios=389707/27324, aggrmerge=0/491, aggrticks=9454912/1121628, aggrin_queue=10576496, aggrutil=100.00% xvdd: ios=389707/27324, merge=0/491, ticks=9454912/1121628, in_queue=10576496, util=100.00% lvmcache (4.11.6-1, full cache device) $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.11.6-1 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [4904KB/0KB/0KB /s] [1226/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1723: Wed Jul 26 17:03:19 2017 read : io=2713.5MB, bw=2315.3KB/s, iops=578, runt=1200126msec slat (usec): min=4, max=103, avg=14.92, stdev= 8.59 clat (usec): min=67, max=943720, avg=13802.04, stdev=37388.05 lat (usec): min=76, max=943732, avg=13817.72, stdev=37388.50 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 112], 10.00th=[ 115], 20.00th=[ 119], | 30.00th=[ 127], 40.00th=[ 131], 50.00th=[ 141], 60.00th=[ 155], | 70.00th=[ 201], 80.00th=[17280], 90.00th=[43776], 95.00th=[79360], | 99.00th=[181248], 99.50th=[234496], 99.90th=[366592], 99.95th=[428032], | 99.99th=[585728] lat (usec) : 100=0.35%, 250=72.25%, 500=0.73%, 750=0.26%, 1000=0.22% lat (msec) : 2=0.53%, 4=0.06%, 10=1.18%, 20=6.18%, 50=9.45% lat (msec) : 100=5.33%, 250=3.06%, 500=0.39%, 750=0.02%, 1000=0.01% cpu : usr=0.37%, sys=1.29%, ctx=688778, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=694643/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1724: Wed Jul 26 17:03:19 2017 read : io=2493.3MB, bw=2127.5KB/s, iops=531, runt=1200115msec slat (usec): min=3, max=113, avg=15.20, stdev= 8.84 clat (usec): min=29, max=929665, avg=15021.94, stdev=39251.61 lat (usec): min=76, max=929675, avg=15037.91, stdev=39252.04 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 112], 10.00th=[ 116], 20.00th=[ 124], | 30.00th=[ 129], 40.00th=[ 133], 50.00th=[ 143], 60.00th=[ 159], | 70.00th=[ 217], 80.00th=[19328], 90.00th=[48896], 95.00th=[85504], | 99.00th=[191488], 99.50th=[242688], 99.90th=[374784], 99.95th=[440320], | 99.99th=[602112] lat (usec) : 50=0.01%, 100=0.40%, 250=70.92%, 500=0.78%, 750=0.31% lat (usec) : 1000=0.23% lat (msec) : 2=0.60%, 4=0.08%, 10=1.17%, 20=5.98%, 50=9.80% lat (msec) : 100=5.83%, 250=3.46%, 500=0.43%, 750=0.03%, 1000=0.01% cpu : usr=0.35%, sys=1.20%, ctx=632752, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=638283/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=5206.8MB, aggrb=4442KB/s, minb=2127KB/s, maxb=2315KB/s, mint=1200115msec, maxt=1200126msec Disk stats (read/write): dm-2: ios=1332926/29, merge=0/0, ticks=19182084/4352, in_queue=19187080, util=100.00%, aggrios=445250/12578, aggrmerge=0/0, aggrticks=6224384/4612, aggrin_queue=6229029, aggrutil=100.00% dm-1: ios=60/34939, merge=0/0, ticks=16/6800, in_queue=6820, util=0.28%, aggrios=954236/35982, aggrmerge=0/1722, aggrticks=141920/7080, aggrin_queue=148992, aggrutil=11.70% xvdc: ios=954236/35982, merge=0/1722, ticks=141920/7080, in_queue=148992, util=11.70% dm-0: ios=954176/2767, merge=0/0, ticks=142036/720, in_queue=142780, util=11.53% dm-3: ios=381514/28, merge=0/0, ticks=18531100/6316, in_queue=18537488, util=100.00%, aggrios=381514/13, aggrmerge=0/15, aggrticks=18530984/2632, aggrin_queue=18533840, aggrutil=100.00% xvdd: ios=381514/13, merge=0/15, ticks=18530984/2632, in_queue=18533840, util=100.00% Kernel 4.9.30-2 Baseline SSD (4.9.30-2) $ (cd /srv/ssd/fio && FIOTEST=ssd-4.9.30-2 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=0): [f(2)] [100.0% done] [399.8MB/0KB/0KB /s] [102K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=770: Thu Jul 27 02:27:43 2017 read : io=241655MB, bw=206212KB/s, iops=51553, runt=1200001msec slat (usec): min=1, max=1046, avg= 4.42, stdev= 1.59 clat (usec): min=28, max=4828, avg=148.39, stdev=42.92 lat (usec): min=95, max=4830, avg=153.26, stdev=42.90 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 109], 10.00th=[ 113], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 197], 95.00th=[ 235], | 99.00th=[ 322], 99.50th=[ 350], 99.90th=[ 422], 99.95th=[ 454], | 99.99th=[ 532] lat (usec) : 50=0.01%, 100=0.15%, 250=96.00%, 500=3.84%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=13.51%, sys=36.36%, ctx=33974378, majf=1, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=61863654/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=771: Thu Jul 27 02:27:43 2017 read : io=238123MB, bw=203198KB/s, iops=50799, runt=1200001msec slat (usec): min=1, max=997, avg= 4.39, stdev= 1.60 clat (usec): min=42, max=4827, avg=150.72, stdev=47.68 lat (usec): min=65, max=4832, avg=155.56, stdev=47.66 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 109], 10.00th=[ 113], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 143], | 70.00th=[ 155], 80.00th=[ 171], 90.00th=[ 203], 95.00th=[ 249], | 99.00th=[ 342], 99.50th=[ 386], 99.90th=[ 458], 99.95th=[ 486], | 99.99th=[ 564] lat (usec) : 50=0.01%, 100=0.14%, 250=94.88%, 500=4.95%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=13.44%, sys=35.68%, ctx=34074578, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=60959403/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=479778MB, aggrb=409409KB/s, minb=203197KB/s, maxb=206212KB/s, mint=1200001msec, maxt=1200001msec Disk stats (read/write): xvda: ios=122797792/38, merge=0/33, ticks=18273356/0, in_queue=18319740, util=100.00%

Combined average IOPS were 102,352 or 51,176 per job.

Baseline HDD (4.9.30-2) $ (cd /srv/slow/fio && FIOTEST=hdd-4.9.30-2 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [3847KB/0KB/0KB /s] [961/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=1290: Thu Jul 27 02:57:41 2017 read : io=2348.2MB, bw=2003.7KB/s, iops=500, runt=1200052msec slat (usec): min=2, max=88, avg=12.51, stdev= 7.84 clat (usec): min=28, max=877693, avg=15952.77, stdev=32920.15 lat (usec): min=65, max=877697, avg=15966.13, stdev=32920.23 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 77], 10.00th=[ 86], 20.00th=[ 108], | 30.00th=[ 139], 40.00th=[ 207], 50.00th=[ 996], 60.00th=[ 1464], | 70.00th=[14912], 80.00th=[25472], 90.00th=[47872], 95.00th=[76288], | 99.00th=[156672], 99.50th=[197632], 99.90th=[296960], 99.95th=[346112], | 99.99th=[473088] lat (usec) : 50=0.01%, 100=15.94%, 250=25.74%, 500=3.72%, 750=1.98% lat (usec) : 1000=2.70% lat (msec) : 2=11.26%, 4=0.46%, 10=3.02%, 20=11.04%, 50=14.68% lat (msec) : 100=6.41%, 250=2.83%, 500=0.20%, 750=0.01%, 1000=0.01% cpu : usr=0.47%, sys=1.14%, ctx=589357, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=601124/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=1291: Thu Jul 27 02:57:41 2017 read : io=2094.6MB, bw=1787.3KB/s, iops=446, runt=1200065msec slat (usec): min=2, max=136, avg=12.86, stdev= 8.11 clat (usec): min=57, max=1113.9K, avg=17886.33, stdev=37285.63 lat (usec): min=62, max=1113.1K, avg=17900.05, stdev=37285.72 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 77], 10.00th=[ 87], 20.00th=[ 109], | 30.00th=[ 141], 40.00th=[ 209], 50.00th=[ 1032], 60.00th=[ 1832], | 70.00th=[16064], 80.00th=[27520], 90.00th=[52992], 95.00th=[85504], | 99.00th=[179200], 99.50th=[226304], 99.90th=[346112], 99.95th=[391168], | 99.99th=[536576] lat (usec) : 100=15.88%, 250=25.66%, 500=3.65%, 750=1.82%, 1000=2.45% lat (msec) : 2=10.62%, 4=0.32%, 10=2.87%, 20=11.21%, 50=14.75% lat (msec) : 100=6.98%, 250=3.46%, 500=0.34%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.39%, sys=1.08%, ctx=526250, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=536196/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=4442.7MB, aggrb=3790KB/s, minb=1787KB/s, maxb=2003KB/s, mint=1200052msec, maxt=1200065msec Disk stats (read/write): xvde: ios=1137197/11, merge=0/17, ticks=19176720/2496, in_queue=19179612, util=100.00%

Average combined IOPS were 946.

lvmcache (4.9.30-2, empty cache device) $ (cd /srv/cache/fio && FIOTEST=lvmcache-empty-4.9.30-2 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [93173KB/0KB/0KB /s] [23.3K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=8863: Wed Jul 26 15:33:04 2017 read : io=222011MB, bw=189442KB/s, iops=47360, runt=1200047msec slat (usec): min=1, max=536, avg= 5.50, stdev= 1.29 clat (usec): min=45, max=1066.2K, avg=161.37, stdev=2258.81 lat (usec): min=70, max=1066.2K, avg=167.26, stdev=2258.82 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 109], | 30.00th=[ 114], 40.00th=[ 118], 50.00th=[ 121], 60.00th=[ 124], | 70.00th=[ 129], 80.00th=[ 137], 90.00th=[ 165], 95.00th=[ 197], | 99.00th=[ 282], 99.50th=[ 322], 99.90th=[ 422], 99.95th=[ 1176], | 99.99th=[90624] lat (usec) : 50=0.01%, 100=0.30%, 250=97.69%, 500=1.93%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=10.37%, sys=37.16%, ctx=31754180, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=56834728/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=8864: Wed Jul 26 15:33:04 2017 read : io=6439.3MB, bw=5494.3KB/s, iops=1373, runt=1200133msec slat (usec): min=2, max=363, avg= 7.53, stdev= 2.49 clat (usec): min=1, max=1082.6K, avg=5814.26, stdev=21389.31 lat (usec): min=67, max=1082.6K, avg=5822.26, stdev=21389.42 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 97], 10.00th=[ 104], 20.00th=[ 109], | 30.00th=[ 116], 40.00th=[ 119], 50.00th=[ 123], 60.00th=[ 127], | 70.00th=[ 139], 80.00th=[ 318], 90.00th=[18304], 95.00th=[35072], | 99.00th=[89600], 99.50th=[120320], 99.90th=[218112], 99.95th=[337920], | 99.99th=[593920] lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=5.70% lat (usec) : 250=73.84%, 500=1.37%, 750=0.81%, 1000=0.81% lat (msec) : 2=1.32%, 4=0.26%, 10=1.52%, 20=5.19%, 50=6.12% lat (msec) : 100=2.26%, 250=0.72%, 500=0.05%, 750=0.03%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.43%, sys=1.55%, ctx=1616527, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1648391/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=228450MB, aggrb=194922KB/s, minb=5494KB/s, maxb=189441KB/s, mint=1200047msec, maxt=1200133msec Disk stats (read/write): dm-2: ios=58483119/28, merge=0/0, ticks=18951620/1096, in_queue=18967248, util=100.00%, aggrios=19503443/9385, aggrmerge=0/0, aggrticks=6300254/611164, aggrin_queue=6914877, aggrutil=100.00% dm-1: ios=1/917, merge=0/0, ticks=0/316, in_queue=316, util=0.01%, aggrios=58032708/925, aggrmerge=600/80, aggrticks=7695844/316, aggrin_queue=7697340, aggrutil=99.47% xvdc: ios=58032708/925, merge=600/80, ticks=7695844/316, in_queue=7697340, util=99.47% dm-0: ios=58033307/88, merge=0/0, ticks=7713444/8, in_queue=7724700, util=99.46% dm-3: ios=477022/27152, merge=0/0, ticks=11187320/1833168, in_queue=13019616, util=100.00%, aggrios=477022/26569, aggrmerge=0/598, aggrticks=11187220/1810144, aggrin_queue=12996476, aggrutil=100.00% xvdd: ios=477022/26569, merge=0/598, ticks=11187220/1810144, in_queue=12996476, util=100.00% lvmcache (4.9.30-2, full cache device) $ (cd /srv/cache/fio && FIOTEST=lvmcache-full-4.9.30-2 fio ~/partial.fio) partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [5953KB/0KB/0KB /s] [1488/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=9014: Wed Jul 26 16:09:32 2017 read : io=3180.6MB, bw=2713.1KB/s, iops=678, runt=1200039msec slat (usec): min=3, max=172, avg=13.34, stdev= 7.73 clat (usec): min=65, max=1177.6K, avg=11773.54, stdev=32057.40 lat (usec): min=72, max=1177.7K, avg=11787.57, stdev=32057.79 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 113], 10.00th=[ 118], 20.00th=[ 125], | 30.00th=[ 131], 40.00th=[ 133], 50.00th=[ 145], 60.00th=[ 159], | 70.00th=[ 213], 80.00th=[16512], 90.00th=[36096], 95.00th=[63232], | 99.00th=[148480], 99.50th=[197632], 99.90th=[337920], 99.95th=[415744], | 99.99th=[610304] lat (usec) : 100=0.45%, 250=70.90%, 500=0.60%, 750=0.28%, 1000=0.30% lat (msec) : 2=0.67%, 4=0.04%, 10=1.48%, 20=8.16%, 50=10.28% lat (msec) : 100=4.53%, 250=2.07%, 500=0.23%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.34%, sys=1.45%, ctx=806505, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=814209/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=9015: Wed Jul 26 16:09:32 2017 read : io=3083.6MB, bw=2631.2KB/s, iops=657, runt=1200059msec slat (usec): min=3, max=251, avg=13.31, stdev= 7.88 clat (usec): min=65, max=1195.8K, avg=12144.40, stdev=32642.85 lat (usec): min=72, max=1195.8K, avg=12158.39, stdev=32643.28 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 111], 10.00th=[ 115], 20.00th=[ 122], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 143], 60.00th=[ 159], | 70.00th=[ 221], 80.00th=[17024], 90.00th=[37632], 95.00th=[65280], | 99.00th=[152576], 99.50th=[199680], 99.90th=[342016], 99.95th=[419840], | 99.99th=[610304] lat (usec) : 100=0.49%, 250=70.26%, 500=0.58%, 750=0.27%, 1000=0.30% lat (msec) : 2=0.77%, 4=0.04%, 10=1.51%, 20=8.18%, 50=10.43% lat (msec) : 100=4.74%, 250=2.17%, 500=0.24%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.43%, sys=1.30%, ctx=782175, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=789392/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=6264.7MB, aggrb=5345KB/s, minb=2631KB/s, maxb=2713KB/s, mint=1200039msec, maxt=1200059msec Disk stats (read/write): dm-2: ios=1603515/29, merge=0/0, ticks=19178856/4820, in_queue=19184536, util=100.00%, aggrios=535452/12461, aggrmerge=0/0, aggrticks=6232844/4946, aggrin_queue=6237806, aggrutil=100.00% dm-1: ios=59/34655, merge=0/0, ticks=16/7544, in_queue=7560, util=0.29%, aggrios=1124948/36669, aggrmerge=0/683, aggrticks=166868/8120, aggrin_queue=174932, aggrutil=13.61% xvdc: ios=1124948/36669, merge=0/683, ticks=166868/8120, in_queue=174932, util=13.61% dm-0: ios=1124889/2697, merge=0/0, ticks=166996/736, in_queue=167772, util=13.44% dm-3: ios=481409/31, merge=0/0, ticks=18531520/6560, in_queue=18538088, util=100.00%, aggrios=481409/14, aggrmerge=0/17, aggrticks=18531404/2968, aggrin_queue=18534356, aggrutil=100.00% xvdd: ios=481409/14, merge=0/17, ticks=18531404/2968, in_queue=18534356, util=100.00%
Categories: LUG Community Blogs

Debian Bits: DebConf17 Schedule Published!

Planet HantsLUG - Tue, 25/07/2017 - 00:15

The DebConf17 orga team is proud to announce that over 120 activities have been scheduled so far, including 45- and 20-minute talks, team meetings, and workshops, as well as a variety of other events.

Most of the talks and BoFs will be streamed and recorded, thanks to our amazing video team!

We'd like to remind you that Saturday August 5th is also our much anticipated Open Day! This means a program for a wider audience, including special activities for newcomers, such as an AMA session about Debian, a beginners workshop on packaging, a thoughtful talk about freedom with regard to today's popular gadgets and more.

In addition to the published schedule, we'll provide rooms for ad-hoc sessions where attendees will be able to schedule activities at any time during the whole conference.

The current schedule is available at https://debconf17.debconf.org/schedule/

This is also available through an XML feed. You can use ConfClerk in Debian to consume this, or Giggity on Android devices: https://debconf17.debconf.org/schedule/mobile/

We look forward to seeing you in Montreal!

Categories: LUG Community Blogs

Jonathan McDowell: Learning to love Ansible

Planet ALUG - Mon, 24/07/2017 - 18:41

This post attempts to chart my journey towards getting usefully started with Ansible to manage my system configurations. It’s a high level discussion of how I went about doing so and what I got out of it, rather than including any actual config snippets - there are plenty of great resources out there that handle the actual practicalities of getting started much better than I could.

I’ve been convinced about the merits of configuration management for machines for a while now; I remember conversations about producing an appropriate set of recipes to reproduce our haphazard development environment reliably over 4 years ago. That never really got dealt with before I left, and as managing systems hasn’t been part of my day job since then I never got around to doing more than working my way through the Puppet Learning VM. I do, however, continue to run a number of different Linux machines - a few VMs, a hosted dedicated server and a few physical machines at home and my parents’. In particular I have a VM which handles my parents’ email, and I thought that was a good candidate for trying to properly manage. It’s backed up, but it would be nice to be able to redeploy that setup easily if I wanted to move provider, or do hosting for other domains in their own VMs.

I picked Ansible, largely because I wanted something lightweight and the agentless design appealed to me. All I really need to do is ensure Python is on the host I want to manage and everything else I can bootstrap using Ansible itself. Plus it meant I could use the version from Debian testing on my laptop and not require backports on the stable machines I wanted to manage.

My first attempt was to write a single Ansible YAML file which did all the appropriate things for the email VM; installed Exim/Apache/Roundcube, created users, made sure the appropriate SSH keys were in place, installed configuration files, etc, etc. This did the job, but I found myself thinking it was no better than writing a shell script to do the same things.

Things got a lot better when instead of concentrating on a single host I looked at what commonality was shared between hosts. I started with simple things; Debian is my default distro so I created an Ansible role debian-system which configured up APT and ensured package updates were installed. Then I added a task to setup my own account and install my SSH keys. I was then able to deploy those 2 basic steps across a dozen different machine instances. At one point I got an ARM64 VM from Scaleway to play with, and it was great to be able to just add it to my Ansible hosts file and run the playbook against it to get my basic system setup.

Adding email configuration got trickier. In addition to my parents’ email VM I have my own email hosted elsewhere (along with a whole bunch of other users) and the needs of both systems are different. Sitting down and trying to manage both configurations sensibly forced me to do some rationalisation of the systems, pulling out the commonality and then templating the differences. Additionally I ended up using the lineinfile module to edit the Debian supplied configurations, rather than rolling out my own config files. This helped ensure more common components between systems. There were also a bunch of differences that had grown out of the fact each system was maintained by hand - I had about 4 copies of each Let’s Encrypt certificate rather than just putting one copy in /etc/ssl and pointing everything at that. They weren’t even in the same places on different systems. I unified these sorts of things as I came across them.

Throughout the process of this rationalisation I was able to easily test using containers. I wrote an Ansible role to create systemd-nspawn based containers, doing all of the LVM + debootstrap work required to produce a system which could then be managed by Ansible. I then pointed the same configuration as I was using for the email VM at this container, and could verify at each step along the way that the results were what I expected. It was still a little nerve-racking when I switched over the live email config to be managed by Ansible, but it went without a hitch as hoped.

I still have a lot more configuration to switch to being managed by Ansible, especially on the machines which handle a greater number of services, but it’s already proved extremely useful. To prepare for a jessie to stretch upgrade I fired up a stretch container and pointed the Ansible config at it. Most things just worked and the minor issues I was able to fix up in that instance leaving me confident that the live system could be upgraded smoothly. Or when I want to roll out a new SSH key I can just add it to the Ansible setup, and then kick off an update. No need to worry about whether I’ve updated it everywhere, or correctly removed the old one.

So I’m a convert; things were a bit more difficult by starting with existing machines that I didn’t want too much disruption on, but going forward I’ll be using Ansible to roll out any new machines or services I need, and expect that I’ll find that new deployment to be much easier now I have a firm grasp on the tools available.

Categories: LUG Community Blogs

Andy Smith: 12 hours of lvmcache

Planet HantsLUG - Fri, 21/07/2017 - 20:59

In the previous post I noted that the performance of lvmcache was still increasing and it might be worth testing it for longer than 3 hours.

Here’a a 12 hour test $ cd /srv/cache/fio && FIOTEST=lvmcache-12h fio ~/lvmcache-12h.fio partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [6272KB/0KB/0KB /s] [1568/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=11130: Fri Jul 21 16:37:30 2017 read : io=136145MB, bw=3227.2KB/s, iops=806, runt=43200062msec slat (usec): min=3, max=586402, avg=14.27, stdev=619.54 clat (usec): min=2, max=1517.9K, avg=9897.80, stdev=29334.14 lat (usec): min=71, max=1517.9K, avg=9912.72, stdev=29344.74 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 110], 10.00th=[ 113], 20.00th=[ 119], | 30.00th=[ 124], 40.00th=[ 129], 50.00th=[ 133], 60.00th=[ 143], | 70.00th=[ 157], 80.00th=[11840], 90.00th=[30848], 95.00th=[56576], | 99.00th=[136192], 99.50th=[179200], 99.90th=[309248], 99.95th=[382976], | 99.99th=[577536] lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03% lat (usec) : 250=76.84%, 500=0.26%, 750=0.13%, 1000=0.13% lat (msec) : 2=0.25%, 4=0.02%, 10=1.19%, 20=6.67%, 50=8.59% lat (msec) : 100=3.93%, 250=1.77%, 500=0.18%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.44%, sys=1.63%, ctx=34524570, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=34853153/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=11131: Fri Jul 21 16:37:30 2017 read : io=134521MB, bw=3188.7KB/s, iops=797, runt=43200050msec slat (usec): min=3, max=588479, avg=14.35, stdev=613.38 clat (usec): min=2, max=1530.3K, avg=10017.42, stdev=29196.28 lat (usec): min=70, max=1530.3K, avg=10032.43, stdev=29207.06 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 109], 10.00th=[ 112], 20.00th=[ 118], | 30.00th=[ 124], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 143], | 70.00th=[ 157], 80.00th=[12352], 90.00th=[31360], 95.00th=[57600], | 99.00th=[138240], 99.50th=[179200], 99.90th=[301056], 99.95th=[370688], | 99.99th=[561152] lat (usec) : 4=0.01%, 20=0.01%, 50=0.01%, 100=0.04%, 250=76.56% lat (usec) : 500=0.26%, 750=0.12%, 1000=0.13% lat (msec) : 2=0.26%, 4=0.02%, 10=1.20%, 20=6.75%, 50=8.65% lat (msec) : 100=4.01%, 250=1.82%, 500=0.17%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.45%, sys=1.60%, ctx=34118324, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=34437257/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=270666MB, aggrb=6415KB/s, minb=3188KB/s, maxb=3227KB/s, mint=43200050msec, maxt=43200062msec Disk stats (read/write): dm-2: ios=69290239/1883, merge=0/0, ticks=690078104/246240, in_queue=690329868, util=100.00%, aggrios=23098543/27863, aggrmerge=0/0, aggrticks=229728200/637782, aggrin_queue=230366965, aggrutil=100.00% dm-1: ios=247/64985, merge=0/0, ticks=36/15464, in_queue=15504, util=0.02%, aggrios=53025553/63449, aggrmerge=0/7939, aggrticks=7413340/14760, aggrin_queue=7427028, aggrutil=16.42% xvdc: ios=53025553/63449, merge=0/7939, ticks=7413340/14760, in_queue=7427028, util=16.42% dm-0: ios=53025306/6403, merge=0/0, ticks=7417028/1852, in_queue=7419784, util=16.42% dm-3: ios=16270078/12201, merge=0/0, ticks=681767536/1896032, in_queue=683665608, util=100.00%, aggrios=16270077/12200, aggrmerge=1/1, aggrticks=681637744/1813744, aggrin_queue=683453224, aggrutil=100.00% xvdd: ios=16270077/12200, merge=1/1, ticks=681637744/1813744, in_queue=683453224, util=100.00%

It’s still going up, slowly. The cache hit rate was 76.53%. In the 30 minute test the hit rate was 73.64%.

Over 30 minutes the average IOPS was 1,484.

Over 12 hours the average IOPS was 1,603.

I was kind of hoping to reach the point where the hit rate is so high that it just takes off like bcache does and we start to see tens of thousands of IOPS, but it wasn’t to be.

24 hours of lvmcache

…so I went ahead and ran the same thing for 24 hours.

I’ve skipped the first 2 hours of results since we know what they look like. It appears to still be going up, although the results past 20 hours leave some doubt there.

Here’s the full fio output.

$ cd /srv/cache/fio && FIOTEST=lvmcache-24h fio ~/lvmcache-24h.fio partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [7152KB/0KB/0KB /s] [1788/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=14676: Sat Jul 22 21:34:12 2017 read : io=278655MB, bw=3302.6KB/s, iops=825, runt=86400091msec slat (usec): min=3, max=326, avg=12.43, stdev= 6.97 clat (usec): min=1, max=1524.1K, avg=9673.02, stdev=28748.45 lat (usec): min=71, max=1525.7K, avg=9686.11, stdev=28748.87 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 106], 10.00th=[ 111], 20.00th=[ 116], | 30.00th=[ 119], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 139], | 70.00th=[ 155], 80.00th=[11456], 90.00th=[30336], 95.00th=[55552], | 99.00th=[134144], 99.50th=[177152], 99.90th=[305152], 99.95th=[374784], | 99.99th=[569344] lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.03%, 250=77.16%, 500=0.22%, 750=0.12%, 1000=0.12% lat (msec) : 2=0.23%, 4=0.02%, 10=1.18%, 20=6.65%, 50=8.54% lat (msec) : 100=3.84%, 250=1.70%, 500=0.17%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.47%, sys=1.64%, ctx=70653446, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=71335660/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=14677: Sat Jul 22 21:34:12 2017 read : io=283280MB, bw=3357.4KB/s, iops=839, runt=86400074msec slat (usec): min=3, max=330, avg=12.44, stdev= 6.98 clat (usec): min=2, max=1515.9K, avg=9514.83, stdev=28128.86 lat (usec): min=71, max=1515.2K, avg=9527.92, stdev=28129.29 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 109], 10.00th=[ 112], 20.00th=[ 118], | 30.00th=[ 123], 40.00th=[ 126], 50.00th=[ 133], 60.00th=[ 141], | 70.00th=[ 157], 80.00th=[11328], 90.00th=[29824], 95.00th=[55040], | 99.00th=[132096], 99.50th=[173056], 99.90th=[292864], 99.95th=[362496], | 99.99th=[544768] lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03% lat (usec) : 250=77.29%, 500=0.23%, 750=0.11%, 1000=0.12% lat (msec) : 2=0.23%, 4=0.02%, 10=1.18%, 20=6.65%, 50=8.49% lat (msec) : 100=3.81%, 250=1.66%, 500=0.15%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.47%, sys=1.67%, ctx=71794214, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=72519640/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=561935MB, aggrb=6659KB/s, minb=3302KB/s, maxb=3357KB/s, mint=86400074msec, maxt=86400091msec Disk stats (read/write): dm-2: ios=143855123/29, merge=0/0, ticks=1380761492/1976, in_queue=1380772508, util=100.00%, aggrios=47953627/25157, aggrmerge=0/0, aggrticks=459927326/7329, aggrin_queue=459937080, aggrutil=100.00% dm-1: ios=314/70172, merge=0/0, ticks=40/15968, in_queue=16008, util=0.01%, aggrios=110839338/72760, aggrmerge=0/2691, aggrticks=15300392/17100, aggrin_queue=15315432, aggrutil=16.92% xvdc: ios=110839338/72760, merge=0/2691, ticks=15300392/17100, in_queue=15315432, util=16.92% dm-0: ios=110839024/5279, merge=0/0, ticks=15308540/1768, in_queue=15312588, util=16.93% dm-3: ios=33021544/20, merge=0/0, ticks=1364473400/4252, in_queue=1364482644, util=100.00%, aggrios=33021544/19, aggrmerge=0/1, aggrticks=1364468920/4076, aggrin_queue=1364476064, aggrutil=100.00% xvdd: ios=33021544/19, merge=0/1, ticks=1364468920/4076, in_queue=1364476064, util=100.00%

So, 1,664 average IOPS (825 + 839), 77.05% (110,839,338 / (71,335,660 + 72,519,640)*100) cache hit rate.

Not sure I can be bothered to run a multi-day test on this now!

Categories: LUG Community Blogs

Andy Smith: bcache and lvmcache

Planet HantsLUG - Wed, 19/07/2017 - 07:06
Background

Over at BitFolk we offer both SSD-backed storage and HDD-backed archive storage. The SSDs we use are really nice and basically have removed all IO performance problems we have ever encountered in the past. I can’t begin to describe how pleasant it is to just never have to think about that whole class of problems.

The main downside of course is that SSD capacity is still really expensive. That’s why we introduced the HDD-backed archive storage: for bulk storage of things that didn’t need to have high performance.

You’d really think though that by now there would be some kind of commodity tiered storage that would allow a relatively small amount of fast SSD storage to accelerate a much larger pool of HDDs. Of course there are various enterprise solutions, and there is also ZFS where SSDs could be used for the ZIL and L2ARC while HDDs are used for the pool.

ZFS is quite appealing but I’m not ready to switch everything to that yet, and I’m certainly not going to buy some enterprise storage solution. I also don’t necessarily want to commit to putting all storage behind such a system.

I decided to explore Linux’s own block device caching solutions.

Scope

I’ve restricted the scope to the two solutions which are part of the mainline Linux kernel as of July 2017, these being bcache and lvmcache.

lvmcache is based upon dm-cache which has been included with the mainline kernel since April 2013. It’s quite conservative, and having been around for quite a while is considered stable. It has the advantage that it can work with any LVM logical volume no matter what the contents. That brings the disadvantage that you do need to run LVM.

bcache has been around for a little longer but is a much more ambitious project. Being completely dedicated to accelerating slow block devices with fast ones it is claimed to be able to achieve higher performance than other caching solutions, but as it’s much more complicated than dm-cache there are still bugs being found. Also it requires you format your block devices as bcache before you use them for anything.

Test environment

I’m testing this on a Debian testing (buster) Xen virtual machine with a 20GiB xvda virtual disk containing the main operating system. That disk is backed by a software (md) RAID-10 composed of two Samsung sm863 SSDs. It was also used for testing the baseline SSD performance from the directory /srv/ssd.

The virtual machine had 1GiB of memory but the pagecache was cleared between each test run in an attempt to prevent anything being cached in memory.

A 5GiB xvdc virtual disk was provided, backed again on the SSD RAID. This was used for the cache role both in bcache and lvmcache.

A 50GiB xvdd virtual disk was provided, backed by a pair of Seagate ST4000LM016-1N2170 HDDs in software RAID-1. This was used for the HDD backing store in each of the caching implementations. The resulting cache device was mounted at /srv/cache.

Finally a 50GiB xvde virtual disk also backed on HDD was used to test baseline HDD performance, mounted at /srv/slow.

The filesystem in use in all cases was ext4 with default options. In dom0, deadline scheduler was used in all cases.

TL;DR, I just want graphs

In case you can’t be bothered to read the rest of this article, here’s just the graphs with some attempt at interpreting them. Down at the tests section you’ll find details of the actual testing process and more commentary on why certain graphs were produced.

git test graphs

Times to git clone and git grep.

fio IOPS graphs

These are graphs of IOPS across the 30 minutes of testing. There’s two important things to note about these graphs:

  1. They’re a Bezier curve fitted to the data points which are one per second. The actual data points are all over the place, because achieved IOPS depends on how many cache hits/misses there were, which is statistical.
  2. Only the IOPS for the first job is graphed. Even when using the per_job_logs=0 setting my copy of fio writes a set of results for each job. I couldn’t work out how to easily combine these so I’ve shown only the results for the first job.

    For all tests except bcache (sequential_cutoff=0) you just have to bear in mind that there is a second job working in parallel doing pretty much the same amount of IOPS. Strangely for that second bcache test the second job only managed a fraction of the IOPS (though still more than 10k IOPS) and I don’t know why.

IOPS over time for all tests

Well, those results are so extreme that it kind of makes it hard to distinguish between the low-end results.

A couple of observations:

  • SSD is incredibly and consistently fast.
  • For everything else there is a short steep section at the beginning which is likely to be the effect of HDD drive cache.
  • With sequential_cutoff set to 0, bcache very quickly reaches near-SSD performance for this workload (4k reads, 90% hitting 10% of data that fits entirely in the bcache). This is probably because the initial write put data straight into cache as it’s set to writeback.
  • When starting with a completely empty cache, lvmcache is no slouch either. It’s not quite as performant as bcache but that is still up near the 48k IOPS per process region, and very predictable.
  • When sequential_cutoff is left at its default of 4M, bcache performs much worse though still blazing compared to an HDD on its own. At the end of this 30 minute test performance was still increasing so it might be worth performing a longer test
  • The performance of lvmcache when starting with a cache already full of junk data seems to be not that much better than HDD baseline.
IOPS over time for low-end results

Leaving the high-performers out to see if there is anything interesting going on near the bottom of the previous graph.

Apart from the initial spike, HDD results are flat as expected.

Although the lvmcache (full cache) results in the previous graph seemed flat too, looking closer we can see that performance is still increasing, just very slowly. It may be interesting to test for longer to see if performance does continue to increase.

Both HDD and lvmcache have a very similar spike at the start of the test so let’s look closer at that.

IOPS for first 30 seconds

For all the lower-end performers the first 19 seconds are steeper and I can only think this is the effect of HDD drive cache. Once that is filled, HDD remains basically flat, lvmcache (full cache) increases performance more slowly and bcache with the default sequential_cutoff starts to take off.

SSDs don’t have the same sort of cache and bcache with no sequential_cutoff spikes up too quickly to really be noticeable at this scale.

3-hour lvmcache test

Since it seemed like lvmcache with a full cache device was still slowly increasing in performance I did a 3-hour testing on that one.

Skipping the first 20 minutes which show stronger growth, even after 3 hours there is still some performance increase happening. It seems like even a full cache would eventually promote read hot spots, but it could take a very very long time.

The tests

This being caching, we would only really expect it to be effective for reads of already-known data. Sequential reads on an HDD aren’t too bad but what really makes things feel slow are random reads all over the place: the time taken to seek to a new location on an HDD is really a killer for performance. So, we’d like a test that involves multiple reads of the same data but non-sequential.

git tests

I decided to do a git clone of the Linux kernel and then do a git grep of this, which would be guaranteed to be random access. The git grep step was done repeatedly to see if we could spot the cache being warmed up, also with the kernel’s pagecache being cleared after each step to avoid a situation where the entirety of the kernel source ended up in RAM. The clone process was also timed just to see if there would be any effect on write time.

fio tests

Despite git being a real application, the workload itself (searching for a string over and over) seemed rather too synthetic and simplistic. I decided to also try using the excellent fio to gather some detailed statistics. This is also synthetic of course, relating to no real workload, but it’s quite sophisticated and hopefully of interest.

The fio job spec used in all cases was as follows:

[partial] ioengine=libaio direct=1 iodepth=8 numjobs=2 readwrite=randread random_distribution=zipf:1.2 bs=4k size=2g unlink=1 runtime=1800 time_based=1 per_job_logs=1 log_avg_msec=1000 write_iops_log=/var/tmp/fio-${FIOTEST}

This causes two 2GiB files of random data to be written out and then two processes read from them for 30 minutes with up to 8 IOs in flight at once for each process. Between the use of direct=1 and the fact that there is 4GiB of data and only 1GiB of RAM, that should be enough to avoid testing any memory cache.

log_avg_msec=1000 is specified in order to restrict the IOPS log to one entry per second, otherwise you get one entry per IO. As that is kept in memory until fio completes, unless the number of entries is reduced then due to the sheer number of IOs the SSDs can service the fio process will cause the VM to run out of memory. This directive causes just one averaged entry per 1000ms, which is adequate for graphing.

unlink is used to make sure that data files are deleted after each test run. Leaving the files in place would cause additional promotions into cache, which would favour tests that are run multiple times.

By default when told to do a random read, fio would ensure it reads the entire file once only (before starting again, if the test is time-based). That would not be any use for testing cache warming as there would only ever be one read from each part of the file. To avoid this, a Zipf distribution is used so that read locations are chosen according to a somewhat repetitive distribution, simulating hotspots. I have no real idea about this part so if anyone has any input on that I’d be interested to hear, but my understanding is that a Zipf distribution with exponent 1.2 should cause about 91% of the reads to be from about 10% of the blocks.

In fact I would be very interested to hear of other fio job configurations that people think would be useful here.

If you’re unfamiliar with the tool, fio output can appear quite daunting. There’s a wealth of information in there though, so it’s worth looking into if you ever need to test storage performance. Too many times people just reach for dd or bonnie as performance testing tools but apart from the most trivial of tests they are just too simple. I found this article by Albert Tobey useful in understanding fio‘s output.

Each test run was performed against:

  1. A purely SSD-backed filesystem.
  2. A purely HDD-backed filesystem.
  3. A filesystem on top of a bcache device, sequential_cutoff=4M (default).
  4. A filesystem on top of a bcache device, sequential_cutoff=0.
  5. A filesystem on top of an lvmcache device, empty cache.
  6. A filesystem on top of an lvmcache device, cache full of junk.
Baseline SSD git grep test $ cd /srv/ssd $ /usr/bin/time -f '%U user %S system %E elapsed %P CPU' \ > git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Cloning into 'linux-stable'... remote: Counting objects: 5948690, done. remote: Compressing objects: 100% (124799/124799), done. remote: Total 5948690 (delta 427756), reused 549517 (delta 425675) Receiving objects: 100% (5948690/5948690), 1.21 GiB | 39.61 MiB/s, done. Resolving deltas: 100% (4961965/4961965), done. Checking out files: 100% (59844/59844), done. 195.45 user 22.10 system 2:37.89 elapsed 137% CPU $ cd /srv/ssd/linux-stable $ for i in {1..20}; do > echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null ; \ > /usr/bin/time -f '%U user %S system %E elapsed %P CPU' \ > git --no-pager grep -q blah ; \ > sleep 1; \ > done 0.98 user 1.28 system 0:02.28 elapsed 99% CPU 0.95 user 1.15 system 0:02.18 elapsed 96% CPU 0.89 user 1.30 system 0:02.23 elapsed 98% CPU 1.04 user 1.13 system 0:02.25 elapsed 96% CPU 0.95 user 1.22 system 0:02.24 elapsed 96% CPU 0.95 user 1.23 system 0:02.23 elapsed 97% CPU 1.00 user 1.16 system 0:02.24 elapsed 96% CPU 1.00 user 1.16 system 0:02.23 elapsed 96% CPU 0.89 user 1.28 system 0:02.22 elapsed 97% CPU 0.97 user 1.20 system 0:02.24 elapsed 97% CPU 1.00 user 1.18 system 0:02.23 elapsed 97% CPU 0.96 user 1.21 system 0:02.23 elapsed 97% CPU 0.96 user 1.22 system 0:02.24 elapsed 97% CPU 1.01 user 1.17 system 0:02.24 elapsed 97% CPU 0.94 user 1.22 system 0:02.24 elapsed 96% CPU 1.00 user 1.16 system 0:02.24 elapsed 96% CPU 0.98 user 1.18 system 0:02.24 elapsed 96% CPU 0.92 user 1.24 system 0:02.24 elapsed 97% CPU 0.92 user 1.28 system 0:02.25 elapsed 98% CPU 1.09 user 1.07 system 0:02.23 elapsed 97% CPU

The initial git clone took 157.89 seconds and each git grep reliably took about 2.24 seconds.

fio test $ cd /srv/ssd/fio && FIOTEST=ssd fio ~/partial.fio partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [409.8MB/0KB/0KB /s] [105K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=29306: Wed Jul 19 05:17:39 2017 read : io=379033MB, bw=215628KB/s, iops=53906, runt=1800001msec slat (usec): min=2, max=1547, avg= 4.38, stdev= 1.68 clat (usec): min=26, max=4350, avg=141.54, stdev=29.67 lat (usec): min=94, max=4353, avg=146.39, stdev=29.65 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 108], 10.00th=[ 112], 20.00th=[ 119], | 30.00th=[ 124], 40.00th=[ 129], 50.00th=[ 135], 60.00th=[ 141], | 70.00th=[ 149], 80.00th=[ 161], 90.00th=[ 179], 95.00th=[ 197], | 99.00th=[ 243], 99.50th=[ 262], 99.90th=[ 314], 99.95th=[ 338], | 99.99th=[ 406] lat (usec) : 50=0.01%, 100=0.14%, 250=99.08%, 500=0.77%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=14.30%, sys=39.70%, ctx=50601996, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=97032448/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=29307: Wed Jul 19 05:17:39 2017 read : io=358188MB, bw=203769KB/s, iops=50942, runt=1800001msec slat (usec): min=2, max=225, avg= 4.46, stdev= 1.71 clat (usec): min=21, max=4385, avg=150.08, stdev=43.00 lat (usec): min=69, max=4391, avg=155.01, stdev=42.98 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 110], 10.00th=[ 115], 20.00th=[ 121], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 137], 60.00th=[ 145], | 70.00th=[ 155], 80.00th=[ 171], 90.00th=[ 199], 95.00th=[ 237], | 99.00th=[ 322], 99.50th=[ 350], 99.90th=[ 414], 99.95th=[ 438], | 99.99th=[ 498] lat (usec) : 50=0.01%, 100=0.10%, 250=95.82%, 500=4.06%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=13.80%, sys=37.97%, ctx=48695297, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=91696255/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=737221MB, aggrb=419396KB/s, minb=203769KB/s, maxb=215627KB/s, mint=1800001msec, maxt=1800001msec Disk stats (read/write): xvda: ios=188724848/22, merge=0/28, ticks=27221784/0, in_queue=27267336, util=100.00%

This provides a nice illustration of just how fast the SSDs are: each process managed a very consistent 52 thousand IOPS and between the two of them they managed to read ~720GiB of data in 30 minutes.

Baseline HDD git grep test $ cd /srv/slow $ /usr/bin/time -f '%U user %S system %E elapsed %P CPU' \ > git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Cloning into 'linux-stable'... remote: Counting objects: 5948690, done. remote: Compressing objects: 100% (124799/124799), done. remote: Total 5948690 (delta 427756), reused 549517 (delta 425675) Receiving objects: 100% (5948690/5948690), 1.21 GiB | 38.36 MiB/s, done. Resolving deltas: 100% (4961965/4961965), done. Checking out files: 100% (59844/59844), done. 194.61 user 21.48 system 3:42.23 elapsed 97% CPU $ cd /srv/slow/linux-stable $ for i in {1..20}; do > echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null ; \ > /usr/bin/time -f '%U user %S system %E elapsed %P CPU' \ > git --no-pager grep -q blah ; \ > sleep 1; \ > done 1.22 user 1.60 system 0:23.60 elapsed 11% CPU 1.35 user 1.32 system 0:23.08 elapsed 11% CPU 1.24 user 1.43 system 0:22.49 elapsed 11% CPU 1.15 user 1.57 system 0:22.37 elapsed 12% CPU 1.25 user 1.44 system 0:22.09 elapsed 12% CPU 1.26 user 1.48 system 0:22.60 elapsed 12% CPU 1.20 user 1.52 system 0:22.47 elapsed 12% CPU 1.21 user 1.48 system 0:22.82 elapsed 11% CPU 1.14 user 1.55 system 0:22.45 elapsed 12% CPU 1.28 user 1.49 system 0:22.32 elapsed 12% CPU 1.21 user 1.48 system 0:22.43 elapsed 12% CPU 1.26 user 1.51 system 0:22.95 elapsed 12% CPU 1.21 user 1.49 system 0:22.33 elapsed 12% CPU 1.23 user 1.45 system 0:22.54 elapsed 11% CPU 1.19 user 1.51 system 0:22.08 elapsed 12% CPU 1.26 user 1.42 system 0:22.42 elapsed 12% CPU 1.20 user 1.51 system 0:22.47 elapsed 12% CPU 1.19 user 1.53 system 0:22.42 elapsed 12% CPU 1.15 user 1.56 system 0:22.48 elapsed 12% CPU 1.24 user 1.47 system 0:22.10 elapsed 12% CPU

The initial git clone took 222.23 seconds and each git grep reliably took about 22.53 seconds.

Of course, a significant part of the clone time was spent downloading the data.

fio test $ cd /srv/slow/fio && FIOTEST=hdd fio ~/partial.fio partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [3795KB/0KB/0KB /s] [948/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=29158: Wed Jul 19 04:33:19 2017 read : io=3276.3MB, bw=1863.8KB/s, iops=465, runt=1800078msec slat (usec): min=2, max=132, avg=13.00, stdev= 8.10 clat (usec): min=59, max=830656, avg=17151.07, stdev=36630.33 lat (usec): min=64, max=830668, avg=17164.94, stdev=36630.39 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 78], 10.00th=[ 88], 20.00th=[ 111], | 30.00th=[ 143], 40.00th=[ 205], 50.00th=[ 988], 60.00th=[ 1496], | 70.00th=[15424], 80.00th=[26240], 90.00th=[50432], 95.00th=[82432], | 99.00th=[177152], 99.50th=[224256], 99.90th=[346112], 99.95th=[403456], | 99.99th=[544768] lat (usec) : 100=15.10%, 250=26.82%, 500=3.79%, 750=1.99%, 1000=2.47% lat (msec) : 2=10.84%, 4=0.39%, 10=2.91%, 20=10.92%, 50=14.68% lat (msec) : 100=6.53%, 250=3.21%, 500=0.34%, 750=0.02%, 1000=0.01% cpu : usr=0.39%, sys=1.17%, ctx=822685, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=838728/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=29159: Wed Jul 19 04:33:19 2017 read : io=3328.4MB, bw=1893.4KB/s, iops=473, runt=1800076msec slat (usec): min=2, max=157, avg=13.20, stdev= 8.22 clat (usec): min=4, max=1046.5K, avg=16882.45, stdev=35737.35 lat (usec): min=64, max=1046.6K, avg=16896.52, stdev=35737.42 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 78], 10.00th=[ 88], 20.00th=[ 111], | 30.00th=[ 143], 40.00th=[ 211], 50.00th=[ 996], 60.00th=[ 1528], | 70.00th=[15296], 80.00th=[25728], 90.00th=[49920], 95.00th=[81408], | 99.00th=[173056], 99.50th=[218112], 99.90th=[329728], 99.95th=[391168], | 99.99th=[522240] lat (usec) : 10=0.01%, 50=0.01%, 100=15.03%, 250=26.59%, 500=3.86% lat (usec) : 750=2.07%, 1000=2.49% lat (msec) : 2=10.70%, 4=0.36%, 10=3.03%, 20=11.45%, 50=14.46% lat (msec) : 100=6.49%, 250=3.17%, 500=0.30%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.47%, sys=1.15%, ctx=834258, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=852044/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=6604.6MB, aggrb=3757KB/s, minb=1863KB/s, maxb=1893KB/s, mint=1800076msec, maxt=1800078msec Disk stats (read/write): xvde: ios=1690665/12, merge=0/15, ticks=28764332/2848, in_queue=28767992, util=100.00%

Both jobs together managed an average of 938 IOPS here, which is pretty weak but still considerably more than a pair of HDDs should be able to do (100–150 IOPS each, so 300 if you were amazingly lucky). This indicates that the drive cache must have been involved, and indeed the IOPS rate does very rapidly increase within the first few seconds of the test run but reaches a stable maximum within 20 seconds. That marks the time when the disk cache is full and cache hit rate can’t be maintained.

A total of 6,604.6MiB was read during the 30 minute test.

bcache Setup

By default bcache uses writethrough cache mode, where writes go to both the backing store and the cache and are only acknowledged once they’ve completed at the backing store. In this mode bcache will never accelerate writes, but it is safe to lose the cache device at any time.

I felt comfortable switching the bcache device to writeback mode as the underlying storage used for the cache is already reliable. In writeback mode the writes are acknowledged as soon as they are completed on the cache device and are later sequentially written to backing device.

This should result in higher write performance as well as the higher read performance we’d expect from using a cache.

$ sudo make-bcache -B /dev/xvdd UUID: cf0353eb-ac08-45bd-a3d5-f8b92fbc0fb8 Set UUID: 8c06d317-e735-41f5-8dec-52da26a37883 version: 1 block_size: 1 data_offset: 16 $ sudo mkfs.ext4 -L cached /dev/bcache0 mke2fs 1.43.4 (31-Jan-2017) Discarding device blocks: done Creating filesystem with 13107198 4k blocks and 3276800 inodes Filesystem UUID: 7db8568e-282b-4fa2-bf19-a42143708a0d Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (65536 blocks): done Writing superblocks and filesystem accounting information: done $ sudo make-bcache -C /dev/xvdc UUID: ae581052-4449-460d-8cf5-b11d6467403d Set UUID: 2fcc7937-9f5f-4089-9400-552675929ea5 version: 0 nbuckets: 10240 block_size: 1 bucket_size: 1024 nr_in_set: 1 nr_this_dev: 0 first_bucket: 1 $ sudo bcache-super-show /dev/xvdc sb.magic ok sb.first_sector 8 [match] sb.csum 206ADF979437131E [match] sb.version 3 [cache device] dev.label (empty) dev.uuid ae581052-4449-460d-8cf5-b11d6467403d dev.sectors_per_block 1 dev.sectors_per_bucket 1024 dev.cache.first_sector 1024 dev.cache.cache_sectors 10484736 dev.cache.total_sectors 10485760 dev.cache.ordered yes dev.cache.discard no dev.cache.pos 0 dev.cache.replacement 0 [lru] cset.uuid 2fcc7937-9f5f-4089-9400-552675929ea5 $ echo 2fcc7937-9f5f-4089-9400-552675929ea5 | sudo tee /sys/block/bcache0/bcache/attach >/dev/null $ echo writeback | sudo tee /sys/block/bcache0/bcache/cache_mode >/dev/null $ sudo mount /dev/bcache0 /srv/cache $ sudo chown andy: /srv/cache git grep test $ cd /srv/cache $ /usr/bin/time -f '%U user %S system %E elapsed %P CPU' \ > git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Cloning into 'linux-stable'... remote: Counting objects: 5948690, done. remote: Compressing objects: 100% (124799/124799), done. remote: Total 5948690 (delta 427755), reused 549518 (delta 425675) Receiving objects: 100% (5948690/5948690), 1.21 GiB | 40.30 MiB/s, done. Resolving deltas: 100% (4961964/4961964), done. Checking out files: 100% (59844/59844), done. 194.88 user 21.30 system 3:34.10 elapsed 100% CPU $ cd /srv/cache/linux-stable $ for i in {1..20}; do > echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null ; \ > /usr/bin/time -f '%U user %S system %E elapsed %P CPU' \ > git --no-pager grep -q blah ; \ > sleep 1; \ > done 1.03 user 1.40 system 0:03.02 elapsed 80% CPU 1.01 user 1.25 system 0:02.55 elapsed 88% CPU 1.05 user 1.21 system 0:02.63 elapsed 85% CPU 0.90 user 1.34 system 0:02.56 elapsed 87% CPU 0.96 user 1.28 system 0:02.59 elapsed 86% CPU 0.90 user 1.35 system 0:02.54 elapsed 88% CPU 1.04 user 1.22 system 0:02.51 elapsed 90% CPU 1.00 user 1.25 system 0:02.50 elapsed 89% CPU 0.93 user 1.34 system 0:02.54 elapsed 89% CPU 1.02 user 1.26 system 0:02.53 elapsed 90% CPU 0.91 user 1.43 system 0:02.67 elapsed 88% CPU 0.94 user 1.39 system 0:02.57 elapsed 90% CPU 1.04 user 1.28 system 0:02.57 elapsed 90% CPU 1.00 user 1.37 system 0:02.58 elapsed 91% CPU 1.02 user 1.33 system 0:02.84 elapsed 82% CPU 1.11 user 1.21 system 0:02.58 elapsed 90% CPU 1.03 user 1.30 system 0:02.57 elapsed 90% CPU 0.96 user 1.37 system 0:02.58 elapsed 90% CPU 1.02 user 1.36 system 0:02.62 elapsed 90% CPU 0.92 user 1.41 system 0:02.62 elapsed 89% CPU

The initial git clone took 214.10 seconds and each git grep on average took about 2.61 seconds. It can be seen that the first run is slightly slower but very quickly things settle down.

The git grep results are very encouraging being only 11.6% of the time taken for the baseline HDD tests, or almost 9x faster. Each run was only 0.37 seconds slower than when run directly against SSD.

The initial clone results are disappointing though, being only 8.13 seconds faster than when run against baseline HDD. It looks like most data must have bypassed the cache and waited for HDD. bcache does bypass cache for sequential writes which is what the git clone would have done, and we can prove this:

$ cat /sys/block/bcache0/bcache/stats_total/bypassed 270M

So yes, there we go, all 270MiB bypassed the cache and waited on HDD completion.

We can give bcache hints about when to consider writes sequential, and at the extreme end tell it to never consider writes sequential, so let’s try that.

$ echo 0 | sudo tee /sys/block/bcache0/bcache/sequential_cutoff >/dev/null $ cd /srv/cache $ rm -r linux-stable $ /usr/bin/time -f '%U user %S system %E elapsed %P CPU' \ > git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Cloning into 'linux-stable'... remote: Counting objects: 5948690, done. remote: Compressing objects: 100% (124799/124799), done. remote: Total 5948690 (delta 427756), reused 549517 (delta 425675) Receiving objects: 100% (5948690/5948690), 1.21 GiB | 39.51 MiB/s, done. Resolving deltas: 100% (4961965/4961965), done. Checking out files: 100% (59844/59844), done. 195.10 user 23.53 system 2:39.87 elapsed 136% CPU $ cat /sys/block/bcache0/bcache/stats_total/bypassed 299M

This time only 29MiB of data bypassed cache, and the time for clone was 159.87 seconds. That’s 62.36 seconds faster than against HDD, and only 1.98 seconds slower than against SSD!

fio test – sequential_cutoff=4M $ cd /srv/cache/fio && FIOTEST=bcache-sc4m fio ~/partial.fio partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [119.9MB/0KB/0KB /s] [30.7K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=8640: Thu Jul 20 18:13:08 2017 read : io=45237MB, bw=25734KB/s, iops=6433, runt=1800041msec slat (usec): min=3, max=5707, avg= 6.14, stdev=10.97 clat (usec): min=2, max=1238.4K, avg=1235.14, stdev=10675.04 lat (usec): min=70, max=1238.5K, avg=1241.70, stdev=10675.36 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108], | 30.00th=[ 111], 40.00th=[ 117], 50.00th=[ 119], 60.00th=[ 121], | 70.00th=[ 122], 80.00th=[ 124], 90.00th=[ 133], 95.00th=[ 151], | 99.00th=[35584], 99.50th=[67072], 99.90th=[148480], 99.95th=[185344], | 99.99th=[305152] lat (usec) : 4=0.01%, 50=0.01%, 100=0.37%, 250=97.11%, 500=0.08% lat (usec) : 750=0.02%, 1000=0.02% lat (msec) : 2=0.03%, 4=0.01%, 10=0.13%, 20=0.65%, 50=0.87% lat (msec) : 100=0.46%, 250=0.24%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=1.81%, sys=6.79%, ctx=10722687, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=11580733/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=8641: Thu Jul 20 18:13:08 2017 read : io=43743MB, bw=24885KB/s, iops=6221, runt=1800037msec slat (usec): min=3, max=5703, avg= 6.27, stdev=10.53 clat (usec): min=2, max=1081.7K, avg=1277.44, stdev=10884.69 lat (usec): min=70, max=1081.8K, avg=1284.13, stdev=10884.99 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 103], 10.00th=[ 104], 20.00th=[ 106], | 30.00th=[ 107], 40.00th=[ 109], 50.00th=[ 111], 60.00th=[ 117], | 70.00th=[ 120], 80.00th=[ 122], 90.00th=[ 129], 95.00th=[ 145], | 99.00th=[37632], 99.50th=[69120], 99.90th=[150528], 99.95th=[189440], | 99.99th=[317440] lat (usec) : 4=0.01%, 20=0.01%, 50=0.01%, 100=0.68%, 250=96.73% lat (usec) : 500=0.07%, 750=0.02%, 1000=0.02% lat (msec) : 2=0.03%, 4=0.01%, 10=0.13%, 20=0.68%, 50=0.90% lat (msec) : 100=0.48%, 250=0.25%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=1.74%, sys=6.57%, ctx=10425775, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=11198290/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=88981MB, aggrb=50618KB/s, minb=24884KB/s, maxb=25734KB/s, mint=1800037msec, maxt=1800041msec Disk stats (read/write): bcache0: ios=22775854/880, merge=0/0, ticks=28668080/36768, in_queue=197531580, util=100.00%, aggrios=11393419/289009, aggrmerge=0/182, aggrticks=14329508/435284, aggrin_queue=14764868, aggrutil=100.00% xvdc: ios=22230796/570128, merge=0/348, ticks=2656456/37312, in_queue=2693812, util=67.39% xvdd: ios=556042/7890, merge=0/17, ticks=26002560/833256, in_queue=26835924, util=100.00%

The ~15k IOPS per process achieved towards the end of this test are quite impressive and suggest that there is more scope for performance increase in a longer test. The combined average IOPS across 30 minutes were “only” 12.7k though. Vastly superior to a HDD and we would be glad to take it.

22,779,023 IOs were issued (11,580,733+11,198,290) of which 22,230,796 (97.6%) hit the cache. That’s a really great result, but we learned earlier that setting sequential_cutoff to 0 allows us to get data we write straight into the cache, so let’s see how that changes things.

fio test – sequential_cutoff=0 $ cd /srv/cache/fio && FIOTEST=bcache-sc0 fio ~/partial.fio partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [276.1MB/0KB/0KB /s] [70.1K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=9190: Thu Jul 20 20:34:49 2017 read : io=367029MB, bw=208799KB/s, iops=52199, runt=1800001msec slat (usec): min=3, max=350673, avg= 5.49, stdev=54.90 clat (usec): min=22, max=558335, avg=145.49, stdev=593.01 lat (usec): min=67, max=558342, avg=151.41, stdev=597.19 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 106], 10.00th=[ 109], 20.00th=[ 116], | 30.00th=[ 120], 40.00th=[ 123], 50.00th=[ 127], 60.00th=[ 133], | 70.00th=[ 139], 80.00th=[ 153], 90.00th=[ 185], 95.00th=[ 223], | 99.00th=[ 310], 99.50th=[ 338], 99.90th=[ 442], 99.95th=[ 868], | 99.99th=[20352] lat (usec) : 50=0.01%, 100=0.27%, 250=96.55%, 500=3.11%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=13.23%, sys=41.79%, ctx=46793234, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=93959531/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=9191: Thu Jul 20 20:34:49 2017 read : io=116645MB, bw=66357KB/s, iops=16589, runt=1800020msec slat (usec): min=3, max=86150, avg= 6.31, stdev=22.07 clat (usec): min=16, max=581877, avg=473.49, stdev=3466.14 lat (usec): min=67, max=581884, avg=480.26, stdev=3466.62 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 107], 10.00th=[ 110], 20.00th=[ 116], | 30.00th=[ 120], 40.00th=[ 123], 50.00th=[ 126], 60.00th=[ 131], | 70.00th=[ 135], 80.00th=[ 143], 90.00th=[ 161], 95.00th=[ 183], | 99.00th=[14656], 99.50th=[23680], 99.90th=[47360], 99.95th=[59648], | 99.99th=[94720] lat (usec) : 20=0.01%, 50=0.01%, 100=0.23%, 250=97.81%, 500=0.37% lat (usec) : 750=0.03%, 1000=0.02% lat (msec) : 2=0.02%, 4=0.02%, 10=0.18%, 20=0.66%, 50=0.58% lat (msec) : 100=0.08%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=4.86%, sys=16.81%, ctx=24164179, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=29861123/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=483674MB, aggrb=275153KB/s, minb=66357KB/s, maxb=208798KB/s, mint=1800001msec, maxt=1800020msec Disk stats (read/write): bcache0: ios=123818417/28, merge=0/0, ticks=27945560/5768, in_queue=429639892, util=100.00%, aggrios=61919282/259095, aggrmerge=0/25, aggrticks=13927904/125270, aggrin_queue=14059068, aggrutil=100.00% xvdc: ios=123345393/500268, merge=0/35, ticks=17064408/30904, in_queue=17107112, util=99.99% xvdd: ios=493172/17923, merge=0/15, ticks=10791400/219636, in_queue=11011024, util=100.00%

123,820,654 (93,959,531+29,861,123) IOs issued, 123,345,393 (99.62%) hit cache. It’s just ~2% higher than the previous test, but what a difference it makes. The IOPS log indicates that performance very quickly increases to near-SSD levels, averaging 68.8k IOPS across 30 minutes.

Interestingly unlike all other tests (even unlike bcache with the default sequential_cutoff), the second job doesn’t manage anywhere near the same level of performance as the first job. It’s 52.2k IOPS vs. 16.6k IOPS. Perhaps at this high level of cache utilisation there is some sort of locking going on. Might be worth following up with upstream.

lvmcache Setup $ sudo pvcreate /dev/xvdc Physical volume "/dev/xvdc" successfully created. $ sudo pvcreate /dev/xvdd Physical volume "/dev/xvdd" successfully created. $ sudo vgcreate cachedvg /dev/xvd{d,c} Volume group "cachedvg" successfully created $ sudo lvcreate -L50m -n lv_cache_meta cachedvg /dev/xvdc Rounding up size to full physical extent 52.00 MiB Logical volume "lv_cache_meta" created. $ sudo lvcreate -L4g -n lv_cache cachedvg /dev/xvdc Logical volume "lv_cache" created. $ sudo lvcreate -l100%PVS -n lv cachedvg /dev/xvdd Logical volume "lv" created. $ sudo lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert lv cachedvg -wi-a----- 50.00g lv_cache cachedvg -wi-a----- 4.00g lv_cache_meta cachedvg -wi-a----- 52.00m $ sudo lvconvert --type cache-pool --cachemode writeback --poolmetadata cachedvg/lv_cache_meta cachedvg/lv_cache Converted cachedvg/lv_cache to cache pool. $ sudo lvconvert --type cache --cachepool cachedvg/lv_cache cachedvg/lv Logical volume cachedvg/lv is now cached. $ sudo lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices lv cachedvg Cwi-a-C--- 50.00g [lv_cache] [lv_corig] 0.00 1.01 0.00 lv_corig(0) [lv_cache] cachedvg Cwi---C--- 4.00g 0.00 1.01 0.00 lv_cache_cdata(0) [lv_cache_cdata] cachedvg Cwi-ao---- 4.00g /dev/xvdc(13) [lv_cache_cmeta] cachedvg ewi-ao---- 52.00m /dev/xvdc(0) [lv_corig] cachedvg owi-aoC--- 50.00g /dev/xvdd(0) [lvol0_pmspare] cachedvg ewi------- 52.00m /dev/xvdc(1037) $ sudo mkfs.ext4 -L cached /dev/cachedvg/lv mke2fs 1.43.4 (31-Jan-2017) Creating filesystem with 13106176 4k blocks and 3276800 inodes Filesystem UUID: bff28530-339a-403d-889d-dd42f4465ab5 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (65536 blocks): done Writing superblocks and filesystem accounting information: done $ sudo mount /dev/cachedvg/lv /srv/cache $ sudo chown andy: /srv/cache git grep test $ cd /srv/cache $ /usr/bin/time -f '%U user %S system %E elapsed %P CPU' \ > git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Cloning into 'linux-stable'... remote: Counting objects: 5948690, done. remote: Compressing objects: 100% (124799/124799), done. remote: Total 5948690 (delta 427755), reused 549518 (delta 425675) Receiving objects: 100% (5948690/5948690), 1.21 GiB | 35.05 MiB/s, done. Resolving deltas: 100% (4961964/4961964), done. Checking out files: 100% (59844/59844), done. 196.00 user 23.40 system 2:43.13 elapsed 134% CPU $ cd /srv/cache/linux-stable $ for i in {1..20}; do > echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null ; \ > /usr/bin/time -f '%U user %S system %E elapsed %P CPU' \ > git --no-pager grep -q blah ; \ > sleep 1; \ > done 1.00 user 1.58 system 0:06.41 elapsed 40% CPU 0.93 user 1.46 system 0:04.22 elapsed 56% CPU 1.09 user 1.30 system 0:04.09 elapsed 58% CPU 1.02 user 1.39 system 0:03.89 elapsed 62% CPU 0.97 user 1.41 system 0:03.89 elapsed 61% CPU 1.03 user 1.34 system 0:03.52 elapsed 67% CPU 1.06 user 1.30 system 0:03.60 elapsed 65% CPU 0.94 user 1.46 system 0:03.79 elapsed 63% CPU 1.05 user 1.33 system 0:03.28 elapsed 72% CPU 1.00 user 1.39 system 0:03.81 elapsed 62% CPU 1.03 user 1.37 system 0:03.25 elapsed 74% CPU 1.05 user 1.37 system 0:03.30 elapsed 73% CPU 1.07 user 1.30 system 0:03.18 elapsed 74% CPU 1.07 user 1.32 system 0:03.28 elapsed 72% CPU 0.96 user 1.46 system 0:02.95 elapsed 82% CPU 1.07 user 1.31 system 0:03.14 elapsed 76% CPU 0.98 user 1.40 system 0:03.04 elapsed 78% CPU 0.96 user 1.42 system 0:03.19 elapsed 74% CPU 1.00 user 1.40 system 0:03.12 elapsed 77% CPU 0.89 user 1.48 system 0:02.89 elapsed 82% CPU

The initial git clone took 163.13 seconds, not far off native SSD timings (157.68 seconds). Each git grep on average took about 3.35 seconds. These started off slower (6.41 seconds) but settled down with the last 15 runs averaging at 3.29 seconds.

fio test – empty cache $ cd /srv/cache/fio && FIOTEST=lvmcache-empty fio ~/partial.fio partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [373.5MB/0KB/0KB /s] [95.7K/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=6652: Thu Jul 20 12:14:06 2017 read : io=333831MB, bw=189912KB/s, iops=47478, runt=1800001msec slat (usec): min=2, max=245, avg= 6.41, stdev= 2.17 clat (usec): min=29, max=234093, avg=159.69, stdev=401.74 lat (usec): min=69, max=234098, avg=166.57, stdev=401.75 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 108], 10.00th=[ 113], 20.00th=[ 119], | 30.00th=[ 124], 40.00th=[ 129], 50.00th=[ 135], 60.00th=[ 143], | 70.00th=[ 153], 80.00th=[ 175], 90.00th=[ 221], 95.00th=[ 274], | 99.00th=[ 370], 99.50th=[ 410], 99.90th=[ 502], 99.95th=[ 996], | 99.99th=[16768] lat (usec) : 50=0.01%, 100=0.16%, 250=92.76%, 500=6.97%, 750=0.05% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.02%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=12.04%, sys=44.39%, ctx=42073879, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=85460611/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=6653: Thu Jul 20 12:14:06 2017 read : io=313896MB, bw=178572KB/s, iops=44642, runt=1800001msec slat (usec): min=2, max=335, avg= 6.55, stdev= 2.25 clat (usec): min=22, max=212646, avg=170.22, stdev=721.29 lat (usec): min=63, max=212656, avg=177.25, stdev=721.32 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 108], 10.00th=[ 112], 20.00th=[ 118], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 131], 60.00th=[ 137], | 70.00th=[ 145], 80.00th=[ 157], 90.00th=[ 177], 95.00th=[ 197], | 99.00th=[ 258], 99.50th=[ 314], 99.90th=[12096], 99.95th=[15424], | 99.99th=[28032] lat (usec) : 50=0.01%, 100=0.22%, 250=98.57%, 500=0.91%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.02%, 10=0.11%, 20=0.11%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=11.36%, sys=42.77%, ctx=41146564, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=80357365/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=647726MB, aggrb=368484KB/s, minb=178571KB/s, maxb=189912KB/s, mint=1800001msec, maxt=1800001msec Disk stats (read/write): dm-2: ios=165815070/5, merge=0/0, ticks=27172896/564, in_queue=27297688, util=100.00%, aggrios=55273566/7259, aggrmerge=0/0, aggrticks=8986252/1498, aggrin_queue=9007888, aggrutil=100.00% dm-1: ios=2/19047, merge=0/0, ticks=0/3072, in_queue=3072, util=0.14%, aggrios=165379217/17690, aggrmerge=0/4083, aggrticks=23890596/3616, aggrin_queue=23926592, aggrutil=100.00% xvdc: ios=165379217/17690, merge=0/4083, ticks=23890596/3616, in_queue=23926592, util=100.00% dm-0: ios=165379215/2726, merge=0/0, ticks=23917816/860, in_queue=23979056, util=100.00% dm-3: ios=441483/6, merge=0/0, ticks=3040940/564, in_queue=3041536, util=75.31%, aggrios=441483/5, aggrmerge=0/1, aggrticks=3040896/464, aggrin_queue=3041312, aggrutil=75.30% xvdd: ios=441483/5, merge=0/1, ticks=3040896/464, in_queue=3041312, util=75.30%

We can see that the first job issued 85,460,611 IOs and the second job issued 80,357,365 IOs, for a total of 165817976 IOs. Of these, 165,379,217 (99.74%) came from xvdc, the cache (SSD) device. Only 441,483 (0.27%) came from xvdd, the backing (HDD) device. Again this is very aggressive cache population, with a combined average of 92k IOPS being achieved over 30 minutes.

80k IOPS are achieved within 20 seconds of the start of testing, which is probably helped by the disk cache, but the steady maximum of ~90k IOPS is still achieved by 40 seconds in. Average IOPS ended up at 88% that of the SSD.

This is incredible performance, but I could only get it to happen the first time, against an empty cache device. Since the next test run will unlink the previous run’s files, the cache device will at that point be full of data that is of no use as it’s been deleted. The next test will write out 4GiB of new, uncached data.

fio test – full cache $ cd /srv/cache/fio && FIOTEST=lvmcache-full fio ~/partial.fio partial: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... fio-2.16 Starting 2 processes partial: Laying out IO file(s) (1 file(s) / 2048MB) partial: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 2 (f=2): [r(2)] [100.0% done] [6072KB/0KB/0KB /s] [1518/0/0 iops] [eta 00m:00s] partial: (groupid=0, jobs=1): err= 0: pid=6857: Thu Jul 20 12:58:00 2017 read : io=5266.1MB, bw=2996.2KB/s, iops=749, runt=1800111msec slat (usec): min=3, max=117, avg=12.93, stdev= 7.50 clat (usec): min=58, max=1060.8K, avg=10663.38, stdev=31025.49 lat (usec): min=73, max=1060.8K, avg=10676.99, stdev=31025.92 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 112], 10.00th=[ 117], 20.00th=[ 125], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 137], 60.00th=[ 153], | 70.00th=[ 179], 80.00th=[14144], 90.00th=[33024], 95.00th=[59136], | 99.00th=[144384], 99.50th=[191488], 99.90th=[337920], 99.95th=[411648], | 99.99th=[618496] lat (usec) : 100=0.29%, 250=74.34%, 500=0.46%, 750=0.20%, 1000=0.21% lat (msec) : 2=0.47%, 4=0.03%, 10=1.30%, 20=7.26%, 50=9.23% lat (msec) : 100=4.09%, 250=1.88%, 500=0.22%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.48%, sys=1.52%, ctx=1335232, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1348336/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 partial: (groupid=0, jobs=1): err= 0: pid=6858: Thu Jul 20 12:58:00 2017 read : io=5171.2MB, bw=2941.7KB/s, iops=735, runt=1800133msec slat (usec): min=3, max=320, avg=13.05, stdev= 7.56 clat (usec): min=63, max=1145.3K, avg=10861.20, stdev=30827.62 lat (usec): min=72, max=1145.3K, avg=10874.93, stdev=30828.07 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 116], 10.00th=[ 120], 20.00th=[ 126], | 30.00th=[ 131], 40.00th=[ 133], 50.00th=[ 141], 60.00th=[ 155], | 70.00th=[ 185], 80.00th=[14912], 90.00th=[33536], 95.00th=[59136], | 99.00th=[142336], 99.50th=[189440], 99.90th=[333824], 99.95th=[411648], | 99.99th=[602112] lat (usec) : 100=0.33%, 250=73.29%, 500=0.50%, 750=0.22%, 1000=0.22% lat (msec) : 2=0.52%, 4=0.03%, 10=1.38%, 20=7.58%, 50=9.63% lat (msec) : 100=4.18%, 250=1.88%, 500=0.21%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.43%, sys=1.55%, ctx=1310390, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=1323812/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: io=10438MB, aggrb=5937KB/s, minb=2941KB/s, maxb=2996KB/s, mint=1800111msec, maxt=1800133msec Disk stats (read/write): dm-2: ios=2671993/27, merge=0/0, ticks=28764200/8380, in_queue=28773988, util=100.00%, aggrios=891886/15335, aggrmerge=0/0, aggrticks=9387100/6312, aggrin_queue=9393456, aggrutil=100.00% dm-1: ios=173/42639, merge=0/0, ticks=16/9316, in_queue=9336, util=0.24%, aggrios=1967684/42191, aggrmerge=0/3788, aggrticks=291076/9440, aggrin_queue=300476, aggrutil=15.68% xvdc: ios=1967684/42191, merge=0/3788, ticks=291076/9440, in_queue=300476, util=15.68% dm-0: ios=1967511/3340, merge=0/0, ticks=291184/1240, in_queue=292496, util=15.55% dm-3: ios=707976/28, merge=0/0, ticks=27870100/8380, in_queue=27878536, util=100.00%, aggrios=707976/12, aggrmerge=0/16, aggrticks=27870028/2908, aggrin_queue=27872968, aggrutil=100.00% xvdd: ios=707976/12, merge=0/16, ticks=27870028/2908, in_queue=27872968, util=100.00%

Testing with an already-full cache produces dramatically different results. We can see that a total of 2,672,148 (1,348,336+1,323,812) IOs were issued, with 1,967,684 (73.64%) coming from the cache. 707,976 (26.49%) came from the backing device. Although on the face of it that sounds like a good hit rate, it was poor enough to severely limit the average IOPS to 1484 across both jobs. We saw how much the difference was between 99.6% and 97.6% cache hit rate, and now we see what the difference is between ~99% and ~73%.

1,484 IOPS represents a 158% increase on the baseline HDD IOPS of 938 so it’s nothing to be sniffed at, and as I say it may be the case that it only gets better over much longer time span.

Even after this 30 minute test the performance of the lvmcache was still very slowly going up, so I decided to run another test across 3 hours instead of 30 seconds. The performance did continue to increase very very slowly peaking at about 1,640 aggregate IOPS after 3 hours. It seems like it is quite difficult to evict data from the cache in favour of new data that is being read, whereas when the cache was empty it promoted everything that was read immediately. Possibly if this was left for long enough there would eventually be enough cache hits to achieve similar performance to the first run.

bcache did not suffer from this problem: every test run against bcache was the same, except for the effect of preloading the cache device by messing with sequential_cutoff. I suspect that this is because bcache is a write cache; when set to writethrough or writeback mode, writes are going through the cache and immediately being cached. lvmcache is not a write cache; it will only promote things when they are read.

It seems like lvmcache needs a very large number of reads before it will evict data from the cache and bring new data in.

Further tests?

If you have any further tests you’d like to suggest please leave a comment or a tweet. I think it would be good to come up with more fio tests for this, but bear in mind that it really needs to test random read IO with a repetitive element so that a cache can actually be effective. If you read every part of 100GiB equally as often then there will be no hot spots and a comparatively small cache will not be effective, no matter what technology is used.

Also real-world benchmarks would be good, so perhaps a really complex SQL SELECT on a data set that won’t fit in (1GiB) memory but will fit in (5GiB) SSD cache.

It might be interesting to use a cache device much smaller than the working set. These tests laid out two 2GiB files then used a Zipf distribution to randomly read 10% of each (so two sets of ~205MiB data) while the cache devices were 4 or 5GiB in size. It is difficult to guess what the hot spots would be on a production file system, but a 90/10 or 80/20 distribution does seem reasonable.

What may be less reasonable is starting off with such a small data set. What would it be like to try to cache 1TiB with 10GiB of SSD? I’d rather not write out 1TiB of data to check, but I could simulate by writing out 51GiB (5%) and having a cache device that is only 512MiB. On a 90/10 distribution it would still want to cache ~5.2GiB and only have 512MiB to put it in.

Conclusions

SSDs are lovely but they’re still really expensive. Although they are fun for turning out impressive benchmarks, in the real world you only need things to be fast enough. If you want consistently great performance and you have the money, buy some decent SSDs. If you don’t have the money then there is a whole world of trade-offs.

bcache from what I can measure provides really impressive performance but I am concerned about relying upon it. The existing implementation is no longer the focus of the main developer’s attention as the developer has now moved on to bcachefs, a complete filesystem in its own right that builds upon the bcache concept.

I’m really conservative when it comes to filesystems and I know it can take years, even decades, to get to feature parity with all the nasty bugs hammered out. I also worry about continued maintenance of a separate filesystem.

lvmcache on the other hand is providing only a very modest performance boost. In many cases it won’t be worth it. I do like its simplicity though and I’m already very familiar with LVM.

The way that any logical volume can have a cache attached without needing to start over with the filesystem makes lvmcache very flexible; it also means that some volumes can be cached and others not. Furthermore with a slow PV and a fast PV both in the volume group, if necessary an individual logical volume can be allocated entirely from the fast PV.

I may experiment with bcache some more in situations where I can quickly restore the data in event of calamity, or fairly easily move to a different solution if bcache becomes abandonware. I may experiment with lvmcache in places where a modest speedup would be welcome.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Yay, finished my degree at last

Planet ALUG - Tue, 18/07/2017 - 22:56

A little while back, in June, I sat my last exam for what I hoped would be the last module in my degree. For seven years, I've been working on a degree with the Open University and have been taking advantage of the opportunity to have a somewhat self-directed course load by taking the 'Open' degree track. When asked why I bothered to do this, I guess my answer has been a little varied. In principle it's because I felt like I'd already done a year's worth of degree and didn't want it wasted, but it's also because I have been, in the dim and distant past, overlooked for jobs simply because I had no degree and thus was an easy "bin the CV".

Fed up with this, I decided to commit to the Open University and thus began my journey toward 'qualification' in 2010. I started by transferring the level 1 credits from my stint at UCL back in 1998/1999 which were in a combination of basic programming in Java, some mathematics including things like RSA, and some psychology and AI courses which at the time were aiming at a degree called 'Computer Science with Cognitive Sciences'.

Then I took level 2 courses, M263 (Building blocks of software), TA212 (The technology of music) and MS221 (Exploring mathematics). I really enjoyed the mathematics course and so...

At level 3 I took MT365 (Graphs, networks and design), M362 (Developing concurrent distributed systems), TM351 (Data management and analysis - which I ended up hating), and finally finishing this June with TM355 (Communications technology).

I received an email this evening telling me the module result for TM355 had been posted, and I logged in to find I had done well enough to be offered my degree. I could have claimed my degree 18+ months ago, but I persevered through another two courses in order to qualify for an honours degree which I have now been awarded. Since I don't particularly fancy any ceremonial awarding, I just went through the clicky clicky and accepted my qualification of 'Batchelor of Science (Honours) Open, Upper Second-class Honours (2.1)' which grants me the letters 'BSc (Hons) Open (Open)' which, knowing me, will likely never even make it onto my CV because I'm too lazy.

It has been a significant effort, over the course of the past few years, to complete a degree without giving up too much of my personal commitments. In addition to earning the degree, I have worked, for six of the seven years it has taken, for Codethink doing interesting work in and around Linux systems and Trustable software. I have designed and built Git server software which is in use in some universities, and many companies, along with a good few of my F/LOSS colleagues. And I've still managed to find time to attend plays, watch films, read an average of 2 novel-length stories a week (some of which were even real books), and be a member of the Manchester Hackspace.

Right now, I'm looking forward to a stress free couple of weeks, followed by an immense amount of fun at Debconf17 in Montréal!

Categories: LUG Community Blogs
Syndicate content