Perpetual IT

Max Kellermann, a coder and security researcher for German content management software creators CM4all, has just published a fascinating report about a Linux kernel bug that was patched recently.

He called the vulnerability Dirty Pipe, because it involves insecure interaction between a true Linux file (one that’s saved permanently on disk) and a Linux pipe, which is a memory-only data buffer that can be used like a file.

Very greatly simplified, if you have a pipe that you are allowed to write to and a file that you aren’t…

…then, sometimes, writing into the pipe’s memory buffer may inadvertently also modify the kernel’s temporary in-memory copies – the so-called cache pages – of various parts of the disk file.

Annoyingly, even if the file is flagged as “read only” by the operating system itself, modifying its underlying kernel cache is treated as a “write”.

As a result, the modified cache buffer is flushed back to disk by the kernel, permanently updating the contents of the stored file, despite any operating system permissions applied to it.

Even a physically unwritable file, such as one on a CD-ROM or an SD card with the write-enable switch turned off, will appear to have been modified for as long as the corrupted cache buffers are kept in memory by the kernel.

Which versions are affected?

For those running Linux who want to cut to the chase and check if they’re patched, Kellermann reports that this bug was introduced (at least in its current, easily exploitable form) in kernel 5.8.

That means three officially supported kernel flavours are definitely at risk: 5.10, 5.15 and 5.16.

The bug was patched in 5.10.102, 5.15.25 and 5.16.11, so if you have a version that is at or above one of those, you’re OK.

Apparently, Android is affected too, and although a fix for the vulnerability was incorporated into the kernel source code on 2022-02-24, neither Google’s Android Security Bulletin for March 2022, nor the company’s Pixel-specific notes, mention this bug, now dubbed CVE-2022-0847.

Of all the many officially supported Android handsets we’ve surveyed so far, however, only the Pixel 6 series seems to use kernel 5.10, with most devices back on one of the older-but-apparently-not-vulnerable Linux 5.4 or 4.x versions.

User-friendly log files

Intriguingly, Kellermann discovered the vulnerablity due to intermittent corruption of HTTP log files on his company’s network.

He had a server process that would regularly take daily logfiles, compressed using the Unix-friendly gzip utility, and convert them into monthly logfiles in the Windows-friendly ZIP format for customers to download.

ZIP files support, and typically use, gzip compression internally, so that raw gzip files can actually be used as the individual components inside a ZIP archive, as long as ZIP-style control data is added at the start and end of the file, and in between each internal gzipped chunk.

So, to save both time and CPU power, Kellermann was able to avoid temporarily decompressing each day’s logfile for each customer, only to recompress it immediately into the all-inclusive ZIP file.

He ended up creating a writable Linux pipe to which he could export the all-in-one ZIP archive, and then he’d read from each gzip file in turn, sending them one-by-one into the output pipe, with the needed headers and trailers inserted at the right points.

For extra efficiency, he used the special Linux function splice(), which tells the kernel to read data from a file and write it into a pipe directly from kernel memory, which avoids the overhead of a traditional read()-and-then-write() loop.

Reading-and-writing using the traditional read() and write() functions means copying data from the kernel into a memory buffer assigned by the user, and then copying that buffer straight back into the kernel, so the data is copied around in memory at least twice, even though it’s not modified in the process. To avoid this overhead, splice() and its companion function sendfile() were introduced to Linux for use when programmers wanted to move data unaltered between two file system objects. For large files on a busy server, this is faster and reduces load.

Corrupted bytes, 8-at-a-time

Occasionally, however, Kellermann would find that the last 8 bytes of one of the original gzip files would get corrupted, even though he was only ever reading from these files.

All his output was going into the writable “output pipe” used to create the combined ZIP file.

There was nothing in his code that even tried to write to any of the input files, which were opened “read only” and should therefore have been protected by the operating system anyway.

One telltale he spotted was that the corrupted 8 bytes almost always showed up in the last gzip file of any month, and were always 50 4B 01 02 1E 03 14 00 in hexadecimal.

Threat researchers will recognise 50 4B 01 02 right away, because 50 4B comes out as PK in ASCII characters, short for Phil Katz, the creator of the ZIP file format.

Also commonly seen in malware analysis involving ZIP files are those bytes 01 02 immediately after the PK – that’s a special marker that denotes “what follows is a block of data in the end-of-archive ZIP trailer”.

The bytes 1E 03, in case you’re wondering, denote that the file was created on a Unix-like system (0x03), following zipfile specification 3 (0x1E is 30 in decimal, interpreted as version 3.). The 14 00 after that denotes that PKZIP 2.0 or later is needed to uncompresss (0x14 is 20 in decimal, interpreted as 2.0).

In other words, Kellerman was able to infer that the data bleeding into the very end of occasional “read only” gzip files was always the start of the additional data that he was adding at the end of his writable, all-in-one ZIP file.

No matter how carefully he perused at his own code, however, he couldn’t see how he could perpetrate this corruption with a bug of his own, even if he wanted to.

After all, the side-effect of the bug was that his software ended up corrupting 8 bytes
at the end of a file that the kernel itself was supposed to stop him writing to anyway.

As he writes of his feelings at the time:

Blaming the Linux kernel (i.e. somebody else’s code) for data corruption must be the last resort. That is unlikely. The kernel is an extremely complex project developed by thousands of individuals with methods that may seem chaotic; despite of this, it is extremely stable and reliable. But this time, I was convinced that it must be a kernel bug.

But with perseverance, he was able to create two minimalist programs, with just three and five lines of code respectively, that reproduced the misbehaviour in a way that could only be blamed on the kernel.

Following that, he was able to construct a proof-of-concept attack that allows an unprivileged user to modify even a well-locked-down file such as your list of trusted SSH keys, or the list of “known good” digital signatures you’re willing to connect to for updates.

Even worse, it seems that this bug, given its low-level nature, can be used inside a virtualised container (where any running program is not supposed to have write access to any objects outside its “sandbox” or “jail”) to modify files that would usually be off limits.

The good news, of course, is that Kellermans’s careful digging led not only to uncovering the bug and understanding its cause, but also to helping the community devise a patch to close the hole.

What to do?

• If you’re a Linux 5.x user. Check your kernel version. You want 5.10.102, 5.15.25 or 5.16.11 (or above). If your distro uses older kernel versions with its own security patches “backported”, check with your distro maker for details. Otherwise, simply run the command uname -r to print your kernel release.
• If you’re an Android user. We don’t quite know what to tell you at the time of writing [2022-03-08T17:00Z], given the variety of kernel versions in use by various products and vendors. So far, the only mainstream device we’re aware of that has kernel 5.10 is the Google Pixel 6 series, apparently at 5.10.43, and with no mention of a fix for this bug in the latest Pixel Update Bulletin. A few other devices (e.g. Google Pixel 5) are on kernel 5.4, with the majority of the rest from Google and other vendors alike still on a 4.x version. You can view your kernel version at Settings > About phone > Android version > Kernel version – if you are on 5.10, we suggest you keep your eye on the Android Security Bulletins overview page.
• If you’re a programmer hunting bugs. When tracking down otherwise inexplicable behaviour, develop the art of creating the smallest “repro” you can – that’s the jargon term for code that reliably reproduces the incorrect behaviour so that others can investigate more easily. Eliminate as many possible variables and uncertainties from the bug-hunting equation as you can before handing it over to someone else. They will thank you for it.

Popular open-source computer hardware company Adafruit Industries accidentally exposed customer data…

…via the GitHub account of a former employee.

As you’ve probably figured out already, Adafruit is named after after Ada Lovelace, a nineteenth-century British intellectual who was a computer programmer long before any programmable computers existed.

As mysterious as that might sound, the story is both uplifting and disappointing in equal measure. In the 1830s, British inventor Charles Babbage designed a general-purpose computer that he dubbed the Difference Engine. While he was busy trying to construct the device, Ada started wrestling with how it might be used. She outlined numerous programming principles that today we take for granted, such as loops and subroutines for commonly repeated computations, essentially coding various algorithms that would run on it. She even began pondering the issues of artificial intelligence and whether computing machines might ever truly be considered capable of independent thought and creativity. (Her considered conclusion, dubbed Lady Lovelace’s Objection by twentieth-century computer scientist Alan Turing, was: “No.”)
Unfortunately, Babbage’s computer – which was, of nineteenth century necessity, entirely mechnical – turned out to be unbuildable: the lathes and milling machines of the day just weren’t up to the precision required to allow its many cogs and levers to operate reliably in unison. The cumulative effects of backlash in the mechanism meant that it never worked, so the Victorian age never acquired giant steampunk computers, and Ada’s code was never executed on an actual device.

The company sells a wide range of miniature open-source hardware boards and kits for hobbyists and professionals alike. (Think Raspberry Pi and Arduino, along with loads more custom hardware that’s even smaller and even funkier.)

What happened?

According to Adafruit’s public report:

The inadvertent disclosure involved an auditing data set used for employee training becoming public, on a GitHub repository associated with an inactive former employee’s account who was learning data analysis. The repository contained some names, email addresses, shipping/billing addresses and/or whether orders were placed successfully via credit card processor and/or PayPal, as well as details for some orders. There were no user passwords or financial information such as credit cards in the data analysis set.

Reading between the lines of the company’s notification, it sounds as though the leaked data had been sitting around in public for at least two years, given that the database entries exposed don’t go past 2019.

(Unfortunately, the report doesn’t say who reported the leaked data, when it was discovered, how obviously exposed it was, when the ex-employee concerned left the company, when the data was extracted from the company’s live data, or how many customers or records were involved.)

Adafruit claims that it got onto the job of removing the offending information within 15 minutes of hearing about the problem, contacting the ex-employee to get the data deleted, and kicking off an analysis to try to figure out who else might have seen it, and what they might have done with it.

From the report, it sounds as though the results of the forensic analysis were inconclusive – the company wasn’t able to specify with certainty whether the data was accessed or not, but it did comment: “[W]e are unaware of any actual misuse of the information”.

Nevertheless, Adafruit published a reminder that breaches of this sort, once reported, do provide a powerful pretext for cybercriminals.

The company is warning customers to watch out for apparently believable phishing campaigns that “warn” potential “victims” to take corrective action such as resetting their passwords via a handily-supplied but fake website, and for to bogus callers claiming to be offering “official support” and requesting personal information “for confirmation”:

As a reminder, for your security, we will never send you a link to reset your password as part of a security alert, our customer support team will never contact you asking for your password. If you receive an email of this nature, or otherwise suspect that someone is attempting to gain access to your account or solicit your personal information, or have any other questions about this process, please contact us at security@adafruit.com.

If phishing criminals do have access to actual names, addresses and order details from a company database breach, then their fraudulent emails can be made even more believable by including genuine historical data as believable but bogus “proof” that their scam warnings are real.

What to do?

• If you’re a customer and you bought any Adafruit products before 2020. Take Adafruit’s own advice and be aware of possible phishing attempts that try to scare you with “urgent actions” allegedly necessitated by this breach.
• If you’ve had a breach at your own company. By all means, use Adafruit’s official report as a partial example of how to respond, but try to include in your notification some of the content that Adafruit omitted. Firstly, offer a genuine apology – after all, if you aren’t sorry, why should customers think you care more about their security now than you did before? Secondly, if you have conducted or are still in the process of conducting a forensic analysis, as Adafruit claims, be clear what you have already found (including admitting what you don’t know and are unlikely ever to discover), or provide a date by which you expect to follow up with further details.
• If you need test or training data to use outside your live systems. Don’t simply skim off real data and distribute it outside your secure servers for those purposes. Numerous tools exist both for redacting genuine data so that it reflects reality without revealing personal details, and for generating realistic but artificial data that is suitable for training.
• If you’re entrusted with access to genuine company data. Don’t copy it or upload it anywhere other than official company locations. Especially don’t upload it to personal cloud accounts, such as GitHub storage – even if your motivations are honest and your intentions impeccable – where the company can’t fulfil its own data protection obligations, and can’t reliably revoke your access to it if you leave.

Mozilla has published Firefox 97.0.2, an “out-of-band” update that closes two bugs that are officially listed as critical.

Mozilla reports that both of these holes are already actively being exploited, making them so-called zero-day bugs, which means, in simple terms, that the crooks got there first:

We have had reports of attacks in the wild abusing [these] flaw[s].

Access to information about the bugs is still restricted to Mozilla insiders, presumably to make it harder for attackers to get at the technical details of how to exploit these security holes.

Assuming that the existing zero-day exploits are not widely known (these days, true zero-days are often jealously guarded by their discoverers because they’re considered both scarce and valuable), temporarily limiting access to the source code changes does provide some protection against copycat attacks.

As we’ve mentioned many times before on Naked Security, finding and exploiting a zero-day hole when you know where to start looking, and what to start looking for, is very much easier than discovering such a bug from scratch.

The bugs are listed as:

• CVE-2022-26485. Use-after-free in XSLT parameter processing. This bug has apparently already been exploited for remote code exection (RCE), implying that attackers with no existing privileges or accounts on your computer could trick you into running malware code of their choice simply by luring you to an innocent-looking but booby-trapped website.
• CVE-2022-26486, Use-after-free in WebGPU IPC Framework. This bug has apparently already been exploited for what’s known as a sandbox escape. This sort of security hole can typically be abused on its own (for example, to give an attacker access to files that are supposed to be off limits), or in combination with an RCE bug to allow implanted malware to escape from the security confines imposed by your browser, thus making an already bad situation even worse.

Use-after-free bugs occur when one part of a program signals its intention to stop using a chunk of memory that was allocated to it…

…but carries on using it anyway, thus potentially trampling on data that other parts of the program are now relying on.

In the best case, a use-after-free bug typically leads to corrupted data or to a program crash, either of which can be considered a security problems in its own right.

In the worst case, a use-after-free leads to remote code execution, where the data that’s trampled on is wilfully modified by the attackers to trick the program into running untrusted code from outside.

What to do?

Go to the About Firefox dialog to check your current version.

If you are out of date then Firefox will offer to fetch the update and then present a [Restart Firefox] button; click the button, or exit and restart the browser, to deploy the update.

The version numbers you want are: Firefox 97.0.2 (if you are using the regular release), or Firefox 91.6.1 ESR (if you are using the extended support release), or Firefox 97.3.0 for Android.

If you’re on Android, check for updates via the Play Store.

If you’re a Linux user where Firefox is managed by your distro, check your distro creator.

Note that if you are not yet on the latest major version (97.0 for regular Firefox, or 91.6 for the Extended Support Release), you may need to complete the update in multiple stages, so be sure to re-visit the About Firefox dialog after each update has been installed, to make sure you have finished all needed update-and-restart cycles.

Just over a year ago, graphics card behemoth Nvidia announced an unexpected software “feature”: anti-cryptomining code baked into the drivers for its latest graphics processing units (GPUs).

Simply put, if the driver software thinks you’re using the GPU to perform calculations related to Ethereum cryptocurrency calculations, it cuts the execution speed of your code in half.

This restriction isn’t meant to protect you from yourself, for example to limit hardware damage if you try to drive the GPU too hard and cause it to overheat dangerously.

This is all about managing supply and demand.

Unfortunately for keen gamers, who love powerful GPUs because they improve their gaming experience with faster and more realistic graphics, cryptocurrency mining syndicates love good GPUs even more.

That’s because GPUs greatly accelerate the mining of Ethereum-based cryptocurrencies, with calculation speeds (or hashrates, as they are known in the jargon) anywhere from five to ten times higher than a normal CPU from the same amount of electricity.

Even more unfortunately for gamers, who might buy one or two GPUs each at a time, mining syndicates use their purchasing power to buy up GPUs in bulk.

This, in turn, encourages scalpers to buy in bulk too, aiming to sell their “second hand” cards well above new retail prices when official supplies run out.

Nvidia decided to appease its many avid gaming fans – surely the company’s most loyal long-term GPU customers, given that they actually want graphics cards for doing graphics – by splitting its processor card line in two.

Mining XOR Gaming

As Nvidia said last year:

To address the specific needs of Ethereum mining, we’re announcing the NVIDIA CMP [Cryptocurrency Mining Processor] product line for professional mining. CMP products, which don’t do graphics, are [… ]optimized for the best mining performance and efficiency. They don’t meet the specifications required of a GeForce GPU and thus don’t impact the availability of GeForce GPUs to gamers.

The idea is that GeForce GPUs run at full speed if used for graphics, but if used for Ethereum mining are deliberately hobbled by Nvidia’s Lite Hash Rate system, or LHR for short.

Public opinion at the time of the announcement was sharply divided, as a quick look at the many comments on last year’s article will reveal.

Naked Security readers reacted in many ways.

A gamer called Trillian said, “Good on Nvidia!”

Others claimed this LHR behaviour was unfair because they used their GPU cards for a mix of gaming and mining (intermingled, intriguingly, with comments from readers who claimed those claims were made up).

And a commenter called J Riley Castine was even more critical, wanting to know, “How is such a move […] not a violation of anti-trust laws?”

Exit light, enter night

Well, it looks as though this year-old community divide over LHR has spilled over into outright cybercrime.

Popular technology website Tom’s Hardware, amongst numerous other commenters, is reporting that cybercrime gang Lapsus$ claims to have hacked Nvidia and stolen a terabyte’s worth of data…

…only to issue what amounts to an unusual ransomware demand: Remove the Lite Hash Rate limiter, or else!

According to an IM screenshot posted by Tom’s Hardware, the alleged hackers wrote:

Hello,

We decided to help mining and gaming community, we want nvidia to push an update for all 30 series firmware that remove every lhr limitations otherwise we will leak hw folder.

If they remove the lhr we will forget about hw folder (it’s a big folder) We both know lhr impact mining and gaming.

Thanks.

The hw folder (hw is short for “computer hardware”) alluded to above is the claimed 1TB of allegedly stolen data, apparently including card schematics, driver and firmware code, internal documentation, and more.

Ironically, in the same message thread, these hackers also claim to be selling their own “LHR unlocker” for some Nvidia cards, although the underground market for such a cracking tool would clearly evaporate if Nvidia were to remove the LHR restrictions for everyone.

Perhaps the alleged existence of this darkweb LHR unlocker is supposed to make Nvidia feel even more pressurised, on the grounds that an LHR bypass could be made public anyway, so the company might as well go along with the blackmail demand?

What to do?

It’s hard to know what to believe when messages of this sort start circulating.

Did the hackers actually get in to start with? Did they really manage to steal the information they’re claiming? Was this a conventional ransomware attack, aiming at both stealing and scrambling data for extra leverage? If so, and we therefore assume that the data scrambling part was thwarted, why should we believe any of the boasts in the messages? Do the crooks really have an LHR unlocker of their own to add to the drama?

We may never know the answers to these questions, but we can learn from the allegations anyway, which reiterate the importance of defence-in-depth.

Defence-in-depth not only involves multiple layers of proactive protection aimed at early threat detection and prevention, but ideally also needs ongoing threat assessment and response, in order to figure out what really happened if anomalies are detected.

As the self-styled Nvidia hackers say:

We were into nvidia systems for about a week, we fastly escalated to admin of a lot of systems. We grabbed 1TB of data.

Whether that’s is true or not in this case, it does describe the nature of many modern cyberattacks, which aren’t simply automated “smash, gran and run” sallies any more.

Modern cyberintrusions typically involve human-led network exploration, privilege escalation, and data exfiltration, often over an extended period.

Intruders with administrator powers often introduce backdoors along the way, or add extra network accounts for themselves, thus giving themselves a quiet and easy way back in next time…

…if you don’t take the trouble to seek-and-destroy the boobytraps they left behind this time.

Learn more about Sophos Managed Threat Response here:
Sophos MTR – Expert Led Response  ▶
24/7 threat hunting, detection, and response  ▶