xz backdoor

A malicious backdoor was discovered in xz library that implements LZMA compression. xz, among many other places, is used, indirectly, in sshd. My attempt to explain what happened.

Timeline

Around the year of 2005, Lasse Collin together with other people, creates the xz library that implements the LZMA compression algorithm. LZMA is a lossless compression algorithm. Over the time, xz becomes widely adopted in compressing data, and you’ve probably had to download at least one file that has the extension of .tar.xz. Most of the work on xz was done by Lasse Collin, who, as we learn now, had personal struggle and not a lot of time to maintain the project. This setup was perfect for some social engineering.

At the end of 2021, we meet another person named Jia Tan. Jia Tan starts to create little patches to xz, but they are not merged.

Forward to mid 2022, another person, named Jigar Kumar, complains that the development on xz has stalled, and that the original maintainer should give the project to someone else. An exchange of accusations from Jigar Kumar, and explanations from Lasse Collin, take place in the following months from April till early July 2022. Eventually, Jia Tan becomes a maintainer of xz.

It’s important to note few things.

First, Lasse Collin is a one person that maintains a free open source library. A lot has been said on the state of open source and how maintainers treated unfairly. Companies build a billion dollar products on top of free and open source software, while rarely giving anything back. While the accusations from Jigar Kumar were probably part of social engineering, it’s not uncommon that maintainers of OSS are being harassed and demanded to patch their projects or keep developing them. There are no indications that Lasse Collin was part of the execution, and everything points out to the fact that he is just a guy who maintained a fairly popular open source library, and had personal health issues and lack of time.

Secondly, there are no proofs that both Jia Tan and Jigar Kumar are real individuals (rather than state actors); not the same person; didn’t know each other; none of this was planned. Jigar Kumar does not appear anywhere else on the internet apart from a few mailing list exchanges where he pushes Lasse Collin to give up the project to another person. Jia Tan does not appear anywhere else on the internet apart from small commits to xz about 2 years ago in order to build credibility and gain trust.

I’ll skip some other technical milestones in the timeline. You can read a detailed explanation by Russ Cox.

The Backdoor

The TL;DR; version is that Jia Tan goes on to create a very sophisticated backdoor that hides inside compressed xz (irony?) files that are, presumably, used for testing. This is smart, because nobody would suspect a compressed file inside tests/ folder to be malicious. In fact, it’s common practice to have mock data for tests, and these things are rarely checked during PRs, as we assume they are just test data.

The backdoor, which is hidden in multiple compressed files, taps into the build process of the xz library—essentially replaces the RSA_public_decrypt function symbol with a malicious function that gives an attacker, who possess a particular private key, remote execution capabilities.

This means that when SSH server is launching, it tries to load a dynamic library and looks for RSA_public_descrypt function, which is a legit function. However, if the distributed xz library is malicious, then it modifies the way the dynamic library resolves RSA_public_descrypt function, and instead it loads a malicious code. This code listens to a particular payload that is encrypted by a private key, that apparently only Jia Tan have, and using the embedded public key inside the code, it validates the payload (by checking that it was signed by Jia Tan’s private key)—and if so, it executes whatever is in the payload.

SSH runs as root. It has to, because it needs to de-escalate your privileges down to the particular user account that you log in with. This means that using remote code execution backdoor, one could bypass the entire account management and gain root access to a Linux machine that runs SSH. In case if you are not aware, this means millions, if not more, of machines all around the world. From web servers, to networking equipment, industrial equipment management systems (manufacturing, power plants, military and civilian objects), personal and home appliances, etc. Everything that runs SSH using systemd, on a rolling release Linux distribution, is vulnerable.

The backdoor was assigned the CVE-2024-3094 identifier, and has the base score of 10.0 CRITICAL, the highest possible severity.

Discovery

The backdoor was, accidentally, discovered by Microsoft employee and PostgerSQL contributor Andres Freund, when he was benchmarking PostgreSQL and noticed that his SSH connection took a hundred milliseconds longer and used more CPU cycles. He first thought that it has to do with a compromised Debian package, but later found out that the entire upstream xz repository has been compromised. If he hadn’t discovered it, there is a good chance that this vulnerability could have been reached major Linux distributions, and even production servers and systems.

There are discoveries now that Jia Tan was in talks with major Linux distributions to push his malicious version of xz. One such example, is this bug request for Ubuntu to update to xz to 5.6.1.

Andres shared his finding in an Openwall mailing list.

Who is behind the backdoor?

As of the time of publishing this article, there are no evidence that Jia Tan/Jigar Kumar are real individuals, and the consensus in the security community, is that a state actor is behind this attack. It seems like this backdoor was at least 2 years in the making, carefully planned and executed. Neither Jia Tan nor Jigar Kumar exist outside the few posts on mailing list of xz. Jia Tan’s role was to write the backdoor, while Jigar Kumar’s role was to pressure Lasse Collin to pass xz to another maintainer.

While this attack could have been carried out by an individual, or two, the fact that it was almost three years in the making, and the level of sophistication, points out to either an organized group, or a state actor. I could name at least 6 state actors who have the ability and the desire to plan and execute such attack, given the geopolitical situation in the past 3 years. I will, however, refrain from doing so, as the situation is still unfolding.

What’s next?

I want to come back to XKCD #2347. It clearly illustrates the modern digital infrastructure and the role that OSS plays in it.

We tend to forget that, but most of the modern digital infrastructure is run on the backs of hobby projects that few people maintain in their free time. These people are prone to losing interest, burn out, and other personal problems. And they are, like everyone else, prone to social engineering.

It’s unfortunate that Lasse Collin found himself in that situation. At one point, his GitHub account was even suspended (but it seems like it was reinstated few days ago). I don’t think he had any malicious intents, and he was just a guy maintaining a fairly popular open source library.

But we also need to remember that while the world became a safer place, there are still malicious actors in it. It is impossible to audit every single line of code out there.

Time will tell how we will move forward with OSS and our digital infrastructure, but as of today, we are thanking Andres who cares about CPU cycles, even in a world where everyone seems to solve their performance problems with “just buy more RAM”, and “let’s spin another EC2 instance”.

Timeline

The Backdoor

Discovery

Who is behind the backdoor?

What’s next?

Further reading

You might also like...

The Curse of the Senior Software Engineer

Serving Astro with Rust

Rendering emails with Svelte

How to stay junior forever

The Bloat in Software Engineering