Hi friends,
​

This is the story of how one mysterious hacker, took advantage of an inactive negligible library in Linux, and the maintainers emotional stress.

Slowly, week after week, bit by bit lines of seemingly random testing code were added, in front of everyone’s eyes, to the Linux upstream.

Thanks to one curious developer, who had an itch because of a tiny lag in his login process, sending him deep into the rabbit hole of Linux SSH and the mystic world of ZX, we would have never heard of this critical 10-graded security hole.

2 years earlier // 19 May 2022

A random developer starts shooting emails on the community board asking whether the niche XZ utility in Linux is still maintained.

They wanted to know whether they should update their own version because they now want new features.

What’s XZ?

Xz is a compression utility - similar in concept to making .zip files. Its main use is lossless compression for command line utilities, which is to say that it guarantees when it is uncompressed the result is a byte-for-byte clone of the original data. It’s used by a lot of important security software, and is included as a library for many other utilities. A library is just a term used for tools used by other tools.

The core maintainer “Lasse” then says that he’s not actively working on it but if bugs will be reported he’ll make sure they’re fixed.

He also says that he know this isn’t not optimal but towards the end mentions a developer by the name “Jia Tan”, saying they were a big help with XZ utils and “might have a bigger role in the future…”:

Others respond as well with questions until one guy, pops out of no where, his name is “Jigar Kumar” and he’s making very pushy comments towards a new maintainer, saying that people should either fork or establish a new leader, finishing by saying it’s “sad to see … a repo like this” in his words.

The same Kumar goes on in another reply to say that he “doubts we’ll see a release this year”, telling the maintainer that “you ignore the many patches..” and “you choke your repo”:

Months later, a security researcher will start questioning the identity of this “Kumar” and how he emerged out of no where and disappeared off the face of the earth just as fast.

Related or not, “Jia Tan” keeps working and slowly, unofficially, becomes the de-facto maintainer of XZ utils. A small compression library, used by every major Linux distro.

Over two years “Jia Tan” added changes, slowly, bit by bit, with commits that didn’t come up on any radar:

They even contributed a PR that was merged into Google’s oss-fuzz project to disable information coming up from the library.

Google’s project’s purpose is “uncovering programming errors in software… (that)… can have serious security implications” 🤔 https://github.com/google/oss-fuzz/pull/10667

He then went on and used yet another alias to ask for XZ updates from the community, pushing the fixes even further. Fixes, that will soon be identified as a backdoor in the Linux SSH protocol.

The back door part comes into play with one of the main ways xz is used - SSH. SSH is an encrypted protocol between two machines where text commands can be exchanged, allowing a user to interact with a server. It’s a very common utility in the Linux world and the security of this communication is critical. The back door means that the connection is no longer private and could allow an attacker to insert their own text commands into the secure connection.

Fri, 29 Mar 2024 08:51:26 -0700

Andres Freund, A software developer at Microsoft, randomly notices “odd symptoms” when his SSH login process took a tiny bit longer. How much? 600 milliseconds. 600 milliseconds between the world, and one of the greatest zero-day Linux vulnerabilities ever found.

Andres noticed the CPU levels went noticeably higher than expected and went down the rabbit hole, investigating the library leading to the issue and reporting his findings with a clear message:

The reason this was ever discovered is that the injected code was heavily reliant on in-memory computational processes, which replaced symbols in real time, effectively allowing other parties in on the SSHd process.

Quoting Andres again : > “This is the quite slow step that made me look into the issue.”

The next day 2024-03-29

The news blow up, every major news outlet, security news, Linux distros and even cloud providers started sharing the news, either covering the story, urging users to update their systems.

Keep in mind this is still VERY fresh. New information is discovered by the minute. And while the project is already disabled on Github, if you follow the news closely you’ll keep getting updates revealing information over this absolutely crazy story.

It’s still unclear who’s the mysterious Jia Ten, whether the other names are aliases or maybe, maybe, this is a nation backed actor, or even a group, slowly building their own zero days vulnerability into one of the worlds most sensitive protocols!

What should you do?

The silver lining here, is that staying out of the infection zone is rather easy, downgrade of upgrade from the infected version, or just upgrade your linux to latest will solve the problem. Moreover, Not all distros even use the compromised library, it’s mostly Debian but others as well. If you want to be 100% sure, as said - just upgrade and you’ll be fine.

Moreover, in the majority of cases the vulnerable version didn’t get released into any major stable version so most users should be safe, but it’s still wise to monitor the situation and make sure all systems and utils are up to date.

What can we learn from this?

1. The hard lesson we get to re-learn every year: Open source is not magic, nor a security guarantee! We’ve all seen the implication of the “everything npm” package earlier in 2023 which as only a joke that got out of hand, but famous other vulnerabilities like log4shell and the often forgotten heartbleed bug (https://heartbleed.com/) should have taught us to not trust every package even if it’s coming from a famous upstream, or your local package manager!
2. Focused engineers save lives. yes, I took it this far, focus is critical, and curious engineers like Andres saved the day / year for MAJOR system’s most of us aren’t even aware of. I can come up with a bunch of critical civil infrastructure running linux what would have surely got infected with this at some point, and if not for Andres this could have never even come up.
3. Does this make you think of how many exiting bugs like this we currently have without even knowing? How many zero day bugs are out there ready to be used?
   Systems are not as safe as they seem to be, and this is great reminder! I hope you enjoyed this, and I’ll see you on the next one!

Special thanks to Andres, who was diligant, focused and curious enough to help the world fix one of the worst malicious bugs ever! Be like Andres!

Links:

• ​https://openwall.com/lists/oss-security/2024/03/29/4​
• ​https://lwn.net/ml/oss-security/20240329155126.kjjfduxw2yrlxgzm@awork3.anarazel.de/​
• ​https://x.com/kdrag0n/status/1773950974480314574?s=20​
• ​https://aws.amazon.com/security/security-bulletins/AWS-2024-002/​
• ​https://openssf.org/blog/2024/03/30/xz-backdoor-cve-2024-3094/​
• ​https://duo.com/decipher/red-hat-warns-of-malicious-code-in-xz-utils​
• ​https://www.theregister.com/2024/03/29/malicious_backdoor_xz/​
• ​https://unit42.paloaltonetworks.com/threat-brief-xz-utils-cve-2024-3094/​

Thank you for reading, I hope you liked it, and maybe even learned something new.
Enjoy your weekend!