Ten ideas to fix Magento's reverse brand ambassadors


Ecommerce people have been making long hours, as more stores were hacked in 2016. Magento saw a rise of Shoplift and brute force attacks. Stores that follow best practices should not worry. But a majority of stores do not keep up with patches and strong passwords, and get hacked sooner or later. These negligent merchants become reverse brand ambassadors. Which is ultimately a bad thing for everybody but.. Demandware.

The bottomline is: how to secure thousands of badly maintained stores? Nobody has found a really good solution yet, and - interestingly - there has been little public debate so far. Here are 10 ideas to get the conversation started!

  1. Crawl all 300K contact forms of vulnerable stores to reach the beneficial owner. So far, Magento has been contacting people who entered an email on the download page (ie, technicians). The response rate has been “very low”. Academic research claims only 5.8% of actual website owners can be reached in case of a vulnerability issue. Now, I think we have an edge with Magento because of the standardized contact form implementation (easy to scrape). Second, the recipient of the contact form has likely a larger interest in fixing the site than the tech who did the initial install but is no longer on the project.

  2. Encourage hosting companies (HSPs) to enforce secure installations. Magento Inc could promote HSPs that actively engage in protecting their merchants’ security. All of Magento’s official hosting partners are “committed to security” but few actively block the latest threats. HSPs can leverage intrusion prevention systems (IPS), and assist their customers with patching and incident response. Hacked, abandoned stores should be locked-down pro-actively to stop them spreading malware.

  3. Publish a nematode: a friendly, autonomous worm that crawls the Internet and fixes vulnerable stores. Slightly unconventional, but nevertheless proven valuable in the past. A well written nematode would fix all Shoplift-susceptible stores globally within 30 minutes. In fact, this is slowly happening right now, but by the bad guys: hacker groups are already closing the Shoplift hole after they got in, to keep competing hackers out.

  4. Government could tighten incident disclosure laws, so that hacked stores would actually notify their customers upon data theft. At the moment, this is certainly not the case, and merchants get away with sloppy maintenance. (Proof: I have placed canary orders at stores that I knew were hacked, and I was never notified afterwards. Were you ever?) This would be a great incentive for merchants to spend money on maintenance. Current laws are lobbied down to be very vague, and where the law actually dicates disclosure, enforcement agencies are seriously understaffed (eg in Holland).

  5. Set up a malware signature collection, and empower system administrators to easily detect security breaches (and submit new signatures). So far, two HSPs have shared their malware rules: NBS and Byte. The effectivity of these rules would get a boost if more HSPs joined. Both Magemojo and Lexiconn have already pledged to contribute.

  6. Set up a searchable archive of ecosystem vulnerabilities, similar to wpvulndb for Wordpress. At the moment, only Magento Inc and a few vendors have processes to disclose vulnerability information. And this information is not standardized or even accessible from a single place. Most vendors fix security bugs but write a small note deep in their release notes (if at all). Standardizing and vetting security announcements is quite a lot of work, but let’s start with a low-effort centralized repository of statements. Once we have that, somebody will publish a n98-magerun plugin the same day.

  7. Implement and enforce an immutable document root in Magento (and offload somewhat volatile data such as media to a CDN). This would prohibit attackers to hide malware on the server. The required technical changes are no deal breaker for the Magento core, but several 3rd party modules assume a writeable document root. Supporters include Max Pronko, Ivan Cherpurnyi and Fabrizio Branco.

  8. Magento could enforce Content Security Policy headers by default. This would make it harder for hackers to inject malicious javascript in a site. Scott Arciszewski has written a proposal for this. Max Chadwick also has a few insights on restricting “Miscellaneous HTML” in the backend.

  9. Magento could encourage Google to block compromised sites in Chrome. At the moment, Google’s Safe Browsing team has about 10% of compromised stores in its filter. Blocking is a rude measure, but without it, the hijacking of private data continues and total damage increases.

  10. Magento Inc could hire more vulnerability researchers to find bugs before they get released. Magento has many top of the bill engineers already, but hunting security flaws is a specialized art. To identify the best Magento bug hunters globally, one would simply have a look at the credits in the latest patch releases. I posit that if Magento Inc paid the best bug hunter $200k per year and s/he would find just a single RCE exploit in unreleased code, it would be an absolute bargain.

  11. Organize a dedicated MageStackDay to increase patch rates globally. Or, as Anna Völkl called it, a MagePatchDay! Enthusiastic volunteers would teach merchants why and how to get their security on par. Especially the “why”.

  12. Give small time merchants a discount for Shopify. The long tail of Magento installations are people who chose Magento just because it is a “free download”, but have only 3 SKUs. They will not pay an agency $20k for proper maintenance and will certainly not pay Magento Inc for an Enterprise license. So it would be good for everybody if these stores converted to a SaaS solution with zero maintenance.

Yes, more than ten, the ideas just kept coming! I must still have missed a couple though. Are you a concerned professional? What works and what doesn’t?


Fixing Jekyll --watch and 404 errors


When you edit drafts locally, a very handy Jekyll feature is the –watch flag, which triggers Jekyll to rebuild whenever you save your post:

$ bundle exec jekyll serve --drafts --incremental --watch

However, on my Ubuntu 16.04 desktop, this yields erratic 404 errors.

jekyll 404

After cursing Jekyll for a few weeks, I decided to dig into it using my favorite tool strace. And indeed, Jekyll removes the draft after a save. When run with --verbose, it becomes clear:

Skipping: _drafts/world-domination-howto.md has a future date

I added some debug to lib/jekyll/readers/post_reader.rb and it appears that the file’s ctime is a few microseconds off from the its mtime. Curious?

A quick fix is to run Jekyll with the --watch --future option.

A thorough fix probably involves digging into the internals of my ecryptfs home folder (which is a standard Ubuntu feature by the way).


Magento and upcoming SSL requirements


At the end of January, Google will release Chrome 56 with two important changes:

  1. Sites using SHA1 SSL certificates will be blocked
  2. Sites accepting passwords or credit cards without SSL will show a warning

Other browsers such as Firefox have also announced stricter SSL handling.

Old SHA1 SSL certificates

Old SSL certificates use SHA1 hashing, which is considered insecure. New certificates use SHA256 or even SHA512. The quest to eliminate insecure SSL certificates has been going on for a few years. Browsers have displayed incrementally intrusive warnings. The latest Chrome warning is a red mark in your address bar:

chrome sha1 warning

But from January onwards, Chrome 56 will actually block you from visiting the site.

Half a year ago I wrote that 37.000 Magento sites were using insecure SSL certificates. Now, that was an awful lot. The good news is that certificates have to be renewed periodically, and SSL providers have not handed out insecure certs since 2016.

So what is the verdict today? I have reran my scan of all 305K known Magento stores globally and detected .. only 432 stores with outdated SSL! These stores have an old insecure certificate with multi-year validity (beyond 2016) and their owners probably forgot about them.

To summarize, less than 0.2% of Magento stores will break once Google releases Chrome 56. I think that is an amazing result.

No SSL at all

Until now, Chrome showed a small exclamation mark for non-SSL stores. Chrome 56 will show a slightly bigger warning:

chrome non ssl warning

While this isn’t exactly terrifying yet, it will hopefully encourage a couple more store owners to go SSL-only. Because that isn’t quite commonplace: out of 305.000 Magento stores, only 31.000 (<11%) enforce SSL site-wide.

In June 2017 I will run a new scan. How many Magento sites will be HTTPS only by then? Place your bet in the comments below.


5900 online stores found skimming [analysis]


Update Dec 1st: already 2300 stores have been fixed! Thanks to everybody who tirelessly notified and fixed stores.


Last week I showed how the Senate Republicans were skimmed for 6 months (and then it was quietly fixed). But this is just one example of thousands of stores that have been compromised and are still being skimmed.

real skimming

How it works

Online skimming is just like physical skimming: your card details are stolen so that other people can spend your money. However, online skimming is more effective because a) it is harder to detect and b) it is near impossible to trace the thieves.

In short: hackers gain access to a store’s source code using unpatched software flaws in various popular e-commerce software. Once a store is under control of a perpetrator, a (Javascript) wiretap is installed that funnels live payment data to an off-shore collection server (mostly in Russia). This wiretap operates transparently for customers and the merchant. Skimmed credit cards are then sold on the dark web for the going rate of $30 per card .

Online skimming gains popularity

Online skimming is a new form of card fraud. In November 2015, the first case was reported. Upon investigating, I scanned a sample of 255K online stores globally and found 3501 stores to be skimmed. It is now ten months later. Are the culprits in jail yet? Not quite, here are the numbers of compromised stores:

November 20153501 
March 20164476+28%
September 20165925+69%

Victims vary from car makers (Audi ZA) to government (NRSC, Malaysia) to fashion (Converse, Heels.com), to pop stars (Bjork) to NGOs (Science Museum, Washington Cathedral).

754 stores who are skimming today, were already skimming in 2015. Apparently you can skim cards undisturbed for months.

Culprits get professional

In 2015, reported malware cases were all minor variations of the same code base. In March 2016, another malware variety was discovered (report in Dutch). Today, at least 9 varieties and 3 distinct malware families can be identified (see my collection of samples). This suggests that multiple persons or groups are involved.

One reason that many hacks go unnoticed is the amount of effort spent on obfuscating the malware code. Earlier malware cases contained pretty readable Javascript but in the last scan more sophisticated versions were discovered. Some malware uses multi-layer obfuscation, which would take a programmer a fair bit of time to reverse engineer. Add to this that most obfuscation includes some level of randomness, which makes it difficult to implement static filtering.

To trick the casual observer, the malware has sometimes been disguised as UPS code:

ups disguise

Another sign of malware sophistication is the maturity of the payment detection algorithm. The first malware just intercepted pages that had checkout in the URL. Newer versions also check for popular payment plugins such as Firecheckout, Onestepcheckout and Paypal.

Replies from worried merchants

I have manually reported several compromised shops and got some curious responses:

We don’t care, our payments are handled by a 3rd party payment provider

If someone can inject Javascript into your site, your database is most likely also hacked.

Thanks for your suggestion, but our shop is totally safe. There is just an annoying javascript error.

Or:

Our shop is safe because we use https

Solutions

New cases could be stopped right away if store owners would upgrade their software regularly. But this is costly and most merchants are not aware of the risks.

Besides, that would not repair the current hausse of abuse. While stores could be contacted on an individual basis, it is a lot of work and nobody does it. Companies such as Visa or Mastercard could revoke the payment license of affected merchants. But it would be way more efficient if Google would add the compromised sites to its Chrome Safe Browsing blacklist. Visitors would be greeted with a red warning and induce the store owner to quickly resolve the situation. I have submitted all my malware samples to Google’s Safe Browsing team but only a small part of the detected malware has been blocked so far.

safebrowsing

Are you a merchant?

If your store is compromised (check MageReport), find a competent programmer or development agency and send them here: how to recover a hacked store.

Read more:


Senate Republicans were skimmed for six months, quietly fix store


Did you order anything from the Senate Republicans in the last half year? In that case, your name and credit card details have been skimmed and sent to a Russian server. And subsequently sold on the dark web for $30.

Update Oct 6th: The Republicans have rushed to secure their store today. But no word about the skimming between March 16th and October 5th.

See a short video where I demonstrate how the skimming works. And read on to find out how I traced the culprits to a hornet’s nest of criminal activity.

I think I’ll pass on the Never Hillary sticker for now.

The crime scene

So our evidence consists of one compromised Republican store, which was fitted with hidden skimming software at least 6 months ago (dissection of the malware here). And we have two Russian credit card harvesters with the rather boring names jquery-cloud.net (March) and jquery-code.su (October).

Follow the money

The older harvester jquery-cloud.net was registered in December 2015 by an American lady with a Chinese fax number and a fake email address. The newer harvester, jquery-code.su, is registered anonymously per 24th of August.

Both domain names are hosted by a company called Dataflow, as is shown by the nameservers and IP addresses. Curiously, the Dataflow network and the jquery-cloud.net domain name were created in the same week:

route:          80.87.205.0/24
descr:          DDoS Protected Network DATAFLOW.SU
origin:         AS203624
mnt-by:         MNT-DATAFLOWSU
created:        2015-12-28T22:37:25Z

A hornet’s nest

Dataflow has a Russian language website but is registered in Belize on November 3rd, 2015. It advertises with:

Offshore […] Solutions with protection from DDoS to 350 Gbit : Belize, Panama, Seychelles

Its office is registered here:

This address shows up in the Panama Papers and is - coincidentally - also the home of a trust office called Alpha Offshore, who

is an international provider of legal corporate tax planning services. Mainly, we focus on registering companies in countries that use preferential taxation policies and in offshore jurisdictions

Dataflow has a very small network of just 2 blocks (512 IPs) and you can look up what else runs on that network. Its owners deserve praise for collecting about every kind of online fraud known to man: money laundering, synthetic drug trade, darknet messaging, phishing and spam.

Estimated black market yield

Money Power Respect

I do not know how many credit cards were stolen from the Republican store but I can make an educated guess. According to TrafficEstimates, the Republican store has received some 350K visits per month lately. A conservative conversion ratio of 1% yields 3500 stolen credit cards per month, or 21K stolen credits cards since March. Black market value per card is between $4 and $120, so I assume a modest $30 per card. The villains could have made roughly $600K on this store alone.

Note, this is just the criminal yield. The monetary loss for society is higher, as credit card companies reimburse their clients for fraudulent deductions (actual deductions are much higher than the black market value!) and conduct investigations. They shift these fraud handling costs to their clients, so that merchants pay a higher transaction fee and, in turn, shift this to their customer (you).

Conclusions

This clever form of card skimming has been going for a while, at least since March. The culprits are hiding behind an shelf company in Belize. Their business is growing rapidly, which I will illustrate in a next post.

Economics and culture of credit card laundering.

Donald Trump’s view on cyber security.


Pagination