Hacker, Interrupted

In early November, while refining our anomaly-detection service, I found a data leak on our network. Our internal network is well equipped with ExtraHop Reveal(x) Ultra: a Discover appliance, which captures wire data from the network for metric analysis, a connected Explore appliance, which indexes and stores transaction records, and a connected Trace appliance, which indexes and stores packets.

By leveraging these network traffic analysis capabilities, I was able to take moderately suspicious behavior and dig into the details of an exfiltration attack that would have been otherwise impossible to find.

Indicator of Compromise

Less than a day after I expanded our command-and-control detector to include web socket connections, I was rewarded with the following detection: a potential malware infection.

Reducing false positives plays a big part in creating accurate detectors, so I wasn't overly alarmed. I began my standard investigation process with measured expectations.

Checking the Endpoints

I checked our internal applications and learned that the affected device was an Ubuntu workstation for one of our employees. A search through the American Registry for Internet Numbers (ARIN) revealed that the peer IP address belonged to a droplet in the DigitalOcean cloud infrastructure.

It isn't unusual for an employee's workstation to communicate with a peer at a cloud provider as part of normal internet browsing, but the direct IP connection was unusual.

Scanning Wire Data

I queried the Trace appliance for all of the stored packets between the Ubuntu workstation and the unknown peer device for the previous few days and found a large number of outbound websocket connections over a nonstandard port—6332.

I downloaded the packets as a PCAP and opened the file in Wireshark for analysis. The Origin header for the direct IP connections in the packets was initiated by a Chrome extension. Still suspicious, but no clear evidence of foul play.

I continued my investigation by checking the message payloads, to determine what information was being sent off-site. The payloads were obfuscated and unreadable, instead of being encoded as UTF-8 text per the WebSocket RFC specifications.

This was officially a red flag.

Analyzing the Code

I extracted the extension ID from the HTTP Origin header and found the related extension on the Chrome Web Store.

The low user count raised concerns, but because Chrome extensions are written in JavaScript, I was most interested in looking at the code. I was able to Google for the source code without installing the extension.

It took a while to find the offending code because it was buried at the bottom of a minified library.

By deciphering and executing the code, I was able to reproduce the websocket URL I'd found in the PCAP.

This was all I needed to verify that the Chrome extension was exfiltrating data from our internal machine to an unknown endpoint in the DigitalOcean cloud. The deliberately hidden code was obviously suspicious: time to notify our Security team.

Securing the Leak

We quarantined the compromised workstation and began to search for other compromised machines. We queried the Explore appliance for all transaction records over the nonstandard port by the suspicious peer IP address and found that two additional machines were compromised. They were quickly quarantined.

We downloaded the related packets for those records for further analysis. One of the transactions was related to the same employee, because the extension had synced to their laptop, and the other was for a second employee. We revoked their credentials and reset passwords, while we investigated the full extent of the leak.

Assessing the Damage

Having secured our network against further attacks, we dug deeper into the exfiltration: what had they stolen? We analyzed the extension code and found that the latest version collected URLs through the WebRequest API, compiled them into batches, and transmitted the latest batch every 12 seconds.

There was no guarantee that a previous version of the extension didn't contain additional malicious functionality. So we cross-checked the last time the extension was updated with a record query for the transactions stored on the Explore appliance. Luckily, neither of our users had updated the extension since the initial installation.

We further downloaded all packets sent to the malicious host and decoded them. Only URLs were sent and none of them contained sensitive information.

How Bad Things Happen to Technical People

So, how did our employees and 25,000+ other tech-savvy users end up installing this bogus extension?

Simple deception. Postman is the name of a popular Chrome application, which is different from the extension. It's an easy difference to miss and the application isn't even included in search results from the Chrome Extension Store.

We reported the malicious extension (along with several other extensions by the same hacker that also contained malicious code) to Google. A month later, after a post to a public hacker news forum, Google removed the extension. We also reported the control server to DigitalOcean, but haven't heard back.

We were lucky: only URLs were stolen, even though the attacker had sufficient permissions to access and exfiltrate user credentials and sensitive corporate information. If the leak had remained undetected, it might have been open for months, during which the attacker could have silently pushed updates containing additional malicious functionality to tens of thousands of users.

As an engineer, I don't often have the opportunity to see our products from an end-user's perspective. The leak gave me a chance to see the strength of the ExtraHop platform, and to see how each appliance's capability was essential to quickly resolving what could have been a potential nightmare. It took us less than half an hour from the initial detection to completely mitigate the leak.

In the next post in this two-part series, I'll walk you through my adventures in writing my own Chrome malware in order to see just how much damage a more sophisticated attacker could do.