Internet Exposure of memcached

 memcached (UDP/11211)

It's an easy-to-use DDoS Howitzer AND a NoSQL database!

TLDR

• WHAT IT IS: An in-memory key-value store, used usually in caching website assets for geographically distributed websites.
• HOW MANY: 68,337 discovered nodes. 68,337 (100%) have version fingerprints
• VULNERABILITIES: 13 CVEs since 2011, but it has a wicked amplification DDoS issue we cover in the Exposure Information section.
• ADVICE: Use it! Just don’t expose it to the internet.ALTERNATIVES: Redis and etcd are two similar, alternative in-memory key-value stores with characteristics similar to memcached.

Memcached is an in-memory key-value store for small chunks of arbitrary data (i.e., strings, binary objects) from results of database calls, API calls, or web page rendering. Its simple design has made it wildly popular, as it promotes quick deployment and ease of development.

Discovery details

Project Sonar found 68,337 exposed memcached hosts, and we did a double-take when we saw that South Africa is in third place, since we don’t often find it in any other top 10 lists of exposure. Most (97%) of these SA nodes are in two autonomous systems: Icidc Network (87%) and Internet Keeper Global (10%), and a majority of hosts in each autonomous system appear to have similar exposure counts in both nginx and SSH.

As noted in our section on Redis, Amazon has a cloud “cache” service offering that can also be configured to use memcached directly or emulate the exquisitely diminutive memcached protocol, and Alibaba has a managed memcached service offering, so it is no surprise finding some in those environments, but it is somewhat disconcerting that these instances are exposed to the internet. What’s more surprising is that OVH goes out of its way to help you secure memcached and, yet, ~1,500 folks apparently did not get that rather crucial memo.

Exposure information

Why all the fuss about memcached? Well, way back in 2018, an “Insufficient Control of Network Message Volume” vulnerability in the UDP portion of memcached was used to launch the largest distributed denial-of-service amplification attack ever, and disrupted many major internet services. Cloudflare summed up the flaw quite well in its blog at that time:

"There are absolutely zero checks, and the data WILL be delivered to the client, with blazing speed! Furthermore, the request can be tiny and the response huge (up to 1MB)."

Attacker’s view

While we do not have a memcached honeypot, we can see connection attempts on UDP 11211 and peek at the packet captures to see if memcached commands (most often, the stats one we use, though we occasionally see what look like misconfigured clients that are connecting to what they think are their memcached servers). The daily memcached command connections we see are mostly from Shadowserver, which is (correctly) self-described as a “nonprofit security organization working altruistically behind the scenes to make the internet more secure for everyone,” despite their scary name. They also scan the internet every day to try to get a handle on exposure and help organizations (for free) get a picture of their attack surface. We heart Shadowserver.

During the first third of 2020, we saw spikes of near 80,000 data-less UDP connections on 11211 across a handful of days, but none of this appears truly malicious in nature.

Our advice

IT and IT security teams should never expose memcached to the internet, and should ensure via playbooks and automation that development, test, and production memcached environments are rigidly controlled.

Cloud providers should continue to offer secure service alternatives to self-hosted memcached, ensure their provider-maintained machine images are kept patched with a default configuration of memcached only available on non-internet-exposed interfaces, and—frankly—not allow memcached to be exposed to the internet on host in their sphere of network control.

Government cybersecurity agencies should provide regular reminders about the dangers of memcached, offer guidance on how to run memcached safely, monitor for malicious use of memcached, and strongly encourage ISPs and cloud providers to block connections to memcached’s default port.

Links:

• https://www.rapid7.com/blog/post/2020/12/07/nicer-protocol-deep-dive-internet-exposure-of-memcached/

Internet Exposure of etcd

etcd (TCP/2379)

Gleaming the Kube(rnetes)

TLDR

• WHAT IT IS: Another distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines.
• HOW MANY: 2,560 discovered nodes. 2,560 (100%) have version fingerprints
• VULNERABILITIES: Two low-to-moderate CVEs since 2018.
• ADVICE: Use it! Just don’t expose it to the internet.
• ALTERNATIVES: Redis and memcached are two similar, alternative in-memory key-value stores with characteristics similar to etcd.

The etcd key-value service is part of the Kubernetes ecosystem and is designed to hold system/service configuration and state information. The Kubernetes API Server uses etcd's watch API to monitor the cluster and roll out critical configuration changes or simply restore any divergences of the state of the cluster back to what was declared by the deployer. It exposes a JSON API over the HTTP protocol.

We’re including etcd for completeness (since we’ve mentioned in the previous blogs on Redis and memcached), but the sample size is way too small to dig into, since we have no data on which ones are honeypots and which ones are real.

Just like the other two key-value databases, etcd should never be exposed to the internet. Unlike the previous two services, etcd tends to be purpose-driven for Kubernetes orchestration environments, which is another great reason not to expose it to the internet directly.

Links:

• https://www.rapid7.com/blog/post/2020/12/10/nicer-protocol-deep-dive-internet-exposure-of-etcd/

Incident Mgmt Vs Vulnerability Mgmt

Everyone agrees that Incident Management is different than Vulnerability Management. But yet still people mix them up when things happen.

Recently, SolarWinds supply chain attack has becomes the hottest topic in the month of December 2020 for Cybersecurity world. With the SUNBURST or SuperNova backdoors found, it is obviously becomes an incident (rather than a vulnerability) if your company is using the affected products. With the investigation is still ongoing, more affected products might be included in the list.

But, why people try to manage the SolarWinds case with vulnerability management? Is it because everyone is more familiar with vulnerability remediation process.

Yes, both incident management and vulnerability management has the remediation process. However it is performed by different teams. The remediation process for an incident is performed by Cybersecurity team, while the remediation process for a vulnerability is performed by IT team.

I guess people just thought that the all remediation process are the same.

Why people mix them up so easy?

I think it is due to sophisticated Cybersecurity team structure. And there is monthly remediation reminder that continuously reminding us on the remediation process. People just like to engage with what they are familiar with.

If your company is using a vulnerability management process to handle the SolarWinds attack or incident, then you should start think about simplifying your Cybersecurity teams. 

Retirement of Enhanced Security Admin Environment ESAE

Today, I just found that Microsoft has stopped their ESAE strategic architecture. Microsoft has recommended new strategy to complements any existing ESAE implementation. 

The Enhanced Security Admin Environment (ESAE) architecture (often referred to as red forest, admin forest, or hardened forest) is an approach to provide a secure environment for Windows Server Active Directory (AD) administrators.

In early 2018, where I first learned about ESAE. After a deep dive evaluation, my conclusion is, it is too costly to implement such a solution, in term of operating cost, for most of the company (even it is a secure strategy). This is simply because the strategies have never consider the essential security concept. #EssentialSecurity 

And after 3 years, Microsoft has admitted that the ESAE architectural pattern is only valid or applicable in a limited set of scenarios. Any organization who implement ESAE architecture, must accept the increased technical complexity and operational costs of the solution. 

The ESAE retirement post can be found at https://docs.microsoft.com/en-us/security/compass/esae-retirement

GUI bug in KennaSecurity

There is a minor bug found at the Kenna's user interface that I encounter today.

To simulate the bug, from the Dashboard, look for any Kenna meter that has zero asset count (or you can create one), click on it to explore the asset/vuln list. And you will found that most of the filters are missing at the Explore page, such as:

• (Asset Filters) Active/inactive
• (Asset Filters) Tag
• (Asset Filters) Sort by (everything)
• (Vulnerability Filters) Status
• (Vulnerability Filters) Classification
• (Vulnerability Filters) Connector names .....
• (Vulnerability Filters) custom fields.....

 😞😞