Priorities

Remember how often the word unprecedented got used in 2020?

Unfortunately, 2021 has started off in an unprecedented manner as well. Solar Winds, Microsoft Exchange, Big-IP to name but a few of the unprecedented things that we’ve been dealing with since the ball dropped on New Years Eve.

Now is not the time for people who aren’t serious about their work to be involved in IT and especially IT Security.

That’s why this story bothers me so much.

First of all, let’s get this straight: I’m deliberately not naming names in this story because I don’t want to out anyone - I’m a hell of a nice guy that way. Not that they don’t deserve to be outed - they do. But I was young and stupid once too - we all were, so I’m somewhat apt to give that combination a pass.

Also, and this is important, I’m working off of a very limited view of the individuals involved. I don’t know them, I don’t know their mindset or their struggles - I only know what I can interpret from their social media posts - something that, I’ll admit, is likely to paint a horribly inaccurate portrait.

But, while this certainly may not be an accurate portrait of this unnamed individual, I do believe that it accurately portrays something about our industry in general - and that’s why I’m telling this story.

So, let’s start at the beginning…

Hi, my name is Tom, and I have a knack for finding compromised systems on the Internet. I find them, and then I try to contact the system owners (as opposed to 0wners) to get them fixed. Hey… it’s a hobby.

Generally speaking, I have two main tools that I use find compromised systems:

• I run a bunch of different types of honeypot systems
• I do a lot of Google searches

This story starts with the latter.

Using some Google search-fu, I found a website that had been 0wned by purveyors of boner pills. Unless you’re new to the Internet, this shouldn’t be surprising. Folks selling all manner of borderline-illicit things whack websites left and right on the ‘Net, placing links from the compromised sites pointing back to their own storefronts. The idea is to boost their site’s rankings in search engine results - a process known as Search Engine Optimization (SEO) Hacking. (Lest I get a slew of emails again, I am compelled to state that there are legitimate SEO practices that can be used to boost a site’s search engine ranking. These do not, however, involve hacking other sites. Just sayin’…)

The more popular and more legit a site is, the better target it represents for SEO hacking. If you’re trying to boost your site toward the top of the search engine rankings, you want it to have links from popular and legitimate sites. And lots of those links.

As best as I can tell, the site I found had been compromised for at least 7 months. It had, according to Google, about 21,800 "pages" linking to various boner pill storefronts. Please note that I put the word "pages" in quotes (<-- there, I did it again). That's because the compromised site was run on WordPress (the WebApp Hacker's BFF^™). As such, the whole concept of "pages" is a little vague... YourFlyIsOpen.com is a statically generated site and thus has an individual html file for each page. WordPress sites are dynamically generated and can just have a bunch of URLs listed in a database that point to a single chunk of crappy markup/HTML stored in that same database. When WordPress gets a request for one of those stored URLs, it just vomits out the associated HTML. Thus, the idea of "pages" becomes a little bit slippery...

All of that being said, this is pretty run-of-the-mill stuff on the ‘Net in 2021. I find sites like this all of the time and I do my very best to find someone to get them cleaned up. That’s where this one went a little off the rails.

I sent multiple emails.

I called and left voicemails.

Nothing.

Sometimes, for whatever reason, a specific site compromise bothers me. For most sites, I’m just fine to make an effort and then let it go if nothing happens. Every once in a while, I just can’t walk away. This was one of those.

I decided to do a little digging. Helpfully, the site had one of those Our Team sections that listed a bunch of their IT folks. That was what I used for firing off the emails and phone calls.

I decided to use some of those names to poke around a little bit on Twitter. I figured maybe I could use the Big Blue Bird to give someone a nudge to do something about their compromised site. On my third try, I had what looked like a hit.

I pulled up a list of their Tweets and plowed down through them in order to see if I could find any details that might verify that this was, indeed, a person associated with the compromised site.

That’s when I got aggravated.

A few months back, they had Tweeted triumphantly about the results of a phishing exercise that they had performed. They were absolutely gleeful about the fact that they had “gotten hits” from several members of the organization who would now need to perform remedial phishing training.

What. The. Hell.

Generally, I’m not one for tossing around Biblical quotes, but this one just seems so darned appropriate:

Thou hypocrite, first cast out the beam out of thine own eye; and then shalt thou see clearly to cast out the mote out of thy brother’s eye. - Matthew 7:5

As a practice, Security has a problem with beams, motes, and priorities.

For the past 20 years, it’s been my privilege to work with all manner of folks who are trying to do better - trying to up their game. These are serious folks, working in serious organizations, being serious about doing their best.

Unfortunately, there are still a ton of organizations out there that are content to just go along, doing what they’re doing, generally being unserious about their jobs. These folks (and thus their organizations) have priorities that are all askew.

Let me spell this out: If you’re an IT professional and you get an email from me, telling you about a system or site compromise, you likely aren’t being serious about your job. You need to check your priorities.

Put aside for a moment that your site got whacked… hell, people make mistakes, those things happen. The bothersome thing here is that someone else found out about it before you did. You can’t be monitoring your network to an appropriate level if someone else figures this stuff out before you.

Let’s try to get some priorities in order:

1. If your organization provides publicly available services (web, email, VPN, etc…) your first priority is to make sure that those are deployed securely.
2. Monitor your frickin’ logs. No, this isn’t as fun or as flashy as a pentest or a phishing exercise, but it’s your job and it’ll find your problems before someone else (like me!) does. Seriously, that’s your second priority.
3. Unless and until you’ve done the work of actually locking down your environment to the level of generally accepted best practices don’t even think about pentests. Professional pentesters will eat you alive.
4. Make sure you’ve gotten the beam out of your own eye before looking for the mote in someone else’s. Don’t even think about running a phishing exercise unless you’re checking your logs and you’re damned sure your website isn’t compromised.
5. Get yourself educated. Before you can educate your users (and you should be educating your users), you need to understand more about your job. Security is constantly evolving, if you’re not, you and your organization will be left behind.

A couple of bonus chunks of advice:

• Phishing exercise should never be gotcha games.
• What is there to be gleeful about when someone takes a phishing bait? That just means you failed to properly educate them.

Security 101 in a nutshell:

• Deploy your stuff securely
• Monitor your stuff to make sure it stays secure

Please note: Stuff wasn’t my first wording…

Priorities are important. Make sure you understand yours.

Now, if you’ll excuse me, I think I’ve got something in my eye.

-TL
Tom Liston
Owner, Principal Consultant
Bad Wolf Security, LLC
Mastodon: @tliston@infosec.exchange
Twitter (yes, I know… X): @tliston
March 22, 2021

Postscript

Because I am - as stated earlier - a hell of a nice guy, I’m going to leave you with some tricks that you absolutely should be using to monitor your organization’s website. This isn’t a substitute for monitoring your logs, but it’s something you should be doing in addition to monitoring your logs.

• Do a Google search. Using Google’s site: keyword, anchor your search to only pages on your site.
  • site:yourwebsite.com
• Check out the number of results at the top of the first page. If this is your first time looking at your site, you may not have a clue how many pages your site has. Remember: There’s a good reason they say “About X results.” Use this number as a reference/order of magnitude only. But if this number changes dramatically the next time you do this, you need to figure out why.
• Look through all of the results and make sure there isn’t anything unexpected in there…
  • I never said this was going to be easy…
  • If your site has an inordinate number of pages, you can narrow things down a bit by adding some well chosen search terms. I generally throw in the word buy. Remember though, you could be missing things.
• For some additional fun, you can paste &as_qdr=y15 at the end of the Google search URL. That’ll tag each search result with the last time that Google noticed that the page was updated.

Seriously. GO DO THIS. Because if you don’t, I will…