For any website manager, one of the most stressful moments in the job comes when the website gets hacked into. Even more so for smaller websites that don’t have a team of experts who look after such things.

A website like this one for example, which suffered a hack last Thursday.

It was an unusual attack, in that it didn’t try to delete the site or post “oh so funny” threats from Anonymous, which seemed to be all the rage a few years back. This attack actually went to a lot of effort to stop me taking the site down for maintenance, because their plan was to flood the pages with adverts, and earn an income from my distress.

I’ve flagged the fraudster to Google, as it’s their AdSense code being used in this way. Google wont tell me what will happen, but at a best they’ll close the fraudulent account without paying anything.

Of course, I am out of pocket as well.

But, it’s the helpless frustration that comes from playing whack-a-mole with the hacker as you desperately try to find how they got in, and stop them screwing the website, while it is still live to visitors who are now complaining about the nasty adverts on there, that’s so stressful.

In the olden days of the internet, in the halcyon times of a couple of years ago, there was a lot of hacking going on, but most of it was “drive by shootings” by simple scripts usually downloaded off the internet by spotty faced kids who sat in bedrooms thinking they were being terrifically clever by using code someone else had written.

A lot of that dried up as websites and software got faster at fixing bugs when they were found.

These days hacking is harder, and hence it’s usually the clever people doing it. Which sadly also means that unlike the old days when fixing the problem was a simple task, these days it’s much harder to find the flaw and repair it. It also means that with a lot more effort needed to find flaws, that the hacker invests a lot more work in making it a damn nuisance to repair.

In this situation, a series of template files were being changed to add the annoying adverts to the website, and if I deleted the files, within a minute or two, the entire folder would reappear. Renaming the folder and changing how the website architecture worked, to give me some breathing space to fix things properly lasted, oh, about 5 minutes.

It became apparent that the hacker had a script that was able to find template references in other files, then attack the templates, no matter where they were.

It also became apparent quickly on that there was a copy of the template files somewhere on the server that the hacker had made, so they could override any changes I was making.

So far, this looked like an annoying, but effective “bot” attack.

What was puzzling was that changing the entire website architecture didn’t stop the attack. If they were pinging a file maliciously inserted into the website to launch the attacks, then renaming every folder on the server should have stopped them, at least until they found the new folder name.

In the end I decided to turn the website off by amending a few core files to point everything to a maintenance page, and got to work migrating the website to a new server, which would hopefully be clean enough to get the site working again.

Then it got interesting

Those core files, edited in a way that was obvious to a human, but not a computer, were being edited back again a while later. Even changing file names subtly saw them being renamed back a while later.

While the attack looked like a computer bot, and the speed of its responses suggested as much, this new development was clearly the actions of a human who had access to the website and was now fighting my attempts to turn the website off.

After all, if I did that then they’d lose a day or two of fraudulent income.

What had been me fighting a bit of computer code maliciously inserted into the website and causing havoc was now a fight between two humans — one seemingly in Albania and the other here in London. My moves to shut-down the website being undone a hour or so later by my adversary.

This made the fight more personal. It’s one thing to be dealing with a computer code being mean, but when you can see that an actual human is editing your website as you watch, then it’s a more tangible experience. And a much more annoying one.

While working to migrate all the clean code to a new webserver, I continued to work on trying to fix the old, and finally, it looked like a change I made resulted in silence from the hacker. This was interesting, changing a folder which was in a location that wasn’t accessible externally to be unavailable to the server as well by altering its permissions, blocked the hacker.

Watching for a few hours, and calm reigned

I am wary of declaring victory, but after a couple of hours of watching, and no more attacks, then it was time to tentatively decide to switch the website on again.

The folder was in a non-web addressable location, which is to say, the webserver could access it, but no one else could. This is a modest security precaution, as it’s where uploaded images for the events listing would be saved. Making sure that only images could be uploaded, and putting them somewhere that makes it harder for a malicious payload to be triggered is a routine thing to do.

Somehow, the hacker had uploaded a malicious payload disguised as an image to the folder, and it was merrily playing havoc with the website.

But most frustrating is knowing that it’s my own fault that the hacker got into the website. I wrote that bit of code, and took all the precautions I know to take, but it was evidently not good enough.

I shouldn’t beat myself up too much, as the biggest cleverest of websites get hacked as well, and I am never going to be able to stop a determined attack — but it still rankles that at some level, it was my code that created the vulnerability.

A little over a million lines of code are on this website, and it takes just one glitch in there to allow a hacker inside.

Repairing a hack like this takes a lot of effort, just to find the cause, let alone find a fix.

I was recently involved in a big problem at work, and described as “cold as ice” in how I responded… which is to say here is a problem, let’s find a solution.

It’s how I look externally, while inside my stomach is churning like mad, because when a website is being hacked, as much as you really want to, running around wailing and crying wont change the result. Be methodically emotionless, fix the problem and don’t let anyone see your stomach making you feel sick and the massive levels of stress you pile on yourself.

When it’s fixed (hopefully), then the stresses boil over and you curl up in a corner and cry.

Fortunately websites being hacked is a decreasing problem

Just a few years ago, I doubt I went more than six months without a serious hack on a website, such were the quantity of holes in code back then. Over the years, websites have got better at finding exploits and fixing them.

I am still not entirely sure how to enable the functionality I want, without opening that door once more to a risk of the website being hacked again, so for a while you’ll see a lot of missing images on the website while I ponder more calmly how to deploy a long-term solution.

I am still wary of declaring this attack to be over, but if it is indeed finished, then with luck, it’ll be another couple of years before the website is hacked again.

But one day… it’ll happen again.

Also on ianVisits

Tagged with:

Whats's on in London: today or tomorrow or this weekend

18 comments on “When your website gets hacked
  1. Maurice Reed says:

    I wondered why so-called Google ads were right in the middle of the screen.

    Glad you got the problem fixed.

  2. Hi Ian

    This is not your fault!

    I have followed your site for many years and have found the event lisying and the blogs useful and informative. I am glad that you had the skills and fortitude to bring the site back.

    Would you consider publishing the details of the hackers?

    Very best wishes

    Bill

  3. Jo W says:

    A very interesting explanation,Ian,although I confess to not understanding most of the ‘tech’. * Glad you’ve got it sorted for now. It must feel like having a burglary. Please keep up your flow of information,if you can. We find it useful when considering walks in town etc.
    * I’m just getting past the turn it off turn it on stage,myself.

  4. Peter Jones says:

    Sorry to hear this. Good luck in the fight against the hacker. It’s a great site, so don’t give up

  5. Maurice Reed says:

    I think that all you can do is regularly back up all the web pages and database to your own pc. Then if the site gets hacked you can go into the hosting control panel and wipe the content, and hopefully the injected code, and then restore the content from your clean backup.

    • Ian Visits says:

      I have many back-ups, with snapshots taken twice per day.

      But a) which of them is the clean one, and b) restoring the site leaves the vulnerability unfixed, which makes it very difficult to know where the hole was in order to fix it.

  6. Malcolm Corbett says:

    Am reading this whilst on the phone to our bank to cancel a card used to book a BA flight through their website which was subsequently hacked. Even (especially) the big boys have sites with vulnerable code that gets regularly exploited.

    Well done for getting the site back so quickly!

  7. Jimbo says:

    As someone who runs a few websites, I feel your pain. After being hacked a number of times, I have taken to using disposable servers and offline copies of the source, I can then destroy a site and rebuild it quickly. Last time, I went through this loop 3 times before working out how they got in. Fortunately, I no longer have advertising which removes most of the attack vectors.

  8. Westville13 says:

    That’s really beastly. Well done for sticking to it (and sticking it to your hacker). And I do entirely understand your feelings – I find reading a Sherlock Holmes story or two helps me to calm down after dealing with a crisis caused by someone else’s malevolence.

  9. Greg T says:

    VERY WELL DONE in screwing with whomever was attempting to screw with you.
    Very annoying – another site I read regularly, a text-only blog by a fairly well-known author, with decent security precautions, still gets hit from time to time, causing greif all round.
    Hope it’s a long time to the next one.
    Oh & full marks for the cat pictures & comments – they are wonderful computer supervisors, aren’t they?

  10. Andrew Gwilt says:

    Screw the hackers who have to start causing misery to everyone on the internet. Including phishing spam & malicious email that contains viruses that will affect your computer and anything you stored on your computer. Luckily I got Bullguard anti-virus system and a external hard drive for storing photos & music without having to lose everything. But your website is amazing.

  11. JP says:

    Bravo ragazzo. These things are sent to try us.

  12. Howard says:

    backup backup backup backup

  13. Rich G says:

    Think the hack might be back. 🙁 Each page is currently showing to me a white-on-red floating “X” icon in the middle, linked to HTML div class=”modal-Anodaz” and adsbygoogle.

  14. chris says:

    Sorry to hear about this. It’s a great site and a fantastic resource to those of us who use it.
    It’s my first go-to website to look at what I might do every weekend.

  15. ADS says:

    ah – i was wondering why the icons had disappeared in the weekly emails !

    good like fighting this one – i hope you can declare victory soon.

  16. Endean0 says:

    Sorry to hear about the hack, B@st**ds! Did this result in the deletion of the Alleyways, Courtyards and passages site? This was my go to site and I miss it. Hope all continues to go well and thanks for such an interesting collection of info.

Leave a Reply

Your email address will not be published. Required fields are marked *

*