The past week is one I'd rather forget at present, due to a range of technical issues with Techwatch, our flagship technology news site.
We were hit by a DDoS attack at the end of February, occurring periodically over 3 days, a matter that was reported in The Register due to the possibility of multiple tech sites being attacked for extortion purposes.
We applied some aggressive IP filtering - basically, allowed access only to known UK IP ranges - and that seemed to do the trick long enough to make the DDoS attackers get bored, and the rules were relaxed.
Then last Wednesday the attackers returned in force with an even bigger botnet, with traffic spiking at 1.5 GBps. The attacks continued three times per day, and while we were able to cope with the morning and afternoon attacks (a combination of a beefy server and aggressive IP rules again), the evening attacks were just too large and aggresive.
Because Techwatch is advertising driven, it was seriously threatening the site's income, and it was disappointing that the datacenter we are currently with was not able to stop the DDoS attacks.
So I looked around for DDoS protection services and eventually took up on a recommendation from BurstNet for Black Lotus, and ordered a new server with DDoS protection.
A couple of hours after paying for this, the original datacentre emailed that they'd found a way to successfully null route the DDoS attacks rather than domain.
Sod's law, but oh well - let's get the site moved anyway.
Then the real nightmare began.
Despite sharing the same basic software installations, different releases were used and after uploading all of the static files successfully, everything stalled badly when trying to move the site databases.
Then to cap it off, when trying to abort the move, a still unexplained error led to problems with the DNS - I thought TW was down most of this Monday, but luckily it was still online for some people - just not for me, even though I desperately needed to admin the site after the aborted move.
A few days of complete stress, but finally managed to get everything running smoothly again.
If I were my own boss, I would have sacked myself for the cock up.
In the meantime, still scheduling a move but taking far more car and time with planning, and will be working with a sys op friend directly to oversee the move.
And I'm finally just glad this week is almost over, that TW is still online, and I don't have to fight command line prompts today. :)
Comments