Dreamhost, the low-cost hosting provider and domain registrar earlier found database breach on its system back on January 20, which they later confirmed that were a series of attacks that results in some of their customers’ FTP passwords. And, on Sunday at 4 am PST, the company has been reporting outage problems, mainly as Web, SSH and FTP services were down for many of its virtual private servers, shared and dedicated machines which continued throughout the day.
DreamHost’s CEO, Simon Anderson stated, “We run Debian OS and have used autoupdates to ensure security packages are installed as soon as they are available. We’ve had some breakage in the past from this approach, but nothing major. However last night’s autoupdate went badly wrong, removing essential packages from dedicated, VPS and some shared servers. Our monitoring and support team flagged the issue fast, and we scrambled our admin, dev and NOC teams to reinstall the packages that had been removed by autoupdate, reboot servers, fix package dependencies, and test that individual services were live. Given the number of services affected, this took a long time to complete. Rest assured we had all hands working on the issue, but I know it was still a frustrating experience for customers.
To mitigate the risk of anything like this happening again, we’re immediately switching off autoupdates, and moving to a manual process where we’ll only push out Debian updates after significant testing. There’s always a balance to be struck between speed, efficiency, security and issue prevention, but this event has shown us that we need to take a different approach. Again, my apologies for the downtime experienced today. We’re acutely focused on adjusting our processes and systems to ensure we do a better job going forward.”
Stating on January 30, the company on its blog pointed out that, “There are still a few VPS guests we are still continuing to work on. An Apache restart is fixing most issues.” DreamHost plays host to thousands of small websites and personal blogs across the Web, and for many of them, it was a surprise to find their sites offline for most of the day.
(Image Source: Dreamhost logo, FTP featured image)
