Simon Says... (Written by CEO Simon Anderson!) As Brett has said, January was not a good month for our level of service. And on behalf of all of us here at DreamHost, I want to apologize and let you know what the issues were, and what we're doing to fix them. First up, security. We're now a mid-sized global web hosting company with over 300,000 customers and 1.2 million domains under management. That means we get our fair share of hack attempts from around the world. Sometimes it's just kids looking to prove themselves, other times it's more malicious than that - people wanting to take down customer blogs or websites because they don't agree with the opinions expressed on them; people wanting to steal other folks domains, that kind of thing. We put a lot of people and dollars into our security software and systems and protect our customers from these attempts every day. But in January a hacker got through our defenses and we took the rapid action of resetting all customers' FTP/shell passwords to prevent any malicious activity. In the process we identified a whole slate of improvements to our security systems, software and data management that we've already been implementing. Security is one of those issues that we can't give too much detail on, but rest assured we're on it and have upped our game substantially so that this kind of event is prevented in future. And as outlined in the newsletter below, we're also asking you to participate by using strong passwords and other best practices. The second issue in January was related to our network. Given our unlimited bandwidth policy for shared, VPS, and dedicated customers, we have a network that has been designed for speed using tried and tested network equipment. However, the complexity of our routing and switching systems has increased as our global customer base has grown. As a result, we experienced some unplanned network downtime in January due to network configuration errors, plus the testing of some new distributed denial of service (DDoS) equipment. We've already been in the process of a focused effort since late last year to upgrade and streamline our network for the next 5 years of customer needs. But these issues have accelerated our planned efforts in this area. We'll be investing over 1 million dollars in network upgrades in 2012, including a new US east coast data center to better serve our European and east coast customers. Third, we had an issue with software updates in January that resulted in some service downtime. The issue arose because of the way we have approached security updates for the operating system that runs on most of our servers. We've typically tested security updates to ensure they install correctly, but our testing of whether they break anything has typically been "light" due to the fact that we haven't seen any such issues in the past. Well this has proved to be a flawed approach, because one such update in January went badly wrong, and automatically removed some critical software on production servers, causing downtime and an understandable amount of customer angst. We've immediately changed our software update process to fully test all software going into production, and we'll be acutely focused going on ensuring that any necessary software upgrades are performed efficiently and quickly in planned maintenance windows going forward, with minimal downtime. Finally, a quick word about our team. We have very dedicated, smart and super friendly people on our support, systems administration, development, engineering and data center operations teams. They work long hours and are absolutely passionate about delivering you an awesome and flexible web hosting service at a great price. When there are service issues like we've seen in the last month, they pull out all the stops â?? nights, weekends, 24/7 â?? to fix the issues and plan out changes to give you a better experience for you in future. I'm fully committed to ensuring we deliver a reliable, flexible and useful service for you going forward. And I believe we're adapting and focusing on the right priorities and investments to continue to deliver a great hosting service and experience. Best, Simon @DreamHostSimon -- Duaine Hechler Piano, Player Piano, Pump Organ Tuning, Servicing& Rebuilding Reed Organ Society Member Florissant, MO 63034 (314) 838-5587 dahechler at att.net www.hechlerpianoandorgan.com -- Home& Business user of Linux - 11 years
This PTG archive page provided courtesy of Moy Piano Service, LLC