kgale
April 22, 2011
11:31am
As you may have heard, Drupal Gardens experienced an extended outage beginning on Thursday, April 21st due to a widely publicized outage at Amazon Web Services. Some customers continue to be impacted. See further details on the outage itself in this post, http://www.drupalgardens.com/content/drupal-gardens-outage-due-amazon-we....
The purpose of this message is to acknowledge the poor job we've done in communicating with our Gardens customers through the extended outage. Our team was caught unprepared to manage crisis communications for our Drupal Gardens customers and we are focusing attention on how we change our process and systems to do far better going forward. We did a better job with customers of our other cloud-based services, but even so, we found much room for improvement across the board. I assure you that we fully grasp the magnitude of this event and we recognize the task ahead of us to fix it and regain your confidence. Know that while we won't always get it right, we will certainly always make it right.
For starters, we are extending subscriptions for all annual subscribers by one month. For those on monthly plans, your next invoice will be $0.00. Drupal Gardens lives on Amazon Web Services because we chose to build our infrastructure on top of the rich set of services they offer. While it is true that Amazon suffered a massive outage impacting us and many others such as we never imagined possible, we also understand most of you either don't know or don't care about those details of our service. And nor should you. What you sign up for is an awesome experience with building and running your site and it is our job to take care of the plumbing. We've done a tremendous amount of engineering to create the Drupal Gardens experience and we are disappointed to have our customers impacted in this way no matter what the reason. As a company, we are looking very hard at what we can do to further insulate or mitigate the impact of infrastructure issues on our customers.
On the communications side, for a company working to be transparent and collaborative we did a lousy job and we know it. We could have posted to the forums or tweeted far earlier than we did and it is clear our process and communication outlets for Drupal Gardens customers are lacking. We need a status board on separate infrastructure, an RSS feed, e-mail alerts, online postings and processes that let you know something is up as soon as we know it. Drupal makes it easy to do these things, and though we have several of these in place for other services, it is a bit embarrassing that we don't yet have them in place for Drupal Gardens.
The good news is Drupal makes communication easy, so it won't take us long to make improvements. We are formulating a plan which we will post back here on Monday for feedback. Again, I apologize and promise you will see an effective change shortly.
Kent Gale
Senior Director Client Advisory Team | Acquia
Twitter: @kentgale
Status: Unresolved
Comments
1:39pm
Thanks, Kent.
Your apologies are appreciated, and the "we're sorry" gift of a free month of hosting shows that your company cares. Even though it was Amazon that was responsible, it is also refreshing that DG didn't just point the finger and shrug it off. Mistakes happen, ans sometimes we are all caught off guard, but it is your response to this kind of thing that shows your customers that you're committed to making things better in the long run -- and that counts. :)
-Kendall
www.kendallsdesign.com
1:06pm
In days where most would deflect or delay, your response is appreciated. Web is up, some delays still being experienced...please keep us posted....
Guidance Aviation
Airplane and Helicopter Flight Training
1:15pm
Apology accepted. Thanks for working your buns off for us DG.
CK
1:24pm
1:27pm
Your apology is accepted. Thanks for the update and information. Please keep us posted regarding the implementation of better communications. It does amaze me that with all the knowledge and experience at both Drupal Gardens and Amazon that such an event could result in such as extended downtime (at least for my site). Can you please post details about back ups and redundancies that are in place now and what this event might change.
1:27pm
1:46pm
I also appreciate your honesty and constructive communication.
We are new at DG and we are very enthusiastic, and every day I get some new amazing impacts about the power of DG. But the last days were really horrable and I hope we can launch our website this weekend, because the world is waiting for the magic of play :-)
Keep up the good work guys!
Greetings from Austria, Europe
Alexander
Webmaster of AngeliqueFelix.com - the magic of play.
AngeliqueFelix.com
The Magic of Play
3:19pm
Thank you for getting on this quickly and the attention to the community is appreciated. We chose DG because of the low technical requirement and automation of so many web-maintenance tasks. As a new(er) service, I think the customer that I've placed on DG is going to be more indicative of your client base - low technical skills, low technical needs, and quick and clear responses to issues.
Keep up the good work.
3:36pm
At least your learning from your mistakes. I am one of the clients whose client site is down http://www.fivevalleysrestoration.com and I appreciate that you'll be giving my accounts one free month of hosting to make this outage experience right for me and my clients. Please also update us as soon as things are fixed.
3:55pm
My sites are all still down. I can't jump into the apology love fest just yet. The free month is a nice gesture but to be honest I would really like to see the custom url extended for people who configured before the March deadline. I believe I read online that as long as they were configured before the deadline that it would not be taken away from us when the product exited beta?
I have a few real simple low traffic sites that I did under the beta program that now require me to buy an annual package or lose my custom urls. Not cool since I can't really bill the customers for it now.
I have one customer whom I just sent a notice letting them know we had to upgrade their hosting plan etc etc the day before this happened. They were not thrilled but seemed to be dealing with it. Then they called me at 6am on the cloud implosion day and are not really happy.
10:54pm
Though my site is still down, and just as we launched a major online initiative, I have to say I am impressed with this email. Telling someone of what had happened, they exclaimed that I should complain and demand "x". I responded that given the size of Drupal and the nature of the incident, that I would be wasting my time. But then I get this email. Drupal - this is the first time in a long while that I actually feel that a major corporation that could easily survive without my business took how I might feel into consideration.
Thank you. And while I do hope my website is back online soon, I will also be telling all those I chat to about the excellent customer serive I get from Drupal. Thank you!
12:42am
We believe all websites are back up at this point. Please let us know if you have a site that is not up and functioning properly.
2:59am
Hi, kgale, I would recommend everyone to check their pages, I lost content on my page and that is serious. I posted the issue on this post.
http://www.drupalgardens.com/content/i-lost-content-my-page-and-i-lost-my-last-4-blog-entries
8:00am
^Regarding my last post, I can say now that the missing content is now restored in my page.
1:30pm
we are back again and also the missed content from yesterday is back. thx guys!
AngeliqueFelix.com
The Magic of Play
8:40pm
8:45pm
Hi Benno,
I will start looking into this, but I think it would best be handled in its own thread.
George Cassie
Gardens Advisory Team
4:15am
Thank you for keeping us updated on the progress. We know mistakes happen and the steps you have taken in response show your proffesionalism. Despite the problems, we are assured that we have made the right choice in trusting our website with DG.
Greetings from Taiwan,
Marco
9:33pm
Hello folks,
In the wake of last week's outage, we have been planning how to more effectively communicate with our Drupal Gardens members. Here's where we stand:
We'll keep you updated on the progress of these items as the pieces fall into place. If you have any questions or comments, please let us know.