Confession… this Sunday, I woke up to my cell phone ringing nearby my bed-side. The first thought that crosses your mind as your phone rings in the middle of the night is that either something really bad happened to someone close to you… or something really bad happened to a technology you run your business on. Either way, the event is emotionally charged, and no doubt your woes in either case are dramatic and ongoing.
Fortunately, my call this weekend was the latter. A cloud provider we use for integration between our billing system and our Customer Relationship Management (CRM) system ended up having a serious issue. The issue was so severe, it forced us to resort to a “in case of emergency, break glass” recovery plan. After nearly 24 hours of work on a Sunday, we came out of the event, unscathed apart from the sleep loss and hours a few of us had to work.
LESSON 1: Make sure their customer support culture matches your need.
So onto today’s topic, responsible Cloud Customer Management. My biggest irritation during Sunday’s event was the manner in which my provider treated me. Out of the gates, we were discussing the issue with their support teams and their support teams swore to the ends of the earth the problem was on our side. We went through a series of ridiculous steps to try and address the issue each of which was painful to implement, and anyone with understanding of the technology knew it was painfully obvious it wasn’t going to fix the issue. But in my case, the burden of proof to show there was an issue with their product rested on my shoulders. It wasn’t enough to say that there was a problem… I had to prove the problem was theirs.
This gets to the heart of evaluating a cloud vendor. When selecting their product, do some work with their customer support teams and really analyze if they are a customer support organization or a technical support organization. The two are very different:
– A customer support oriented organization prioritizes quality of interactions and sympathy with customers issues. Most organizations that are successful with this approach believe as soon as the phone rings that their organization is responsible for fixing the issue, no matter what the cause was.
– A technical support oriented organization prioritizes depth of knowledge and the ability to communicate around deep details over the customer support organization. Most organizations that are successful with this approach have deep technical resources answering the phone and in most cases can answer nearly any question without escalation.
In my case, I ran head-strong into a technical support organization and one the slammed the door on my face. It’s unfortunate to run into that challenge especially as it pertains to the promise of cloud. Cloud providers should realize that business are using their services in a manner that is outside their span of control where at the end of the day, any issue that needs addressing must encourage a cooperative problem solving environment where the customer is a trusted member in the problem solving environment. Alas, my situation was frustrating, be sure you make sure your cloud providers are helping you to the right degree.
LESSON 2: Cloud providers need to maintain their own equipment ownership.
Beyond support though, the resolution of my issue came through a networking change that had to occur. Based upon the resolution, my assumption is that it was at the firewall level as they had to open up some new IP addresses for the fix. Herein lies my second recommendation when evaluating vendors… make sure the vendor has the span of control of their environment. In my case, if any of you said that I had to open up an IP address to our Cloud Systems whether it be our XML servers, IT.Shavlik.com or VMware Go, it’d take me all of about 15 minutes to document the change (we’re ITIL at a core) and then another 15 minutes to get the change into our firewall. I have a team of three people who can make the change in the dark as they are familiar with our equipment and rules. In my vendor’s case, the change had to be escalated through a third party vendor (a big one at that!). This inevitably caused massive delays in my fix… they had to escalate it, and get a network engineer who had presumably no familiarity with the setup to make the change. What should have been a fast and safe emergency fix was littered with time for them to learn the system, document the change and get it pushed out. Approximately four hours later, the issue was fixed and I had access again.
Cloud providers have to realize their infrastructure has to be managed like it is their own. Our datacenter is plugged into that need and as such, we can make prudent changes quickly and we own access to our core equipment to manage and maintain our environment. How a cloud provider would have to go to a service provider and escalate a change like that on a firewall is shocking to me. Oh well, lesson number two learned for me. For you all out there, make sure your providers have span-of-control ownership over their environments from firewalls, routers, switches, load balancers, servers, intrusion detection systems (IDS) on back. It’s the only way they can truly control the circumstance and get you out of it successfully.
LESSON 3: The backup plan… Always have one.
So by this point, shame on me. Two big strikes against my vendor. In good news, I had a backup plan. Make sure you have one. Having a cloud provider run into an issue is going to happen. There are things we can’t control, and we try hard, truly we do, to make sure we don’t run into issues. Inevitably, they happen though. When things go wrong, have a plan to get around it or work without the system for a bit. Good SaaS providers take minutes to get back online… other ones takes hours. Stay away from them… find the good ones.
Oh well, if nothing else from this blog entry, let me tell you, I have clarity now. It was a sobering reminder to me to make sure the right choices are made. For all of those out there that have had this happen, yes, even though I’ve been consuming the cloud for years, it still happens to even me. This weekend, I got lucky… it was an important element of my infrastructure, but not my backbone. I can’t imagine losing my CRM for that period of time… Oh well, that which doesn’t kill us makes us stronger… right?