The Cloud Crisis: Thinking about the unthinkable… When clouds fail!

Confession… this Sunday, I woke up to my cell phone ringing nearby my bed-side.  The first thought that crosses your mind as your phone rings in the middle of the night is that either something really bad happened to someone close to you… or something really bad happened to a technology you run your business on.  Either way, the event is emotionally charged, and no doubt your woes in either case are dramatic and ongoing.

Fortunately, my call this weekend was the latter.  A cloud provider we use for integration between our billing system and our Customer Relationship Management (CRM) system ended up having a serious issue.  The issue was so severe, it forced us to resort to a “in case of emergency, break glass” recovery plan.  After nearly 24 hours of work on a Sunday, we came out of the event, unscathed apart from the sleep loss and hours a few of us had to work.

LESSON 1:  Make sure their customer support culture matches your need.

So onto today’s topic, responsible Cloud Customer Management.  My biggest irritation during Sunday’s event was the manner in which my provider treated me.  Out of the gates, we were discussing the issue with their support teams and their support teams swore to the ends of the earth the problem was on our side.  We went through a series of ridiculous steps to try and address the issue each of which was painful to implement, and anyone with understanding of the technology knew it was painfully obvious it wasn’t going to fix the issue.  But in my case, the burden of proof to show there was an issue with their product rested on my shoulders.  It wasn’t enough to say that there was a problem… I had to prove the problem was theirs.

This gets to the heart of evaluating a cloud vendor.  When selecting their product, do some work with their customer support teams and really analyze if they are a customer support organization or a technical support organization.  The two are very different:

–          A customer support oriented organization prioritizes quality of interactions and sympathy with customers issues.  Most organizations that are successful with this approach believe as soon as the phone rings that their organization is responsible for fixing the issue, no matter what the cause was.

–          A technical support oriented organization prioritizes depth of knowledge and the ability to communicate around deep details over the customer support organization.  Most organizations that are successful with this approach have deep technical resources answering the phone and in most cases can answer nearly any question without escalation.

In my case, I ran head-strong into a technical support organization and one the slammed the door on my face.  It’s unfortunate to run into that challenge especially as it pertains to the promise of cloud.  Cloud providers should realize that business are using their services in a manner that is outside their span of control where at the end of the day, any issue that needs addressing must encourage a cooperative problem solving environment where the customer is a trusted member in the problem solving environment.  Alas, my situation was frustrating, be sure you make sure your cloud providers are helping you to the right degree.

LESSON 2:  Cloud providers need to maintain their own equipment ownership.

Beyond support though, the resolution of my issue came through a networking change that had to occur.  Based upon the resolution, my assumption is that it was at the firewall level as they had to open up some new IP addresses for the fix.  Herein lies my second recommendation when evaluating vendors… make sure the vendor has the span of control of their environment.  In my case, if any of you said that I had to open up an IP address to our Cloud Systems whether it be our XML servers, IT.Shavlik.com or VMware Go, it’d take me all of about 15 minutes to document the change (we’re ITIL at a core) and then another 15 minutes to get the change into our firewall.  I have a team of three people who can make the change in the dark as they are familiar with our equipment and rules.  In my vendor’s case, the change had to be escalated through a third party vendor (a big one at that!).  This inevitably caused massive delays in my fix… they had to escalate it, and get a network engineer who had presumably no familiarity with the setup to make the change.  What should have been a fast and safe emergency fix was littered with time for them to learn the system, document the change and get it pushed out.  Approximately four hours later, the issue was fixed and I had access again.

Cloud providers have to realize their infrastructure has to be managed like it is their own.  Our datacenter is plugged into that need and as such, we can make prudent changes quickly and we own access to our core equipment to manage and maintain our environment.  How a cloud provider would have to go to a service provider and escalate a change like that on a firewall is shocking to me.  Oh well, lesson number two learned for me.  For you all out there, make sure your providers have span-of-control ownership over their environments from firewalls, routers, switches, load balancers, servers, intrusion detection systems (IDS) on back.  It’s the only way they can truly control the circumstance and get you out of it successfully.

LESSON 3:  The backup plan… Always have one.

So by this point, shame on me.  Two big strikes against my vendor.  In good news, I had a backup plan.  Make sure you have one.  Having a cloud provider run into an issue is going to happen.  There are things we can’t control, and we try hard, truly we do, to make sure we don’t run into issues.  Inevitably, they happen though.  When things go wrong, have a plan to get around it or work without the system for a bit.  Good SaaS providers take minutes to get back online… other ones takes hours.  Stay away from them… find the good ones.

Oh well, if nothing else from this blog entry, let me tell you, I have clarity now.  It was a sobering reminder to me to make sure the right choices are made.  For all of those out there that have had this happen, yes, even though I’ve been consuming the cloud for years, it still happens to even me.  This weekend, I got lucky… it was an important element of my infrastructure, but not my backbone.  I can’t imagine losing my CRM for that period of time… Oh well, that which doesn’t kill us makes us stronger… right?

Deploying Software from NetChk Protect

A question came up on the forum earlier this week regarding deploying software from NetChk Protect.  Specifically newer IE versions in this case.  Our newest SE wrote up a quick doc (his first official documentation excersise here at Shavlik) to walk them through the process and I posted it there.  I was looking at the Google Analytics for the blogs today and have seen that other people are searching for the same type of information so I wanted to share it on the Product Blog as well. 

Thanks goes out to Dan in the Shavlik SE Team for the doc!

Chris Goettl
Agile Product Owner

Free Antivirus Delivered by IT.Shavlik for Home users and Small Business

It’s a typical Thursday morning, and as I come into the office I open up my e-mailbag of customer questions for the week and begin to churn through them. Over the past few months a new pattern has emerged where people are discussion Anti-Virus (AV) and the issues that they run into with it. Common complaints such as “It slows down my computer” or “It’s eating up network bandwidth” typically top the scales with the “it’s so expensive” not too far behind. No matter how you slice it, the requests center around a few key things:

  • It’s expensive
  • It’s a performance hog
  • It’s a network challenge

Here at Shavlik, we’ve worked with our partner Sunbelt a long time for Enterprise Anti-virus. Their engine is market leading in quality and performance which mitigates many of those concerns, but for the home user, I have to admit, there needs to be something easier to use, more cost effective, and gets high quality results without a lot of interaction.

Well, how about FREE? No joke, Free. Here at Shavlik we always look for products that serve our enterprises but also our smaller businesses and home users. Our first offering in this camp is now emerging from our Shavlik Labs (http://labs.shavlik.com). We’ve teamed up IT.Shavlik with Immunet (http://www.immunet.com) to launch their Immunet Protect product which is a community based Anti-Virus product and deliver it free of charge.

How does it work? – It’s really simple… it relies on a community of users that run the product and using the combined collective intelligence of their data, is able to ensure protection across the community. – It’s fast, effective, and at the “free” cost definitely affordable.

For those of you interested, give it a download and let me know what you think. On the main application, go to “My Community” and click “Add People” and add me rob.juncker@shavlik.com and you’ll tap into the growing community of users that use the Immunet Protect Product in Partnership with IT.Shavlik.

Let us know your thoughts!  (Click this link to be directed straight to our download page)

Adobe Releases Critical Security Bulletin Out-of-Band, Again

Adobe has released their out-of-band security bulletin for Adobe Acrobat and Reader.  Adobe Security Bulletin APSB10-28 addresses two critical zero-day vulnerabilities.  You should look at patching your Adobe Acrobat and Reader products as soon as possible today.
 
Last year, Adobe announced a planned maintenance schedule to address security vulnerabilities.  The original plan called for a quarterly update of their products to coincide with Microsoft Patch Tuesday.  I for one applauded Adobe for standardizing their patching practices.  It is important for vendors to establish baselines and expectations for their customers.
 
Well, a year later and Adobe’s patching process has progressed but still has much more work to do.  On one hand, Adobe is changing their release cycle to address critical zero-day vulnerabilities.  On the other hand, it can be a bit challenging for administrators to adjust their patch maintenance windows for Adobe’s patch releases.
 
Taking a look over the past year, you can see that it is rare for Adobe to stick to their planned release cadence:
 
APSB-1002: Acrobat/Reader 9.3.0, Released: 1/12/2010, Patch Day/On Schedule, 9 vulnerabilities fixed
APSB-1007: Acrobat/Reader 9.3.1, Released: 2/15/2010, Out-of-Band, 2 vulnerabilities fixed
APSB-1009: Acrobat/Reader 9.3.2, Released: 4/13/2010, Patch Day/On Schedule, 15 vulnerabilities fixed
APSB-1015: Acrobat/Reader 9.3.3, Released: 6/29/2010, Out-of-Band, 17 vulnerabilities fixed
APSB-1017: Acrobat/Reader 9.3.4, Released: 8/19/2010, Out-of-Band, 2 vulnerabilities fixed
APSB-1021: Acrobat/Reader 9.4.0, Released: 10/5/2010, Out-of-Band, 23 vulnerabilities fixed
APSB-1028: Acrobat/Reader 9.1.0, Released: 11/16/2010, Out-of-Band, 2 vulnerabilities fixed
 
This is not to say Adobe isn’t getting better at their security patch cycle.  There are many non-Microsoft vendors who struggle with what Adobe has been getting right:

  • Being extremely transparent in their process by disclosing a lot of information regarding their process, (just check out the history on their blog)
  • Gives advanced notification ranging from days to weeks for administrators to plan ahead instead of surprising their customers
  • Sticking to the patch release estimation dates and releasing patches

As Adobe works on their process more, I am excited to see what additional tweaks they make.  Maybe they will increase their release cycle to more than quarterly considering the number shown this year.
 
– Jason Miller

Feature Requests

Hello from Minnesota!

We just got our first taste of winter here.  We went from a green to a white landscape this weekend.  Half a foot of snow and 400 fender benders and spin outs later… but that is not what I am here to talk about.

Lets talk Feature Requests!

As the Agile Product Owner it is my job to try and ensure that you get the best experience out of our products as possible.  New features, enhancements to existing features, and bug fixes all cross my desk and get my attention to some degree.  I have worked in our Support Team as well as the Sales Engineers.  I also conducted the Shavlik Classroom Training for the past couple of years, so I have a pretty good idea of what our customers need and want in general.

The problem is that I cannot hear from everyone face to face or over the phone.  I browse the forums as well and respond to customers out there, but it is hard to understand the requests coming in when I have no facts on the requester.  The forums are somewhat anonymous of any details that could really help me to understand the person who has the problem or is requesting a feature.  How big are they?  What type of company?  Servers or workstations or both?  Some of this information is important to understand when taking in a request.

I urge anyone with a request to use our feature request site to submit enhancements requests, new feature requests, behavioural changes, etc.  The more people we have requesting the better we can understand where our time is best spent.

Just to give you an idea of how effective this system is I have looked at our feature requests for NetChk Protect pre 7.6.  There were over 500 open feature requests submitted by customers or by Shavlik Support, SEs, Devs, or Sales Reps on behalf of our customers.  With the release of NetChk Protect 7.6 we implemented 56 of these requests.

We are still working on verifying what requests will be closed out in our 7.8 release in Q1 of 2011, but currently it is looking like 40+ requests are currently scheduled to be closed out.

Keep the requests coming.  They help us to understand your needs.

Chris Goettl
Agile Product Owner

Adobe Releasing Critical Security Bulletin Out-Of-Band

Adobe announced today they will be releasing their quarterly Security Bulletin out-of-band for the Adobe Reader and Acrobat programs.  This Security Bulletin will address two critical issues.

First, the Security Bulletin will address Security Advisory APSB10-26.  This Security Advisory pertains to a zero-day vulnerability for Adobe Reader and Acrobat.  This vulnerability for Adobe Flash has already been addressed with a patch.   The patch will also address a potential issue in Adobe Reader that was publicly disclosed.

Adobe is planning on releasing a Security Bulletin next Tuesday, November 16, 2010 to cover both of these vulnerabilities.  There has been a lot of news lately regarding both of the vulnerabilities, so you should look to patch Acrobat and Reader as soon as the bulletin is released.

– Jason Miller

NetChk Protect 7.6 Hotfix Available

This hotfix is available for download through the forum currently.  See the list of resolved issues below and go to the Shavlik Forum and login to download the hotfix.

This hotfix will release as a deployable patch in NetChk Protect 7.6 in an upcoming XML release.  It was not included in the November Patch Tuesday release.

Resolved issues:

• Resolved an issue where the ‘Item History’ report and the ‘Seat License Status’ report were not available in the report gallery.
• Scan fails to import properly due to file locking issue on the scan information importer.
• Resolved an issue where deployments would fail when deploying patches that could not be downloaded.
• Resolved an issue where scans would hang on importing new definitions: Error – ‘Object synchronization method was called from an unsynchronized block of code.’
• Resolved an issue where deployments fail when installing or upgrading to 7.6 on a disconnected network.

Chris Goettl
Agile Product Owner
Shavlik Technologies

November 2010 Patch Tuesday Overview

In the November 2010 edition of Patch Tuesday, Microsoft has released 3 new bulletins addressing 11 vulnerabilities.  Only one of these vulnerabilities, addressed in Security Bulletin MS10-087, is publicly known.  This zero-day vulnerability was previously discussed in Microsoft Security Advisory 2269637.  Microsoft typically follows a large patch release month with a lighter patch month, so a release this small is not completely unexpected. 
 
For the November Patch Tuesday, there is one bulletin IT administrators should focus on first and foremost.  Security Bulletin MS10-087 affects all supported versions of Microsoft Office.  This bulletin addresses 5 vulnerabilities and is rated as critical.  If a maliciously crafted RFT formatted document is previewed with Microsoft Office, an attacker can gain remote code execution.  Although this vulnerability is not publicly known, we are likely to see exploit attempts against this vulnerability in the near future.  RTF document attachments are typically not blocked and used as a common shared file format like PDF Files.
 
The second bulletin released this Patch Tuesday cycle, MS10-088, affects older versions of Microsoft PowerPoint and PowerPoint Viewer addressing 2 vulnerabilities.  Opening a malicious PowerPoint document can lead to remote code execution.
 
The last bulletin Microsoft has released, MS10-089, addresses 4 vulnerabilities in their Microsoft Forefront Unified Access Gateway product.  With these vulnerabilities, an attacker can gain Elevation of Privilege with a successful exploit.  The update for this product is currently only available for manual download through the Microsoft Download Center.  Administrators should assess their networks and identify any systems with UAG installed and manually apply the patch as it will not be automatically applied with Windows Update.  On a good note, most companies will not have many systems with this software program installed.  However, as this is a high profile product, administrators should know if this program exists and the machine it is installed on.
 
It is also important to note MS10-089 also applies to Intelligent Application Gateway (IAG).  A patch is not being supplied publicly to cover the vulnerabilities as Microsoft is asking customers using this product to contact their OEM for a fix.  IAG is typically deployed to networks from authorized distributors.  Any company currently running IAG on their network should review the Microsoft Security Bulletin for more information.

– Jason Miller

AUTOMATIC Home Office and Small Business Patching

Every few weeks, we get an email from a small business or home user with a small slug of machines they want to make sure are patched. It awesome to see everyone care about patching, but implementing a routine patching plan is sometimes a tough thing to do. All of the emails we see always say something to the effect of:

  1. Patching isn’t fun, but I know it needs to get done.
  2. How do I do it in a non-intrusive way?

 
After thinking about this issue for quite some time, we now have a great answer. On Tuesday this week, we’ll be announcing the release of IT.Shavlik’s Site Manager. Site Manager is designed for the Small office IT administrator or home user who wants to patch, but not have to think about logging into the IT.Shavlik site to scan and deploy patches.  — We automate the entire process for you using your existing IT.Shavlik credentials.

The concept of Site Manager is simple; set it, and forget it. We want to make it so you can download the patching app once, set it up on one computer in your office, and then Site Manager takes over. In Site Manager, we’ll find the machines you can scan on your network, you can set a time for those machines to be scanned, and when that time is crossed, we’ll take care of scanning all the selected machines, and patch them for you automatically. — If you want to view the results, they can either be retrieved online using your IT.Shavlik account, or alternatively delivered to you in a forthcoming release.

No longer do small businesses have to think about patching. — You’re a few clicks away from making the process automated.  Get started by registering via the  “Join IT Now” button on IT.Shavlik at http://it.shavlik.com and click “Forget IT”

New Version Of Adobe Flash Available

Adobe released an update to Adobe Flash today for both version 9.x and 10.x.  Adobe Security bulletin APSB10-26 addresses 18 vulnerabilities.  The most notable vulnerability fixed is CVE-2010-3654.  This vulnerability was detailed in the Adobe Security Advisory APSA10-05.

Adobe is stating there are still no attacks against Adobe Flash for the zero day exploit.  All exploits to date have been against Adobe Acrobat and Reader.  Although, you should look to patch any Adobe Flash installations as soon as possible.

The next round of updates for vulnerability CVE-2010-3654 is as follows:

November 9, 2010:  Flash Player 10 for Android

Week of November 15, 2010: Adobe Reader/Acrobat

– Jason Miller