Sources close to the company have confirmed to The Enterprise Software Observer that recent outage problems at Salesforce.com are being caused by the on-demand company pushing its Oracle database to its limits. This raises a number of questions about the company’s future direction that investors are likely ask at tomorrow’s 5:00 pm eastern quarterly earning conference call.
Since December the company has experienced a number of outages, the most serious occurring January 30th when some parts of the system were down for many hours.
At the Root of the Problem
In a letter to Salesforce customers co-founder and Executive Vice President of Technology Parker Harris wrote:
“Some of our customers experienced intermittent service availability which was caused by a problem in our database cluster. This issue required Salesforce.com to restart each database instance in the cluster, resulting in a pause in service.”
Salesforce pushes its Oracle database pretty hard. At the moment the company claims to host 351,000 subscribers from 18,700 companies which makes the size of its database gargantuan. Since Salesforce also operates worldwide the database is not only enormous, it must be available 24 hours a day 7 days a week. Since few organizations have more than 350,000 people the number of users is also extraordinary large. The result is millions and millions of queries and other database transactions a day.
By way of a rough comparison, RightNow Technologies (Nasdaq: RNOW) which is about one-third Salesforce’s size averages over 1 billon queries per month. This transaction volume makes RightNow one of MySQL’s three largest implementations in the world. At the recent Enterprise Software Summit, CEO Greg Gianforte told delegates that his data base administrators were on the phone with the open source developers “almost daily”.
In response to the outage, irate users launched their own web site called gripeforce (gripeforce.blogspot.com) to complain about the outages.
“We've had service go down at *least* every Monday since Jan 9.”
“We use salesforce.com and have invested a lot of time and money into this system. Although the features and functionality is great, we're not very pleased with the reliability.”
“SFDC is down, yet again... that's twice in the same day. Today's situation appeared to be resolved when all of the sudden... Down again!”
The publicity savvy firm has been forced to make public acknowledgement of the problem and launch a site so interested parties could monitor the situation (trust.salesforce.com). At a recent investor conference Chairman and co-founder Marc Benioff disclosed:
“In the month of December, our availability fell to a 99.5% uptime. But that is still not where we want to be.”
By almost any measure 99.5% is an impressive statistic, but with 44,640 minutes in the month it translates into almost 4 hours of downtime. For the full year, being available 99.5% of the time means Salesforce is not available for an entire work week. Since December 2005 the site has been less, not more, available.
This has others in the industry worried. When asked what most concerned him at last week’s Enterprise Software Summit, RightNow founder and CEO Greg Gianforte responded that he hoped Salesforce.com would fix its outage problems quickly. Given the intense competition between the two companies and the fact that RightNow stands to directly benefit from customer defections, Gianforte’s expression of support for Salesforce surprised even seasoned industry figures at the conference.
“Given Salesforce’s high profile in the market,” said Gianforte “Continued outages could set the entire software as a service industry back”.
Ironically Salesforce’s biggest and most bitter competitor, Oracle Corporation - the company which should gain the most customers from continued outages – also has the most to lose. Some of Oracle’s best engineering talent is working overtime to help Salesforce address its availability issues.
The Industry’s Fair Haired Child
Until very recently Salesforce was a company that could do no wrong. Starting in 1999, the company began providing its products directly to end users over the internet rather than through corporate IT. Thanks to an innovative business model and brilliant marketing, the company’s number of subscribers close to doubled year after year.
As that number increased Salesforce came to occupy the pole position in the software industry as the most emulated and envied company in the space. As the foremost champion of the software as a service or on-demand model, the company became the darling of investors who in the last five months doubled its share price from $19.32 on September 1st to $ 42.56 on January 31st. In the 15 trading days since the January 30 outage the stock price has dropped 16%.
This very growth began to create its own set of problems. Beginning on December 20th customers and prospects attempting to log on began to receive a rather ominous message:
In an industry that has seen many strange bedfellows, Salesforce and Oracle working shoulder to shoulder to fix and ultimately advance the Oracle 10g database is one of the strangest. With the acquisition of first PeopleSoft and then Siebel Systems, Oracle jumped from being a minor player in the CRM market to becoming the biggest in terms of both revenue and seats deployed.
In public, the two companies have squared off directly against one another. During an autumn conference call with investors Marc Benioff dismissed Oracle as a competitive threat saying:
"They have created a culture of acquisition instead of innovation. It's a lot easier just to write big checks than it is to innovate…. Now, the same thing that happened to PeopleSoft will happen to Siebel; it will die."
Not surprisingly, Larry Ellison has been no less vocal. At Oracle OpenWorld in September, he said to reporters:
"We want to go after Salesforce.com as much as we can. I'm an investor in Salesforce, and I want to see my investment go to zero."
To understand why Salesforce picked Oracle to provide a key part of its infrastructure, it is helpful to know the history of the two companies. Salesforce founder Marc Benioff was Larry Ellison’s long time protégé at Oracle. When he launched Salesforce in 1999, Ellison invested $1 million of his own money and served as Chairman of the board. It was only later the two men had a falling out.
At the time of Salesforce’s launch Oracle’s CRM offering was anemic, but its database offering was very strong at the time and continues to set records in performance benchmarks so it was a natural choice for the young startup company. In a recent Winter Corporation survey Oracle was found to run the world's largest commercial database with over 100 trillion byes of data.
Source: Oracle Corporation (1)
In July 2005 a joint press release, the two companies renewed their commitment to one another and announced that Salesforce had selected the 10g Database and Oracle Grid Computing to “help drive salesforce.com's success in delivering reliable, scalable application services around the clock.” In the release Benioff was quoted as saying:
"We are proud to run our systems, and our company on the Oracle platform."
There may be other Oracle databases that are larger, other Oracle databases that must be available more often, other Oracle databases that have a larger number of users, but with their Salesforce’s need for high performance in all three areas has it pushing the envelope.
Again ironically, overcoming the problems Salesforce is encountering is helping Oracle build a better database. Not only will this benefit all of Oracle’s high performance customers, it will also help it own CRM On-Demand business when it eventually converts from IBM’s DB2 database.
The Future Direction of the Industry
On Wednesday’s February 22nd quarterly conference call with investors, Saleforce executives are likely to be asked a lot of hard questions.
- What impact are the outages likely to have on Salesforce’s rate of customer churn?
- Was it wise to be so heavily dependent on its principle competitor for key technology?
- Would an open source solution like mySQL serve Salesforce better?
- What are the costs of converting likely to be?
- Can current technology sustain the company’s 100% year-over-year growth?
The answers to these questions are likely to set the future direction of the software as a service business for years to come.
 Benchmark conducted on a single Sun Fire E25K server with 72 UltraSPARC IV+ 1.5 GHz processors and Sun's Solaris™ 10 Operating System (OS),and Oracle Database 10g Release 2.