NoSQL ... No Kidding

Cloud computing puts a lot of demands over databases. While traditional databases are very good in what they do, they may not be the best fit for cloud computing.

Consider this scenario. Your cloud offering gains popularity and results in hundreds/thousands of new users overnight. You do want to make sure that your cloud offering can support those users and dramatic increase in demand. You might have started with single in house server. Now you must scale well by making use of multiple nodes. Adding new hardware and distributing the load is not trivial process. It also has cost implications. Research in this area suggests that it comes down to fulfilling the 3 important characteristics

* Consistency
This means that each client must always has the same view of the data.

* Availability
This means that all clients must be always read and write.

* Partition tolerance
This means that the system must work well across physical network partitions.

While it is desirable that DBMS should address all the three requirements, it is not possible. The underlying constrains and issues are well captured and described in what is popularly referred to as CAPS theorem.

So, depending on which characteristics are important, more desirable solutions are available. Nathan Hurst has done great job in classifying them.



The solutions are now collectively referred as NoSQL databases (sometimes distributed databases). The NoSQL databses are getting increasing popular to an extent that somebody who is starting out with new cloud offering should think about the value they bring in from the inception. I will try to discuss some of the popular distributed databases in subsequent post.

Deploying at the speed of light

In SaaS environment, it is very important to make sure the service is "always" available. The downtimes in the service impacts wide range of users especially if they are coming from a different timezone. Downtime between 3 AM to 6 AM in US ET may not seem disruptive to anybody in US. However for somebody who is working in Asia or Australia this might eat into major chunk of one's productive hours.

Anyway, the point I am at is in order to keep the service up for the most of the time, it is important to minimize the downtimes. Downtimes are usually needed for emergency patches or software refreshes. That is usually the only way to roll out new features and services. In SaaS environment, it might be daunting task to rollout new upgrade to a number of services. The downtime might then range from few minutes to hours depending on complexity of the environment.

The presentation by Larry Gadea demonstrates some of the challenges that are involved in deploying new builds to servers.

For starters, in case of Twitter,the deployment time was around 40 minutes for a server. With a number of innovative ideas the team was able to reduce it few seconds.

Here is the first general issue. If the farm is to be updated by copying the updates from one central server, usually the central server is the main bottleneck.

One way to solve the issue would be to use tree distribution. However it runs into a problem if one or more nodes in the tree fail resulting in servers that are not updated.


Twitter was able to use an approach that involved using techniques below



The new approach integrated with existing deployment app, Capistrano, resulted in getting the deployment time by about 99%. You can see it here

Twitter's Countdown to OAuth

OAuth 1.0 has been around for some time now. The adoption of OAuth has grown tremendously in last couple of years. The standard is evolving and there is significant interest in OAuth 2.0.

While companies are enthusiastically embracing OAuth, they are yet to dump their existing api authorization framework (example: salesforce.com). One can argue that drastic change like discontinuing support for non-OAuth based auth framework will impact hundreds (or thousands) of third party apps and might hurt the company. It is interesting though to understand what Twitter has announced. Twitter is doing courageous thing by publicaly announcing they are dumping BasicAuth in favor of OAuth. In other words, if you are to call Twitter APIs, you must use OAuth. The efforts needs to be appreciated. This is coming from a company which receives 75% of requests in the form APIs.

So, what really prompted Twitter to make that switch?

First of all, one can see that it is probably in Twitter's DNA to do the things right way. Twitter has a prime example of a company that is using Open Standards the way they are to be used. This has been obvious the way they did 'Sign in With Twitter' via OAuth rather than proprietary methods employed by Facebook in 'Facebook Connect'.

Now, let's dive into some technical aspects. In BasicAuth, user's credentials need to be sent with the API calls and it makes itself vulnerable to Man-in-the-Middle attacks. While this can be addressed by using SSL (secure transport layer), unless you do it right, it is problematic.

The next issue is around password management. Using BasicAuth means the API caller is forced to send the user's userid and password. That means users do need to share their passwords with API caller (external app). How many of us use the same password for multiple site? Would you be willing to share your password with third party app that you are not much familiar with? What happens if the api caller's website is breached? When you change your password, what happens to integration?....Once you start thinking on those lines, you will quickly realize that it is always problematic to share the passwords and trust different parties to manage the secrets. These slew of problems simply go away with OAuth.

Finally.. the service providers get enormous flexibility and traceability with OAuth. They can associate each API call to not only user who is making the call but also to each third party app that is making the call. It helps them to understand and analyze who is making those API calls.

Read more at http://www.countdowntooauth.com