For enterprises, what comes first? 

Adopting a solution that is not highly available and then try to make it enterprise ready or start with high availability as core feature then add the power of analytics?

In my past few articles I wrote about the challenges to adopting Hadoop in the enterprise and what it would take to make it enterprise ready. One of the points I highlighted is the NameNode high availability or the lack of it. In Hadoop Distributed File System (HDFS), NameNode is the metadata server that has the location information for data blocks distributed across DataNodes. If NameNode fails, the cluster would be unavailable to analytics applications.

The Hadoop community has been working on a solution to add High Availability to HDFS. The solution entails adding another NameNode with shared storage and changing DataNodes to send...

The biggest driver for adoption of Hadoop is its promise of unlocking value from an enterprise’s vast data store. Use cases that show incremental revenue from data analysis are very well publicized.  Every organization strives to achieve that and wants to leverage the power of data analytics to drive its revenues. Promises aside, Hadoop storage has severe issues that calls  into question its place in the enterprise datacenter.

1. Increased storage and server sprawl – Hadoop cluster is built  with numerous commodity hosts, each with its own direct attached storage. Just when datacenter architects have spent considerable time and resources consolidating their datacenters and reducing footprint through server consolidation, virtualization and private cloud,  Hadoop requires them to build out a massively parallel system with hundreds or even thousands of compute nodes. Managing these numerous nodes and keeping them up to date...

Suppose you read the following headline in a major newspaper article, what would you think?

Student Test Scores Tied to Number of Bathrooms in their Homes

Let us say, this article is also associated with a chart showing this relation

Look at those near perfect correlations. Should we start adding more bathrooms to help our children?

Except there is no such study but very close.  The x-axis is actually income level of the family. While we see a nice positive correlation between income and test scores, Harvard Economics Professor Greg Mankiw warns us about the spurious correlation using...

Does Big Data ( big on volume and variety) mean better insights?
Taking this to the extreme, does Big Data eliminate application of thought?

A recent New York Times article writes,

Big Data, which should probably be called Big Analysis, is about looking at that information in novel ways to find new patterns for prediction.

I agree with their call but let us not try to change an accepted terminology. What Times article states is the fact that we are able to look at data in new ways with newer tools. The value add comes more from analytics applications that help in answering the question at hand.

At the extreme I refer to above, we see those who favor relying on volume of data, on the Bigness of Big Data, to tell us what to do. The next logical step for them is to include every possible data source and every bit of data in the analysis...