BIG DATA AND THE CHALLENGES AHEAD

                            BIG DATA AND THE CHALLENGES AHEAD
*********************************************************************************
Today, I would like to talk about the next revolution in IT platform: BIG DATA. Now, we have been hearing this thing for a very long time: what is it ,from
where did it come, why do we do need it  & what are the challenges and complications its posing to the current infra-structure/resources/technologies.

The data which is big in VOLUME (in PB :Peta bytes) ,moves with high VELOCITY & is unstructured /semi-structured /structured i.e. VARYING in nature is termed BIG DATA.
COMPLEXITY is another aspect of BIG DATA which means it must be able to traverse across multiple data centres , cloud and geographical zones.

As per Garter's definition :"BIG DATA are high volume, high velocity, and/or high variety information assets that require new forms of processing to enable
enhanced decision making, insight discovery and process optimisation".

It is quite staggering to know that the amount of data generated in the last 2 years is equivalent to the data which was present before that.Now, where did we
get this data from ?  The source data has grown in size and is being gathered by ubiquitous information-sensing mobile devices, aerial sensory technologies
(remote sensing), software logs, cameras, microphones, radio-frequency identification readers, social media (Facebook, Twitter, LinkedIn, Blogger etc), emails ,
and wireless sensor networks, in the form of voice, images, videos or text.

The next question arises that why do we need such prodigious data ? The answer is simple : The business needs to outperform the competition.
They want to spot business trends, analyse data patterns, understand their customer buying trends or behaviour and reform their analytical approach from
the traditional business intelligence and warehousing techniques.
To stay ahead in the market, a player should not only be aware of his own data but also his opponent's. He also needs to keep an hawk eye on the various
strategies applied by the competition by analysing their data. With such a huge data he can create thousands of reports, predict patterns and can take
business decisions.

To handle 'BIG DATA' we definitely have big Challenges ahead. I believe to solve new set of problems we should have new set of methodologies and tools/technologies
as we can't rely on traditional relational databases , designs and techniques to allay this 'new elephant' in the jungle.
The multiple challenges with BIG DATA are:

1. to capture from multiple source systems
2. to curate and architect a design
3. to store data in size greater than Peta bytes
4. to search the data in acceptable time frame
5. to share between concurrent users
6. to transfer across multiple systems/servers
7. to analyse and visualise data of such infinite magnitude.

Currently, it is beyond the ability of the contemporary tools to manage all these activities in a tolerable elapsed time. BIG DATA is difficult to work with using most relational database management systems, desktop statistics and visualisation packages, requiring instead massively parallel
software running on tens, hundreds, or even thousands of servers.

But do we have people and the skills ?
Recently heard that  organisations like Google are doing 10PB sorts on 8000 machines in just over 6 hours – we know the technical scope for BIG DATA
exists and eventually will flow down to the masses, and such scale will likely be achievable by most organisations in the next decade.Surely, the rise of data scientists is imminent.

There are and there will be people who can build such distributed systems or systems which can manage data having such orders of magnitude and operate on them to capture the data but what we lack today are the people who knows what to do with this data. The people who know how to
provide you patterns, discoveries, factoids & astonishing visualisations when provided with few PB of data. People with data analysis and knowledge discovery
skills who would be at the center & driving the BIG DATA revolution are hard to find.