What is BiG Data

                       

1.    Big data is a term that describes the large volume of structured, semi-structured, un-structured data that is generating from our day to day life

2.    Big data technologies talks not only about storing the Huge data, it also talks about how data will be processed and how the business is using that data to take decisions





3.    70% of the data has grown in recent 4-5 years only

E.g.

1.    The New York Stock Exchange generates about one terabyte of new trade data per day.

2.    Facebook hosts approximately 10 billion photos, taking up one petabyte of storage.

4.    Different kind of computer memory sizes are available

            Computer Memory size:

1 Bit                =         1 Byte

8 Bits               =         1 Byte

1024 Byte       =         1 KB (Kilo Byte)

1024 KB          =         1 MB( Mega Byte)

1024 MB        =         1 GB(Giga Byte)

1024 GB         =         1 TB(1 Terra Byte)

1024 TB          =         1 PB( Peta Byte)

1024 PB          =         1 EB(Exa Byte)

1024 EB          =         1 ZB(Zetta Byte) currently ZB of data generating everyday

1024 ZB          =         1 YB(Yotta Byte)

1024 YB          =         1 (Bronto Byte)

1024 BB          =         1 (Geop Byte)

Geop Byte is the highest memory as of today

5.    Big data can be defined by using 4 V’s

1. Volume
2. Velocity
3. Variety
4. Veracity
5. Value

Volume: Large amount of data (MB-> GB-> TB-> PB->EB-> ZB…)

e.g Whatsapp

Consider Whatsapp previously chat history coming from users are in some GB or TB

Nowadays data is coming in EB/ZB

Velocity: Rate of data coming and processing (Velocity =(volume/Time))

e.g Facebook:

Consider Facebook users upload more than 500 million photos/videos a day. Velocity is the measure of how fast the data is coming in and how fast the data is processing.

Facebook has to handle a tsunami of photographs/videos every day. It has to ingest it all, process it, file it, and somehow, later, be able to retrieve it.

Variety: Structured Data, Semi structured data, Unstructured

Structured:  Data which can be stored in database SQL in table with rows and columns

Semi structured: It doesn’t reside in a relational database but that does have some organizational properties that make it easier to analyze

E.g XML, JSON

Unstructured: Now a day’s world has around 80% of unstructured data. It often include text and multimedia content. Examples include e-mail messages, word processing documents, videos, photos, audio files, presentations

E.G. Twitter tweets, Face book scraps, videos, images, Radar data, Plan

Veracity:  Messiness of the data. With many forms of big data, quality and accuracy are less controllable (e.g. twitter posts with hash tags, abbreviations, typos..) so Big data and analytics technologies now allow us to work with these types of data

Value: turn raw data into value

Why BiG Data

Big data doesn’t mean that how much data you have, but what you do with it. We can take any kind of data from any source and analyze it to find answers with less cost and less time. With Existing Data warehouse system, we take some portion of data (like recent 2-3 years sales data) and analyzing the trends of data to take the business decision. But with the Big data technologies we can analyze the whole dataset and we can take better business decisions

When we combine big data with analytics technologies (Machine learning R), you can accomplish business-related tasks such as:

• Fraud Detections in banking system
• Finding the most churn customer in telecommunications
• etc...

Who uses Big Data

Almost all industries are using the Big Data.

Ø  Banking
Ø  Education
Ø  Government
Ø  Health Care
Ø  Manufacturing
Ø  Retail

 How Big Data Works
With Big Data technologies (Hadoop), we can store, process, and analyze the large amount of data with

Ø  Less cost of hardware
Ø  Fast processing

Storage + Processing + Analyze =Big Data

Hadoop specially designed to handle Big Data. We will discuss about hadoop in next session
                                                                                                                           Session 2