TECHTalksPro
  • Home
  • Business
    • Internet
    • Market
    • Stock
  • Parent Category
    • Child Category 1
      • Sub Child Category 1
      • Sub Child Category 2
      • Sub Child Category 3
    • Child Category 2
    • Child Category 3
    • Child Category 4
  • Featured
  • Health
    • Childcare
    • Doctors
  • Home
  • SQL Server
    • SQL Server 2012
    • SQL Server 2014
    • SQL Server 2016
  • Downloads
    • PowerShell Scripts
    • Database Scripts
  • Big Data
    • Hadoop
      • Hive
      • Pig
      • HDFS
    • MPP
  • Certifications
    • Microsoft SQL Server -70-461
    • Hadoop-HDPCD
  • Problems/Solutions
  • Interview Questions

Wednesday, July 13, 2016

What is BiG Data

 Chitchatiq     7/13/2016 03:27:00 PM     Big Data, Hadoop, SQL Server     No comments   

                       
1.    Big data is a term that describes the large volume of structured, semi-structured, un-structured data that is generating from our day to day life




2.    Big data technologies talks not only about storing the Huge data, it also talks about how data will be processed and how the business is using that data to take decisions





3.    70% of the data has grown in recent 4-5 years only
E.g.
1.    The New York Stock Exchange generates about one terabyte of new trade data per day.
2.    Facebook hosts approximately 10 billion photos, taking up one petabyte of storage.

4.    Different kind of computer memory sizes are available
            Computer Memory size:

1 Bit                =         1 Byte
8 Bits               =         1 Byte
1024 Byte       =         1 KB (Kilo Byte)
1024 KB          =         1 MB( Mega Byte)
1024 MB        =         1 GB(Giga Byte)
1024 GB         =         1 TB(1 Terra Byte)
1024 TB          =         1 PB( Peta Byte)
1024 PB          =         1 EB(Exa Byte)
1024 EB          =         1 ZB(Zetta Byte) currently ZB of data generating everyday
1024 ZB          =         1 YB(Yotta Byte)
1024 YB          =         1 (Bronto Byte)
1024 BB          =         1 (Geop Byte)
Geop Byte is the highest memory as of today


5.    Big data can be defined by using 4 V’s

  1. Volume
  2. Velocity
  3. Variety
  4. Veracity
  5. Value


Volume: Large amount of data (MB-> GB-> TB-> PB->EB-> ZB…)

e.g Whatsapp
Consider Whatsapp previously chat history coming from users are in some GB or TB
Nowadays data is coming in EB/ZB

Velocity: Rate of data coming and processing (Velocity =(volume/Time))
e.g Facebook:
Consider Facebook users upload more than 500 million photos/videos a day. Velocity is the measure of how fast the data is coming in and how fast the data is processing.
Facebook has to handle a tsunami of photographs/videos every day. It has to ingest it all, process it, file it, and somehow, later, be able to retrieve it.

Variety: Structured Data, Semi structured data, Unstructured

Structured:  Data which can be stored in database SQL in table with rows and columns

Semi structured: It doesn’t reside in a relational database but that does have some organizational properties that make it easier to analyze
E.g XML, JSON

Unstructured: Now a day’s world has around 80% of unstructured data. It often include text and multimedia content. Examples include e-mail messages, word processing documents, videos, photos, audio files, presentations
E.G. Twitter tweets, Face book scraps, videos, images, Radar data, Plan

Veracity:  Messiness of the data. With many forms of big data, quality and accuracy are less controllable (e.g. twitter posts with hash tags, abbreviations, typos..) so Big data and analytics technologies now allow us to work with these types of data

Value: turn raw data into value

Why BiG Data
Big data doesn’t mean that how much data you have, but what you do with it. We can take any kind of data from any source and analyze it to find answers with less cost and less time. With Existing Data warehouse system, we take some portion of data (like recent 2-3 years sales data) and analyzing the trends of data to take the business decision. But with the Big data technologies we can analyze the whole dataset and we can take better business decisions
When we combine big data with analytics technologies (Machine learning R), you can accomplish business-related tasks such as:

  • Fraud Detections in banking system
  • Finding the most churn customer in telecommunications
  • etc...
Who uses Big Data
Almost all industries are using the Big Data.
Ø  Banking
Ø  Education
Ø  Government
Ø  Health Care
Ø  Manufacturing
Ø  Retail
 How Big Data Works
With Big Data technologies (Hadoop), we can store, process, and analyze the large amount of data with
Ø  Less cost of hardware
Ø  Fast processing
Storage + Processing + Analyze =Big Data
Hadoop specially designed to handle Big Data. We will discuss about hadoop in next session
                                                                                                                           Session 2
  • Share This:  
  •  Facebook
  •  Twitter
  •  Google+
  •  Stumble
  •  Digg
Email ThisBlogThis!Share to XShare to Facebook

Related Posts:

  • File Formats in Hive As Hive uses HDFS to store the structured data in files … multiple file formats are supported by hive with various use cases. 1.  &n… Read More
  • What is BiG Data                         1.    Big data is a term that desc… Read More
  • How to Fix ERROR: CANNOT PARALLELIZE AN UPDATE STATEMENT THAT UPDATES THE DISTRIBUTION COLUMNS Printable View Error: [DataDirect][ODBC Greenplum Wire Protocol driver][Greenplum]ERROR: Cannot parallelize an UPDATE statement that updates the distribution col… Read More
  • SSIS-Sequence, Foreach, For loop containers Recently one of my friend asked me a question like  "when to use sequence container, For loop, and For each loop in SSIS" Here I tried my bes… Read More
  • Powershell Script for SSRS Deployment Scenarios: Deploy SSRS reports using automation #Set variables with configure values $Environment = "DEV" $reportPath ="/" $reportFold… Read More
Newer Post Home

0 comments:

Post a Comment

Popular Posts

  • FAILED: SemanticException [Error 10294]: Attempt to do update or delete using transaction manager that does not support these operations.
    Error: Resolution:         1.       Check if Your table storage format is ORC         2.      Check if yo...
  • Greenplum System-Useful Queries
    Useful Queries: Query to verify the list of segments in a Greenplum system select * from gp_segment_configuration; R...
  • Getting Error while accessing Hive from command line interface
    Some times we  see below error while launching Hive from command line. Error: Logging initialized using configuration in file:/et...
  • SELECT TOP COUNT dynamically
    Scenario: Some times we want to make TOP Count as Dynamically then passing parameter to TOP command will not work. We have to add paramete...
  • How to fix ERROR 1045 : Access denied for user 'root'@'localhost' (using password: YES)
    Please follow below steps to fix HDP Sandbox -Mysql root password issue: Enter mysql -u root UPDATE mysql.user SET Password=PASSWOR...

Facebook

Categories

Best Practices (1) Big Data (5) BigData&Hadoop (6) DAG (1) Error 10294 (1) external tables (1) File Formats in Hive (1) Greenplum (3) Hadoop (5) Hadoop Commands (1) Hive (4) Internal tables (1) interview Questions (1) Managed tables (1) MySQL Installation (1) ORCFILE (1) org.apache.hadoop.hive.ql.exec.MoveTask (1) Powershell (1) Problems&Solutions (15) RCFILE (1) return code 1 (1) SEQUENCEFILE (1) Service 'userhome' (1) Service 'userhome' check failed: java.io.FileNotFoundException (1) SQL Server (27) sqoop (2) SSIS (1) TEXTFILE (1) Tez (1) transaction manager (1) Views (1) What is Hadoop (1)

Blog Archive

  • December (1)
  • November (1)
  • October (2)
  • September (6)
  • August (1)
  • July (3)
  • March (1)
  • February (8)
  • January (4)
  • December (9)
  • August (4)
  • July (1)

Popular Tags

  • Best Practices
  • Big Data
  • BigData&Hadoop
  • DAG
  • Error 10294
  • external tables
  • File Formats in Hive
  • Greenplum
  • Hadoop
  • Hadoop Commands
  • Hive
  • Internal tables
  • interview Questions
  • Managed tables
  • MySQL Installation
  • ORCFILE
  • org.apache.hadoop.hive.ql.exec.MoveTask
  • Powershell
  • Problems&Solutions
  • RCFILE
  • return code 1
  • SEQUENCEFILE
  • Service 'userhome'
  • Service 'userhome' check failed: java.io.FileNotFoundException
  • SQL Server
  • sqoop
  • SSIS
  • TEXTFILE
  • Tez
  • transaction manager
  • Views
  • What is Hadoop

Featured Post

TOP 100 SQL SERVER INTERVIEW QUESTIONS

SQL SERVER INTERVIEW QUESTIONS 1.       What is the Complex task that you handled in your project 2.       What are the diffe...

Pages

  • Home
  • SQL SERVER
  • Greenplum
  • Hadoop Tutorials
  • Contact US
  • Disclaimer
  • Privacy Policy

Popular Posts

  • FAILED: SemanticException [Error 10294]: Attempt to do update or delete using transaction manager that does not support these operations.
    Error: Resolution:         1.       Check if Your table storage format is ORC         2.      Check if yo...
  • Greenplum System-Useful Queries
    Useful Queries: Query to verify the list of segments in a Greenplum system select * from gp_segment_configuration; R...

Copyright © 2025 TECHTalksPro
Designed by Vasu