Setting up Development Environment

• Setting up Development environment on User's laptop to be able to develop and execute programs
• Setting up Eclipse ( Basics of Eclipse like, import, create project, add JARs) to understand basics of Eclipse for Map Reduce and Spark development
• Installing Maven & Gradle to understand building tools
• Installing Putty, FileZilla/WinSCP to get ready to access EduPristine's Big Data Cloud

• Case Study: XYZ Telecom need to set up approprioate directory Structure along with permissions on various files on Linux file system
• Setting up, accessing and verifying Linux server access over SSH
• Transferring files over FTP or SFTP
• Creating directory structure and Setting up permissions
• Understanding File name pattern and move using regular expressions
• Changing file owners, permissions
• Reviewing mock file generator utility written in Shell Script, enhancing it to be more useful
• Case Study: Developing a simulator to generate mock data using Python
• Understand your domain requirement, describe need of required fields and possible values, file format etc
• Preparing configuration file that can be changed to fit any requirement
• Developing python script to generate mock data in configuration file

Case Study: Design and Develop Phone Book in Java

• Identifying Classes and Methods for Phone Book
• Implementing design into Java Code using Eclipse
• Compiling and Executing Java Program
• Enhancing the code with each learnings, like Inheritence, Method overloading
• Further enhancing the code to initialize PhoneBook from a Text File by using Java file reading

Case Study: Handling huge data set in HDFS to make it accessible to right user and remove non-functional requirements like backups, cost, high availability etc.

• Understanding the problem statement and challenges persisting to such large data to perceive the need of Distributed File System
• Understanding HDFS architecture to solve problems
• Understanding configuration and creating directory structure to get a solution of the given problem statement
• setup appropriate permissions to secure data for appropriate users

Case Study: Developing automation tool for HDFS file management

• Setting up Java Development with HDFS libraries to use HDFS Java APIs
• Coding to develope menu driven HDFS file management utility and schedule to run for file management in HDFS cluster

Sqoop

Case Study: Develop automation utility to migrate huge RDBMS warehouse implemented in MySQL to Hadoop cluster

• Creating and loading data into RDBMS table to understand RDBMS setup
• Preparing data to experiment with Sqoop imports
• Importing using Sqoop Command in HDFS file system to understand simple imports
• Importing using Sqoop command in Hive table to import data into Hive partitioned table and perform ETL
• Exporting using Sqoop from Hive/ HDFS to RDBM to store the output of Hive ETL into RDBMS
• Wrapping Sqoop commands into Unix Shell Script To be able to build and use automated utility for day to day use

Map-Reduce

Case Study: Processing 4G usage data of a Telecom Operator to find out potential customers for various promotional offers

• Cleaning data, ETL and Aggregation
• Exploring data set using known tools like linux commands to understand the nature of data
• Setting up Eclipse project, maven dependencies to add required Map Reduce Libraries
• Coding, packaging and deploying project on hadoop cluster to understand how to deploy/ run map reduce on Hadoop Cluster

Case Study: Process a structured data set to find some insights

• Finding out per driver total miles and hours driven
• Creating Table, Loading Data, Selecting Query to load, query and cleaning of data
• Which driver has driven maximum & minimum miles
• Joining Tables, Saving Query results to table to explore and use right type of table type, partition schema, buckets
• Discussing optimum file format for hive table
• Using right file format, type of table, partition scheme to optimize query performance
• Using UDFs to reuse domain specific implementations

Case Study: Perform ETL processing on Data Set to find some insights

• Loading and exploring Movie - 100K data set to load data set, explore it and associate schema to it
• Using grunt, Loading data set, defining schema
• Finding Simple Statistic from given Data Set to clean up the data
• Filtering and modifying data schema
• Finding gender distribution in users
• Aggregrating and looping
• Finding top 25 movies by rating, joining data sets and saving to HDFS to perform aggregration
• Dumping, Storing, joining, sorting
• Filtering function for complex condition to reuse domain specific functionalities & avoid rewriting code
• Using UDFs

Case Study: Build a model to predict production error/ failure (huge servers - applications/ software) with good speed by using computation power efficiently while considering processor challenges

• Loading and performing pre-processing to convert unstructured data to some structured data format
• Cleaning data, filtering out bad records, converting data to more usable format
• Aggregating data based on Response Code to find out server' performace from logs
• Filtering, Joining and aggregating data to find top 20 Frequent Hosts that generates errors

Spark Project

Case Study: Build a model (using Python) to predict production error/ failure (huge servers - applications/ software) with good speed by using computation power efficiently while considering processor challenges

• Loading and performing pre-processing to convert unstructured data to some structured data format
• Cleaning data, filtering out bad records, converting data to more usable format
• Aggregating data based on Response Code to find out server' performace from logs
• Filtering, Joining and aggregating data to find top 20 Frequent Hosts that generates errors

Case Study: Setting up Data processing pipeline to work as per schedule in Hadoop Eco System comprising of multiple components like sqoop job, hive scripts, pig scripts, spark jobs etc.

• Setting up Oozie workflow to tigger a script, then Sqoop Job followed by Hive Job
• Executing workflow to run complete ETL pipeline

Case Study: Find out top 10 customers by expenditure, top 10 most buying brands, and monthly sales from data stored in Hbase which is in Key value pair

• Designing Hbase Table Schema to model table structure, decide families in table as per data
• Deciding families in table as per data
• Bulk Loading & Programatically Loading data using Java APIs to populate data into Hbase table
• Querying and Showing data on UI to integret Hbase with UI/Reporting

Project: ETL processing of retail logs

• To find demand of a given product
• Trend and seasonality of a product
• Understand performance of the chain

Project: Creating 360 degree view (past, present and future) of the customer for a retail company - avoiding repetition or re-keying of information, to view customer history, establishing context and initiating desired actions

• Explorating and checking basic quality of data to understand data and need for filtering/pre-processing of data
• Loading data into RDBMS table to simulate real world scenarios where data is persisted in RDBMS table
• Developing & executing Sqoop Job to ingest Data in Hadoop Cluster to perfrom further actions
• Developing & executing Pig script to perform required ETL processing on ingested Data
• Developing & executing Hive Queries to get reports out of processed data

Project: Twitter Sentiment Analytics - Collect and real time data (JSON format), and perform sentiment analysis on continuously flowing streaming data

• Creating and setting up Twitter App to generate Twitter Auth tokens to access twitter by APIs
• Building Flume Source to pull twits from Twitter
• Setting up Flume agent with Kafka sink to persist twits into Distributed Kafka topic
• Setting up Flume agent with Kafka Source and HDFS as Sink to backup twits on HDFS for batch processing
• Building & Executing Spark Job to perform Sentiment Analytics in real time on each incoming twits
• Creating Hive table on Twits and perform basic queries to understand Hive Serdes and dealing with Semi-structured data in Hive
• Writing and Executing Impala Queries to understand and overcome Impala's limitation of not being able to use Hive Serdes

Project: Machine Learning with TensorFlow - Build a solution which can recognize images on search words and can run on distributed computing like Hadoop/ Spark etc. for a photo storage company

• Setting up development environment to be able to use TensorFlow java APIs
• Developing Java Code using TensorFlow Inception Model using Java API to develop image recognition program in Java
• Developing Python code to train TensorFlow model to learn to training TensorFlow model, for a domain specific problem

Project: Developing a Chat-bot to offer an artificially intelligent customer help desk for an insurance company

• Identify client's most frequently asked questions and answers
• Develop a training data set
• Build TensorFlow NLP model which can understand questions
• Train the model
• Implement & Run the model

Real Time Analytics, Unstructured Data Ingestion

An open source database that uses a document-oriented data model

Exam pattern, CV preparation & Imp topics