CHAPTER 1: INFORMATION SYSTEM

CHAPTER 2: CONTROL,AUDIT AND SECURITY OF INFORMATION SYSTEM

CHAPTER 3: ENTERPRISE MANAGEMENT SYSTEM

OLD QUESTION BANK SOLUTION

CHAPTER 5: PLANNING FOR IS

CHAPTER 6: IMPLEMENTATION OF INFORMATION SYSTEM

CHAPTER 7: WEB BASED INFORMATION SYSTEM AND NAVIGATION

OLD QUESTION BANK

CHAPTER 8: SCALABLE AND EMERGING INFORMATION SYSTEM TECHNIQUES

IS CASE STUDY TOPICS

CHAPTER 4: DECISION SUPPORT AND INTELLIGENT SYSTEM

IS PRACTICE QUESTION

MapReduce and Hadoop systems

MAPREDUCE

MapReduce is a programming model and processing framework designed for parallel processing and distributed computing of large datasets. It was introduced by Google and has become a fundamental concept in big data processing. The MapReduce model is commonly used to process and analyze massive amounts of data across a distributed cluster of computers. 

Map Function:

  • The Map function is responsible for processing input data and emitting a set of key-value pairs. It takes the input data and applies a user-defined operation to produce intermediate key-value pairs.
  • Shuffle and Sort:
    • After the Map phase, the framework groups and sorts the intermediate key-value pairs based on the keys. This process is known as the Shuffle and Sort phase.
  • Reduce Function:
    • The Reduce function takes the sorted key-value pairs produced by the Shuffle and Sort phase and performs a user-defined operation on each group of values associated with the same key. The result is a set of output key-value pairs.
  • Input Split:
    • The input data is divided into smaller chunks called input splits. Each input split is processed by a separate instance of the Map function.
  • Partitioning:
    • During the Shuffle and Sort phase, the framework partitions the data based on keys, distributing it to the appropriate Reducer tasks.
  • Master Node:
    • The master node is responsible for coordinating the MapReduce job. It assigns tasks to worker nodes, manages the execution flow, and collects the final output.
  • Worker Nodes:
    • Worker nodes execute the Map and Reduce tasks. They process the input data, perform the necessary computations, and communicate with the master node.

Working of MapReduce

  • Map Phase:
    • Input data is divided into input splits, and multiple Map tasks are executed concurrently across the cluster. The Map function processes each input split and emits intermediate key-value pairs.
  • Shuffle and Sort Phase:
    • The intermediate key-value pairs are shuffled and sorted based on keys. This ensures that all values associated with the same key end up in the same group.
  • Reduce Phase:
    • Multiple Reduce tasks are executed concurrently, each processing a subset of the key-value pairs. The Reduce function is applied to the values associated with each key, producing the final output.

Advantages of MapReduce:

  • Parallel Processing:
    • MapReduce enables the parallel processing of data across a distributed cluster, providing scalability for handling large datasets.
  • Fault Tolerance:
    • The framework automatically handles node failures by restarting failed tasks on other nodes, ensuring fault tolerance.
  • Distributed Computing:
    • MapReduce abstracts the complexities of distributed computing, allowing developers to focus on defining the Map and Reduce operations.
  • Scalability:
    • MapReduce is designed to scale horizontally, allowing organizations to add more nodes to the cluster as data volumes grow.

Application:

  • Big Data Processing:
    • MapReduce is commonly used in big data frameworks such as Apache Hadoop for processing and analyzing large datasets.
  • Log Analysis:
    • Analyzing logs from various sources to extract valuable insights and patterns.
  • Search Engines:
    • Search engines use MapReduce to index and process large amounts of web data.
  • Machine Learning:
    • Training and processing data for machine learning models, especially in distributed machine learning frameworks.

HADOOP

 

Hadoop is an open-source framework designed for distributed storage and processing of large data sets. It is part of the Apache Software Foundation and provides a scalable, fault-tolerant, and cost-effective solution for handling big data. Hadoop is particularly well-suited for processing and analyzing data across distributed clusters of commodity hardware. 

Components of Hadoop :

  • Hadoop Distributed File System (HDFS):
    • HDFS is the distributed file system used by Hadoop for storing vast amounts of data across multiple nodes in a Hadoop cluster. It provides high fault tolerance by replicating data across nodes.
  • MapReduce:
    • MapReduce is a programming model and processing engine for parallel and distributed data processing. It allows developers to write programs that process large amounts of data in parallel across a Hadoop cluster.
  • YARN (Yet Another Resource Negotiator):
    • YARN is the resource management layer of Hadoop. It manages and schedules resources (CPU, memory) across different applications running on a Hadoop cluster. YARN decouples the resource management and job scheduling functionalities, providing flexibility in running diverse workloads.
  • Hadoop Common:
    • Hadoop Common includes the libraries, utilities, and APIs that support the other Hadoop modules. It provides the foundation for the Hadoop ecosystem and ensures compatibility across different Hadoop components.

Features of Hadoop:

  • Scalability:
    • Hadoop can scale horizontally by adding more nodes to a cluster, making it suitable for handling growing amounts of data.
  • Fault Tolerance:
    • Hadoop ensures fault tolerance by replicating data across multiple nodes in the HDFS. If a node fails, the data can be retrieved from its replicated copies on other nodes.
  • Data Locality:
    • Hadoop aims to process data where it is stored to minimize data movement and reduce network overhead. This concept is known as data locality.
  • High Throughput:
    • Hadoop is designed to provide high throughput for data processing. It can efficiently process large volumes of data by distributing the workload across multiple nodes.

Applications:

  • Big Data Analytics:
    • Hadoop is widely used for processing and analyzing large datasets, enabling organizations to derive valuable insights from their data.
  • Log Processing:
    • Analyzing logs generated by applications or systems to identify patterns, troubleshoot issues, or gain operational insights.
  • Data Warehousing:
    • Hadoop, along with tools like Apache Hive and Apache HBase, is used for building cost-effective and scalable data warehouses.
  • Machine Learning:
    • Hadoop serves as a platform for distributed machine learning, allowing organizations to train and deploy machine learning models at scale.
  • Genomic Data Analysis:
    • Hadoop is applied in bioinformatics for processing and analyzing large volumes of genomic data.

Working of Hadoop:

 

Hadoop works on a distributed computing model and is designed to handle large volumes of data by distributing the processing across a cluster of commodity hardware. The key components of Hadoop, including the Hadoop Distributed File System (HDFS) and the MapReduce programming model, work together to enable distributed storage and processing. Here is an overview of how Hadoop works:

  • Data Storage with HDFS:
    • The first step in the Hadoop process is storing large datasets in the Hadoop Distributed File System (HDFS). HDFS divides large files into smaller blocks (typically 128 MB or 256 MB) and replicates these blocks across multiple nodes in the Hadoop cluster for fault tolerance.
  • Data Processing with MapReduce:
    • Once the data is stored in HDFS, the processing is done using the MapReduce programming model. The MapReduce process consists of two main phases: the Map phase and the Reduce phase.
    • Map Phase:
      • Input data is divided into splits, and multiple Map tasks are created to process these splits concurrently.
      • The user-defined Map function is applied to each split, producing a set of intermediate key-value pairs.
    • Shuffle and Sort:
      • The intermediate key-value pairs are shuffled and sorted based on keys. This ensures that all values associated with the same key end up in the same group.
    • Reduce Phase:
      • Multiple Reduce tasks are created to process the sorted key-value pairs.
      • The user-defined Reduce function is applied to each group of values associated with the same key, producing the final output.
  • Resource Management with YARN:
    • YARN (Yet Another Resource Negotiator) is responsible for managing resources in the Hadoop cluster. It allocates resources (CPU, memory) to different Map and Reduce tasks based on their requirements.