Download Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data ... PDF

Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data ...
Name: Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data ...
Pages: 135
Year: 2016
Language: English
File Size: 8.5 MB
Downloads: 0
Page 2

About Cisco Validated Designs 2 About Cisco Validated Designs The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit http://www.cisco.com/go/designzone. ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO. CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R) 2016 Cisco Systems, Inc. All rights reserved.


Page 3

About Cisco Validated Designs 3 Table of Contents About Cisco Validated Designs .................................................................................................................................................. 2 Executive Summary .................................................................................................................................................................... 6 Solution Overview ....................................................................................................................................................................... 7 Introduction ............................................................................................................................................................................ 7 Solution .................................................................................................................................................................................. 7 Audience ................................................................................................................................................................................ 7 Solution Summary .................................................................................................................................................................. 7 MapR Converged Data Platform ........................................................................................................................................ 8 Lambda Architecture Combining Real time and Batch Processing .................................................................................. 9 MapR Reference Architecture .......................................................................................................................................... 10 Technology Overview ............................................................................................................................................................... 12 Cisco UCS Integrated Infrastructure for Big Data with MapR and MapR Streams ............................................................... 12 Cisco UCS 6200 Series Fabric Interconnects .................................................................................................................. 12 Cisco UCS 6300 Series Fabric Interconnects .................................................................................................................. 12 Cisco UCS C Series Rack Mount Servers ....................................................................................................................... 12 Cisco UCS Virtual Interface Cards (VICs) ........................................................................................................................ 13 Cisco UCS Manager ........................................................................................................................................................ 14 MapR Converged Data Platform 5.1 .................................................................................................................................... 15 MapR Enterprise Grade Platform Services ...................................................................................................................... 16 MapR Open Source Technologies ................................................................................................................................... 18 Solution Design......................................................................................................................................................................... 20 Requirements ....................................................................................................................................................................... 20 Rack and PDU Configuration ............................................................................................................................................... 20 Port Configuration on Fabric Interconnects ...................................................................................................................... 21 Server Configuration and Cabling for Cisco UCS C Series M4........................................................................................ 21 Software Distributions and Versions ..................................................................................................................................... 23 MapR ............................................................................................................................................................................... 23 Red Hat Enterprise Linux (RHEL) .................................................................................................................................... 23 Software Versions ................................................................................................................................................................ 23 Fabric Configuration ............................................................................................................................................................. 24 Performing Initial Setup of Cisco UCS 6296 Fabric Interconnects ....................................................................................... 24 Configure Fabric Interconnect A ...................................................................................................................................... 24 Configure Fabric Interconnect B ...................................................................................................................................... 25 Logging Into Cisco UCS Manager .................................................................................................................................... 26 Upgrading UCSM Software to Version 3.1(1g) ................................................................................................................ 26 Adding a Block of IP Addresses for KVM Access ............................................................................................................ 26 Enabling Uplink Ports ....................................................................................................................................................... 27


Page 4

About Cisco Validated Designs 4 Configuring VLANs ............................................................................................................................................................... 28 Enabling Server Ports ...................................................................................................................................................... 30 Creating Pools for Service Profile Templates ....................................................................................................................... 31 Creating an Organization ................................................................................................................................................. 31 Creating MAC Address Pools ............................................................................................................................................... 32 Creating a Server Pool ......................................................................................................................................................... 33 Creating Policies for Service Profile Templates ................................................................................................................... 35 Creating Host Firmware Package Policy .......................................................................................................................... 35 Creating QoS Policies ...................................................................................................................................................... 36 Creating the Local Disk Configuration Policy ................................................................................................................... 39 Creating Server BIOS Policy ............................................................................................................................................ 40 Creating the Boot Policy .................................................................................................................................................. 42 Creating Power Control Policy ......................................................................................................................................... 43 Creating a Service Profile Template ..................................................................................................................................... 45 Configuring the Storage Provisioning for the Template .................................................................................................... 46 Configuring Network Settings for the Template ................................................................................................................ 47 Configuring the vMedia Policy for the Template ............................................................................................................... 54 Configuring Server Boot Order for the Template .............................................................................................................. 55 Configuring Server Assignment for the Template ............................................................................................................ 57 Configuring Operational Policies for the Template ........................................................................................................... 58 Installing Red Hat Enterprise Linux 7.2 ................................................................................................................................ 60 Post OS Install Configuration ............................................................................................................................................... 83 Setting Up Password less Login ...................................................................................................................................... 83 Configuring /etc/hosts ...................................................................................................................................................... 84 Creating a Red Hat Enterprise Linux (RHEL) 7.2 Local Repo.......................................................................................... 86 Creating the Red Hat Repository Database. .................................................................................................................... 87 Setting up ClusterShell .................................................................................................................................................... 88 Installing httpd .................................................................................................................................................................. 90 Set Up all Nodes to use the RHEL Repository ................................................................................................................. 90 Configuring DNS .................................................................................................................................................................. 91 Upgrading the Cisco Network Driver for VIC1227 ............................................................................................................ 92 Setting up JAVA ............................................................................................................................................................... 93 NTP Configuration ................................................................................................................................................................ 95 Enabling Syslog.................................................................................................................................................................... 97 Setting ulimit .................................................................................................................................................................... 97 Disabling SELinux ............................................................................................................................................................ 98 Set TCP Retries ............................................................................................................................................................... 98 Disabling the Linux Firewall ............................................................................................................................................. 99 Disable Swapping ............................................................................................................................................................ 99


Page 5

About Cisco Validated Designs 5 Disable Transparent Huge Pages .................................................................................................................................... 99 Disable IPv6 Defaults ..................................................................................................................................................... 100 Configuring Data Drives ................................................................................................................................................. 100 Cluster Verification and Micro Benchmark ......................................................................................................................... 101 Running the Cluster Verification Script .......................................................................................................................... 101 Change Permissions to Executable ............................................................................................................................... 105 Running STREAM Benchmark ....................................................................................................................................... 105 Running MapR RPCtest ................................................................................................................................................. 106 Running IOzone Benchmark .......................................................................................................................................... 108 Installing MapR................................................................................................................................................................... 110 Planning the Cluster ....................................................................................................................................................... 110 MapR Services ............................................................................................................................................................... 110 Node Types .................................................................................................................................................................... 111 Hostnames and Roles .................................................................................................................................................... 112 Preparing Packages and Repositories ........................................................................................................................... 113 RPM Repositories for MapR Core Software................................................................................................................... 113 RPM Repositories for Hadoop Ecosystem Tools ........................................................................................................... 113 MapR Software Installation ............................................................................................................................................ 116 Installing MapR packages .............................................................................................................................................. 117 Verification of Installation ............................................................................................................................................... 118 Formatting Disks with the disksetup Script..................................................................................................................... 119 Identify and Format the Data Disks for MapR ................................................................................................................ 119 Bringing Up the Cluster .................................................................................................................................................. 121 Initialization Sequence ................................................................................................................................................... 121 Installing Spark .............................................................................................................................................................. 123 Installing the Cluster License ......................................................................................................................................... 125 Using Web based MCS to Install the License ................................................................................................................ 125 Installing a License from the Command Line (optional) ................................................................................................. 127 Restarting MapR Services after License Installation ...................................................................................................... 127 Verifying Cluster Status ................................................................................................................................................. 128 Enabling MapR Streams ................................................................................................................................................ 128 Installing Additional Hadoop Components ..................................................................................................................... 128 Troubleshooting ............................................................................................................................................................. 129 Conclusion ......................................................................................................................................................................... 130 Bill of Materials ....................................................................................................................................................................... 131 About the Authors ................................................................................................................................................................... 135 Acknowledgements ............................................................................................................................................................ 135


Page 6

Executive Summary Apache Hadoop is a framework that allows distributed processing of large data sets with custom applications for both big data and analytics and is one of the fastest growing technologies providing a competitive advantage for businesses across industries. Previously, the primary method for tapping into the value of big data was through batch processing of the dataset. Recent improvements in technology now allow the ability for fast interactive analysis and real time processing of streaming data. The challenge now is to design and build a reliable big data system that simultaneously handles batch processing, interactive analysis and real time processing of streaming data. This has led to the development of the Lambda Architecture. Lambda Architecture is a framework for designing big data applications with a generic architecture with built in capabilities for fault tolerance against hardware failures, software bugs, etc., and it supports use cases that address both low latency queries, and scaling and sizing of the system with manageable extensibility to accommodate new features. The MapR Converged Data Platform integrates the power of Hadoop and Spark with global event streaming, real time database capabilities and enterprise storage for developing and running innovative data applications built around the Lambda Architecture. This platform is powered by one of secure and open data infrastructures, including MapR Streams: a global publish subscribe event streaming system for big data. MapR Streams is the first big data scale streaming system built into a converged data platform. It makes data available instantly to stream processing and other applications, and is the only big data streaming system to support global event replication reliably at IoT scale. The MapR Converged Data Platform allows enterprises to build reliable, real time applications by providing: a single cluster for streams, file storage database and analytics, persistence of streaming data, providing direct access to batch and interactive frameworks, a unified security framework for data in motion and data at rest with authentication, authorization and encryption, and a utility grade reliability with self healing and no single point of failure architecture. The Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data Platform enables the next generation of big data architecture by providing simplified and centralized management, industry leading performance, and a linearly scaling infrastructure and software platform. The configuration detailed in the document can be scaled to clusters of various sizes depending on the application demand. Up to 80 servers (5 racks) can be supported with no additional switching in a single Cisco UCS domain. Scaling beyond 5 racks (80 servers) can be implemented by interconnecting multiple Cisco UCS domains using Nexus 9000 Series switches or Cisco Application Centric Infrastructure (ACI), scalable to thousands of servers and to hundreds of petabytes of storage, and managed from a single pane using Cisco UCS Central.


Page 7

Solution Overview 7 Solution Overview Introduction Big data technology has evolved from exclusively processing with batch jobs against large data sets to processing with fast interactive analysis and processing of real time streaming data. the tools to develop robust, reliable applications as defined by the Lambda Architecture, and the ability to economically administer and support these systems. The MapR Converged Data Platform integrates the power of Hadoop and Spark with global event streaming, real time database capabilities and enterprise storage for developing and running innovative data applications. MapR was engineered for the data center with IT operations in mind. MapR enables big data applications using Hadoop, Spark and more to serve business critical needs that cannot afford to lose data, must run on a 24x7 basis and require immediate recovery from node and site failures. The Cisco UCS Integrated Infrastructure for Big Data and Analytics and MapR Converged Data Platform support these capabilities for the broadest set of applications from batch analytics to interactive querying and real time streaming. Solution This CVD describes a scalable architecture and deployment procedures for the MapR Converged Data Platform on the Cisco UCS Integrated Infrastructure for Big Data and Analytics. As one of the technology leaders in Hadoop, the MapR Converged Data Platform distribution provides enterprise class big data solutions that are fast to develop and easy to administer. With significant investment in critical technologies, MapR offers a complete Hadoop platform a platform that is fully optimized for performance and scalability. Deployed as part of a comprehensive data center architecture, the Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR fundamentally transforms the way that organizations do business with Hadoop technology by delivering a powerful and flexible infrastructure that: increases business and IT agility, reduces total cost of ownership (TCO), and delivers exceptional return on investment (ROI) at scale. The solution is built on the Cisco UCS Integrated Infrastructure for Big Data and Analytics and includes computing, storage, network and unified management capabilities to help companies manage the vast amount of data they collect today. The Cisco Unified Computing System infrastructure uses Cisco UCS 6200/6300 Series Fabric Interconnects and Cisco UCS C Series Rack Servers. This architecture is specifically designed for performance and linear scalability for big data workloads. Audience This document describes the architecture and deployment procedures for the MapR Converged Data Platform on a 64 Cisco UCS C240 M4 node cluster based on Cisco UCS Integrated Infrastructure for Big Data and Analytics. The intended audience of this document includes, but is not limited to, sales engineers, field consultants, professional services, IT managers, partner engineering and customers who want to deploy the MapR Converged Data Platform on Cisco UCS Integrated Infrastructure for Big Data and Analytics. Solution Summary This CVD describes in detail the process of installing the MapR Converged Data Platform 5.1 and the configuration details of the cluster. It also details application configuration for MapR, and the installation of additional services, like Spark, MapR Steams, etc.


Page 8

Solution Overview 8 The current version of Cisco UCS Integrated Infrastructure for Big Data and Analytics offers the following configurations depending on the compute and storage requirements as shown in Table 1. Table 1 Cisco UCS Integrated Infrastructure for Big Data and Analytics Configuration Details Performance Optimized Option 1 (UCS SL CPA4 P1) Performance Optimized Option 2 (UCS SL CPA4 P2) Performance Optimized Option 3 (UCS SL CPA4 P3) Capacity Optimized Option 1 UCS SL CPA4 C1 Capacity Optimized Option 2 UCS SL CPA4 C2 2 Cisco UCS 6296 UP, 96 port Fabric Interconnect. 2 Cisco UCS 6296 UP, 96 port Fabric Interconnect. 2 Cisco UCS 6332 Fabric Interconnect. 2 Cisco UCS 6296 UP, 96 port Fabric Interconnect. 2 Cisco UCS 6296 UP, 96 port Fabric Interconnect. 16 Cisco UCS C240 M4 Rack Servers (SFF), each with: 16 Cisco UCS C240 M4 Rack Servers (SFF), each with: 16 Cisco UCS C240 M4 Rack Servers (SFF), each with: 16 Cisco UCS C240 M4 Rack Servers (LFF), each with: 16 Cisco UCS C240 M4 Rack Servers (LFF), each with: 2 Intel Xeon processors E5 2680 v4 CPUs (14 cores on each CPU) 2 Intel Xeon processors E5 2680 v4 CPUs (14 cores on each CPU) 2 Intel Xeon processors E5 2680 v4 CPUs (14 cores on each CPU) 2 Intel Xeon processors E5 2620 v4 CPUs (8 cores each CPU) 2 Intel Xeon processors E5 2620 v4 CPUs (8 cores each CPU) 256 GB of memory 256 GB of memory 256 GB of memory 128 GB of memory 256 GB of memory Cisco 12 Gbps SAS Modular Raid Controller with 2 GB flash based write cache (FBWC) Cisco 12 Gbps SAS Modular Raid Controller with 2 GB flash based write cache (FBWC) Cisco 12 Gbps SAS Modular Raid Controller with 2 GB flash based write cache (FBWC) Cisco 12 Gbps SAS Modular Raid Controller with 2 GB flash based write cache (FBWC) Cisco 12 Gbps SAS Modular Raid Controller with 2 GB flash based write cache (FBWC) 24 1.2 TB 10K SFF SAS drives (460 TB total) 24 1.8 TB 10K SFF SAS drives (691 TB total) 24 1.8 TB 10K SFF SAS drives (691 TB total) 12 6 TB 7.2K LFF SAS drives (1152 TB total) 12 8 TB 7.2K LFF SAS drives (1536 TB total) 2 240 GB 6 Gbps 2.5 inch Enterprise Value SATA SSDs for Boot 2 240 GB 6 Gbps 2.5 inch Enterprise Value SATA SSDs for Boot 2 240 GB 6 Gbps 2.5 inch Enterprise Value SATA SSDs for Boot 2 240 GB 6 Gbps 2.5 inch Enterprise Value SATA SSDs for Boot 2 240 GB 6 Gbps 2.5 inch Enterprise Value SATA SSDs for Boot Cisco UCS VIC 1227 (with 2 10 GE SFP+ ports) Cisco UCS VIC 1227 (with 2 10 GE SFP+ ports) Cisco UCS VIC 1387 (with 2 40 GE QSFP ports) Cisco UCS VIC 1227 (with 2 10 GE SFP+ ports) Cisco UCS VIC 1227 (with 2 10 GE SFP+ ports) MapR Converged Data Platform The MapR Converged Data Platform (Figure 1) integrates Hadoop and Spark with real time database capabilities, global event streaming and scalable enterprise storage to power a new generation of big data applications. The MapR Platform delivers enterprise grade security, reliability and real time performance while dramatically lowering both hardware and operational costs of your most important applications and data.


Page 9

Solution Overview 9 Figure 1The MapR Converged Data Platform MapR supports dozens of open source projects and is committed to using industry standard APIs to provide a frictionless method of developing and deploying new applications that can meet the most stringent production runtime requirements. Enterprise Grade Platform Services MapR Platform Services are the core data handling capabilities of the MapR Converged Data Platform. Modules include MapR FS, MapR DB and MapR Streams. Its enterprise friendly design provides a familiar set of file and data management services, including a global namespace, high availability, data protection, self healing clusters, access control, real time performance, secure multi tenancy, and management and monitoring. Open Source Engines and Tools MapR packages a broad set of Apache open source ecosystem projects that enable big data applications. The goal is to provide an open platform that provides the right tool for the job. MapR tests and integrates open source ecosystem projects such as Spark, Hive, Drill, HBase and Mesos, among others. Commercial Engines & Applications One of the key developer benefits of the MapR Converged Data Platform is its basis on well known, open APIs and interfaces. This enables commercial software vendors such as SAP Hana and SAS to easily deploy large scale applications onto the MapR Platform. It also means that even small teams of developers can create enterprise grade software products by exploiting the built in protections of the MapR Platform in combination with mature commercial processing engines. Lambda Architecture Combining Real time and Batch Processing Big data architectures are commonly separated into two mutually exclusive models: traditional batch processing using MapReduce and real time processing using a technology like Storm or Spark Streaming. Often, business requirements drive the adoption of one of these architectures and the popular way to combine these models has been to use the Lambda Architecture. This approach combines real time and batch layers providing the best of both worlds. It also has many additional benefits. The Lambda Architecture serves a wide range of workloads and use cases, including batch processing, interactive analysis and low latency real time processing, and also creates a robust system that is fault tolerant against hardware failures, software issues and human error, as well as being linearly scalable.


Page 10

Solution Overview 10 Figure 2The Lambda Architecture The Lambda Architecture as shown in Figure 2 has three major components. First, the Batch Layer manages the dataset, which is immutable and append only. Being immutable makes it easy to recover from software issues and human error; append only simplifies the database design and performance tuning. This layer also pre computes views of the data, called batch views, used to satisfy query requirements. Second, the Serving Layer indexes the batch views so that they can be queried with low latency, i.e., interactively and in an ad hoc fashion. Third, the Speed Layer handles all needs that require low latency. It uses fast, incremental algorithms that deal with recent data only. All real time stream data processing happens in the speed layer. Each of these layers can be implemented using various big data technologies. The batch layer datasets are stored in the distributed filesystem (MapR FS) and use MapReduce (or Spark) to create batch views. The serving layer uses NoSQL technologies like HBase. Lastly, the speed layer can be implemented using real time processing technologies like Storm or Spark Streaming. The MapR Converged Data Platform provides all the technologies to implement this architecture while also high speed streaming data can be written directly to the Hadoop storage while allowing the real time processing applications to run as independent services within the cluster. This creates a very resilient architecture. The real time processing applications become subscribers to the incoming data feeds. If the application goes down due to some failure, there is no data loss. A new instance of the application picks up the data stream where the original left off. MapR Reference Architecture Figure 3 shows the base configuration of 64 nodes with SFF (1.8TB) drives.


Tags: Download Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data ... PDF, Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data ... free pdf download, Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data ... Pdf online download, Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data ... 2016 download, Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data ....pdf, Cisco UCS Integrated Infrastructure for Big Data and Analytics with MapR Converged Data ... read online.
About | Contact | DMCA | Terms | Privacy | Mobile Specifications
Copyright 2021 FilePdf