呼延:Simple Yet Powerful Hadoop.pdf
《呼延:Simple Yet Powerful Hadoop.pdf》由会员分享,可在线阅读,更多相关《呼延:Simple Yet Powerful Hadoop.pdf(43页珍藏版)》请在三一文库上搜索。
1、Leslie Huyan VP, Internet Technology, Schmap Inc. Hadoop Community Contributor Simple Yet Powerful Hadoop Skills biggest cluster: 2000 nodes (2*4cpu boxes with 4TB disk each); used to support research for Ad Systems and Web Search Facebook: To store copies of internal log and dimension data source
2、s and use it as a source for reporting/analytics and machine learning; 320 machine cluster with 2,560 cores and about 1.3 PB raw storage; Hadoop Deep Insights Hadoop core consists of HDFS Map/Reduce What Is HDFS A file system Fundamental underlying HDFS - Structure Files split into 128MB blocks Bloc
3、ks replicated across several datanodes (usually 3) Single namenode stores metadata (file names, block locations, etc) Optimized for large files, sequential reads Files are append-only Namenode Datanodes 1 2 3 4 1 2 4 2 1 3 1 4 3 3 2 4 File1 18 Namenode B replication Rack1 Rack2 Client Blocks Datanod
4、es Datanodes Client Write Read Metadata ops Metadata(Name, replicas) (/home/foo/data,6. Block ops Double Masters - Optimization HDFS - Data Replication HDFS is designed to store very large files across machines in a large cluster. Each file is a sequence of blocks. All blocks in the file except the
5、last are of the same size. Blocks are replicated for fault tolerance. Block size and replicas are configurable per file. The Namenode receives a Heartbeat and a BlockReport from each DataNode in the cluster. BlockReport contains all the blocks on a Datanode. HDFS - Replica Placement The placement of
6、 the replicas is critical to HDFS reliability and performance. Optimizing replica placement distinguishes HDFS from other distributed file systems. Rack-aware replica placement: Goal: improve reliability, availability and network bandwidth utilization Research topic Many racks, communication between
7、 racks are through switches. Network bandwidth between machines on the same rack is greater than those in different racks. Namenode determines the rack id for each DataNode. Replicas are typically placed on unique racks Simple but non-optimal Writes are expensive Replication factor is 3 Another rese
8、arch topic? Replicas are placed: one on a node in a local rack, one on a different node in the local rack and one on a node in a different rack. 1/3 of the replica on a node, 2/3 on a rack and 1/3 distributed evenly across remaining racks. HDFS API HDFS provides Java API for application to use. Pyth
9、on access is also used in many applications. A C language wrapper for Java API is also available. A HTTP browser can be used to browse the files of a HDFS instance. Map/Reduce Framework provided by Hadoop to process large amount of data across a cluster of machines in a parallel manner Comprises of
10、three classes Mapper class Reducer class Driver class Tasktracker/ Jobtracker Reducer phase will start only after mapper is done Takes (k,v) pairs and emits (k,v) pair MapReduce: High Level JobTracker MapReduce job submitted by client computer Master node TaskTracker Slave node Task instance TaskTra
11、cker Slave node Task instance TaskTracker Slave node Task instance In our case: circe.rc.usf.edu Map/Reduce job flow MapReduce Terminology Job A “full program” - an execution of a Mapper and Reducer across a data set Task An execution of a Mapper or a Reducer on a slice of data a.k.a. Task-In-Progre
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- 呼延:Simple Yet Powerful Hadoop 呼延 Simple
链接地址:https://www.31doc.com/p-3332733.html