Spark Shuffle Introduction-蘑菇街.pdf
《Spark Shuffle Introduction-蘑菇街.pdf》由会员分享,可在线阅读,更多相关《Spark Shuffle Introduction-蘑菇街.pdf(33页珍藏版)》请在三一文库上搜索。
1、SPARK SHUFFLE INTRODUCTION 天火蘑菇街 About Me Spark / Hadoop / Hbase / Phoenix contributor For spark mainly contributes in: Yarn Shuffle BlockManager Scala2.10 update Standalone HA Various other fixes. Weibo 冷冻蚂蚁 Why Spark is fast(er) Whom do we compare to? What do we mean by fast? fast to write fast
2、to run Why Spark is fast(er) cont. But the figure in previous page is some how misleading. The key is the flexible programming mode. Which lead to more reasonable data flow. Which lead to less IO operation. Especially for iterative heavy workloads like ML. Which potentially cut off a lot of shuffle
3、operations needed. But, you wont always be lucky. Many app logic did need to exchange a lot of data. In the end, you will still need to deal with shuffle And which usually impact performance a lot. What is shuffle 7 Shuffle overview Aggregator Aggregator Aggregator Aggregator Aggregator Aggregator H
4、ow does shuffle come into the picture Spark run job stage by stage. Stages are build up by DAGScheduler according to RDDs ShuffleDependency e.g. ShuffleRDD / CoGroupedRDD will have a ShuffleDependency Many operator will create ShuffleRDD / CoGroupedRDD under the hook. Repartition/CombineByKey/GroupB
5、y/ReduceByKey/cogroup many other operator will further call into the above operators e.g. various join operator will call cogroup. Each ShuffleDependency maps to one stage in Spark Job and then will lead to a shuffle. So everyone should have seen this before join union groupBy map Stage 3 Stage 1 St
6、age 2 A: B: C: D: E: F: G: why shuffle is expensive When doing shuffle, data no longer stay in memory only For spark, shuffle process might involve data partition: which might involve very expensive data sorting works etc. data ser/deser: to enable data been transfer through network or across proces
7、ses. data compression: to reduce IO bandwidth etc. DISK IO: probably multiple times on one single data block E.g. Shuffle Spill, Merge combine History Spark 0.6-0.7, same code path with RDDs persistent method, can choose MEMORY_ONLY and DISK_ONLY (default). Spark 0.8-0.9: separate shuffle code path
8、from BM and create ShuffleBlockManager and BlockObjectWriter only for shuffle, now shuffle data can only be written to disk. Shuffle optimization: Consolidate shuffle write. Spark 1.0, pluggable shuffle framework. Spark 1.1, sort-based shuffle implementation. Spark 1.2 netty transfer service reimple
9、mentation. sort- based shuffle by default Spark 1.2+ on the go: external shuffle service etc. LOOK INSIDE 13 Pluggable Shuffle Framework ShuffleManager Manage shuffle related components, registered in SparkEnv, configured through SparkConf, default is sort (pre 1.2 is hash), ShuffleWriter Handle shu
10、ffle data output logics. Will return MapStatus to be tracked by MapOutputTracker. ShuffleReader Fetch shuffle data to be used by e.g. ShuffleRDD ShuffleBlockManager Manage the mapping relation between abstract bucket and materialized data block. High level data flow BlockManager HashShuffleManager D
11、iskBlockManager FileShuffleBlockManager Local File System SortShuffleManager IndexShuffleBlockManager GetBlockData BlockTransferService GetBlockData Direct mapping or mapping by File Groups Map to One Data File and One Index File per mapId Just do one-one File mapping 15 Hash Based Shuffle - Shuffle
12、 Writer Basic shuffle writer Map TaskMap TaskMap TaskMap Task File File File File File File File File File File File File File File File File Aggregator Aggregator Aggregator Aggregator Each bucket is mapping to a single file 16 Hash Based Shuffle - Shuffle Writer Consolidate Shuffle Writer Each buc
13、ket is mapping to a segment of file Aggregator Aggregator Aggregator Aggregator 17 Hash Based Shuffle - Shuffle Writer Basic Shuffle Writer M * R shuffle spill files Concurrent C * R opened shuffle files. If shuffle spill enabled, could generate more tmp spill files say N. Consolidate Shuffle Writer
14、 Reduce the total spilled files into C * R if (M C) Concurrent opened is the same as the basic shuffle writer. Memory consumption Thus Concurrent C * R + N file handlers. Each file handler could take up to 32100KB+ Memory for various buffers across the writer stream chain. 18 Sort Based Shuffle - Sh
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- Spark Shuffle Introduction-蘑菇街 Introduction 蘑菇
链接地址:https://www.31doc.com/p-3331036.html