大數據運維圖解教程

大數據平臺運維是大數據應用人才培養的基本技能之一。本書講解了大數據平臺運維過程中的各個主要階段及其任務，主要包括安裝部署、優化監控、架構原理、生態系統、運維工具等。本書內容全面，涵蓋了 Hadoop 生態絕大部分組件的運維，兼顧基礎理論與運維實踐經驗。主要特色是將知識點凝練到圖形中，通過視覺記憶，看圖聯想，理解知識點的含義和知識點之間的聯系，以達到長時記憶、學以致用的目的。通過 800 多道習題，加強系統性和實踐指導性。本書按照國家“1+X”大數據平臺運維職業技能等級標準編寫，可作為“1+X”職業技能等級證書配套教材，可作為《大數據導論》《大數據運維》的課程教材，同時也適合有意向從事大數據應用工作的廣大學習者和愛好者閱讀。

目 錄

第1 章 大數據運維概述

1.1 從大數據說起 ····················································································· 1

1.1.1 大數據產生、影響及挑戰 ······································································ 1

1.1.2 大數據概念、特徵及價值 ····································································· 14

1.1.3 大數據技術、產業及應用 ····································································· 18

1.2 大數據技術生態 ················································································· 24

1.2.1 分佈式集群概述 ················································································· 24

1.2.2 Hadoop 生態系統 ················································································ 30

1.3 大數據運維 ······················································································· 37

第2 章 集群基礎環境搭建與運維

2.1 集群規劃 ·························································································· 41

2.2 虛擬機與Linux ·················································································· 44

2.3 網絡配置與時鐘同步 ··········································································· 55

2.4 免密 ································································································ 60

2.5 JDK 和MySQL 部署 ··········································································· 62

第3 章 分佈式集群Hadoop 運維

3.1 Hadoop 平臺安裝與部署 ····································································· 69

3.1.1 Hadoop 安裝準備 ················································································ 69

3.1.2 Hadoop 安裝與部署 ············································································· 71

3.2 分佈式存儲組件HDFS ········································································ 75

3.2.1 HDFS 結構 ······················································································· 75

3.2.2 HDFS 讀寫原理 ················································································· 83

3.2.3 HDFS 操作與監控 ·············································································· 86

3.3 離線分佈式計算引擎MapReduce ·························································· 90

3.3.1 MapReduce 結構與原理 ········································································ 90

3.3.2 MapReduce 部署與優化 ······································································ 100

3.4 集群資源管理YARN ········································································· 101

3.4.1 YARN 的結構與原理 ········································································· 101

3.4.2 YARN 部署、優化與監控 ··································································· 105

第4 章 分佈式系統協調器Zookeeper

4.1 Zookeeper 結構 ··············································································· 108

IV 大數據運維圖解教程

4.2 Zookeeper 原理 ··············································································· 111

4.3 Zookeeper 部署與優化 ······································································ 117

第5 章 數據採集組件運維

5.1 日誌採集組件Flume ········································································· 121

5.2 數據遷移組件Sqoop ········································································ 128

5.3 發布訂閱消息組件Kafka ··································································· 132

第6 章 數據處理組件運維

6.1 NoSQL 數據庫HBase ······································································ 137

6.1.1 NoSQL ·························································································· 137

6.1.2 HBase 架構與原理 ············································································ 141

6.1.3 HBase 部署與優化 ············································································ 148

6.2 數據倉庫引擎Hive ··········································································· 153

6.2.1 Hive 架構與原理 ·············································································· 154

6.2.2 Hive 部署與優化 ·············································································· 166

6.3 內存計算引擎Spark ········································································· 169

6.3.1 Spark 生態 ······················································································ 169

6.3.2 Spark 架構與原理 ············································································· 173

6.3.3 Spark 部署與優化 ············································································· 179

第7 章 大數據平臺運維工具

7.1 Ambari 大數據環境搭建利器 ······························································ 187

7.1.1 安裝 ······························································································ 187

7.1.2 使用 ······························································································ 192

7.2 Hive SQL 監控系統—Hive Falcon ···················································· 198

7.2.1 安裝 ······························································································ 198

7.2.2 使用 ······························································································ 199

7.3 統一日誌監控系統EKL ····································································· 202

7.3.1 安裝 ······························································································ 202

7.3.2 使用 ······························································································ 207

參考文獻

附錄A Web 監控埠

附錄B 大數據平臺運維任務

附錄C 大數據運維三十六計