Tag: Spark
-
Day 9
Spark (Con’t) Execution step Code written, execute Driver Program launch Cluster manager provision resources Create a job using DAG as a pipeline (aka RDD lineage) Each item is call Task :: Partition and distributed to worker node Program are distributed into worker node Tutorial 9 Tutorial 10
-
Day 8
Spark Cluster computing framework, processing framework using memory. To overcome map reduce performance issue. Cluster manager resource allocation base on job (request). Spark create Resilient Distributed Datasets (RDD) :: partition once it receive the data. Once RDD is ready, it uses graph transformation (Directed Acyclic Graph DAG) Consists of 2 phases: Transformation and Action **…
