https://www.bencode.io/categories/spark/ Recent content in Spark on Benny Simmonds Hugo -- 0.149.1 en-us Mon, 02 Jul 2018 19:41:10 +0000 https://www.bencode.io/posts/2018-07-02-spark/ Mon, 02 Jul 2018 19:41:10 +0000 https://www.bencode.io/posts/2018-07-02-spark/ <p>Recently I&rsquo;ve had the opportunity to dig into Apache Spark, thanks to some training from Brian Bloechle from Cloudera.</p> <p>What is spark?</p> <blockquote> <p>Fast, flexible, and developer friendly, Apache Spark is the leading platform for large scale SQL, batch processing, stream processing, and machine learning.</p></blockquote> <p><a href="https://spark.apache.org/docs/2.3.0/api/java/">Java</a>, <a href="https://spark.apache.org/docs/2.3.0/api/scala/#package">Scala</a>, <a href="https://spark.apache.org/docs/2.3.0/api/python/pyspark.html">Python</a> and <a href="https://spark.apache.org/docs/2.3.0/api/R/">R</a> are first class citizens when its comes to consuming the various Spark API&rsquo;s. I&rsquo;ll cover PySpark in more detail.</p> <p>Spark is an agnostic processing engine, that can target a number of cluster managers including Spark Standalone, Hadoop&rsquo;s YARN, Apache Mesos and Kubernetes. In the context of Spark, some useful surrounding ecosystem to be aware of:</p>