Data Processing and Enrichment in Spark Streaming with Python and Kafka
In my previous blog post I introduced Spark Streaming and how it can be used to process 'unbounded' datasets. The example I did was a very »
Getting Started with Spark Streaming, Python, and Kafka
Last month I wrote a series of articles in which I looked at the use of Spark for performing data transformation and manipulation. This was in »
ETL Offload with Spark and Amazon EMR - Part 5 - Summary
This is the final article in a series documenting an exercise that we undertook for a client recently. Currently running an Oracle-based datawarehouse platform, the client »
ETL Offload with Spark and Amazon EMR - Part 3 - Running pySpark on EMR
In the previous articles (here, and here) I gave the background to a project we did for a client, exploring the benefits of Spark-based ETL processing »
ETL Offload with Spark and Amazon EMR - Part 2 - Code development with Notebooks and Docker
In the previous article I gave the background to a project we did for a client, exploring the benefits of Spark-based ETL processing running on Amazon's »
Using Jupyter Notebooks with Big Data Discovery 1.2
New in Big Data Discovery 1.2 is the addition of BDD Shell, an integration point with Python. This exposes the datasets and BDD functionality in »
Running Big Data Discovery Shell and Jupyter Notebook on Big Data Lite VM 4.5
New in Big Data Discovery 1.2 is the addition of BDD Shell, an integration point with Python. This exposes the datasets and BDD functionality in »