Rittman Mead Blog

An Introduction to Apache Drill

Apache Drill [https://drill.apache.org/] is an engine that can connect to many different data sources, and provide a SQL interface to them. It's not just a wanna-be SQL interface that trips over at anything complex - it's a hugely functional one [https://drill.apache.

Big Data

Using SparkSQL and Pandas to Import Data into Hive and Big Data Discovery

Big Data Discovery [https://www.oracle.com/big-data/big-data-discovery/index.html] (BDD) is a great tool for exploring, transforming, and visualising data stored in your organisation’s Data Reservoir. I presented a workshop on it at a recent conference [https://speakerdeck.com/rmoff/unlock-the-value-in-your-big-data-reservoir-using-oracle-big-data-discovery-and-oracle-big-data-spatial-and-graph] , and got an interesting question from

Big Data

Forays into Kafka - Enabling Flexible Data Pipelines

One of the defining features of “Big Data” from a technologist’s point of view is the sheer number of tools and permutations at one’s disposal. Do you go Flume or Logstash? Avro or Thrift? Pig or Spark? Foo or Bar? (I made that last one up). This wealth

Technical

Trickle-Feeding Log Files to HDFS using Apache Flume

In some previous articles on the blog I’ve analysed Apache webserver log files sitting on a Hadoop cluster using Hive [https://www.rittmanmead.com/blog/2014/04/simple-data-manipulation-and-reporting-using-hive-impala-and-cdh5/] , Pig [https://www.rittmanmead.com/blog/2014/05/simple-hadoop-dataflows-using-apache-pig-and-cdh4-6/] and most recently, Apache Spark [https://www.rittmanmead.com/blog/2014/05/