In our previous post, we discussed about getting started with knowledge graph where we only saw how to install neo4j in Docker. In this post, we… Read More »Knowledge Graph part-2 : Modelling tabular data as graph
Hey readers, I am learning Data Engineering from last few months and I thought of sharing my learning with you all. Recently I made a project… Read More »Real world application project for Big Data – with Apache Spark and AWS-EMR
Hey readers, in previous post I have explained How to create a python ETL Project. In this post, I will explain how we can schedule/productionize our… Read More »Data Engineering Part 2 – Productionizing Big data ETL with Apache Airflow
Hey Readers, I am a Data Science Student and recently I have started learning more about Data Engineering. Data Science and Data Engineering teams co-exist… Read More »Data Engineering Part 1 – How to become a Big Data Engineer
In the last post we discussed about how to generate synthetic data. Here we will talk about how to query S3 data via Hive. Provide… Read More »Query S3 data via Hive on local box
It was an eventful evening yesterday when I gave my first ever talk. I talked on Analytics with Apache Spark and Zeppelin on Amazon… Read More »Public Speaking at Web Analytics Wednesday Meetup, Sydney
Here is a collection of spark configs that have helped make the job runs faster. Most of the configs come with trade-offs but work very… Read More »Handpicked Spark configs to make the job runs faster
This is a 3 post series on querying Kaggle data on EMR cluster. I will be using Apache Zeppein for the data exploration, and internally… Read More »Query Kaggle data via Apache Spark and Zeppelin via EMR cluster
This is part-3 of the blog series – How to analyze Kaggle data with Apache Spark and Zeppelin. In the first part we saw how to copy Kaggle data… Read More »Part 3: Query Kaggle data via Apache Zeppelin
This is part-2 of the blog series – How to analyze Kaggle data with Apache Spark and Zeppelin. In the first part we saw how… Read More »Part 2: How to create EMR cluster with Apache Spark and Apache Zeppelin
This is part-1 of the blog series — How to analyze Kaggle data with Apache Spark and Zeppelin. This post provides a brief description on how to… Read More »Part 1: How to copy Kaggle data to Amazon S3
Sometimes we quickly need to check the schema of a parquet file, or to head the parquet file for some sample records. Here are some… Read More »How to view content of parquet files on S3/HDFS from Hadoop cluster using parquet-tools