Hey Readers, I am a Data Science Student and recently I have started learning more about Data Engineering. Data Science and Data Engineering teams co-exist in all data organisations. The Data Scientists are responsible for getting insights from data and applying machine learning on data, while the Data Engineers help the Data Scientist get access …
In the last post we discussed about how to generate synthetic data. Here we will talk about how to query S3 data via Hive. Provide AWS configuration to Hadoop and Hive We need to add the following configuration to the Hadoop and Hive config files. hive-site.xml You can find hive-site.xml in HIVE_HOME. You can find …
This is part-2 of the blog series – How to analyze Kaggle data with Apache Spark and Zeppelin. In the first part we saw how to copy Kaggle data to Amazon S3. We would now like to analyze our data on EMR. I’m choosing Spark and Zeppelin for this task. What is EMR ? Amazon EMR …
[Fatal Error] total number of created files now is 900320, which exceeds 900000. Killing the job. tldr; quick fix – but probably not the right thing to do always: SET hive.exec.max.created.files=900000; So my config increases the default partitions and files created limit: set hive.exec.dynamic.partition=true; set hive.exec.max.dynamic.partitions=100000; SET hive.exec.max.dynamic.partitions.pernode=100000; set hive.exec.dynamic.partition.mode=nonstrict; SET hive.exec.max.created.files=900000; Correct thing to do: …
Data import in Hive by default expects a directory name in its query specified by LOCATION keyword. By default Hive picks up all the files from the dir and imports into itself. If the directory does not contain files, rather consists of sub directories Hive blows up with the exception: java.io.IOException:java.io.IOException: Not a file: /path/to/data/* …