[Fatal Error] total number of created files now is 900320, which exceeds 900000. Killing the job.
tldr; quick fix – but probably not the right thing to do always:
So my config increases the default partitions and files created limit:
set hive.exec.dynamic.partition=true; set hive.exec.max.dynamic.partitions=100000; SET hive.exec.max.dynamic.partitions.pernode=100000; set hive.exec.dynamic.partition.mode=nonstrict; SET hive.exec.max.created.files=900000;
Correct thing to do:
Investigate why Hive is creating these many files. Most partitions should be within 100000, so hitting the limit sounds suspicious.
This happens when we misplace the wrong columns in the partition column and the wrong data creates thousands of useless partitions. Verify the query && check output data location to check what exact files and partitions are written to S3/Hdfs.
If everything looks alright and you think you genuinely need to increase the partitions goto the above tip to increate the max file limit.
Yash Sharma is a Big Data & Machine Learning Engineer, A newbie OpenSource contributor, Plays guitar and enjoys teaching as part time hobby.
Talk to Yash about Distributed Systems and Data platform designs.