Debugging : Hive Dynamic partition Error : [Fatal Error] total number of created files now is 100028, which exceeds 100000. Killing the job.

[Fatal Error] total number of created files now is 900320, which exceeds 900000. Killing the job. tldr; quick fix – but probably not the right thing to do always: SET hive.exec.max.created.files=900000; So my config increases the default partitions and files created limit: set hive.exec.dynamic.partition=true; set hive.exec.max.dynamic.partitions=100000; SET hive.exec.max.dynamic.partitions.pernode=100000; set hive.exec.dynamic.partition.mode=nonstrict; SET hive.exec.max.created.files=900000; Correct thing to do: …

More