Debugging : Hive Dynamic partition Error : [Fatal Error] total number of created files now is 100028, which exceeds 100000. Killing the job.

[Fatal Error] total number of created files now is 900320, which exceeds 900000. Killing the job. tldr; quick fix – but probably not the right thing to do always: SET hive.exec.max.created.files=900000; So my config increases the default partitions and files created limit: set hive.exec.dynamic.partition=true; set hive.exec.max.dynamic.partitions=100000; SET hive.exec.max.dynamic.partitions.pernode=100000; set hive.exec.dynamic.partition.mode=nonstrict; SET hive.exec.max.created.files=900000; Correct thing to do: …

More

Hive – Selected data import/query – Files and folders (mapred.input.dir.recursive)

Data import in Hive by default expects a directory name in its query specified by LOCATION keyword. By default Hive picks up all the files from the dir and imports into itself. If the directory does not contain files, rather consists of sub directories Hive blows up with the exception: java.io.IOException:java.io.IOException: Not a file: /path/to/data/* …

More