emr - MRJob - Limit Number of Task Attemps -


in myjob, how limit number of task attempts (if task fails)?

i have long running tasks (have increased timeout, accordingly), want job end after 2 failed attempts @ same task, rather 4-5.

i couldn't find in docs:

http://mrjob.readthedocs.org/en/latest//en/latest/guides/configs-reference.html

for map jobs, can set mapreduce.map.maxattempts in hadoop 2. reduce jobs, set mapreduce.reduce.maxattempts (source). equivalents in hadoop 1 are: mapred.map.max.attempts , mapred.reduce.max.attempts.

if using conf file in mrjob, can set as:

runners:     emr:         jobconf:             mapreduce.map.maxattempts: 2 

Comments

Popular posts from this blog

twig - Using Twigbridge in a Laravel 5.1 Package -

jdbc - Not able to establish database connection in eclipse -

Kivy: Swiping (Carousel & ScreenManager) -