Parallel run and write in python
Do a parallel imap_unordered with numProc processes
- func should always return something
- Tasks should be an iterable which items are acceptable args for func
- The output of func is loggedgradually in outFileName
- If bar is true, show a progress bar
- If errFileName is not None, all exceptions produced by func are intercepted and gradually logged in errFileName
- numProc sets the number of process to use
- if ordered is true, results are outputed in the same order as tasks
- chunksize : chunkszise for imap
- if debug is set to true, errors are also printed to stderr
How to use
As a subpackage
from pyParallelRunAndWrite import parallelRunAndWrite as prw
# The following dependencies are only useful for testFunc
import os
from time import sleep
from random import randint
def testFunc(arg):
ret=str(arg)+str(os.getpid())
sleep(randint(2,5))
if(randint(0,100)%2 ==0):
raise Exception("COUCOU"+ret)
return ret
prw.parallelRunAndWrite(testFunc, range(5), "testplop", numProc=2, errFileName="errors")By installing it
pip3 install -e .Then :
import parallelRunAndWrite as prw
# The following dependencies are only useful for testFunc
import os
from time import sleep
from random import randint
def testFunc(arg):
ret=str(arg)+str(os.getpid())
sleep(randint(2,5))
if(randint(0,100)%2 ==0):
raise Exception("COUCOU"+ret)
return ret
prw.parallelRunAndWrite(testFunc, range(5), "testplop", numProc=2, errFileName="errors")