Skip to content
Snippets Groups Projects
Select Git revision
  • bbf400ce0bf93ff9740b6aab9365903b0d676898
  • master default protected
  • dev
3 results

Readme.md

Blame
  • David Beniamine's avatar
    bbf400ce
    History

    Parallel run and write in python

    Do a parallel imap_unordered with numProc processes

    • func should always return something
    • Tasks should be an iterable which items are acceptable args for func
    • The output of func is loggedgradually in outFileName
    • If bar is true, show a progress bar
    • If errFileName is not None, all exceptions produced by func are intercepted and gradually logged in errFileName
    • numProc sets the number of process to use
    • if ordered is true, results are outputed in the same order as tasks
    • chunksize : chunkszise for imap
    • if debug is set to true, errors are also printed to stderr

    How to use

    from pyParallelRunAndWrite import parallelRunAndWrite as prw
    # The following dependencies are only useful for testFunc
    import os
    from time import sleep
    from random import randint
    
    def testFunc(arg):
        ret=str(arg)+str(os.getpid())
        sleep(randint(2,5))
        if(randint(0,100)%2 ==0):
            raise Exception("COUCOU"+ret)
        return ret
    
    prw.parallelRunAndWrite(testFunc, range(5), "testplop",  numProc=2, errFileName="errors")