nipype.pipeline.plugins.legacymultiproc module

Parallel workflow execution via multiprocessing

Support for child processes running as non-daemons based on http://stackoverflow.com/a/8963618/1183453

class nipype.pipeline.plugins.legacymultiproc.LegacyMultiProcPlugin(plugin_args=None)

Bases: nipype.pipeline.plugins.base.DistributedPluginBase

Execute workflow with multiprocessing, not sending more jobs at once than the system can support.

The plugin_args input to run can be used to control the multiprocessing execution and defining the maximum amount of memory and threads that should be used. When those parameters are not specified, the number of threads and memory of the system is used.

System consuming nodes should be tagged:

memory_consuming_node.mem_gb = 8
thread_consuming_node.n_procs = 16

The default number of threads and memory are set at node creation, and are 1 and 0.25GB respectively.

Currently supported options are:

  • non_daemon : boolean flag to execute as non-daemon processes
  • n_procs: maximum number of threads to be executed in parallel
  • memory_gb: maximum memory (in GB) that can be used at once.
  • raise_insufficient: raise error if the requested resources for
    a node over the maximum n_procs and/or memory_gb (default is True).
  • scheduler: sort jobs topologically ('tsort', default value)
    or prioritize jobs by, first, memory consumption and, second, number of threads ('mem_thread' option).
  • maxtasksperchild: number of nodes to run on each process before
    refreshing the worker (default: 10).
class nipype.pipeline.plugins.legacymultiproc.NonDaemonForkContext

Bases: multiprocessing.context.ForkContext

Process

alias of NonDaemonForkProcess

class nipype.pipeline.plugins.legacymultiproc.NonDaemonForkProcess(group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)

Bases: nipype.pipeline.plugins.legacymultiproc.NonDaemonMixin, multiprocessing.context.ForkProcess

class nipype.pipeline.plugins.legacymultiproc.NonDaemonForkServerContext

Bases: multiprocessing.context.ForkServerContext

Process

alias of NonDaemonForkServerProcess

class nipype.pipeline.plugins.legacymultiproc.NonDaemonForkServerProcess(group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)

Bases: nipype.pipeline.plugins.legacymultiproc.NonDaemonMixin, multiprocessing.context.ForkServerProcess

class nipype.pipeline.plugins.legacymultiproc.NonDaemonMixin

Bases: object

daemon
class nipype.pipeline.plugins.legacymultiproc.NonDaemonPool(processes=None, initializer=None, initargs=(), maxtasksperchild=None, context=None)

Bases: multiprocessing.pool.Pool

class nipype.pipeline.plugins.legacymultiproc.NonDaemonSpawnContext

Bases: multiprocessing.context.SpawnContext

Process

alias of NonDaemonSpawnProcess

class nipype.pipeline.plugins.legacymultiproc.NonDaemonSpawnProcess(group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)

Bases: nipype.pipeline.plugins.legacymultiproc.NonDaemonMixin, multiprocessing.context.SpawnProcess

nipype.pipeline.plugins.legacymultiproc.process_initializer(cwd)

Initializes the environment of the child process

nipype.pipeline.plugins.legacymultiproc.run_node(node, updatehash, taskid)

Function to execute node.run(), catch and log any errors and return the result dictionary

Parameters:
  • node (nipype Node instance) – the node to run
  • updatehash (boolean) – flag for updating hash
  • taskid (int) – an identifier for this task
Returns:

result – dictionary containing the node runtime results and stats
Return type:

dictionary