a lightweight parallel task engine

View the Project on GitHub Illumina/pyflow

pyFlow is a tool to manage tasks in the context of a task dependency graph. It has some similarities to make. pyFlow is not a program – it is a python module, and workflows are defined using pyFlow by writing regular python code with the pyFlow API.

Use case:

pyFlow has been optimized to be lightweight and simple to use for prototype/RD workflows.


Release Distributions:

All release tarballs are distributed on the pyflow releases page


pyflow's only requirement is python. pyflow is supported on python 2 versions 2.4+, except note that python 2.7.2 should not be used due to a critical multithread bug in the python interpreter which impacts many pyflow runs.

Getting Started

To use an existing pyflow workflow or develop a new one, you may need to download or generate the latest pyflow installation tarball (see top-level README.txt on git repository)

To develop a new pyflow workflow: Start by downloading the latest pyflow tarball (from version history section below). See pyflow/README.txt Look at the demo programs. If new to pyflow the recommended order is:

Example Workflow Graph:

Example workflow_task graph (rendered to pdf)

Developer Documentation: