Package src :: Module pyflow

[frames] | no frames]

Module pyflow

source code

pyflow -- a lightweight parallel task engine

Version: v1.1.15

Author: Christopher Saunders

Classes

[hide private]

GlobalSync
Control total memory usage in non-local run modes by limiting the number of simultaneous subprocess calls

GlobalConstants

LogState
A simple logging enum

LogGlobals

StrFileObject
fakes a filehandle for library functions which write to a stream, and captures output in a string

ExpWaiter
Convenience object to setup exponentially increasing wait/polling times

Bunch
generic struct with named argument constructor

TaskNodeConstants

DotConfig
A static container of configuration data for dot graph output

Command
Commands can be presented as strings or argument lists (or none)

StoppableThread
Thread class with a stop() method.

ModeInfo
Stores default values associated with each runmode: local,sge,...

RunMode

RetryParam
parameters pertaining to task retry behavior

RunningTaskStatus
simple object allowing remote task threads to communicate their status back to the TaskManager

BaseTaskRunner
Each individual command-task or sub workflow task is run on its own thread using a class inherited from BaseTaskRunner

WorkflowTaskRunner
Manages a sub-workflow task

CommandTaskRunner
Parent to local and SGE TaskRunner specializations for command tasks

LocalTaskRunner

QCaller
Calls to both qsub and qstat go through this run() method so that we can time them out:

SGETaskRunner

TaskFileWriter
This class runs on a separate thread and is responsible for updating the state and info task files

TaskManager
This class runs on a separate thread from workflowRunner, launching jobs based on the current state of the TaskDAG

CmdPayload

WorkflowPayload

TaskNode
Represents an individual task in the task graph

TaskDAG
Holds all tasks and their dependencies.

DataDirException

WorkflowRunnerThreadSharedData
All data used by the WorkflowRunner which will be constant over the lifetime of a TaskManager instance.

WorkflowRunner
This object is designed to be inherited by a class in client code.

Functions

[hide private]

getPythonVersion()

source code

getPyflowVersion()

source code

_isWindows()

source code

isWindows()

source code

forceRename(src, dst)
dst is only overwritten in a single atomic operation on *nix on windows, we can't have atomic rename, but we can recreate the behavior otherwise

source code

cleanEnv()
clear bash functions out of the env

source code

ensureDir(d)
make directory if it doesn't already exist, raise exception if something else is in the way:

source code

timeStampToTimeStr(ts)
converts time.time() output to timenow() string

source code

timeStrNow()

source code

timeStrToTimeStamp(ts)

source code

isInt(x)

source code

isString(x)

source code

isIterable(x)

source code

lister(x)
Convert input into a list, whether it's already iterable or not.

source code

setzer(x)
convert user input into a set, handling the pathological case that you have been handed a single string, and you don't want a set of letters:

source code

hardFlush(ofp)

source code

log(ofpList, msgList, linePrefix=None)
General logging function.

source code

getThreadName()

source code

isMainThread()

source code

getTracebackStr()

source code

getExceptionMsg()

source code

cmdline()

source code

msgListToMsg(msgList)
convert string or list of strings into a single string message

source code

verifyEmailAddy(x)

source code

isLocalSmtp()
return true if a local smtp server is available

source code

sendEmail(mailTo, mailFrom, subject, msgList)

source code

boolToStr(b)

source code

argToBool(x)
convert argument of unknown type to a bool:

source code

hashObjectValue(obj)
This function hashes objects values -- the hash will be the same for two objects containing the same methods and data, so it corresponds to 'A==B' and *not* 'A is B'.

source code

namespaceJoin(a, b)
join two strings with a separator only if a exists

source code

namespaceLabel(namespace)
provide a consistent naming scheme to users for embedded workflows

source code

lockMethod(f)
method decorator acquires/releases object's lock

source code

stackDump(dumpfp)
adapted from haridsv @ stackoverflow:

source code

taskStateHeader()

source code

taskStateParser(stateFile)

source code

taskInfoHeader()

source code

taskInfoParser(infoFile)

source code

getTaskInfoDepSet(s)

source code

writeDotGraph(taskInfoFile, taskStateFile, workflowClassName)
write out the current graph state in dot format

source code

writeDotScript(taskDotScriptFile, taskInfoFileName, taskStateFileName, workflowClassName)
write dot task graph creation script

source code

getSGEJobsDefault()

source code

Variables

[hide private]

moduleDir = '/home/csaunders/devel/pyflow/pyflow/src'

pyver = sys.version_info(major=2, minor=7, micro=12, releasele...

pythonVersion = '2.7.12.final.0'

emailRegex = re.compile(r'(?i)(?:^|\s)[-a-z0-9_\.]+@(?:[-a-z0-...

namespaceSep = '+'

__package__ = 'src'

Function Details

[hide private]

cleanEnv()

source code

clear bash functions out of the env

without this change the shellshock security update causes pyflow SGE jobs to fail with the behavior of current (201512) versions of SGE qsub

lister(x)

source code

Convert input into a list, whether it's already iterable or not. Make an exception for individual strings to be returned as a list of one string, instead of being chopped into letters Also, convert None type to empty list:

log(ofpList, msgList, linePrefix=None)

source code

General logging function.

Parameters:

ofpList - A container of file objects to write to
msgList - A container of (or a single) multi-line log message string. Final newlines are not required
linePrefix - A prefix to add before every line. This will come *after* the log function's own '[time] [hostname]' prefix.

Returns:

Returns a boolean tuple of size ofpList indicating the success of writing to each file object

Variables Details

[hide private]

pyver

Value:

sys.version_info(major=2, minor=7, micro=12, releaselevel='final', ser
ial=0)

emailRegex

Value:

re.compile(r'(?i)(?:^|\s)[-a-z0-9_\.]+@(?:[-a-z0-9]+\.)+[a-z]{2,6}(?:\
s|$)')