scipipe.org

What It Is

SciPipe is a Go library for writing scientific workflows, aimed at scenarios where multiple command-line programs need to be combined into complex pipelines based on dependencies. It was originally created for bioinformatics and cheminformatics problems, but the documentation also states that it can be applied to any pipeline problem made up of command-line applications. It follows a Flow-Based Programming approach, where files flow between process nodes and nodes can run independently.

Core Capabilities

In terms of features and use cases, SciPipe emphasizes flexibility, robustness, and reproducibility. It can restart after interruptions without overwriting existing outputs, and it generates hierarchical JSON audit logs for each output file, recording the command, parameters, timestamp, execution duration, and upstream file history used to produce that file. It supports both pipeline-level parallelism and parallel execution of multi-input tasks, making use of multi-core CPUs. It also supports inter-process streaming to reduce wasted disk space. File naming is highly customizable, which suits research workflows that require traceable results and standardized naming conventions.

Language, Integration, and Documentation

SciPipe’s main interface is a Go API, including components such as NewWorkflow, NewProc, port connections, and Run. Users need to write workflows in Go, while still being able to wrap arbitrary command-line programs and create reusable Go components. Workflows can be distributed as Go code or compiled into self-contained executables, making them convenient to run locally or in compute environments. The documentation is relatively comprehensive, covering installation, Hello World, basic concepts, workflow writing, parameters, splitting/merging files, scatter/gather, HPC resource managers, resource constraints, workflow graphs, and video tutorials. However, the documentation explicitly states that Common Workflow Language is not yet supported, and some workflow design features are still incomplete.

Pricing and Open Source

The documentation does not mention commercial pricing or paid plans. The project is installed via its GitHub repository and go install, and provides entry points such as an Issue Tracker and Contributing guide, making it an open-source library. It is not a SaaS product, so there is no typical hosted subscription or payment flow. Self-hosting essentially means compiling and running it in the user’s own Go environment.

Pros, Cons, and Who It’s For

Its strengths include solid auditability, natural command-line integration, parallel and streaming processing that fit scientific computing well, and convenient binary deployment after Go compilation. The drawbacks are that users need to know Go, which may not be the easiest starting point for researchers accustomed to Python/R; the size of the ecosystem and current project activity should be verified further on GitHub. It is suitable for developers building reproducible, auditable, and compilable scientific pipelines, especially users in bioinformatics, cheminformatics, and high-performance batch processing. For access from China, the documentation provides no network-related information, and GitHub resources may be affected by local network conditions. If a broader ecosystem is a priority, it may be worth comparing with Nextflow, Snakemake, or CWL.