Quaich
is a
snakemake
based workflow for reproducible and flexible analysis of Hi-C data. Quaich uses multi-resolution
cooler
(.mcool) files as its input. These files can be generated efficiently by the
distiller
data processing pipeline.
Quaich
takes advantage of the
open2c
ecosystem for analysis of C data, primarily making use of command line tools from
cooltools
.
Quaich
also makes use of
chromosight
and
mustache
to call Hi-C peaks (peaks, dots) as well as
coolpuppy
to generate lots of pileups.
Snakemake
is a workflow manager for reproducible and scalable data analyses, based around the concept of rules. Rules used in
Quaich
are defined in the
Snakefile
.
Quaich
then uses a
yaml config file
to specify which rules to run, and which parameters to use for those rules.
Each rule in the
Quaich
Snakefile specifies inputs, outputs, resources, and threads. The best values for resources and threads depend on whether
quaich
is run locally or on a cluster. Outputs of one rule are often used as inputs to another. For example, the rule
make_expected_cis
calls
cooltools compute-expected
on a mcool for a set of regions at specified resolutions to output tsv. This output is then used in make_saddles, make_pileups, and call_loops_cooltools. For reprodcibility and easy setup wherever possible, the rules use
snakemake wrappers
instead of using shell/python code directly. This means every rule will have its own dedicated conda environment that is defined as part of the wrapper, and it is created the first time the pipeline is run.
rule make_expected_cis: input: cooler=lambda wildcards: coolfiles_dict[wildcards.sample], view=lambda wildcards: config["view"], output: f"{expected_folder}/{{sample}}_{{resolution,[0-9]+}}.expected.tsv", params: extra=lambda wildcards: config["expected"]["extra_args_cis"], threads: 4 resources: mem_mb=lambda wildcards, threads: threads * 8 * 1024, runtime=60, wrapper: "v2.0.0/bio/cooltools/expected_cis"
Quaich
groups similar rules together in config.yaml, which is read as a python dictionary by the Snakefile. Parameters for individual rules are passed as indented (key, value) pairs. For example, call_dots configures three methods of calling dots in Hi-C: cooltools, chromosight, and mustache. The parameters for each specific rule are underneath, the shared parameters are below (in this case, the resolutions, and whether to generate pileups for the dot calls).
Do
always specifies if the workflow should attempt to produce the output for this rule. In most cases there is an
extra
argument, where any arbitrary command line arguments not exposed through the YAML can be passed to the underlying tool.
call_dots:
methods:
cooltools:
do: True
extra: "--max-loci-separation 10000000 --fdr 0.02"
chromosight:
do: True
extra: ""
mustache:
do: True
max_dist: 10000000
extra: "-pt 0.05 -st 0.8"
resolutions:
- 10000
pileup: True
quaich
config.yaml has four main sections:
-
genome
-
samples and annotations Here you can set up what comparisons between samples to perform. The .tsv file with samples can contain any arbitrary columns, and you can provide names of those columns to
fields_do_differ
andfields_to_match
, this will generate all combinations of samples to compare. For example, to compare different cell types the default config contains
fields_to_match: null
fields_to_differ:
cell_type: hESCs
This will not try to match any specific fields between samples, but will ensure that for comparisons the value of the cell_type column is different between samples, and the
hESCs
data will be used as the "reference" dataset, with the rest compared to this one.
If the samples.tsv file had a
sex
column, we could ensure to compare only samples on the same sex by providing
fields_to_match: sex
-
i/o Here you can configure where the inputs are located and where the results are saved.
-
snakemake rule configurations
The following analyses can be configured:
-
expected: calculates cis and trans expected contact frequencies using cooltools
-
eigenvector: calculates cis eigenvectors using cooltools for all resolutions within specified resolution_limits .
-
saddle: calculates saddles, reflecting average interaction preferences, from cis eigenvectors for each sample using cooltools.
-
pentads: calculates pentads, as another way to visualize and quantify compartment strength (https://doi.org/10.1186/s12859-022-04654-6)
-
pileups: does pretty much any pileup analysis imaginable using coolpup.py :) (https://doi.org/10.1093/bioinformatics/btaa073)
-
call_dots: three methods of calling dots, at specified resolutions, and postprocess output to standardized bedpe format. Implemented callers are cooltools, mustache and chromosight. Which samples are used can be defined in the samples.tsv configuration file.
-
insulation: calculates diamond insulation score for specified resolutions and window sizes, using cooltools.
-
compare_boundaries: generates differential boundaries, used as input for pileups.
-
call_TADs: combines lists of strong boundaries for specified samples into a list across window sizes for each resolution, filtered by length, used as input for pileups. Which samples are used can be defined in the samples.tsv configuration file.
Authors
- Ilya Flyamer (@phlya)
Usage
If you use this workflow in a paper, don't forget to give credits to the authors by citing the URL of this (original) repository.
Step 1: Obtain a copy of this workflow
-
Create a new github repository using this workflow as a template .
-
Clone the newly created repository to your local system, into the place where you want to perform the data analysis.
Step 2: Configure workflow
Configure the workflow according to your needs via editing the files in the
config/
folder. Adjust
config.yaml
to configure the workflow execution, and
samples.tsv
to specify your sample setup. If you want to use any external bed or bedpe files for pileups, describe them in the
annotations.tsv
file, and pairings of samples with annotations in
samples_annotations.tsv
.
Step 3: Install Snakemake and other requirements
Install Snakemake and other requirements using conda :
conda env create -f workflow/envs/environment.yml
This will create an environment
quaich
where you can launch the pipeline.
For Snakemake installation details, see the instructions in the Snakemake documentation .
Step 4: Execute workflow
Activate the conda environment:
conda activate quaich
Test your configuration by performing a dry-run via
snakemake --use-conda --configfile config/test_config.yml -n
Execute the workflow locally via
snakemake --use-conda --configfile config/test_config.yml --cores $N
using
$N
cores or run it in a cluster environment via
snakemake --use-conda --configfile config/test_config.yml --cluster qsub --jobs 100
or
snakemake --use-conda --configfile config/test_config.yml --drmaa --jobs 100
For slurm, consider using the following arguments:
--cluster "sbatch -A CLUSTER_ACCOUNT -t CLUSTER_TIME -p CLUSTER_PARTITION -N CLUSTER_NODES -J JOBNAME" --jobs NUM_JOBS_TO_SUBMIT
Alternatively, you might want to look into snakemake profiles already available for your HPC scheduler online, for example, here or elsewhere.
See the Snakemake documentation for further details.
Step 5: Investigate results not available yet
After successful execution, you can create a self-contained interactive HTML report with all results via:
snakemake --report report.html
This report can, e.g., be forwarded to your collaborators. An example (using some trivial test data) can be seen here .
Step 6: Commit changes
Whenever you change something, don't forget to commit the changes back to your github copy of the repository:
git commit -a
git push
Step 7: Obtain updates from upstream
Whenever you want to synchronize your workflow copy with new developments from upstream, do the following.
-
Once, register the upstream repository in your local copy:
git remote add -f upstream git@github.com:snakemake-workflows/quaich.git
orgit remote add -f upstream https://github.com/snakemake-workflows/quaich.git
if you do not have setup ssh keys. -
Update the upstream version:
git fetch upstream
. -
Create a diff with the current version:
git diff HEAD upstream/master workflow > upstream-changes.diff
. -
Investigate the changes:
vim upstream-changes.diff
. -
Apply the modified diff via:
git apply upstream-changes.diff
. -
Carefully check whether you need to update the config files:
git diff HEAD upstream/master config
. If so, do it manually, and only where necessary, since you would otherwise likely overwrite your settings and samples.
Step 8: Contribute back
In case you have also changed or added steps, please consider contributing them back to the original repository:
-
Fork the original repo to a personal or lab account.
-
Clone the fork to your local system, to a different place than where you ran your analysis.
-
Copy the modified files from your analysis to the clone of your fork, e.g.,
cp -r workflow path/to/fork
. Make sure to not accidentally copy config file contents or sample sheets. Instead, manually update the example config files if necessary. -
Commit and push your changes to your fork.
-
Create a pull request against the original repository.
Testing
Test cases are in the subfolder
.test
. They are automatically executed via continuous integration with
Github Actions
.
Code Snippets
116 117 118 119 120 121 122 123 124 | run: if not path.exists(params.file): get_file(str(params.file), output[0]) if wildcards.ext == "mcool": verify_view_cooler( cooler.Cooler( f"{output[0]}::{cooler.fileops.list_coolers(output[0])[0]}" ) ) |
65 66 67 68 69 70 | run: dots = pd.concat((map(read_dots, input.dots))).reset_index(drop=True) dots = dedup_dots(dots)[ ["chrom1", "start1", "end1", "chrom2", "start2", "end2"] ] dots.to_csv(output[0], sep="\t", header=False, index=False) |
88 89 | wrapper: "v2.6.0/bio/cooltools/dots" |
118 119 | shell: """TAB=$(printf '\t') && cat {input} | sed "1s/.*/chrom1${{TAB}}start1${{TAB}}end1${{TAB}}chrom2${{TAB}}start2${{TAB}}end2${{TAB}}FDR${{TAB}}detection_scale/" > {output}""" |
40 41 | wrapper: "v2.6.0/bio/cooltools/eigs_cis" |
65 66 | wrapper: "v2.6.0/bio/cooltools/eigs_trans" |
89 90 | wrapper: "v2.6.0/bio/cooltools/genome/gc" |
110 111 | wrapper: "v2.6.0/bio/cooltools/genome/binnify" |
15 16 | wrapper: "v2.6.0/bio/cooltools/expected_cis" |
33 34 | wrapper: "v2.6.0/bio/cooltools/expected_trans" |
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | run: insWT = pd.read_csv(input.insulation_WT, sep="\t") insWT = insWT[~insWT["is_bad_bin"]].drop(columns=["is_bad_bin"]) insKO = pd.read_csv(input.insulation_KO, sep="\t") insKO = insKO[~insKO["is_bad_bin"]].drop(columns=["is_bad_bin"]) ins = pd.merge( insWT, insKO, suffixes=(f"_{wildcards.sampleWT}", f"_{wildcards.sampleKO}"), on=["chrom", "start", "end"], ) ins = ins[ins[f"is_boundary_{wildcards.window}_{wildcards.sampleWT}"]] ins["boundary_strength_fold_change"] = ( ins[f"boundary_strength_{wildcards.window}_{wildcards.sampleKO}"] / ins[f"boundary_strength_{wildcards.window}_{wildcards.sampleWT}"] ) diff_ins = ins[ ( # Boundary much stronger in WT vs KO ins["boundary_strength_fold_change"] <= 1 / config["compare_boundaries"]["fold_change_threshold"] ) | ( # OR there is a strong boundary in WT and not in KO ins[f"is_boundary_{wildcards.window}_{wildcards.sampleWT}"] & ~ins[f"is_boundary_{wildcards.window}_{wildcards.sampleKO}"] ) ] diff_ins[ [ "chrom", "start", "end", f"log2_insulation_score_{wildcards.window}_{wildcards.sampleWT}", f"log2_insulation_score_{wildcards.window}_{wildcards.sampleKO}", f"boundary_strength_{wildcards.window}_{wildcards.sampleWT}", f"boundary_strength_{wildcards.window}_{wildcards.sampleKO}", "boundary_strength_fold_change", ] ].to_csv(output[0], header=False, index=False, sep="\t") |
71 72 73 74 75 76 77 78 | run: ins = pd.read_csv(input.insulation, sep="\t") tads = bioframe.merge(ins[ins[f"is_boundary_{wildcards.window}"] == False]) tads = tads[ (tads["end"] - tads["start"]) <= config["TADs"]["max_tad_length"] ].reset_index(drop=True) tads.to_csv(output[0], header=False, index=False, sep="\t") |
92 93 94 95 96 97 | run: ins = pd.read_csv(input.insulation, sep="\t") boundaries = ins.loc[ins[f"is_boundary_{wildcards.window}"]] boundaries[["chrom", "start", "end"]].to_csv( output[0], header=False, index=False, sep="\t" ) |
117 118 | wrapper: "v2.6.0/bio/cooltools/insulation" |
18 19 20 21 22 23 24 25 26 27 28 29 30 31 | run: from coolpuppy.lib import io as cpio pentads1 = cpio.load_pileup_df(input.pentads1) pentads2 = cpio.load_pileup_df(input.pentads2) merged = pentads1.merge( pentads2, on=["name1", "name2", "local"] + config["pentads"]["groupby"], suffixes=["_1", "_2"], ) merged["data"] = merged["data_1"] / merged["data_2"] merged["store_stripes"] = merged["store_stripes_1"] merged = merged.drop(columns=["store_stripes_1", "store_stripes_2"]) cpio.save_pileup_df(output.pantads_ratio, merged) |
49 50 51 52 53 | run: from coolpuppy.lib import io as cpio pentads = cpio.load_pileup_df_list([input.pentads_local, input.pentads_distal]) cpio.save_pileup_df(output.pentads, pentads) |
85 86 | wrapper: "v2.6.0/bio/coolpuppy" |
22 23 | wrapper: "v2.6.0/bio/coolpuppy" |
27 28 | wrapper: "v2.6.0/bio/cooltools/saddle" |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" from snakemake.shell import shell ## Extract arguments view = snakemake.input.get("view", "") if view: view = f"--view {view}" expected = snakemake.input.get("expected", "") if expected: expected = f"--expected {expected}" extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=True, stderr=True) resolution = snakemake.params.get("resolution", snakemake.wildcards.get("resolution")) if not resolution: raise ValueError("Please specify resolution either as a wildcard or as a parameter") shell( "(coolpup.py" " {snakemake.input.cooler}::resolutions/{resolution}" " {snakemake.input.features}" " {expected}" " --features-format {snakemake.params.features_format}" " {view}" " -p {snakemake.threads}" " {extra}" " -o {snakemake.output}) {log}" ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" from snakemake.shell import shell ## Extract arguments view = snakemake.input.get("view", "") if view: view = f"--view {view}" expected = snakemake.input.get("expected", "") extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=False, stderr=True) resolution = snakemake.params.get( "resolution", snakemake.wildcards.get("resolution", 0) ) if not resolution: raise ValueError( "Please specify ressolution either as a wildcard or as a parameter" ) shell( "(cooltools dots" " {snakemake.input.cooler}::resolutions/{resolution} " " {expected} " " {view} " " -p {snakemake.threads} " " {extra} " " -o {snakemake.output}) {log}" ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" from snakemake.shell import shell import tempfile ## Extract arguments view = snakemake.input.get("view", "") if view: view = f"--view {view}" track = snakemake.input.get("track", "") track_col_name = snakemake.params.get("track_col_name", "") if track and track_col_name: track = f"--phasing-track {track}::{track_col_name}" elif track: track = f"--phasing-track {track}" extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=False, stderr=True) bigwig = snakemake.output.get("bigwig", "") if bigwig: bigwig = "--bigwig" resolution = snakemake.params.get("resolution", snakemake.wildcards.get("resolution")) assert ( resolution ), "Please specify resolution either as a `wildcard` or as a `parameter`" with tempfile.TemporaryDirectory() as tmpdir: shell( "cooltools eigs-cis" " {snakemake.input.cooler}::resolutions/{resolution} " " {track}" " {view} " " {bigwig}" " {extra} " " -o {tmpdir}/out" " {log}" ) shell("mv {tmpdir}/out.cis.vecs.tsv {snakemake.output.vecs}") shell("mv {tmpdir}/out.cis.lam.txt {snakemake.output.lam}") if bigwig: shell("mv {tmpdir}/out.cis.bw {snakemake.output.bigwig}") |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" from snakemake.shell import shell import tempfile ## Extract arguments # view = snakemake.input.get("view", "") # Not yet implemented # if view: # view = f"--view {view}" view = "" track = snakemake.input.get("track", "") track_col_name = snakemake.params.get("track_col_name", "") if track and track_col_name: track = f"--phasing-track {track}::{track_col_name}" elif track: track = f"--phasing-track {track}" extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=False, stderr=True) bigwig = snakemake.output.get("bigwig", "") if bigwig: bigwig = "--bigwig" resolution = snakemake.params.get("resolution", snakemake.wildcards.get("resolution")) assert ( resolution ), "Please specify resolution either as a `wildcard` or as a `parameter`" with tempfile.TemporaryDirectory() as tmpdir: shell( "cooltools eigs-trans" " {snakemake.input.cooler}::resolutions/{resolution} " " {track}" " {view} " # Not yet implemented, hardcoded to "" " {bigwig}" " {extra} " " -o {tmpdir}/out" " {log}" ) shell("mv {tmpdir}/out.trans.vecs.tsv {snakemake.output.vecs}") shell("mv {tmpdir}/out.trans.lam.txt {snakemake.output.lam}") if bigwig: shell("mv {tmpdir}/out.trans.bw {snakemake.output.bigwig}") |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" from snakemake.shell import shell ## Extract arguments view = snakemake.input.get("view", "") if view: view = f"--view {view}" extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=False, stderr=True) resolution = snakemake.params.get( "resolution", snakemake.wildcards.get("resolution", 0) ) if not resolution: raise ValueError("Please specify resolution either as a wildcard or as a parameter") shell( "(cooltools expected-cis" " {snakemake.input.cooler}::resolutions/{resolution} " " {view} " " {extra} " " -p {snakemake.threads} " " -o {snakemake.output}) {log}" ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" from snakemake.shell import shell ## Extract arguments view = snakemake.input.get("view", "") if view: view = f"--view {view}" extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=False, stderr=True) resolution = snakemake.params.get( "resolution", snakemake.wildcards.get("resolution", 0) ) if not resolution: raise ValueError("Please specify resolution either as a wildcard or as a parameter") shell( "(cooltools expected-trans" " {snakemake.input.cooler}::resolutions/{resolution} " " {view} " " -p {snakemake.threads} " " {extra} " " -o {snakemake.output}) {log}" ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" from snakemake.shell import shell ## Extract arguments binsize = snakemake.params.get("binsize", snakemake.wildcards.get("binsize", 0)) if not binsize: raise ValueError("Please specify binsize either as a wildcard or as a parameter") extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=False, stderr=True) shell( "(cooltools genome binnify" " {snakemake.input.chromsizes} {binsize} " " {extra} " " > {snakemake.output}) {log}" ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" from snakemake.shell import shell ## Extract arguments extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=False, stderr=True) shell( "(cooltools genome gc" " {snakemake.input.bins} {snakemake.input.fasta} {extra} > {snakemake.output})" " {log} " ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" import sndhdr from snakemake.shell import shell ## Extract arguments window = snakemake.params.get("window", "") if isinstance(window, list): window = " ".join([str(w) for w in window]) else: window = str(window) view = snakemake.input.get("view", "") if view: view = f"--view {view}" chunksize = snakemake.params.get("chunksize", 20000000) extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=False, stderr=True) resolution = snakemake.params.get( "resolution", snakemake.wildcards.get("resolution", 0) ) if not resolution: raise ValueError("Please specify resolution either as a wildcard or as a parameter") shell( "(cooltools insulation" " {snakemake.input.cooler}::resolutions/{resolution} " " {window} --chunksize {chunksize} " " {view} " " -p {snakemake.threads} " " {extra} " " -o {snakemake.output}) {log}" ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | __author__ = "Ilya Flyamer" __copyright__ = "Copyright 2022, Ilya Flyamer" __email__ = "flyamer@gmail.com" __license__ = "MIT" from snakemake.shell import shell from os import path import tempfile ## Extract arguments view = snakemake.input.get("view", "") if view: view = f"--view {view}" track = snakemake.input.get("track", "") track_col_name = snakemake.params.get("track_col_name", "") if track and track_col_name: track = f"{track}::{track_col_name}" expected = snakemake.input.get("expected", "") range = snakemake.params.get("range", "--qrange 0 1") extra = snakemake.params.get("extra", "") log = snakemake.log_fmt_shell(stdout=False, stderr=True) resolution = snakemake.params.get( "resolution", snakemake.wildcards.get("resolution", 0) ) if not resolution: raise ValueError("Please specify resolution either as a wildcard or as a parameter") fig = snakemake.output.get("fig", "") if fig: ext = path.splitext(fig)[1][1:] fig = f"--fig {ext}" with tempfile.TemporaryDirectory() as tmpdir: shell( "(cooltools saddle" " {snakemake.input.cooler}::resolutions/{resolution} " " {track} " " {expected} " " {view} " " {range} " " {fig} " " {extra} " " -o {tmpdir}/out)" " {log}" ) shell("mv {tmpdir}/out.saddledump.npz {snakemake.output.saddle}") shell("mv {tmpdir}/out.digitized.tsv {snakemake.output.digitized_track}") if fig: shell("mv {tmpdir}/out.{ext} {snakemake.output.fig}") |
Support
- Future updates
Related Workflows





