BatchConvert: Efficient Image Data Conversion into OME-TIFF or OME-Zarr Formats with Parallel Processing and Automated Storage Transfer

public public 1yr ago Version: main @ 03e32fe 0 bookmarks
View Workflow
Report
batchconvert
batchconvert
batchconvert
batchconvert
View Workflow

BatchConvert

A command line tool for converting image data into either of the standard file formats OME-TIFF or OME-Zarr.

The tool wraps the dedicated file converters bfconvert and bioformats2raw to convert into OME-TIFF or OME-Zarr, respectively. The workflow management system NextFlow is used to perform conversion in parallel for batches of images.

The tool also wraps s3 and Aspera clients (go-mc and aspera-cli, respectively). Therefore, input and output locations can be specified as local or remote storage and file transfer will be performed automatically. The conversion can be run on HPC with Slurm.

Installation & Dependencies

Important note: The package has been so far only tested on Ubuntu 20.04.

The minimal dependency to run the tool is NextFlow, which should be installed and made accessible from the command line.

If conda exists on your system, you can install BatchConvert together with NextFlow using the following script:

git clone https://github.com/Euro-BioImaging/BatchConvert.git && \ source BatchConvert/installation/install_with_nextflow.sh

If you already have NextFlow installed and accessible from the command line (or if you prefer to install it manually e.g., as shown here ), you can also install BatchConvert alone, using the following script:

git clone https://github.com/Euro-BioImaging/BatchConvert.git && \ source BatchConvert/installation/install.sh

Other dependencies (which will be automatically installed):

  • bioformats2raw (entrypoint bioformats2raw)
  • bftools (entrypoint bfconvert)
  • go-mc (entrypoint mc)
  • aspera-cli (entrypoint ascp)

These dependencies will be pulled and cached automatically at the first execution of the conversion command. The mode of dependency management can be specified by using the command line option --profile or -pf . Depending on how this option is specified, the dependencies will be acquired / run either via conda or via docker/singularity containers.

Specifying --profile conda (default) will install the dependencies to an environment at ./.condaCache and use this environment to run the workflow. This option requires that miniconda/anaconda is installed on your system.

Alternatively, specifying --profile docker or --profile singularity will pull a docker or singularity image with the dependencies, respectively, and use this image to run the workflow. These options assume that the respective container runtime (docker or singularity) is available on your system. If singularity is being used, a cache directory will be created at the path ./.singularityCache where the singularity image is stored.

Finally, you can still choose to install the dependencies manually and use your own installations to run the workflow. In this case, you should specify --profile standard and make sure the entrypoints specified above are recognised by your shell.

Code Snippets

22
23
24
"""
batchconvert_cli.sh "$inpath.name" "${inpath.baseName}.ome.tiff"
"""
NextFlow From line 22 of modules/modules.nf 
42
43
44
45
46
47
48
49
50
51
"""
if [[ -d "${inpath}/tempdir" ]];
    then
        batchconvert_cli.sh "${inpath}/tempdir/${pattern_file}" "${pattern_file.baseName}.ome.tiff"
    else
        batchconvert_cli.sh "$inpath/$pattern_file.name" "${pattern_file.baseName}.ome.tiff"
fi
# rm -rf ${inpath}/tempdir &> /dev/null
# rm -rf ${inpath}/*pattern &> /dev/null
"""
NextFlow From line 42 of modules/modules.nf 
67
68
69
"""
batchconvert_cli.sh "$inpath.name" "${inpath.baseName}.ome.zarr"
"""
NextFlow From line 67 of modules/modules.nf 
88
89
90
91
92
93
94
95
96
97
"""
if [[ -d "${inpath}/tempdir" ]];
    then
        batchconvert_cli.sh "${inpath}/tempdir/${pattern_file.name}" "${pattern_file.baseName}.ome.zarr"
    else
        batchconvert_cli.sh "$inpath/$pattern_file.name" "${pattern_file.baseName}.ome.zarr"
fi
# rm -rf ${inpath}/tempdir &> /dev/null
# rm -rf ${inpath}/*pattern &> /dev/null
"""
NextFlow From line 88 of modules/modules.nf 
109
110
111
112
113
"""
sleep 5;
mc -C "./mc" alias set "${params.S3REMOTE}" "${params.S3ENDPOINT}" "${params.S3ACCESS}" "${params.S3SECRET}" &> /dev/null;
parse_s3_filenames.py "${params.S3REMOTE}/${params.S3BUCKET}/${source}/"
"""
NextFlow From line 109 of modules/modules.nf 
126
127
128
129
130
131
132
133
134
135
136
"""
sleep 5;
localname="\$(basename $local)" && \
mc -C "./mc" alias set "${params.S3REMOTE}" "${params.S3ENDPOINT}" "${params.S3ACCESS}" "${params.S3SECRET}";
if [ -f $local ];then
    mc -C "./mc" cp $local "${params.S3REMOTE}"/"${params.S3BUCKET}"/"${params.out_path}"/"\$localname";
elif [ -d $local ];then
    mc -C "./mc" mirror $local "${params.S3REMOTE}"/"${params.S3BUCKET}"/"${params.out_path}"/"\$localname";
fi
echo "${params.S3REMOTE}"/"${params.S3BUCKET}"/"${params.out_path}"/$local > "./transfer_report.txt";
"""
NextFlow From line 126 of modules/modules.nf 
145
146
147
148
149
"""
sleep 5;
mc -C "./mc" alias set "${params.S3REMOTE}" "${params.S3ENDPOINT}" "${params.S3ACCESS}" "${params.S3SECRET}";
mc -C "./mc" mirror "${params.S3REMOTE}"/"${params.S3BUCKET}"/"${source}" "transferred/${source}";
"""
NextFlow From line 145 of modules/modules.nf 
160
161
162
163
164
"""
sleep 5;
mc -C "./mc" alias set "${params.S3REMOTE}" "${params.S3ENDPOINT}" "${params.S3ACCESS}" "${params.S3SECRET}";
mc -C "./mc" cp "${s3path}" "${s3name}";
"""
NextFlow From line 160 of modules/modules.nf 
173
174
175
176
"""
ascp -P33001 -l 500M -k 2 -i $BIA_SSH_KEY -d $local bsaspera_w@hx-fasp-1.ebi.ac.uk:${params.BIA_REMOTE}/${params.out_path};
echo "${params.BIA_REMOTE}"/"${params.out_path}" > "./transfer_report.txt";
"""
NextFlow From line 173 of modules/modules.nf 
186
187
188
"""
ascp -P33001 -l 500M -k 2 -i $BIA_SSH_KEY -d bsaspera_w@hx-fasp-1.ebi.ac.uk:${params.BIA_REMOTE}/$source ".";
"""
NextFlow From line 186 of modules/modules.nf 
198
199
200
"""
ascp -P33001 -i $BIA_SSH_KEY bsaspera@fasp.ebi.ac.uk:$source transferred;
"""
NextFlow From line 198 of modules/modules.nf 
271
272
273
274
275
276
277
278
279
280
281
"""
if [[ "${params.pattern}" == '' ]] && [[ "${params.reject_pattern}" == '' ]];then
    create_hyperstack --concatenation_order ${params.concatenation_order} ${inpath}
elif [[ "${params.reject_pattern}" == '' ]];then
    create_hyperstack --concatenation_order ${params.concatenation_order} --select_by ${params.pattern} ${inpath}
elif [[ "${params.pattern}" == '' ]];then
    create_hyperstack --concatenation_order ${params.concatenation_order} --reject_by ${params.reject_pattern} ${inpath}
else
    create_hyperstack --concatenation_order ${params.concatenation_order} --select_by ${params.pattern} --reject_by ${params.reject_pattern} ${inpath}
fi
"""
NextFlow From line 271 of modules/modules.nf 
290
291
292
293
294
295
296
297
298
299
300
"""
if [[ "${params.pattern}" == '' ]] && [[ "${params.reject_pattern}" == '' ]];then
    create_hyperstack --concatenation_order ${params.concatenation_order} ${inpath}
elif [[ "${params.reject_pattern}" == '' ]];then
    create_hyperstack --concatenation_order ${params.concatenation_order} --select_by ${params.pattern} ${inpath}
elif [[ "${params.pattern}" == '' ]];then
    create_hyperstack --concatenation_order ${params.concatenation_order} --reject_by ${params.reject_pattern} ${inpath}
else
    create_hyperstack --concatenation_order ${params.concatenation_order} --select_by ${params.pattern} --reject_by ${params.reject_pattern} ${inpath}
fi
"""
NextFlow From line 290 of modules/modules.nf 
318
319
320
321
"""
rm -rf "${inpath}/tempdir" &> /dev/null
rm -rf "${inpath}/*pattern" &> /dev/null
"""
NextFlow From line 318 of modules/modules.nf 
332
333
334
335
"""
mc alias set "${params.S3REMOTE}" "${params.S3ENDPOINT}" "${params.S3ACCESS}" "${params.S3SECRET}";
mc mirror "${params.S3REMOTE}"/"${params.S3BUCKET}"/"${source}" "transferred";
"""
NextFlow From line 332 of modules/modules.nf 
344
345
"""
"""
NextFlow From line 344 of modules/modules.nf 
359
360
"""
"""
NextFlow From line 359 of modules/modules.nf 
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
"""
if [[ "${params.merge_files}" == "True" ]];
    then
        create_hyperstack --concatenation_order ${params.concatenation_order} --select_by ${params.pattern} ${inpath};
        if [[ "${params.concatenation_order}" == "auto" ]];
            then
                batchconvert_cli.sh $inpath/*pattern "${inpath.baseName}.ome.zarr"
        elif ! [[ "${params.concatenation_order}" == "auto" ]];
            then
                batchconvert_cli.sh $inpath/tempdir/*pattern "${inpath.baseName}.ome.zarr"
        fi
elif [[ "${params.merge_files}" == "False" ]];
    then
        batchconvert_cli.sh $inpath "${inpath.baseName}.ome.zarr"
fi
rm -rf "${inpath}/tempdir" &> /dev/null
rm -rf "${inpath}/*pattern" &> /dev/null
"""
NextFlow From line 377 of modules/modules.nf
ShowHide 18 more snippets with no or duplicated tags.

Login to post a comment if you would like to share your experience with this workflow.

Do you know this workflow well? If so, you can request seller status , and start supporting this workflow.

Free

Created: 1yr ago
Updated: 1yr ago
Maitainers: public
URL: https://github.com/Euro-BioImaging/BatchConvert.git
Name: batchconvert
Version: main @ 03e32fe
Badge:
workflow icon

Insert copied code into your website to add a link to this workflow.

Downloaded: 0
Copyright: Public Domain
License: MIT License
  • Future updates

Related Workflows

psychip_snakemake
Show Details View Workflow
ENCODE pipeline for histone marks developed for the psychENCODE project
public

public

psychip pipeline is an improved version of the ENCODE pipeline for histone marks developed for the psychENCODE project. The o...
raw sequence reads Alignment Sequence alignment report macs2 ucsc-bedclip bedGraphToBigWig BEDTools BWA Picard SAMtools Snakemake
Free
74
ncov_2
Show Details View Workflow
Near-real time tracking of SARS-CoV-2 in Connecticut
public

public

Repository containing scripts to perform near-real time tracking of SARS-CoV-2 in Connecticut using genomic data. This pipeli...
JSON nextclade Augur Biopython FOCUS Pandas Snakemake bs4 epiweeks geopy matplotlib numpy pycountry pycountry-convert uszipcode
Free
28
cellranger-snakemake-gke
Show Details View Workflow
snakemake workflow to run cellranger on a given bucket using gke.
public

public

A Snakemake workflow for running cellranger on a given bucket using Google Kubernetes Engine. The usage of this workflow ...
macs2 ucsc-bedclip bedGraphToBigWig BEDTools BWA Picard SAMtools Snakemake
Free
74
atlas
Show Details View Workflow
ATLAS - Three commands to start analyzing your metagenome data
public

public

Metagenome-atlas is a easy-to-use metagenomic pipeline based on snakemake. It handles all steps from QC, Assembly, Binning, t...
raw sequence reads Genome assembly Annotation track checkm2 gunc prodigal snakemake-wrapper-utils MEGAHIT Atlas BBMap Biopython BioRuby Bwa-mem2 cd-hit CheckM DAS Diamond eggNOG-mapper v2 MetaBAT 2 Minimap2 MMseqs MultiQC Pandas Picard pyfastx SAMtools SemiBin Snakemake SPAdes SqueezeMeta TADpole VAMB CONCOCT ete3 gtdbtk h5py networkx numpy plotly psutil utils metagenomics
Free
175
rna-seq-star-deseq2
Show Details View Workflow
RNA-seq workflow using STAR and DESeq2
public

public

This workflow performs a differential gene expression analysis with STAR and Deseq2. The usage of this workflow is described ...
raw sequence reads Gene expression profiling ID list snakemake-wrapper-utils biomaRt PCAtools BiocParallel BWA Cutadapt DESeq2 GFFutils MultiQC Pandas SAMtools Snakemake STAR cli stringr tidyverse RNA-Seq
Free
41
dna-seq-gatk-variant-calling
Show Details View Workflow
This Snakemake pipeline implements the GATK best-practices workflow
public

public

This Snakemake pipeline implements the GATK best-practices workflow for calling small germline variants. The usage of thi...
VCF raw sequence reads Variant calling genetic variants gatk rust-bio-tools snakemake-wrapper-utils tabix BCFtools BWA FastQC MultiQC Pandas Picard SAMtools Snakemake Trimmomatic Variant Effect Predictor (VEP) common matplotlib numpy seaborn DNA
Free
57