Pipeline Run Tracking

External Link

Site

Creating ability to track the running of the Bioinformatics Core Facility’s open source Nextflow pipeline regardless of run environment.The tool utilizes the uses of a custom built REST-API methods with AWS API Gateway which writes to a AWS DynamoDB non-SQL database. A front-end website was built using the static site generator Hugo, though it utilizes custom htmp, css, js code. Continuous integration/continuous deployment is used to automatically build/test/deploy to a AWS S3 bucket, where it is serverlessly hosted. The site queries the database with though the REST-API.

Role:

  • Engineer
    • Created the Phase 1 proof of concept independently
    • Architect the methods for further development by additional engineering team

Phase 1 (complete):

  • Build API and database to track pipeline runs
  • Engineer method for submitting “started” status to the database by Nextflow pipelines regardless of run environment
  • Build front-end website which displays run information of several pipelines
  • Create CI based build/test/deploy of website

Phase 2 (in process):

  • Develop methods for Nextflow deploying status updates
  • Integrate into all BICF pipelines (see BICF Pipelining project)