Back to search results
Header Image

Data Engineering and Aggregation Lead

Wellcome Sanger Institute

Are you an experienced, pragmatic and forward-thinking Data Engineering and Aggregation Lead looking to contribute towards solving real global health challenges? If so, this may be a once in a lifetime opportunity - and we are looking for someone like you! Come and join us at the world-famous Wellcome Sanger Institute in the newly formed Genomic Surveillance Unit. No science background needed!

You will lead a team of data engineers supporting the aggregation, management, storage and integration of genomic surveillance data and related metadata with public health entities and research partners.

About the Role

As the Data Engineering and Aggregation Lead, your primary objective will be to take data from sequencing informatics pipelines and integrate it with multiple sources of heterogeneous data to deliver a range of high quality, analysis-ready genetic data products to third party organisations and the Analysis team via multiple channels.

In this role you will:

  • Lead a team of principal, senior, and junior data engineers.
  • Develop operational relationships with health and government organisations around the world.
  • Engage with the GSU leadership teams and other internal stakeholders to enable requirements gathering
  • Work closely with Sanger IDS (IT shared service) and GSU to implement a data management strategy
  • Leverage off data architect skills to harmonise data infrastructure to support our products
  • Develop and recruit flexible agile teams

Essential Skills

About You

  • A natural leader when it comes to data management technology and people
  • Able to relate to scrum team members and motivate them to achieve great results
  • Passionate about data engineering and using technology to deliver according to organisation requirements
  • Demonstrable experience when it comes to using data engineering tools to achieve strategic objectives
  • Flexible in your approach and role-model service-oriented behaviour in all you do
  • Able to interpret the needs of an organisation and translate them into actionable plans - collaborating closely with your fellow principals as well as your colleagues in the data architecture, data engineering, and software development space

Other information

About the Tech

We currently develop and run our pipelines in Apache Spark. This registry is backed up in Gitlab. We are trialling the use of Prometheus and Grafana for monitoring and alerting. We use Kafka as a message broker and Prefect as an orchestration tool primarily leveraging off the scheduler and dependency management modules. As most of our code is Python and therefore we use the PEP 8 style guide. We use Gitlab CI/CD for test pipelines and deployment along with Terraform and Ansible to provision and configure services on OpenStack. We are an agile team running our Kanban board in Jira and we’re always looking for ways to improve our way of working.

About Us

Wellcome Sanger is a world-leading genomics research institute and our work helps improve human health and understand life on Earth. The newly launched Genomic Surveillance Unit (GSU) is the first service-delivery unit within Sanger that is working towards using genomic surveillance as a practical tool for local disease control and to support countries in being pandemic-prepared. Collaboration is at the heart of what we do and we are connected to a network of researchers, clinicians and public health agencies across the globe through a number of multi-centre projects. We provide a framework for generating, integrating and sharing genetic and genomic data, and for investigating key questions about COVID-19, malaria biology and epidemiology. Have a look here to get more insight into COVID-19 genomic surveillance and its impact.

Data at Sanger is generated based on organic samples that are processed through sequencing informatics pipelines and combined with multiple heterogeneous data sources. One data stream provides an end-to-end view of the samples’ lifecycle to manage our operations while another stream delivers a range of highly available, scalable, and near real-time data products to third party organisations, public health entities and research partners. 

It is the responsibility of the Data Engineering team to ensure that all necessary data points are consolidated from multiple internal/external systems, analytics outputs, and partners using the most appropriate data architecture. Our goal is for our product portfolio to contain datasets that are interoperable with the receiving organisation’s existing systems and processes.

Additional Information

We have adopted a hybrid model to support a balance of remote and office working. You can find out more about our inspiring campus here.

Interviews will be taking place virtually or on the campus depending on candidate location. This approach may vary for individuals located overseas and/or where a visa is required and starting will be based on a number of factors, we will be able to provide specialist advice to those affected candidates.

Please apply with your CV and cover letter outlining:

  • Your suitability for the role
  • Your experience as a leader and manager in the big data space
  • Your approach to team leadership
  • Any references to prior projects you’ve been involved in that you believe are relevant for consideration
Location: Cambridge, Hinxton
Salary: £58,000 to £80,000 per annum. Grade: PSG Band 4
Hours: Full Time
Contract Type: Permanent
Placed On: 5th August 2022
Closes: 28th August 2022
Job Ref: 84965
 
   
We value your feedback on the quality of our adverts. If you have a comment to make about the overall quality of this advert, or its categorisation then please send us your feedback
Show all jobs for Wellcome Sanger Institute …
Advert information

Type / Role:

Subject Area(s):

Location(s):

Job tools
 

PhD Alert Created

Job Alert Created

Your PhD alert has been successfully created for this search.

Your job alert has been successfully created for this search.

Ok Ok

PhD Alert Created

Job Alert Created

Your PhD alert has been successfully created for this search.

Your job alert has been successfully created for this search.

Manage your job alerts Manage your job alerts

Account Verification Missing

In order to create multiple job alerts, you must first verify your email address to complete your account creation

Request verification email Request verification email

jobs.ac.uk Account Required

In order to create multiple alerts, you must create a jobs.ac.uk jobseeker account

Create Account Create Account

Alert Creation Failed

Unfortunately, your account is currently blocked. Please login to unblock your account.

Email Address Blocked

We received a delivery failure message when attempting to send you an email and therefore your email address has been blocked. You will not receive job alerts until your email address is unblocked. To do so, please choose from one of the two options below.

Max Alerts Reached

A maximum of 5 Job Alerts can be created against your account. Please remove an existing alert in order to create this new Job Alert

Manage your job alerts Manage your job alerts

Creation Failed

Unfortunately, your alert was not created at this time. Please try again.

Ok Ok

Create PhD Alert

Create Job Alert

When you create this PhD alert we will email you a selection of PhDs matching your criteria.When you create this job alert we will email you a selection of jobs matching your criteria. Our Terms and Conditions and Privacy Policy apply to this service. Any personal data you provide in setting up this alert is processed in accordance with our Privacy Notice

Create PhD Alert

Create Job Alert

When you create this PhD alert we will email you a selection of PhDs matching your criteria.When you create this job alert we will email you a selection of jobs matching your criteria. Our Terms and Conditions and Privacy Policy apply to this service. Any personal data you provide in setting up this alert is processed in accordance with our Privacy Notice

 
 

Browser Upgrade Recommended

jobs.ac.uk has been optimised for the latest browsers.

For the best user experience, we recommend viewing jobs.ac.uk on one of the following:

Google Chrome Firefox Microsoft Edge