Back to search results

PhD Studentship: Affordable Training of Large Language Models

The University of Edinburgh

Qualification Type: PhD
Location: Edinburgh
Funding for: UK Students
Funding amount: Tuition fees plus stipend
Hours: Full Time
Placed On: 5th June 2023
Closes: 1st September 2023
 

Recent developments in large language models (LLMs) have caught the attention of the public. LLMs such as OpenAI's GPT-4 and Google's Bard are able to generate remarkably realistic, coherent text based on a user's input and have the potential to be general-purpose tools used throughout society e.g. for customer service, summarising text, answering questions, writing contracts or translating between languages.

However, LLMs are prohibitively expensive to train. GPT-3 (which is significantly smaller than its successor, GPT-4) has an estimated training time of 355-GPU years and an estimated training cost of $4.6M [1]. Only large, wealthy institutions can train these models and thereby control how they are trained and who gets access to them. This is undemocratic.

Very recent work provides hope however. In [2] the authors explore the promising idea of “cramming”: the training of a LLM on a single GPU in a day. In [3] the authors use synthetic data to train “small” language models that can produce consistent stories at little cost. There is a huge discrepancy in quality between these models and their expensive counterparts, however.

In this PhD, the student will investigate affordable LLM training i.e. with limited compute and/or data, inspired by [2,3]. Avenues of research could include (i) generating training data that facilitates fast training e.g. through dataset distillation [4]; (ii) exploring neural architecture search to develop models that are "aware" of being resource-constrained while being trained; (iii) developing novel cost-effective training algorithms, (iv) leveraging and tuning open-source LLMs.

The successful student will have opportunities for collaboration within and outside Edinburgh’s School of Engineering e.g. with colleagues in the Institute for Digital Communications, The Bayesian and Neural Systems Group, and Edinburgh NLP

Further Information: 

The University of Edinburgh is committed to equality of opportunity for all its staff and students, and promotes a culture of inclusivity. Please see details here: https://www.ed.ac.uk/equality-diversity

https://elliotjcrowley.github.io/

https://www.bayeswatch.com 

References

[1] https://lambdalabs.com/blog/demystifying-gpt-3

[2] https://arxiv.org/abs/2212.14034

[3] https://arxiv.org/abs/2305.07759

[4] https://arxiv.org/abs/1811.10959

Closing Date: 

Friday, September 1, 2023
Principal Supervisor:  Assistant Supervisor:  Eligibility: 

Minimum entry qualification - an Honours degree at 2:1 or above (or International equivalent) in a relevant science or engineering discipline, possibly supported by an MSc Degree. Further information on English language requirements for EU/Overseas applicants.

What we’re looking for

An enthusiastic, creative student with a solid academic record and strong programming skills. AI is an exciting, fast-moving field so be prepared to think on your feet (and at times laterally)!

Essential Criteria:

  • A 2.1 or higher numerate degree
  • Excellent communication skills
  • Programming skills

Desirable Criteria:

  • Knowledge of machine learning
  • An understanding of large language models
  • Experience programming in Python
Funding: 

Tuition fees + stipend are available for applicants who qualify as Home applicants (International students can apply, but the funding only covers the Home fee rate)

Further information and other funding options.

Informal Enquiries: 
We value your feedback on the quality of our adverts. If you have a comment to make about the overall quality of this advert, or its categorisation then please send us your feedback
Advert information

Type / Role:

Subject Area(s):

Location(s):

PhD tools
 

PhD Alert Created

Job Alert Created

Your PhD alert has been successfully created for this search.

Your job alert has been successfully created for this search.

Ok Ok

PhD Alert Created

Job Alert Created

Your PhD alert has been successfully created for this search.

Your job alert has been successfully created for this search.

Manage your job alerts Manage your job alerts

Account Verification Missing

In order to create multiple job alerts, you must first verify your email address to complete your account creation

Request verification email Request verification email

jobs.ac.uk Account Required

In order to create multiple alerts, you must create a jobs.ac.uk jobseeker account

Create Account Create Account

Alert Creation Failed

Unfortunately, your account is currently blocked. Please login to unblock your account.

Email Address Blocked

We received a delivery failure message when attempting to send you an email and therefore your email address has been blocked. You will not receive job alerts until your email address is unblocked. To do so, please choose from one of the two options below.

Max Alerts Reached

A maximum of 5 Job Alerts can be created against your account. Please remove an existing alert in order to create this new Job Alert

Manage your job alerts Manage your job alerts

Creation Failed

Unfortunately, your alert was not created at this time. Please try again.

Ok Ok

Create PhD Alert

Create Job Alert

When you create this PhD alert we will email you a selection of PhDs matching your criteria.When you create this job alert we will email you a selection of jobs matching your criteria. Our Terms and Conditions and Privacy Policy apply to this service. Any personal data you provide in setting up this alert is processed in accordance with our Privacy Notice

Create PhD Alert

Create Job Alert

When you create this PhD alert we will email you a selection of PhDs matching your criteria.When you create this job alert we will email you a selection of jobs matching your criteria. Our Terms and Conditions and Privacy Policy apply to this service. Any personal data you provide in setting up this alert is processed in accordance with our Privacy Notice

 
 
 
More PhDs from The University of Edinburgh

Show all PhDs for this organisation …

More PhDs like this
Join in and follow us

Browser Upgrade Recommended

jobs.ac.uk has been optimised for the latest browsers.

For the best user experience, we recommend viewing jobs.ac.uk on one of the following:

Google Chrome Firefox Microsoft Edge