Skip to content

[GitHub Trending] FareedKhan-dev/train-llm-from-scratch

8.6 relevance
Score Breakdown
technical depth
9
novelty
5
actionability
9
community
7
strategic
6
personal
8

Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.

Tutorial for training LLM from scratch, highly actionable and relevant.

AI/ML github.com
A straightforward method for training your LLM, from downloading data to generating text. - FareedKhan-dev/train-llm-from-scratch
Summary

FareedKhan-dev's open-source repository implements a transformer from scratch in PyTorch, based on the 'Attention is All You Need' paper, and provides scripts to train billion- or million-parameter LLMs on a single GPU. The 13M parameter model trains on The Pile dataset and includes a detailed GPU memory comparison for scaling up to 2B parameters. The author, seeking a PhD position, structures the code with modular components (MLP, attention, transformer block) and offers step-by-step explanations.

Author

FareedKhan-dev