Giles Thomas
@gpjt
Building LLMs, learning in public / Founded and led @PythonAnywhere / PSF Fellow
가입 June 2007
198 팔로잉 중 1.6K 팬
I extended the GPT-2-style code from @rasbt's "Build a Large Language Model (from Scratch)" so that it was a 6-expert (2 active) mixture-of-experts, and trained it from scratch over 8 days. It worked well! Full writeup with maths and code at
더 보기
