๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Hyunjong Ok
@HyunjongOk
Google PhD Fellow | PhD Student at POSTECH
๊ฐ€์ž… November 2023
61 ํŒ”๋กœ์ž‰ ์ค‘    102 ํŒฌ
๐Ÿš€ Stop using Transformers for weight-space programming If you are not training with gradients on a GPU, why default to a Transformer? When an LLM writes network weights directly, letting it also design the architecture achieves: ๐Ÿ“‰ ~1,800ร— fewer parameters at matched accuracy on ARC (AI2 Reasoning Challenge), compared to the best Transformer the same LLM wrote. โšก Up to 17ร— faster GPU inference with a memory architecture built for these tables, compared to a Transformer of the same size. ๐Ÿงต Paper & code at the end ๐Ÿ‘‡
๋” ๋ณด๊ธฐ