PyTorch-native NeMo AutoModel handles transformer pretraining in @nvidia's end-to-end workflow for building a transaction foundation model. The workflow combines GPU-accelerated data processing and tokenization, decoder-only model pretraining, embedding extraction, and XGBoost fraud classification. On the synthetic @IBM TabFormer dataset, combining raw features with learned embeddings increased Average Precision by 41.76% over the raw-feature baseline. 🔗 Read the full post:

HIPPO targets a quiet contaminant in RL for reasoning: pre-RL data overlap. When the RL dataset overlaps with pretraining or SFT corpora, the model can exploit the shortcut of recalling a memorized answer and then fabricating post-hoc reasoning to match, so the reward goes up while genuine reasoning does not. The framework injects hints and uses a pairwise objective designed to break that shortcut, forcing the model to reason toward the answer rather than reverse-engineer a justification for one it already memorized. To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Optimization Paper:

iLLaDA is an 8B masked diffusion language model trained from scratch with fully bidirectional attention, keeping the masked-diffusion objective all the way through pretraining and SFT rather than bolting diffusion onto an autoregressive base. Improved Large Language Diffusion Models Paper:

🤖 Expensive robot data is scarce, so why not learn from our everyday first-person videos? This work turns human hand motion into robot actions to pretrain VLA models. 📰 Title: ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining 🔗 URL: 💡 Overview ACE-Ego-0 unifies robot demonstrations with egocentric human videos (Ego4D, EPIC-KITCHENS, and more) to pretrain Vision-Language-Action (VLA) models, trained on over 6,000 hours of combined data. 🔍 Challenges Solved Robot demonstrations are costly to collect, while human videos are cheap and abundant. But the two differ in action space, embodiment structure, temporal dynamics, and supervision quality, so naively mixing them breaks training. 🛠 Methodology & Proposed Approach ・Unifies actions in head-camera coordinates with 6D rotations, treating the human hand as an end-effector ・Encodes robot URDFs into morphology tokens via a GNN to absorb structural differences ・Chunks actions by consistent physical duration instead of fixed steps for temporal alignment ・Applies a reliability-weighted loss to noisy human videos, focusing on trustworthy position channels 📊 Use Cases / Results On RoboCasa it hits 72.8% average success (vs GR00T-N1.6 at 47.6%) and ~91% on RoboTwin 2.0. On a real bimanual robot it reaches 78.3% average (π0.5: 71.7%). Strikingly, on a task with only 34 robot demos, adding 419 human video episodes lifted success from 10% to 40%, a 4x gain. #RobotLearning# #VLA#

OpenAI is hiring Noam Shazeer, per The Information. Shazeer was a key author of the original Transformer paper, co-founded and later worked on Gemini pretraining at Google DeepMind. He will reportedly focus on new AI model architectures and the evolution of transformers at OpenAI.

Congrats to the @cursor_ai team on the launch of Composer 2! We are proud to see Kimi-k2.5 provide the foundation. Seeing our model integrated effectively through Cursor's continued pretraining & high-compute RL training is the open model ecosystem we love to support. Note: Cursor accesses Kimi-k2.5 via @FireworksAI_HQ ' hosted RL and inference platform as part of an authorized commercial partnership.

Does anyone have a clue of how much post-training compute is being used right now? V4-Pro is ≈1e25 class model (as are its peers). Over 2 months, could they have spent another 1e25 on rollouts? More? What is the pretraining share at this point?

Build and train an LLM "from scratch" yourself and you truly understand what's happening inside 🛠️ A complete educational implementation that runs on a single GPU. Title: FareedKhan-dev/train-llm-from-scratch URL: 🛠️ Overview An educational repository that implements a Transformer from scratch in PyTorch, based on "Attention is All You Need." It promises you can train your own million- to billion-parameter LLM on a single GPU. ❓ Challenges Solved LLMs are ubiquitous, but hands-on chances to train one from scratch and understand its internals are rare. ・Just using off-the-shelf frameworks leaves the Transformer's mechanics opaque ・Learners needed an end-to-end resource spanning pretraining through post-training alignment 💡 Content & Structure It covers the entire LLM lifecycle. ・Data acquisition and preprocessing (from The Pile) ・Core Transformer architecture (embeddings, attention, feed-forward networks) ・Model training (with DDP for distributed processing) ・Post-training alignment: SFT, reward modeling, PPO, DPO, GRPO ・Text generation and inference Code is organized into src/models, scripts, data_loader, configs, and a Streamlit ui. The stack is PyTorch, tiktoken, HDF5, and NumPy. 🌍 Use Cases / Audience For developers and researchers who want hands-on understanding of LLM training — from those with limited GPUs (starting at 13M parameters) to those targeting multi-billion-parameter models on enterprise hardware. #LLM# #MachineLearning#

There are 5 core elements of the intersection of AI and crypto that I find most compelling. In no particular order: 1. Consumer AI apps that maintain data privacy 2. Distributed training of AI models comparable to those of the frontier labs 3. Data protocols to support model pretraining 4. Tokenized compute 5. Identity attestation All of these categories have liquid tokens with demonstrated traction, revenue, and accelerating growth. Very likely in my opinion that they materially outperform the broader market from here