my most important lessons on using voice input on agents talking with voice is auto regressive, like an LLM: - we speak one word at a time - we think and talk simultaneously - and we can’t easily take back what we said this created some interesting characteristics: - what we say first will influence what we say next. try starting your sentence with “the most important two things i want you to do are…” and you’ll find yourself cornered into a certain way of framing your message - we generally don’t have a ton of time to think while talking. and the faster we talk, the less thinking we do. this is why people who trained themselves to take pause are often more eloquent and perceived as thoughtful - we often perceive talking as higher stake than typing because correction is a lot harder these traits made voice input a terrible method for writing content. say you want to write a blog post, and using voice means you have to think through most things upfront, and randomly derail the content during talking due to choice of words, and can’t correct what’s written (unless you fallback to typing in the end) BUT.. when using agents, most of these traits don’t apply! agents can understand messy rambling just fine. they can understand your corrections if you said “no no scratch that, let me rephrase”. and they couldn’t care less if you take a pause so when using voice input on agents, you have to think of it differently from talking to real people or writing real content my advice: - don’t try to speak only the “final result”. speak your thinking process out loud this gives yourself a lot more room to reason through the concepts and arrive at something that actually represents what you want it also gives the agent more context on why you made certain choices you can even intentionally use phrases like “let’s think through this..”, or “here’s why..” to elicit more thinking - correct yourself often when you realized something you said earlier was wrong, you can either just stop the voice input and discard what you said so far, or start correcting yourself the good thing about agents is that they don’t judge. they are trained to never shame you for making mistakes or walking back your decisions once you started doing this more, you will perceive voice input as low stake, which makes it feel more natural and less demanding on the other hand, typing is text diffusion - you think in your head and type out the results, and you constantly refine what you already wrote it’s great for writing polished content, or precise text such as code. but it’s not as good for prompting agents because all your thinking tokens were hidden so the agent only sees the final “what” but rarely the “why”

youtube discovered i decided to start eating healthy, so it decided to recommend a michelin chef cooking ramen to me at 11pm... watching the first half i was like "i do that too" the second half just showed me why i'm not a michelin chef

how to set the right reasoning effort level i get asked this a lot, and i learned that because the underlying mechanism is not obvious, a lot of people don’t have the right mental model to think about how to use the knob so a quick crash course here: reasoning effort level in mainstream LLMs today means a “ceiling”, not a “floor”! setting a high effort level does NOT mean every prompt you send will use a lot of thinking tokens. you can try this very easily - launch your agent in an empty directory, use high reasoning, and say “hi” - unless your harness is totally messed up, the agent will respond almost instantly how does that work? the most typical implementation is that in post-training, the models receive a penalty for how many thinking tokens they use. how strong the penalty is depends on the thinking effort level that’s set at low reasoning, the penalty is very strong, so any long thinking traces get punished into oblivion at high reasoning, the penalty is weak so longer thinking traces don’t get punished too much, and are allowed to survive more often as long as they get good results the most important thing to learn there is - thinking tokens is always a penalty, not a reward. there’s never a rule that says “at high reasoning i’ll punish you if you think too little” - this is because more thinking tokens always means higher cost which is always a bad thing and should not be rewarded so with that in mind, you should think of reasoning effort as “how much are you allowed to think”, not “how much do i want you to think” setting it at low means “for this task you are not allowed to think much”. you would want this when 1) the use case requires low latency; or 2) you already know the task does not require additional reasoning setting it at high means “you are allowed to think more when needed”. the model will still try to think as little as possible according to what its training data teaches it. you want this when 1) the use case allows slower response; and 2) you are not sure how much thinking is needed for this task for some models, “max” is special because it forces thinking mode to be on, which forbids “zero thinking”. i don’t have a full inventory for which models do this vs not. and i generally avoid this mode because i don’t think it’s a good idea to say “every request has to have thinking tokens no matter what” practically speaking, i mostly only use two modes: - medium, when i already know the task is well defined for example, implementing a spec planned by a highly intelligent model. i avoid low because 1) i rarely require low latency; 2) even when implementing well defined tasks sometimes the intermediate context is still ambiguous this is also my default for firstmate because highly ambiguous problems are typically handled by crewmates, and i do want firstmate to be fast - xhigh, whenever the task is not well defined yet. this is most often used for planning and investigative work. i do this to give enough room for the model to decide how much thinking it needs hope this is helpful!