I’m at AIE NYC this week, and I’ll be giving a talk on evaluating subjective / open-ended tasks! Talk details: - Wednesday, October 14, 3:20pm - 3:38pm, Turing Room (Riverside Ballroom) - I’ll follow up with slides, links, full details, reading material, and more resources after the talk. Motivation. Evaluating LLMs is difficult, and much of this difficulty comes from noise. How do we know if a better eval result is due to a better model or just attributable to noise / variance? We need a methodology for building evaluations that yield confident / useful information despite the non-determinism of LLMs. Subjective tasks. The problem of noise in evaluations becomes even harder when working with subjective tasks. Recent LLM research focuses heavily on verifiable tasks, where correctness is judged with a deterministic check (e.g., string matches, symbolic verification, test cases, etc.). For subjective tasks, even defining correctness is an open question. We must clearly define what constitutes a correct solution and try to capture this with an LLM judge or rubric. Then, even our verifier is non-deterministic, thus making the evaluation process noisier. The subjectivity of the evaluation process makes it really easy to build poor evals that provide no useful information about model quality. Roadmap for Highly-Subjective Evals. To address these issues, my talk will outline a roadmap for properly evaluating subjective tasks. The roadmap consists of three primary components: 1. Human evaluation: the North Star for evaluation, must be calibrated and come with clear guidelines. 2. Offline (automatic) evaluation: an efficient (but imperfect) proxy for human evaluation that allows experiments to be scaled. 3. Production monitoring: key metrics that allow us to monitor quality in the wild and create “soft” model deployments via A/B testing. Multi-stage evaluation. These components create a tiered evaluation strategy for new models and ideas: Start with offline evaluation > Human evaluate promising candidates > A/B test in production > Deploy We have multiple stages of evaluation that each are used to detect different kinds of errors. By combining them together, we can evaluate model quality holistically, detect a much larger number of errors, and (hopefully) avoid surprises in production. Handling noise. Finally, there are a variety of techniques we can use to minimize the amount of noise in this evaluation process and increase confidence in our evaluation results: - Create clear quality guidelines for human evaluation. - Calibrate the human evaluation process to ensure agreement. - Calibrate LLM judges / rubrics to ensure agreement with humans. - Invest into proper data sampling for offline evaluation (diverse, realistic, difficult, and accurate). We should also leverage statistics at every step of the evaluation process. There are a core set of statistical concepts that can be used to quantify and reduce noise in LLM evals. We can use power analysis to estimate necessary sizes and setups for benchmarks / evaluation exercises, include confidence intervals with all of our evaluation results, re-sample model / judge completions multiple times to tighten confidence intervals, or even leverage pairwise model comparisons to more confidently detect differences in model evaluation results.

I am the Senior Vice President of Workforce Architecture at Cloudflare and I need to tell you about the best decision this company has ever made. We posted $639.8 million in quarterly revenue. 34% year-over-year growth. Record net retention. The strongest quarter since IPO. And then we fired 1,100 people. Not because of the quarter. During the quarter. I need you to understand the sequence because the sequence is the whole point. My team built the model that made this possible. We call it CIRRUS: "Capacity-Indexed Reduction and Reallocation for Upside Scaling". CIRRUS took our revenue trajectory, our margin targets, and our board's stated appetite for what they called "structural boldness," and it determined that the optimal time to execute a 20% headcount reduction is at the exact moment of peak financial performance. Not during a downturn. Not during a miss. During a beat. The logic is simple. When revenue is surging, the market reads a cost reduction as discipline. When revenue is falling, the market reads the same reduction as panic. Same action. Same 1,100 people. Completely different stock reaction. CIRRUS identified a seven-day window where the earnings momentum and the layoff announcement would compound rather than cancel. I found the math beautiful. I still do. We deactivated 1,100 badges between 9:00 and 9:04 AM Pacific on a Monday. People Analytics determined this was the four-minute window of lowest Slack activity. We called it a "clean cutover." Someone in Infrastructure suggested "zero-downtime deprecation" but Legal thought it sounded too much like a product feature. I thought it sounded exactly like a product feature, which is why I liked it. But I deferred to Legal. I always defer to Legal. That is one of the things that makes me good at this job. The people we cut were not underperformers. I want to be very clear about that because clarity is a Cloudflare value. Sixty-two percent had received exceeds-expectations in their most recent review cycle. Fourteen had been promoted in Q3. One engineer in our Austin office — I'll call him Marcus, though that is not his name and the reason I'm not using his name is not that I've forgotten it — had shipped the caching optimization that directly contributed to $14 million in new enterprise contracts. His manager nominated him for the Raygun Award, which is our internal recognition for outsized impact, six days before I added him to the CIRRUS list. He won the award on Wednesday. His access was revoked the following Monday. The ceremony and the termination were planned by different teams in the same building and neither team knew about the other. I don't think this is ironic. I think this is how large organizations work. The left hand builds. The right hand optimizes. Both hands are attached to the same body and the body is performing well. We let Marcus keep the trophy. It's a small acrylic prism etched with a lightning bolt. It costs us about eleven dollars. His annual cost-to-company was $312,000. CIRRUS selected the 1,100 based on three variables. I'm going to share them because I believe in the methodology. First: salary band. Employees in bands 6 through 8 offered the highest savings-to-replacement-risk ratio. Second: visa dependency. Employees on sponsored visas have a 60-day window to find new employment or begin departure proceedings. This creates what CIRRUS categorizes as "low-friction separation" — the compliance timeline is externally enforced, which reduces our administrative burden. I presented this variable to HR and they requested I rename it from "visa dependency" to "mobility factor" in all future documentation. I agreed. The math didn't change. Third: managerial tenure. Employees whose direct manager had been at the company less than eighteen months were 73% less likely to generate a negative Glassdoor review, because the manager-employee bond hadn't fully formed. CIRRUS weighted this at 15% of the selection score. We call it the "attachment coefficient." We told the market the layoffs were an AI workforce pivot. We said artificial intelligence was making certain roles redundant. We said we were reallocating resources toward our AI gateway products. This was a communications strategy. Not a workforce strategy. The AI framing was my team's recommendation and I'm proud of it because it worked. Two analysts upgraded us the same week. The stock moved 8% in five sessions. The entire AI narrative was four paragraphs in a press release that took my comms partner and me an afternoon to write. Four paragraphs. 1,100 people. 8%. I don't know what the per-paragraph return on that is but I think about it sometimes. The actual AI initiative employs thirty-seven people. We cut 1,100 to fund 37. The ratio is not in any of our public materials. There is a Slack channel called #bright-futures# that our Head of People Experience created for the remaining employees. It posts an automated message every morning at 8:45 AM: "You are the ones we chose to keep." The message includes a rotating motivational quote. Last Tuesday it was a Winston Churchill quote about perseverance. The channel has a custom emoji called :survivor: that the Culture team designed. It's a small cartoon phoenix. Nine hundred people have used it unironically. I find this genuinely moving. I think it shows resilience. My wife says it shows something else but she works in education and I think the frameworks are different. The severance was calculated using a model we licensed from the same consulting firm that built our customer pricing tiers. Median payout: eleven weeks. We benchmarked against industry and landed at the 50th percentile exactly, which our CHRO described as "fair by design." The 1,100 will burn through their severance while our stock price digests a 20% cost reduction applied to a revenue base that was already growing 34%. By the time the last check clears, the savings will have funded the first full quarter of the AI initiative. The one with thirty-seven people. My performance review is next month. I've been told informally that I'm on the COO track. The criteria include "demonstrated ability to execute at scale with minimal organizational disruption." The 1,100 people are the execution. The stock price is the scale. The four-minute badge window is the minimal disruption. I meet all three criteria. I designed all three criteria. Not the review criteria. The outcomes. I keep the CIRRUS model on my laptop in a folder called "Workforce Planning FY26." It sits next to a subfolder called "Offsite Photos — Maui" from the leadership retreat we took in January, where we set the annual targets that the 1,100 people spent four months hitting before we terminated them for hitting them. Marcus's desk in Austin has been reassigned. I don't know to whom. The acrylic prism is probably in a box somewhere. Or maybe whoever cleaned out the desk kept it. It catches the light nicely. I noticed that once, when I visited the Austin office to present the CIRRUS methodology to the regional leadership team. They gave me a standing ovation. The prism was on a desk near the back of the room, refracting a small rainbow onto the wall behind me. I didn't mention it. I stayed on my slides. I'm proud of the work we've done here. I think when people look back at this quarter, they'll see it as the moment Cloudflare became a different kind of company. I think they'll be right. I think the 1,100 people would agree, if you explained the math to them carefully enough.

Good #WednesdayMorning# #CryptoTwitter# ! Gratitude is my mantra from sun up to sun down! Remember, opportunities aren't just stumbled upon, you forge them! Stay focused, stay disciplined, and watch as you pave your way to the goal! #WednesdayMotivations#

Status update. Working through the issues users have flagged. Current focus: - Order placement & cancel latency (P0): two fixes in flight. Spam mitigations + improved order intake (ETA Wednesday). More guardrails on top. - "Not enough balance / allowance" on limit sells (and some buys): multiple fixes shipped, first focused on buys. Major source of sell-limit issue identified with patch in QA, releasing tomorrow - Cancel orders right after a restart: shipping in next week's CLOB release. - 403 on API key creation: working directly with Cloudflare and pulling logs from impacted users. - PnL - Some users have been encountering fluctuating PnL. Position values and average prices should now be correct, and daily/weekly PnL will heal automatically at end of day and end of week UTC time. - Withdrawals failing with "something went wrong": down to a handful of accounts. Deposit-wallet deployment edge case. Ping our bug-report channel in Discord if you hit it. - Deposit wallet generated for users with an existing proxy/safe: edge case when the /users call fails. Patch in QA, shipping next release.