Joshua Xu
@joshua_xu_
Co-Founder and CEO @HeyGen | Make visual storytelling accessible to all.
755 Following 27.9K Followers
If your AI characters sound fake, the fix is almost always in how you write and DIRECT the dialogue in your prompt.
I recently started on the dialogue-heavy scenes for my AI feature film about Beethoven, and too many of the lines kept coming out wrong.
They’d sound too theatrical, too monotone, or too much like a practice read-through between actors.
(all this despite the fact that Seedance 2.5 is fantastic when it comes to realism.)
So through a whole lot of trial and error, I've found that these 5 things made the biggest impact on the quality of the dialogue delivery in my generations:
1. Paying close attention to the written punctuation used.
2. Breaking long lines into shorter ones, with intentional pauses.
3. Using bodily actions as punctuation (this one gave the biggest improvement BY FAR! In the attached clip, note the sniffles in between Beethoven's lines. That alone made the delivery feel much more natural.)
4. Using "anchor gestures" (he lifts his finger as if to say "one sec")
5. Just split your generations in two (stupid simple, but it really makes a difference.)
So if your characters keep sounding like they're reading from a teleprompter, give this a try!
Show more
Our OSS skills just got a huge refresh
update HyperFrames skills to unlock /reverse-any-video and the top generative models with Pro
sign in and get access to 10,000+ music tracks, sound effects, 75,000+ stock images and tts for free
npx hyperframes skills update
Show more
This is a great take.
I want to recommend a video on this topic too. This new video from @AliAbdaal is the clearest picture I've seen of where agentic editing is today.
Ali spent a day learning AI editing end-to-end and tested it against his own editor, and ran HyperFrames vs Remotion head-to-head.
If you want a real sense of where we are with visual examples, I highly recommend watching this:
Clip from Boss Battle attached below (42:18–43:13) 👇
Show more
A few thoughts on agentic video editing I’ve had over the last few days:
1. 99% of ads should be edited agentically now. If your editors are still editing in the timeline, they’re falling behind. I got a message from my senior VE saying that he’s putting out 5x edits/day now at the same quality he was before.
2. The conversation with video editors will be uncomfortable because the timeline is all they’ve ever known. Some will be hesitant and some will be fearful for their job. I think the best way to address this is head on. The tech is inevitable, but having taste is still going to be incredibly valuable to train and give feedback to agents. Those who have this will become more valuable, those who don’t will struggle.
3. Opus 5.5 can natively edit all of your ads without needing any MCPs or plugins. I’ve tried the alternatives and couldn’t find anything that was significantly better than doing everything inside of Claude.
4. Have your senior editor(s) build skills for each style of edit (e.g. mashups, UGC, AI animations…). You can build these by feeding 4-5 inspo edits to Claude and tell it to make a skill out of it. I prefer building lots of mini-skills to one big video editing skill, so that the context stays as relevant as possible.
5. Once you have a base skill (e.g. ugly ads edits), the best way to train it is through feedback loops. Have the skill re-edit the last 25 ads each senior edited then put them in an artifact, and have the senior VE give feedback on each. Keep doing this until the skill is refined and it understands what a good edit is.
6. If you do this, there’s no reason why you shouldn’t be able to make edits that need little/no feedback. Most of ours are approved after v1 or v2, sometimes v3.
7. Only A-player editors should be training the skills.
8. Re adoption in the team, the AI champion builds the v1. Then get out of the way. I’m not the best editor on my team, so my job is to build the harness that lets the best editors train the skills, not build the skills myself.
9. If you want to supercharge this process, have your top editors record their screen as they edit. Turn on Gmeet or loom and get them to narrate the reasoning behind their process. E.g. why they chose this cut, pacing or b-roll clip…this is all context that gets fed into your skills.
10. It takes time to build, train and refine these skills. You have to go through the dip to get out the other side. Budget time in for senior VEs to work on this - otherwise you’ll end up with skills that put out subpar edits and don’t save you much/any time.
11. Operate under the assumption that this is the worst video editing is ever going to be. Opus 5.5 can do most of our editing today…what will Opus 6/Fable 6 be able to do? The editing skills you build are only going to become more valuable as the tech evolves.
I still don’t think people realise how big video editing being unblocked is. Things that I thought were 3-6 months away are now on the table.
Will continue to share more as I learn. CRAZY, crazy times.
Show more
I'm not reading 300 pages, so I let Opus 5.5 create this video in @HyperFrames_ Studio to explain the 10 most interesting pieces of the S1 filing
Make an editorial-style "quality vs. cost" scatter plot of the text-to-video models on the Artificial Analysis leaderboard
HeyGen Video 1 went up against today's leading video models in a blind test on @ArtificialAnlys.
Our first general-purpose video model ranked #1# in text to video.
Try it:
Show more
HeyGen Video 1 went up against today's leading video models in a blind test on @ArtificialAnlys.
Our first general-purpose video model ranked #1# in text to video.
Try it:
Show more
the editing techniques and styles you thought were unattainable
now just one reference prompt away:
- the hook from an Instagram reel
- the b-roll style from a TikTok
- the captions from a YouTube Short
then automatically apply them to your footage with your story
get started ↓
Show more
I've been thinking about something as more and more AI agents start generating videos.
A lot of people are asking: if agents can create hundreds of videos, how are we supposed to review all of them?
I actually think that's the wrong way to look at it.
Today, most agents communicate with us through text. They do research, analyze information, and give us a long report to read.
But why does everything have to come back as text?
I think it depends on what the agent is trying to do.
If an agent needs me to make a decision, confirm something, or give feedback, text is probably the best format. It's quick, precise, and easy to interact with.
But if an agent is trying to explain something to me, that's a different story.
For example, recently I wanted to understand how JEV works, what it does, and the different use cases. Instead of spending an hour going through documents and links, I just asked my agent to research it and explain everything back to me in a video.
And I loved that experience.
I could just watch and listen. The combination of visuals, audio, and explanation made it much easier for me to build an understanding of the topic.
It got me thinking that we've been looking at agent-generated video mostly as a content creation problem.
But I think there's a much bigger opportunity here.
Video could become one of the most natural ways for agents to communicate with humans.
Not every interaction needs to be a video. Text is still great for back-and-forth communication.
But when an agent has done the work and wants to present what it learned, why shouldn't it just explain it to me through video?
I think we're going to see a lot more of this.
And personally, I'm pretty excited about that future.
Show more
BREAKING: HeyGen Video 1.0 (1080p, ultraquality) by @HeyGen, a post-trained model on MiniMax H3 by @MiniMax_AI, ranks #3# on both Design Arena's Video and Image to Video leaderboards, making HeyGen the #2# Video lab overall!
In Video, HeyGen Video 1.0 sits just behind Gemini Omni Flash by @GoogleDeepMind and ahead of Wan 3.0 by @Alibaba_Wan. In Image to Video, it also sits just behind MiniMax H3 Max by @MiniMax_AI, while generating faster than 71% of models.
Congrats to the @HeyGen team!
Show more
This month we launched HeyGen Video. Then HeyGen Voice went straight to #1# on Artificial Analysis.
At the same time, more and more of the model layer is going open source.
So I keep getting two questions. How will video and voice models play out from here? And if anyone can get a great model, where is the real competitive advantage?
Here's how I see it.
First, where I think this is going.
Open models will win in video and voice, the same way they did in LLMs. In LLMs, closed models led the frontier first. Then open models caught up. Today, open models serve most of the tokens.
I believe video and voice will follow the same path. It's simply a better world to live in. Open models mean more people building models, more apps, and more value for the end user.
And it's just getting started.
A year ago, the best open video models were second-tier. MiniMax H3 (Aug 2026) changed that. For the first time, an open video model was released at the frontier. That was the tipping point for the industry.
Voice is moving too. Fish Audio released their open model this week. I haven't tested it yet, but the demos look great.
So if the model is no longer the moat, what is?
I think LLMs and coding are the best reference we have. The model is shared. The advantage comes from what you build around it. Here's what that looks like in video.
1. Private video data is untouched.
Enterprises sit on huge amounts of private video and voice data. In the AI world, this is the least developed context of all. Open models finally let companies build on it.
2. Cost matters much more in video than in LLMs.
In LLMs, frontier models cost more than open models, but the value is so big that cost almost doesn't matter. Video is different. A frontier video model can cost close to $1, sometimes up to $10, for a few seconds of video.
The demand is there. Price is blocking it.
Open models give companies the chance to build very efficient inference, push the cost down, and unlock that demand. We're already seeing it. We released HeyGen Video as the fastest and cheapest model on the market, and it has been #1# on OpenRouter for more than a week in a row.
3. Speed without losing quality is the real skill.
Most inference companies know how to make models faster. That's low-level work: CUDA, GPU optimization, GPU scheduling.
Video is a different game. The hard part is the trade-off between quality and speed, and that is decided in post-training.
Post-training is a completely different game from pre-training. It's full of nuance. You curate the data. You steer the model in a specific direction. You make it faster while losing as little quality as possible.
We've been doing this at HeyGen for years. Cheap and best are no longer opposites.
Open video and voice models are just getting started. This is the most exciting time to be building in video.
Show more
everyone has a video they wish they could make
introducing "Reference Any Video" in HyperFrames Studio
now agents can tell your story in ANY format referenced
plus Brand kits: scan your site once for your colors, fonts and logo then applied to your video automatically
try now ↓
Show more
At HeyGen, we make one promise to our customers: your video will look like you, sound like you, and move like you.
"Sound like you" is the hard part.
First, the voice has to be yours. That's why we built our own voice model, and it's now ranked #1# in the arena.
But being similar is not enough. Almost every voice model today is trained on podcast data: a studio, a professional mic, a quiet room. That's great for voice agents, phone calls and voiceovers.
Video is a different problem. You might be outside. You might be in a busy office. A car might drive by behind you. And you speak differently to a camera than you do on the phone or a podcast.
So when you put a studio voice into a real scene, it feels off. People can't say why, but they stop believing it.
Some teams try to fix this with multimodal models. So far the quality isn't there, and customers can't get controllability they want.
We took a different path. We trained our voice model entirely on video data. Years of post-training avatar video models gave us a large proprietary video dataset to learn from. The model reads the scene and adapts both the sound and the delivery to match it.
Today we're launching it: the world’s first video- adaptive voice model, together with state-of-the-art audio quality.
Video that looks real should sound real too.
Show more
HeyGen Voice is the world’s best voice model, ranked #1# on @ArtificialAnlys, and the first built for video.
It performs your script in your voice, with your pacing and expression, adapting to the acoustics of the scene.
The research behind it:
Show more
another amazing video made with HyperFrames!
I gave Gemini 4 Argon our Drive folder with launch videos, our brand + motion guidelines, and the challenge to make a cool trailer for itself. I love Argon's creativity - I did not write any of the text. Built with HyperFrames, music by Lyria (also prompted by Argon) 🚀
Show more
Today we’re launching HeyGen Voice, our first in-house voice model, ranked #1# on @ArtificialAnlys.
We built it for authentic performance: your voice, natural and expressive, sounding like you at your best.
Incredibly proud of our team. Try it in HeyGen or build with our API:
Show more
We’ve been turning our articles into social-ready videos. Here’s how to do it with @hyperframes_:
1. Give Claude the article and ask for an opening hook and a sequence of scenes. Confirm the choices against the piece.
2. Ask for one still image per scene, with the onscreen text, and colors from the cover image. Check the story before approving animation.
3. Have Claude animate the approved storyboard with Hyperframes. For an animated cover, use MiniMax in @FLORA and give Claude the resulting clip.
4. Watch the preview. Flag unreadable text, cropped letters, or scenes that pass too quickly. Review again after each round of changes.
Once the video works, ask Claude to save the process and your corrections as a reusable skill—including the approval steps.
For @kplikethebird’s Compound Writing guide, @beckyisj made an 18-second video showing flying papers from the cover, followed by the writing process in a loop.
Read the full piece:
Show more
Since we launched @HyperFrames_ Studio on Monday, we’ve seen many great videos from customers: product launches, explainers, tutorials. not demo, but real published videos.
Long-form explainers and 30s product launches are our current sweet spot (rapidly expanding to more format). If you’re making either one, try the Studio. It’s the Opus 4.6 moment for AI coding, but for video.
Show more
10 things AI video editing did this week that used to eat hours of my life while creating content:
1. cut 34 min of raw footage + screen share down to a 16:00 video
2. synced my multicam clips, perfectly timed and formatted
3. matched the color grade between my iPhone and my DSLR footage
4. iterated motion graphics for a whole video in Higgsfield
5. turned an excel spreadsheet into a motion graphics presentation
6. found 10 potential shorts in a podcast... then chopped them up
7. pulled still frames for my thumbnail and exported the best ones
8. grabbed archival footage + memes from across the web. didn't download a thing
9. found music and sound effects and timed them to the cut
10. made me ask IT for more usage credits. FOUR separate times
a few of these came from @HyperFrames_ Studio, the editor @HeyGen dropped this week (that's it in the vid). my full review + tutorial is coming soon 👀
fellow creators: what's your #11#?
Show more
this is huge for presentation use cases. you can now turn any doc or deck into a motion video in minutes.
opus 5.5 is really good at video composition. excited to see claude motion launch natively in claude, and proud to be a partner for the editing step.
want more control? export your claude motion to hyperframes studio and keep editing.
seamless.
Show more
Claude Dashboards and Claude Motion are in beta today.
Ask Claude to turn your data into live dashboards and your ideas into animated explainers.
We're the #1# video model on OpenRouter today!
Today we launch Code2Video, our first benchmark built on HyperFrames.
can a model write code that becomes a video worth watching?
How we designed it:
