ShortsAIHow a long video becomes a vertical clip
faster-whisper transcribes the video with word-level timestamps, PySceneDetect finds the scene boundaries, and Google Gemini 3.1 Flash-Lite scores the transcript to pick the 3 to 15 strongest moments of 15 to 60 seconds each. Each moment is then cut with FFmpeg and reframed to 9:16 with MediaPipe face tracking.
The two stages that decide whether a clip is usable are moment scoring and reframing. Everything else is mechanical.
ShortsAI is free, and every stage below runs on every video you paste at shortsai.app.
What is ShortsAI?
ShortsAI is a free AI clip generator that turns long videos (podcasts, webinars, livestreams, interviews) into vertical 9:16 clips for TikTok, Instagram Reels and YouTube Shorts.
The pipeline, stage by stage
Transcription with word-level timestamps
faster-whisper transcribes the source audio and returns a timestamp for every individual word, not just every sentence. Word-level timing is what makes both the clip boundaries and the burned-in subtitles land on the right frame.
Scene boundary detection
PySceneDetect scans the video for cuts and hard visual transitions. These boundaries stop a clip from starting mid-shot, which is the most common reason an auto-generated clip reads as machine-made.
Viral moment scoring
Google Gemini 3.1 Flash-Lite receives the timestamped transcript and the scene boundaries together, and returns the 3 to 15 strongest segments of 15 to 60 seconds. Each one is scored on hook strength, emotional payload and how well it stands alone without the surrounding context.
Vertical reframing
Each clip is cropped from 16:9 to 9:16 in one of two modes. TRACK mode follows a single subject with MediaPipe face detection and a YOLOv8 fallback, damped by a stabiliser that holds the camera still inside a safe zone instead of chasing every head movement. GENERAL mode handles group shots and landscapes by keeping the full width over a blurred backdrop.
Subtitles, hooks and effects
Subtitles are generated from the word-level transcript and burned in with FFmpeg. An AI-written hook overlay covers the first seconds. Optional Gemini-generated FFmpeg filter chains handle colour and transitions.
Dubbing and publishing
ElevenLabs dubbing translates the audio into 30+ languages while preserving the speaker's voice, and the dubbed track is re-transcribed so the subtitles match the new language. Finished clips post directly to TikTok, Instagram Reels and YouTube Shorts.
Why does moment scoring need the transcript and the scenes together?
A transcript alone finds a good sentence but has no idea whether the shot cuts halfway through it. Scene boundaries alone find clean cuts with nothing worth saying between them. Passing both to the model at once is what lets it pick a segment that is both quotable and visually intact, which is the difference between a clip a person would watch and one that merely starts and stops in the right places.
Why does the camera hold still instead of following the face exactly?
Because a crop that tracks a face frame by frame produces visible swinging, and the swinging reads as amateur even when the framing is technically correct. The reframing keeps a safe zone around the subject and only moves the crop when they leave it, then damps the movement on the way. A speaker tracker sits on top to stop the crop from flipping between people every time someone nods, and to hold position through brief occlusions.
How long does it take?
On a typical CPU, an 8-minute source video takes roughly 5 to 8 minutes end to end. On an NVIDIA GPU the same video takes about 50 seconds. The gap is almost entirely transcription and encoding; the model call is a small fraction of it.
What does it cost to run?
ShortsAI comes in two editions and they are priced very differently, so it is worth being precise. ShortsAI has no paid tier at all. Free at shortsai.app, with nothing to install. You paste your own Google Gemini API key (free tier: 1,500 requests a day) in Settings and it stays encrypted in your browser. No plans, no watermark, no usage cap, no account. The site is supported by ads. You bring your own Google Gemini API key, whose free tier covers 1,500 requests a day; the site is supported by ads.
Common questions
- What AI model does ShortsAI use to find viral moments?
- Google Gemini 3.1 Flash-Lite. It receives the word-level transcript with timestamps together with PySceneDetect scene boundaries, and returns 3 to 15 segments of 15 to 60 seconds scored on hook strength, emotional payload and whether the segment stands alone without surrounding context.
- How does the automatic vertical cropping work?
- Two modes. TRACK mode follows a single subject using MediaPipe face detection with a YOLOv8 fallback, stabilised so the crop holds still inside a safe zone instead of following every movement. GENERAL mode handles group shots and landscapes by preserving the full width over a blurred backdrop.
- Can it dub clips into other languages?
- Yes, into more than 30 languages via ElevenLabs, preserving the original speaker's voice characteristics. The dubbed audio is then re-transcribed so the burned-in subtitles match the new language rather than the original.