Watching Videos
Learn how to read what an online video shows, scene by scene, next to what it says, with the VidNavigator API and SDKs.
Overview
A transcript tells you what a video says. Many videos say little, or say something different from what they show: a product demo without a voice-over, a price written on screen, a recipe filmed in silence, a clip whose caption contradicts its footage.
The /watch endpoint reads the picture as well. In one response you get:
- the transcript of the video, as on
/transcript, - its scenes, in order: for each one its
startandendin seconds, adescriptionof what is visible, andon_screen_text, the text readable on screen.
It works with every platform supported by /transcript and /transcribe, YouTube included.
Prerequisites
- A valid VidNavigator API key.
- The Python or JavaScript SDK, version 2.4.0 or later, or any HTTP client.
Watch a video
from vidnavigator import VidNavigatorClient
client = VidNavigatorClient()
result = client.watch_video(video_url="https://www.tiktok.com/@nasa/video/7300000000000000000")
for scene in result.data.scenes:
print(f"[{scene.start:.0f}s - {scene.end:.0f}s] {scene.description}")
if scene.on_screen_text:
print(" on screen:", scene.on_screen_text)[0s - 4s] A studio with a news desk. A presenter in a dark suit faces the camera.
[4s - 12s] A presenter stands in front of a world map and points at Europe.
on screen: BREAKING NEWS
[12s - 24s] Waves break over a sea wall; a few people walk away from the shore.
on screen: LIVE - 18:42Both SDKs run the call as a background job, so a long video works like a short one. Over HTTP, the synchronous POST /watch is limited to 10 minutes of video: use POST /watch/async beyond that.
Read the speech and the picture together
Scenes and transcript segments carry the same clock, in seconds from the beginning of the video. To know what was on screen while something was said, look for the scene that contains the start of the segment:
result = client.watch_video(video_url=url)
def scene_at(seconds):
for scene in result.data.scenes:
if scene.start <= seconds < scene.end:
return scene
return None
for segment in result.data.transcript or []:
scene = scene_at(segment.start)
print(f"[{segment.start:.0f}s] SAID: {segment.text}")
if scene:
print(f" SHOWN: {scene.description}")A video without speech
A video without speech is not an error. The scenes are returned, transcript is empty and transcript_error says why. The same applies to a YouTube video that has no captions.
result = client.watch_video(video_url=url)
if result.data.transcript_error:
print("No transcript:", result.data.transcript_error.error)
print(len(result.data.scenes), "scenes")When you already hold the transcript, or do not need it, ask for the scenes alone:
result = client.watch_video(video_url=url, include_transcript=False)Choose the level of detail
detail sets how closely the picture is followed. At every level the number of frames analyzed depends on the video: few on a fixed shot, many on a fast-cut clip.
detail | Use it for |
|---|---|
low | A rough idea of the picture: a talk, an interview, a long video that mostly shows the same thing. |
auto | The default. Suits most videos. |
high | Fast cuts, or something brief or small on screen: a slide shown for a moment, a price tag, a logo. |
The response gives the number of frames that were analyzed in frames_analyzed.
Many videos, or long ones
Submit the jobs, then collect the results. Each job runs on VidNavigator's side, so your code only waits:
urls = [
"https://www.tiktok.com/@user/video/1234567890",
"https://www.instagram.com/reel/C86ZvEaqRmo/",
]
jobs = [client.submit_watch_video(video_url=url) for url in urls]
for url, job in zip(urls, jobs):
result = job.result()
print(url, "-", len(result.data.scenes), "scenes")To be called back instead of waiting, pass a webhook_url: the job sends a watch.completed or watch.failed event. See Webhooks.
Billing
Watching a video is charged by the number of frames analyzed: 1 credit = 1,000 frames, with at least 334 frames per hour of video. The fetch of the video is charged as on /transcript, and speech-to-text is charged only when the video has no captions. At the default level, a fixed shot costs about 0.33 credit per hour of video, and an hour of television programme typically between 0.5 and 1 credit.
Pass include_usage=True (Python) or include_usage: true (JavaScript, HTTP) to see what a call was charged. See Billing on the endpoint page.
Limits
- A live stream, or a clip of a few seconds, has no picture to read: the call fails with
frames_not_available. - Videos longer than 2 hours are not accepted (
video_too_long), except on YouTube. - The picture is followed scene by scene: something visible for an instant can be missed. Use
detail="high"when that matters. - Uploaded files are not supported by this endpoint.
Next steps
Watch endpoint
Parameters, response fields and errors.
Async jobs
Submit, poll, and resume background jobs.