Skip to Content

Watching Videos

Learn how to read what an online video shows, scene by scene, next to what it says, with the VidNavigator API and SDKs.

Overview

A transcript tells you what a video says. Many videos say little, or say something different from what they show: a product demo without a voice-over, a price written on screen, a recipe filmed in silence, a clip whose caption contradicts its footage.

The /watch endpoint reads the picture as well. In one response you get:

  • the transcript of the video, as on /transcript,
  • its scenes, in order: for each one its start and end in seconds, a description of what is visible, and on_screen_text, the text readable on screen.

It works with every platform supported by /transcript and /transcribe, YouTube included.

Prerequisites

  • A valid VidNavigator API key.
  • The Python or JavaScript SDK, version 2.4.0 or later, or any HTTP client.

Watch a video

from vidnavigator import VidNavigatorClient

client = VidNavigatorClient()

result = client.watch_video(video_url="https://www.tiktok.com/@nasa/video/7300000000000000000")

for scene in result.data.scenes:
    print(f"[{scene.start:.0f}s - {scene.end:.0f}s] {scene.description}")
    if scene.on_screen_text:
        print("   on screen:", scene.on_screen_text)
[0s - 4s] A studio with a news desk. A presenter in a dark suit faces the camera.
[4s - 12s] A presenter stands in front of a world map and points at Europe.
   on screen: BREAKING NEWS
[12s - 24s] Waves break over a sea wall; a few people walk away from the shore.
   on screen: LIVE - 18:42

Both SDKs run the call as a background job, so a long video works like a short one. Over HTTP, the synchronous POST /watch is limited to 10 minutes of video: use POST /watch/async beyond that.

Read the speech and the picture together

Scenes and transcript segments carry the same clock, in seconds from the beginning of the video. To know what was on screen while something was said, look for the scene that contains the start of the segment:

result = client.watch_video(video_url=url)

def scene_at(seconds):
    for scene in result.data.scenes:
        if scene.start <= seconds < scene.end:
            return scene
    return None

for segment in result.data.transcript or []:
    scene = scene_at(segment.start)
    print(f"[{segment.start:.0f}s] SAID:  {segment.text}")
    if scene:
        print(f"        SHOWN: {scene.description}")

A video without speech

A video without speech is not an error. The scenes are returned, transcript is empty and transcript_error says why. The same applies to a YouTube video that has no captions.

result = client.watch_video(video_url=url)

if result.data.transcript_error:
    print("No transcript:", result.data.transcript_error.error)

print(len(result.data.scenes), "scenes")

When you already hold the transcript, or do not need it, ask for the scenes alone:

result = client.watch_video(video_url=url, include_transcript=False)

Choose the level of detail

detail sets how closely the picture is followed. At every level the number of frames analyzed depends on the video: few on a fixed shot, many on a fast-cut clip.

detailUse it for
lowA rough idea of the picture: a talk, an interview, a long video that mostly shows the same thing.
autoThe default. Suits most videos.
highFast cuts, or something brief or small on screen: a slide shown for a moment, a price tag, a logo.

The response gives the number of frames that were analyzed in frames_analyzed.

Many videos, or long ones

Submit the jobs, then collect the results. Each job runs on VidNavigator's side, so your code only waits:

urls = [
    "https://www.tiktok.com/@user/video/1234567890",
    "https://www.instagram.com/reel/C86ZvEaqRmo/",
]

jobs = [client.submit_watch_video(video_url=url) for url in urls]

for url, job in zip(urls, jobs):
    result = job.result()
    print(url, "-", len(result.data.scenes), "scenes")

To be called back instead of waiting, pass a webhook_url: the job sends a watch.completed or watch.failed event. See Webhooks.

Billing

Watching a video is charged by the number of frames analyzed: 1 credit = 1,000 frames, with at least 334 frames per hour of video. The fetch of the video is charged as on /transcript, and speech-to-text is charged only when the video has no captions. At the default level, a fixed shot costs about 0.33 credit per hour of video, and an hour of television programme typically between 0.5 and 1 credit.

Pass include_usage=True (Python) or include_usage: true (JavaScript, HTTP) to see what a call was charged. See Billing on the endpoint page.

Limits

  • A live stream, or a clip of a few seconds, has no picture to read: the call fails with frames_not_available.
  • Videos longer than 2 hours are not accepted (video_too_long), except on YouTube.
  • The picture is followed scene by scene: something visible for an instant can be missed. Use detail="high" when that matters.
  • Uploaded files are not supported by this endpoint.

Next steps

Watch endpoint

Parameters, response fields and errors.

Async jobs

Submit, poll, and resume background jobs.

On this page