Skip to Content

Extracting Structured Data

Turn any online video or uploaded file into clean, validated JSON using a JSON Schema you define, or from a plain-words description of what you need.

If transcripts are for humans, extraction is for software. The Extract API lets you define the exact fields you want as a standard JSON Schema (or describe them in plain words), then returns structured JSON that matches your schema so you can send the result straight into a database, CRM, workflow, or AI pipeline.

What This Guide Helps You Do

By the end of this guide, you will know how to:

  • choose the right extraction endpoint
  • design a schema the model can follow reliably, or reuse a Pydantic model or Zod schema you already have
  • use what_to_extract to improve focus, or on its own when you have no schema yet
  • understand auto-transcription, caching, and billing
  • move from one test video to a repeatable production workflow

Why This Is More Than Just an LLM Prompt

The real challenge is not only extracting structured data. It is getting the video content into a usable text layer in the first place.

With VidNavigator, the workflow is end to end:

  • fetch the transcript when the platform already exposes one
  • auto-transcribe videos with speech-to-text when no transcript exists, YouTube included
  • run AI extraction on that text using your schema
  • return validated JSON you can use immediately in software

That is the main difference. You are not expected to manually retrieve captions, run a separate speech-to-text step, clean the text, and then prompt a model yourself.

Compared with building that flow manually, VidNavigator gives you:

  • transcript retrieval and speech-to-text built into the same extraction workflow
  • one API call instead of stitching together multiple tools
  • a fixed response shape defined by your schema
  • validated output instead of free-form text
  • prompt compilation that is cached for 2 hours for repeated schemas

This is what makes video data extraction practical at scale, especially for platforms where transcripts are inconsistent, missing, or not easily accessible.

Choose the Right Endpoint

Online Videos

Use /v1/extract/video when you have a public video URL from YouTube, Instagram, TikTok, Facebook, X, Vimeo, Dailymotion, Loom, Snapchat, and similar platforms.

Uploaded Files

Use /v1/extract/file when the content is already in your VidNavigator library and you want to extract from a file_id.

extract/video can auto-transcribe videos that have no transcript, YouTube included, when transcribe=true (the default). extract/file does not transcribe for you, so the file must already have a transcript.

Long videos: when speech-to-text is needed, the synchronous /v1/extract/video accepts media up to 10 minutes and rejects longer media with 400 video_too_long. Use /v1/extract/video/async — same body, no duration cap, result by polling or webhook. See Async Jobs.

How Extraction Works

The Extract API runs a simple 2-step pipeline:

  1. You send a schema, a what_to_extract instruction, or both. With what_to_extract alone, the schema is written from it first.
  2. VidNavigator compiles an optimized extraction plan and caches it for 2 hours.
  3. The plan is applied to the transcript for the current video or file.
  4. The response is validated so the output matches your schema.

That cache matters. The first call for a new schema usually has a small compilation overhead. Reusing the same schema on later calls is faster because the compiled prompt is reused automatically.

Start with a Small Schema

The fastest way to succeed is to start with 2-4 fields, validate the output, then expand.

A schema is a standard JSON Schema whose root is an object: the fields go under properties. Here is a good starter schema for product review videos:

{
  "type": "object",
  "properties": {
    "products_mentioned": {
      "type": "array",
      "description": "Products mentioned in the video",
      "items": {
        "type": "string",
        "description": "Product name"
      }
    },
    "main_claim": {
      "type": "string",
      "description": "The single most important claim or conclusion made in the video"
    },
    "sentiment": {
      "type": "string",
      "description": "Overall sentiment toward the main product discussed",
      "enum": ["positive", "neutral", "negative"]
    }
  }
}

Quickstart

Extract from an Online Video

Use this when you have a public video URL:

from vidnavigator import VidNavigatorClient

client = VidNavigatorClient()  # reads VIDNAVIGATOR_API_KEY

resp = client.extract_video_data(
    video_url="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    what_to_extract="Focus on product names, pricing, and the speaker's overall verdict",
    schema={
        "type": "object",
        "properties": {
            "products_mentioned": {
                "type": "array",
                "description": "Products mentioned in the video",
                "items": {"type": "string", "description": "Product name"},
            },
            "pricing_signals": {
                "type": "array",
                "description": "Any price points, pricing plans, or cost references",
                "items": {"type": "string", "description": "Pricing detail"},
            },
            "verdict": {
                "type": "string",
                "description": "Final recommendation or overall verdict from the speaker",
            },
        },
    },
)
print(resp.data)              # a dict shaped like your schema
print(resp.video_info.title)  # the video's metadata

The 2.1.0 SDKs run video extraction through /extract/video/async, so videos of any length work with the same call. To run many extractions in parallel, use client.submit_extract_video_data(...) (Python) or vn.extractVideo.submit(...) (JavaScript). The SDK result contains the extracted data and the video's metadata in video_info (resp.video_info in Python, video_info in JavaScript).

Extract from an Uploaded File

Use this when the transcript already lives in your VidNavigator workspace:

resp = client.extract_file_data(
    file_id="file_abc123",
    what_to_extract="Extract action items and owners from this meeting recording",
    schema={
        "type": "object",
        "properties": {
            "action_items": {
                "type": "array",
                "description": "Tasks agreed on during the meeting",
                "items": {"type": "string", "description": "Action item"},
            },
            "owners": {
                "type": "array",
                "description": "People assigned to specific tasks",
                "items": {"type": "string", "description": "Person name and task if available"},
            },
            "deadline_mentions": {
                "type": "array",
                "description": "Any due dates or timing commitments mentioned",
                "items": {"type": "string", "description": "Deadline detail"},
            },
        },
    },
)
print(resp.data)

Reuse a Pydantic Model or a Zod Schema

Because schema is standard JSON Schema, the models you already have can be passed as they are: a Pydantic model's model_json_schema() in Python, a Zod schema converted to JSON Schema in JavaScript. Field descriptions become the instructions for each field, Literal and z.enum become enums, and nested models (sent as $ref / $defs) are followed.

from typing import Literal, Optional
from pydantic import BaseModel, Field

class Product(BaseModel):
    name: str = Field(description="Product name")
    price: Optional[float] = Field(None, description="Price in the currency shown")

class Review(BaseModel):
    products: list[Product] = Field(description="Products shown or named in the video")
    sentiment: Literal["positive", "negative", "neutral"] = Field(
        description="Overall sentiment toward the main product"
    )

resp = client.extract_video_data(
    video_url="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    schema=Review.model_json_schema(),
)
print(resp.data)  # {"products": [...], "sentiment": "positive"}

A field the video does not give comes back as null, even when the schema lists it as required. Validate data with your model only if its fields accept null.

Without a Schema

When you do not have a schema yet, send what_to_extract alone, in plain words. The schema is written from it and returned beside the data, as a JSON Schema: schema in the API answer, resp.extraction_schema in Python, schema in JavaScript. It needs version 2.6.0 of the SDKs.

resp = client.extract_video_data(
    video_url="https://www.tiktok.com/@user/video/1234567890",
    what_to_extract="Every product shown or mentioned, with its price and whether it is sponsored",
)
print(resp.data)               # {"products": [{"name": "Wireless earbuds", "price": 49.99, "sponsored": True}]}
print(resp.extraction_schema)  # {"type": "object", "properties": {"products": {...}}, ...}

# Later requests: pass the schema back to get exactly the same fields
resp = client.extract_video_data(video_url=other_url, schema=resp.extraction_schema)

The answer carries the data and the schema it follows:

{
  "status": "success",
  "data": {
    "products": [
      { "name": "Wireless earbuds", "price": 49.99, "sponsored": true }
    ]
  },
  "schema": {
    "type": "object",
    "properties": {
      "products": {
        "type": "array",
        "description": "Every product shown or mentioned in the video",
        "items": {
          "type": "object",
          "properties": {
            "name": { "type": "string", "description": "Name of the product" },
            "price": { "type": "number", "description": "Price stated for the product" },
            "sponsored": { "type": "boolean", "description": "Whether the product is presented as sponsored" }
          },
          "required": ["name", "price", "sponsored"]
        }
      }
    },
    "required": ["products"]
  },
  "video_info": { "title": "My favorite tech this month" }
}
  • The same text gives the same fields, so repeated calls with the same what_to_extract stay consistent.
  • To edit the fields (rename one, add an enum, tighten a description), start from the returned schema and send it as schema.
  • With both a schema and what_to_extract, the schema sets the fields and the text guides the extraction. With neither, the request is rejected with 400 missing_parameter.
  • The async endpoint returns it as schema beside result on GET /extract/video/{task_id}.

Fields That Are on Screen, Not Said

By default the fields are filled from what is said. With include_scenes the extraction also reads what the video shows — its scenes, with the text readable on screen — so a field can come from a price tag, a product on a shelf, a caption or a slide. It needs version 2.5.0 of the SDKs.

resp = client.extract_video_data(
    video_url="https://www.dailymotion.com/video/x84243e",
    schema={
        "type": "object",
        "properties": {
            "dish": {"type": "string", "description": "Name of the dish"},
            "ingredients": {
                "type": "array",
                "description": "Every ingredient, with its quantity",
                "items": {"type": "string", "description": "One ingredient and its quantity"},
            },
        },
    },
    include_scenes=True,
)
print(resp.data["ingredients"])  # seven ingredients, read from the text on screen
print(resp.frames_analyzed)      # 30

This video only has music: without include_scenes, the same schema comes back empty.

With include_scenes, the picture is also looked at a second time with your fields in mind, so that a name on a lower third, a price on a label or a logo in a corner is read even when the scene descriptions leave it out. This second look is part of the extraction and costs nothing more.

The frames are charged on top of the extraction (1 credit per 2,000 frames analyzed, with at least 667 frames per hour of video) and given back if the extraction fails. On an uploaded file, include_scenes uses the scenes kept with the file: see Scenes of a file.

Where Each Value Comes From

Every extraction says where its values come from. Beside data, evidence gives, for each field that has a value, the moments of the video it was taken from: start and end in seconds, and in, which says how it was found:

  • speech: said in the video;
  • picture: shown or written on screen (with include_scenes);
  • post: read in the post's own text, its title, channel or description. These have no times.
{
  "data": {"people": ["Reid Wiseman", "Christina H. Koch"], "summary": "Two astronauts talk about the mission."},
  "evidence": {
    "people": [{"start": 0.0, "end": 37.0, "in": "picture"}, {"in": "post"}],
    "summary": [{"start": 0.0, "end": 24.5, "in": "speech"}]
  }
}

Use it to check a value, or to send someone to the exact moment in a player. A field left empty has no entry. The SDKs read it from version 2.7.0 (resp.evidence in Python, evidence on the JavaScript result).

When to Use what_to_extract

Your schema defines the output shape. what_to_extract tells the model where to focus. On its own, without a schema, it also decides the fields: see Without a Schema.

Use it next to a schema when:

  • the transcript covers multiple topics and you care about one of them
  • you want the model to prioritize a specific lens such as pricing, claims, objections, or action items
  • the field descriptions are correct but still too broad

Good examples:

  • Focus on feature claims, competitor mentions, and pricing strategy.
  • Extract only factual claims and cited statistics. Ignore jokes and off-topic banter.
  • Prioritize buyer intent, objections, and next steps from the sales call.

Schema Format

schema is a standard JSON Schema whose root is an object, {"type": "object", "properties": {...}}, as Pydantic, Zod or a structured-output definition produce it. It is the only format accepted.

typeExtra keysExample value
string, number, integer, boolean—"upbeat", 4.5, 3, true
arrayitems: the schema of one element["love", "commitment"]
objectproperties: the sub-fields{"name": "Rick", "role": "singer"}
  • Enums: enum restricts a field to a list of values: {"type": "string", "enum": ["positive", "negative", "neutral"]}. A const is an enum of one value.
  • Descriptions: give each property a description saying what to put in it. Without one, the field's title or its name is used.
  • References: local $refs (#/$defs/... or #/definitions/..., what model_json_schema() emits for nested models) are followed.
  • Nullable types such as ["number", "null"] or anyOf: [{"type": "number"}, {"type": "null"}] count as their type.
  • Hints: format, pattern, minimum, maximum and similar keywords are passed on to guide the extraction.
  • required and additionalProperties change nothing: every field is returned, and a field the video does not give is null.

Limits:

  • maximum 10 fields per level (root fields, and sub-fields of each object)
  • maximum 3 nesting levels
  • not supported: anyOf / oneOf between several types, recursive schemas

A schema that breaks these rules, or is not a JSON Schema with an object at its root, is rejected with 400 invalid_schema before anything is transcribed or billed. The error message says what to fix.

Write descriptions like instructions, not labels. Primary pricing strategy discussed, in one sentence is much better than pricing.

Best Practices That Improve Results

1. Be specific in field descriptions

Weak:

{
  "type": "object",
  "properties": {
    "topic": {
      "type": "string",
      "description": "Topic"
    }
  }
}

Better:

{
  "type": "object",
  "properties": {
    "primary_topic": {
      "type": "string",
      "description": "Primary topic discussed in the video, in 5 to 10 words"
    }
  }
}

2. Use enums when the answer should come from a fixed list

If a field is really a classification, add an enum with the allowed values instead of leaving a free string. That makes downstream automation much easier.

3. Start simple, then expand

Do not begin with a 10-field nested schema unless you already know it works. Start with the 2-3 fields that matter most, test them on a few videos, then add more.

4. Write descriptions in the output language you want

The response is returned in the same language as your schema descriptions. If you write field descriptions in French, the output will also come back in French.

Ready-to-Use Ideas

Common extraction patterns include:

  • lead generation: companies, buyer intent, pain points, next steps
  • market research: competitor mentions, feature claims, pricing strategy, objections
  • creator analysis: hooks, quotes, sponsored mentions, content format
  • RAG ingestion: dense summary, entities, claims, topics, language code
  • fact-checking: claims, statistics, cited sources, controversy markers

Online Videos vs. Uploaded Files

POST /extract/video

Choose this endpoint when:

  • you already have a public video URL
  • you want VidNavigator to fetch the transcript for you
  • you want automatic speech-to-text fallback when a video has no transcript

Important behavior:

  • transcribe=true by default
  • automatic transcription applies to every platform, YouTube included
  • when speech-to-text runs, media longer than 10 minutes must go through POST /extract/video/async

POST /extract/file

Choose this endpoint when:

  • the media is already uploaded to your VidNavigator account
  • you want to extract from private or internal content
  • you already completed transcription earlier in your workflow

Typical file workflow:

  1. Upload the file.
  2. Generate a transcript if needed.
  3. Call /v1/extract/file with the resulting file_id.

Billing and Performance

Each extraction consumes analysis_request units. One unit covers up to 15,000 total tokens; longer transcripts scale with:

ceil(total_tokens / 15000)

A few practical notes:

  • the first call for a new schema includes prompt compilation overhead
  • repeated calls with the same schema reuse the cached plan for 2 hours
  • extract/video may also trigger transcription_hour billing when auto-transcription is used
  • failed requests are reverted and are not billed

Troubleshooting

transcript_not_available

This usually means the source does not have a usable transcript and auto-transcription is either disabled or unavailable.

  • Keep transcribe=true on /extract/video: a video without a transcript is then transcribed from its audio.
  • For uploaded files, transcribe the file before calling /extract/file.

video_too_long

The video needs speech-to-text and is longer than 10 minutes. Send the same request to /v1/extract/video/async and collect the result by polling or webhook.

invalid_schema

This usually means the schema is not a JSON Schema with "type": "object" and properties at its root, a field has no type or an unsupported one (types are the lowercase JSON Schema types listed above), nesting is too deep, or the schema exceeds the limits above. The error message says which field to fix.

missing_parameter

The request has neither a schema nor a what_to_extract. Send at least one of them.

schema_generation_failed

A request with what_to_extract alone could not get its schema written. This is temporary: retry, or pass a schema.

Results are too broad

Tighten the schema descriptions and add a more targeted what_to_extract instruction.

First request is slower than later requests

That is expected when a schema is new. The compiled extraction plan is cached for 2 hours, so repeated schemas are faster.

Next Steps

On this page