Skip to main content

How to return structured data from a model

It is often useful to have a model return output that matches some specific schema. One common use-case is extracting data from arbitrary text to insert into a traditional database or use with some other downstrem system. This guide will show you a few different strategies you can use to do this.

Prerequisites

This guide assumes familiarity with the following concepts:

The .withStructuredOutput() method

There are several strategies that models can use under the hood. For some of the most popular model providers, including Anthropic, Google VertexAI, Mistral, and OpenAI LangChain implements a common interface that abstracts away these strategies called .withStructuredOutput.

By invoking this method (and passing in JSON schema or a Zod schema) the model will add whatever model parameters + output parsers are necessary to get back structured output matching the requested schema. If the model supports more than one way to do this (e.g., function calling vs JSON mode) - you can configure which method to use by passing into that method.

Let’s look at some examples of this in action! We’ll use Zod to create a simple response schema.

Pick your chat model:

Install dependencies

yarn add @langchain/openai 

Add environment variables

OPENAI_API_KEY=your-api-key

Instantiate the model

import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
model: "gpt-4o-mini",
temperature: 0
});
import { z } from "zod";

const joke = z.object({
setup: z.string().describe("The setup of the joke"),
punchline: z.string().describe("The punchline to the joke"),
rating: z.number().optional().describe("How funny the joke is, from 1 to 10"),
});

const structuredLlm = model.withStructuredOutput(joke);

await structuredLlm.invoke("Tell me a joke about cats");
{
setup: "Why don't cats play poker in the wild?",
punchline: "Too many cheetahs.",
rating: 7
}

One key point is that though we set our Zod schema as a variable named joke, Zod is not able to access that variable name, and therefore cannot pass it to the model. Though it is not required, we can pass a name for our schema in order to give the model additional context as to what our schema represents, improving performance:

const structuredLlm = model.withStructuredOutput(joke, { name: "joke" });

await structuredLlm.invoke("Tell me a joke about cats");
{
setup: "Why don't cats play poker in the wild?",
punchline: "Too many cheetahs!",
rating: 7
}

The result is a JSON object.

We can also pass in an OpenAI-style JSON schema dict if you prefer not to use Zod. This object should contain three properties:

  • name: The name of the schema to output.
  • description: A high level description of the schema to output.
  • parameters: The nested details of the schema you want to extract, formatted as a JSON schema dict.

In this case, the response is also a dict:

const structuredLlm = model.withStructuredOutput({
name: "joke",
description: "Joke to tell user.",
parameters: {
title: "Joke",
type: "object",
properties: {
setup: { type: "string", description: "The setup for the joke" },
punchline: { type: "string", description: "The joke's punchline" },
},
required: ["setup", "punchline"],
},
});

await structuredLlm.invoke("Tell me a joke about cats", { name: "joke" });
{
setup: "Why was the cat sitting on the computer?",
punchline: "Because it wanted to keep an eye on the mouse!"
}

If you are using JSON Schema, you can take advantage of other more complex schema descriptions to create a similar effect.

You can also use tool calling directly to allow the model to choose between options, if your chosen model supports it. This involves a bit more parsing and setup. See this how-to guide for more details.

Specifying the output method (Advanced)

For models that support more than one means of outputting data, you can specify the preferred one like this:

const structuredLlm = model.withStructuredOutput(joke, {
method: "json_mode",
name: "joke",
});

await structuredLlm.invoke(
"Tell me a joke about cats, respond in JSON with `setup` and `punchline` keys"
);
{
setup: "Why don't cats play poker in the jungle?",
punchline: "Too many cheetahs!"
}

In the above example, we use OpenAI’s alternate JSON mode capability along with a more specific prompt.

For specifics about the model you choose, peruse its entry in the API reference pages.

(Advanced) Raw outputs

LLMs aren’t perfect at generating structured output, especially as schemas become complex. You can avoid raising exceptions and handle the raw output yourself by passing includeRaw: true. This changes the output format to contain the raw message output and the parsed value (if successful):

const joke = z.object({
setup: z.string().describe("The setup of the joke"),
punchline: z.string().describe("The punchline to the joke"),
rating: z.number().optional().describe("How funny the joke is, from 1 to 10"),
});

const structuredLlm = model.withStructuredOutput(joke, {
includeRaw: true,
name: "joke",
});

await structuredLlm.invoke("Tell me a joke about cats");
{
raw: AIMessage {
lc_serializable: true,
lc_kwargs: {
content: "",
tool_calls: [
{
name: "joke",
args: [Object],
id: "call_0pEdltlfSXjq20RaBFKSQOeF"
}
],
invalid_tool_calls: [],
additional_kwargs: { function_call: undefined, tool_calls: [ [Object] ] },
response_metadata: {}
},
lc_namespace: [ "langchain_core", "messages" ],
content: "",
name: undefined,
additional_kwargs: {
function_call: undefined,
tool_calls: [
{
id: "call_0pEdltlfSXjq20RaBFKSQOeF",
type: "function",
function: [Object]
}
]
},
response_metadata: {
tokenUsage: { completionTokens: 33, promptTokens: 88, totalTokens: 121 },
finish_reason: "stop"
},
tool_calls: [
{
name: "joke",
args: {
setup: "Why was the cat sitting on the computer?",
punchline: "Because it wanted to keep an eye on the mouse!",
rating: 7
},
id: "call_0pEdltlfSXjq20RaBFKSQOeF"
}
],
invalid_tool_calls: [],
usage_metadata: { input_tokens: 88, output_tokens: 33, total_tokens: 121 }
},
parsed: {
setup: "Why was the cat sitting on the computer?",
punchline: "Because it wanted to keep an eye on the mouse!",
rating: 7
}
}

Prompting techniques

You can also prompt models to outputting information in a given format. This approach relies on designing good prompts and then parsing the output of the models. This is the only option for models that don’t support .with_structured_output() or other built-in approaches.

Using JsonOutputParser

The following example uses the built-in JsonOutputParser to parse the output of a chat model prompted to match a the given JSON schema. Note that we are adding format_instructions directly to the prompt from a method on the parser:

import { JsonOutputParser } from "@langchain/core/output_parsers";
import { ChatPromptTemplate } from "@langchain/core/prompts";

type Person = {
name: string;
height_in_meters: number;
};

type People = {
people: Person[];
};

const formatInstructions = `Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}

Where people is an array of objects, each with a name and height_in_meters field.
`;

// Set up a parser
const parser = new JsonOutputParser<People>();

// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
[
"system",
"Answer the user query. Wrap the output in `json` tags\n{format_instructions}",
],
["human", "{query}"],
]).partial({
format_instructions: formatInstructions,
});

Let’s take a look at what information is sent to the model:

const query = "Anna is 23 years old and she is 6 feet tall";

console.log((await prompt.format({ query })).toString());
System: Answer the user query. Wrap the output in `json` tags
Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}

Where people is an array of objects, each with a name and height_in_meters field.

Human: Anna is 23 years old and she is 6 feet tall

And now let’s invoke it:

const chain = prompt.pipe(model).pipe(parser);

await chain.invoke({ query });
{ people: [ { name: "Anna", height_in_meters: 1.83 } ] }

For a deeper dive into using output parsers with prompting techniques for structured output, see this guide.

Custom Parsing

You can also create a custom prompt and parser with LangChain Expression Language (LCEL), using a plain function to parse the output from the model:

import { AIMessage } from "@langchain/core/messages";
import { ChatPromptTemplate } from "@langchain/core/prompts";

type Person = {
name: string;
height_in_meters: number;
};

type People = {
people: Person[];
};

const schema = `{{ people: [{{ name: "string", height_in_meters: "number" }}] }}`;

// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
[
"system",
`Answer the user query. Output your answer as JSON that
matches the given schema: \`\`\`json\n{schema}\n\`\`\`.
Make sure to wrap the answer in \`\`\`json and \`\`\` tags`,
],
["human", "{query}"],
]).partial({
schema,
});

/**
* Custom extractor
*
* Extracts JSON content from a string where
* JSON is embedded between ```json and ``` tags.
*/
const extractJson = (output: AIMessage): Array<People> => {
const text = output.content as string;
// Define the regular expression pattern to match JSON blocks
const pattern = /```json(.*?)```/gs;

// Find all non-overlapping matches of the pattern in the string
const matches = text.match(pattern);

// Process each match, attempting to parse it as JSON
try {
return (
matches?.map((match) => {
// Remove the markdown code block syntax to isolate the JSON string
const jsonStr = match.replace(/```json|```/g, "").trim();
return JSON.parse(jsonStr);
}) ?? []
);
} catch (error) {
throw new Error(`Failed to parse: ${output}`);
}
};

Here is the prompt sent to the model:

const query = "Anna is 23 years old and she is 6 feet tall";

console.log((await prompt.format({ query })).toString());
System: Answer the user query. Output your answer as JSON that
matches the given schema: ```json
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}
```.
Make sure to wrap the answer in ```json and ``` tags
Human: Anna is 23 years old and she is 6 feet tall

And here’s what it looks like when we invoke it:

import { RunnableLambda } from "@langchain/core/runnables";

const chain = prompt
.pipe(model)
.pipe(new RunnableLambda({ func: extractJson }));

await chain.invoke({ query });
[
{ people: [ { name: "Anna", height_in_meters: 1.83 } ] }
]

Next steps

Now you’ve learned a few methods to make a model output structured data.

To learn more, check out the other how-to guides in this section, or the conceptual guide on tool calling.


Was this page helpful?


You can also leave detailed feedback on GitHub.