How to return structured data from a model
It is often useful to have a model return output that matches some specific schema. One common use-case is extracting data from arbitrary text to insert into a traditional database or use with some other downstrem system. This guide will show you a few different strategies you can use to do this.
This guide assumes familiarity with the following concepts:
The .withStructuredOutput()
method
There are several strategies that models can use under the hood. For
some of the most popular model providers, including
Anthropic, Google
VertexAI,
Mistral, and
OpenAI LangChain implements a
common interface that abstracts away these strategies called
.withStructuredOutput
.
By invoking this method (and passing in JSON schema or a Zod schema) the model will add whatever model parameters + output parsers are necessary to get back structured output matching the requested schema. If the model supports more than one way to do this (e.g., function calling vs JSON mode) - you can configure which method to use by passing into that method.
Let’s look at some examples of this in action! We’ll use Zod to create a simple response schema.
Pick your chat model:
- OpenAI
- Anthropic
- MistralAI
- Groq
- VertexAI
Install dependencies
- npm
- yarn
- pnpm
npm i @langchain/openai
yarn add @langchain/openai
pnpm add @langchain/openai
Add environment variables
OPENAI_API_KEY=your-api-key
Instantiate the model
import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({
model: "gpt-4o-mini",
temperature: 0
});
Install dependencies
- npm
- yarn
- pnpm
npm i @langchain/anthropic
yarn add @langchain/anthropic
pnpm add @langchain/anthropic
Add environment variables
ANTHROPIC_API_KEY=your-api-key
Instantiate the model
import { ChatAnthropic } from "@langchain/anthropic";
const model = new ChatAnthropic({
model: "claude-3-5-sonnet-20240620",
temperature: 0
});
Install dependencies
- npm
- yarn
- pnpm
npm i @langchain/mistralai
yarn add @langchain/mistralai
pnpm add @langchain/mistralai
Add environment variables
MISTRAL_API_KEY=your-api-key
Instantiate the model
import { ChatMistralAI } from "@langchain/mistralai";
const model = new ChatMistralAI({
model: "mistral-large-latest",
temperature: 0
});
Install dependencies
- npm
- yarn
- pnpm
npm i @langchain/groq
yarn add @langchain/groq
pnpm add @langchain/groq
Add environment variables
GROQ_API_KEY=your-api-key
Instantiate the model
import { ChatGroq } from "@langchain/groq";
const model = new ChatGroq({
model: "mixtral-8x7b-32768",
temperature: 0
});
Install dependencies
- npm
- yarn
- pnpm
npm i @langchain/google-vertexai
yarn add @langchain/google-vertexai
pnpm add @langchain/google-vertexai
Add environment variables
GOOGLE_APPLICATION_CREDENTIALS=credentials.json
Instantiate the model
import { ChatVertexAI } from "@langchain/google-vertexai";
const model = new ChatVertexAI({
model: "gemini-1.5-flash",
temperature: 0
});
import { z } from "zod";
const joke = z.object({
setup: z.string().describe("The setup of the joke"),
punchline: z.string().describe("The punchline to the joke"),
rating: z.number().optional().describe("How funny the joke is, from 1 to 10"),
});
const structuredLlm = model.withStructuredOutput(joke);
await structuredLlm.invoke("Tell me a joke about cats");
{
setup: "Why don't cats play poker in the wild?",
punchline: "Too many cheetahs.",
rating: 7
}
One key point is that though we set our Zod schema as a variable named
joke
, Zod is not able to access that variable name, and therefore
cannot pass it to the model. Though it is not required, we can pass a
name for our schema in order to give the model additional context as to
what our schema represents, improving performance:
const structuredLlm = model.withStructuredOutput(joke, { name: "joke" });
await structuredLlm.invoke("Tell me a joke about cats");
{
setup: "Why don't cats play poker in the wild?",
punchline: "Too many cheetahs!",
rating: 7
}
The result is a JSON object.
We can also pass in an OpenAI-style JSON schema dict if you prefer not to use Zod. This object should contain three properties:
name
: The name of the schema to output.description
: A high level description of the schema to output.parameters
: The nested details of the schema you want to extract, formatted as a JSON schema dict.
In this case, the response is also a dict:
const structuredLlm = model.withStructuredOutput({
name: "joke",
description: "Joke to tell user.",
parameters: {
title: "Joke",
type: "object",
properties: {
setup: { type: "string", description: "The setup for the joke" },
punchline: { type: "string", description: "The joke's punchline" },
},
required: ["setup", "punchline"],
},
});
await structuredLlm.invoke("Tell me a joke about cats", { name: "joke" });
{
setup: "Why was the cat sitting on the computer?",
punchline: "Because it wanted to keep an eye on the mouse!"
}
If you are using JSON Schema, you can take advantage of other more complex schema descriptions to create a similar effect.
You can also use tool calling directly to allow the model to choose between options, if your chosen model supports it. This involves a bit more parsing and setup. See this how-to guide for more details.
Specifying the output method (Advanced)
For models that support more than one means of outputting data, you can specify the preferred one like this:
const structuredLlm = model.withStructuredOutput(joke, {
method: "json_mode",
name: "joke",
});
await structuredLlm.invoke(
"Tell me a joke about cats, respond in JSON with `setup` and `punchline` keys"
);
{
setup: "Why don't cats play poker in the jungle?",
punchline: "Too many cheetahs!"
}
In the above example, we use OpenAI’s alternate JSON mode capability along with a more specific prompt.
For specifics about the model you choose, peruse its entry in the API reference pages.
(Advanced) Raw outputs
LLMs aren’t perfect at generating structured output, especially as
schemas become complex. You can avoid raising exceptions and handle the
raw output yourself by passing includeRaw: true
. This changes the
output format to contain the raw message output and the parsed
value
(if successful):
const joke = z.object({
setup: z.string().describe("The setup of the joke"),
punchline: z.string().describe("The punchline to the joke"),
rating: z.number().optional().describe("How funny the joke is, from 1 to 10"),
});
const structuredLlm = model.withStructuredOutput(joke, {
includeRaw: true,
name: "joke",
});
await structuredLlm.invoke("Tell me a joke about cats");
{
raw: AIMessage {
lc_serializable: true,
lc_kwargs: {
content: "",
tool_calls: [
{
name: "joke",
args: [Object],
id: "call_0pEdltlfSXjq20RaBFKSQOeF"
}
],
invalid_tool_calls: [],
additional_kwargs: { function_call: undefined, tool_calls: [ [Object] ] },
response_metadata: {}
},
lc_namespace: [ "langchain_core", "messages" ],
content: "",
name: undefined,
additional_kwargs: {
function_call: undefined,
tool_calls: [
{
id: "call_0pEdltlfSXjq20RaBFKSQOeF",
type: "function",
function: [Object]
}
]
},
response_metadata: {
tokenUsage: { completionTokens: 33, promptTokens: 88, totalTokens: 121 },
finish_reason: "stop"
},
tool_calls: [
{
name: "joke",
args: {
setup: "Why was the cat sitting on the computer?",
punchline: "Because it wanted to keep an eye on the mouse!",
rating: 7
},
id: "call_0pEdltlfSXjq20RaBFKSQOeF"
}
],
invalid_tool_calls: [],
usage_metadata: { input_tokens: 88, output_tokens: 33, total_tokens: 121 }
},
parsed: {
setup: "Why was the cat sitting on the computer?",
punchline: "Because it wanted to keep an eye on the mouse!",
rating: 7
}
}
Prompting techniques
You can also prompt models to outputting information in a given format.
This approach relies on designing good prompts and then parsing the
output of the models. This is the only option for models that don’t
support .with_structured_output()
or other built-in approaches.
Using JsonOutputParser
The following example uses the built-in
JsonOutputParser
to parse the output of a chat model prompted to match a the given JSON
schema. Note that we are adding format_instructions
directly to the
prompt from a method on the parser:
import { JsonOutputParser } from "@langchain/core/output_parsers";
import { ChatPromptTemplate } from "@langchain/core/prompts";
type Person = {
name: string;
height_in_meters: number;
};
type People = {
people: Person[];
};
const formatInstructions = `Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}
Where people is an array of objects, each with a name and height_in_meters field.
`;
// Set up a parser
const parser = new JsonOutputParser<People>();
// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
[
"system",
"Answer the user query. Wrap the output in `json` tags\n{format_instructions}",
],
["human", "{query}"],
]).partial({
format_instructions: formatInstructions,
});
Let’s take a look at what information is sent to the model:
const query = "Anna is 23 years old and she is 6 feet tall";
console.log((await prompt.format({ query })).toString());
System: Answer the user query. Wrap the output in `json` tags
Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}
Where people is an array of objects, each with a name and height_in_meters field.
Human: Anna is 23 years old and she is 6 feet tall
And now let’s invoke it:
const chain = prompt.pipe(model).pipe(parser);
await chain.invoke({ query });
{ people: [ { name: "Anna", height_in_meters: 1.83 } ] }
For a deeper dive into using output parsers with prompting techniques for structured output, see this guide.
Custom Parsing
You can also create a custom prompt and parser with LangChain Expression Language (LCEL), using a plain function to parse the output from the model:
import { AIMessage } from "@langchain/core/messages";
import { ChatPromptTemplate } from "@langchain/core/prompts";
type Person = {
name: string;
height_in_meters: number;
};
type People = {
people: Person[];
};
const schema = `{{ people: [{{ name: "string", height_in_meters: "number" }}] }}`;
// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
[
"system",
`Answer the user query. Output your answer as JSON that
matches the given schema: \`\`\`json\n{schema}\n\`\`\`.
Make sure to wrap the answer in \`\`\`json and \`\`\` tags`,
],
["human", "{query}"],
]).partial({
schema,
});
/**
* Custom extractor
*
* Extracts JSON content from a string where
* JSON is embedded between ```json and ``` tags.
*/
const extractJson = (output: AIMessage): Array<People> => {
const text = output.content as string;
// Define the regular expression pattern to match JSON blocks
const pattern = /```json(.*?)```/gs;
// Find all non-overlapping matches of the pattern in the string
const matches = text.match(pattern);
// Process each match, attempting to parse it as JSON
try {
return (
matches?.map((match) => {
// Remove the markdown code block syntax to isolate the JSON string
const jsonStr = match.replace(/```json|```/g, "").trim();
return JSON.parse(jsonStr);
}) ?? []
);
} catch (error) {
throw new Error(`Failed to parse: ${output}`);
}
};
Here is the prompt sent to the model:
const query = "Anna is 23 years old and she is 6 feet tall";
console.log((await prompt.format({ query })).toString());
System: Answer the user query. Output your answer as JSON that
matches the given schema: ```json
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}
```.
Make sure to wrap the answer in ```json and ``` tags
Human: Anna is 23 years old and she is 6 feet tall
And here’s what it looks like when we invoke it:
import { RunnableLambda } from "@langchain/core/runnables";
const chain = prompt
.pipe(model)
.pipe(new RunnableLambda({ func: extractJson }));
await chain.invoke({ query });
[
{ people: [ { name: "Anna", height_in_meters: 1.83 } ] }
]
Next steps
Now you’ve learned a few methods to make a model output structured data.
To learn more, check out the other how-to guides in this section, or the conceptual guide on tool calling.