Building AI Agents From Scratch
If you have never built an AI agent before, the concept might feel intimidating. With complex framework abstractions, jargon, and multi-layered libraries floating around, it is easy to think that agentic AI requires thousands of lines of sophisticated software infrastructure.
The truth is much simpler: building a fully functional AI agent is straightforward once you understand the core concept. Stripped of all hype, an AI agent is just a language model wrapped in a stateful loop that knows how to call functions.
In this tutorial, you will build a complete AI agent from scratch in plain Python using the OpenAI SDK. Step-by-step, you will give an LLM memory, teach it to use custom tools, and construct an execution loop that solves multi-step goals.
The Four Primitives of an AI Agent
We will build an Agent class around four core primitives:
- System Instructions: The prompt that defines the model’s persona, goal, and operating rules.
- Memory: A list of messages that preserves context across multiple turns.
- Tools: Python functions paired with JSON schemas so the model knows how to request external actions.
- Execution Loop: A control loop that receives user goals, runs requested tools, and feeds results back to the LLM until the task is complete.
graph TD
A[1. System Instructions & User Goal] --> B[2. Memory]
B --> C[3. LLM Reasoning]
C --> D{Tool Call Requested?}
D -- Yes --> E[4. Tool Execution]
E -->|Feed Result Back| B
D -- No --> F[Final Output to User]
Figure 1: The four core primitives of an agentic execution loop.
Before we get started, you need an OpenAI API Key (paid) or other compatible OpenAI API Key. To help get started for free, you can get it from OpenRouter. You can use any free model available on OpenRouter (look for models tagged with :free on OpenRouter Models, such as nvidia/nemotron-3-nano-30b-a3b:free, meta-llama/llama-3.3-70b-instruct:free, etc.) or any standard OpenAI model if using OpenAI directly. Set these variables in your .env file:
OPENAI_API_KEY=<your-api-key>
OPENAI_BASE_URL=https://openrouter.ai/api/v1
OPENAI_MODEL=nvidia/nemotron-3-nano-30b-a3b:free1. LLM API Calls Are Stateless by Default
When you call an LLM API, the model does not remember your previous messages. Every HTTP request to the LLM starts completely fresh.
Let’s set up your environment and see this stateless behavior in action:
uv add openai python-dotenvHere is a baseline Agent class that sends each prompt in isolation:
import os
import openai
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
class Agent:
"""Simple AI Agent that can answer questions."""
def __init__(self):
self.client = OpenAI(
api_key=os.getenv("OPENAI_API_KEY"),
base_url=os.getenv("OPENAI_BASE_URL")
)
self.model = os.getenv("OPENAI_MODEL", "nvidia/nemotron-3-nano-30b-a3b:free")
self.system_message = (
"You are a helpful assistant"
)
def chat(self, message: str) -> str:
"""Receive a user message and return the LLM's response."""
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": self.system_message},
{"role": "user", "content": message}
],
temperature=0.1,
max_tokens=1024
)
return response.choices[0].message.contentIf we ask two related questions back-to-back, the problem immediately becomes clear:
agent = Agent()
print(agent.chat("I have 4 bananas. How many do you have?"))
print('---')
print(agent.chat("I ate 1 banana. How many are left?"))I don't have any bananas. I am an AI, so I don't have physical items or a fruit bowl. But thanks for asking!
---
It depends on how many bananas you had to start with! If you had just 1, there are 0 left. If you had more, how many did you have originally?Because the second API request had no record of the first turn, the model had no idea we started with 4 bananas.
2. Giving the Agent Memory
To solve context loss, the agent needs memory. For short term memory, this is simply a Python list storing every user message and assistant reply. On every new turn, we pass the entire history list to the API.
graph LR
subgraph Isolated Prompts
U1[User Prompt 1] --> M1[LLM]
U2[User Prompt 2] --> M2[LLM]
end
subgraph Stateful Memory
U3[User Prompt 1] --> S[Memory Array]
A1[Assistant Reply 1] --> S
U4[User Prompt 2] --> S
S --> M3[LLM with Full History]
end
Figure 2: Isolated API calls vs. stateful conversation history.
Let’s update the Agent class to retain conversation history:
class Agent:
"""An AI agent with short-term memory for multi-turn conversations."""
def __init__(self):
self.client = OpenAI(
api_key=os.getenv("OPENAI_API_KEY"),
base_url=os.getenv("OPENAI_BASE_URL")
)
self.model = os.getenv("OPENAI_MODEL", "nvidia/nemotron-3-nano-30b-a3b:free")
self.system_message = (
"You are a helpful assistant"
)
self.messages = [] # Short-term memory
def chat(self, message: str) -> str:
"""Process a user message while maintaining conversation history."""
self.messages.append({"role": "user", "content": message})
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": self.system_message},
*self.messages
],
temperature=0.1,
max_tokens=1024
)
assistant_message = response.choices[0].message.content
self.messages.append({"role": "assistant", "content": assistant_message})
return assistant_messageRe-running our multi-turn test now works as expected:
agent = Agent()
print(agent.chat("I have 4 bananas. How many do you have?"))
print('---')
print(agent.chat("I ate 1 banana. How many are left?"))I don't have any bananas. I am an AI, so I don't have physical possessions. But you've got 4!
---
You started with **4 bananas**, and you ate **1**.
So: **4 - 1 = 3**.
You have **3 bananas left**!
*(Simple math, but I love the banana math!)*But what if we ask for some realtime data?
print(agent.chat("What is the weather today in Jakarta?"))I’m not able to access real‑time data, so I can’t give you the exact weather conditions in Jakarta right now.
For the most up‑to‑date forecast, you can check a weather service such as:
- **Weather.com**: just type "Jakarta, Indonesia" into the search bar.
- **AccuWeather**: provides hourly and daily forecasts.
- **Google**: type "Jakarta weather" and the current conditions will appear at the top of the results.
- **Local Indonesian sites** like **BMKG** (the Indonesian Meteorology, Climatology, and Geophysics Agency) for official forecasts.
If you need help interpreting the forecast or planning activities around typical Jakarta weather patterns (e.g., rainy season vs. dry season), just let me know!Memory solves conversation flow, but the model is still limited to what it learned during training. If a user asks for real-time information, such as current weather, the model needs tools.
3. Defining Tools with Function Schemas
An LLM cannot directly run Python code on your computer. Instead, function calling works through a contract:
- JSON Schema: You send a JSON dictionary describing your tool’s name, purpose, and parameter types to the LLM.
- LLM Choice: If the LLM decides it needs the tool, it responds with a structured JSON payload containing the function name and arguments.
- Python Execution: Your Python code reads that payload, runs the actual Python function, and sends the output back to the model.
Here is a simple mock weather tool that implements this pattern:
class WeatherTool:
"""A tool for retrieving current weather metrics by city name."""
def get_schema(self) -> dict:
"""Returns the JSON schema describing this tool to LLM."""
return {
"type": "function",
"function": {
"name": "check_weather",
"description": (
"Retrieves current weather details for a given city. "
"Use this tool whenever the user asks about weather, temperature, "
"or conditions in a specific city."
),
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The target city name, e.g. 'Jakarta', 'Bandung', 'Surabaya'."
}
},
"required": ["city"]
}
}
}
def execute(self, city: str) -> dict:
"""Mock backend execution returning structured weather data."""
weather_db = {
"jakarta": {"city": "Jakarta", "temperature": 32, "condition": "Partly cloudy", "humidity": 70},
"bandung": {"city": "Bandung", "temperature": 24, "condition": "Cloudy", "humidity": 85},
"surabaya": {"city": "Surabaya", "temperature": 34, "condition": "Sunny", "humidity": 65},
"yogyakarta": {"city": "Yogyakarta", "temperature": 28, "condition": "Light rain", "humidity": 80}
}
city_key = city.lower().strip()
if city_key in weather_db:
return weather_db[city_key]
return {
"error": f"Weather data for '{city}' is unavailable. Available cities: Jakarta, Bandung, Surabaya, Yogyakarta."
}4. Building the Core Agent Execution Loop
Now we combine memory, tools, and a control loop into a single working agent.
The agent needs to handle two scenarios when calling OpenAI:
- Scenario A: The LLM returns a normal text answer -> We stop and show the user.
- Scenario B: The LLM requests a tool call -> We execute the Python function, format the result into a
role: "tool"message, append it to memory, and ask the LLM again.
Here is the complete Agent class:
import os
import json
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
class Agent:
"""An AI agent capable of handling tool calls and stateful memory."""
def __init__(self, tools=None):
self.client = OpenAI(
api_key=os.getenv("OPENAI_API_KEY"),
base_url=os.getenv("OPENAI_BASE_URL"),
)
self.model = os.getenv("OPENAI_MODEL", "nvidia/nemotron-3-nano-30b-a3b:free")
self.system_message = (
"You are a helpful and friendly assistant. "
"If the user asks about the weather in a city, ALWAYS use the 'check_weather' tool. "
"Never guess or make up weather data."
)
self.messages = []
self.tools = tools or []
self.tool_map = {
tool.get_schema()["function"]["name"]: tool
for tool in self.tools
}
def _get_tool_schemas(self):
return [tool.get_schema() for tool in self.tools]
def step(self, new_input):
"""Process new input (user prompt string or list of tool results) and call the LLM."""
if isinstance(new_input, str):
self.messages.append({"role": "user", "content": new_input})
elif isinstance(new_input, list):
# Append tool execution result messages directly to memory
self.messages.extend(new_input)
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": self.system_message},
*self.messages
],
tools=self._get_tool_schemas() if self.tools else None,
tool_choice="auto" if self.tools else None,
temperature=0.1,
max_tokens=1024
)
assistant_message = response.choices[0].message
# Store the assistant response cleanly in conversation history
assistant_dict = {
"role": "assistant",
"content": assistant_message.content or ""
}
if assistant_message.tool_calls:
assistant_dict["tool_calls"] = [
{
"id": tc.id,
"type": "function",
"function": {
"name": tc.function.name,
"arguments": tc.function.arguments
}
}
for tc in assistant_message.tool_calls
]
self.messages.append(assistant_dict)
return assistant_messageNext, let’s write run_agent, which drives the execution loop until the task is complete (bounded by max_turns to prevent infinite loops):
def run_agent(user_input: str, tools: list, max_turns: int = 10) -> str:
"""Execute the agent loop until a final text output is returned."""
agent = Agent(tools=tools)
current_input = user_input
for i in range(max_turns):
print(f"\n{'='*50}\nIteration {i + 1}\n{'='*50}")
print(f"Input: {current_input}")
response = agent.step(current_input)
# Check if the LLM requested tool execution
if response.tool_calls:
print(f"Agent decision: {response.content or '(requesting tool execution)'}")
tool_results = []
for tool_call in response.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
print(f"Calling Tool: {tool_name}")
print(f" Arguments: {tool_args}")
# Execute Python function locally with safe lookup
tool = agent.tool_map.get(tool_name)
if not tool:
result = {"error": f"Tool '{tool_name}' not found."}
else:
result = tool.execute(**tool_args)
print(f"Tool Output: {result}")
# Format tool output for OpenAI API specs
tool_results.append({
"role": "tool",
"tool_call_id": tool_call.id,
"name": tool_name,
"content": json.dumps(result)
})
current_input = tool_results
else:
print(f"Final Output: {response.content}")
return response.content
return "Reached maximum iteration limit."5. Testing the Agent: Handling Multi-Step Workflows
Let’s test our agent with a prompt that requires two distinct tool calls to answer:
weather_tool = WeatherTool()
run_agent(
"I'm planning to visit Bandung tomorrow. What's the weather like? Is it cooler than Surabaya?",
tools=[weather_tool]
)Here is what happens step-by-step during execution:
==================================================
Iteration 1
==================================================
Input: I'm planning to visit Bandung tomorrow. What's the weather like? Is it cooler than Surabaya?
Agent decision: (requesting tool execution)
Calling Tool: check_weather
Arguments: {'city': 'Bandung'}
Tool Output: {'city': 'Bandung', 'temperature': 24, 'condition': 'Cloudy', 'humidity': 85}
==================================================
Iteration 2
==================================================
Input: [{'role': 'tool', 'tool_call_id': 'call_ghi789', 'name': 'check_weather', 'content': '{"city": "Bandung", "temperature": 24, "condition": "Cloudy", "humidity": 85}'}]
Agent decision: (requesting tool execution)
Calling Tool: check_weather
Arguments: {'city': 'Surabaya'}
Tool Output: {'city': 'Surabaya', 'temperature': 34, 'condition': 'Sunny', 'humidity': 65}
==================================================
Iteration 3
==================================================
Input: [{'role': 'tool', 'tool_call_id': 'call_jkl012', 'name': 'check_weather', 'content': '{"city": "Surabaya", "temperature": 34, "condition": "Sunny", "humidity": 65}'}]
Final Output:
Here's the weather comparison:
- Bandung: Cloudy, 24°C, 85% humidity
- Surabaya: Sunny, 34°C, 65% humidity
Bandung is significantly cooler than Surabaya by 10°C.sequenceDiagram
participant User
participant Agent as Agent Loop
participant LLM as LLM API
participant Tool as WeatherTool
User->>Agent: Compare weather in Bandung & Surabaya
Agent->>LLM: Send prompt + tool schemas
LLM-->>Agent: Request tool check_weather(city='Bandung')
Agent->>Tool: execute(city='Bandung')
Tool-->>Agent: Returns 24°C, Cloudy
Agent->>LLM: Send tool result + history
LLM-->>Agent: Request tool check_weather(city='Surabaya')
Agent->>Tool: execute(city='Surabaya')
Tool-->>Agent: Returns 34°C, Sunny
Agent->>LLM: Send tool result + history
LLM-->>Agent: Returns final text response
Agent-->>User: Bandung (24°C) is 10°C cooler than Surabaya (34°C)
Figure 3: Sequence diagram showing how memory and tool execution enable multi-step reasoning.
Notice how the agent autonomously broke down the user prompt:
- It realized it needed Bandung’s weather first, so it called
check_weather(city='Bandung'). - After seeing Bandung’s result, it realized it still needed Surabaya’s weather, so it called
check_weather(city='Surabaya'). - With both facts stored in memory, it synthesized the final comparison response.
Note on Parallel Tool Calling: Depending on the model and settings, modern LLMs often emit multiple tool calls in a single response turn (parallel tool calling) when requests can be queried at the same time. The execution loop’s
for tool_call in response.tool_calls:loop is already structured to handle both single and parallel tool calls seamlessly.
What Agent Frameworks Cover (And Why Fundamentals Matter)
Now that you’ve built an agent from scratch, you have unlocked the core mental model that powers every modern AI agent system.
When you look at popular frameworks like LangChain, LlamaIndex, CrewAI, or AutoGen, you will realize they aren’t doing magic under the hood. They are simply building production abstraction layers on top of the exact primitives you just wrote.
What Frameworks Provide
- Unified Model Interfaces: Standardized abstractions across different LLM providers (OpenAI, Anthropic, Gemini, Ollama, etc.). Since each provider has different SDK syntax and payload structures, frameworks allow you to swap models without rewriting your tool schemas or message loop.
- Pre-Built Integrations: Instant connections to search engines, databases, file parsers, and external APIs.
- Advanced Memory Management: Automatic sliding windows, vector memory (RAG), and persistent database storage rather than a raw Python list.
- Multi-Agent Orchestration: Standardized patterns for handing off tasks between multiple specialized agents.
- Observability & Telemetry: Out-of-the-box tracking for latency, token counts, and step-by-step tool traces.
Why Understanding the Fundamentals Comes First
Frameworks speed up development, but understanding the core primitives is essential:
- Debugging with Confidence: When a framework agent gets stuck, hallucinates tool arguments, or fails silently, you won’t be trapped behind black-box abstractions. You’ll know how to inspect the message array, fix the JSON schema, or adjust the control loop.
- Avoiding Over-Engineering: For many tasks, like a simple CLI automation or internal API tool, a ~100 line custom agent loop is faster, cleaner, and far easier to maintain than pulling in heavy framework dependencies.
Conclusion
Building an AI agent is surprisingly simple once you grasp the underlying concept:
- Store conversation history in a stateful message list.
- Expose Python functions using JSON schemas.
- Drive interaction using an execution loop that feeds tool results back to the LLM until the goal is solved.
By building this foundation yourself, you can now evaluate high-level frameworks with complete clarity, or write your own lightweight agent loops whenever you need maximum control.
Resources & Further Reading
- OpenAI Function Calling Guide: Official specifications for tool schemas and structured outputs.
- Anthropic: Building Effective Agents: In-depth guide on workflows, agentic primitives, and evaluation loops.