How to Deploy MultiAgent workflow using CrewAI on TrueFoundry

In this guide, we’ll show you how to deploy a CrewAI agent on TrueFoundry, a platform designed to simplify AI deployment with minimal DevOps or MLOps expertise. TrueFoundry automates infrastructure management, scaling, and monitoring, allowing you to focus on deriving insights rather than handling deployment complexities. With just a few clicks, you can transform natural language requests into SQL queries and dynamic charts, making data exploration seamless and intelligent. No manual querying required!

If you would like to try this out directly, please visit the TrueFoundry platform and navigate to Live Demos and CrewAI-Streamlit: Live demo of our agent workflow

Architecture Overview

This project consists of several key components working together:

Query Agent

• Uses GPT-4o for natural language understanding
• Generates appropriate SQL queries for ClickHouse
• Executes the SQL query against a pre-configured database
• Returns the data in tabular format as input for the visualization agent

Visualization Agent: A second AI agent that

• Determines the most appropriate visualization type given the data
• Generates plots using matplotlib/seaborn
• Handles formatting and styling of visualizations

FastAPI Backend: RESTful API that

• Coordinates between agents using CrewAI
• Manages asynchronous job processing
• Serves plot images and results

Streamlit Frontend: User interface that

• Provides an intuitive query interface
• Displays real-time processing status
• Shows interactive visualizations

Data Flow

User submits a natural language query through Streamlit.

• Query Agent uses CrewAI with GPT-4o to generate SQL queries for ClickHouse
• Executes the SQL query against ClickHouse database
• Results are returned in tabular format as input for the Visualization Agent
• Visualization Agent generates visualizations and returns images for display

Getting Started

Clone the Repository

First, navigate to the TrueFoundry Getting Started Examples repository and clone it:

git clone <https://github.com/truefoundry/getting-started-examples.git>

Navigate to the CrewAI Plot Agent Directory:

cd getting-started-examples/plot_agent/crewai_plot_agent

Environment Setup

Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies:

pip install uv
uv pip install -r requirements.txt

Configure Environment Variables

Create a .env file:

# Truefoundry LLMGateway Configuration if using Truefoundry LLM Gateway for calling models
LLM_GATEWAY_BASE_URL=your_llm_gateway_base_url_here
LLM_GATEWAY_API_KEY=your_llm_gateway_api_key_here

# OPENAI API Configuration if not using Truefoundry LLM Gateway
OPENAI_API_KEY=<your_openai_api_key_here>
CLICKHOUSE_HOST=your_clickhouse_host
CLICKHOUSE_PORT=443
CLICKHOUSE_USER=your_user
CLICKHOUSE_PASSWORD=your_password
CLICKHOUSE_DATABASE=default
CREWAI_VERBOSE=true

Note: When using the TrueFoundry LLM Gateway, the model ID format should be provider-name/model-name (e.g., openai-main/gpt-4o). Make sure your .env file contains the correct LLM Gateway credentials as shown in the Environment Configuration section.

To get clickhouse credentials, create an account on clickhouse, sign in and create a service. After clicking the service you will see a connect button on the middle of left sidebar, which you can click to see the credentials as shown below. You can either create a database by uploading your files or use a predefined one.

CrewAI Agent implementation

@CrewBase
class CrewaiPlotAgent():
	"""CrewaiPlotAgent crew"""

	# Learn more about YAML configuration files here:
	# Agents: https://docs.crewai.com/concepts/agents#yaml-configuration-recommended
	# Tasks: https://docs.crewai.com/concepts/tasks#yaml-configuration-recommended
	agents_config = 'config/agents.yaml'
	tasks_config = 'config/tasks.yaml'

	# def process_output(self, output):
	# 	# Modify output after the crew finishes
	# 	output.raw += "\nProcessed after kickoff."
	# 	print("Output", output ,"sdfsdfsdfsd")
	# 	return output

	# If you would like to add tools to your agents, you can learn more about it here:
	# https://docs.crewai.com/concepts/agents#agent-tools
	@agent
	def sql_writer(self) -> Agent:
		return Agent(
			config=self.agents_config['sql_writer'],
			verbose=True,
			tools=[ClickHouseTool()]
		)

	@agent
	def plot_writer(self) -> Agent:
		return Agent(
			config=self.agents_config['plot_writer'],
			verbose=True,
			tools=[PlotTools()],
			pydantic_output=PlotResult,
		)

	# To learn more about structured task outputs, 
	# task dependencies, and task callbacks, check out the documentation:
	# https://docs.crewai.com/concepts/tasks#overview-of-a-task
	@task
	def sql_task(self) -> Task:
		return Task(
			config=self.tasks_config['sql_task'],
		)

	@task
	def plot_task(self) -> Task:
		return Task(
			config=self.tasks_config['plot_task'],
			# callback=self.process_output,
			allow_code_execution=True,
			output_pydantic=PlotResult,
			# output_file='plot.png'
		)

	@crew
	def crew(self) -> Crew:
		"""Creates the CrewaiPlotAgent crew"""
		# To learn how to add knowledge sources to your crew, check out the documentation:
		# https://docs.crewai.com/concepts/knowledge#what-is-knowledge

		return Crew(
			agents=self.agents, # Automatically created by the @agent decorator
			tasks=self.tasks, # Automatically created by the @task decorator
			process=Process.sequential,
			verbose=True,
			# output_pydantic=True,
			# process=Process.hierarchical, # In case you wanna use that instead https://docs.crewai.com/how-to/Hierarchical/
		)
	

The decorators @Crewbase, @agent, @task etc. are for enabling tracing in truefoundry mentioned in detail later.

Running the Services

Start CrewAI Workflow

crewai run

Start FastAPI Backend:

python api.py

Start Streamlit UI (new terminal):

streamlit run app.py

Deployment on TrueFoundry

Prerequisites

Install TrueFoundry CLI:

pip install -U "truefoundry"

Login to TrueFoundry:

tfy login --host "<https://app.truefoundry.com>"

Deployment Steps

1. Navigate to Deployments section in TrueFoundry.

2. Click Service at the bottom.
3. Select your cluster workspace.
4. You can deploy from your laptop, GitHub, or Docker. If deploying from your laptop, ensure you have completed the prerequisites above.
5. The TrueFoundry platform will generate a deploy.py file and add it to your project. You’ll need to edit this file to add your environment variables. Find the env section in the generated file and add your credentials:
6. Use the generated deploy.py and edit the env section:

env={
    "OPENAI_API_KEY": "your_openai_api_key",
    "CLICKHOUSE_HOST": "your_clickhouse_host",
    "CLICKHOUSE_PORT": "443",
    "CLICKHOUSE_USER": "your_user",
    "CLICKHOUSE_PASSWORD": "your_password",
    "CLICKHOUSE_DATABASE": "default",
    "CREWAI_VERBOSE": "true"
},

Replace placeholders with your credentials and environment configurations.

Testing Deployment

Send a test query:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "Show me the cost trends by model over the last week"}' \
  <https://crewai-plot-agent-demo-8000.aws.demo.truefoundry.cloud/query>

Successful response example:

{
  "job_id": "1234-abcd-5678-efgh"
}

API Endpoints

• Submit a Query:

curl -X POST <http://localhost:8000/query> -H "Content-Type: application/json" -d '{"query": "Your query here."}'

• Check Query Status:

curl -X GET <http://localhost:8000/status/{job_id}>

• Retrieve Plot Image:

curl -X GET <http://localhost:8000/plot/{job_id}> > plot.png

Frontend and CORS

Configure CORS in FastAPI:

from fastapi.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

Define Environment Variable in Streamlit:

import os

FASTAPI_ENDPOINT = os.getenv("FASTAPI_ENDPOINT", "<http://localhost:8000>")

Post-deployment Notes

• Test API connectivity from Streamlit to FastAPI.
• Update Streamlit’s .env file with the FastAPI endpoint.
• Confirm CORS settings allow Streamlit requests.

Monitor and manage your deployment through TrueFoundry by:

• Viewing logs
• Monitoring resource usage
• Setting auto-scaling rules
• Checking backend health (/health), API documentation (/docs), and metrics at /metrics

Add traces to your agent

Tracing helps you understand what’s happening under the hood when an agent run is called. You get to understand the path, tools calls made, context used, latency taken when you run your agent using Truefoundry’s tracing functionality by add very few lines of code.

You need to install the following

 pip install traceloop-sdk

And then add the necessary environment variables to enable tracing

"TRACELOOP_BASE_URL": "<your_host_name>/api/otel" # "https://internal.devtest.truefoundry.tech/api/otel"
"TRACELOOP_HEADERS"="Authorization=Bearer%20<your_tfy_api_key>"

In your codebase where you define your agent, you just need these lines to enable tracing

from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow, agent, task
Traceloop.init(app_name="crew-ai")

Then add decorator to the agent and workflows like the following

@agent(name="sql_and_plot_workflow")
@workflow(name="plotting workflow")
@task(name="execute sql query")

With these steps, your CrewAI agent workflow is now successfully deployed on TrueFoundry!

‍