How to actually make tools with output_type="image"?

[inikishev]

I tried to have my tool return PIL.Image, and I tried setting output_type = 'image' and returning file name or PIL.Image. In all cases all the agent sees is a string that states "Stored image.png in memory". An example on the docs of how to make a tool with an image output would be appreciated

[rkfshakti]

I tried to have my tool return PIL.Image, and I tried setting output_type = 'image' and returning file name or PIL.Image. In all cases all the agent sees is a string that states "Stored image.png in memory". An example on the docs of how to make a tool with an image output would be appreciated

The output_type="image" decorator tells the Gradio UI how to render the output, but the tool itself still needs to return something the agent can use. Returning a PIL.Image object won't work because the agent (an LLM) can only read text.

The correct approach is to:

Save the image to a file
Return the file path as a string
Set output_type = "image" so the UI knows to render it

from smolagents import tool

@tool
def generate_plot(data: str) -> str:
"""Generate a plot from the given data and return the image path."""
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

values = [float(x) for x in data.split(",")]
fig, ax = plt.subplots()
ax.plot(values)
ax.set_title("Generated Plot")

path = "/tmp/plot_output.png"
fig.savefig(path, dpi=100, bbox_inches="tight")
plt.close(fig)

return path

generate_plot.output_type = "image"

Key points:

The function returns a string (the file path), not a PIL.Image
output_type = "image" must be set on the tool after defining it
The path must be an absolute path the agent can access
Using matplotlib.use("Agg") prevents display backend issues in headless environments
The Gradio UI will render the image from the path
If you're using the Gradio UI (like the one in the smolagents demo), this should work out of the box. The agent will see the path string, but the UI layer handles the actual image rendering.