Promptfoo is an open-source CLI and library for evaluating LLM apps. Docker Model Runner makes it easy to manage, run, and deploy AI models using Docker. The Docker MCP Toolkit is a local gateway that lets you set up, manage, and run containerized MCP servers and connect them to AI agents.
Together, these tools let you compare models, evaluate MCP servers, and even perform LLM red-teaming from the comfort of your own dev machine. Let’s look at a few examples to see it in action.
Prerequisites
Before jumping into the examples, we’ll first need to enable Docker MCP Toolkit in Docker Desktop, enable Docker Model Runner in Docker Desktop, pull a few models with docker model, and install promptfoo.
- Enable Docker MCP Toolkit in Docker Desktop.
- Enable Docker Model Runner in Docker Desktop.
- Use the Docker Model Runner CLI to pull the following models
docker model pull ai/gemma3:4B-Q4_K_M
docker model pull ai/smollm3:Q4_K_M
docker model pull ai/mxbai-embed-large:335M-F164. Install Promptfoo
npm install -g promptfooWith the prerequisites complete, we can get into our first example.
Using Docker Model Runner and promptfoo for Prompt Comparison
Does your prompt and context require paying for tokens from an AI cloud provider or will an open source model provide 80% of the value for a fraction of the cost? How will you systematically re-assess this dilemma every month when your prompt changes, a new model drops, or token costs change? With the Docker Model Runner provider in promptfoo, it’s easy to set up a Promptfoo eval to compare a prompt across local and cloud models.
In this example, we’ll compare & grade Gemma3 running locally with DMR to Claude Opus 4.1 with a simple prompt about whales. Promptfoo provides a host of assertions to assess and grade model output. These assertions range from traditional deterministic evals, such as contains, to model-assisted evals, such as llm-rubric. By default, the model-assisted evals use Open AI models, but in this example, we’ll use local models powered by DMR. Specifically, we’ve configured smollm3:Q4_K_M to judge the output and mxbai-embed-large:335M-F16 to perform embedding to check the output semantics.
Source: Read the original article here.
Discover more tech insights on ByteTrending.
Discover more from ByteTrending
Subscribe to get the latest posts sent to your email.
