Docker Model Runner Now Generally Available
We’re excited to share that Docker Model Runner is now generally available (GA)! In April 2024, Docker introduced the Beta release of Docker Model Runner, making it easy to manage, run, and distribute local AI models (specifically LLMs). Though only a short time has passed since then, the product has evolved rapidly, with continuous enhancements driving the product to a reliable level of maturity and stability. Notably, this evolution demonstrates our commitment to providing developers with powerful tools for leveraging artificial intelligence.
This blog post takes a look back at the most important and widely appreciated capabilities Docker Model Runner brings to developers, and looks ahead to share what they can expect in the near future. For those new to the concept, the initial announcement provides a good overview.
What is Docker Model Runner?
Docker Model Runner (DMR) is built for developers first, making it easy to pull, run, and distribute large language models (LLMs) directly from Docker Hub (in an OCI-compliant format) or HuggingFace (if models are available in the GGUF format, in which case they will be packaged as OCI Artifacts on-the-fly by the HuggingFace backend). Furthermore, this integration simplifies the process of working with AI locally.
Tightly integrated with Docker Desktop and Docker Engine, DMR lets you serve models through OpenAI-compatible APIs, package GGUF files as OCI artifacts, and interact with them using either the command line, a graphical interface, or developer-friendly (REST) APIs. Therefore, developers can seamlessly incorporate AI into their workflows.
Whether you’re creating generative AI applications, experimenting with machine learning workflows, or embedding AI into your software development lifecycle, Docker Model Runner delivers a consistent, secure, and efficient way to work with AI models locally. As a result, teams can accelerate innovation and reduce complexity.
Check the official documentation to learn more about Docker Model Runner and its capabilities.
Why Docker Model Runner?
Docker Model Runner makes it easier for developers to experiment and build AI applications, including agentic apps, using the same Docker commands and workflows they already use every day. No need to learn a new tool! For example, existing Docker expertise can be immediately applied to managing LLMs.
Unlike many new AI tools that introduce complexity or require additional approvals, Docker Model Runner fits cleanly into existing enterprise infrastructure. It runs within your current security and compliance boundaries, so teams don’t have to jump through hoops to adopt it. Moreover, this allows for easier integration with existing systems.
Model Runner supports OCI-packaged models, allowing you to store and distribute models through any OCI-compatible registry, including Docker Hub. And for teams using Docker Hub, enterprise features like Registry Access Management (RAM) provide policy-based access controls to help enforce guardrails at scale. Consequently, security is significantly enhanced.
11 Docker Model Runner Features Developers Love Most
Below are the features that stand out the most and have been highly valued by the community. Notably, these features reflect developer feedback and a focus on usability.
Powered by llama.cpp
Currently, DMR is built on top of llama.cpp, which we plan to continue supporting. At the same time, DMR is designed with flexibility in mind, and support for additional inference engines (such as MLX or vLLM) is under consideration for future releases. As a result of this flexible design, Docker Model Runner can adapt to evolving AI technologies.
GPU acceleration across macOS and Windows platforms
Harness the full power of your hardware with GPU support: Apple Silicon on macOS, NVIDIA GPUs on Windows, and even ARM/Qualcomm acceleration — all seamlessly managed through
Source: Read the original article here.
Discover more tech insights on ByteTrending.
Discover more from ByteTrending
Subscribe to get the latest posts sent to your email.
