Skip to main content

Announcing KServe v0.21 - Direct KEDA Autoscaling, Kernel Cache, Standalone Tokenizer, and Canary Traffic Splitting

· 11 min read
Filippe Spolti
Filippe Spolti
Approver, KServe; Senior Software Engineer, Red Hat

Published on November 4, 2026

We are excited to announce the release of KServe v0.21. This release focuses on autoscaling flexibility, operational hardening, and deeper llm-d integration, with key highlights including:

  • Direct KEDA autoscaling for LLMInferenceService, with true scale-to-zero
  • Kernel Cache initial code donation with image signature verification
  • Standalone vLLM tokenizer deployment replacing the UDS tokenizer sidecar
  • Canary traffic splitting on RawDeployment HTTPRoutes
  • Rollout strategy controls for GPU-constrained rolling updates
  • LocalModelCache support for LoRA adapters
  • oci+fetch:// KServe-side OCI image pull and ModelScope (modelscope://) storage support
  • TLS hardening: cluster TLS profile flags, hot reload of rotated certificates, and TLS for transformers
  • Python 3.13 support and Go 1.26 upgrade

🚀 LLMInferenceService Enhancements​

📈 Direct KEDA Autoscaling and Scale-to-Zero​

Until now, KEDA-based scaling on LLMInferenceService was only available through the Workload Variant Autoscaler (WVA) path. KServe v0.21 adds a direct KEDA path via the new spec.scaling.keda field (#5839), letting users define their own KEDA triggers (CPU, memory, custom Prometheus queries, etc.) without WVA. The controller creates the ScaledObject from the user-defined triggers. spec.scaling.keda and spec.scaling.wva are mutually exclusive, and decode/prefill workloads must use the same scaling mode.

spec:
  scaling:
    minReplicas: 1
    maxReplicas: 4
    keda:
      idleReplicaCount: 0
      triggers:
        - type: prometheus
          metadata:
            serverAddress: http://prometheus.monitoring:9090
            query: sum(vllm:num_requests_waiting{namespace="my-ns"})
            threshold: "5"

In addition, idleReplicaCount: 0 is now accepted for both the WVA-mediated and direct KEDA paths (#5996), enabling true KEDA scale-to-zero. triggerAuthName can now also be used without authModes (#6004).

🔤 Standalone vLLM Tokenizer Deployment​

The UDS tokenizer sidecar inside the scheduler pod has been replaced by a standalone tokenizer Deployment and Service running vLLM's /v1/completions/render endpoint over HTTP (#5712). This follows the llm-d router v0.9.0+ architecture and enables independent scaling, lifecycle management, and health checking of the tokenizer.

The monolithic precise-prefix-cache-scorer plugin is decomposed into the new 3-plugin pipeline (token-producer + precise-prefix-cache-producer + prefix-cache-scorer). Existing configurations are migrated automatically with a deprecation warning event, and a new TokenizerReady condition reports the tokenizer status.

🔄 Rollout Strategy for Rolling Updates​

A new rolloutStrategy field (#5916) lets users configure maxUnavailable and maxSurge for both single-node Deployments and multi-node LeaderWorkerSets, including prefill workloads. This enables rolling updates on GPU-constrained clusters without spare hardware, or faster rollouts with additional surge capacity:

spec:
  rolloutStrategy:
    maxSurge: 0
    maxUnavailable: 1

💾 LocalModelCache for LoRA Adapters​

LoRA adapter URIs can now be served from LocalModelCache and LocalModelNamespaceCache (#5690). Cached adapters are mounted from PVCs instead of being downloaded at pod startup, independently of whether the base model is cached. Cache deletion is blocked while an LLMInferenceService references a cached adapter.

🧭 Scheduler (EPP) Configuration as Presets​

The default EndpointPicker configuration is now shipped as LLMInferenceServiceConfig presets instead of hardcoded strings (#5921), updated to the latest llm-d router plugins. Two presets mirror the llm-d guides: an optimized baseline profile and a prefill/decode disaggregation profile. Rendered config templates are now also validated through the admission chain before any child resources are created (#5918).

🔗 Working KV Transfer for Prefill/Decode​

Prefill/decode disaggregated deployments now get a NixlConnector injected automatically (kv_producer on prefill, kv_consumer on decode) when no other --kv-transfer-config is set (#6027). Previously, decode silently recomputed the prompt KV produced by prefill. User-supplied configuration still takes precedence.

📋 Additional LLMInferenceService Improvements​

  • Workloads automatically labeled with the InferencePool selector when an InferencePool ref is set (#5624)
  • Migration to new llm-d metrics flags and the metrics-data-source plugin (#5877, #5930)
  • EndpointPickerConfig apiVersion migration (#5774)
  • LoRA adapter validation and disambiguation of colliding mount paths (#6092, #6085)
  • Traffic splitting group fixes for stopped and terminating members and pool matching (#5847, #6149, #6156)
  • Agentic tool calling guide and samples (#5906)

🧠 Kernel Cache​

KServe v0.21 includes the initial code donation of Kernel Cache (#5590), starting with the MCV CLI tool and API for building kernel cache container images, extracting caches, and running preflight checks. Pre-compiled GPU kernel caches can significantly reduce model server startup times by avoiding repeated kernel compilation.

To ensure that only trusted kernel caches are used, this release also introduces a security verification framework (#6099), cert-mode signature verification (#6103), and a signing contract (#6118).

See the Kernel Cache documentation to get started.

🔧 InferenceService and Platform Improvements​

🐦 Canary Traffic Splitting on RawDeployment HTTPRoutes​

Building on the canary rollout support introduced in v0.20, HTTPRoutes in RawDeployment mode now carry weighted backend refs for canaries (#5912). The stable predictor receives the remaining weight after all canary percentages, and HTTPRoute names and hostnames remain stable through promotion — only backend refs change:

# HTTPRoute: my-model-predictor (20% canary)
backendRefs:
  - name: my-model-predictor
    weight: 80
  - name: my-model-v2-predictor
    weight: 20

Canary traffic is now gated on readiness to prevent transient 503s (#5984), canary weights are restricted to predictor rules (#5944), and orphaned autoscaler and OpenTelemetry resources are cleaned up on promotion (#5899).

🔍 InferenceService Tracing API​

InferenceService now has an optional top-level spec.tracing field (#6108), aligned with the LLMInferenceService tracing API. Setting tracing: {} applies defaults (otlp exporter to http://otel-collector:4317, parentbased_traceidratio sampler at 0.05), with admission validation for sampler names and exporter endpoints. This release establishes the API contract; workload injection will follow in a future release.

📦 Storage Enhancements​

  • oci+fetch:// (#5739) — Step 3 of the OCI storage harmonization roadmap (#4083). An init container pulls the OCI image with oras-py and extracts the /models/ subtree, without depending on container runtime features. This works on older Kubernetes versions and restricted environments, and supports private registries through imagePullSecrets.

  • ModelScope support (#5330) — download models directly from ModelScope using the modelscope:// URI scheme, mirroring the existing Hugging Face integration:

    storageUri: "modelscope://qwen/Qwen2-0.5B-Instruct"

  • IBM Power (ppc64le) support for the storage initializer (#5235)

  • AWS SDK migrated from v1 to v2 in the agent (#5438)

🎯 DRA Support for ServingRuntime​

ServingRuntime and ClusterServingRuntime now support pod-level resourceClaims for Dynamic Resource Allocation (#5828). Runtime defaults are merged with InferenceService predictor overrides, so DRA-backed InferenceServices can be scheduled correctly.

🧩 Independent CRD Installation​

CRD management has been restructured so CRDs can be installed independently from the controllers with proper cert-manager webhook annotations (#5843). All existing installation methods continue to work.

Additional Enhancements​

  • Python 3.13 support (#5897)
  • AutoGluon server injects a synthetic item ID for two-column time series inputs (#6009)
  • Headless InferenceService status URL now reports the correct port (#5968)
  • KEDA ScaledObject is skipped for components without autoscaling configured (#6029)
  • Storage resources templated in the inferenceservice-config ConfigMap (#5697)
  • JWE decryption supports both full JWK and raw key formats for confidential model serving (#6034)

🔒 Security Improvements​

TLS Hardening​

  • Cluster TLS security profile integration (#5791) — new --tls-min-version and --tls-cipher-suites flags for webhook and metrics servers across all KServe controllers, defaulting to hardened TLS 1.2. The --enable-http2 flag has been removed from the llmisvc controller.
  • Rotated TLS certificates are reloaded by the model server without a restart (#6106).
  • TLS for transformers via the KSERVE_TLS_CERT_FILE and KSERVE_TLS_KEY_FILE environment variables, enabling end-to-end TLS (#6045), plus CA bundle injection for httpx (#5932).

Hardening and CVE Fixes​

  • Controller manager ClusterRoles hardened by removing unused permissions (#5785)
  • AutoGluon artifact loading secured and path traversal prevented (#5803, #5802)
  • Agent crash prevented when model spec unmarshalling fails (#5700)
  • Pillow — CVE-2026-40192, CVE-2026-54058, CVE-2026-59200 (#5959, #5998, #6010)
  • cryptography — CVE-2026-69247 (#5957, #5999)
  • black — CVE-2026-32274 (#5953)
  • go-jose — GHSA-78h2-9frx-2jm8 (#5985)
  • golang.org/x/crypto — CVE-2026-46595 (#5993)
  • aiohttp — zip bomb DoS (#5971)
  • kin-openapi — nil-pointer panic in request validation (#5992)
  • pyasn1 — CVE-2026-59885 (#6011)
  • urllib3 — CVE-2026-21441 (#6023)
  • opentelemetry-go — CVE-2026-39883 (#6019)
  • setuptools — CVE-2025-47273 (#6055)

📦 Dependency Upgrades​

  • llm-d images upgraded to the latest v0.10 release (#6026)
  • Envoy AI Gateway upgraded to v1.1.0 (#6122)
  • KEDA upgraded to v2.20.2 (#6153)
  • WVA bumped to v0.8.0 (#5722)
  • Go upgraded to 1.26 (#6127)
  • pyarrow now allows versions newer than 20 (#6125)

🏗️ Infrastructure and CI​

  • Makefile split into fragment files and kustomize image overrides refactored (#6058, #6060)
  • Updated install scripts for older versions and a new install script (#6061)
  • Kustomize validation workflow for manifests (#5878)
  • CLI flag drift detection when image tags are bumped (#6035)
  • Shell script linting with shellcheck (#6038)
  • Native ARM image builds (#6070)
  • Doc samples verified against the real admission chain (#6039)
  • Per-test namespace isolation and expanded canary, KEDA, and weighted InferencePool e2e coverage (#5832, #5849, #6018, #5886)
  • Timeouts on all GitHub Actions jobs (#5933)

🔍 Release Notes​

For the complete list of all merged pull requests, bug fixes, and known issues, visit the GitHub release pages:

🙏 Acknowledgments​

We extend our gratitude to all 39 contributors who made this release possible, including 15 first-time contributors. Your efforts continue to drive the advancement of KServe as a leading platform for serving machine learning models.

  • New Contributors: @aneeshkp, @ugiordan, @unichronic, @iasthc, @harshhh817, @mgonzalezo, @0-Zaid-0, @anxkhn, @noah4477, @mateenali66, @xrwang8, @Wojciech-Rebisz, @ThatsMrTalbot, @NoOverflow, @Sunidhi-Gaonkar1 — welcome and thank you!
  • Core Contributors: The KServe maintainers and regular contributors
  • Community: Everyone who reported issues, provided feedback, and tested features

🤝 Join the Community​

We invite you to explore the new features in KServe v0.21 and contribute to the ongoing development of the project:

Happy serving!

The KServe team is committed to making machine learning model serving simple, scalable, and standardized. Thank you for being part of our community!