Engineering

Kubernetes 1.38 Enters Code Freeze Ahead of December 16: Breaking Removals, DRA Graduations, and Gang Scheduling Status

Kubernetes 1.38 has entered code and test freeze ahead of its planned December 16, 2026 release, locking down scope around v1.38.0-beta.0. While AI workload primitives like CompositePodGroup (v1beta1) and DRA Device Binding Conditions (GA) advance, several removed feature gates and stricter JWK/kubelet validations will break unverified upgrades. Here is what actually shipped in the 1.38 changelog, how the scheduling architecture works, and how to test your clusters before December 16.

By 0xScorPionSon

9 min read

Kubernetes 1.38 Enters Code Freeze Ahead of December 16: Breaking Removals, DRA Graduations, and Gang Scheduling Status

Kubernetes 1.38 code freeze and AI scheduling architecture infographic

Executive Summary

On November 17, 2026, Kubernetes 1.38 passed its code and test freeze (12:00 UTC following the November 16 Anywhere-on-Earth cutoff), locking down feature scope ahead of the planned December 16, 2026 general availability release. The v1.38.0-beta.0 pre-release tag shipped on November 4, 2026, and the project's deprecations and removals blog is scheduled for November 23, 2026. For platform engineering teams, 1.37 ("Garhwal", latest patch 1.37.2 on October 28, 2026) remains the current production baseline, while 1.34 reached end of life on October 27, 2026.

The 1.38 cycle centers on two operational realities: major Dynamic Resource Allocation (DRA) and workload-aware scheduling graduations for GPU and AI clusters, alongside strict feature-gate removals and security-driven validation changes that will prevent control-plane and node components from starting if clusters carry stale flags.

Confidence note: Release dates, milestone statuses, and changelog entries in this article are verified directly from the upstream CHANGELOG-1.38.md, the Kubernetes v1.37 DRA blog, and post-KubeCon release tracking from shattered.io and ajayk.sh. Because v1.38.0 has not yet shipped, final GA promotion of gang scheduling gates remains subject to the December 16 release notes. No universal 1.38 cluster-wide latency benchmark has been published by SIG Release.


Release Timeline and Support Baseline

Kubernetes maintains three active minor branches at any time. As of November 18, 2026, the supported upstream branches are 1.37, 1.36, and 1.35, with 1.34 having exited support on October 27, 2026.

Version Status (Nov 18, 2026) Latest Tag / Milestone Key Date
1.38 Code & test freeze (stabilization) v1.38.0-beta.0 Target GA: Dec 16, 2026
1.37 ("Garhwal") Current stable v1.37.2 EOL: Oct 28, 2027
1.36 Maintained v1.36.6 EOL: Jun 28, 2027
1.35 Maintained v1.35.10 EOL: Feb 28, 2027
1.34 End of life v1.34.13 (final patch) EOL: Oct 27, 2026

Because kubeadm and the Kubernetes version skew policy allow only one minor version step per control-plane upgrade, any self-managed cluster still running 1.34 must step through 1.35, 1.36, and 1.37 sequentially—it cannot jump directly to 1.38.


Technical Mechanics and Architecture

1. Workload-Aware Scheduling and CompositePodGroup

Traditional Kubernetes scheduling evaluates one Pod at a time. For distributed AI training or disaggregated inference, placing 60 out of 64 worker Pods leaves those 60 accelerators idle while waiting on the remaining 4. Kubernetes 1.37 introduced Beta gang scheduling behind the GenericWorkload feature gate (disabled by default) using Workload and PodGroup primitives.

In 1.38, the upstream CHANGELOG-1.38.md records several structural promotions:

  • CompositePodGroup promoted to scheduling.k8s.io/v1beta1 (#143088): Enables hierarchical, multi-role workload trees (such as prefill and decode pools in disaggregated inference) with garbage-collection finalizer protection (scheduling.k8s.io/compositepodgroup-protection, #141021) and recursive status propagation (#141860).
  • Topology-Aware Workload Scheduling (TAS) promoted to Beta (#140295): Controlled by TopologyAwareWorkloadScheduling (off by default in v1.38.0-beta.0), with a new percentageOfPlacementsToScore scheduler setting (#139859) to evaluate an adaptive subset of placements in large clusters rather than scoring every domain exhaustively.
  • Gang policy consistency validation (#142719, #142830, #143495): kube-scheduler and admission validation now enforce hierarchy depth/size limits, prevent a Gang parent from holding a Basic child, and disallow All disruption mode when the scheduling policy is not gang.
+-----------------------------------------------------------------------+
|                         kube-apiserver                                |
|  Workload (scheduling.k8s.io/v1beta1)                                 |
|   └── CompositePodGroup (root) ──> PodGroup A (Prefill, minCount: 4)  |
|                                └── PodGroup B (Decode,  minCount: 8)  |
+-----------------------------------+-----------------------------------+
                                    | Watch / Informers
                                    v
+-----------------------------------------------------------------------+
|                        kube-scheduler                                 |
|  1. QueueingHint / PreEnqueue: checks quorum & DRA ResourceClaims     |
|  2. PlacementFeasible: evaluates gang quorum across topology domains  |
|  3. TAS Scoring: adaptive subset via percentageOfPlacementsToScore    |
|  4. Preemption: workload-aware victim selection (DisruptionMode)      |
+-----------------------------------+-----------------------------------+
                                    | Bind Pods + ResourceClaims
                                    v
+-----------------------------------------------------------------------+
|                    Worker Nodes (kubelet + DRA)                       |
|  DRA Drivers (DeviceBindingConditions GA, PartitionableDevices GA)    |
|  -> Bind GPUs + NICs sharing PCIe / NUMA topology                     |
+-----------------------------------------------------------------------+

2. Dynamic Resource Allocation (DRA) Graduations

Dynamic Resource Allocation replaces opaque integer device counts (nvidia.com/gpu: 1) with structured ResourceClaim and ResourceSlice objects capable of matching device capabilities, PCIe root affinity, and resource.kubernetes.io/numaNode attributes. In v1.38.0-beta.0, CHANGELOG-1.38.md documents the following DRA graduations:

  • Device Binding Conditions (KEP #5007) graduate to GA (#142478).
  • DRAConsumableCapacity and DRAFractionalCapacityRange graduate to GA (#143309): Workloads can safely request fractional device capacity within driver-defined policy ranges.
  • DRAPartitionableDevices graduates to GA and is locked to true (#143478).
  • DRADeviceTaints and DRADeviceTaintRules are locked to true (GA) (#140256): Cluster administrators can taint degraded accelerators cluster-wide, and 1.38 rejects configurations attempting to set these gates to false (or warns/ignores before removal in 1.41).
  • DynamicResourceAllocation feature gate removed (#141946): DRA is now unconditionally enabled across API server, scheduler, and kubelet.
  • SchedulerPreQueueingHints promoted to Beta, enabled by default (#141732): Uses a pod informer index so ResourceClaim updates requeue only affected pods rather than scanning all unschedulable pods. As noted in the Kubernetes v1.37 DRA blog, this turns an $O(N^2)$ requeue path into $O(1)$.

3. Defensive Configuration: Testing GenericWorkload Safely in kind

Never toggle beta scheduling gates on a production control plane to observe behavior. Use a disposable kind cluster against v1.38.0-beta.0 (or v1.37.2) to validate how GenericWorkload and TopologyAwareWorkloadScheduling interact with your Job and PodGroup manifests:

# kind-1.38-scheduling.yaml — Safe local test cluster for 1.38 beta gates
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  GenericWorkload: true
  TopologyAwareWorkloadScheduling: true
nodes:
  - role: control-plane
  - role: worker
  - role: worker

Evidence: Breaking Changes, Security Fixes, and Scale Signals

Breaking Changes and Urgent Upgrade Notes in CHANGELOG-1.38.md

Every 1.38 pre-release section in CHANGELOG-1.38.md flags changes that can break cluster startup, authentication, or workloads on upgrade:

  1. Removed GA feature gates cause startup errors if still passed: SupplementalGroupsPolicy (#142093), SeparateTaintEvictionController (#141789), PodSchedulingReadiness (#141741), RecoverVolumeExpansionFailure (#139079), NodeSwap (#143587), ExecProbeTimeout (#143335), and SystemdWatchdog (#143337) have been removed. Passing --feature-gates=<Gate>=... to kube-apiserver, kube-controller-manager, kube-scheduler, or kubelet will fail component startup.
  2. OIDC JWK RSA key size limit (CVE-2026-116587, #143657): Updating github.com/go-jose/go-jose/v4 to v4.1.7 (and v4.1.9 in beta.0) rejects RSA keys larger than 4096 bits during JWK parsing. External OIDC identity providers using larger RSA keys will stop authenticating until rotated.
  3. Kubelet seccompProfile fail-closed (#141958): Pods reaching kubelet with an unknown or empty seccompProfile.type now fail with CreateContainerConfigError instead of falling back to Unconfined.
  4. Kubelet env-file quote validation (#143221): kubelet now rejects env-file entries with # immediately after a closing single quote (e.g., KEY='value'#suffix). Add whitespace before inline comments or move literal # characters inside quotes.
  5. User namespace ID range enforcement (#141345): kubelet refuses to start if /etc/subuid or /etc/subgid ranges for the kubelet user end at or past 4294967295 (2^32 - 1), requiring a reduction by at least one idsPerPod block (65536 by default).
  6. Go 1.27.3 header-count adjustment (#142607): Kubernetes 1.38 is built with Go 1.27.3 (#143201). Because Go 1.27 enforces a default limit of 500 request header values, kube-apiserver explicitly raises this to 16384 to avoid HTTP 431 errors for impersonated users with large group memberships.

Control-Plane Scale and Conformance Context

At KubeCon + CloudNativeCon North America 2026 (November 9–12), AWS Head of Containers Mike Stefaniak reported sustained scheduling rates above 300 Pods per second in large AWS clusters following 1.37 watch-cache and HPA lock-contention fixes, and noted that an AWS-co-authored proposal to compress API objects before storing them in etcd was merged as an alpha feature for 1.38. Note that the 300 Pods/sec figure is an AWS-reported internal cluster result rather than an upstream SIG Scalability benchmark.

On the governance side, Kubernetes graduated AI Conformance from a working group to a permanent SIG Architecture subproject on October 29, 2026, alongside launching the Workload Conformance Working Group on October 27, 2026 to standardize multi-cloud GPU and DRA behavior.


Hardening Playbook: Pre-Freeze Upgrade Audit

Run these checks on 1.37 clusters now—before the November 23 deprecations blog and December 16 GA—to ensure 1.38 upgrades remain routine maintenance rather than an outage.

1. Audit Component Manifests for Removed Feature Gates

Inspect all static pod manifests and systemd unit files on control-plane and worker nodes for feature gates removed in 1.38:

# Scan control-plane manifests and kubelet flags for removed 1.38 feature gates
grep -rnE "SupplementalGroupsPolicy|SeparateTaintEvictionController|PodSchedulingReadiness|RecoverVolumeExpansionFailure|NodeSwap|ExecProbeTimeout|SystemdWatchdog|DynamicResourceAllocation|DRADeviceTaints=false|DRADeviceTaintRules=false" \
  /etc/kubernetes/manifests/ /var/lib/kubelet/config.yaml /etc/systemd/system/kubelet.service.d/

2. Verify OIDC Issuer JWK Key Sizes

If your cluster authenticates users via an external OIDC provider, inspect the JWKS endpoint to verify no RSA signing key exceeds 4096 bits (CVE-2026-116587 mitigation in go-jose v4.1.7+):

# Check RSA modulus length in bits from your OIDC JWKS endpoint
curl -sS https://idp.example.com/.well-known/jwks.json \
  | jq -r '.keys[] | select(.kty=="RSA") | "\(.kid): \((.n | gsub("-";"+") | gsub("_";"/") | @base64d | length) * 8) bits"'

3. Scan Cluster Workloads for Deprecated DRA APIs and seccompProfile Edge Cases

Confirm no manifests or controllers still reference resource.k8s.io/v1beta1 or malformed seccompProfile blocks:

# Check cluster resources for Workload/DRA APIs and seccompProfile anomalies
kubectl api-resources | grep -iE "workload|podgroup|resourceclaim"
kubectl get pods -A -o json \
  | jq -r '.items[] | select(.spec.securityContext.seccompProfile.type == "" or (.spec.containers[]?.securityContext.seccompProfile.type? == "")) | "\(.metadata.namespace)/\(.metadata.name)"'

4. Validate subuid/subgid Ranges for User-Namespace Pods

On Linux nodes using spec.hostUsers: false, verify /etc/subuid and /etc/subgid do not end at or above 4294967295:

awk -F: '$1=="kubelet" {print $1, $2, $3, "end=" $2+$3-1}' /etc/subuid /etc/subgid

5. Update kubeadm Package Repositories

Verify nodes point to pkgs.k8s.io rather than the frozen legacy apt.kubernetes.io or yum.kubernetes.io repositories before pulling 1.37.x or 1.38.0-beta.0 packages.

6. Rollback Notes

If a 1.38 test or canary control-plane upgrade encounters issues, remember that kubeadm does not support automated minor-version downgrades. Snapshot etcd before upgrading (ETCDCTL_API=3 etcdctl snapshot save), keep staticOrder out of WebhookAdmissionConfiguration if you may roll back to 1.37 (per CHANGELOG-1.38.md, 1.37 rejects staticOrder at startup), and drain nodes before adjusting /etc/subuid or kubelet checkpoint formats.


Frequently Asked Questions

When is Kubernetes 1.38 scheduled to release?

SIG Release targets December 16, 2026 for v1.38.0 general availability. Code and test freeze took place November 16–17, 2026, and the official deprecations and removals blog is scheduled for November 23, 2026.

Is gang scheduling guaranteed to be GA (stable) in Kubernetes 1.38?

Not confirmed. While CompositePodGroup moved to v1beta1 and multiple reports list KEP-4671 gang scheduling as a 1.38 stable target, v1.38.0-beta.0 kept the GenericWorkload gate off by default. Wait for the final December 16 release notes before planning production dependencies.

What DRA features are stable in Kubernetes 1.38?

Per CHANGELOG-1.38.md, 1.38 graduates Device Binding Conditions (KEP #5007), DRAConsumableCapacity, DRAFractionalCapacityRange, and DRAPartitionableDevices to GA, locks DRADeviceTaints and DRADeviceTaintRules to true, and removes the DynamicResourceAllocation feature gate because DRA is unconditionally enabled.

Why would OIDC authentication break after upgrading to Kubernetes 1.38?

Kubernetes 1.38 updates github.com/go-jose/go-jose/v4 to address CVE-2026-116587, which enforces a maximum RSA key size of 4096 bits during JWK parsing. External OIDC issuers using larger RSA keys will fail token validation.

Can I upgrade directly from Kubernetes 1.34 or 1.35 to 1.38?

No. Kubernetes kubeadm and the version skew policy allow only one minor version step per control-plane upgrade (1.34 -> 1.35 -> 1.36 -> 1.37 -> 1.38).


Sources

  1. GitHub (kubernetes/kubernetes) — CHANGELOG-1.38.md (v1.38.0-beta.0) (Updated November 2026)
  2. Kubernetes Official Blog — Kubernetes v1.37: DRA Updates (Published September 3, 2026)
  3. Shattered.io — Kubernetes 1.38 Due Dec. 16: 3 AI Scheduling Bets (Updated November 17, 2026)
  4. Ajay K — Kubernetes 1.37 and 1.38: What's New in 2026 and What Got Removed (Published November 10, 2026)
  5. Cloud Native Now — Kubernetes Is Strengthening its Position as the AI Platform of Choice (Published November 13, 2026)

More on Engineering

Engineering

Microsoft agent-framework PyPI Hijack: .pth Startup Execution & Cloud Credential Theft

On November 17, 2026, attackers published two hijacked releases (1.26.1 and 1.26.2) of Microsoft's official agent-framework PyPI package during a ~45-minute window. Both releases dropped .pth startup hooks that execute on any Python invocation—even pip—to harvest environment variables, credential files, Azure IMDS tokens, and CI OIDC credentials before exfiltrating them to webhook.site.

11 min read

Engineering

Cloudflare Workers VPC Adds Custom CA and mTLS Support, Ending verify_ca Workarounds for Private Origins

Previously, connecting Cloudflare Workers VPC to internal APIs or private databases signed by an enterprise CA meant downgrading TLS verification to verify_ca or disabled—and origins requiring client certificates were unreachable. On November 18, 2026, Cloudflare added custom CA and mutual TLS (mTLS) client certificate support across both HTTP and TCP VPC Services, including Hyperdrive PostgreSQL connections to Amazon RDS. Here is how the certificate architecture works, how to configure it with Wrangler 4.164.0+ or Terraform 5.29.0+, and how to audit existing services.

11 min read

Tell us what is not working

A thirty-minute call, no deck and no obligation. You will leave with a straight answer about whether this is worth building - including if that answer is no.

Start a project

Direct line

hello@codebumble.net
Dhaka, Bangladesh  :   local timeCalls in English or Bangla