Skip to content

Commit c582aa1

Browse files
aojeaaojea
committed
ServiceCIDR documentation
Add documentation about the new ServiceCIDR feature and how to handle cluster updates of the default ServiceCIDR
1 parent 75b1f4a commit c582aa1

File tree

2 files changed

+230
-1
lines changed

2 files changed

+230
-1
lines changed

‎content/en/docs/tasks/network/extend-service-ip-ranges.md

Lines changed: 104 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,13 @@ This document shares how to extend the existing Service IP range assigned to a c
1919

2020
{{< version-check >}}
2121

22+
{{< note >}}
23+
While you can use this feature with an earlier version, the feature is only GA and officially supported since v1.33.
24+
{{< /note >}}
25+
2226
2327

24-
## API
28+
## Extend Service IP Ranges
2529

2630
Kubernetes clusters with kube-apiservers that have enabled the `MultiCIDRServiceAllocator`
2731
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) and have the
@@ -196,3 +200,102 @@ kubectl get servicecidr newcidr1
196200
```
197201
Error from server (NotFound): servicecidrs.networking.k8s.io "newcidr1" not found
198202
```
203+
204+
## Kubernetes Service CIDR Policies
205+
206+
Cluster administrators can implement policies to control the creation and
207+
modification of ServiceCIDR resources within the cluster. This allows for
208+
centralized management of the IP address ranges used for Services and helps
209+
prevent unintended or conflicting configurations. Kubernetes provides mechanisms
210+
like Validating Admission Policies to enforce these rules.
211+
212+
### Preventing Unauthorized ServiceCIDR Creation/Update using Validating Admission Policy
213+
214+
There can be situations that the cluster administrators want to restrict the
215+
ranges that can be allowed or to completely deny any changes to the cluster
216+
Service IP ranges.
217+
218+
{{< note >}}
219+
The default "kubernetes" ServiceCIDR is created by the kube-apiserver
220+
to provide consistency in the cluster and is required for the cluster to work,
221+
so it always must be allowed.
222+
{{}}
223+
224+
#### Restrict Service CIDR ranges to some specific ranges
225+
226+
The following is an example of a `ValidatingAdmissionPolicy` that only allows
227+
ServiceCIDRs to be created if they are subranges of the given `allowed` ranges.
228+
(So the example policy would allow a ServiceCIDR with `cidrs: ['10.96.1.0/24']`
229+
or `cidrs: ['2001:db8:0:0:ffff::/80', '10.96.0.0/20']` but would not allow a
230+
ServiceCIDR with `cidrs: ['172.20.0.0/16']`.) You can copy this policy and change
231+
the value of `allowed` to something appropriate for you cluster.
232+
233+
...
234+
235+
Consult the [CEL documentation](https://kubernetes.io/docs/reference/using-api/cel/)
236+
to learn more about CEL if you want to write your own validation `expression`.
237+
238+
```yaml
239+
apiVersion: admissionregistration.k8s.io/v1
240+
kind: ValidatingAdmissionPolicy
241+
metadata:
242+
name: "servicecidrs.default"
243+
spec:
244+
failurePolicy: Fail
245+
matchConstraints:
246+
resourceRules:
247+
- apiGroups: ["networking.k8s.io"]
248+
apiVersions: ["v1","v1beta1"]
249+
operations: ["CREATE", "UPDATE"]
250+
resources: ["servicecidrs"]
251+
matchConditions:
252+
- name: 'exclude-default-servicecidr'
253+
expression: "object.metadata.name != 'kubernetes'"
254+
variables:
255+
- name: allowed
256+
expression: "['10.96.0.0/16','2001:db8::/64']"
257+
validations:
258+
- expression: "object.spec.cidrs.all(currentCIDR, variables.allowed.exists(allowedCIDR, cidr(allowedCIDR).containsCIDR(currentCIDR)))"
259+
# For all CIDRs (i) listed in the spec.cidrs of the submitted ServiceCIDR
260+
# object, check if there exists at least one CIDR (j) in the predefined
261+
# allowed list such that the allowed CIDR (j) fully contains the submitted
262+
# CIDR (i)."
263+
---
264+
apiVersion: admissionregistration.k8s.io/v1
265+
kind: ValidatingAdmissionPolicyBinding
266+
metadata:
267+
name: "servicecidrs-binding"
268+
spec:
269+
policyName: "servicecidrs.default"
270+
validationActions: [Deny,Audit]
271+
```
272+
273+
#### Restrict any usage of the ServiceCIDR API
274+
275+
The following example demonstrates how to use a `ValidatingAdmissionPolicy` and
276+
its binding to restrict the creation of any new Service CIDR ranges, excluding the default "kubernetes" ServiceCIDR:
277+
278+
```yaml
279+
apiVersion: admissionregistration.k8s.io/v1
280+
kind: ValidatingAdmissionPolicy
281+
metadata:
282+
name: "servicecidrs.deny"
283+
spec:
284+
failurePolicy: Fail
285+
matchConstraints:
286+
resourceRules:
287+
- apiGroups: ["networking.k8s.io"]
288+
apiVersions: ["v1","v1beta1"]
289+
operations: ["CREATE", "UPDATE"]
290+
resources: ["servicecidrs"]
291+
validations:
292+
- expression: "object.metadata.name == 'kubernetes'"
293+
---
294+
apiVersion: admissionregistration.k8s.io/v1
295+
kind: ValidatingAdmissionPolicyBinding
296+
metadata:
297+
name: "servicecidrs-deny-binding"
298+
spec:
299+
policyName: "servicecidrs.deny"
300+
validationActions: [Deny,Audit]
301+
```

‎content/en/docs/tasks/network/reconfigure-default-service-ip-ranges.md

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
---
2+
reviewers:
3+
- thockin
4+
- dwinship
5+
min-kubernetes-server-version: v1.33
6+
title: Kubernetes Default Service CIDR Reconfiguration
7+
content_type: task
8+
---
9+
10+
11+
{{< feature-state feature_gate_name="MultiCIDRServiceAllocator" >}}
12+
13+
This document shares how to reconfigure the default Service IP range(s) assigned
14+
to a cluster.
15+
16+
## {{% heading "prerequisites" %}}
17+
18+
{{< include "task-tutorial-prereqs.md" >}}
19+
20+
{{< version-check >}}
21+
22+
23+
24+
## Kubernetes Default Service CIDR Reconfiguration
25+
26+
This document explains how to manage the Service IP address range within a
27+
Kubernetes cluster, which also influences the cluster's supported IP families
28+
for Services.
29+
30+
The IP families available for Service ClusterIPs are determined by the
31+
`--service-cluster-ip-range` flag to kube-apiserver. For a better understanding of Service IP address allocation, refer to the
32+
[Services IP address allocation tracking](https://kubernetes.io/docs/reference/networking/virtual-ips/#ip-address-objects) documentation.
33+
34+
Since Kubernetes 1.33, the Service IP families configured for the cluster are
35+
reflected by the `ServiceCIDR` object named `kubernetes`. The `kubernetes` `ServiceCIDR`
36+
object is created by the first kube-apiserver instance that starts, based on its
37+
configured `--service-cluster-ip-range` flag. To ensure consistent cluster behavior, all kube-apiserver instances must be configured with the same `--service-cluster-ip-range` values, which must match the default kubernetes ServiceCIDR object.
38+
39+
### Kubernetes Service CIDR Reconfiguration Categories
40+
41+
We can categorize Service CIDR reconfiguration into the following scenarios:
42+
43+
* **Extending the existing Service CIDRs:** This can be done dynamically by
44+
adding new ServiceCIDR objects without the need of reconfiguration of the
45+
kube-apiserver. Please refer to the dedicated documentation on
46+
[Extending Service IP
47+
Ranges](https://kubernetes.io/docs/tasks/network/extend-service-ip-ranges/).
48+
49+
* **Single-to-dual-stack conversion preserving the primary service CIDR:** This
50+
involves introducing a secondary IP family (IPv6 to an IPv4-only cluster, or
51+
IPv4 to an IPv6-only cluster) while keeping the original IP family as
52+
primary. This requires an update to the kube-apiserver configuration and a
53+
corresponding modification of various cluster components that need to handle
54+
this additional IP family. These components include, but are not limited to,
55+
kube-proxy, the CNI or network plugin, service mesh implementations, and DNS
56+
services.
57+
58+
* **Dual-to-single conversion preserving the primary service CIDR:** This
59+
involves removing the secondary IP family from a dual-stack cluster,
60+
reverting to a single IP family while retaining the original primary IP
61+
family. In addition to the reconfiguration of the components to match the
62+
new IP family, you might need to address Services that were explicitly
63+
configured to use the removed IP family.
64+
65+
* **Anything that results in changing the primary service CIDR:** Completely
66+
replacing the default ServiceCIDR is a complex operation. If the new
67+
ServiceCIDR does not overlap with the existing one, [it will require
68+
renumbering all existing Services and changing the `kubernetes.default`
69+
service](#Illustrative Reconfiguration Steps). The case where the primary IP
70+
family also changes is even more complicated, and may require to change
71+
multiple cluster components (kubelet, network plugins, etc.) to match the new
72+
primary IP family.
73+
74+
### Manual Operations for Replacing the Default Service CIDR
75+
76+
Reconfiguring the default Service CIDR necessitates manual steps performed by
77+
the cluster operator, administrator, or the software managing the cluster
78+
lifecycle. These typically include:
79+
80+
1. **Updating** the kube-apiserver configuration: Modify the
81+
`--service-cluster-ip-range` flag with the new IP range(s).
82+
2. **Reconfiguring** the network components: This is a critical step and the
83+
specific procedure depends on the different networking components in use. It
84+
might involve updating configuration files, restarting agent pods, or
85+
updating the components to manage the new Service CIDR(s) and the desired IP
86+
family configuration for Pods. Typical components can be the implementation
87+
of Kubernetes Services, such as kube-proxy, and the configured networking
88+
plugin, and potentially other networking components like service mesh
89+
controllers and DNS servers, to ensure they can correctly handle traffic and
90+
perform service discovery with the new IP family configuration.
91+
3. **Managing existing Services:** Services with IPs from the old CIDR need to
92+
be addressed if they are not within the new configured ranges. Options
93+
include recreation (leading to downtime and new IP assignments) or
94+
potentially more complex reconfiguration strategies.
95+
4. **Recreating internal Kubernetes services:** The `kubernetes.default`
96+
service must be deleted and recreated to obtain an IP address from the new
97+
Service CIDR if the primary IP family is changed or replaced by a different
98+
network.
99+
100+
### Illustrative Reconfiguration Steps
101+
102+
The following steps describe a controlled reconfiguration focusing on the
103+
completely replacement of the default Service CIDR and the recreation of the
104+
`kubernetes.default` Service:
105+
106+
1. Start the kube-apiserver with the initial `--service-cluster-ip-range`.
107+
2. Create initial Services that obtain IPs from this range.
108+
3. Introduce a new Service CIDR as a temporary target for reconfiguration.
109+
4. Mark the `kubernetes` default Service CIDR for deletion (it will remain
110+
pending due to existing IPs and finalizers). This prevents new allocations
111+
from the old range.
112+
5. Recreate existing Services. They should now be allocated IPs from the new,
113+
temporary Service CIDR.
114+
6. Restart the kube-apiserver with the new Service CIDR(s) configured and shut
115+
down the old instance.
116+
7. Delete the `kubernetes.default` service. The new kube-apiserver will
117+
recreate it within the new Service CIDR.
118+
119+
## {{% heading "whatsnext" %}}
120+
121+
* **Kubernetes Networking Concepts:**
122+
[https://kubernetes.io/docs/concepts/cluster-administration/networking/](https://kubernetes.io/docs/concepts/cluster-administration/networking/)
123+
* **Kubernetes Dual-Stack Services:**
124+
[https://kubernetes.io/docs/concepts/services-networking/dual-stack/](https://kubernetes.io/docs/concepts/services-networking/dual-stack/)
125+
* **Extending Kubernetes Service IP Ranges:**
126+
[https://kubernetes.io/docs/tasks/network/extend-service-ip-ranges/](https://kubernetes.io/docs/tasks/network/extend-service-ip-ranges/)

0 commit comments

Comments
 (0)