VPC Nat Gateway Functionality from KuebOVN#985
Conversation
github-actions
bot
requested review from
akashraj4261,
dariavladykina and
jillian-maroket
|
There was a problem hiding this comment.
Pull request overview
Adds a new documentation page describing how to use Kube-OVN VPC NAT Gateway to provide outbound (SNAT) and inbound (DNAT) connectivity for overlay VMs, including an end-to-end example configuration.
Changes:
- Introduces a new doc page for VPC NAT Gateway concepts and workflow.
- Provides example manifests for NADs, VPC/Subnets, VpcNatGateway, and iptables-based EIP/SNAT/DNAT rules.
- Includes verification steps and expected outputs (StatefulSet/Pod details, interfaces, iptables rules).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| Kube-OVN supports NAT via Kubernetes custom resources (CRDs), not just IP-tables directly. For example, resources like OvnEip, OvnSnatRule, OvnDnatRule (or their iptables-based equivalents) are used to define NAT behavior declaratively. In the context of Harvester, the VM orchestration (compute, storage, VM lifecycle) is handled by Harvester; networking — including routing, NAT, VPC/subnets — is handled by Kube-OVN. This separation allows for more scalable, flexible and clean networking. | ||
|
|
||
| ## KubeOVN as Secondary CNI |
There was a problem hiding this comment.
This heading has inconsistent spacing and product naming. Elsewhere the docs use "Kube-OVN"; consider renaming the heading accordingly and removing the extra space after the ## so it renders consistently in navigation/TOC.
| ## KubeOVN as Secondary CNI | |
| ## Kube-OVN as Secondary CNI |
| ``` | ||
| kubectl get net-attach-def vswitchexternal1 -o yaml | ||
| apiVersion: k8s.cni.cncf.io/v1 | ||
| kind: NetworkAttachmentDefinition | ||
| metadata: | ||
| labels: | ||
| network.harvesterhci.io/ready: "true" | ||
| network.harvesterhci.io/type: OverlayNetwork | ||
| name: vswitchexternal | ||
| namespace: kube-system |
There was a problem hiding this comment.
The example command is querying vswitchexternal1, but the manifest shown below sets metadata.name: vswitchexternal. Readers following this will likely get a NotFound error; please make the resource name consistent (either change the command or the manifest name).
| kind: VpcNatGateway | ||
| apiVersion: kubeovn.io/v1 | ||
| metadata: | ||
| annotations: | ||
| k8s.v1.cni.cncf.io/networks: default/vswitchinternal |
There was a problem hiding this comment.
For consistency with the other YAML snippets in this doc (and most Kubernetes examples), put apiVersion before kind. Also consider fixing the extra indentation under metadata.annotations to make the YAML easier to copy/paste without confusion.
| kind: VpcNatGateway | |
| apiVersion: kubeovn.io/v1 | |
| metadata: | |
| annotations: | |
| k8s.v1.cni.cncf.io/networks: default/vswitchinternal | |
| apiVersion: kubeovn.io/v1 | |
| kind: VpcNatGateway | |
| metadata: | |
| annotations: | |
| k8s.v1.cni.cncf.io/networks: default/vswitchinternal |
| ``` | ||
|
|
||
| ##### Create a VM and attach it to the vswitchinternal overlay network and add the following default route in the guest os. | ||
| (172.20.10.254 is the ip addresses on net1 interface on vpc nat gw pod) |
There was a problem hiding this comment.
Typo: "ip addresses" should be singular here (and consider capitalizing "IP").
| (172.20.10.254 is the ip addresses on net1 interface on vpc nat gw pod) | |
| (172.20.10.254 is the IP address on net1 interface on vpc nat gw pod) |
|
|
||
| Flexible networking for VPCs / overlay networks: Using NAT (SNAT / DNAT) with Kube-OVN means you can build isolated private subnets / VPCs and still allow controlled egress (outbound) or ingress (inbound) traffic. This is especially relevant for VM workloads managed by Harvester, where VMs may need internet access or to expose services externally. | ||
|
|
||
| Kube-OVN supports NAT via Kubernetes custom resources (CRDs), not just IP-tables directly. For example, resources like OvnEip, OvnSnatRule, OvnDnatRule (or their iptables-based equivalents) are used to define NAT behavior declaratively. In the context of Harvester, the VM orchestration (compute, storage, VM lifecycle) is handled by Harvester; networking — including routing, NAT, VPC/subnets — is handled by Kube-OVN. This separation allows for more scalable, flexible and clean networking. |
There was a problem hiding this comment.
Use the standard term "iptables" here (instead of "IP-tables") to match common Linux/Kube-OVN terminology and avoid confusion when cross-referencing commands like iptables-save.
| Kube-OVN supports NAT via Kubernetes custom resources (CRDs), not just IP-tables directly. For example, resources like OvnEip, OvnSnatRule, OvnDnatRule (or their iptables-based equivalents) are used to define NAT behavior declaratively. In the context of Harvester, the VM orchestration (compute, storage, VM lifecycle) is handled by Harvester; networking — including routing, NAT, VPC/subnets — is handled by Kube-OVN. This separation allows for more scalable, flexible and clean networking. | |
| Kube-OVN supports NAT via Kubernetes custom resources (CRDs), not just iptables directly. For example, resources like OvnEip, OvnSnatRule, OvnDnatRule (or their iptables-based equivalents) are used to define NAT behavior declaratively. In the context of Harvester, the VM orchestration (compute, storage, VM lifecycle) is handled by Harvester; networking — including routing, NAT, VPC/subnets — is handled by Kube-OVN. This separation allows for more scalable, flexible and clean networking. |
| externalvswitch.kube-system.ovn.kubernetes.io/routes: '[{"dst":"0.0.0.0/0","gw":"10.115.55.254"}]' | ||
| internalvswitch.default.ovn.kubernetes.io/ip_address: 172.20.10.254 | ||
| internalvswitch.default.ovn.kubernetes.io/logical_switch: internalsubnet | ||
| internalvswitch.default.ovn.kubernetes.io/routes: '[{"dst":"10.55.0.0/16","gw":"172.20.10.1"},{"dst":"10.115.48.0/21","gw":"172.20.10.1"}]' | ||
| internalvswitch.default.ovn.kubernetes.io/vpc_nat_gw: gw1 | ||
| k8s.v1.cni.cncf.io/networks: default/internalvswitch, kube-system/externalvswitch | ||
| ovn.kubernetes.io/ip_address: 172.20.10.254 | ||
| ovn.kubernetes.io/logical_switch: internalsubnet |
There was a problem hiding this comment.
The StatefulSet/pod output uses internalvswitch/externalvswitch and internalsubnet/externalsubnet, but the earlier examples define vswitchinternal/vswitchexternal and subnetinternal/subnetexternal. This mismatch makes it hard to follow which names should be used; please align the examples/outputs to the same object names (or add an explicit note explaining why the names differ).
| externalvswitch.kube-system.ovn.kubernetes.io/routes: '[{"dst":"0.0.0.0/0","gw":"10.115.55.254"}]' | |
| internalvswitch.default.ovn.kubernetes.io/ip_address: 172.20.10.254 | |
| internalvswitch.default.ovn.kubernetes.io/logical_switch: internalsubnet | |
| internalvswitch.default.ovn.kubernetes.io/routes: '[{"dst":"10.55.0.0/16","gw":"172.20.10.1"},{"dst":"10.115.48.0/21","gw":"172.20.10.1"}]' | |
| internalvswitch.default.ovn.kubernetes.io/vpc_nat_gw: gw1 | |
| k8s.v1.cni.cncf.io/networks: default/internalvswitch, kube-system/externalvswitch | |
| ovn.kubernetes.io/ip_address: 172.20.10.254 | |
| ovn.kubernetes.io/logical_switch: internalsubnet | |
| vswitchexternal.kube-system.ovn.kubernetes.io/routes: '[{"dst":"0.0.0.0/0","gw":"10.115.55.254"}]' | |
| vswitchinternal.default.ovn.kubernetes.io/ip_address: 172.20.10.254 | |
| vswitchinternal.default.ovn.kubernetes.io/logical_switch: subnetinternal | |
| vswitchinternal.default.ovn.kubernetes.io/routes: '[{"dst":"10.55.0.0/16","gw":"172.20.10.1"},{"dst":"10.115.48.0/21","gw":"172.20.10.1"}]' | |
| vswitchinternal.default.ovn.kubernetes.io/vpc_nat_gw: gw1 | |
| k8s.v1.cni.cncf.io/networks: default/vswitchinternal, kube-system/vswitchexternal | |
| ovn.kubernetes.io/ip_address: 172.20.10.254 | |
| ovn.kubernetes.io/logical_switch: subnetinternal |
|
|
||
| ``` | ||
|
|
||
| ##### Verify SNAT filter iptable rule created inside the VPC NAT gateway pod |
There was a problem hiding this comment.
Typo: "iptable" should be "iptables" (the command examples below use iptables-legacy-save).
| ##### Verify SNAT filter iptable rule created inside the VPC NAT gateway pod | |
| ##### Verify SNAT filter iptables rule created inside the VPC NAT gateway pod |
| ``` | ||
|
|
||
| Ping from VM (inside guest os) to 8.8.8.8 must be successful | ||
| The traffic from VM reaches net1 of vpc nat gw pod and with route installed egress out of net2 and hits the iptable rule for SNAT and translates 172.20.10.0/24 subnet ip to 10.115.55.200 for external connectivity. |
There was a problem hiding this comment.
Typo: "iptable" should be "iptables".
| The traffic from VM reaches net1 of vpc nat gw pod and with route installed egress out of net2 and hits the iptable rule for SNAT and translates 172.20.10.0/24 subnet ip to 10.115.55.200 for external connectivity. | |
| The traffic from VM reaches net1 of vpc nat gw pod and with route installed egress out of net2 and hits the iptables rule for SNAT and translates 172.20.10.0/24 subnet ip to 10.115.55.200 for external connectivity. |
| test nginx locally on the VM | ||
| curl http://127.0.0.1 | ||
|
|
||
| Now curl -k http://10.115.55.200:8888 from external must be successful |
There was a problem hiding this comment.
curl -k disables TLS certificate verification, but this example uses plain HTTP (http://). Either drop -k or switch the example to https:// if TLS is intended; as written, -k is misleading.
| Now curl -k http://10.115.55.200:8888 from external must be successful | |
| Now curl http://10.115.55.200:8888 from external must be successful |
w13915984028
left a comment
w13915984028
left a comment
There was a problem hiding this comment.
Thanks for the PR, some suggestions.
|
|
||
| Flexible networking for VPCs / overlay networks: Using NAT (SNAT / DNAT) with Kube-OVN means you can build isolated private subnets / VPCs and still allow controlled egress (outbound) or ingress (inbound) traffic. This is especially relevant for VM workloads managed by Harvester, where VMs may need internet access or to expose services externally. | ||
|
|
||
| Kube-OVN supports NAT via Kubernetes custom resources (CRDs), not just IP-tables directly. For example, resources like OvnEip, OvnSnatRule, OvnDnatRule (or their iptables-based equivalents) are used to define NAT behavior declaratively. In the context of Harvester, the VM orchestration (compute, storage, VM lifecycle) is handled by Harvester; networking — including routing, NAT, VPC/subnets — is handled by Kube-OVN. This separation allows for more scalable, flexible and clean networking. |
There was a problem hiding this comment.
(compute, storage, VM lifecycle)
->
(compute, storage, VM lifecycle, L2 VLAN based networking, underlay networking)
networking — including routing, NAT, VPC/subnets
->
overlay networking — including routing, NAT, VPC/subnets
|
|
||
| ### Example Configuration and Working | ||
|
|
||
| #### Create a network attachment definition (tenant or internal network) |
There was a problem hiding this comment.
is the configuration process available on UI now? when true, please also add descriptions about how to create things on UI step by step
There was a problem hiding this comment.
UI is planned for v1.9.0
There was a problem hiding this comment.
please mention this on doc, UI is not available on v180, thanks.
| "/run/openvswitch/kube-ovn-daemon.sock", "provider": "vswitchinternal.default.ovn"}' | ||
| ``` | ||
|
|
||
| #### Create a network attachment definition (external network) |
There was a problem hiding this comment.
it will be helpful on readability to have
diagram to show the traffic in/out
or
diagram to show the raltionships of those CRDs
rrajendran17
force-pushed
the
vpcnatgw
branch
2 times, most recently
from
5f0bdff to
7997126
Compare
w13915984028
left a comment
w13915984028
left a comment
There was a problem hiding this comment.
LGTM, the steps and details are very helpful, thanks.
Signed-off-by: Renuka Devi Rajendran <renuka.rajendran@suse.com>
|
@w13915984028 I have updated the flow diagram to be even more clearer.Can you please review and approve? We can merge this. |
Problem:
Add documentation for VPC Nat Gateway functionality for external and inbound access for overlay VMs
Solution:
Add documentation for VPC Nat Gateway functionality for external and inbound access for overlay VMs
Related Issue(s):
harvester/harvester#9455
Test plan:
Additional documentation or context