Skip to main content

Using Private Clusters in AWS

Private cluster support in AWS can be achieved using one of these methods:

  • Direct peering between the control plane and the workspace networks
  • Hub and spoke model provided by AWS Transit Gateway

This topic provides a high-level summary for private cluster support in AWS, and a worked example of the process.

Summary of the private cluster process​

Enabling private cluster support in AWS includes these high-level tasks. Click the links for more information.

The sections below provide an example of these and other tasks.

Example assumptions​

For the purposes of the example in this topic, we assume the following requirements:

  • Connectivity between Wayfinder and private clusters will be facilitated by direct peering.
  • We have been allocated a network range of 10.20.0.0/16, which will be sliced up and allocated on demand to the clusters.
  • We intend to use a dedicated virtual network 10.80.0.0/24, residing in another account, where a VPN will be situated for users.
  • We need to peer all our clusters with our CI network 10.70.0.0/24, which is running in another account.
  • The CI solution is managed by another team that wants to manually approve peering.
  • We want to lock down the authentication module to only speak to CI and VPN.

Create the IP allocation for clusters​

This step is optional, but highly recommended.

An IP allocation for the cloud provider sets up a pool of address spaces, which is sliced up and allocated on demand to the clusters. This ensures that you don't create peers with overlapping address spaces, or networks that may overlap with shared services.

To add an IP allocation using the UI or CLI, see detailed instructions in Add a network allocation.

note

Currently we have no support for IP management of the EKS service address range, which defaults to 172.20.0.0/16. When creating a peering between networks, ensure that you don't overlap this range. Otherwise, the result will be a peering status of success, but the internal routing for kubernetes will ignore the whole range.

To create an IP allocation for EKS:

  1. Create a resource file and apply it using the CLI:

    wf apply -f FILENAME

    Here is an example for EKS: 

    ---
    apiVersion: networks.appvia.io/v1alpha1
    kind: AssignableNetwork
    metadata:
      name: eks
      namespace: ws-admin
    spec:
      provider: eks
      networks:
      - defaultMask: 21
        range: 10.20.0.0/16 # we should be able to fit 32 clusters in the range
        type: node
      - defaultMask: 18
        min: 21
        max: 24
        type: node

For more information, see IP Management.

Create a plan for private clusters​

This step is optional, but if skipped, it is the responsibility of the workspaces to ensure the clusters are defined as private, i.e., spec.enablePrivateCluster is set to true in yaml, or the Private cluster setting is enabled in the UI.

To create a plan for private clusters on EKS:

  1. Create a resource file, of kind: Plan, and apply it with the CLI command:

    wf apply -f FILENAME.

    Below is a sample plan. You can prevent workspaces from editing a parameter when creating a cluster by setting editable to false for that parameter.

    apiVersion: compute.appvia.io/v1alpha1
    kind: Plan
    metadata:
      name: eks-default
      namespace: ws-admin
    spec:
      allocation:
        type: all
      policies:
      - editable: false
        path: spec.networking.authorizedNetworks
      - editable: false
        path: spec.networking.authorizedMasterNetworks
      - editable: false
        path: spec.enablePrivateCluster
      - editable: false
        path: spec.networking.clusterIPV4Cidr
      summary: Default EKS Template
      template:
        addons:
          ingress:
            enabled: false
        enablePrivateCluster: true
        networking:
          authorizedMasterNetworks:
          - cidr: 0.0.0.0/0
            name: default
          authorizedNetworks:
          - cidr: 10.70.0.0/24
            name: ci
          - cidr: 10.80.0.0/24
            name: vpn
          clusterIPV4Cidr: auto
        nodePools:
        - autoscaling:
            enabled: true
            maxSize: 10
            minSize: 1
          diskSize: 30
          image: AL2_x86_64
          machine: t3.medium
          name: compute
          size: 2
          spot:
            enabled: true
        provider: EKS
        region: eu-west-2
        version: latest

For detailed instructions to create a cluster plan, see:

Create a peering connection from workspace clusters back to Wayfinder​

Wayfinder requires direct connectivity to the Kubernetes API. This is the cloud managed Kubernetes API, not the authentication module. Since we've just removed all public access to the endpoint we need to create a peering back to the management network, shown in the example below.

---
apiVersion: networking.appvia.io/v1alpha1
kind: PeeringRule
metadata:
  name: management
spec:
  filters:
    allocation:
      type: all
    selectors:
      matchExpressions:
        - key: networking.appvia.io/peering
          operator: In
          values: ["true"]
        - key: appvia.io/provider
          operator: In
          values: ["aws"]
  connection:
    type: peering
    peering:
      enableAutoApproval: true

Create the cloud account for VPN and CI​

For the purpose of this example, we can assume the VPN and CI system reside in the same AWS account, but in different networks. So we need a cloud account with the correct permissions to handle automatic acceptance.

note

If the networks resided in the same account as Wayfinder we could skip this step and reuse the ws-admin/management cloud account instead. Also, auto acceptance of external networks (CI and VPC under this example) is not a strict requirement. You can disable the feature and perform the necessary peering acceptance and routing change out-of-band. In this scenario the cluster(s) will be in a holding pattern waiting for the peering to be accepted.

To create the cloud account for VPN and CI:

  1. Create a cloud account using yaml, and apply the file using the Wayfinder CLI:

    wf apply -f FILENAME

    Here is a sample file. The important element to note here is the role required in spec.roles.

    ---
    apiVersion: cloudaccess.appvia.io/v1alpha1
    kind: CloudAccount
    metadata:
      name: shared-services
      namespace: ws-admin
    spec:
      accountType: shared
      allocation:
        type: none
      cloud: aws
      defaultRegion: eu-west-2
      features:
      - NetworkManager
      identifier: <AWS_ACCOUNT_ID>
      identityCred:
        name: management
        namespace: ws-admin
      name: shared
      orgIdentifier: ""
      providerDetails:
        type: AWSAccount
      roles:
      - role: NetworkManager
      stages:
      - nonprod
      - prod

    The above creates a cloud account linked to the identity that Wayfinder runs under. We don't have access yet.

  2. We need a user with permissions in the account to delegate the NetworkManager role to us.

    1. Switch into an AWS profile/session that has IAM access in the target account.
    2. Run the command: wf setup roles -w admin --cloud-account shared-services --feature NetworkManager.

    The required role is provisioned behind the scenes in the target account, and a trust policy is created between the account and the cloud managed identity Wayfinder is running under. (The cloud managed identity is provided by EKS pod identity service.)

Create the peering rules for CI and VPN​

The following example creates the peering rules necessary for CI and VPN.

To create peering rules for CI and VPN:

  1. Create resource files of kind: peeringRule, one for VPN and one for CI (see examples below). Then apply the files using the CLI:

    wf apply -f <FILENAME>

    Example peering rule for VPN:

    ---
    apiVersion: networking.appvia.io/v1alpha1
    kind: peeringRule
    metadata:
      name: vpn
    spec:
      filters:
        allocation:
          type: all
        selectors:
          matchExpressions:
            - key: appvia.io/provider
              operator: in
              values: ["aws"]

      # use the credential created above
      cloudAccount:
        name: shared-services
        namespace: ws-admin

      # define the connection to the vpc network
      connection:
        type: peering
        peering:
          enableAutoApproval: true
          network:
            account: <aws_account_id>
            identifier: <aws_vpc_id>
            location: <aws_region>
            routes:
              - 10.80.0.0/24
            routeTableSelectors:
              tag:name: vpn-main

    Example peering rule for CI:

    ---
    apiVersion: networking.appvia.io/v1alpha1
    kind: peeringRule
    metadata:
      name: ci
    spec:
      filters:
        allocation:
          type: all
        selectors:
          matchExpressions:
            - key: appvia.io/provider
              operator: in
              values: ["aws"]
            - key: networking.appvia.io/peering
              operator: in
              values: ["true"]

      # use the credential created above
      cloudAccount:
        name: shared-services
        namespace: ws-admin

      # define the connection to the vpc network - here we are not automatically
      # accepting the request, this must be done out-of-baud
      connection:
        type: peering
        peering:
          enableAutoApproval: false
          network:
            account: <aws_account_id>
            identifier: <aws_vpc_id>
            location: <aws_region>
            routes:
              - 10.70.0.0/24

Create a private cluster​

Once the steps above are complete, workspaces can create private clusters, using the cluster plan eks-default (a plan that has private cluster enabled):

Last updated on 28 Dec 2023