ACI Firewall Migration - Gateways to ACI¶
Estimated time to read: 22 minutes
- Originally Written: January, 2024
Info
This post uses the Nexus as Code (NaC) project which makes it very easy to configure ACI fabrics through a YAML file . More details and examples can be found at https://developer.cisco.com/docs/nexus-as-code/#!aci-introduction
You can find sample code in the following repository
https://github.com/conmurphy/aci-firewall-migration-scenarios
This post gives a basic example of migrating default gateways from a firewall to an ACI fabric while still using the firewall for security policies. The specific steps may differ depending on the environment.
The migration has five high level steps.
- Create a new interface and VRF for PBR
- Create a new interface and L3Out OSPF peering
- Create an Endpoint Security Group and contracts
- Migrate the gateways from the firewall to ACI, add the subnets to the ESG, advertise the migrated subnets from ACI
- Repeat step 4 until all subnets have been migrated
Note
Traffic continues to use the current firewall routing/bridging/security while the configuration in steps 1, 2, and 3 is implemented.
Benefits of moving to an L3 ACI Fabric¶
There are a number of benefits of migrating from an L2 ACI fabric to an L3 ACI fabric including:
- Distributed default gateway
- Rather than having traffic traverse the fabric and be routed at the firewall, ACI can provide a default gateway that spans all of the leaf switches assigned to the tenant. This means if a workload moves from one leaf to another the default gateway will still be available in the new leaf switch.
- Leverage solutions for their primary purposes
- You'll see later that Policy-based Redirect is used to send traffic to the firewall. This can be all traffic or alternatively just a selection of traffic (e.g. web). This allows you to use the fabric for hardware based routing and bridging while sending only the required traffic to the firewall for inspection and security policies.
- Easier to separate duties e.g. networking vs security
- Network admins can use ACI to setup the networking connectivity to the firewall while the security teams can separately manage firewall rules and policies.
- Roadmap to more granular application security policies
- The example design classifies all subnets in an Endpoint Security Group (ESG). ESGs allow network and security administrators to implement security based on applications rather than on network segments or IP addresses. This allows for simpler network designs as there is no longer a requirement to insert security devices into the routing path.
- Roadmap to application visibility
- ESGs allow network admins to understand where the application endpoint is connected.
Info
For more information on Endpoint Security Groups, their benefits, example designs, and how to implement ESGs see the following resources.
Cisco ACI Endpoint Security Group (ESG) Design Guide
ACI: the foundation of an internal private cloud – BRKDCN-2984
Current Deployment¶
In the current design the ACI fabric is in L2 (i.e. no unicast routing on the bridge domains). The firewall handles the security policy, subnet gateways, routing between subnets, and routing to the external network.
Setup Notes
- This setup may not exactly match your environment and is for example purposes only
- Always schedule and perform the tasks in a maintenance window
- A virtual FTD is used as the firewall
- In the legacy environment, one interface on the firewall maps to one subnet to simplify the setup. The endpoint VM interface (e.g.
192.168.100.10) is in the same portgroup as the legacy firewall gateway interface (e.g.192.168.100.1). - In this example there's a VM
10.10.103.10outside of the ACI fabric that is connected to the firewall to simulate external traffic.
Nexus as Code Configuration - legacy_subnets.nac.yaml¶
Key points about this design
- There are no contracts in the current design. All routing and security policies are applied on the firewall, not the ACI fabric.
- The ACI bridge domains have Unicast Routing disabled and flooding enabled
Here is the Nexus as Code configuration (legacy_subnets.nac.yaml) to configure the bridge domains and endpoint groups. It assume a tenant and VRF have already been configured.
Configuration
---
apic:
tenants:
- name: conmurph
managed: false
bridge_domains:
- name: 192.168.100.0_24
alias: legacy-subnet-1
vrf: vrf-01
unknown_unicast: flood
unicast_routing: false
- name: 192.168.101.0_24
alias: legacy-subnet-2
vrf: vrf-01
unknown_unicast: flood
unicast_routing: false
- name: 192.168.102.0_24
alias: legacy-subnet-3
vrf: vrf-01
unknown_unicast: flood
unicast_routing: false
application_profiles:
- name: network-segments
managed: false
endpoint_groups:
- name: 192.168.100.0_24
bridge_domain: 192.168.100.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
- name: 192.168.101.0_24
bridge_domain: 192.168.101.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
- name: 192.168.102.0_24
bridge_domain: 192.168.102.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
The decision is made to migrate the gateways from the firewalls to the ACI fabric, while still using the firewall for security policies and connectivity to the external networks. It will be a phased migration over a period of time, rather than a single migration of all gateways to ACI.
Step 1: PBR link and new VRF¶
There are a number of steps to perform before the gateways are migrated. Since it's a phased migration, the following connectivity needs to be provided.
- Migrated subnet to migrated subnet
- Migrated subnet to legacy subnet
- Migrated subnet to external subnet
- Legacy subnet to legacy subnet
- Legacy subnet to external subnet
As shown above, in the current setup legacy subnets communicate with each other through the firewall. Security policy is applied in the firewall if required.
When the routing is moved to the ACI fabric there needs to be a method to ensure that communication between two migrated subnets (now existing in ACI) still traverses the firewall to enforce security policies.
This is implemented using Policy-based Redirect. PBR provides many capabilities but in this example scenario, will redirect traffic between two endpoints or subnets to the firewall for inspection.
The configuration below uses a service bridge domain (i.e. another VLAN/subnet 6.6.6.0_24) in one-arm mode, although there are other methods to configure PBR depending on your requirements.
Since there is only a single arm back to the ACI fabric, a default route (0.0.0.0/0) is configured on the firewall to point to the service bridge domain IP (6.6.6.1/24).
Don't overlook the addition of a second VRF
A new VRF, vrf-pbr, is configured on the firewall. This VRF contains the PBR interface and the default route pointing to 6.6.6.1/24
The new VRF is used because the firewall may already contain a default route pointing to the external network.
Nexus as Code Configuration - pbr.nac.yaml¶
Key points about this step
- The legacy subnets are left in place for the time being to ensure traffic still flows
- The service graph will be deployed using one-arm mode with an Intra-ESG contract (PVLANs) at a later stage. Therefore the same configuration is used for the
providerandconsumerdevice selection policies. The contract,to-firewall-pbr, is selected in the service graph code below but the actual configuration will also be at a later stage.
Here is the Nexus as Code configuration (pbr.nac.yaml) to configure the service bridge domain and PBR service graph.
Note
The configuration below is using a virtual firewall with the VMM integration having already been setup
Configuration
---
apic:
tenants:
- name: conmurph
managed: false
bridge_domains:
- name: 6.6.6.0_24
alias: pbr_bd
vrf: vrf-01
subnets:
- ip: 6.6.6.1/24
application_profiles:
- name: network-segments
managed: false
endpoint_groups:
- name: 6.6.6.0_24
alias: pbr_bd
bridge_domain: 6.6.6.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
services:
service_graph_templates:
- name: conmurph-ftdv-routed-1
template_type: FW_ROUTED
redirect: true
device:
tenant: conmurph
name: conmurph-ftdv-routed-1
l4l7_devices:
- name: conmurph-ftdv-routed-1
context_aware: single-Context
type: VIRTUAL
vmware_vmm_domain: ucsc-c220m5-vds-01
function: GoTo
managed: false
service_type: FW
concrete_devices:
- name: conmurph-ftdv-routed-1
vcenter_name: <your-vcenter-name> # this is assuming a virtual firewall already setup
vm_name: conmurph-ftdv-routed-1 # name of the virtual firewall VM
interfaces:
- name: client
vnic_name: Network adapter 9 # network adapter on the VM which is used for PBR
logical_interfaces:
- name: client
concrete_interfaces:
- device: conmurph-ftdv-routed-1
interface_name: client
redirect_policies:
- name: client
l3_destinations:
- ip: 6.6.6.2
mac: 00:50:56:a1:2e:9b # MAC address of the network adapter 9 from above
device_selection_policies:
- contract: to-firewall-pbr
service_graph_template: conmurph-ftdv-routed-1
consumer:
l3_destination: true
redirect_policy:
name: client
logical_interface: client
bridge_domain:
name: 6.6.6.0_24
provider:
l3_destination: true
redirect_policy:
name: client
logical_interface: client
bridge_domain:
name: 6.6.6.0_24
Step 2: L3Out link¶
Recall that in the current design all routing is performed on the firewall. The firewall knows how to route to all legacy subnets and external networks. This is not the case once the migration takes place.
After a subnet is migrated, there needs to be a way for ACI to advertise to the firewall that it now knows about the migrated subnet. Likewise, the firewall needs to tell ACI about the legacy and external networks to which it still connects.
This is achieved through either static or dynamic route peering or in the case of ACI, an L3Out. Basic OSPF configuration is used for this example.
The L3Out will provide connectivity from the migrated subnets to the legacy and external subnets. Once you have applied the configuration you should confirm the neighbour adjacency is active.
Note
The legacy to legacy and legacy to external routing is still performed on the firewall as in the original design
Warning
Only the peering is setup at this stage. The advertisement of subnets will take place when the gateway is migrated
Nexus as Code Configuration - l3out.nac.yaml¶
Key points about this step
- The legacy subnets are left in place for the time being to ensure traffic still flow.
- Note that the configuration share previously and also below does not yet advertise the migrated routes. This will be configured at a later stage. Additionally, the legacy and external routes should not be advertised at this stage from the firewall.
- Note that in the configuration below the L3Out provides a contract,
to-firewall-l3out, but this does not currently exist. It will be created in the next step.
Here is the Nexus as Code configuration (l3out.nac.yaml) to configure the service bridge domain and PBR service graph.
Note
- The configuration below assumes an L3Out domain and VLAN pool have already been setup. In this example the domain is
conmurph.vrf-01 - This example was configured using a
floating SVI. For more information see the following post Basic L3Out with Floating SVI
Configuration
---
apic:
tenants:
- name: conmurph
managed: false
l3outs:
- name: floating-l3out-to-firewall
vrf: vrf-01
domain: conmurph.vrf-01
ospf:
area: 0
area_type: regular
node_profiles:
- name: border-leafs
nodes:
- node_id: 101
router_id: 101.2.1.1
- node_id: 102
router_id: 102.2.1.1
interface_profiles:
- name: ucsc-c220m5-vds-01
ospf:
policy: floating-l3out-to-firewall
interfaces: # floating SVI
- node_id: 101
vlan: 500
floating_svi: true
ip: 172.16.100.1/24
paths:
- vmware_vmm_domain: ucsc-c220m5-vds-01
floating_ip: 172.16.100.3/24
- node_id: 102
vlan: 500
floating_svi: true
ip: 172.16.100.2/24
paths:
- vmware_vmm_domain: ucsc-c220m5-vds-01
floating_ip: 172.16.100.3/24
external_endpoint_groups:
- name: all-ext-subnets
contracts:
providers:
- to-firewall-l3out
subnets:
- prefix: 0.0.0.0/1
- prefix: 128.0.0.0/1
policies:
ospf_interface_policies:
- name: floating-l3out-to-firewall
network_type: bcast
Step 3: Endpoint Security Groups and Contracts¶
This step will include two parts. First create the Endpoint Security Group (ESG) with Intra-ESG contract for the PBR connection and then add another contract from the ESG to L3Out.
Step 3a: Endpoint Security Group and Intra-ESG contract¶
It was previously mentioned that a PBR service graph will be used to redirect traffic between migrated subnets through the firewall. A service graph is attached at to a contract. This could be achieved by providing and consuming contracts between individual subnets, however the example in this post uses a different approach.
One catch-all Endpoint Security Group (ESG) will be configured, with each migrated subnet being mapped to this ESG. An Intra-ESG contract (with the PBR service graph attached) is then added to the ESG such that any traffic between endpoints in any of the migrated subnets or within a subnet itself would be sent to the firewall.
This simplifies the amount of configuration required to redirect migrated subnets.
Note
Although the example uses a single catch-all ESG, more granular ESGs could be added instead. Alternatively this could be configured at a later stage. See the ESG whitepaper and Cisco Live presentation above for further details
Info
The previously named legacy_subnets.nac.yaml configuration will be changed to legacy_subnets_esgs_contracts.nac.yaml to represent the new configuration (although it will still contain the legacy subnets). It will be updated to include the ESG configuration as well as the to-firewall-pbr and to-firewall-l3out contracts.
Step 3b: ESG to L3Out contract¶
Traffic between the migrated subnet and the legacy or external subnets will not use the PBR interface. It will use the L3Out. Therefore a contract needs to be added between the ESG and the L3Out to permit the traffic to flow.
Nexus as Code Configuration - legacy_subnets_esgs_contracts.nac.yaml¶
Key points about this step
- The legacy subnets are left in place for the time being to ensure traffic still flows
- The
epg_selectors(commented out) will be added in the next steps. This step is to first create the ESG before migrating the subnet intra_esg_isolation(Private VLAN) is enabled to force all traffic between endpoints in a subnet or between migrated subnets to be sent to the firewall for inspection.- The
to-firewall-pbrandto-firewall-l3outcontracts will permit the migrated subnets to the PBR bridge domain or to the L3Out
Here is the Nexus as Code configuration for the updated legacy_subnets_esgs_contracts.nac.yaml file to configure the ESGs and contracts.
Configuration
---
apic:
tenants:
- name: conmurph
managed: false
bridge_domains:
- name: 192.168.100.0_24
alias: legacy-subnet-1
vrf: vrf-01
unknown_unicast: flood
unicast_routing: false
- name: 192.168.101.0_24
alias: legacy-subnet-2
vrf: vrf-01
unknown_unicast: flood
unicast_routing: false
- name: 192.168.102.0_24
alias: legacy-subnet-3
vrf: vrf-01
unknown_unicast: flood
unicast_routing: false
application_profiles:
- name: network-segments
managed: false
endpoint_groups:
- name: 192.168.100.0_24
bridge_domain: 192.168.100.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
- name: 192.168.101.0_24
bridge_domain: 192.168.101.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
- name: 192.168.102.0_24
bridge_domain: 192.168.102.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
- name: network-security-groups
managed: false
endpoint_security_groups:
- name: migrated-subnets
vrf: vrf-01
intra_esg_isolation: true
contracts:
intra_esgs:
- to-firewall-pbr
consumers:
- to-firewall-l3out
# epg_selectors:
# - application_profile: network-segments
# endpoint_group: 192.168.100.0_24
# - application_profile: network-segments
# endpoint_group: 192.168.101.0_24
filters:
- name: icmp-src-any-to-dst
entries:
- name: src-any-to-dst
ethertype: ip
protocol: icmp
contracts:
- name: to-firewall-pbr
subjects:
- name: icmp
filters:
- filter: icmp-src-any-to-dst
service_graph: conmurph-ftdv-routed-1
- name: to-firewall-l3out
subjects:
- name: icmp
filters:
- filter: icmp-src-any-to-dst
Step 4: The migration¶
Up until this point the legacy gateways have remained on the firewall and traffic has continued to flow. With PBR, the L3Out, the ESG, and contracts in place, it's now time to migrate the subnet gateways to the ACI fabric.
There are three high level steps for this process in no specific order.
- Swap subnet gateway to ACI
- Add subnet to ESG
- Advertise subnets between ACI and FW
The Nexus as Code Configuration will be shared at the end of the three steps.
Step 4a: Swap subnet gateway to ACI¶
This removes the gateway of one or more subnets from the firewall and configures it in ACI. In this example 192.168.100.0 and 192.168.101.0 will be migrated.
Key points about this step
- Unicast routing will be enabled on the migrated subnets
- The gateway IP which was on the firewall interface is now configured on the ACI bridge domain
Step 4b: Add EPG (i.e. subnet) to ESG¶
Previously a catch-all ESG was created. This step now maps the EPG i.e. the subnet, into the ESG
Step 4c: Advertise subnets between ACI and FW¶
Now that the gateway has moved you can advertise the migrated subnet from ACI to the firewall and the legacy or external subnets from the firewall to ACI.
Key points about this step
- Subnets are advertised in ACI by checking the "Advertise Externally" checkbox in the UI or using the
publicflag in Nexus as Code - Additionally you need to associate the bridge domain to the L3Out
Nexus as Code - subnets_esgs_contracts.nac.yaml and Firewall OSPF¶
Info
The previously named legacy_subnets_esgs_contracts.nac.yaml configuration file will be changed to subnets_esgs_contracts.nac.yaml to represent the new configuration. It will be updated with the gateways, EPG to ESG mapping, and the subnet advertisement configuration.
Configuration
---
apic:
tenants:
- name: conmurph
managed: false
bridge_domains:
- name: 192.168.100.0_24
alias: migrated-subnet-1
vrf: vrf-01
subnets:
- ip: 192.168.100.1
public: true
l3outs:
- floating-l3out-to-firewall
- name: 192.168.101.0_24
alias: migrated-subnet-2
vrf: vrf-01
subnets:
- ip: 192.168.101.1
public: true
l3outs:
- floating-l3out-to-firewall
- name: 192.168.102.0_24
alias: legacy-subnet-3
vrf: vrf-01
unknown_unicast: flood
unicast_routing: false
application_profiles:
- name: network-segments
managed: false
endpoint_groups:
- name: 192.168.100.0_24
bridge_domain: 192.168.100.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
u_segmentation: true
- name: 192.168.101.0_24
bridge_domain: 192.168.101.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
u_segmentation: true
- name: 192.168.102.0_24
bridge_domain: 192.168.102.0_24
vmware_vmm_domains:
- name: ucsc-c220m5-vds-01
deployment_immediacy: immediate
resolution_immediacy: immediate
- name: network-security-groups
managed: false
endpoint_security_groups:
- name: migrated-subnets
vrf: vrf-01
intra_esg_isolation: true
contracts:
intra_esgs:
- to-firewall-pbr
consumers:
- to-firewall-l3out
epg_selectors:
- application_profile: network-segments
endpoint_group: 192.168.100.0_24
- application_profile: network-segments
endpoint_group: 192.168.101.0_24
filters:
- name: icmp-src-any-to-dst
entries:
- name: src-any-to-dst
ethertype: ip
protocol: icmp
contracts:
- name: to-firewall-pbr
subjects:
- name: icmp
filters:
- filter: icmp-src-any-to-dst
service_graph: conmurph-ftdv-routed-1
- name: to-firewall-l3out
subjects:
- name: icmp
filters:
- filter: icmp-src-any-to-dst
Step n: Continue until all subnets migrated¶
You can continue with the same process in steps 4a, 4b, and 4c to migrate the remaining subnets and gateways.
Packet Flows¶
The following section is to confirm the configuration is setup correctly as well as high level diagrams for the various packet flows.
Info
-
The route table on the left-hand side is for the firewall
defaultVRF. This provides connectivity to the migrated, legacy, and external networks (via default route0.0.0.0/0). -
The route table on the right-hand side is for the firewall
pbrVRF. Since there is only a single interface in this VRF and therefore only one way in and out, there is a single default route pointing back to the ACI fabric. i.e. once the traffic is sent to the firewall, to get anywhere else it must go back to the ACI fabric. As previously mentioned, the separate VRFs allow two default routes. One pointing back to the ACI fabric for PBR traffic, and one pointing to the external network for any traffic needing to egress the fabric. -
The third route table is from one of the ACI leaf switches,
aci-dev-01-leaf-102
Migrated subnet to migrated subnet¶
Migrated subnet to legacy subnet¶
Migrated subnet to external subnet¶
Legacy subnet to external subnet¶
