Building a Cisco Secure AI Factory¶
Estimated time to read: 30 minutes
- Originally Written: May, 2026
Overview¶
For the last 6 months I've been working with colleagues to build an internal lab that can be used by our data center teams for customer demos, proof of concepts, training, and testing of new product features.
As you can imagine, much of the focus has been on AI/ML and GPU workloads so I'm documenting some of the design and setup of the "Secure AI Factory" component for future reference. I plan on writing a more comprehensive post on the rest of the lab setup including the overall requirements, planning, design, implementation and things we learnt along the way.
Note
We built the lab to our requirements and in some cases, due to infrastructure availablilty and the knowledge that it's a lab not a production environment, consolidated components or modified them to meet our needs. It's always best to follow the release notes, install guides, and validated designs if this is for a production environment.
This is the general product stack and if you want to find out more on any of the products have a look at https://www.cisco.com/site/us/en/solutions/artificial-intelligence/secure-ai-factory/index.html.
Considerations¶
There are two main constraints we had when designing the lab.
- It needs to support some long running demos which also require GPU resources
- It's a shared environment for ~100 people to use. Although we don't expect that many parallel users we need as much multi-tenancy as possible so someone can build/train/test without breaking someone elses experience
Environment setup¶
This is a high level diagram of the infrastructure layout and as mentioned above we built some things differently than you would in a prod network because of constraints. It's also an evolving environment so we plan to improve it in the future.
Here's a list of the hardware/software we currently have and some of the configuration.
Openshift¶
We have two Openshift clusters, one on UCS-X and the other with a C845A. The UCS-X contains 2 x AMD Mi210s in one compute module and 2 x Nvidia H100s in the other compute module. We have a third compute module for additional CPU/memory. The second cluster has 8 x Nvidia H200 NVLs in the 845A and two additional compute modules from the same UCS-X chassis. These additional servers are also for additional CPU/memory resources; I originally tried to build a single node cluster but I quickly hit the 250 default pod limit once the Openshift, Openshift AI, Run:ai backend, and Run:ai cluster system pods were installed.
All nodes in both clusters have control and worker node roles to keep the resource requirements down. We may rebuild with dedicated control plane nodes in the future if we hit scaling issues
We are trying to automate as much as possible so we used the iserver tool from one of our colleagues. This helped us to provision both Openshift clusters from a config file, including validating the image, the network settings, and converted from the Openshift OVN CNI to the Cilium CNI as part of the install. It also means we can redeploy the cluster just by re-running iserver with our config file.
Each team receives their own Openshift project/namespace in each cluster and we have one for our "shared models" (more on this later). RBAC is tied to Active Directory and each group is mapped to their respective namespace.
Intersight and UCS¶
The UCS-X and C845A in these pods are connected to Cisco Intersight along with all other rack servers and blade servers in our lab. The servers were deployed from a service profile template so made the initial setup pretty quick. To expand the second cluster with an additional two servers I just needed to provision a new service profile from the template and assign them to the blades. Since we're using SaaS we also have it linked to our Service Contract and we get Proactive RMAs for any component issues.
ACI for the frontend¶
ACI was chosen because of the strong multi-tenancy and automation capabilities. Each team receives their own tenant so they can configure the network for their use case in isolation. The user logins are tied to AD and users receives read/write access to their respective tenant and read only to the entire fabric. The Openshift clusters ecxist in the shared-services tenant.
The entire fabric was deployed using Network as Code.
Each tenant has a separate configuration repo, we have one for access policies, and one for fabric policies. See https://github.com/spsharman/aci-dev-01 for an example of what this structure looks like.The tenant configuration will be covered in a different post but below is an example of the configuration used for the connections to the Openshift clusters.
Both clusters have Cilium install and we used BGP to peer with the fabric (ACI L3Out). Since we know which IPs are used by various Openshift namespaces (see the Cilium section below), we can assign those subnets to their own security groups (e.g. look for ocpai-01-ns-demo-video-streaming-platform in the config below). We can then apply a contract to these security groups to provide more granular segmentation policies, including redirect traffic using this contract through a firewall (look for service_graph: infra-ftdv-01).
This gives the network/security admins visibility and security of the service/pod traffic.
We also use two VRFs (vrf-external and vrf-internal) so we can control which routes are leaked/learned from the external network.
There's a great eBook, Designing Isovalent Enterprise for Cisco ACI & Nexus which includes this design and more details.
Example config for shared-services tenant
---
apic:
tenants:
- name: shared-services
managed: false
security_domains:
- shared-services
match_rules:
- name: 198.18.193.240_28
description: shared-firewall-services
prefixes:
- ip: 198.18.193.240/28
- name: 198.18.193.0_25
description: shared-services
prefixes:
- ip: 198.18.193.0/25
- name: 198.18.194.0_25
description: shared-services
prefixes:
- ip: 198.18.194.0/25
- name: 198.18.194.192_27
description: openshift-ai-01
prefixes:
- ip: 198.18.194.192/27
- name: 198.18.194.224_27
description: openshift-ai-02
prefixes:
- ip: 198.18.194.224/27
- name: 198.18.200.0_28
description: static-demo-video-analytics-cilium ocpai-01
prefixes:
- ip: 198.18.200.0/28
- name: 198.18.200.16_28
description: shared-models ocpai-01
prefixes:
- ip: 198.18.200.16/28
vrfs:
- name: vrf-external
leaked_external_prefixes:
- prefix: 0.0.0.0/0
destinations:
- tenant: shared-services
vrf: vrf-internal
- name: vrf-internal
leaked_internal_prefixes:
- prefix: 198.18.193.240/28 # shared firewall services
public: true
destinations:
- {tenant: shared-services, vrf: vrf-external, public: true}
- prefix: 198.18.193.0/25 # shared services
public: true
destinations:
- {tenant: shared-services, vrf: vrf-external, public: true}
- prefix: 198.18.194.0/25 # shared services
public: true
destinations:
- {tenant: shared-services, vrf: vrf-external, public: true}
leaked_external_prefixes:
- prefix: 198.18.194.192/27 # openshift cluster 01 nodes
to_prefix_length: 32
destinations:
- {tenant: shared-services, vrf: vrf-external}
- prefix: 198.18.200.0/28 # video analytics cilium demo - openshift cluster 01 nodes
to_prefix_length: 32
destinations:
- {tenant: shared-services, vrf: vrf-external}
- prefix: 198.18.200.16/28 # shared models ns - openshift cluster 01 nodes
to_prefix_length: 32
destinations:
- {tenant: shared-services, vrf: vrf-external}
- prefix: 198.18.194.224/27 # openshift cluster 02 nodes
to_prefix_length: 32
destinations:
- {tenant: shared-services, vrf: vrf-external}
bridge_domains:
- name: 198.18.193.240_28
alias: shared-firewall-services
vrf: vrf-internal
subnets:
- ip: 198.18.193.241/28
ip_dataplane_learning: false
- name: 198.18.193.0_25
alias: shared-services
vrf: vrf-internal
subnets:
- ip: 198.18.193.1/25
- name: 198.18.194.0_25
alias: shared-services
vrf: vrf-internal
subnets:
- ip: 198.18.194.1/25
application_profiles:
- name: external-subnets
endpoint_security_groups:
- name: external-subnets
vrf: vrf-external
ip_external_subnet_selectors:
- ip: 0.0.0.0/1
shared: true # leaks pc tag into vrf-internal
- ip: 128.0.0.0/1
shared: true # leaks pc tag into vrf-internal
contracts:
consumers:
- permit-to-tn-shared-services
- permit-to-ocpai-01-nodes
- permit-to-ocpai-01-ns-demo-video-streaming-platform
- permit-to-ocpai-01-ns-shared-models
- permit-to-ocpai-02-nodes
- name: network-segments
endpoint_security_groups:
- name: all-shared-services
vrf: vrf-internal
epg_selectors:
- application_profile: network-segments
description: shared-services
endpoint_group: 198.18.193.0_25
- application_profile: network-segments
description: shared-services
endpoint_group: 198.18.194.0_25
- application_profile: network-segments
endpoint_group: 198.18.194.160_27
- application_profile: network-segments
endpoint_group: 198.18.194.128_28
- application_profile: network-segments
endpoint_group: 198.18.194.144_28
contracts:
providers:
- permit-to-tn-shared-services
consumers:
- permit-to-ocpai-01-nodes
- permit-to-ocpai-02-nodes
- permit-to-ocpai-01-ns-shared-models
- permit-to-ocpai-01-ns-demo-video-streaming-platform
- name: ocpai-01-nodes
vrf: vrf-internal
ip_external_subnet_selectors:
- ip: 198.18.194.192/27
shared: true
contracts:
providers:
- permit-to-ocpai-01-nodes
consumers:
- permit-to-tn-shared-services
- permit-to-ocpai-01-ns-demo-video-streaming-platform
- permit-to-ocpai-01-ns-shared-models
- permit-to-ocpai-02-nodes
- name: ocpai-01-ns-demo-video-streaming-platform
vrf: vrf-internal
ip_external_subnet_selectors:
- ip: 198.18.200.0/28
shared: true
contracts:
providers:
- permit-to-ocpai-01-ns-demo-video-streaming-platform
consumers:
- permit-to-tn-shared-services
- permit-to-ocpai-01-nodes
- permit-to-ocpai-01-ns-shared-models
- permit-to-ocpai-02-nodes
- name: ocpai-01-ns-shared-models
vrf: vrf-internal
ip_external_subnet_selectors:
- ip: 198.18.200.16/28
shared: true
contracts:
providers:
- permit-to-ocpai-01-ns-shared-models
consumers:
- permit-to-tn-shared-services
- permit-to-ocpai-01-nodes
- permit-to-ocpai-02-nodes
- permit-to-ocpai-01-ns-demo-video-streaming-platform
- name: ocpai-02-nodes
vrf: vrf-internal
ip_external_subnet_selectors:
- ip: 198.18.194.224/27
shared: true
contracts:
providers:
- permit-to-ocpai-02-nodes
consumers:
- permit-to-tn-shared-services
- permit-to-ocpai-01-nodes
- permit-to-ocpai-01-ns-shared-models
- permit-to-ocpai-01-ns-demo-video-streaming-platform
endpoint_groups:
- name: 198.18.193.0_25
alias: shared-services
bridge_domain: 198.18.193.0_25
physical_domains:
- shared-services.phys-dom
static_ports:
- node_id: 1101
node2_id: 1102
channel: vpc-ucs-test-01-6536-A
vlan: 193
deployment_immediacy: immediate
- node_id: 1101
node2_id: 1102
channel: vpc-ucs-test-01-6536-B
vlan: 193
deployment_immediacy: immediate
- node_id: 1101
node2_id: 1102
channel: vpc-ucs-prod-01-6454-A
vlan: 193
deployment_immediacy: immediate
- name: 198.18.194.0_25
alias: shared-services
bridge_domain: 198.18.194.0_25
physical_domains:
- shared-services.phys-dom
static_ports:
- node_id: 1101
node2_id: 1102
channel: vpc-ucs-test-01-6536-A
vlan: 194
deployment_immediacy: immediate
- node_id: 1101
node2_id: 1102
channel: vpc-ucs-test-01-6536-B
vlan: 194
deployment_immediacy: immediate
filters:
- name: icmp
entries:
- name: icmp
ethertype: ip
protocol: icmp
- name: web
entries:
- name: http
ethertype: ip
protocol: tcp
destination_from_port: http
destination_to_port: http
- name: all-protocols
entries:
- name: any
ethertype: unspecified
contracts:
- name: permit-to-tn-shared-services
scope: tenant
subjects:
- name: permit-any
filters:
- filter: all-protocols
log: true
- name: permit-to-ocpai-01-ns-demo-video-streaming-platform
scope: tenant
subjects:
- name: permit-any
filters:
- filter: all-protocols
log: true
service_graph: infra-ftdv-01
- name: permit-to-ocpai-01-ns-shared-models
scope: tenant
subjects:
- name: permit-any
filters:
- filter: all-protocols
log: true
#service_graph: infra-ftdv-01
- name: permit-to-ocpai-02-ns-shared-models
scope: tenant
subjects:
- name: permit-any
filters:
- filter: all-protocols
log: true
- name: permit-to-ocpai-01-nodes
scope: tenant
subjects:
- name: permit-any
filters:
- filter: all-protocols
log: true
- name: permit-to-ocpai-02-nodes
scope: tenant
subjects:
- name: permit-any
filters:
- filter: all-protocols
log: true
l3outs:
- name: l3out-to-core-01
vrf: vrf-external
domain: shared-services.routed-dom
bgp:
name: bgp-l3out-to-core
node_profiles:
- name: border-leafs
nodes:
- node_id: 1101
router_id: 101.2.1.1
router_id_as_loopback: false
- node_id: 1102
router_id: 102.2.1.1
router_id_as_loopback: false
interface_profiles:
- name: aci-site-01-pod-01-leaf-switches
interfaces:
- node_id: 1101
node2_id: 1102
channel: vpc-routed-interfaces-to-core
vlan: 94
svi: true
ip_a: 172.16.1.186/29
ip_b: 172.16.1.187/29
ip_shared: 172.16.1.189/29
bgp_peers:
- ip: 172.16.1.185/29
local_as: 65510
remote_as: 65500
external_endpoint_groups:
- name: all-subnets # still need this otherwise the l3out config isn't pushed
export_route_map:
name: default-export
type: global
contexts:
- name: 198.18.193.0_25 # aci shared-services
match_rules:
- 198.18.193.0_25
- name: 198.18.193.240_28 # aci shared-firewall-services
match_rules:
- 198.18.193.240_28
- name: 198.18.194.0_25 # aci shared-services
match_rules:
- 198.18.194.0_25
- name: 198.18.194.192_27 # ocpai-01 cluster nodes
match_rules:
- 198.18.194.192_27
- name: 198.18.194.224_27 # ocpai-02 cluster nodes
match_rules:
- 198.18.194.224_27
- name: 198.18.200.0_28 # static-demo-video-analytics-cilium ocpai-01 cluster
match_rules:
- 198.18.200.0_28
- name: 198.18.200.16_28 # shared-models ocpai-01 cluster
match_rules:
- 198.18.200.16_28
- name: l3out-to-ocp-ai-01
vrf: vrf-internal
domain: shared-services.routed-dom
bgp:
name: bgp-l3out-to-ocp-ai-01
node_profiles:
- name: border-leafs
nodes:
- node_id: 1101
router_id: 101.2.1.1
router_id_as_loopback: false
- node_id: 1102
router_id: 102.2.1.1
router_id_as_loopback: false
interface_profiles:
- name: aci-site-01-pod-01-leaf-switches
interfaces:
- node_id: 1101
node2_id: 1102
channel: vpc-ucs-test-01-6536-A
vlan: 301
svi: true
ip_a: 198.18.194.194/27
ip_b: 198.18.194.195/27
ip_shared: 198.18.194.193/27
bgp_peers:
- ip: 198.18.194.192/27
local_as: 65551
remote_as: 65552
- node_id: 1101
node2_id: 1102
channel: vpc-ucs-test-01-6536-B
vlan: 301
svi: true
ip_a: 198.18.194.194/27
ip_b: 198.18.194.195/27
ip_shared: 198.18.194.193/27
bgp_peers:
- ip: 198.18.194.192/27
local_as: 65551
remote_as: 65552
external_endpoint_groups:
- name: ocp-ai-01-nodes # still need this otherwise the l3out config isn't pushed
- name: l3out-to-ocp-ai-02
vrf: vrf-internal
domain: shared-services.routed-dom
bgp:
name: bgp-l3out-to-ocp-ai-02
node_profiles:
- name: border-leafs
nodes:
- node_id: 1101
router_id: 101.2.1.1
router_id_as_loopback: false
- node_id: 1102
router_id: 102.2.1.1
router_id_as_loopback: false
interface_profiles:
- name: aci-site-01-pod-01-leaf-switches
interfaces:
- node_id: 1101
mode: regular
floating_svi: true
vlan: 302
paths:
- floating_ip: 198.18.194.228/27
physical_domain: shared-services.phys-dom
ip: 198.18.194.226/27
ip_shared: 198.18.194.225/27 # node default gateway
bgp_peers:
- ip: 198.18.194.224/27
local_as: 65551
remote_as: 65553
- node_id: 1102
mode: regular
floating_svi: true
vlan: 302
paths:
- floating_ip: 198.18.194.228/27
physical_domain: shared-services.phys-dom
ip: 198.18.194.227/27
ip_shared: 198.18.194.225/27 # node default gateway
bgp_peers:
- ip: 198.18.194.224/27
local_as: 65551
remote_as: 65553
external_endpoint_groups:
- name: ocp-ai-02-nodes # still need this otherwise the l3out config isn't pushed
GPU Software Deployment¶
Now that I've covered the networking and compute foundation we should look at the AI/ML demo use cases and some of the software. running on Openshift.
Coming from a networking/infrastructure background, the last couple of months have involved a lot of learning as I worked through some of the software we needed to install in the cluster.
We have access to NVIDIA AI Enterprise software (and will deploy the equivalent AMD AI Enterprise in the future) and also the H100s/H200s. There were a few components to install before the GPUs could be used with our demo workloads. This included the Node Feature Discovery and the NVIDIA Operators.
Example GPU Setup Manifest
# This manifest deploys the NVIDIA GPU Operator on OpenShift from scratch,
# including namespaces, operator groups, subscription, and a fully-detailed
# ClusterPolicy.
#
# ---------------------------------------------------------
# 1. Node Feature Discovery (NFD) Namespace & OperatorGroup
# ---------------------------------------------------------
apiVersion: v1
kind: Namespace
metadata:
name: openshift-nfd
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: openshift-nfd
namespace: openshift-nfd
spec:
targetNamespaces:
- openshift-nfd
---
# ---------------------------------------------------------
# NFD Operator Subscription
# ---------------------------------------------------------
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: nfd
namespace: openshift-nfd
spec:
channel: "stable"
name: nfd
source: redhat-operators
sourceNamespace: openshift-marketplace
installPlanApproval: Automatic
---
# ---------------------------------------------------------
# NodeFeatureDiscovery CR
# ---------------------------------------------------------
apiVersion: nfd.openshift.io/v1
kind: NodeFeatureDiscovery
metadata:
name: nfd-instance
namespace: openshift-nfd
spec:
operand:
servicePort: 12000
workerConfig:
configData: |
sources:
pci:
deviceClassWhitelist:
- "0200"
- "03"
- "0300"
- "0302"
- "0380"
deviceLabelFields:
- "vendor"
---
# ---------------------------------------------------------
# 2. NVIDIA GPU Operator Namespace & OperatorGroup
# ---------------------------------------------------------
apiVersion: v1
kind: Namespace
metadata:
name: gpu-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: nvidia-gpu-operator
namespace: gpu-operator
spec:
targetNamespaces:
- gpu-operator
---
# ---------------------------------------------------------
# 3. Operator Subscription
# ---------------------------------------------------------
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: gpu-operator-certified
namespace: gpu-operator
spec:
channel: "v25.10"
name: gpu-operator-certified
source: certified-operators
sourceNamespace: openshift-marketplace
installPlanApproval: Automatic
startingCSV: "gpu-operator-certified.v25.10.1"
---
# ---------------------------------------------------------
# 4. ClusterPolicy — Full Configuration
# ---------------------------------------------------------
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
name: gpu-cluster-policy
spec:
# ----------------------------
# Operator-level settings
# ----------------------------
operator:
defaultRuntime: crio
runtimeClass: nvidia
use_ocp_driver_toolkit: true
initContainer: {}
# ----------------------------
# CDI (Container Device Interface)
# ----------------------------
cdi:
default: false
enabled: false
# ----------------------------
# Daemonset update strategy
# ----------------------------
daemonsets:
rollingUpdate:
maxUnavailable: "1"
updateStrategy: RollingUpdate
# ----------------------------
# GPU Driver
# ----------------------------
driver:
enabled: true
kernelModuleType: auto
useNvidiaDriverCRD: false
certConfig:
name: ""
kernelModuleConfig:
name: ""
licensingConfig:
nlsEnabled: true
secretName: ""
repoConfig:
configMapName: ""
virtualTopology:
config: ""
upgradePolicy:
autoUpgrade: true
maxParallelUpgrades: 1
maxUnavailable: 25%
drain:
enable: true
force: true
deleteEmptyDir: true
timeoutSeconds: 300
podDeletion:
force: true
deleteEmptyDir: true
timeoutSeconds: 300
waitForCompletion:
timeoutSeconds: 0
# ----------------------------
# Container Toolkit
# ----------------------------
toolkit:
enabled: true
installDir: /usr/local/nvidia
# ----------------------------
# Device Plugin
# ----------------------------
devicePlugin:
enabled: true
config:
default: ""
name: ""
mps:
root: /run/nvidia/mps
# ----------------------------
# DCGM (Data Center GPU Manager)
# ----------------------------
dcgm:
enabled: true
# ----------------------------
# DCGM Exporter (metrics)
# ----------------------------
dcgmExporter:
enabled: true
config:
name: ""
serviceMonitor:
enabled: true
# ----------------------------
# GPU Feature Discovery
# ----------------------------
gfd:
enabled: true
# ----------------------------
# MIG (Multi-Instance GPU)
# ----------------------------
mig:
strategy: single
migManager:
enabled: true
# ----------------------------
# Node Status Exporter
# ----------------------------
nodeStatusExporter:
enabled: true
# ----------------------------
# GDRCopy
# ----------------------------
gdrcopy:
enabled: false
# ----------------------------
# GPU Direct Storage (GDS)
# ----------------------------
gds:
enabled: false
# ----------------------------
# Sandbox / Virtualization
# ----------------------------
sandboxDevicePlugin:
enabled: true
sandboxWorkloads:
defaultWorkload: container
enabled: false
# ----------------------------
# VFIO Manager
# ----------------------------
vfioManager:
enabled: true
# ----------------------------
# vGPU
# ----------------------------
vgpuManager:
enabled: false
vgpuDeviceManager:
enabled: true
# ----------------------------
# Validator
# ----------------------------
validator:
plugin:
env: []
When this is working correctly you should see pods in the openshift-nfd namespace and node labels with various features will be assigned.
~# oc get pods -n openshift-nfd
NAME READY STATUS RESTARTS AGE
nfd-controller-manager-9b6d8bccf-tr9g4 1/1 Running 0 30h
nfd-gc-64d68c6fcf-xfqxl 1/1 Running 0 29h
nfd-master-7977c86865-wsvlt 1/1 Running 0 29h
nfd-worker-5fjlh 1/1 Running 0 29h
nfd-worker-qlszh 1/1 Running 0 29h
nfd-worker-zcmqf 1/1 Running 0 29h
Node features
root@openshift-installer:~# oc get nodes
NAME STATUS ROLES AGE VERSION
ucs-test-01-6536-1-1 Ready control-plane,master,worker 82d v1.34.2
ucs-test-01-6536-1-3 Ready control-plane,master,worker 82d v1.34.2
ucs-test-01-6536-1-5 Ready control-plane,master,worker 82d v1.34.2
root@openshift-installer:~# oc describe node ucs-test-01-6536-1-3
Name: ucs-test-01-6536-1-3
Roles: control-plane,master,worker
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
feature.node.kubernetes.io/cpu-cpuid.ADX=true
feature.node.kubernetes.io/cpu-cpuid.AESNI=true
feature.node.kubernetes.io/cpu-cpuid.AMXBF16=true
feature.node.kubernetes.io/cpu-cpuid.AMXFP16=true
feature.node.kubernetes.io/cpu-cpuid.AMXFP8=true
feature.node.kubernetes.io/cpu-cpuid.AMXINT8=true
feature.node.kubernetes.io/cpu-cpuid.AMXTILE=true
feature.node.kubernetes.io/cpu-cpuid.AVX=true
feature.node.kubernetes.io/cpu-cpuid.AVX10_128=true
feature.node.kubernetes.io/cpu-cpuid.AVX10_256=true
feature.node.kubernetes.io/cpu-cpuid.AVX10_512=true
feature.node.kubernetes.io/cpu-cpuid.AVX10_VERSION=1
feature.node.kubernetes.io/cpu-cpuid.AVX2=true
feature.node.kubernetes.io/cpu-cpuid.AVX512BF16=true
feature.node.kubernetes.io/cpu-cpuid.AVX512BITALG=true
feature.node.kubernetes.io/cpu-cpuid.AVX512BW=true
feature.node.kubernetes.io/cpu-cpuid.AVX512CD=true
feature.node.kubernetes.io/cpu-cpuid.AVX512DQ=true
feature.node.kubernetes.io/cpu-cpuid.AVX512F=true
feature.node.kubernetes.io/cpu-cpuid.AVX512FP16=true
feature.node.kubernetes.io/cpu-cpuid.AVX512IFMA=true
feature.node.kubernetes.io/cpu-cpuid.AVX512VBMI=true
feature.node.kubernetes.io/cpu-cpuid.AVX512VBMI2=true
feature.node.kubernetes.io/cpu-cpuid.AVX512VL=true
feature.node.kubernetes.io/cpu-cpuid.AVX512VNNI=true
feature.node.kubernetes.io/cpu-cpuid.AVX512VPOPCNTDQ=true
feature.node.kubernetes.io/cpu-cpuid.AVXVNNI=true
feature.node.kubernetes.io/cpu-cpuid.BHI_CTRL=true
feature.node.kubernetes.io/cpu-cpuid.CETIBT=true
feature.node.kubernetes.io/cpu-cpuid.CETSS=true
feature.node.kubernetes.io/cpu-cpuid.CLDEMOTE=true
feature.node.kubernetes.io/cpu-cpuid.CMPSB_SCADBS_SHORT=true
feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8=true
feature.node.kubernetes.io/cpu-cpuid.ENQCMD=true
feature.node.kubernetes.io/cpu-cpuid.FLUSH_L1D=true
feature.node.kubernetes.io/cpu-cpuid.FMA3=true
feature.node.kubernetes.io/cpu-cpuid.FSRM=true
feature.node.kubernetes.io/cpu-cpuid.FXSR=true
feature.node.kubernetes.io/cpu-cpuid.FXSROPT=true
feature.node.kubernetes.io/cpu-cpuid.GFNI=true
feature.node.kubernetes.io/cpu-cpuid.IA32_ARCH_CAP=true
feature.node.kubernetes.io/cpu-cpuid.IA32_CORE_CAP=true
feature.node.kubernetes.io/cpu-cpuid.IBPB=true
feature.node.kubernetes.io/cpu-cpuid.IDPRED_CTRL=true
feature.node.kubernetes.io/cpu-cpuid.LAHF=true
feature.node.kubernetes.io/cpu-cpuid.MCDT_NO=true
feature.node.kubernetes.io/cpu-cpuid.MD_CLEAR=true
feature.node.kubernetes.io/cpu-cpuid.MOVBE=true
feature.node.kubernetes.io/cpu-cpuid.MOVDIR64B=true
feature.node.kubernetes.io/cpu-cpuid.MOVDIRI=true
feature.node.kubernetes.io/cpu-cpuid.MOVSB_ZL=true
feature.node.kubernetes.io/cpu-cpuid.OSXSAVE=true
feature.node.kubernetes.io/cpu-cpuid.PCONFIG=true
feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_CYCLES=true
feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_INSTRUCTIONS=true
feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_REFCYCLES=true
feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_TOPDOWN_SLOTS=true
feature.node.kubernetes.io/cpu-cpuid.PREFETCHI=true
feature.node.kubernetes.io/cpu-cpuid.PSFD=true
feature.node.kubernetes.io/cpu-cpuid.RRSBA_CTRL=true
feature.node.kubernetes.io/cpu-cpuid.SERIALIZE=true
feature.node.kubernetes.io/cpu-cpuid.SHA=true
feature.node.kubernetes.io/cpu-cpuid.SPEC_CTRL_SSBD=true
feature.node.kubernetes.io/cpu-cpuid.STIBP=true
feature.node.kubernetes.io/cpu-cpuid.STOSB_SHORT=true
feature.node.kubernetes.io/cpu-cpuid.SYSCALL=true
feature.node.kubernetes.io/cpu-cpuid.SYSEE=true
feature.node.kubernetes.io/cpu-cpuid.TME=true
feature.node.kubernetes.io/cpu-cpuid.TSXLDTRK=true
feature.node.kubernetes.io/cpu-cpuid.VAES=true
feature.node.kubernetes.io/cpu-cpuid.VMX=true
feature.node.kubernetes.io/cpu-cpuid.VPCLMULQDQ=true
feature.node.kubernetes.io/cpu-cpuid.WAITPKG=true
feature.node.kubernetes.io/cpu-cpuid.WBNOINVD=true
feature.node.kubernetes.io/cpu-cpuid.X87=true
feature.node.kubernetes.io/cpu-cpuid.XGETBV1=true
feature.node.kubernetes.io/cpu-cpuid.XSAVE=true
feature.node.kubernetes.io/cpu-cpuid.XSAVEC=true
feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT=true
feature.node.kubernetes.io/cpu-cpuid.XSAVES=true
feature.node.kubernetes.io/cpu-cstate.enabled=true
feature.node.kubernetes.io/cpu-hardware_multithreading=true
feature.node.kubernetes.io/cpu-model.family=6
feature.node.kubernetes.io/cpu-model.id=173
feature.node.kubernetes.io/cpu-model.vendor_id=Intel
feature.node.kubernetes.io/cpu-pstate.scaling_governor=performance
feature.node.kubernetes.io/cpu-pstate.status=active
feature.node.kubernetes.io/cpu-pstate.turbo=true
feature.node.kubernetes.io/kernel-config.NO_HZ=true
feature.node.kubernetes.io/kernel-config.NO_HZ_FULL=true
feature.node.kubernetes.io/kernel-selinux.enabled=true
feature.node.kubernetes.io/kernel-version.full=5.14.0-570.78.1.el9_6.x86_64
feature.node.kubernetes.io/kernel-version.major=5
feature.node.kubernetes.io/kernel-version.minor=14
feature.node.kubernetes.io/kernel-version.revision=0
feature.node.kubernetes.io/memory-numa=true
feature.node.kubernetes.io/memory-nv.present=true
feature.node.kubernetes.io/pci-10de.present=true
feature.node.kubernetes.io/pci-10de.sriov.capable=true
feature.node.kubernetes.io/pci-1137.present=true
feature.node.kubernetes.io/pci-1a03.present=true
feature.node.kubernetes.io/rdma.available=true
feature.node.kubernetes.io/storage-nonrotationaldisk=true
feature.node.kubernetes.io/system-os_release.ID=rhel
feature.node.kubernetes.io/system-os_release.OPENSHIFT_VERSION=4.21
feature.node.kubernetes.io/system-os_release.OSTREE_VERSION=9.6.20260112-0
feature.node.kubernetes.io/system-os_release.VERSION_ID=9.6
feature.node.kubernetes.io/system-os_release.VERSION_ID.major=9
feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=6
host-model-cpu.node.kubevirt.io/GraniteRapids=true
kubernetes.io/arch=amd64
kubernetes.io/hostname=ucs-test-01-6536-1-3
kubernetes.io/os=linux
kubevirt.io/ksm-enabled=false
kubevirt.io/schedulable=true
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
node-role.kubernetes.io/worker=
node.openshift.io/os_id=rhel
nvidia.com/cuda.driver-version.full=580.105.08
nvidia.com/cuda.driver-version.major=580
nvidia.com/cuda.driver-version.minor=105
nvidia.com/cuda.driver-version.revision=08
nvidia.com/cuda.driver.major=580
nvidia.com/cuda.driver.minor=105
nvidia.com/cuda.driver.rev=08
nvidia.com/cuda.runtime-version.full=13.0
nvidia.com/cuda.runtime-version.major=13
nvidia.com/cuda.runtime-version.minor=0
nvidia.com/cuda.runtime.major=13
nvidia.com/cuda.runtime.minor=0
nvidia.com/gfd.timestamp=1779050846
nvidia.com/gpu-driver-upgrade-state=upgrade-done
nvidia.com/gpu.compute.major=9
nvidia.com/gpu.compute.minor=0
nvidia.com/gpu.count=2
nvidia.com/gpu.deploy.container-toolkit=true
nvidia.com/gpu.deploy.dcgm=true
nvidia.com/gpu.deploy.dcgm-exporter=true
nvidia.com/gpu.deploy.device-plugin=true
nvidia.com/gpu.deploy.driver=true
nvidia.com/gpu.deploy.gpu-feature-discovery=true
nvidia.com/gpu.deploy.mig-manager=true
nvidia.com/gpu.deploy.node-status-exporter=true
nvidia.com/gpu.deploy.nvsm=true
nvidia.com/gpu.deploy.operator-validator=true
nvidia.com/gpu.family=hopper
nvidia.com/gpu.machine=UCSX-210C-M8
nvidia.com/gpu.memory=95830
nvidia.com/gpu.mode=compute
nvidia.com/gpu.present=true
nvidia.com/gpu.product=NVIDIA-H100-NVL
nvidia.com/gpu.replicas=0
nvidia.com/gpu.sharing-strategy=none
nvidia.com/mig-1g.12gb.count=4
nvidia.com/mig-1g.12gb.engines.copy=1
nvidia.com/mig-1g.12gb.engines.decoder=1
nvidia.com/mig-1g.12gb.engines.encoder=0
nvidia.com/mig-1g.12gb.engines.jpeg=1
nvidia.com/mig-1g.12gb.engines.ofa=0
nvidia.com/mig-1g.12gb.memory=11008
nvidia.com/mig-1g.12gb.multiprocessors=16
nvidia.com/mig-1g.12gb.product=NVIDIA-H100-NVL-MIG-1g.12gb
nvidia.com/mig-1g.12gb.replicas=1
nvidia.com/mig-1g.12gb.sharing-strategy=none
nvidia.com/mig-1g.12gb.slices.ci=1
nvidia.com/mig-1g.12gb.slices.gi=1
nvidia.com/mig-3g.47gb.count=3
nvidia.com/mig-3g.47gb.engines.copy=3
nvidia.com/mig-3g.47gb.engines.decoder=3
nvidia.com/mig-3g.47gb.engines.encoder=0
nvidia.com/mig-3g.47gb.engines.jpeg=3
nvidia.com/mig-3g.47gb.engines.ofa=0
nvidia.com/mig-3g.47gb.memory=47488
nvidia.com/mig-3g.47gb.multiprocessors=60
nvidia.com/mig-3g.47gb.product=NVIDIA-H100-NVL-MIG-3g.47gb
nvidia.com/mig-3g.47gb.replicas=1
nvidia.com/mig-3g.47gb.sharing-strategy=none
nvidia.com/mig-3g.47gb.slices.ci=3
nvidia.com/mig-3g.47gb.slices.gi=3
nvidia.com/mig.capable=true
nvidia.com/mig.config=shared-models-layout
nvidia.com/mig.config.state=success
nvidia.com/mig.strategy=mixed
nvidia.com/mps.capable=false
nvidia.com/vgpu.present=false
scheduling.node.kubevirt.io/tsc-frequency-2900000000=true
The NVIDIA operator will deploy new pods into the gpu-operator namespace and when all of those are running and healthy it should be time to deploy your first workloads.
~# oc get pods -n gpu-operator
NAME READY STATUS RESTARTS AGE
gpu-feature-discovery-5xl2l 1/1 Running 0 2d22h
gpu-operator-54846444f4-4n94q 1/1 Running 1 22d
nvidia-container-toolkit-daemonset-fkcbv 1/1 Running 0 2d22h
nvidia-dcgm-7c4hd 1/1 Running 0 2d22h
nvidia-dcgm-exporter-jx4kn 1/1 Running 0 2d22h
nvidia-device-plugin-daemonset-8cl22 1/1 Running 0 2d22h
nvidia-driver-daemonset-9.6.20260112-0-vl44n 2/2 Running 2 22d
nvidia-mig-manager-jmszr 1/1 Running 0 2d22h
nvidia-node-status-exporter-rm9bh 1/1 Running 1 22d
nvidia-operator-validator-wrr8w 1/1 Running 0 2d22h
NVIDIA NIMs¶
Before looking at the setup I want to explain the NIM concept as I understand it. I like to compare it to a traditional webserver (not sure if this is exactly accurate so let me know if it's not).
There are a various web servers (e.g. Nginx, Apache) you can deploy so that when some requests your page you serve the HTML/JS/CSS assets. In the same way, when I send an inferencing request to a model (e.g. llama-8b), there needs to be an engine which provides the inferencing/serving functionality. In this case it's not returning a webpage but a response to your request (e.g. generate me a poem or is this transaction fraudulent?).
There are different open source inferencing libraries such as vLLM, Ollama, and Triton which you can setup yourself and then configure/optimize along with installing or pulling the models you want to host.
NVIDIA Inference Microservices (NIMs) and AMD Inference Microservices (AIMs) are a way to simplify this process by wrapping the vLLM + Model + API + other tools into a single container image. The configuration is also optimized and tested.
My experience with NIMs
I'll explain it further later but I found that not all NIMs are equal. When I started testing with some generated traffic I saw degraded performance in one of the NIMs because of the software/configuration it was using.
Since it's a packaged container you just reference the image name and deploy it like a standard container/pod. The NIMs are found at https://catalog.ngc.nvidia.com/, although you'd need a login and NVAIE license to get access. Once the pod is running you can look through the logs to watch it load. It should also show some of the default configuration. You can find more information at https://docs.nvidia.com/nim/large-language-models/latest/about-nim-llm/overview.html
Cilium¶
apiVersion: v1
kind: Service
metadata:
name: llama-nim
namespace: shared-models
labels:
app: llama-nim
component: nim
advertise: bgp-shared-models
annotations:
io.cilium/lb-ipam-ips: "198.18.200.17"
spec:
type: LoadBalancer
selector:
app: llama-nim
ports:
- name: http
port: 80
targetPort: 8000
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: llama-8b-nim
namespace: shared-models
spec:
host: llama-8b-nim.ocpai-01.my-domain.com
port:
targetPort: http
tls:
caCertificate: |
certificate: |
insecureEdgeTerminationPolicy: Redirect
key: |
termination: edge
to:
kind: Service
name: llama-nim
weight: 100
wildcardPolicy: None