Knowledge Base
GPU Servers & Cloud Computing FAQ
Straight answers to the questions buyers, operators, and AI teams ask us most often.
Hardware & GPUs
Choosing the right silicon for training, fine-tuning, or inference.
Cloud vs On-Prem
When to rent, when to buy, and when to do both.
Power & Cooling
The constraint that decides most deployments.
Pricing & TCO
Sticker price is only a third of what you'll actually spend.
Networking & Storage
The unglamorous half of any AI cluster.
Security & Compliance
What changes when the workload is AI.
Glossary
Key terms you'll encounter when buying, renting, or operating GPU infrastructure.
- Accelerator
- A specialized processor designed to offload specific computations from the CPU. In AI infrastructure, this almost always means a GPU (NVIDIA, AMD) or an AI-specific chip (TPU, Habana Gaudi).
- B200 / Blackwell
- NVIDIA's second-generation data-center GPU architecture following Hopper. B200 delivers roughly 3× the inference throughput of H100 at similar power, with native FP4 and FP6 support for quantized models.
- Bandwidth
- The rate at which data can be read from or written to memory. GPU memory bandwidth (measured in TB/s) is often the bottleneck in transformer inference, not raw compute.
- Colocation (Colo)
- Renting space, power, and cooling in a third-party data center while owning the servers yourself. The middle ground between cloud rental and a fully owned facility.
- FP8 / FP4
- Low-precision numeric formats that let models run faster and fit in less memory. FP8 is mainstream for training and inference; FP4 is emerging for ultra-low-latency serving of quantized models.
- GB200 NVL72
- NVIDIA's rack-scale system combining 72 Blackwell GPUs and 36 Grace CPUs in a single liquid-cooled rack. Designed for training frontier models at the 100B+ parameter scale.
- HBM (High Bandwidth Memory)
- Stacked memory dies connected directly to the GPU die via an interposer. HBM3E on H200 delivers ~4.8 TB/s of bandwidth — an order of magnitude faster than standard DDR5.
- HGX
- NVIDIA's reference architecture for multi-GPU server boards. HGX H200 and HGX B200 are the standard 8-GPU building blocks used by Dell, HPE, Supermicro, and other OEMs.
- Inference
- The process of running a trained model to generate outputs — text, images, predictions — in response to user requests. Often the dominant cost in production AI workloads.
- InfiniBand
- A high-speed networking standard (now owned by NVIDIA) used to connect GPU nodes in training clusters. NDR = 400 Gb/s, XDR = 800 Gb/s. Lower latency and more deterministic than Ethernet for collectives.
- MI350X
- AMD's flagship data-center GPU. Ships with 288 GB of HBM3E per GPU, giving it the highest single-GPU memory capacity currently available. A strong alternative for inference-heavy workloads.
- Neocloud
- A cloud provider specializing in GPU compute rather than general-purpose services. CoreWeave, Lambda, Crusoe, and Nebius are leading examples. Typically 30–50% cheaper than hyperscalers for GPU hours.
- NVLink
- NVIDIA's high-speed GPU-to-GPU interconnect. Inside a single node, NVLink lets 8 GPUs share memory and communicate at up to 900 GB/s. Essential for training large models across multiple GPUs.
- RoCEv2
- RDMA over Converged Ethernet v2. A standard that lets Ethernet networks behave like InfiniBand for GPU clusters — lower CPU overhead and direct memory access between nodes. Cheaper but harder to tune than InfiniBand.
- TCO
- Total Cost of Ownership. The full cost of owning and operating hardware over its useful life, including purchase price, power, cooling, colocation, networking, and support — not just the invoice from the OEM.
- Training
- The process of teaching a neural network by feeding it large datasets and adjusting billions of parameters through backpropagation. Far more compute-intensive than inference, often requiring multi-node clusters.
- Hyperscaler
- The largest cloud providers — AWS, Azure, and Google Cloud — offering broad service catalogs, global footprints, and deep enterprise integrations. Typically the most expensive GPU option per hour.
- Fine-Tuning
- Adapting a pre-trained model to a specific task or dataset with a smaller, targeted training run. Requires far less compute than training from scratch but still needs significant GPU memory.
- Node
- A single server chassis, usually containing 8 GPUs, CPUs, system memory, and networking. The basic building block of a GPU cluster.
- Cluster
- A networked group of GPU nodes working together. A small cluster might be 4–8 nodes; a large training cluster can span hundreds of nodes with InfiniBand interconnects.
- Latency
- The delay between a request being sent and a response being received. In GPU networking, low latency is critical for synchronizing gradients across nodes during distributed training.
- Throughput
- The total amount of work completed per unit of time. In inference, throughput is measured in tokens per second or queries per second — higher is better for cost-efficient serving.
- Parameters
- The adjustable values inside a neural network that store what the model has learned. A '70B parameter' model has 70 billion weights, dictating how much GPU memory is needed to load it.
- Quantization
- Reducing the numeric precision of model weights (e.g., from FP16 to FP8 or FP4) to shrink memory footprint and speed up inference. Often done with minimal accuracy loss.
- PDU
- Power Distribution Unit — the rack-mounted device that splits incoming facility power into individual outlets for servers. GPU nodes need high-amperage PDUs; rack-scale systems may need 3-phase PDUs.
- Parallel Filesystem
- A high-performance storage system (Lustre, BeeGFS, WEKA, VAST) that lets thousands of clients read and write simultaneously. Standard for feeding training data to large GPU clusters without I/O bottlenecks.
- Direct-to-Chip Liquid Cooling
- A cooling method where coolant flows through cold plates directly attached to the GPU and CPU dies. Required for B300, GB200, and GB300 because air alone cannot dissipate 120 kW per rack.
Still have questions?
Our team helps teams spec, source, and deploy GPU infrastructure — from a single node to full rack-scale clusters.