The H100 NVL has a full 6144-bit memory interface (1024-bit for each HBM3 stack) and memory speed up to 5.1 Gbps. This means that the maximum throughput is 7.8GB/s, more than twice as much as the H100 SXM. Large Language Models require large buffers and higher bandwidth will certainly have an impact as well.
NVIDIA H100 NVL for Large Language Model Deployment is ideal for deploying massive LLMs like ChatGPT at scale. The new H100 NVL with 96GB of memory with Transformer Engine acceleration delivers up to 12x faster inference performance at GPT-3 compared to the prior generation A100 at data center scale.
Ships in 2 weeks from payment. Verify with live chat agent in advance for availability as stock and pricing is volatile and changes every 48-72 hours. All sales final. No returns or cancellations. For bulk inquiries, consult a live chat agent or call our toll-free number.
The NVIDIA GPU-powered H100 NVL graphics card is said to feature a dual-GPU NVLINK interconnect with each chip featuring 96 GB of HBM3e memory. The GPU is able to process up to 175 Billion ChatGPT parameters on the go. Four of these GPUs in a single server can offer up to 10x the speed up compared to a traditional DGX A100 server with up to 8 GPUs.
| Specification | H100 SXM | H100 PCIe | H100 NVL^2 |
|---|---|---|---|
| FP64 | 34 teraFLOPS | 26 teraFLOPS | 68 teraFLOPS |
| FP64 Tensor Core | 67 teraFLOPS | 51 teraFLOPS | 134 teraFLOPS |
| FP32 | 67 teraFLOPS | 51 teraFLOPS | 134 teraFLOPS |
| TF32 Tensor Core | 989 teraFLOPS | 756teraFLOPS | 1,979 teraFLOPS’ |
| BFLOAT16 Tensor Core | 1,979 teraFLOPS | 1,513 teraFLOPS | 3,958 teraFLOPS |
| FP16 Tensor Core | 1,979 teraFLOPS | 1,513 teraFLOPS | 3,958 teraFLOPS |
| FP8 Tensor Core | 3,958 teraFLOPS | 3,026 teraFLOPS | 7,916 teraFLOPS |
| INT8 Tensor Core | 3,958 TOPS | 3,026 TOPS | 7,916 TOPS |
| GPU memory | 80GB | 80GB | 188GB |
| GPU memory bandwidth | 3.35TB/s | 2TB/s | 7.8TB/s |
| Decoders | 7 NVDEC | 7 NVDEC | 14 NVDEC |
| 7 JPEG | 7 JPEG | 14 JPEG | |
| Max thermal design power (TDP) | Up to 700W (configurable) | 300-350W (configurable) | 2x 350-400W (configurable) |
| Multi-Instance GPUs | Up to 7 MIGS @ 10GB each | Up to 7 MIGS @ 10GB each | Up to 14 MIGS @ 12GB each |
| Form factor | SXM | PCle | 2x PCIe |
| Interconnect | NVLink: 900GB/s PCIe Gen5: 128GB/s | Dual-slot air-cooled NVLink: 600GB/s PCIe Gen5: 128GB/s | Dual-slot air-cooled NVLink: 600GB/s PCIe Gen5: 128GB/s |
| Server options | NVIDIA HGX H100 Partner and NVIDIA-Certified Systems with 4 or 8 GPUs NVIDIA DGX H100 with 8 GPUS | Partner and NVIDIA-Certified Systems with 1-8 GPUs | Partner and NVIDIA-Certified Systems with 2-4 pairs |
| NVIDIA AI Enterprise | Add-on | Included | Add-on |
No reviews available.
