How to build a local AI server with used GPUs: the RTX 3090 & Tesla P40 guide

Local Large Language Model (LLM) inference throughput is constrained by memory bandwidth ($B/s$) during the token generation phase and total VRAM capacity during model allocation. While modern enterprise GPUs like the Nvidia H100 offer multi-terabyte-per-second memory bandwidth, secondary-market 24 GB cards—specifically the consumer-grade and the legacy datacenter —provide cost-effective entry points for local inference servers.

However, mixing or selecting between Ampere (GA102) and Pascal (GP102) silicon introduces severe architectural, thermal, and instruction-set trade-offs. Deploying these cards efficiently requires an understanding of CUDA compute capabilities, memory architecture, power delivery, and execution runtime constraints.

1. Silicon Microarchitecture: GA102 vs. GP102

The underlying architecture dictates which quantizations, kernel optimizations, and inference runtimes your host system can execute.

+-------------------------------------------------------------------------+
|                              HOST CPU / RAM                             |
+-------------------------------------------------------------------------+
       |                                                 |
  PCIe Gen4 x16                                    PCIe Gen3 x16
  (31.5 GB/s bidirectional)                        (15.75 GB/s bidirectional)
       |                                                 |
+--------------+                                  +--------------+
|   RTX 3090   |                                  |  Tesla P40   |
| (GA102 Ampere|                                  |(GP102 Pascal)|
| 24GB GDDR6X  |<==== NVLink 3.0 (112.5 GB/s) ====>| 24GB GDDR5   |
|  936 GB/s)   |   (Unsupported across generations|  346 GB/s)   |
+--------------+    Must run independent topologies|--------------+

Nvidia RTX 3090 (Ampere GA102)

Compute Capability: 8.6
Memory Architecture: 24 GB GDDR6X on a 384-bit bus yielding 936 GB/s of memory bandwidth.
Instruction Support: Native hardware acceleration for FP32, FP16, BF16, INT8, and INT4 operations via 3rd-generation Tensor Cores.
Optimization Ecosystem: Fully supports FlashAttention-2, PagedAttention, ExLlamaV2 (EXL2 format), and AWQ.

Nvidia Tesla P40 (Pascal GP102)

Compute Capability: 6.1
Memory Architecture: 24 GB GDDR5 on a 384-bit bus yielding 346 GB/s of memory bandwidth.
Instruction Support: Zero Tensor Cores. High FP32 throughput (12 TFLOPS), native INT8 vector instructions (DP4A) executing at 47 TOPS, but catastrophically crippled FP16 hardware performance (1/64th execution rate of FP32, yielding ~0.18 TFLOPS FP16).
Optimization Ecosystem: Completely incompatible with FlashAttention, FP16-based AWQ, EXL2, or modern BF16 kernels. Relying on FP16 execution causes inference speed to drop below 0.5 tokens per second. The P40 must be targeted exclusively using INT8/INT4 quantized models via CPU-offloaded or pure DP4A INT8 execution runtimes like .

The Token Generation Math

In auto-regressive decoding, inference speed is bounded by memory bandwidth:

$$\text{Theoretical Max Speed (Tokens/s)} = \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Footprint in VRAM (GB)}}$$

For a Llama-3-8B model quantized to 4-bit (Q4_K_M, occupying roughly 5.5 GB VRAM):

RTX 3090: $\frac{936 \text{ GB/s}}{5.5 \text{ GB}} \approx 170 \text{ tokens/sec}$ (Actual real-world performance: ~120–140 t/s due to software overhead).
Tesla P40: $\frac{346 \text{ GB/s}}{5.5 \text{ GB}} \approx 62.9 \text{ tokens/sec}$ (Actual real-world performance: ~25–35 t/s due to DP4A execution latency).

2. Electrical, Thermal, and Mechanical Integration

Integrating these cards into standard ATX or rackmount chassis presents distinct mechanical and power engineering challenges.

Tesla P40 Integration Constraints

Power Pinout Hazard: The Tesla P40 uses an EPS-12V (CPU 8-pin) power input, not a standard PCIe 8-pin GPU cable. Plugging a standard PCIe 8-pin cable into a P40 short-circuits the power supply's 12V rail and can destroy the card's VRMs. You must use an 8-pin PCIe to 8-pin EPS adapter cable wired specifically for Tesla cards.
Cooling Dynamics: The P40 is a passive server card designed for 1U/2U enterprise servers with high-pressure chassis fans. Operating a P40 on a open bench or desktop case without active cooling will result in thermal throttling ($>80^\circ\text{C}$) within 45 seconds of matrix multiplication.
- Fix: A 3D-printed shroud attached to a 50mm x 20mm high-static-pressure blower fan (e.g., Delta BFB1012HH delivering $>25 \text{ CFM}$ at $>1.5 \text{ inch-H}_2\text{O}$ static pressure) wired directly to a 12V molex or 4-