Deploying modern language models in high-performance computing (HPC) environments presents a paradox: while HPC clusters possess immense parallel compute power, dedicated GPU nodes are heavily contested, expensive, and often queued for days. For many batch analysis tasks—such as code synthesis or log classification—relying entirely on GPU availability stalls workflows.
At HiPeC (High Performance Computing Lab, IIIT Delhi), our goal was to optimize transformer inference specifically for CPU architectures without sacrificing accuracy.
The Optimization Pipeline
Our target benchmark was **HPCorpus**, a comprehensive dataset of parallel computing source code (OpenMP, MPI, and CUDA). Standard 16-bit float evaluation of open-weight models proved too slow for interactive usage. We attacked this across three fronts:
- **Grid-Ada LoRA Fine-Tuning**: Rather than uniform parameter adaptation, we partitioned low-rank update matrices across parameter blocks based on gradient sensitivity grids.
- **GGUF Multi-Level Quantization**: We compressed weights down to 4-bit and 5-bit representations, isolating sensitive attention projection matrices in higher bit-depths.
- **ONNX Runtime + OpenMP Core Affinity**: We converted the fine-tuned weights into ONNX format and tuned thread pinning across physical NUMA nodes to minimize cross-socket cache thrashing.
Results & Takeaways
The combination achieved **87.25% accuracy on HPCorpus**, matching unquantized FP16 checkpoints while achieving a 4.2× reduction in memory footprint and sustaining sub-50ms per-token latency on standard Intel Xeon CPU nodes.
Moving compute closer to the memory hierarchy and understanding cache line contention remains the most effective lever in production systems engineering.