Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>We're back - and cloud GPUs! | SupraLabs Blog</title> | |
| <style> | |
| :root { | |
| --bg: #0f0f0f; | |
| --surface: #1a1a1a; | |
| --border: #333; | |
| --text: #e0e0e0; | |
| --accent: #536bfe; | |
| --muted: #888; | |
| --font-mono: 'JetBrains Mono', 'Fira Code', monospace; | |
| } | |
| * { margin: 0; padding: 0; box-sizing: border-box; } | |
| body { | |
| background-color: var(--bg); | |
| color: var(--text); | |
| font-family: 'Inter', -apple-system, sans-serif; | |
| line-height: 1.6; | |
| padding: 2rem; | |
| } | |
| code, pre, .mono { font-family: var(--font-mono); } | |
| .container { max-width: 900px; margin: 0 auto; } | |
| header { | |
| border-bottom: 2px solid var(--border); | |
| padding-bottom: 2rem; | |
| margin-bottom: 3rem; | |
| display: flex; | |
| justify-content: space-between; | |
| align-items: flex-end; | |
| } | |
| .logo-area h1 { | |
| font-size: 1.2rem; | |
| text-transform: uppercase; | |
| letter-spacing: 2px; | |
| color: var(--accent); | |
| line-height: 1; | |
| display: flex; | |
| align-items: center; | |
| gap: 10px; | |
| } | |
| .logo-area a { text-decoration: none; color: inherit; } | |
| .logo-area { display: flex; align-items: center; gap: 10px; font-weight: bold; font-size: 1.2rem; } | |
| nav a { | |
| color: var(--text); | |
| text-decoration: none; | |
| margin-left: 1.5rem; | |
| font-size: 0.9rem; | |
| border-bottom: 1px solid transparent; | |
| } | |
| nav a:hover { border-bottom: 1px solid var(--accent); } | |
| .post-header { margin-bottom: 3rem; } | |
| .post-header h2 { font-size: 3rem; line-height: 1.1; margin-bottom: 1rem; font-weight: 800; } | |
| .post-meta { font-family: var(--font-mono); color: var(--accent); font-size: 0.9rem; margin-bottom: 2rem; } | |
| .post-content { | |
| background: var(--surface); | |
| border: 1px solid var(--border); | |
| padding: 3rem; | |
| margin-bottom: 4rem; | |
| } | |
| .post-content h2 { font-size: 1.8rem; margin: 2.5rem 0 1rem 0; color: var(--accent); } | |
| .post-content h2:first-child { margin-top: 0; } | |
| .post-content p { margin-bottom: 1.5rem; font-size: 1.1rem; color: var(--text); } | |
| .post-content ul { margin-bottom: 1.5rem; padding-left: 1.5rem; } | |
| .post-content li { margin-bottom: 0.5rem; font-size: 1.1rem; } | |
| .post-content strong { color: #fff; } | |
| .post-content code { | |
| background: #111; | |
| border: 1px solid var(--border); | |
| padding: 2px 6px; | |
| border-radius: 3px; | |
| font-size: 0.95em; | |
| color: var(--accent); | |
| } | |
| .callout { | |
| border-left: 3px solid var(--accent); | |
| background: #111; | |
| padding: 1rem 1.5rem; | |
| margin: 2rem 0; | |
| font-family: var(--font-mono); | |
| font-size: 0.95rem; | |
| color: #ccc; | |
| } | |
| .callout span { display: block; color: var(--muted); font-size: 0.8rem; margin-bottom: 0.4rem; } | |
| /* IO example pairs */ | |
| .io-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin: 2rem 0; } | |
| .io-pair { | |
| background: #111; | |
| border: 1px solid var(--border); | |
| padding: 1.2rem; | |
| } | |
| .io-pair .io-label { | |
| font-family: var(--font-mono); | |
| font-size: 0.72rem; | |
| color: var(--muted); | |
| margin-bottom: 0.5rem; | |
| } | |
| .io-pair .io-user { color: #ccc; font-size: 0.95rem; margin-bottom: 0.8rem; line-height: 1.5; } | |
| .io-pair .io-result { | |
| font-family: var(--font-mono); | |
| color: var(--accent); | |
| font-size: 0.9rem; | |
| font-weight: 700; | |
| } | |
| /* Flow diagram */ | |
| .flow { | |
| display: flex; | |
| align-items: center; | |
| justify-content: center; | |
| gap: 0.8rem; | |
| margin: 2rem 0; | |
| flex-wrap: wrap; | |
| } | |
| .flow-box { | |
| background: #111; | |
| border: 1px solid var(--border); | |
| padding: 0.8rem 1.2rem; | |
| font-family: var(--font-mono); | |
| font-size: 0.82rem; | |
| text-align: center; | |
| color: #ccc; | |
| } | |
| .flow-box.accent { border-color: var(--accent); color: var(--accent); } | |
| .flow-arrow { color: var(--accent); font-size: 1.2rem; } | |
| .table-wrap { overflow-x: auto; margin: 2rem 0; } | |
| table { width: 100%; border-collapse: collapse; font-family: var(--font-mono); font-size: 0.88rem; } | |
| th { background: #111; color: var(--accent); padding: 0.75rem 1rem; text-align: left; border: 1px solid var(--border); } | |
| td { padding: 0.7rem 1rem; border: 1px solid var(--border); color: var(--text); } | |
| tr:nth-child(even) td { background: #111; } | |
| .tags { display: flex; gap: 0.5rem; margin-top: 2rem; flex-wrap: wrap; } | |
| .tag { font-family: var(--font-mono); font-size: 0.7rem; padding: 2px 8px; border: 1px solid var(--border); border-radius: 4px; color: var(--muted); } | |
| footer { margin-top: 6rem; padding-bottom: 2rem; font-size: 0.8rem; color: var(--muted); text-align: center; } | |
| @media (max-width: 600px) { | |
| .post-header h2 { font-size: 2rem; } | |
| .post-content { padding: 1.5rem; } | |
| header { flex-direction: column; align-items: flex-start; gap: 1rem; } | |
| nav a { margin-left: 0; margin-right: 1rem; } | |
| .io-grid { grid-template-columns: 1fr; } | |
| .flow { flex-direction: column; } | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <div class="container"> | |
| <header> | |
| <div class="logo-area" style="font-size: 1.5em;"> | |
| <a href="./index.html"><h1><img src="./image.png" style="height: 2em"> SupraLabs_</h1></a> | |
| </div> | |
| <nav> | |
| <a href="./index.html#news">News</a> | |
| <a href="https://huggingface.co/SupraLabs" target="blank">HuggingFace</a> | |
| <a href="./index.html#hardware">Hardware</a> | |
| </nav> | |
| </header> | |
| <article> | |
| <div class="post-header"> | |
| <div class="post-meta">// 2026-08-29 | Research</div> | |
| <h2>We're back.<br>And we audited<br>cloud GPU math.</h2> | |
| </div> | |
| <div class="post-content"> | |
| <p>This blog was dark for a while. Not dead - parked. We are back. First post after the pause is not a launch. It is a spreadsheet with teeth.</p> | |
| <p>Cloud GPU listings mix <strong>dense</strong> and <strong>sparse</strong> BF16 numbers like they are the same unit. They are not. NVIDIA datasheets often lead with 2:4 sparsity (roughly 2x dense). Sort by that mix and you pick the wrong card for a long run 😭. We re-checked VRAM, memory bandwidth, architecture, and BF16 dense TFLOPS against vendor datasheets. Hourly prices were already trusted (in Runpod). Everything else got rebuilt.</p> | |
| <h2>The dense / sparse trap</h2> | |
| <p>Training almost never uses structured sparsity the way the marketing peak assumes. If you pay for FLOPs, you should compare <strong>BF16 dense (FP32 accumulate)</strong>. Sparse peaks belong in a footnote, not in the ranking column.</p> | |
| <p>Biggest corrections from the first pass:</p> | |
| <ul> | |
| <li>RTX PRO 4000 is ~161 dense TFLOPS, not ~358. Bandwidth is 672 GB/s.</li> | |
| <li>RTX 5090 419 is sparse; dense is 209.5.</li> | |
| <li>H100 NVL is 835.5 dense, not the SXM 989 number copied across SKUs.</li> | |
| <li>L40 dense is 181, not the L40S 362 figure.</li> | |
| <li>H200 is 141 GB, not 143. SXM and NVL share the same die and HBM3e stack.</li> | |
| <li>B200 bandwidth is 7.7 TB/s (often rounded to 8). B300 keeps ~2250 BF16 dense; the Ultra bump is mostly FP4 and 288 GB.</li> | |
| <li>MI300X 1307 BF16 dense was already right - and that moves it to the top of $/FLOP once everyone else is densified.</li> | |
| </ul> | |
| <p>Ada workstation cards were the worst offenders: Tensor numbers in the PDF are often FP8-with-sparsity. Divide by 8 and you get a usable BF16 dense estimate (RTX 4000 Ada ~41, RTX 2000 Ada ~24).</p> | |
| <h2>How to pick a GPU for a long run</h2> | |
| <p>Fixed work W (tokens, epochs, whatever). Price p in $/h. Peak dense BF16 F. Bandwidth B. Arithmetic intensity I (FLOPs per byte). Utilization η (MFU, often 0.3-0.5 in training).</p> | |
| <div class="callout"> | |
| <span>// cost</span> | |
| cost = p · W / F_eff<br> | |
| F_eff = η · min(F, I · B)<br> | |
| pick argmin p / F_eff subject to VRAM ≥ model + opt + acts | |
| </div> | |
| <p>That collapses to two sorts:</p> | |
| <ul> | |
| <li><strong>Compute-bound</strong> (big-batch training): minimize <strong>$ per BF16 TFLOP</strong>.</li> | |
| <li><strong>Memory-bound</strong> (decode, tiny batches): minimize <strong>$ per TB/s</strong>.</li> | |
| </ul> | |
| <p>If the model does not fit, multiply by a parallel tax. PCIe without NVLink is ugly: TP=2 on a 70B BF16 decode can lose 35-55% vs a single fat card. Consumer GDDR also has no ECC - a bit flip in a 48-hour run is a real failure mode. Checkpoint.</p> | |
| <p>Peak is not MFU. L40S looks cheap on 362 TFLOPS sitting on 0.86 TB/s until the kernel is bandwidth-starved. MI300X wins on paper if ROCm actually delivers; measured FP16/BF16 is often 45-85% of peak depending on the stack.</p> | |
| <h2>The table</h2> | |
| <p>Hourly $ kept as-is. Specs renormalized to BF16 <strong>dense</strong>. Sorted by $ per BF16 TFLOP. Scroll sideways if your viewport is not a cinema screen ... lol.</p> | |
| <div class="table-wrap"> | |
| <table> | |
| <thead> | |
| <tr><th>GPU</th><th>Architecture</th><th>VRAM</th><th>Data throughput in TB/s</th><th>BF16 TFLOPS</th><th>Price in $ per hour</th><th>Price in $ per TB data throughput</th><th>Price in $ per BF16 TFLOP</th></tr> | |
| </thead> | |
| <tbody> | |
| <tr><td>MI300X</td><td>CDNA 3</td><td>192 GB</td><td>5.3</td><td>1307</td><td>2.39</td><td>0.000125</td><td>0.00000051</td></tr> | |
| <tr><td>RTX A5000</td><td>Ampere</td><td>24 GB</td><td>1</td><td>111</td><td>0.27</td><td>0.000098</td><td>0.00000068</td></tr> | |
| <tr><td>RTX A4500</td><td>Ampere</td><td>20 GB</td><td>0.64</td><td>~95</td><td>0.25</td><td>0.000109</td><td>0.00000073</td></tr> | |
| <tr><td>L40S</td><td>Ada Lovelace</td><td>48 GB</td><td>1</td><td>362</td><td>0.99</td><td>0.000318</td><td>0.00000076</td></tr> | |
| <tr><td>A40</td><td>Ampere</td><td>48 GB</td><td>1</td><td>150</td><td>0.44</td><td>0.000176</td><td>0.00000082</td></tr> | |
| <tr><td>B200</td><td>Blackwell</td><td>180 GB</td><td>7.7</td><td>2250</td><td>6.79</td><td>0.000245</td><td>0.00000084</td></tr> | |
| <tr><td>RTX A4000</td><td>Ampere</td><td>16 GB</td><td>0</td><td>~77</td><td>0.25</td><td>0.000155</td><td>0.00000091</td></tr> | |
| <tr><td>H100 SXM</td><td>Hopper</td><td>80 GB</td><td>3.35</td><td>989</td><td>3.29</td><td>0.000273</td><td>0.00000092</td></tr> | |
| <tr><td>RTX PRO 4500 (+SE)</td><td>Blackwell</td><td>32 GB</td><td>1</td><td>~215</td><td>0.72</td><td>0.000223</td><td>0.00000093</td></tr> | |
| <tr><td>RTX A6000</td><td>Ampere</td><td>48 GB</td><td>1</td><td>155</td><td>0.53</td><td>0.000192</td><td>0.00000095</td></tr> | |
| <tr><td>B300</td><td>Blackwell Ultra</td><td>288 GB</td><td>8</td><td>2250</td><td>7.89</td><td>0.000274</td><td>0.00000097</td></tr> | |
| <tr><td>RTX PRO 4000</td><td>Blackwell</td><td>24 GB</td><td>1</td><td>~161</td><td>0.57</td><td>0.000236</td><td>0.00000098</td></tr> | |
| <tr><td>RTX PRO 6000 WK</td><td>Blackwell</td><td>96 GB</td><td>1.79</td><td>~500</td><td>1.89</td><td>0.000293</td><td>0.00000105</td></tr> | |
| <tr><td>H100 NVL</td><td>Hopper</td><td>94 GB</td><td>3.9</td><td>836</td><td>3.19</td><td>0.000227</td><td>0.00000106</td></tr> | |
| <tr><td>H100 PCIe</td><td>Hopper</td><td>80 GB</td><td>2</td><td>756</td><td>2.89</td><td>0.000401</td><td>0.00000106</td></tr> | |
| <tr><td>H200 NVL</td><td>Hopper</td><td>141 GB</td><td>4.8</td><td>989</td><td>3.79</td><td>0.000219</td><td>0.00000106</td></tr> | |
| <tr><td>L4</td><td>Ada Lovelace</td><td>24 GB</td><td>0.3</td><td>121</td><td>0.49</td><td>0.000454</td><td>0.00000112</td></tr> | |
| <tr><td>RTX PRO 6000 SE</td><td>Blackwell</td><td>96 GB</td><td>1.79</td><td>~500</td><td>2.09</td><td>0.000324</td><td>0.00000116</td></tr> | |
| <tr><td>PRO 6000 MIG 48GB</td><td>Blackwell (MIG)</td><td>48 GB</td><td>~0.90</td><td>~250</td><td>1.09</td><td>0.000338</td><td>0.00000121</td></tr> | |
| <tr><td>A100 PCIe</td><td>Ampere</td><td>80 GB</td><td>1.94</td><td>312</td><td>1.39</td><td>0.000200</td><td>0.00000124</td></tr> | |
| <tr><td>RTX 4090</td><td>Ada Lovelace</td><td>24 GB</td><td>1.01</td><td>165</td><td>0.74</td><td>0.000204</td><td>0.00000124</td></tr> | |
| <tr><td>L40</td><td>Ada Lovelace</td><td>48 GB</td><td>1</td><td>181</td><td>0.82</td><td>0.000264</td><td>0.00000126</td></tr> | |
| <tr><td>RTX 6000 Ada</td><td>Ada Lovelace</td><td>48 GB</td><td>0.96</td><td>182.5</td><td>0.84</td><td>0.000243</td><td>0.00000128</td></tr> | |
| <tr><td>H200 SXM</td><td>Hopper</td><td>141 GB</td><td>4.8</td><td>989</td><td>4.59</td><td>0.000266</td><td>0.00000129</td></tr> | |
| <tr><td>PRO 6000 MIG 24GB</td><td>Blackwell (MIG)</td><td>24 GB</td><td>~0.45</td><td>~125</td><td>0.59</td><td>0.000366</td><td>0.00000131</td></tr> | |
| <tr><td>RTX 5090</td><td>Blackwell</td><td>32 GB</td><td>1.79</td><td>209.5</td><td>0.99</td><td>0.000153</td><td>0.00000131</td></tr> | |
| <tr><td>A100 SXM</td><td>Ampere</td><td>80 GB</td><td>2.04</td><td>312</td><td>1.59</td><td>0.000217</td><td>0.00000142</td></tr> | |
| <tr><td>RTX 4000 Ada</td><td>Ada Lovelace</td><td>20 GB</td><td>0.36</td><td>~41</td><td>0.28</td><td>0.000216</td><td>0.00000190</td></tr> | |
| <tr><td>RTX 3090</td><td>Ampere</td><td>24 GB</td><td>1</td><td>71</td><td>0.5</td><td>0.000148</td><td>0.00000196</td></tr> | |
| <tr><td>RTX 2000 Ada</td><td>Ada Lovelace</td><td>16 GB</td><td>0</td><td>~24</td><td>0.24</td><td>0.000298</td><td>0.00000278</td></tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <h2>What we would actually rent</h2> | |
| <p>Jobs that fit in 20-24 GB: A4500 / A5000 still crush $/FLOP. Real training: MI300X if the software stack is ROCm-shaped, otherwise B200 or H100 SXM among NVIDIA. H200 NVL is the sleeper - H100-SXM compute, 141 GB, 4.8 TB/s, almost the same $ per dense TFLOP as H100 PCIe/NVL.</p> | |
| <p>Do not sort sparse marketing peaks and call it research. We did that once. Then we fixed it. Life is crazy 😭😂</p> | |
| <div class="callout"> | |
| <span>// notes</span> | |
| Prices --> as provided ($/h)<br> | |
| FLOPS --> BF16 dense, vendor datasheets<br> | |
| BW / VRAM --> NVIDIA / AMD product pages<br> | |
| Formula --> roofline, not vibes | |
| </div> | |
| <div class="tags"> | |
| <span class="tag">#we're-back</span> | |
| <span class="tag">#gpu</span> | |
| <span class="tag">#bf16</span> | |
| <span class="tag">#roofline</span> | |
| <span class="tag">#cloud</span> | |
| <span class="tag">#mi300x</span> | |
| <span class="tag">#blackwell</span> | |
| <span class="tag">#research</span> | |
| </div> | |
| </div> | |
| </article> | |
| <footer> | |
| <p class="mono">© 2026 SupraLabs // Built for the community.</p> | |
| </footer> | |
| </div> | |
| </body> | |
| </html> |