Ever witnessed a fighter jet spewing hot flames as it kicks into afterburner? In that moment, efficiency is deliberately sacrificed for maximum acceleration.
In the midst of combat, efficiency means nothing when your life is on the line. The jet engine must keep roaring, before the pilot gets taken down by the enemy (and potentially meets their maker).
A GPU faces a similar fate. When pushed to consume hundreds of watts to churn out LLM tokens at the user’s breakneck speed, there’s no choice but to run as fast as possible, even if sweat is pouring and muscle fatigue reaches its peak.
Fortunately, nvidia-smi, with its pl (power limit) option, can be used to set an upper limit on power consumption, so the GPU doesn’t go completely overboard. Those last few dozen watts often don’t make a significant difference in performance, but they definitely contribute to heat generation, which needs to be monitored.
From the graph (measured with llama-bench, for the Mistral-7B-Instruct model, Q4), it’s evident that pushing the power further doesn’t lead to increased LLM speed. 250, 300, or even 350 watts, it’s more or less the same. Meanwhile, dropping to 200 watts does slightly decrease the speed, but it’s very worthwhile considering the power consumption is reduced by a third.
Saving energy is always a wise choice!
- Kiambu Web Design Services
- Kiambu SEO Services
- Kiambu Digital Marketing Services
- Kiambu Social Media Marketing Services
- Kiambu Lipa Pole Pole Phones
- Nyali Web Design Services
- Nyali SEO Services
- Nyali Lipa Pole Pole Phones
- Mombasa Lipa Pole Pole Phones
- Meru Web Design Services
- Meru SEO Services
- Meru Digital Marketing Services
- Meru Social Media Marketing Services
- CBD Nairobi Web Design Services
- Westlands Web Design Services
- Outer Ring Road Web Design Services
- Outer Ring Road SEO Services
- Thika Road Web Design Services
- Thika Road SEO Services
- Thika Road Digital Marketing Services
- Thika Road Lipa Pole Pole Phones
- Langata Web Design Services
- Langata SEO Services
- Langata Lipa Pole Pole Phones
- Mombasa Road Web Design Services
- Mombasa Road SEO Services
- Mombasa Road Lipa Pole Pole Phones
- Mombasa Road Digital Marketing Services
- Mombasa Road Social Media Marketing Services
- Karen Web Design Services
- Karen SEO Services
- Karen Digital Marketing Services
- Garden City Web Design Services
- Thika Road Mall Web Design Services
- Thika Road Mall SEO Services
- Thika Road Mall Lipa Pole Pole Phones
- Eastlands Web Design Services
- Eastlands SEO Services
- Eastlands Lipa Pole Pole Phones
- Donholm Web Design Services
- Donholm SEO Services
- Donholm Lipa Pole Pole Phones
- Ruai Web Design Services
- Ruai SEO Services
- Ruai Lipa Pole Phones
from Blockchain – Techyrack Hub https://ift.tt/oHuDwOk
via IFTTT
No comments:
Post a Comment