Next weekend hack, trying to run bigger LLM (e.g., gpt4-oss 20B) on mixed multi GPU settings: 2-way (x16+x4) and 3-way (x16+x4+x1). All budget GPUs (PCIe 3, under €50 used, except for 1070ti and P4)

Lookie here! The RX470 8G got a companion and together they can run gpt4-oss 20B MXFP4 MoE using llama.cpp for 12-13 tps (on Windows11, on Linux should be 2-3 times higher). each repurposed mining GPU bought for the price of 1 pizza

The tale of 3 mining GPUs 2x rx470 8g + p106-100 6g on x16/X4/x1 PCIe lanes

For some reason, this combo (1070ti,1060,P106-100) doesn't want to go POST. So I had to turn off P106 and replace it with Tesla P4

Finally a 3-GPU combination that passed POST: 1060 (x16), P4 (x4), P106-100(x1)

Now this works (1070ti,P4,P106-100) but I don't know why the other combi doesn't. Perhaps power distribution issue?

Here's the picture of them. 1070ti and P106-100 use diff PCIe power cable coming from same PSU but different slot/rail

Pretty sure power distrib is the problem. Now this works if I replace P106-100 with 1060

Should I try all AMD with additional R9 Fury? I'm not sure if the current power supply setup works though.. but let's try that later

Surely power distribution is the issue! it works now! gist.github.com/pebaryan/b7239…

Strange, this setup passed POST but showing black screen during os boot so I plugged the HDMI to the motherboard but now the only GPU listed are only iGPU and the R9 Fury

Turns out, the second PSU died :(

R9 Fury is so power hungry and the vram is too tight. Rx470 cannot even save it and only with rtx3060 it can run gpt4-oss 20B without spilling. I wonder how bad Tesla K80 for this setup

RTX 3060 might not be able to be combined with other cards on a (x16/x4/x1) config because the X1 slot is covered by its body

RTX 3060 + Tesla P4 running gpt-oss 20B with Vulkan only reaches 40tps token generation. CUDA run crashes llama.cpp, now rebuilding llama.cpp from the latest commit

For context, Tesla P4 & GTX 1070ti running CUDA can reach 48tps

llama.cpp build complete. CUDA version now runs at 58tps! Not bad! more detailed result here: gist.github.com/pebaryan/0f501…