Maybe a more reasonable question: Is there anyone here self-hosting on non-shit hardware? 😅

Is that still true though? My impression is that AMD works just fine for inference with ROCm and llama.cpp nowadays. And you get much more VRAM per dollar, which means you can stuff a bigger model in there. You might get fewer tokens per second compared with a similar Nvidia, but that shouldn’t really be a problem for a home assistant. I believe. Even an Arc a770 should work with IPEX-LLM. Buy two Arc or Radeon with 16 GB VRAM each, and you can fit a Llama 3.2 11B or a Pixtral 12B without any quantization. Just make sure that ROCm supports that specific Radeon card, if you go for team red.

Actually I did. Not thanks to you though.

Probably good, but I want to stay away from anything related to Kubernetes. My experience is that it’s an overkill black hole of constant debugging. Unfortunately. Thanks though!

Looks good. Thanks!

It’s much more powerful though. Based on Org-Mode.

OrgNote