What’s the biggest code base you have ever reviewed? What’s the most recent TLS vulnerability you have encountered, as opposed to the last vulnerability in other parts of your OS? Code being swapped by the server, maybe, but are you saying you do a code review every time you update a package or dependency of some other project? This is only less secure in some inconceivably convoluted chain of events that no practical person could enact. No sane person does what you’re saying. Everyone has to trust someone else with code blindly at some point.
Does the server have to run headless, or can one person still run games locally on it while another has a remote session? That is, would I be able to play on the monitor/mouse/keyboard directly attached, while my partner has a session on a laptop?