I have a similar setup and in the end just went for a cheap chinese („ESKEVE“) 4x HDMI 2.0 KVM switch that supports 4K60. No problems except HDR does not work properly, which is not useful on my monitor anyway. Laptops are on USB-C docks with HDMI 2.0 out, PC uses a passive DP to HDMI cable.
Well then. Why not make them not shitty?