I've had to resort to exactly this, the KVM controls the mouse and keyboard, while dual HDMI my monitor and switch using its remote, as the KVM just won't work with my 4K monitor (tried it on my TV too, didn't work, tried it on office 1080p monitors, did work, even though the KVM says it supports 4K).Ah sorry, my montiors have dual hdmi and displayport so not used to not just pushing a button and connecting, also trying to find a kvm for modern monitors doesn't seem so easy.
If I were going to do it again, I would see if I could stomach the Level1 Tech KVM's, but they are pricey.
This is the one I am currently using - https://www.wootware.co.za/openbox-d-link-dkvm-210h-2-port-kvm-black-switch.html
