Lower GPU Performance and jittery input latency

Hey, I have successfully installed the driver, and am using Looking Glass on an optimal setting. Using GPU Passthrough I have done this. When I just use IDD Sample Driver, I get significantly lower FPS in games, and worse input latency/ visible stuttering ?
When I plug an input into the GPU then use it, with the Sample Driver disabled, everything acts as it should.