Yes, using a different passphrases or/and prefixes shouldn’t affect neither latency nor speed, both are depends on location, your bandwidth and hardware power (CPU/RAM), it also depends on used protocol - if you have a wide bandwidth, the native protocol can be better, if it’s limited or you do not have resources on the device where you plan to use uplink, then you can use S3 protocol.
See