Will VM-Host Affinity affect DRS Scheduler?

We conduct a VMWare Health Check recently by a consultant.

He opines that VM-Host affinity will affect DRS Scheduler performance and suggests us to use VM-VM affinity.  We just wonder how much performance degradation will be caused by having VM-Host affinity ?

Thanks