We study the Hall response of two-dimensional lattice systems of charged fermions in a transverse magnetic field, in the ballistic coherent limit. We identify a setup in which this response vanishes over wide regions of parameter space: the "Landauer-Büttiker" setup commonly studied for coherent quantum transport, consisting of a strip contacted to biased ideal reservoirs of charges. We show that this effect does not rely on particle-hole symmetry, and is robust to a variety of perturbations including variations of the transverse magnetic field, chemical potential, and temperature. We trace this robustness back to a topological property of the Fermi surface: the number of Fermi points with positive velocity of the system. We argue that the mechanism leading to a vanishing Hall response applies to noninteracting and interacting systems alike, which we verify in concrete examples using density-matrix renormalization group simulations.