Distributed hydrological models account for spatial heterogeneity by discretizing the watershed into unique units based on the watershed characteristics. However, parameter estimation is one of the major tasks in the application of distributed hydrological models. The existing calibration methods for distributed hydrological models do not consider the spatial variability of the parameters across the basin, and, therefore, do not guarantee good simulations on locations other than the calibration outlets. This study proposes a calibration approach which preserves the heterogeneity of the parameters across the basin. The basic simulation units of the distributed models are grouped in this approach based on the land use and soil type, and a random perturbation of the parameters is performed in these groups during calibration. The proposed method is demonstrated through a case study of two watersheds in the USA using soil and water assessment tool (SWAT) model. The results indicate that the calibrated model simulations in the upstream gauged locations (other than that used for calibration) are much better in the proposed approach, in contrast to the currently employed calibration method. Nonetheless, it is also observed that the proposed calibration approach would be more effective in watersheds that have higher spatial heterogeneity. © 2021, The Author(s), under exclusive licence to Springer-Verlag GmbH, DE part of Springer Nature.