The general problem of dissipation in macroscopic large-amplitude collective motion and its relation to the energy diffusion of the intrinsic degrees of freedom of a nucleus is studied. By applying the cranking approach to the nuclear many-body system, a set of coupled dynamical equations for the collective classical variables and the quantum mechanical occupancies of the intrinsic nuclear states is derived. Different dynamical regimes of the intrinsic nuclear motion and its consequences on time properties of the collective dissipation are discussed. The approach is applied to the descent of the nucleus from the fission barrier.