This paper is concerned with the problem of global exponential passivity for quaternion-valued memristor-based neural networks (QVMNNs) with time-varying delay. The QVMNNs can be seen as a switched system due to the memristor parameters are switching according to the states of the network. This is the first time that the global exponential passivity of QVMNNs with time-varying delay is investigated. By means of a nondecomposition method and structuring novel Lyapunov functional in form of quaternion self-conjugate matrices, the delay-dependent passivity criteria are derived in the forms of quaternion-valued linear matrix inequalities (LMIs) as well as complex-valued LMIs. Furthermore, the asymptotical stability criteria can be obtained from the proposed passivity criteria. Finally, a numerical example is presented to illustrate the effectiveness of the theoretical results.