The idea of random matrix theory is applicable not only to the level statistics but also to various physical observables. Taking the dynamical conductivity in isolated quantum dots with diffusive dynamics, we investigate analytically intertwining effects of the time-reversal invariance, level repulsion and quantum interference. We clarify an ambivalent role of the time-reversal invariance at finite frequency by a new invariant analysis respecting the symmetry of the effective field theory. A subtlety of the operator insertion, and the fast-slow mode separation within the effective field description is pointed out.