Do task demands change the way we extract information from a stimulus, or only how we use this information for decision making? In order to answer this question for visual word recognition, we used EEG/MEG as well as fMRI to determine the latency ranges and spatial areas in which brain activation to words is modulated by task demands. We presented letter strings in three tasks (lexical decision, semantic decision, silent reading), and measured combined EEG/MEG as well as fMRI responses in two separate experiments. EEG/MEG sensor statistics revealed the earliest reliable task effects at around 150 ms, which were localized, using minimum norm estimates (MNE), to left inferior temporal, right anterior temporal and left precentral gyri. Later task effects (250 ms and 480 ms) occurred in left middle and inferior temporal gyri. Our fMRI data showed task effects in left inferior frontal, posterior superior temporal and precentral cortices. Although there was some correspondence between fMRI and EEG/MEG localizations, discrepancies predominated. We suggest that fMRI may be less sensitive to the early short-lived processes revealed in our EEG/MEG data. Our results indicate that task-specific processes start to penetrate word recognition already at 150 ms, suggesting that early word processing is flexible and intertwined with decision making.