The Mediterranean Sea is one of the most oligotrophic marine areas on earth where nitrogen fixation has formally believed to play an important role in carbon and nitrogen fluxes. Although this view is under debate, the diazotrophs responsible for this activity have still not been investigated in the open sea. In this study, we characterised the surface distribution and species richness of unicellular and filamentous diazotrophs across the Mediterranean Sea by combining microscopic counts with size fractionated in situ hybridization (TSA-FISH), and 16S rDNA and <i>nifH</i> genes phylogenies. These genetic analyses were possible owing to the development of a new PCR protocol adapted to scarce microorganisms that can detect as few as 1 cell ml⁻¹ in cultures. Low concentrations of diazotrophic cyanobacteria were detected and this community was dominated at 99.9% by picoplankton hybridized to the Nitro821 probe, specific for unicellular diazotrophic cyanobacteria (UCYN). Among filamentous cyanobacteria only 0.02 filament ml⁻¹ of <i>Richelia</i> were detected in the eastern basin, while small (0.7–1.5 μm) and large (2.5–3.2 μm) Nitro821-targeted cells were recovered at all stations with a mean concentration of 3.5 cell ml⁻¹. The affiliation of the small Nitro821-targeted cells to UCYN-A was confirmed by 16S and <i>nifH</i> phylogenies in the western Mediterranean Sea. In the central and the eastern Mediterranean Sea no 16S rDNA and <i>nifH</i> sequence from UCYN was obtained as cells concentration were close to, or below PCR detection limit. <i>Bradyrhizobium</i> sequences dominated <i>nifH</i> clone libraries from picoplanktonic size fractions. A few sequences of &gamma;-proteobacteria were also detected in the central Mediterranean Sea. While low phosphate and iron concentrations could explain the absence of <i>Trichodesmium</i> sp., the factors that prevent the development of UCYN-B and C remain unknown. We also propose that the dominating picoplankters probably developed specific strategies, such as associations with protists or particles, and/or photosynthetic activity, to acquire carbon for sustaining diazotrophy.