Land cover classification of very high resolution (VHR) imagery over urban areas is an extremely challenging task. Impervious land covers such as buildings, roads, and parking lots are spectrally too similar to be separated using only the spectral information of VHR imagery. Additional information, therefore, is required for separating such land covers by the classifier. One source of additional information is the vector data, which are available in archives for many urban areas. Further, the object-based approach provides a more effective way to incorporate vector data into the classification process as the misregistration between different layers is less problematic in object-based compared to pixel-based image analysis. In this research, a hierarchical rule-based object-based classification framework was developed based on a small subset of QuickBird (QB) imagery coupled with a layer of height points called Spot Height (SH) to classify a complex urban environment. In the rule-set, different spectral, morphological, contextual, class-related, and thematic layer features were employed. To assess the general applicability of the rule-set, the same classification framework and a similar one using slightly different thresholds applied to larger subsets of QB and IKONOS (IK), respectively. Results show an overall accuracy of 92% and 86% and a Kappa coefficient of 0.88 and 0.80 for the QB and IK Test image, respectively. The average producers’ accuracies for impervious land cover types were also 82% and 74.5% for QB and IK.