Prostate cancer is one of the leading causes of death around the world. The manual Gleason grading of prostate cancer after histological analysis of stained tissue slides is rigorous, time-consuming and also suffers from subjectivity among experts. Image-based computer-assisted diagnosis can serve pathologists to efficiently diagnose cancer in early stages. We have proposed a Hybrid Unified Deep Learning Architecture to grade the prostate cancer accurately and quickly. For the feature analysis technique, we have implemented the shearlet transform in addition to original RGB images. We have introduced saliency maps of images using a Deep Convolutional Generative Adversarial Network (DCGAN) by applying semantic segmentation technique with the salient maps provided by pathology experts. Our proposed architecture is a combination of Convolutional Neural Netowork (CNN), Recurrent Neural Netowrk (RNN) and fine-tuned VGGnet. We have introduced a novel approach of utilizing LSTM-RNN for the sequential subband images of the shearlet coefficients. Our hybrid framework is a computationally high-cost architecture to train but proved to be highly accurate and faster in the testing phase. With our approach, we have achieved an accuracy of 0.98 ± 0.02 for Gleason grading of prostate cancer on the dataset provided by Jafari-Khouzani and Soltanian-Zadeh which is used in successive research work.