Immunoassays such as the enzyme-linked immunosorbent assay (ELISA) are utilized extensively for detecting protein biomarkers and small molecules in healthcare, environmental monitoring, and food analysis. Unfortunately, the current strategies for immunoassays often require sophisticated apparatus such as a microplate reader, which might not be available in resource-limited areas. To mitigate this problem, we designed a compact smartphone based-device and a multicolor response immunosensor. First, we designed a compact and cost-effective 3D-printed attachment, where a light-emitting diode was used as a light excitation source and a smartphone captured the fluorescent emission signals. Second, by combining quantum dots hybrid and chemical redox reaction, multiple color responses were displayed in the presence of the analyte at different concentrations. Third, solutions with distinct tonality could be readily distinguished by the naked eye and they were suitable for quantitative analysis using the hue-saturation-lightness color space based on a smartphone application. The versatility of the proposed sensing system was demonstrated by implementing an indirect competitive ELISA for analyzing trace drug residues in foodstuffs. The multicolor response of this sensing strategy allows us to visually quantify drug residues in foodstuffs. Moreover, the smartphone-based immunosensor can assess the exact concentration of the analyte by using a self-designed mobile application. The proposed assay provides a highly sensitive performance that the limit of detection was 0.37 ng/mL by visual detection and 0.057 ng/mL using the compact device. Due to its advantages in terms of portability, straightforward visual detection, high sensitivity, and cost effectiveness, the proposed immunosensor has great potential for applications in areas without access to laboratories or expensive infrastructure.