Abstract Background Riverbank erosion has both direct and indirect effects on human life and socio-economy of Bangladesh. The present study investigated the riverbank erosion hazard in study area, its impacts on local people and livelihood vulnerability due to land loss. Methods To evaluate the riverbank erosion hazard in the study area, data have been collected from relevant scientific literatures, different government and non-government organizations, informal interview, questionnaire survey and Focused Group Discussion; and analyzed through different computer program and index. Results From the study, it was found that from 1973 to 2011, about 189.4 km2 lands was eroded from the left bank section and only 23.66 km2 was accreted with a net loss of 155 km2. Instead, right bank of the Padma River behaved in the opposite manner with 166.53 km2 erosion and 134.45 km2 accretions. Comparing to the right bank, left bank was more vulnerable to erosion which destroyed the permanent stable lands. However, the value of the newly accreted char land is very low compare to the main land. Thus this hazard creates a great loss in the local economy. Within the studied time range the monetary loss is about 1414.81 million BDT (17,422,937.16 $). Many wealthy farmers of the study area turn into marginal farmer and even landless due to the erosional hazard. Agricultural land becomes barren land by huge siltation and the cropping pattern has been changed significantly. In addition, the infrastructure and property losses are enormous. From the results of vulnerability index (IPCC-VI) it suggests that the most vulnerable areas are identified as Boyra (0.061), Kanchanpur (0.062), Lesragonj (0.064), Azimnagar (0.067), Sutalori (0.071) and Dhulsonra (0.076) because of more sensitivity and less adaptive capacity. On the other hand, Balara (− 0.017) and Balla (− 0.019) are comparatively least vulnerable comparing to the previous sites. Conclusions As, riverbank erosion is one of the most hazardous disasters in the study area; so treating independent separate policies and program for the vulnerable areas might helpful to support the affected community.