Coral reefs sustain millions of people worldwide, yet in recent years, social, environmental, and climate change have caused major declines in coral reef fisheries. Small-scale coral reef fisheries research has largely focused on community-level drivers of fishing, ignoring the heterogeneities that exist within communities. We used social-ecological indicators from 20 coastal villages in Fiji to identify potential fine-scale, context-appropriate drivers of estimated household fish catch. Indicators were developed based on a review of the literature, discussions with local experts, and a pilot study. Using structural equation models, we found that importance of fishing to income, household fish consumption, livelihood diversity, travel time to market, and coral reef area all positively affect estimated household-level fish catch. Our results contrast with findings from other larger scale studies by identifying that households further from markets had higher fishing frequency. We highlight the role of middlemen in these small-scale fisheries, who have been largely overlooked as drivers of fisheries catch. Our findings emphasize the need for household-level analyses to better understand the complexities in coral reef social-ecological systems to more effectively manage small-scale fisheries in communities.