In the midst of a global fisheries crisis, there has been great interest in the fostering of adaptation and resilience in fisheries, as a means to reduce vulnerability and improve the capacity of fishing society to adapt to change. However, enhanced resilience does not automatically result in improved well-being of people, and adaptation strategies are riddled with difficult choices, or trade-offs, that people must negotiate. This paper uses the context of fisheries to explore some apparent tensions between adapting to change on the one hand, and the pursuit of well-being on the other, and illustrates that trade-offs can operate at different levels of scale. It argues that policies that seek to support fisheries resilience need to be built on a better understanding of the wide range of consequences that adaptation has on fisher well-being, the agency people exert in negotiating their adaptation strategies, and how this feeds back into the resilience of fisheries as a social-ecological system. The paper draws from theories on agency and adaptive preferences to illustrate how agency might be better incorporated into the resilience debate.