Reproductive skew theory seeks to explain how reproduction is divided among group members in animal societies. Existing theory is framed almost entirely in terms of selection, though nonadaptive processes must also play some role in the evolution of reproductive skew. Here I propose that a genetic correlation between helper fecundity and breeder fecundity may frequently constrain the evolution of reproductive skew. This constraint is part of a wider phenomenon that I term ‘caste load’, which is defined as the decline in mean fitness caused by caste-specific selection pressures, that is, differential selection on breeding and nonbreeding individuals. I elaborate the caste load hypothesis using quantitative and population genetic arguments and individual-based simulations. Although selection can sometimes erode genetic correlations and resolve caste load, this may be constrained when mutations have similar pleiotropic effects on breeder and helper traits. I document evidence for caste load, identify putative genomic adaptations to it, and suggest future research directions. The models highlight the value of considering adaptation within the boundaries imposed by genetic architecture and incidentally reaffirm that monogamy promotes the evolutionary transition to eusociality.