Uncovering how inequality emerges from human interaction is imperative for just societies. Here we show that the way social groups interact in face-to-face situations can enable the emergence of degree inequality. We present a mechanism that integrates group mixing dynamics with individual preferences, which reproduces group degree inequality found in six empirical data sets of face-to-face interactions. We uncover the impact of group-size imbalance on degree inequality, revealing a critical minority group size that changes social gatherings qualitatively. If the minority group is larger than this ‘critical mass’ size, it can be a well-connected, cohesive group; if it is smaller, minority cohesion widens degree inequality. Finally, we expose the under-representation of social groups in degree rankings due to mixing dynamics and propose a way to reduce such biases.