Space, time, and number are fundamental to how we act within and reason about the world. These three experiential domains are systematically intertwined in behavior, language, and the brain. Two main theories have attempted to account for cross-domain interactions. A Theory of Magnitude (ATOM) posits a domain-general magnitude system. Conceptual Metaphor Theory (CMT) maintains that cross-domain interactions are manifestations of asymmetric mappings that use representations of space to structure the domains of number and time. These theories are often viewed as competing accounts. We propose instead that ATOM and CMT are complementary, each illuminating different aspects of cross-domain interactions. We argue that simple representations of magnitude cannot, on their own, account for the rich, complex interactions between space, time and number described by CMT. On the other hand, ATOM is better at accounting for low-level and language-independent associations that arise early in ontogeny. We conclude by discussing how magnitudes and metaphors are both needed to understand our neural and cognitive web of space, time and number.