The categories named by spatial terms vary considerably across languages. It is often proposed that underlying this variation is a universal set of primitive spatial concepts that are combined differently in different languages. Despite the inherently cognitive assumptions of this proposal, such spatial primitives have generally been inferred in a top-down manner from linguistic data. Here we show that comparable spatial primitives can be inferred bottom-up from non-linguistic pile-sorting of spatial stimuli by speakers of English, Dutch, and Chichewa. We demonstrate that primitives obtained in this fashion explain meaningful cross-linguistic variation in spatial categories better than primitives designed by hand for that purpose, and reflect both universal and language-specific spatial semantics.