Drawing on multimodal interactional analysis, the ethnography of communication and social semiotics, this paper proposes a model to explore the impact of the discursive interpreter-storyteller interrelationship on both hearing and deaf children's literary experience in the multimodal literacy event of library storytelling combined with sign language interpretation.