Sarcasm detection is a challenging natural language processing task for sentiment analysis. Existing deep learning based sarcasm detection models have not fully considered sentiment semantics, even though sentiment semantics is necessary to improve the performance of sarcasm detection. To deal with the problem, we propose a multi-level memory network using sentiment semantics to capture the features of sarcasm expressions. In our model, we use the first-level memory network to capture sentiment semantics, and use the second-level memory network to capture the contrast between sentiment semantics and the situation in each sentence. Moreover, we use an improved convolutional neural network to improve the memory network in the absence of local information. The experimental results on the Internet Argument Corpus (IAC-V1 and IAC-V2) and Twitter dataset demonstrate the effectiveness of our model.