We investigate whether node embeddings, which are vector representations of graph nodes, can be inverted to approximately recover the graph used to generate them. We present algorithms that invert embeddings from the popular DeepWalk method. In experiments on real-world networks, we find that significant information about the original graph, such as specific edges, is often lost through the process of embedding and inversion; however, community structure is often preserved or even enhanced. Our findings are a step towards a more rigorous understanding of what information embeddings encode about the input graph, and why this information is useful for learning tasks.