Comprehensive guide to Deep Reinforcement Learning (DRL) as applied to wireless communication systems. Deep Reinforcement Learning empowered by deep neural networks (DNNs) has been developing as a promising solution to address high-dimensional and continuous control problems effectively. The integration of DRL into future wireless networks will revolutionize conventional model-based network optimization with model-free approaches and meet various application demands. By interacting with the environment, DRL provides an autonomous decision-making mechanism for the network entities to solve non-convex, complex, model-free problems, e.g. spectrum access, handover, scheduling, caching, data offloading, and resource allocation. This not only reduces communication overhead but also improves network security and reliability. Though DRL has shown great potential to address emerging issues in complex wireless networks, there are still domain-specific challenges that require further investigation.