Deep Reinforcement Learning Hands-On, 2/e (Paperback)

Maxim Lapan



Deep Reinforcement Learning Hands-On, Second Edition is an updated and expanded version of the bestselling guide to the very latest reinforcement learning (RL) tools and techniques. It provides you with an introduction to the fundamentals of RL, along with the hands-on ability to code intelligent learning agents to perform a range of practical tasks.


With six new chapters devoted to a variety of up-to-the-minute developments in RL, including discrete optimization (solving the Rubik's Cube), multi-agent methods, Microsoft's TextWorld environment, advanced exploration techniques, and more, you will come away from this book with a deep understanding of the latest innovations in this emerging field.


In addition, you will gain actionable insights into such topic areas as deep Q-networks, policy gradient methods, continuous control problems, and highly scalable, non-gradient methods. You will also discover how to build a real hardware robot trained with RL for less than $100 and solve the Pong environment in just 30 minutes of training using step-by-step code optimization.


In short, Deep Reinforcement Learning Hands-On, Second Edition, is your companion to navigating the exciting complexities of RL as it helps you attain experience and knowledge through real-world examples.

  • Understand the deep learning context of RL and implement complex deep learning models
  • Evaluate RL methods including cross-entropy, DQN, actor-critic, TRPO, PPO, DDPG, D4PG, and others
  • Build a practical hardware robot trained with RL methods for less than $100
  • Discover Microsoft's TextWorld environment, which is an interactive fiction games platform
  • Use discrete optimization in RL to solve a Rubik's Cube
  • Teach your agent to play Connect 4 using AlphaGo Zero
  • Explore the very latest deep RL research on topics including AI chatbots
  • Discover advanced exploration techniques, including noisy networks and network distillation techniques
  • Second edition of the bestselling introduction to deep reinforcement learning, expanded with six new chapters
  • Learn advanced exploration techniques including noisy networks, pseudo-count, and network distillation methods
  • Apply RL methods to cheap hardware robotics platforms


Maxim Lapan is a deep learning enthusiast and independent researcher. His background and 15 years' work expertise as a software developer and a systems architect lies from low-level Linux kernel driver development to performance optimization and design of distributed applications working on thousands of servers. With vast work experiences in big data, machine learning, and large parallel distributed HPC and non-HPC systems, he is able to explain a number of complicated concepts in simple words and vivid examples. His current areas of interest are in practical applications of deep learning, such as deep natural language processing and deep reinforcement learning. Maxim lives in Moscow, Russian Federation, with his family.


  1. What Is Reinforcement Learning?
  2. OpenAI Gym
  3. Deep Learning with PyTorch
  4. The Cross-Entropy Method
  5. Tabular Learning and the Bellman Equation
  6. Deep Q-Networks
  7. Higher-Level RL libraries
  8. DQN Extensions
  9. Ways to Speed up RL
  10. Stocks Trading Using RL
  11. Policy Gradients – an Alternative
  12. The Actor-Critic Method
  13. Asynchronous Advantage Actor-Critic
  14. Training Chatbots with RL
  15. The TextWorld environment
  16. Web Navigation
  17. Continuous Action Space
  18. RL in Robotics
  19. Trust Regions – PPO, TRPO, ACKTR, and SAC
  20. Black-Box Optimization in RL
  21. Advanced exploration
  22. Beyond Model-Free – Imagination
  23. AlphaGo Zero
  24. RL in Discrete Optimisation
  25. Multi-agent RL