(zhuan) 一些RL的文献（及笔记）

一些RL的文献（及笔记）

copy from: https://zhuanlan.zhihu.com/p/25770890

Introductions

Introduction to reinforcement learning
Index of /rowan/files/rl

ICML Tutorials:
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

NIPS Tutorials:
CS 294 Deep Reinforcement Learning, Spring 2017
https://drive.google.com/file/d/0B_wzP_JlVFcKS2dDWUZqTTZGalU/view

Deep Q-Learning

DQN:
[1312.5602] Playing Atari with Deep Reinforcement Learning (and its nature version)

Double DQN
[1509.06461] Deep Reinforcement Learning with Double Q-learning

Bootstrapped DQN
[1602.04621] Deep Exploration via Bootstrapped DQN

Priority Experienced Replay
http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Applications_files/prioritized-replay.pdf

Duel DQN
[1511.06581] Dueling Network Architectures for Deep Reinforcement Learning

Classic Literature

SuttonBook
http://people.inf.elte.hu/lorincz/Files/RL_2006/SuttonBook.pdf
Book

David Silver's thesis
http://www0.cs.ucl.ac.uk/staff/d.silver/web/Publications_files/thesis.pdf

Policy Gradient Methods for Reinforcement Learning with Function Approximation
https://webdocs.cs.ualberta.ca/~sutton/papers/SMSM-NIPS99.pdf
(Policy gradient theorem)

1. Policy-based approach is better than value based: policy function is smooth, while using value function to pick policy is not continuous.

2. Policy Gradient method.
Objective function is averaged on the stationary distribution (starting from s0).
For average reward, it needs to be truly stationary.
For state-action (with discount), if all experience starts with s0, then the objective is averaged over a discounted distribution (not necessarily fully-stationary). If we starts with any arbitrary state, then the objective is averaged over the (discounted) stationary distribution.
Policy gradient theorem: gradient operator can “pass” through the state distribution, which is dependent on the parameters (and at a first glance, should be taken derivatives, too).

3. You can replace Q^\pi(s, a) with an approximate, which is only accurate when the approximate f(s, a) satisfies df/dw = d\pi/d\theta /\pi
If pi(s, a) is loglinear wrt some features, then f has to be linear to these features and \sum_a f(s, a) = 0 (So f is an advantage function).

4. First time to show the RL algorithm converges to a local optimum with relatively free-form functional estimator.

DAgger
https://www.cs.cmu.edu/~sross1/publications/Ross-AIStats10-paper.pdf

Actor-Critic Models

Asynchronous Advantage Actor-Critic Model
[1602.01783] Asynchronous Methods for Deep Reinforcement Learning

Tensorpack's BatchA3C (ppwwyyxx/tensorpack) and GA3C ([1611.06256] Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU)
Instead of using a separate model for each actor (in separate CPU threads), they process all the data generated by actors with a single model, which is updated regularly via optimization.

On actor-critic algorithms.
http://www.mit.edu/~jnt/Papers/J094-03-kon-actors.pdf
Only read the first part of the paper. It proves that actor-critic will converge to the local minima, when the feature space used to linearly represent Q(s, a) also covers the space spanned by \nabla log \pi(a|s) (compatibility condition), and the actor learns slower than the critic.

https://dev.spline.de/trac/dbsprojekt_51_ss09/export/74/ki_seminar/referenzen/peters-ECML2005.pdf
Natural Actor-Critic
Natural gradient is applied on actor critic method. When the compatibility condition proposed by the policy gradient paper is satisfied (i.e., Q(s, a) is a linear function with respect to \nabla log pi(a|s), so that the gradient estimation using this estimated Q is the same as the true gradient which uses the unknown perfect Q function computed from the ground truth policy), then the natural gradient of the policy's parameters is just the linear coefficient of Q.

A Survey of Actor-Critic Reinforcement Learning Standard and Natural Policy Gradients
https://hal.archives-ouvertes.fr/hal-00756747/document
Covers the above two papers.

Continuous State/Action

Reinforcement Learning with Deep Energy-Based Policies
Use the soft-Q formulation proposed by https://arxiv.org/pdf/1702.08892.pdf (in the math section) and naturally incorporate the entropy term in the Q-learning paradigm. For continuous space, both the training (updating Bellman equation) and sampling from the resulting policy (in terms of Q) are intractable. For the former, they propose to use a surrogate action distribution, and compute the gradient with importance sampling. For the latter, they use Stein variational method that matches a deterministic function a = f(e, s) to the learned Q-distribution. In terms of performance, they are comparable with DDPG. But since the learnt Q could be diverse (multimodal) under maximal entropy principle, it can be used as a common initialization for many specific tasks (Example, pretrain=learn to run towards arbitrary direction, task=run in a maze).

Deterministic Policy Gradient Algorithms
http://jmlr.org/proceedings/papers/v32/silver14.pdf
Silver's paper. Learn an actor to prediction the deterministic action (rather than a conditional probability distribution \pi(a|s)) in Q-learning. When trained with Q-learning, propagate through Q to \pi. Similar to Policy Gradient Theorem (gradient operator can “pass” the state distribution, which is dependent on the parameters), there is also deterministic version of it. Also interesting comparison with stochastic offline actor-critic model (stochastic = \pi(a|s)).

Continuous control with deep reinforcement learning (DDPG)
Deep version of DPG (with DQN trick). Neural network + minibatch → not stable, so they also add target network and replay buffer.

Reward Shaping

Policy invariance under reward transformations: theory and application to reward shaping.
http://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf
Andrew Ng's reward shaping paper. It proves that for reward shaping, policy is invariant if and only if a difference of a potential function is added to the reward.

Theoretical considerations of potential-based reward shaping for multi-agent systems
Theoretical considerations of potential-based reward shaping for multi-agent systems
Potential based reward-shaping can help a single-agent achieve optimal solution without changing the value (or Nash Equilibrium). This paper extends it to multi-agent case.

Reinforcement Learning with Unsupervised Auxiliary Tasks
[1611.05397] Reinforcement Learning with Unsupervised Auxiliary Tasks
ICLR17 Oral. Add auxiliary task to improve the performance of Atari Games and Navigation. Auxiliary task includes maximizing pixel changes and maximizing the activation of individual neurons.

Navigation

Learning to Navigate in Complex Environments
https://openreview.net/forum?id=SJMGPrcle¬eId=SJMGPrcle
Raia's group from DM. ICLR17 poster, adding depth prediction as the auxiliary task and improve the navigation performance (also uses SLAM results as network input)

[1611.05397] Reinforcement Learning with Unsupervised Auxiliary Tasks (in reward shaping)

Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments
Goal: navigation without SLAM.
Learn successor features (Q, V before the last layer, these features have a similar Bellman equation.) for transfer learning: learn k top weights simultaneously while sharing the successor features, using DQN acting on the features). In addition to successor features, also try to reconstruct the frame.

Experiments on simulation.
state: 96x96x four most recent frames.
action: four discrete actions. (still, left, right, straight(1m))
baseline: train a CNN to directly predict the action of A*

Deep Recurrent Q-Learning for Partially Observable MDPs
There is no much performance difference between stacked frame DQN versus DRQN. DRQN may be more robust when the game state is flickered (some are 0)