# Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimism

←

→

**Page content transcription**

If your browser does not render page correctly, please read the page content below

Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimism Wang Chi Cheung 1 David Simchi-Levi 2 Ruihao Zhu 2 Abstract imizes its cumulative rewards, while facing the following challenges: We consider un-discounted reinforcement learn- ing (RL) in Markov decision processes (MDPs) • Endogeneity: At each time step, the reward follows a under drifting non-stationarity, i.e., both the re- reward distribution, and the subsequent state follows a ward and state transition distributions are allowed state transition distribution. Both distributions depend to evolve over time, as long as their respective (solely) on the current state and action, which are influ- total variations, quantified by suitable metrics, do enced by the policy. Hence, the environment can be fully not exceed certain variation budgets. We first characterized by a discrete time Markov decision process develop the Sliding Window Upper-Confidence (MDP). bound for Reinforcement Learning with Confi- • Exogeneity: The reward and state transition distributions dence Widening (SWUCRL2-CW) algorithm, and vary (independently of the policy) across time steps, but establish its dynamic regret bound when the vari- the total variations are bounded by the respective variation ation budgets are known. In addition, we propose budgets. the Bandit-over-Reinforcement Learning (BORL) • Uncertainty: Both the reward and state transition distri- algorithm to adaptively tune the SWUCRL2-CW butions are initially unknown to the DM. algorithm to achieve the same dynamic regret • Bandit/Partial Feedback: The DM can only observe the bound, but in a parameter-free manner, i.e., with- reward and state transition resulted by the current state out knowing the variation budgets. Notably, learn- and action in each time step. ing non-stationary MDPs via the conventional op- It turns out that many applications, such as real-time bid- timistic exploration technique presents a unique ding in advertisement (ad) auctions, can be captured by this challenge absent in existing (non-stationary) ban- framework (Cai et al., 2017; Flajolet & Jaillet, 2017; Bal- dit learning settings. We overcome the challenge seiro & Gur, 2019; Guo et al., 2019; Han et al., 2020). by a novel confidence widening technique that Besides, this framework can be used to model sequen- incorporates additional optimism. tial decision-making problems in transportation (Zhang & Wang, 2018; Qin et al., 2019), wireless network (Zhou & Bambos, 2015; Zhou et al., 2016), consumer choice model- 1. Introduction ing (Xu & Yun, 2020), ride-sharing (Taylor, 2018; Gurvich et al., 2018; Bimpikis et al., 2019; Kanoria & Qian, 2019), Consider a general sequential decision-making framework, healthcare operations (Shortreed et al., 2010), epidemic con- where a decision-maker (DM) interacts with an initially un- trol (Nowzari et al., 2016; Kiss et al., 2017), and inventory known environment iteratively. At each time step, the DM control (Huh & Rusmevichientong, 2009; Bertsekas, 2017; first observes the current state of the environment, and then Zhang et al., 2018; Agrawal & Jia, 2019; Chen et al., 2019a). chooses an available action. After that, she receives an in- stantaneous random reward, and the environment transitions There exists numerous works in sequential decision-making to the next state. The DM aims to design a policy that max- that considered part of the four challenges. The traditional stream of research (Auer et al., 2002a; Bubeck & Cesa- 1 Department of Industrial Systems Engineering and Manage- Bianchi, 2012; Lattimore & Szepesvári, 2018) on stochastic ment, National University of Singapore 2 MIT Institute for Data, multi-armed bandits (MAB) focused on the interplay be- Systems, and Society. Correspondence to: Wang Chi Cheung , David Simchi-Levi , tween uncertainty and bandit feedback (i.e., challenges 3 Ruihao Zhu . and 4), and (Auer et al., 2002a) proposed the classical Upper Confidence Bound (UCB) algorithm. Starting from (Bur- Proceedings of the 37 th International Conference on Machine netas & Katehakis, 1997; Tewari & Bartlett, 2008; Jaksch Learning, Vienna, Austria, PMLR 119, 2020. Copyright 2020 by et al., 2010), a volume of works (see Section 3) have been the author(s).

Non-Stationary Reinforcement Learning Stationary Non-stationary MAB OFU (Auer et al., 2002a) OFU + Forgetting (Besbes et al., 2014; Cheung et al., 2019b) RL OFU (Jaksch et al., 2010) Extra optimism + Forgetting (This paper) Table 1. Summary of algorithmic frameworks of stationary and non-stationary online learning settings. devoted to reinforcement learning (RL) in MDPs (Sutton 2010) to RL in non-stationary MDPs may result in poor & Barto, 2018), which further involves endogeneity. RL in dynamic regret bounds. MDPs incorporate challenges 1,3,4, and stochastic MAB is a special case of MDPs when there is only one state. In the 1.1. Summary of Main Contributions absence of exogeneity, the reward and state transition distri- butions are invariant across time, and these three challenges Assuming that, during the T time steps, the total variations can be jointly solved by the Upper Confidence bound for of the reward and state transition distributions are bounded Reinforcement Learning (UCRL2) algorithm (Jaksch et al., (under suitable metrics) by the variation budgets Br (> 0) 2010). and Bp (> 0), respectively, we design and analyze novel algorithms for RL in non-stationary MDPs. Let Dmax , S, The UCB and UCRL2 algorithms leverage the optimism and A be respectively the maximum diameter (a complexity in face of uncertainty (OFU) principle to select actions it- measure to be defined in Section 2), number of states, and eratively based on the entire collections of historical data. number of actions in the MDP. Our main contributions are: However, both algorithms quickly deteriorate when exo- • We develop the Sliding Window UCRL2 with Con- geneity emerge since the environment can change over time, fidence Widening (SWUCRL2-CW) algorithm. When and the historical data becomes obsolete. To address the the! variation budgets are known, we "prove it attains a challenge of exogeneity, (Garivier & Moulines, 2011b) con- Õ Dmax (Br + Bp )1/4 S 2/3 A1/2 T 3/4 dynamic regret sidered the piecewise-stationary MAB environment where bound via a budget-aware analysis. the reward distributions remain unaltered over certain time • We propose the Bandit-over-Reinforcement Learning periods and change at unknown time steps. Later on, there is (BORL) algorithm that tunes the SWUCRL2-CW a line of research initiated by (Besbes et al., 2014) that stud- algorithm adaptively, and retains the same ied the general non-stationary MAB environment (Besbes ! " Õ Dmax (Br + Bp )1/4 S 2/3 A1/2 T 3/4 dynamic et al., 2014; Cheung et al., 2019a;b), in which the reward regret bound without knowing the variation budgets. distributions can change arbitrarily over time, but the total • We identify an unprecedented challenge for RL in non- changes (quantified by a suitable metric) is upper bounded stationary MDPs with conventional optimistic exploration by a variation budget (Besbes et al., 2014). The aim is to techniques: existing algorithmic frameworks for non- minimize the dynamic regret, the optimality gap compared stationary online learning (including non-stationary ban- to the cumulative rewards of the sequence of optimal actions. dit and RL in piecewise-stationary MDPs) (Jaksch et al., Both the (relatively restrictive) piecewise-stationary MAB 2010; Garivier & Moulines, 2011b; Cheung et al., 2019a) and the general non-stationary MAB settings consider the typically estimate unknown parameters by averaging his- challenges of exogeneity, uncertainty, and partial feedback torical data in a “forgetting” fashion, and construct the (i.e., challenges 2, 3, 4), but endogeneity (challenge 1) are tightest possible confidence regions/intervals accordingly. not present. They then optimistically search for the most favorable In this paper, to address all four above-mentioned chal- model within the confidence regions, and execute the lenges, we consider RL in non-stationary MDPs where bot corresponding optimal policy. However, we first demon- the reward and state transition distributions can change over strate in Section 4.1, and then rigorously show in Section time, but the total changes (quantified by suitable metrics) 6 that in the context of RL in non-stationary MDPs, the are upper bounded by the respective variation budgets. We diameters induced by the MDPs in the confidence regions note that in (Jaksch et al., 2010), the authors also consider constructed in this manner can grow wildly, and may re- the intermediate RL in piecewise-stationary MDPs. Never- sult in unfavorable dynamic regret bound. We overcome theless, we first demonstrate in Section 4.1, and then rigor- this with our novel proposal of extra optimism via the ously show in Section 6 that simply adopting the techniques confidence widening technique (alternatively, in (Cheung for non-stationary MAB (Besbes et al., 2014; Cheung et al., et al., 2020a), an extended version of the current paper, 2019a;b) or RL in piecewise-stationary MDPs (Jaksch et al., the authors demonstrate that one can leverage special

Non-Stationary Reinforcement Learning structures on the state transition distributions in the con- DM, individual Br,t ’s and Bp,t ’s are unknown to the DM text of single item inventory control with fixed cost to throughout the current paper. bypass this difficulty of exploring time-varying environ- Endogeneity: The DM faces a non-stationary MDP in- ments). A summary of the algorithmic frameworks for stance (S, A, T, r, p). She knows S, A, T , but not r, p. stationary and non-stationary online learning settings are The DM starts at an arbitrary state s1 œ S. At time t, provided in Table 1. three events happen. First, the DM observes its current state st . Second, she takes an action at œ Ast . Third, 2. Problem Formulation given st , at , she stochastically transits to another state st+1 which is distributed as pt (·|st , at ), and receives a stochas- In this section, we introduce the notations to be used tic reward Rt (st , at ), which is 1-sub-Gaussian with mean throughout paper, and introduce the learning protocol for rt (st , at ). In the second event, the choice of at is based on our problem of RL in non-stationary MDPs. a non-anticipatory policy . That is, the choice only de- pends on the current state st and the previous observations 2.1. Notations Ht≠1 := {sq , aq , Rq (sq , aq )}t≠1 q=1 . Throughout the paper, all vectors are column vectors, unless Dynamic Regret: The q DM aims to maximize the cumula- specified otherwise. We define [n] to be the set {1, 2, . . . , n} tive expected reward E[ t=1 rt (st , at )], despite the model T for any positive integer n. We denote 1[·] as the indicator uncertainty on r, p and the dynamics of the learning envi- function. For p œ [1, Œ], we use ÎxÎp to denote the p-norm ronment. To measure the convergence to optimality, we of a vector x œ Rd . We denote x ‚ y and x · y as the max- consider an equivalent objective of minimizing the dynamic imum and minimum between x, y œ R, respectively. We regret (Besbes et al., 2014; Jaksch et al., 2010) adopt the asymptotic notations O(·), (·), and (·) (Cor- T ÿ men et al., 2009). When logarithmic factors are omitted, we use Õ(·), ˜ (·), ˜ (·), respectively. With some abuse, these Dyn-RegT ( ) = {ﬂút ≠ E[rt (st , at )]} . (2) t=1 notations are used when we try to avoid the clutter of writing qT out constants explicitly. In the oracle t=1 ﬂút , the summand ﬂút is the optimal long- term average reward of the stationary MDP with state tran- 2.2. Learning Protocol sition distribution pt and mean reward rt . The optimum ﬂút can be computed by solving linear program (9) provided in Model Primitives: An instance of non-stationary MDP Section A.1. We note that the same oracle is used for RL in is specified by the tuple (S, A, T, r, p). The set S is a fi- piecewise-stationary MDPs (Jaksch et al., 2010). nite set of states. The collection A = {As }sœS contains a finite action set As for each state s œ S. We say that Remark 1. When S = 1, (2) reduces to the definition (s, a) is a state-action pair (Besbes et al., 2014) of dynamic regret for non-stationary q if s œ S, a œ As . We de- K-armed bandits. Nevertheless, note S = |S|, A = ( sœS |As |)/S. We denote T as qT different from the bandit the total number of time steps, and denote r = {rt }Tt=1 case, the offline benchmark t=1 ﬂút does not equal to the as the sequence of mean rewards. For each t, we have expected optimum for the non-stationary MDP problem in rt = {rt (s, a)}sœS,aœAs , and rt (s, a) œ [0, 1] for each general. We justify our choice in Proposition 1. state-action pair (s, a). In addition, we denote p = {pt }Tt=1 Next, we review concepts of communicating MDPs and as the sequence of state transition distributions. For each diameters, in order to stipulate an assumption that ensures t, we have pt = {pt (·|s, a)}sœS,aœAs , where pt (·|s, a) is learnability and justifies our offline bnechmark. a probability distribution over S for each state-action pair (s, a). Definition 1 ((Jaksch et al., 2010) Communicating MDPs and Diameters). Consider a set of states S, Exogeneity: The quantities rt ’s and pt ’s vary across dif- a collection A = {As }sœS of action sets, and a ferent t’s in general. Following (Besbes et al., 2014), we transition kernel p̄ = {p̄(·|s, a)}sœS,aœAs . For any quantify the variations on rt ’s and pt ’s in terms of their s, sÕ œ S and stationary policy ﬁ, the hitting time from respective variation budgets Br , Bp (> 0): s to sÕ under ﬁ is the random variable (sÕ |ﬁ, s) := min {t : st+1 = sÕ , s1 = s, s· +1 ≥ p̄(·|s· , ﬁ(s· )) ’· } , T ÿ ≠1 T ÿ ≠1 which can be infinite. We say that (S, A, p̄) is a Br = Br,t , Bp = Bp,t , (1) communicating MDP iff t=1 t=1 D := max min E [ (sÕ |ﬁ, s)] where Br,t = maxsœS,aœAs |rt+1 (s, a) ≠ rt (s, a)| and s,s œS stationary ﬁ Õ Bp,t = maxsœS,aœAs Îpt+1 (·|s, a) ≠ pt (·|s, a)Î1 . We em- is finite. The quantity D is the diameter associated with phasize although Br and Bp might be used as inputs by the (S, A, p̄).

Non-Stationary Reinforcement Learning We make the following assumption throughout. 3.2. RL in Non-Stationary MDPs Assumption 1. For each t œ {1, . . . , T }, the tuple In a parallel work (Ortner et al., 2019), the authors consid- (S, A, pt ) constitutes a communicating MDP with diam- ered a similar setting to ours by applying the “forgetting eter at most Dt . We denote the maximum diameter as principle” from non-stationary bandit settings (Garivier & Dmax = maxtœ{1,...,T } Dt . Moulines, 2011b; Cheung et al., 2019b) to design a learn- ing algorithm. To achieve its dynamic regret bound, the The following qT proposition justifies our choice of offline algorithm by (Ortner et al., 2019) partitions the entire time benchmark t=1 ﬂút . horizon [T ] into time intervals I = {Ik }K , and crucially qmax Ik ≠1 k=1 qmax Ik ≠1 Proposition 1. Consider an instance (S, A, T, p, r) that requires the access to t=min Ik Br,t and t=min Ik Bp,t , satisfies Assumption 1 with maximum diameter Dmax , i.e., the variations in both reward and state transition distribu- and has variation budgets Br , Bp for the rewards and tions of each interval Ik œ I (see Theorem 3 in (Ortner et al., transition kernels respectively. In addition, suppose 2019)). In contrast, the SWUCRL2-CW algorithm and the that T Ø Br +Ó2D max Bp Ëq > 0. ÈÔIt holds that BORL algorithm require significantly less information on the qT variations. Specifically, the SWUCRL2-CW algorithm does t=1 ﬂt Ø max t=1 rt (st , at ) ≠ 4(Dmax + T ú E not need any additional knowledge on the variations except 1) (Br + 2Dmax Bp )T . The maximum is taken over all for Br and Bp , i.e., the variation budgets over the entire non-anticipatory policies ’s. We denote {(st , at )}Tt=1 time horizon as defined in eqn. (1), to achieve its dynamic as the trajectory under policy , where at œ Ast is regret bound (see Theorem 1). This is similar to algorithms determined based on and Ht≠1 ﬁ {st }, and st+1 ≥ for the non-stationary bandit settings, which only require pt (·|st , at ) for each t. the access to Br (Besbes et al., 2014). More importantly, the BORL algorithm (built upon the SWUCRL2-CW algo- The Proposition is proved in Section A.2 of the full ver- rithm) enjoys the same dynamic regret bound even without sion (Cheung et al., 2020b). In fact, our dynamic re- knowing either of Br or Bp (see Theorem 2). bounds are larger than the error term 4(Dmax + gret There also exists some settings that are closely related to, 1) (Br + 2Dmax Bp )T , thus justifying the choice of qT but different than our setting (in terms of exogeneity and ﬂút as the offline benchmark. The offline benchmark qt=1 T feedback). (Jaksch et al., 2010; Gajane et al., 2018) pro- t=1 ﬂt is more convenient for analysis than the expected ú posed solutions for the RL in piecewise-stationary MDPs optimum, since the former can be decomposed to summa- setting. But as discussed before, simply applying their tech- tions across different intervals, unlike the latter where the niques to the general RL in non-stationary MDPs may result summands are intertwined (since st+1 ≥ pt (·|st , at )). in undesirable dynamic regret bounds (see Section 6 for more details). In (Yu et al., 2009; Neu et al., 2010; Arora 3. Related Works et al., 2012; Dick et al., 2014; Jin et al., 2019; Cardoso et al., 2019), the authors considered RL in MDPs with changing 3.1. RL in Stationary MDPs reward distributions but fixed transition distributions. The RL in stationary (discounted and un-discounted reward) authors of (Even-Dar et al., 2005; Yu & Mannor, 2009; Neu MDPs has been widely studied in (Burnetas & Katehakis, et al., 2012; Abbasi-Yadkori et al., 2013; Rosenberg & Man- 1997; Bartlett & Tewari, 2009; Jaksch et al., 2010; Agrawal sour, 2019; Li et al., 2019) considered RL in non-stationary & Jia, 2017; Fruit et al., 2018a;b; Sidford et al., 2018b;a; MDPs with full information feedback. Wang, 2019; Zhang & Ji, 2019; Fruit et al., 2019; Wei et al., 2019). For the discounted reward setting, the authors of (Sid- 3.3. Non-Stationary MAB ford et al., 2018b; Wang, 2019; Sidford et al., 2018a) pro- For online learning and bandit problems where there is posed (nearly) optimal algorithms in terms of sample com- only one state, the works by (Auer et al., 2002b; Garivier plexity. For the un-discounted reward setting, the authors & Moulines, 2011b; Besbes et al., 2014; Keskin & Zeevi, of (Jaksch Ô et al., 2010) established a minimax lower bound 2016) proposed several “forgetting” strategies for different ( Dmax SAT ) on the regret when both the reward and non-stationary MAB settings. More recently, the works by state transition distributions are time-invariant. They also (Karnin & Anava, 2016; Luo et al., 2018; Cheung et al., designed the UCRL2 algorithm Ô and showed that it attains a 2019a;b; Chen et al., 2019b) designed parameter-free algo- regret bound Õ(Dmax S AT ). The authors of (Fruit et al., rithms for non-stationary MAB problems. Another related 2019) proposed the UCRL2B algorithm, which is an im- but different setting is the Markovian bandit (Kim & Lim, proved version of the UCRL2 algorithm. Ô The regret bound 2 2016; Ma, 2018), in which the state of the chosen action of the UCRL2B algorithm is Õ(S Dmax AT +Dmax S 2 A). evolve according to an independent time-invariant Markov The minimax optimal algorithm is provided in (Zhang & Ji, chain while the states of the remaining actions stay un- 2019) although it is not computationally efficient.

Non-Stationary Reinforcement Learning changed. In (Zhou et al., 2020), the authors also considered the diameter of the estimated MDP produced by the naive the case when the states of all the actions are governed by sliding-window UCRL2 algorithm with window size W the same (uncontrollable) Markov chain. can be as large as (W ), which is orders of magnitude larger than Dmax , the maximum diameter of each individ- 4. Sliding Window UCRL2 with Confidence ual MDP encountered by the DM. Consequently, the naive sliding-window UCRL2 algorithm may result in undesirable Widening dynamic regret bound. In what follows, we discuss in more In this section, we present the SWUCRL2-CW algorithm, details how our novel confidence widening technique can which incorporates sliding window estimates (Garivier & mitigate this issue. Moulines, 2011a) and a novel confidence widening tech- nique into UCRL2 (Jaksch et al., 2010). 4.2. Design Overview The SWUCRL2-CW algorithm first specifies a sliding win- 4.1. Design Challenge: Failure of Naive Sliding dow parameter W œ N and a confidence widening parame- Window UCRL2 Algorithm ter ÷ Ø 0. Parameter W specifies the number of previous For stationary MAB problems, the UCB algorithm (Auer time steps to look at. Parameter ÷ quantifies the amount of et al., 2002a) suggests the DM should iteratively execute additional optimistic exploration, on top of the conventional the following two steps in each time step: optimistic exploration using upper confidence bounds. The former turns out to be necessary for handling the drifting 1. Estimate the mean reward of each action by taking the non-stationarity of the transition kernel. time average of all observed samples. 2. Pick the action with the highest estimated mean reward The algorithm runs in a sequence of episodes that partitions plus the confidence radius, where the radius scales in- the T time steps. Episode m starts at time · (m) (in partic- versely proportional with the number of observations ular · (1) = 1), and ends at the end of time · (m + 1) ≠ 1. (Auer et al., 2002a). Throughout an episode m, the DM follows a certain station- ary policy ﬁ̃m . The DM ceases the mth episode if at least The UCB algorithm has been proved to attain optimal regret one of the following two criteria is met: bounds for various stationary MAB settings (Auer et al., 2002a; Kveton et al., 2015). For non-stationary problems, • The time index t is a multiple of W. Consequently, each (Garivier & Moulines, 2011b; Keskin & Zeevi, 2016; Che- episode last for at most W time steps. The criterion en- ung et al., 2019b) shown that the DM could further leverage sures that the DM switches the stationary policy ﬁ̃m fre- the forgetting principle by incorporating the sliding-window quently enough, in order to adapt to the non-stationarity estimator (Garivier & Moulines, 2011b) into the UCB algo- of rt ’s and pt ’s. rithms (Auer et al., 2002a; Kveton et al., 2015) to achieve • There exists some state-action pair (s, a) such that optimal dynamic regret bounds for a wide variety of non- ‹m (s, a), the number of time step t’s with (st , at ) = stationary MAB settings. The sliding window UCB algo- (s, a) within episode m, is at least as many as the total rithm with a window size W œ R+ is similar to the UCB number of counts for it within the W time steps prior to algorithm except that the estimated mean rewards are com- · (m), i.e., from (· (m) ≠ W ) ‚ 1 to (· (m) ≠ 1). This is puted by taking the time average of the W most recent similar to the doubling criterion in (Jaksch et al., 2010), observed samples. which ensures that each episode is sufficiently long so that the DM can focus on learning. As noted in Section 1, (Jaksch et al., 2010) proposed the UCRL2 algorithm, which is a UCB-alike algorithm with The combined effect of these two criteria allows the DM nearly optimal regret for RL in stationary MDPs. It is thus to learn a low dynamic regret policy with historical data tempting to think that one could also integrate the forgetting from an appropriately sized time window. One important principle into the UCRL2 algorithm to attain low dynamic piece of ingredient is the construction of the policy ﬁ̃m , for regret bound for RL in non-stationary MDPs. In particular, each episode m. To allow learning under non-stationarity, one could easily design a naive sliding-window UCRL2 the SWUCRL2-CW algorithm computes the policy ﬁ̃m based algorithm that follows exactly the same steps as the UCRL2 on the history in the W time steps previous to the current algorithm with the exception that it uses only the W most episode m, i.e., from round (· (m)≠W )‚1 to round · (m)≠ recent observed samples instead of all observed samples to 1. The construction of ﬁ̃m involves the Extended Value estimate the mean rewards and the state transition distribu- Iteration (EVI) (Jaksch et al., 2010), which requires the tions, and to compute the respective confidence radius. confidence regions Hr,· (m) , Hp,· (m) (÷) for rewards and transition kernels as the inputs, in addition to an precision Under non-stationarity and bandit feedback, however, we parameter ‘. The confidence widening parameter ÷ Ø 0 show in Proposition 3 of the forthcoming Section 6 that is capable of ensuring the MDP output by the EVI has a

Non-Stationary Reinforcement Learning bounded diameter most of the time. radp,t (s, a) = 2 2S log (SAT /”)/Nt+ (s, a). 4.3. Policy Construction In a nutshell, the incorporation of ÷ > 0 provides an addi- tional source of optimism, and the DM can explore transition To describe SWUCRL2-CW algorithm, we define for each kernels that further deviate from the sample average. This state action pair (s, a) and each round t in episode m, turns out to be crucial for learning MDPs under drifting t≠1 non-stationarity. We treat ÷ as a hyper-parameter at the mo- ÿ Nt (s, a) = 1((sq , aq ) = (s, a)), ment, and provide a suitable choice of ÷ when we discuss q=(· (m)≠W )‚1 our main results. Nt+ (s, a) = max{1, Nt (s, a)}. (3) 4.3.3. E XTENDED VALUE I TERATION (EVI) (JAKSCH ET AL ., 2010). 4.3.1. C ONFIDENCE R EGION FOR R EWARDS . The SWUCRL2-CW algorithm relies on the EVI, which For each state action pair (s, a) and each time step t in solves MDPs with optimistic exploration to near-optimality. episode m, we consider the empirical mean estimator We extract and rephrase a description of EVI in Section A.3 of the full version (Cheung et al., 2020b). EVI inputs the Rq (s, a) 1(sq = s, aq = a) t≠1 ÿ r̂t (s, a) = , confidence regions Hr , Hp for the rewards and the transition q=(· (m)≠W )‚1 Nt+ (s, a) kernels. The algorithm outputs an “optimistic MDP model”, which consists of reward vector r̃ and transition kernel p̃ which serves to estimate the average reward under which the optimal average gain ﬂ̃ is the largest among all ṙ œ Hr , ṗ œ Hp : rq (s, a)1(sq = s, aq = a) t≠1 ÿ r̄t (s, a) = . • Input: Confidence regions Hr for r, Hp for p, and an Nt+ (s, a) q=(· (m)≠W )‚1 error parameter ‘ > 0. • Output: The returned policy ﬁ̃ and the auxiliary out- The confidence region Hr,t = {Hr,t (s, a)}sœS,aœAs is de- put (r̃, p̃, ﬂ̃, “˜ ). In the latter, r̃, p̃, and ﬂ̃ are the selected fined as “optimistic” reward vector, transition kernel, and the corre- Hr,t (s, a) = {ṙ œ [0, 1] : |ṙ ≠ r̂t (s, a)| Æ radr,t (s, a)} , sponding long term average reward. The output “˜ œ RØ0 S (4) is a bias vector (Jaksch et al., 2010). For each s œ S, the with confidence radius quantity “˜ (s) is indicative of the short term reward when the DM starts at state s and follows the optimal policy. radr,t (s, a) = 2 2 log(SAT /”)/Nt+ (s, a). By the design of EVI, for the output “˜ , there exists s œ S such that “˜ (s) = 0. Altogether, we express 4.3.2. C ONFIDENCE W IDENING FOR T RANSITION K ERNELS . EVI(Hr , Hp ; ‘) æ (ﬁ̃, r̃, p̃, ﬂ̃, “˜ ). For each state action pair s, a and each time step t in episode Combining the three components, a formal description of m, consider the empirical estimator the SWUCRL2-CW algorithm is shown in Algorithm 1. 1(sq = s, aq = a, sq+1 = sÕ ) t≠1 ÿ p̂t (sÕ |s, a) = , 4.4. Performance Analysis: The Blessing of More q=(· (m)≠W )‚1 Nt+ (s, a) Optimism which serves to estimate the average transition probability We now analyze the performance of the SWUCRL2-CW al- gorithm. First, we introduce two events Er , Ep , which state t≠1 ÿ pq (sÕ |s, a)1(sq = s, aq = a) that the estimated reward and transition kernels lie in the p̄t (sÕ |s, a) = . respective confidence regions. q=(· (m)≠W )‚1 Nt+ (s, a) Er = {r̄t (s, a) œ Hr,t (s, a) ’s, a, t}, Different from the case of estimating reward, the confidence region Hp,t (÷) = {Hp,t (s, a; ÷)}sœS,aœAs for the transi- Ep = {p̄t (·|s, a) œ Hp,t (s, a; 0) ’s, a, t}. tion probability involves a widening parameter ÷ Ø 0: We prove that Er , Ep hold with high probability. Hp,t (s, a; ÷) (5) Lemma 1. We have Pr[Er ] Ø 1 ≠ ”/2, Pr[Ep ] Ø 1 ≠ ”/2. ={ṗ œ S : Îṗ(·|s, a) ≠ p̂t (·|s, a)Î1 Æ radp,t (s, a) + ÷}, The proof is provided in Section B of the full version (Che- with confidence radius ung et al., 2020b). In defining Ep , the widening parameter

Non-Stationary Reinforcement Learning Algorithm 1 SWUCRL2-CW algorithm Our subsequent analysis shows that ÷ can be suitably cali- 1: Input: Time horizon T , state space S, action space A, brated so that D = O(Dmax ). Next, we state our first main window size W , widening parameter ÷. result, which provides a dynamic regret bound assuming the 2: Initialize t Ω 1, initial state s1 . knowledge of Br , Bp to set W, ÷: 3: for episode m = 1, 2, . . . do Theorem 1. Assuming S > 1, the SWUCRL2-CW algo- 4: Set · (m) Ω t, ‹m (s, a) Ω 0, and N· (m) (s, a) ac- rithm with window size W and confidence widening param- cording to Eqn (3), for all s, a. eter ÷ > 0 satisfies the dynamic regret bound 5: Compute the confidence regions Hr,· (m) , 1 Ô Ô Hp,· (m) (÷) according to Eqns (4, 5). Õ Bp W /÷ + Br W + SAT / W 6: Compute a 1/ · (m)-optimal optimistic policy Ë Ô Ô Ô È2 +Dmax Bp W + S AT / W + T ÷ + SAT /W + T , ﬁ̃m : EVI(Hr,· (m) , Hp,· (m) (÷); 1/ · (m)) æ (ﬁ̃m , r̃m , p̃m , ﬂ̃m , “˜m ). with probability 1 ≠ O(”). Putting W = W ú := 7: while t is not a multiple of W and ‹m (st , ﬁ̃m (st )) < 2 1 1 1 3S A T /(Br + Bp + 1) and ÷ = ÷ú := N·+(m) (st , ﬁ̃m (st )) do 3 2 2 2 (Bp + 1)W /T , the bound specializes to ú 8: Choose action at = ﬁ̃m (st ), observe reward Rt (st , at ) and the next state st+1 . 1 1 2 1 3 2 Õ Dmax (Br + Bp + 1) 4 S 3 A 2 T 4 . (8) 9: Update ‹m (st , at ) Ω ‹m (st , at ) + 1, t Ω t + 1. 10: if t > T then 11: The algorithm is terminated. Proof Sketch. The complete proof is presented in Section 12: end if D of the full version (Cheung et al., 2020b). Proposition 13: end while 2 states that if the confidence region Hp,· (m) (÷) contains 14: end for a transition kernel that induces a MDP with bounded di- ameter D, the EVI supplied with Hp,· (m) (÷) can return a policy with controllable dynamic regret bound. However, ÷ is set to be 0, since we are only concerned with the esti- as we show in Section 6, one in general cannot expect this mation error on p. Next, we bound the dynamic regret of to happen. Nevertheless, we bypass this with our novel each time step, under certain assumptions on Hp,t (÷). To confidence widening technique and a budget-aware analy- facilitate our discussion, we define the following variation sis. We consider the first time step · (m) of each episode measure for each t in an episode m: m : if p· (m) (·|s, a) œ Hp,· (m) (s, a; ÷) for all (s, a), then Proposition 2 can be leveraged; otherwise, the widened t≠1 ÿ t≠1 ÿ confidence region enforces that a considerable amount of varr,t = Br,q , varp,t = Bp,q . variation budget is consumed. q=· (m)≠W q=· (m)≠W Remark 2. When S = {s}, our problem becomes the Proposition 2. Consider an episode m. Condition on non-stationary bandit problem studied by (Besbes et al., events Er , Ep , and suppose that there exists a transition 2014), and we have Dmax = 0 and Bp = 0. By choos- kernel p satisfying two properties: (1) ’s œ S ’a œ As , we 2/3 ing W = W ú = A1/3 T 2/3 /Br , our algorithm has dy- have p(·|s, a) œ Hp,· (m) (s, a; ÷), and (2) the diameter of 1/3 1/3 2/3 (S, A, p) at most D. Then, for every t œ {· (m), . . . , · (m + namic regret Õ(Br A T ), matching the minimax op- 1) ≠ 1} in episode m, we have timal dynamic regret bound by (Besbes et al., 2014) when Br œ [A≠1 , A≠1 T ]. C D ÿ Remark 3. Similar to (Cheung et al., 2019b), if Bp , Br ﬂút ≠ rt (st , at ) Æ pt (sÕ |st , at )˜ “· (m) (sÕ ) ≠ “˜· (m) (st ) are not known, wecan set W and ÷ obliviously as W = 2 1 1 2 1 1 sÕ œS S 3 A 2 T 2 , ÷ = W/T 1 = S 3 A 2 T ≠ 2 to obtain a2dy- (6) 2 1 3 namic regret bound Õ Dmax (Br + Bp + 1)S 3 A 2 T 4 . 1 + + [2varr,t + 4D(varp,t + ÷)] · (m) # $ + 2radr,· (m) (st , at ) + 4D · radp,· (m) (s, a) . (7) 5. Bandit-over-Reinforcement Learning: Towards Parameter-Free The complete proof is in Section C of the full version (Che- As said in Remark 3, in the case of unknown Br and Bp , ung et al., 2020b). Unlike Lemma 1, the parameter ÷ plays the dynamic regret of SWUCRL2-CW algorithm scales lin- an important role in the Proposition. As ÷ increases, the early in Br and Bp , which leads to a (T ) dynamic regret confidence region Hp,· (m) (s, a; ÷) becomes larger for each when Br or Bp = (T 1/4 ). In comparison, Theorem 1 s, a, and the assumed diameter D is expected to decrease. assures us that by using (W ú , ÷ ú ), we can achieve a o(T )

Non-Stationary Reinforcement Learning dynamic regret when Br , Bp = o(T ). For the bandit set- 6. The Perils of Drift in Learning Markov ting, (Cheung et al., 2019b) proposes the bandit-over-bandit Decision Processes framework that uses a separate copy of EXP3 algorithm to tune the window length. Inspired by it, we develop a In stochastic online learning problems, one usually estimates novel Bandit-over-Reinforcement Learning (BORL) algo- a latent quantity by taking the time average of observed rithm, which is parameter free and has dynamic regret bound samples, even when the sample distribution varies across equal to (8). Following (Cheung et al., 2019b), we view time. This has been proved to work well in stationary and the SWUCRL2-CW algorithm as a sub-routine, and “hedge” non-stationary bandit settings (Auer et al., 2002a; Garivier (Bubeck & Cesa-Bianchi, 2012) against the (possibly adver- & Moulines, 2011b; Cheung et al., 2019b;a). To extend to sarial) changes of rt ’s and pt ’s to identify a reasonable fixed RL, it is natural to consider the sample average transition window length and confidence widening parameter. As illus- distribution p̂t , which uses the data in the previous W rounds to estimate the time average transition distribution p̄t to within an additive error Õ(1/ Nt+ (s, a)) (see Lemma 1). In the case of stationary MDPs, where ’ t œ [T ] pt = p, one has p̄t = p. Thus, the un-widened confidence region Hp,t (0) contains p with high probability (see Lemma 1). Consequently, the UCRL2 algorithm by (Jaksch et al., 2010), which optimistic explores Hp,t (0), has a regret that scales linearly with the diameter of p. The approach of optimistic exploring Hp,t (0) is further ex- Figure 1. Structure of the BORL algorithm tended to RL in piecewise-stationary MDPs by (Jaksch et al., 2010; Gajane et al., 2018). The latter establishes a 2/3 O(¸1/3 Dmax S 2/3 A1/3 T 2/3 ) dynamic regret bounds, when trated in Fig. 1, the BORL algorithm divides the whole time there are at most ¸ changes. Their analyses involve partition- horizon into ÁT /HË blocks of equal length H rounds (the ing the T -round horizon into C · T 1/3 equal-length intervals, length of the last block can Æ H), and specifies a set J from where C is a constant dependent on Dmax , S, A, ¸. At least which each pair of (window length, confidence widening pa- CT 1/3 ≠¸ intervals enjoy stationary environments, and opti- rameter) are drawn from. For each block i œ [ÁT /HË], the mistic exploring Hp,t (0) in these intervals yields a dynamic BORL algorithm first calls some master algorithm to select regret bound that scales linearly with Dmax . Bounding the a pair of (window length, confidence widening parameter) dynamic regret of the remaining intervals by their lengths (Wi , ÷i ) œ J, and restarts the SWUCRL2-CW algorithm with and tuning C yield the desired bound. the selected parameters as a sub-routine to choose actions for this block. Afterwards, the total reward of block i is fed In contrast to the stationary and piecewise-stationary set- back to the master, and the “posterior” of these parameters tings, optimistic exploration on Hp,t (0) might lead to unfa- are updated accordingly. vorable dynamic regret bounds in non-stationary MDPs. In the non-stationary environment where pt≠W , . . . , pt≠1 are One immediate challenge not presented in the bandit setting generally distinct, we show that it is impossible to bound the (Cheung et al., 2019b) is that the starting state of each block diameter of p̄t in terms of the maximum of the diameters of is determined by previous moves of the DM. Hence, the pt≠W , . . . , pt≠1 . More generally, we demonstrate the pre- master algorithm is not facing a simple oblivious environ- vious claim not only for p̄t , but also for every p̃ œ Hp,t (0) ment as the case in (Cheung et al., 2019b), and we cannot in the following Proposition. The Proposition showcases use the EXP3 (Auer et al., 2002b) algorithm as the master. the unique challenge in exploring non-stationary MDPs that Nevertheless, the state is observed before the starting of a is absent in the piecewise-stationary MDPs, and motivates block. Thus, we use the EXP3.P algorithm for multi-armed our notion of confidence widening with ÷ > 0. To ease the bandit against an adaptive adversary (Auer et al., 2002b) as notation, we put t = W + 1 without loss of generality. the master algorithm. Owing to its similarity to the BOB algorithm (Cheung et al., 2019b), we defer the design de- Proposition 3. There exists a sequence of non-stationary tails and the proof of dynamic regret bound for the BORL MDP transition distributions p1 , . . . , pW such that 1) The algorithm to Sections E and F of the full version (Cheung diameter of (S, A, pn ) is 1 for each n œ [W ]. 2) The total et al., 2020b), respectively. variations in state transition distributions is O(1). Never- theless, under some deterministic policy, Theorem 2. Assume S > 1, with probability 1 ≠ O(”), • The empirical MDP (S, A, p̂W +1 ) has diameter (W ) the dynamic regret bound of the BORL algorithm is 1 2 1 3 p̃ œ Hp,W +1 (0), the MDP (S, A, p̃) • Further, for every Õ(Dmax (Br + Bp + 1) 4 S 3 A 2 T 4 ). has diameter ( W/ log W )

Non-Stationary Reinforcement Learning Proof. The sequence p1 , . . . , pW alternates between the fol- lowing 2 instances p1 , p2 . Now, define the common state space S = {1, 2} and action collection A = {A1 , A2 }, where A1 = {a1 , a2 }, {A2 } = {b1 , b2 }. We assume all the state transitions are deterministic, and a graphical illus- tration is presented in Fig. 2. Clearly, we see that both instances have diameter 1. Figure 3. (From top to bottom) Underlying policies, transition ker- nels, time steps, and state visits. Finally, for the confidence region Hp,W +1 (0) = {Hp,W +1 (s, a; 0)}s,a constructed without confidence widening, for any p̃ œ Hp,W +1 (0) we have p̃(2|1, a1 ) = 1Ò 2 log W p̃(1|2, b1 ) = O · +1 and p̃(2|1, a2 ) = p̃(1|2, b2 ) = 1Ò 2 Figure 2. Example MDPs (with deterministic transitions). log W O · ≠1 respectively, since the stochastic confidence 1Ò 2 1Ò 2 Now, consider the following two deterministic and station- log W log W radii and dominate the sample ary policies ﬁ 1 and ﬁ 2 : ﬁ 1 (1) = a1 , ﬁ 1 (2) = b2 , ﬁ 2 (1) = · +1 · ≠1 1 a2 , ﬁ 2 (2) = b1 . Since the MDP is deterministic, we have mean · +1 and 0. Therefore, for any p̃ œ Hp,W +1 (0), the p̂W +1 = p̄W +1 . diameter 1Ò of 2the MDP constructed by (S, A, p̃) is at least W log W . In the following, we construct a trajectory where the DM alternates between policies ﬁ 1 , ﬁ 2 during time {1, . . . , W } Remark 4. Inspecting the prevalent OFU guided approach while the underlying transition kernel alternates between for stochastic MAB and RL in MDPs settings (Auer et al., p1 , p2 . In the construction, the DM is almost always at the 2002a; Abbasi-Yadkori et al., 2011; Jaksch et al., 2010; self-loop at state 1 (or 2) throughout the horizon, no matter Bubeck & Cesa-Bianchi, 2012; Lattimore & Szepesvári, what action a1 , a2 (or b1 , b2 ) she takes. Consequently, it 2018), one usually concludes that a tighter design of confi- will trick the DM into thinking that p̂W +1 (1|1, ai ) ¥ 1 for dence region can result in a lower (dynamic) regret bound. each i œ {1, 2}, and likewise p̂W +1 (2|2, bi ) ¥ 1 for each In (Abernethy et al., 2016), this insights has been formal- i œ {1, 2}. Altogether, this will lead the DM to conclude ized in stochastic K-armed bandit settings via a potential that (S, A, p̂W +1 ) constitute a high diameter MDP, since function type argument. Nevertheless, Proposition 3 (to- the probability of transiting from state 1 to 2 (and 2 to 1) gether with Theorem 1) demonstrates that using the tightest are close to 0. confidence region in learning algorithm design may not The construction is detailed as follows. Let W = 4· . In be enough to ensure low dynamic regret bound for RL in addition, let the state transition kernels be p1 from time 1 non-stationary MDPs. to · and from time step 2· + 1 to 3· and be p2 for the remaining time steps. The DM starts at state 1. She follows 7. Conclusion policy ﬁ 1 from time 1 to time 2· , and policy ﬁ 2 from 2· + 1 to 4· . Under the specified instance and policies, it can be In this paper, we studied the problem of non-stationary re- readily verified that the DM takes inforcement learning where the unknown reward and state transition distributions can be different from time to time • action a1 from time 1 to · + 1, as long as the total changes are bounded by some variation • action b2 from time · + 2 to 2· , budgets, respectively. We first incorporated the sliding win- • action b1 from time 2· + 1 to 3· + 1, dow estimator and the novel confidence widening technique • action a2 from time 3· + 2 to 4· . into the UCRL2 algorithm to propose the SWUCRL2-CW As a result, the DM is at state 1 from time 1 to · + 1, and algorithm with low dynamic regret when the variation bud- time 3· + 2 to 4· ; while she is at state 2 from time · + 2 to gets are known. We then designed the parameter-free BORL 3· + 1, as depicted in Fig. 3. We have: algorithm that allows us to enjoy this dynamic regret bound · 1 without knowing the variation budgets. The main ingredient p̂W +1 (1|1, a1 ) = , p̂W +1 (2|1, a1 ) = of the proposed algorithms is the novel confidence widening · +1 · +1 technique, which injects extra optimism into the design of p̂W +1 (1|1, a2 ) = 1, p̂W +1 (2|1, a2 ) = 0 learning algorithms. This is in contrast to the widely held · 1 believe that optimistic exploration algorithms for (station- p̂W +1 (2|2, b1 ) = , p̂W +1 (1|2, b1 ) = · +1 · +1 ary and non-stationary) stochastic online learning settings p̂W +1 (2|2, b2 ) = 1, p̂W +1 (1|2, b2 ) = 0. should employ the lowest possible level of optimism.

Non-Stationary Reinforcement Learning Acknowledgements Cai, H., Ren, K., Zhang, W., Malialis, K., Wang, J., Yu, Y., and Guo, D. Real-time bidding by reinforcement learn- The authors would like to express sincere gratitude to Dy- ing in display advertising. In Proceedings of the ACM lan Foster, Negin Golrezaei, and Mengdi Wang, as well International Conference on Web Search and Data Mining as various seminar attendees for helpful discussions and (WSDM), 2017. comments. Cardoso, A. R., Wang, H., and Xu, H. Large scale markov decision processes with changing rewards. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. References Chen, W., Shi, C., and Duenyas, I. Optimal learning algorithms Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. Improved algo- for stochastic inventory systems with random capacities. In rithms for linear stochastic bandits. In Advances Neural SSRN Preprint 3287560, 2019a. Information Processing Systems 25 (NIPS), 2011. Chen, Y., Lee, C.-W., Luo, H., and Wei, C.-Y. A new algorithm for Abbasi-Yadkori, Y., Bartlett, P. L., Kanade, V., Seldin, Y., and non-stationary contextual bandits: Efficient, optimal, and Szepesvári, C. Online learning in markov decision processes parameter-free. In Proceedings of Conference on Learning with adversarially chosen transition probability distributions. Theory (COLT), 2019b. In Advances in Neural Information Processing Systems 26 Cheung, W. C., Simchi-Levi, D., and Zhu, R. Learning to optimize (NIPS), 2013. under non-stationarity. In Proceedings of International Con- Abernethy, J., Amin, K., and Zhu, R. Threshold bandits, with ference on Artificial Intelligence and Statistics (AISTATS), and without censored feedback. In Advances in Neural 2019a. Information Processing Systems 29 (NIPS), 2016. Cheung, W. C., Simchi-Levi, D., and Zhu, R. Hedging the Agrawal, S. and Jia, R. Optimistic posterior sampling for rein- drift: Learning to optimize under non-stationarity. In forcement learning: worst-case regret bounds. In Advances arXiv:1903.01461, 2019b. URL https://arxiv.org/ in Neural Information Processing Systems 30 (NIPS), pp. abs/1903.01461. 1184–1194. Curran Associates, Inc., 2017. Cheung, W. C., Simchi-Levi, D., and Zhu, R. Non-stationary Agrawal, S. and Jia, R. Learning in structured mdps with con- reinforcement learning: The blessing of (more) optimism. vex cost functions: Improved regret bounds for inventory In arXiv:1906.02922v4 [cs.LG], 2020a. URL https:// management. In Proceedings of the ACM Conference on arxiv.org/abs/1906.02922. Economics and Computation (EC), 2019. Cheung, W. C., Simchi-Levi, D., and Zhu, R. Reinforcement Arora, R., Dekel, O., and Tewari, A. Deterministic mdps with learning for non-stationary markov decision processes: The adversarial rewards and bandit feedback. In Proceedings of blessing of (more) optimism. In arXiv:2006.14389, 2020b. the Twenty-Eighth Conference on Uncertainty in Artificial URL https://arxiv.org/abs/2006.14389. Intelligence, pp. 93–101, 2012. Cormen, T. H., Leiserson, C. E., Rivest, R. L., and Stein, C. Intro- Auer, P., Cesa-Bianchi, N., and Fischer, P. Finite-time analysis of duction to algorithms. In MIT Press, 2009. the multiarmed bandit problem. In Machine learning, 47, Dick, T., György, A., and Szepesvári, C. Online learning in markov 235–256, 2002a. decision processes with changing cost sequences. In Pro- Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. The ceedings of the International Conference on Machine Learn- nonstochastic multiarmed bandit problem. In SIAM Journal ing (ICML), 2014. on Computing, 2002, Vol. 32, No. 1 : pp. 48–77, 2002b. Even-Dar, E., Kakade, S. M., , and Mansour, Y. Experts in a Balseiro, S. and Gur, Y. Learning in repeated auctions with bud- markov decision process. In Advances in Neural Informa- gets: Regret minimization and equilibrium. In Management tion Processing Systems 18 (NIPS), 2005. Science, 2019. Flajolet, A. and Jaillet, P. Real-time bidding with side information. Bartlett, P. L. and Tewari, A. REGAL: A regularization based algo- In Advances in Neural Information Processing Systems 30 rithm for reinforcement learning in weakly communicating (NeurIPS), 2017. mdps. In UAI 2009, Proceedings of the Twenty-Fifth Con- Fruit, R., Pirotta, M., and Lazaric, A. Near optimal exploration- ference on Uncertainty in Artificial Intelligence, Montreal, exploitation in non-communicating markov decision pro- QC, Canada, June 18-21, 2009, pp. 35–42, 2009. cesses. In Advances in Neural Information Processing Sys- Bertsekas, D. Dynamic Programming and Optimal Control. tems 31, pp. 2998–3008. Curran Associates, Inc., 2018a. Athena Scientific, 2017. Fruit, R., Pirotta, M., Lazaric, A., and Ortner, R. Efficient bias- Besbes, O., Gur, Y., and Zeevi, A. Stochastic multi-armed ban- span-constrained exploration-exploitation in reinforcement dit with non-stationary rewards. In Advances in Neural learning. In Proceedings of the 35th International Confer- Information Processing Systems 27 (NIPS), 2014. ence on Machine Learning, volume 80 of Proceedings of Bimpikis, K., Candogan, O., and Saban, D. Spatial pricing in Machine Learning Research, pp. 1578–1586. PMLR, 10–15 ride-sharing networks. In Operations Research, 2019. Jul 2018b. Bubeck, S. and Cesa-Bianchi, N. Regret Analysis of Stochastic and Fruit, R., Pirotta, M., and Lazaric, A. Improved analysis of ucrl2b. Nonstochastic Multi-armed Bandit Problems. Foundations In https://rlgammazero.github.io/, 2019. and Trends in Machine Learning, 2012, Vol. 5, No. 1: pp. Gajane, P., Ortner, R., and Auer, P. A sliding-window algorithm 1–122, 2012. for markov decision processes with arbitrarily changing Burnetas, A. N. and Katehakis, M. N. Optimal adaptive policies for rewards and transitions. 2018. markov decision processes. In Mathematics of Operations Garivier, A. and Moulines, E. On upper-confidence bound policies Research, volume 22, pp. 222–255, 1997. for switching bandit problems. In Proceedings of Interna-

Non-Stationary Reinforcement Learning tional Conferenc on Algorithmic Learning Theory (ALT), Neu, G., Gyorgy, A., and Szepesvari, C. The adversarial stochastic 2011a. shortest path problem with unknown transition probabilities. Garivier, A. and Moulines, E. On upper-confidence bound policies In Proceedings of the Fifteenth International Conference for switching bandit problems. In Algorithmic Learning on Artificial Intelligence and Statistics, volume 22, pp. 805– Theory, pp. 174–188. Springer Berlin Heidelberg, 2011b. 813. PMLR, 21–23 Apr 2012. Guo, X., Hu, A., Xu, R., and Zhang, J. Learning mean-field games. Nowzari, C., Preciado, V. M., and Pappas, G. J. Analysis and In Advances in Neural Information Processing Systems 32 control of epidemics: A survey of spreading processes on (NeurIPS), 2019. complex networks. In IEEE Control Systems Magazine, 2016. Gurvich, I., Lariviere, M., and Moreno, A. Operations in the on- demand economy: Staffing services with self-scheduling Ortner, R., Gajane, P., and Auer, P. Variational regret bounds for capacity. In Sharing Economy: Making Supply Meet De- reinforcement learning. In Proceedings of Conference on mand, 2018. Uncertainty in Artificial Intelligence (UAI), 2019. Han, Y., Zhou, Z., and Weissman, T. Optimal no-regret learning in Puterman, M. L. Markov Decision Processes: Discrete Stochastic repeated first-price auctions. In arXiv:2003.09795, 2020. Dynamic Programming. John Wiley & Sons, Inc., New York, NY, USA, 1st edition, 1994. Hoeffding, W. Probability inequalities for sums of bounded ran- dom variables. In Journal of the American statistical as- Qin, Z. T., Tang, J., and Ye, J. Deep reinforcement learning with sociation, volume 58, pp. 13–30. Taylor & Francis Group, applications in transportation. In Tutorial of the 33rd AAAI 1963. Conference on Artificial Intelligence (AAAI-19), 2019. Huh, W. T. and Rusmevichientong, P. A nonparametric asymptotic Rosenberg, A. and Mansour, Y. Online convex optimization in analysis of inventory planning with censored demand. In adversarial Markov decision processes. In Proceedings of Mathematics of Operations Research, 2009. the 36th International Conference on Machine Learning, volume 97, pp. 5478–5486. PMLR, 2019. Jaksch, T., Ortner, R., and Auer, P. Near-optimal regret bounds for reinforcement learning. In J. Mach. Learn. Res., volume 11, Shortreed, S., Laber, E., Lizotte, D., Stroup, S., and Murphy, J. pp. 1563–1600. JMLR.org, August 2010. P. S. Informing sequential clinical decision-making through reinforcement learning: an empirical study. In Machine Jin, C., Jin, T., Luo, H., Sra, S., and Yu, T. Learning adversar- Learning, 2010. ial markov decision processes with bandit feedback and unknown transition. 2019. Sidford, A., Wang, M., Wu, X., Yang, L. F., and Ye, Y. Near- optimal time and sample complexities for solving dis- Kanoria, Y. and Qian, P. Blind dynamic resource alloca- counted markov decision process with a generative model. tion in closed networks via mirror backpressure. In In Advances in Neural Information Processing Systems 31 arXiv:1903.02764, 2019. (NeurIPS), 2018a. Karnin, Z. and Anava, O. Multi-armed bandits: Competing with Sidford, A., Wang, M., Wu, X., and Ye, Y. Variance reduced optimal sequences. In Advances in Neural Information value iteration and faster algorithms for solving markov Processing Systems 29 (NIPS), 2016. decision processes. In Proceedings of the Annual ACM- Keskin, N. and Zeevi, A. Chasing demand: Learning and earning SIAM Symposium on Discrete Algorithms (SODA), 2018b. in a changing environments. In Mathematics of Operations Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Research, 2016, 42(2), 277–307, 2016. Introduction. A Bradford Book, 2018. Kim, M. J. and Lim, A. E. Robust multiarmed bandit problems. In Taylor, T. On-demand service platforms. In Manufacturing & Management Science, 2016. Service Operations Management, 2018. Kiss, I. Z., Miller, J. C., and Simon, P. L. Mathematics of epidemics Tewari, A. and Bartlett, P. L. Optimistic linear programming on networks. In Springer, 2017. gives logarithmic regret for irreducible mdps. In Advances Kveton, B., Wen, Z., Ashkan, A., and Szepesvári, C. Tight re- in Neural Information Processing Systems 20 (NIPS), pp. gret bounds for stochastic combinatorial semi-bandits. In 1505–1512. 2008. AISTATS, 2015. Wang, M. Randomized linear programming solves the markov Lattimore, T. and Szepesvári, C. Bandit Algorithms. Cambridge decision problem in nearly-linear (sometimes sublinear) University Press, 2018. running time. In Mathematics of Operations Research, Li, Y., Zhong, A., Qu, G., and Li, N. Online markov decision 2019. processes with time-varying transition probabilities and re- Wei, C.-Y., Jafarnia-Jahromi, M., Luo, H., Sharma, H., and wards. In ICML workshop on Real-world Sequential Deci- Jain, R. Model-free reinforcement learning in infinite- sion Making, 2019. horizon average-reward markov decision processes. In Luo, H., Wei, C., Agarwal, A., and Langford, J. Efficient con- arXiv:1910.07072, 2019. textual bandits in non-stationary worlds. In Proceedings of Xu, K. and Yun, S.-Y. Reinforcement with fading memories. In Conference on Learning Theory (COLT), 2018. Mathematics of Operations Research, 2020. Ma, W. Improvements and generalizations of stochastic knap- Yu, J. Y. and Mannor, S. Online learning in markov decision sack and markovian bandits approximation algorithms. In processes with arbitrarily changing rewards and transitions. Mathematics of Operations Research, 2018. In Proceedings of the International Conference on Game Neu, G., Antos, A., György, A., and Szepesvári, C. Online markov Theory for Networks, 2009. decision processes under bandit feedback. In Advances Yu, J. Y., Mannor, S., and Shimkin, N. Markov decision pro- in Neural Information Processing Systems 23 (NIPS), pp. cesses with arbitrary reward processes. In Mathematics of 1804–1812. 2010. Operations Research, volume 34, pp. 737–757, 2009.

Non-Stationary Reinforcement Learning Zhang, A. and Wang, M. Spectral state compression of markov processes. In arXiv:1802.02920, 2018. Zhang, H., Chao, X., and Shi, C. Closing the gap: A learning algorithm for the lost-sales inventory system with lead times. In Management Science, 2018. Zhang, Z. and Ji, X. Regret minimization for reinforcement learn- ing by evaluating the optimal bias function. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. Zhou, X., Chen, N., Gao, X., and Xiong, Y. Regime switching bandits. In arXiv:2001.09390, 2020. Zhou, Z. and Bambos, N. Wireless communications games in fixed and random environments. In IEEE Conference on Decision and Control (CDC), 2015. Zhou, Z., Glynn, P., and Bambos, N. Repeated games for power control in wireless communications: Equilibrium and regret. In IEEE Conference on Decision and Control (CDC), 2016.

You can also read