Þ¥bonds€¬cell_resultsÞ¸Ù$781fa565-398c-4c89-89f3-0595455afc85Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ3update_estimator! (generic function with 2 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšUG—°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$781fa565-398c-4c89-89f3-0595455afc85¹depends_on_disabled_cellsÂ§runtimeÎêµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$37446874-1c28-491b-b3cc-b4ad3282686eŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

Recreation of Figure 2.6 for the non-stationary case over 200000 steps with a drift rate of Ïƒ = 0.01. These parameters can be adjusted above, but by default this is performed over 200,000 steps with the accumulated reward only being measured on the final 100,000 steps. Unlike in the stationary case, the Ïµ-greedy method with Î±=0.1 for updating the Q values performs the best at a very small Ïµ value of $2^{-7}.$ The UCB method is the second best performer but requires a very large c value of 128 compared to ~1 for the stationary case in which it was the best performer. This UCB method also uses the sample average which is not ideal for a non-stationary distribution. That is one of the reasons why it was mentioned earlier in the chapter that it is difficult to adapt the UCB technique to the non-stationary problem. We can use the constant step size method but that doesn't help the fact that the variance estimates are wrong.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšXÄÙ©°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$37446874-1c28-491b-b3cc-b4ad3282686e¹depends_on_disabled_cellsÂ§runtimeÎjjbµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ8average_runs_cum_reward (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšU\‹ë°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872¹depends_on_disabled_cellsÂ§runtimeÎ¹8;µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8fac4109-2e0d-4366-9118-018221e0b910Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ+figure_2_4 (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšSú±°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8fac4109-2e0d-4366-9118-018221e0b910¹depends_on_disabled_cellsÂ§runtimeÎ4ðµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$98f4d5e6-7569-457e-851d-713c572ae400Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ@plot_nonstationary_param_search (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXªÓ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$98f4d5e6-7569-457e-851d-713c572ae400¹depends_on_disabled_cellsÂ§runtimeÎ,oµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$4be9e81b-c3de-4c79-97b5-b41c03e0f187Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÛs

Distribution of Best Action Reward for a Random 10 Armed Bandit, Expected Value Estimate = 1.539

Expected Value of Mean Reward for Arms Ranked from 1 to 10

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšS7ŽL°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$4be9e81b-c3de-4c79-97b5-b41c03e0f187¹depends_on_disabled_cellsÂ§runtimeÎàk±µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$78c45162-dc8e-4faf-b1a9-8c71be86dceeŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚj

Exercise 2.4

If the step-size parameters, $\alpha_n$, are not constant, then the estimate $Q_n$ is a weighted average of previously received rewards with a weighting different from that given by (2.6). What is the weighting on each prior reward for the general case, analogous to (2.6), in terms of the sequence of step-size parameters?

From (2.6): $Q_{n+1} = Q_n + \alpha[R_n - Q_n]$ so here we consider the case where $\alpha$ is not a constant but rather can have a unique value for each step n.

$$\begin{flalign} Q_{n+1}&=Q_n + \alpha_n[R_n - Q_n]\\ &=\alpha_nR_n+(1-\alpha_n)Q_n\\ &=\alpha_nR_n+(1-\alpha_n)[\alpha_{n-1}R_{n-1}+(1-\alpha_{n-1})Q_{n-1}] \tag{using recursive formula from first step}\\ &=\alpha_nR_n+(1-\alpha_n)\alpha_{n-1}R_{n-1}+(1-\alpha_n)(1-\alpha_{n-1})Q_{n-1}\\ &=\alpha_nR_n+(1-\alpha_n)\alpha_{n-1}R_{n-1}+(1-\alpha_n)(1-\alpha_{n-1})[\alpha_{n-2}R_{n-2}+(1-\alpha_{n-2})Q_{n-2}]\\ &=\alpha_nR_n+(1-\alpha_n)\alpha_{n-1}R_{n-1}+(1-\alpha_n)(1-\alpha_{n-1})\alpha_{n-2}R_{n-2}+...\\ &=Q_1\prod_{i=1}^n \left( 1-\alpha_i \right)+\sum_{i=1}^{n} \left[ (R_i\alpha_i)\prod_{j=i+1}^n(1-\alpha_j) \right]\\ \end{flalign}$$

For example if $\alpha_i=1/i$ then the product in the first term is 0 and the formula becomes:

$$\begin{flalign} Q_{n+1} &= \sum_{i=1}^{n} \left[ \frac{R_i}{i}\prod_{j=i+1}^n\frac{j-1}{j} \right]\\ &= \sum_{i=1}^{n} \left[ \frac{R_i}{i}\frac{i}{i+1}\frac{i+1}{i+2}...\frac{n-1}{n} \right]\\ &=\sum_{i=1}^{n} \frac{R_i}{n} \end{flalign}$$

from the expanded product we can see that all of the numerators and denominators cancel out leaving only $\frac{R_i}{n}$ which matches the sample-average as expected for this step-size.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNƒ8°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$78c45162-dc8e-4faf-b1a9-8c71be86dcee¹depends_on_disabled_cellsÂ§runtimeÎ¯µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$30f05bd9-e939-4810-b710-edc7f5975921Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ,updatecoef (generic function with 2 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšU?¢U°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$30f05bd9-e939-4810-b710-edc7f5975921¹depends_on_disabled_cellsÂ§runtimeÎ Ú½µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$2bce1b80-2133-40a0-9367-fc2d491f6245Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ

Visualize Armed Bandit

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšRÀ]B°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$2bce1b80-2133-40a0-9367-fc2d491f6245¹depends_on_disabled_cellsÂ§runtimeÎùÒ‚µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$a15e5d05-a238-440b-9a43-d830c6ea2f4dŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ.param_search (generic function with 2 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšX&üÆ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$a15e5d05-a238-440b-9a43-d830c6ea2f4d¹depends_on_disabled_cellsÂ§runtimeÎåJÃµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$0e06d9cf-b995-40b1-86ee-1c6a56c63bd0Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTeAl°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$0e06d9cf-b995-40b1-86ee-1c6a56c63bd0¹depends_on_disabled_cellsÂ§runtimeÎwÆµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1cde4625-f8ed-4403-835c-95cc30206699Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙž"/home/runner/work/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Chapter-02/parameter_studies"¤mimeªtext/plain¬rootassignee®const datapath²last_run_timestampËAÚšX[*Ã°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1cde4625-f8ed-4403-835c-95cc30206699¹depends_on_disabled_cellsÂ§runtimeÎ— µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$6a00226c-cda1-49d8-92a8-514f2264469cŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

Ideas for Non-Stationary Problem

In the non-stationary problem we can track each average with the constant step size method and this will be our estimate for the q values. Since the value is drifting over time we definitely cannot represent our knowledge of it with a probability distribution, however we can assume that the exponentially decaying average preserves a small enough amount of information that the samples over that period are roughly stationary. If we do that, then we can still use a normal distribution for each q value, but the standard deviation now takes a different form. For the sample average case we had:

$$q_{i} \sim N(\mu, \sigma)$$

$$N_i = \text{number of samples from action i}$$

$$\mu = \frac{1}{N_i}\sum_{j = 1}^N R_j$$

$$\sigma^2 = \frac{1}{N_i}$$

For the constant step size case, our average can be represented as a weighted sum:

$$\mu = (1=\alpha)^N Q_1 + \sum_{i = j}^N \alpha(1-\alpha)^{N-i}R_j$$

where $\alpha$ is the constant step size parameter and $Q_1$ is the initial value for the estimate. This weighted sum implies a different variance for the distribution:

$$\sigma^2 = \sum_j w_j^2$$

where $w_j$ are the weights in the sum and there is still the assumption that each action produces iid rewards with unit variance. So what remains is to calculate what this squared sum of the weights is for this type of average.

$$\begin{flalign} \sigma^2 &= \sum_j w_j^2 \\ &= (1-\alpha)^{2N} + \sum_{j=1}^N \alpha^2(1-\alpha)^{2N-2j} \\ &= (1-\alpha)^{2N} + \alpha^2 \left [ 1 + (1-\alpha)^2 + \cdots + (1-\alpha)^{2N-2} \right ] \\ \end{flalign}$$

If we take $a = (1-\alpha)^2$, then the expression in the brackets is just:

$$\sum_{j = 0}^{N-1} a^j = \frac{a^N - 1}{a - 1} = \frac{(1-\alpha)^{2N} - 1}{(1-\alpha)^2 - 1} = \frac{(1-\alpha)^{2N} - 1}{\alpha(\alpha - 2)}$$

And our overall expression for the variance is:

$$\sigma^2 = (1-\alpha)^{2N} + \frac{\alpha((1-\alpha)^{2N} - 1)}{\alpha - 2}$$

which depends on both $\alpha$ and $N$. The variance is plotted below compared to the variance for the sample mean with the option to change the $\alpha$ parameter.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN°ê°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$6a00226c-cda1-49d8-92a8-514f2264469c¹depends_on_disabled_cellsÂ§runtimeÎ RŒµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$c61630b0-29c1-4183-90d9-57c999187b53Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ-print_power2 (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšX*|°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$c61630b0-29c1-4183-90d9-57c999187b53¹depends_on_disabled_cellsÂ§runtimeÎêµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$0e606680-dd65-444f-bc98-73de4abbcdd4Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTgòI°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$0e606680-dd65-444f-bc98-73de4abbcdd4¹depends_on_disabled_cellsÂ§runtimeÎm¯µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$c695b7f9-76ca-419b-924d-8338a42c8188Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ¹

$$Q_{n+1} = Q_n + \beta_n[R_n - Q_n]$$

where $\beta_n \dot= \alpha / \bar{o}_n $$ and $$\bar{o}_n \dot= \bar{o}_{n-1}+\alpha(1-\bar{o}_{n-1})$

$$\bar{o}_n = \bar{o}_{n-1} + \alpha(1-\bar{o}_{n-1})=\bar{o}_{n-1}(1-\alpha)+\alpha$$

We can expand $\bar{o}_n$ backwards to get an explicit formula.

$$\begin{flalign} \bar{o}_n&=\bar{o}_{n-1}(1-\alpha)+\alpha\\ &=(\bar{o}_{n-2}(1-\alpha) + \alpha)(1-\alpha)+\alpha\\ &=\bar{o}_{n-2}(1-\alpha)^2 + \alpha((1-\alpha)+1)\\ &=(\bar{o}_{n-3}(1-\alpha)+\alpha)(1-\alpha)^2 + \alpha((1-\alpha)+1)\\ &=\bar{o}_{n-3}(1-\alpha)^3+\alpha((1-\alpha^2) + (1-\alpha)+1)\\ &\vdots \\ &=\bar{o}_0(1-\alpha)^n + \alpha\sum_{i=0}^{n-1}(1-\alpha)^i=\alpha\sum_{i=0}^{n-1}(1-\alpha)^i \end{flalign}$$

This sum has an explicit formula as can be seen by:

$$\begin{flalign} S &= 1 + (1-\alpha) + (1-\alpha)^2 + \cdots + (1-\alpha)^{n-1} \\ S(1-\alpha) &= (1-\alpha)+\cdots+(1-\alpha)^n=S-1+(1-\alpha)^n \\ -S\alpha &=-1+(1-\alpha)^n \\ S&=\frac{1-(1-\alpha)^n}{\alpha} \\ \end{flalign}$$

Therefore, $\bar{o}_n=\alpha\frac{1-(1-\alpha)^n}{\alpha}=1-(1-\alpha)^n$, and since $0<\alpha<1$, then $(1 - \alpha)^n \rightarrow 0 \text{ as } n \rightarrow \infty.$

$$\beta_n=\frac{\alpha}{\bar{o}_n}=\frac{\alpha}{1-(1-\alpha)^n} \implies \beta_1=1$$

From exercise 2.4, we have the formula for $Q_n$ with a non-constant coefficient $\alpha_n$ which we can trivially replace here with $\beta_n$

$$Q_n=Q_1\prod_{i=1}^n \left( 1-\beta_i \right)+\sum_{i=1}^{n} \left[ R_i\beta_i\prod_{j=i+1}^n(1-\beta_j) \right]$$

Since $\beta_1=1$, the product associated with $Q_1$ will be 0. Since there is no dependency on the initial value of Q, we can say this formula for updating Q has no initial bias. If we then make the substitution $\beta_n=\frac{\alpha}{1-(1-\alpha)^n}$, we have

$$\begin{flalign} Q_n&=\sum_{i=1}^{n} \left[ R_i\frac{\alpha}{1-(1-\alpha)^i}\prod_{j=i+1}^n \left( 1-\frac{\alpha}{1-(1-\alpha)^j} \right) \right] \\ &=\alpha\sum_{i=1}^n \left[ \frac{R_i}{1-(1-\alpha)^i}\prod_{j=i+1}^n \left( \frac{(1-\alpha)(1-(1-\alpha)^{j-1})}{1-(1-\alpha)^j} \right) \right] \\ &=\alpha\sum_{i=1}^n \left[ \frac{R_i(1-\alpha)^{n-i}}{1-(1-\alpha)^i}\prod_{j=i+1}^n \left( \frac{1-(1-\alpha)^{j-1}}{1-(1-\alpha)^j} \right) \right] \end{flalign}$$

Examining the product term on its own, we can see it simplifies.

$$\prod_{j=i+1}^n \left( \frac{1-(1-\alpha)^{j-1}}{1-(1-\alpha)^j} \right)$$

$$\frac{1-(1-\alpha)^{i}}{1-(1-\alpha)^{i+1}}\frac{1-(1-\alpha)^{i+1}}{1-(1-\alpha)^{i+2}}\cdots\frac{1-(1-\alpha)^{n-1}}{1-(1-\alpha)^{n}}=\frac{1-(1-\alpha)^i}{1-(1-\alpha)^n} \text{ for iâ‰¤n}$$

Replacing this expression for the product in the expression for $Q_n$ we have:

$$Q_n=\alpha\sum_{i=1}^n \left[ \frac{R_i(1-\alpha)^{n-i}}{1-(1-\alpha)^i}\frac{1-(1-\alpha)^i}{1-(1-\alpha)^n}\right]=\frac{\alpha}{1-(1-\alpha)^n}\sum_{i=1}^n R_i(1-\alpha)^{n-i}$$

If we expand this sum going backwards from $i=n$:

$$Q_n=\frac{\alpha}{1-(1-\alpha)^n} \left[ R_n+R_{n-1}(1-\alpha)+R_{n-2}(1-\alpha)^2+\cdots+R_1(1-\alpha)^{n-1} \right]$$

The constant term starts off at $1$ for $n=1$ and approaches $\alpha$ in the limit of $n \rightarrow \infty$. If $0<\alpha<1$, then the coefficients in the sum section for $R_i$ decrease exponentially from 1 for $i=n$ to $(1-\alpha)^{n-1}$ for $i=1.$ So the average over rewards includes every reward back to $R_1$ like the simple average but the coefficients become exponentially smaller approaching 0 as $n \rightarrow \infty$.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNqÉ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$c695b7f9-76ca-419b-924d-8338a42c8188¹depends_on_disabled_cellsÂ§runtimeÎ ´‰µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$865610bb-ee82-4440-9f32-f00d0382783bŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ,run_or_load (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšX;·¢°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$865610bb-ee82-4440-9f32-f00d0382783b¹depends_on_disabled_cellsÂ§runtimeÎ =µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$50fbdc85-82f1-4c52-936b-84eb14951d71Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙAaverage_gradient_stationary_runs (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšT{‘°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$50fbdc85-82f1-4c52-936b-84eb14951d71¹depends_on_disabled_cellsÂ§runtimeÎÜ{eµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$2447c4ea-7752-457c-80da-ac0dd72a64c1Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ*save_data (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšX4N°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$2447c4ea-7752-457c-80da-ac0dd72a64c1¹depends_on_disabled_cellsÂ§runtimeÎ¾±µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$b5a2df21-4525-4320-b8dd-aea5ecdab832Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

Figure 2.5 Parameters

Number of Actions: Reward Offset:

Step-Size Minimum With Baseline:

Step-Size Minimum Without Baseline:

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšT^¿Y°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$b5a2df21-4525-4320-b8dd-aea5ecdab832¹depends_on_disabled_cellsÂ§runtimeÎ /îÈµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$c12dab6a-92f0-41b1-a6b7-c404c74d9a83Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ:action_value_testbed_plot (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšRáµ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$c12dab6a-92f0-41b1-a6b7-c404c74d9a83¹depends_on_disabled_cellsÂ§runtimeÎŠ–Ýµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$36602c38-8b29-4158-b299-94015a333762Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ?

Dependencies and Settings

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNË°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$36602c38-8b29-4158-b299-94015a333762¹depends_on_disabled_cellsÂ§runtimeÎçÚµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ;nonstationary_param_search (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšX ]Ñ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2¹depends_on_disabled_cellsÂ§runtimeÎ®*µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$68470b1d-3cc2-4cb1-8dc2-53227e6300e7Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšThéu°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$68470b1d-3cc2-4cb1-8dc2-53227e6300e7¹depends_on_disabled_cellsÂ§runtimeÎµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$86350532-13f8-4035-bad8-f25f41c93163Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙJ

2.5 Tracking a Nonstationary Problem

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN>…°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$86350532-13f8-4035-bad8-f25f41c93163¹depends_on_disabled_cellsÂ§runtimeÎø‡µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$88e43fed-fcf3-4071-996a-63f63c3d49b4Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚî

Number of Steps to Accumulate Reward:

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšXt¡°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$88e43fed-fcf3-4071-996a-63f63c3d49b4¹depends_on_disabled_cellsÂ§runtimeÎ)Ü»µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$f00ab44e-0d84-40c7-aa23-358c77a013e3Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTeô°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$f00ab44e-0d84-40c7-aa23-358c77a013e3¹depends_on_disabled_cellsÂ§runtimeÎòBµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0adŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ,updatecoef (generic function with 4 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšU@ °persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0ad¹depends_on_disabled_cellsÂ§runtimeÎÍÙµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$46ce2b1f-02cf-4dae-bf02-f67543f38b91Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ7

Parameter Studies

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNË°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$46ce2b1f-02cf-4dae-bf02-f67543f38b91¹depends_on_disabled_cellsÂ§runtimeÎ4êµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$211d9390-df44-4902-9e13-ad6744b9b7ddŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšY á°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$211d9390-df44-4902-9e13-ad6744b9b7dd¹depends_on_disabled_cellsÂ§runtimeÎFo[µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$6d647790-a029-4579-86fa-9c7294aa94eaŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTeß°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$6d647790-a029-4579-86fa-9c7294aa94ea¹depends_on_disabled_cellsÂ§runtimeÎµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ+updatecoef (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšU?j8°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03¹depends_on_disabled_cellsÂ§runtimeÎYµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$51349e41-4696-4bd5-9bc1-cefbb82bea08Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ.sample_action (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTTˆr°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$51349e41-4696-4bd5-9bc1-cefbb82bea08¹depends_on_disabled_cellsÂ§runtimeÎ×_µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTj @°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4¹depends_on_disabled_cellsÂ§runtimeÎçðµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0baŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ/sample_action (generic function with 6 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTm‚°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0ba¹depends_on_disabled_cellsÂ§runtimeÎ&µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$7678b06b-feef-4656-a801-33f630437bfbŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ-average_runs (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšUU~°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$7678b06b-feef-4656-a801-33f630437bfb¹depends_on_disabled_cellsÂ§runtimeÎ[¨µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$3b88ec30-768b-44d0-88ee-b3ed989f22c3Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚJ

Number of Steps to Accumulate Reward (only measured on second half):

Reward Drift Rate Per Step $\sigma$:

By default, a plot will be loaded that matches this search criteria. Check the box below to recalculate the search and save a new plot whever the submit button is clicked.

Recompute Parameter Search:

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšX¹jb°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$3b88ec30-768b-44d0-88ee-b3ed989f22c3¹depends_on_disabled_cellsÂ§runtimeÎP®µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$691aa77a-d6da-4fde-9024-c4195057179dŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÛl

Average performance of the gradient bandit algorithm with and without a reward baseline on the 10-armed bandit testbed when the $q_*(a)$ are chosen to be near 4.0 rather than near 0. For the gradient bandit with a baseline, the offset doesn't affect the curves at all, but if the baseline is removed then the results are worse as seen in the second plot. However, if $\alpha$ is made smaller it seems like it will also converge to a similar success rate just over a longer time. The optimal value of $\alpha$ is much lower than when the baseline is removed which is consistent with slower convergence properties.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšU8On°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$691aa77a-d6da-4fde-9024-c4195057179d¹depends_on_disabled_cellsÂ§runtimeÎœ\‹aµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$276779a3-9332-46bd-b511-a33a2fea4b5fŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚí

Figure Parameters for Exercise 2.5

Action Count:

Exploration Parameter Ïµ:

Constant Step-Size Parameter Î±:

Initial Reward Value:

Reward Drift Standard Deviation:

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšSRC°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$276779a3-9332-46bd-b511-a33a2fea4b5f¹depends_on_disabled_cellsÂ§runtimeÎF+ðµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$b26b3519-6ab4-4cc9-971d-de5e591cac86Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚU

The sample average method fails to improve at finding the optimal action after about 4000 time steps. At this point in the learning process, it selects the optimal action about 45% of the time, while the constant step size averaging method continues to improve and approach the theoretical limit. It will take many more samples for the sample average method to learn the new rewards since each new sample is weighted progressively less. As the problem continues to change, the number of steps required to correct the estimates will grow infinitely large.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNº¡°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$b26b3519-6ab4-4cc9-971d-de5e591cac86¹depends_on_disabled_cellsÂ§runtimeÎa8µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$d0111453-9a66-411d-9966-fc386d1bdcb7Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ©

Exercise 2.11 (programming)

Make a figure analogous to Figure 2.6 for the nonstionary case outlined in Exercise 2.5. Include the constant-step-size Ïµ-greedy algorithm with Î±=0.1. Use runs of 200,000 steps and, as a performance measure for each algorithm and parameter setting, use the average reward over the last 100,000 steps.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNÅ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$d0111453-9a66-411d-9966-fc386d1bdcb7¹depends_on_disabled_cellsÂ§runtimeÎÁcµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$79082409-3182-4e0b-9c8c-37a94543fee9Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÛYê

Figure 2.2

Average performance of the $\epsilon$-greedy action-value methods on the 10-armed testbed. These data are averages over 2000 runs with different bandit problems. All methods use sample averages as their action-value estimates.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšR±É—°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$79082409-3182-4e0b-9c8c-37a94543fee9¹depends_on_disabled_cellsÂ§runtimeÎ;GÌØµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÛå€

Figure 2.1

Shows the reward distribution for each of the 10 arms in the testbed. The mean value is marked with a dashed line for each.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšQÞT*°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802¹depends_on_disabled_cellsÂ§runtimeÎ¡:ngµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$9b625fc0-89bd-4064-a379-225e6a940af7Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ¯

Number of Arms:

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšSYƒ©°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$9b625fc0-89bd-4064-a379-225e6a940af7¹depends_on_disabled_cellsÂ§runtimeÎ¾…µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1f9a98fd-ea29-415c-9f35-add34b513a34Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

Exercise 2.10

Suppose you face a 2-armed bandit task whose true action values change randomly from time step to time step. Specifically, suppose that, for any time step, the true values of actions 1 and 2 are respectively 10 and 20 with probability 0.5 (case A), and 90 and 80 with probability 0.5 (case B). If you are not able to tell which case you face at any step, what is the best expected reward you can achieve and how should you behave to achieve it? Now suppose that on each step you are told whether you are facing case A or case B (although you still donâ€™t know the true action values). This is an associative search task. What is the best expected reward you can achieve in this task, and how should you behave to achieve it?

When we do not know which case we are facing, we can calculate the expected reward for each action across all cases.

$$E[R_1] = 0.5 \times 10 + 0.5 \times 90 = 50$$

$$E[R_2] = 0.5 \times 20 + 0.5 \times 80 = 50$$

Since the expected reward of each action is equal, the best we can do is pick randomly which will have an expected reward of 50.

For the case in which we know if we are in case A or case B, we now can select the best action for each case which has a value of 20 (action 2) for case A and 90 (action 1) for case B. However, we have a 50% probability of facing each case so the best achievable expected reward is.

$$E[R] = 20 \times 0.5 + 90 \times 0.5 = 55$$

To acheive this reward we could apply the action value estimate approach but separate our samples for case A and B. That way we would have 4 estimates representing the expected reward of each action in each case. We could perform any of the exploration strategies mentioned earlier such as Ïµ-greedy action selection but being careful to update the estimate for that case only.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN[«°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1f9a98fd-ea29-415c-9f35-add34b513a34¹depends_on_disabled_cellsÂ§runtimeÎ_§µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ8nonstationary_algorithm (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšS>{–°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5¹depends_on_disabled_cellsÂ§runtimeÎVÕµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚM

N1: 100

N2: 100

Maximum Mean Difference : 1.0

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšXúL°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53¹depends_on_disabled_cellsÂ§runtimeÎzºëµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$649e3d20-e276-4f4b-aeb0-89150f180ef5Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙW

2.9 Associate Search (Contextual Bandits)

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN7 °persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$649e3d20-e276-4f4b-aeb0-89150f180ef5¹depends_on_disabled_cellsÂ§runtimeÎHqµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$32bff269-e893-4907-b589-7ba2ae1314bdŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚñ

2.6 Optmisitic Initial Values

The averaging methods discussed above have some bias towards the initial value of the estimates. We can exploit this by initializing Q with a value much higher than we'd expect to receive as a reward from any action. That way every observed reward at first will be dissappointing thus encouraging the agent to try unvisited actions. See below an example whose performance can be observed under different conditions.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNó°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$32bff269-e893-4907-b589-7ba2ae1314bd¹depends_on_disabled_cellsÂ§runtimeÎòBµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$7c562867-55d3-4b4f-950d-c8efc4a9ff32Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ*loadplots (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXb5~°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$7c562867-55d3-4b4f-950d-c8efc4a9ff32¹depends_on_disabled_cellsÂ§runtimeÎS›fµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8eŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚVŒ

Original Values: $h_i$ Softmax: $\frac{e^{(\beta h_i)}}{\sum_{i = 1} ^k e^{(\beta h_i)}}$

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšXç4°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8e¹depends_on_disabled_cellsÂ§runtimeÎ49µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$60b2079e-0efa-427e-93cf-7f4646fe202eŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTcÚ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$60b2079e-0efa-427e-93cf-7f4646fe202e¹depends_on_disabled_cellsÂ§runtimeÎíµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfaŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ6visualize_bandit_dist (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšR¿Ë°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfa¹depends_on_disabled_cellsÂ§runtimeÎ–8»µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$c72d35fc-c8fd-450e-9b95-d12ece5c2291Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ/get_param_list (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšX ù°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$c72d35fc-c8fd-450e-9b95-d12ece5c2291¹depends_on_disabled_cellsÂ§runtimeÎÍrµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$97c94391-397d-4cf2-88b1-3bea3af56ed3Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ.create_bandit (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšP®R°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$97c94391-397d-4cf2-88b1-3bea3af56ed3¹depends_on_disabled_cellsÂ§runtimeÎJÉµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$839861db-676f-4544-a802-0abb5d0049e1Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTed°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$839861db-676f-4544-a802-0abb5d0049e1¹depends_on_disabled_cellsÂ§runtimeÎÃˆµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$9d36934a-78cb-446b-b3db-1bbd88cf272dŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTc‘m°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$9d36934a-78cb-446b-b3db-1bbd88cf272d¹depends_on_disabled_cellsÂ§runtimeÎó!µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$d24bd737-9e09-441a-aa94-9279c80f566dŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÛÁo

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšX0°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$d24bd737-9e09-441a-aa94-9279c80f566d¹depends_on_disabled_cellsÂ§runtimeÏT{kÇµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$69b560c1-98ad-4cbf-89d2-e0516299bc69Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTkŽ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$69b560c1-98ad-4cbf-89d2-e0516299bc69¹depends_on_disabled_cellsÂ§runtimeÎbÎµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$24759e26-e670-4330-b6e4-b313620660f1Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ0calculate_Ï€vec (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTGÜ@°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$24759e26-e670-4330-b6e4-b313620660f1¹depends_on_disabled_cellsÂ§runtimeÎìœµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$22fa2b71-a98f-4b87-9e0b-9d373cd8915fŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ8stationary_param_search (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšX.3œ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$22fa2b71-a98f-4b87-9e0b-9d373cd8915f¹depends_on_disabled_cellsÂ§runtimeÎãÕµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$f561b0a8-a086-4e1a-bc87-82c4205e89c9Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTg¤r°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$f561b0a8-a086-4e1a-bc87-82c4205e89c9¹depends_on_disabled_cellsÂ§runtimeÎ\µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$85b3dbc4-2e02-4924-b3e5-887a1a557a2bŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ%fmax (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšR¸÷°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$85b3dbc4-2e02-4924-b3e5-887a1a557a2b¹depends_on_disabled_cellsÂ§runtimeÎ2Eµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3eŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ/sample_action (generic function with 4 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTla™°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3e¹depends_on_disabled_cellsÂ§runtimeÎ }µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$cb93c588-3dfa-45f4-9d83-f2de26cb1ceaŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ*

Exercise 2.7: Unbiased Constant-Step-Size Tick

In most of this chapter we have used sample averages to estimate action values because sample averages do not produce the initial bias that constant step sizes do (see analysis leading to (2.6)). However, sample averages are not a completely satisfactory solution because they may perform poorly on nonstationary problems. Is it possible to avoid the bias of constant sample sizes while retaining their advantages on nonstationary problems? One way is to use a step size of $\beta_n \dot= \alpha / \bar{o}_n,$ to process the nth reward for a particular action, where $\alpha>0$ is a conventional constant step size, and $\bar{o}_n$ is a trace of one that starts at 0: $\bar{o}_n \dot= \bar{o}_{n-1}+\alpha(1-\bar{o}_{n-1}), \text{ for } n \geq 0, \text{ with } \bar{o}_0 \dot= 0.$ Carry out an analysis like that in (2.6) to show that $Q_n$ is an exponential recency-weighted average without initial bias.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN5±°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$cb93c588-3dfa-45f4-9d83-f2de26cb1cea¹depends_on_disabled_cellsÂ§runtimeÎhîµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$5712b303-0aa3-4501-b1b5-020136d6e655Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTgq—°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$5712b303-0aa3-4501-b1b5-020136d6e655¹depends_on_disabled_cellsÂ§runtimeÎä%µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$ca726a9d-364d-48e2-8882-20ddbc85b664Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTcdí°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$ca726a9d-364d-48e2-8882-20ddbc85b664¹depends_on_disabled_cellsÂ§runtimeÎ„xµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1e4ac085-7b72-4bad-ad87-21635930a6f7Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

Exercise 2.1

In Ïµ-greedy action selection, for the case of two actions and $\epsilon = 0.5$, what is the probability that the greedy action is selected?

The greedy action could be selected in two cases, each of which has probability 0.5. For case 1 the greedy action is selected with probability 1. For case 2, we select an action randomly, so the probability of selecting the greedy action is $\frac{\text{num greedy actions}}{\text{num total actions}}=0.5$ Since both cases are independent, the probabilities can be summed after multiplying each by the probability of that case which is 0.5 for both.

$$P(a = a_{greedy}) = 0.5 \times (1 + 0.5) = 0.5 + 0.25 = 0.75$$

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNóé°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1e4ac085-7b72-4bad-ad87-21635930a6f7¹depends_on_disabled_cellsÂ§runtimeÎ,áµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$bf3770ea-ee54-4296-ab33-340aea445670Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ¨

Figure 2.6

Parameter study of bandit algorithms on the 10-armed testbed for stationary normally distributed bandit rewards.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNèê°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$bf3770ea-ee54-4296-ab33-340aea445670¹depends_on_disabled_cellsÂ§runtimeÎ±Ãµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$db30e6c0-36bb-4602-a257-5768f3833525Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

Chapter 2: Multi-armed Bandits

Reinforcement learning is distinct from other types of learning in its use of training information to evaluate actions taken rather than simply being instructed. In supervised learning, correct actions would be given explicitely independent of the actual behavior. Evaluative feedback, on the other hand depends entirely on the action taken.

In this chapter, we study evaluative feedback in a nonassociative setting, that is one in which the situation does not change. In doing so, we avoid must of the complexity of the full reinforcement learning problem while introducing a technique that will be important throughtout the rest of the book.

2.1: A k-armed Bandit Problem

Consider a repeated choice among k different options. A numerical reward is chosen from a stationary probability distribution that depends only on the action selected. The objective is to maximize the accumulated reward over some time period, let's say 1000 action selections or time steps. The bandit described here can be specified by $\Pr \{ r \mid a \}$ for all $a$ where $\Pr$ is some probability distribution with a well defined mean value: $\mathbb{E}[r \vert a] = r_a \; \forall a$.

We denote the action selected on time step t as $A_t,$ and the corresponding reward as $R_t.$ The value then of an arbitrary action $a$, denoted $q_*(a),$ is the expected reward given that $a$ is selected:

$$q_*(a) \doteq \mathbb{E}[R_t \vert A_t=a]$$

If we know the expected values, then the problem is trivial, but we assume that we only have estimates of the values at a time step $a$ which we will denote $Q_t(a).$ At any given time step, the greedy action is the one with the highest value estimate. If we take non-greedy actions then we can improve our value estimate for other states. To solve the problem in general we must balance exploiting the action estimated to be the best with exploring the values of other candidate actions. What follows are various methods to balance these two choices.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNŽ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$db30e6c0-36bb-4602-a257-5768f3833525¹depends_on_disabled_cellsÂ§runtimeÎXýµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$4982b489-ca15-4188-90e5-565c45f02e01Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ/sample_action (generic function with 2 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTkúU°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$4982b489-ca15-4188-90e5-565c45f02e01¹depends_on_disabled_cellsÂ§runtimeÎ9Tµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$6ce00349-1cf0-4a80-bddd-c1d26b66d051Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXjÒk°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$6ce00349-1cf0-4a80-bddd-c1d26b66d051¹depends_on_disabled_cellsÂ§runtimeÎÉŽµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$b29624b0-be1d-4cc2-964d-0a050f4c1bedŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXú‡ì°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$b29624b0-be1d-4cc2-964d-0a050f4c1bed¹depends_on_disabled_cellsÂ§runtimeÎò–µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$400e0de1-0101-4531-928f-08ca155da40cŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ-preparetrace (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXB’l°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$400e0de1-0101-4531-928f-08ca155da40c¹depends_on_disabled_cellsÂ§runtimeÎ^²kµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8a04adab-e97e-4ac4-a85e-5eae93b1c37bŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTk@y°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8a04adab-e97e-4ac4-a85e-5eae93b1c37b¹depends_on_disabled_cellsÂ§runtimeÎ'D›µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$3e5a226e-ecdb-43fb-a40a-a262da0ae542Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚÈ

Number of Arms in Testbed

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšQ-f°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$3e5a226e-ecdb-43fb-a40a-a262da0ae542¹depends_on_disabled_cellsÂ§runtimeÎ$‹Ýµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$7a31d1c5-260b-42f4-b997-967150881e21Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTgHA°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$7a31d1c5-260b-42f4-b997-967150881e21¹depends_on_disabled_cellsÂ§runtimeÎmµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8584ece7-badc-486a-9a57-b60e77f92673Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÛ | ¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšSÅ£×°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8584ece7-badc-486a-9a57-b60e77f92673¹depends_on_disabled_cellsÂ§runtimeÎ4X¦>µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$e07a27c5-0c9a-4893-a1cf-cf565ab78761Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ ;

2.4 Incremental Implementation

The sample-average for the action-value estimate is defined as $Q_n \dot = \frac{R_1+R_2+\cdots+R_{n-1}}{n-1}$. To compute this we can maintain a record of every reward and every time we accumulate a new reward recompute the entire average. However that is inefficient in terms of computational and memory resources. It is possible to instead maintain a single value for the estimate at step n (and n itself) and update it incrementally every time we obtain a new reward.

The update formula for $Q_{n+1}$ when we obtain a new reward sample is derived below:

$$\begin{flalign} Q_{n+1} &= \frac{1}{n} \sum_{i=1}^n R_i \\ &= \frac{1}{n} \left ( R_n + \sum_{i=1}^{n-1} R_i \right ) \tag{separate final sum term}\\ &= \frac{1}{n} \left ( R_n + (n-1) \frac{1}{n-1} \sum_{i=1}^{n-1} R_i \right ) \tag{multiply by 1} \\ &= \frac{1}{n} \left ( R_n + (n-1) Q_n \right ) \tag{definition of Q} \\ &= \frac{1}{n} \left ( R_n + n Q_n - Q_n \right ) \\ &= Q_n + \frac{1}{n} \left [ R_n + Q_n \right ] \tag{2.3} \end{flalign}$$

The update rule (2.3) is of a form that occurs frequently whose general form is

$$NewEstimate \leftarrow OldEstimate + StepSize \left [ Target - OldEstimate \right ] \tag{2.4}$$

The expression $[Target - OldEstimate]$ is an error in the estimate. It is reuced by taking a step toward the "Target." The target is presumed to indicate a desireable direction in which to move, though it may be noisy. In the case above, for example, the target is the nth reward.

Notice that the step size parameter in this case is $\frac{1}{n}$ but in general it can be constant or depend on the step count and the action itself. In this case it is denoted $\alpha_t (a)$. The simple bandit algorithm that uses this incremental update rule is implemented above in section 2.3.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN"«°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$e07a27c5-0c9a-4893-a1cf-cf565ab78761¹depends_on_disabled_cellsÂ§runtimeÎ ;9µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$b24a92fc-f6c6-44e4-9afc-fa4249e4ab83Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚk

2.8 Gradient Bandit Algorithms

As an alternative to estimating action values, we can attempt to learn a numerical preference for each action $a$ which we will denote $H_t(a) \in \mathbf{R}.$ This vector of preferences will be converted in a probability distribution using the soft-max distribution.

$$\Pr\{A_t = a\} \doteq \frac{e^{H_t(a)}}{\sum_{b=1}^k e^{H_t(b)}} \doteq \pi_t(a) \tag{2.11}$$

$\pi_t(a)$ is the probability for this agent to select action $a$ at time $t.$ All action preferences are initialized at the same value.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNÜµ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$b24a92fc-f6c6-44e4-9afc-fa4249e4ab83¹depends_on_disabled_cellsÂ§runtimeÎ|îµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$ea6d7cad-47ad-4472-a9e9-1ee33c81058dŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ*update_H! (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTN3°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$ea6d7cad-47ad-4472-a9e9-1ee33c81058d¹depends_on_disabled_cellsÂ§runtimeÎ+»µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$74024d96-d0c7-43c8-8379-caf843cbe4b8Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙFaverage_nonstationary_runs_cum_reward (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXÇ`°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$74024d96-d0c7-43c8-8379-caf843cbe4b8¹depends_on_disabled_cellsÂ§runtimeÎCI¾µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1c9b54cd-08dd-401e-9705-818741844e8dŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

Code Refactoring

Due to the variety of algorithms and parameters for the bandit, I have rewritten the test environment with types that represent the different algorithms. The run simulator will dispatch on the types to correctly simulate that method with its parameters. Some of the previous simluations and plots are generated again. Because of the style used, only one simulation function is needed with the flexibility to select any combination of techniques in the chapter.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNzt°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1c9b54cd-08dd-401e-9705-818741844e8d¹depends_on_disabled_cellsÂ§runtimeÎh6µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$e2597cc6-a6f6-4260-887b-c587cacd3bc8Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTfÌ–°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$e2597cc6-a6f6-4260-887b-c587cacd3bc8¹depends_on_disabled_cellsÂ§runtimeÎØ+µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$47be3ae6-20f7-47d0-aae3-b67154afc1a8Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTd„Ó°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$47be3ae6-20f7-47d0-aae3-b67154afc1a8¹depends_on_disabled_cellsÂ§runtimeÎð|µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$70e40b75-e7d8-4009-af80-3bf4086a28dfŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ.

Figure 2.3 Parameters

Default values match textbook

Number of Actions: Initial Optimistic Estimate:

Exploration Parameter Ïµ: Constant Step Size Î±:

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšSÓ¿è°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$70e40b75-e7d8-4009-af80-3bf4086a28df¹depends_on_disabled_cellsÂ§runtimeÎNâ¬µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$cb2bf56d-5c1d-4265-9480-11523f776a78Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

This expression will look simpler if we treat the probabilities as variables themselves.

$$f(p_i) = \frac{\sum_i p_i q_i}{\sum_i p_i}$$

$$\frac{\partial{f(p_i)}}{\partial p_a} = \frac{q_a - \sum_i p_i q_i}{(\sum_i p_i)^2}$$

Setting this equal to 0 for all the partials implies the following:

$$q_a = \sum_i p_i q_i \quad \forall \quad a$$

Let's say we have 2 actions. Then we have 2 equations with two unknowns.

$$\begin{flalign} q_1 &= p_1 q_1 + p_2 q_2 \implies p_1 = \frac{q_1 - p_2 q_2}{q_1} \\ q_2 &= p_1 q_1 + p_2 q_2 \implies p_2 = \frac{q_2 - p_1 q_1}{q_2} \\ p_1 &= \frac{q_1 - q_2 + p_1 q_1}{q_1} \implies q_1 = q_2 \\ \therefore \\ q &= p_1 q + p_2 q \implies p_1 = p_2 \end{flalign}$$

From the initial equations we see this necessitates all of the q's to be equal, and in this case the optimal solution is for the probabilities to also be equal.

We can also simplify this case with two actions because $p_2 = 1-p_1$ so let's just call $p_1 = p$ and $p_2 = 1-p$. Then we are trying to maximize $(f(p) = pq_1 + (1-p)q_2$ and we can just take the derivative in terms of p and set that to 0.

$$q_1 - q_2 = 0$$

which again gives us an unreasonable constraint on the q values. That is because the only solution with a finite maximum is one in which the probabilities are equal. In all other cases the extremum only occurs at one of the probabilities being infinite. We can fix this by sticking with one variable but forcing it to be a true probability.

$$p(x) = \frac{e^x}{e^x + 1}$$

So now $f(x) = p(x)q_1 + (1-p(x))q_2 = p(x)q_1 + p(-x)q_2$

$$\frac{d f(x)}{d x} = \frac{d p(x)}{d x} q_1 - \frac{d p(x)}{dx} q_2 = \frac{d p(x)}{d x}(q_1 - q_2)$$

Now we can see there are two options for this derivative being 0. We first have the case where the q values are equal which causes the probabilities to be equal. Let's consider the case where $q_1 \neq q_2$. Now we must have:

$$\frac{d p(x)}{dx} = e^{-x}(1+e^{-x})^{-2} = 0$$

Consider first the case of very large x so the exponential term is close to 0. In this case the limit as $x \rightarrow \infty$ approaches $\frac{0}{1}=0$.

Now consider the second case of $x \rightarrow -\infty$. In this case the exponential term is much larger than 1 and the expression simplifies to $\frac{e^{-x}}{e^{-2x}} = e^x$. And the limit of $e^x$ as $x \rightarrow -\infty$ is 0.

So we have shown that the x that solves this maximization can either trend towards positive or negative infinity which corresponds to the probability $p(x) = \{0, 1\}$. These two cases correspond to the case where $q_1 > q_2$ and $q_1 < q_2$ which can be shown by taking the second derivative.

This same argument applies to the softmax as well in the limit such that the solution that maximizes the expected value is one in which the largest q value probability is 1 and all the rest are 0. So if we were to maximize the gradient fully after some set number of samples we'd always arrive at a distribution that only selects the greedy action. In this algorithm the only thing that controls whether we select other actions is the fact that at any given sample point we have yet to converge. This is not a very precise way to control exploration and it isn't surprising that simply adjusting the step size does not yield optimal results.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN!Ÿ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$cb2bf56d-5c1d-4265-9480-11523f776a78¹depends_on_disabled_cellsÂ§runtimeÎüßµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$f9e60b35-84d2-4b4b-8832-6b0f08152396Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ³

Shows the reward mean for the arms at each ranking from 1 to 10. Each arm starts at 0 mean reward and is perturbed by a normal random variable with Ïƒ = 0.01 at each step. The rewards for an arm at a particular ranking seem to track the standard deviation of the overall distribution for the drift process. The functional form of this is $\sqrt{n}$ where $n$ is the number of steps so far. Depending on the number of arms, the multiplicative factor on the curve changes but for the 10 armed case, the second best arm seems to match the value for 1 standard deviation above the mean of 0.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšSÆò°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$f9e60b35-84d2-4b4b-8832-6b0f08152396¹depends_on_disabled_cellsÂ§runtimeÎ®¤µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$083c721c-70dd-4ca3-8160-fc0b0531914fŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ‚

2.3 The 10-armed Testbed

To roughly assess the relative effectiveness of the greedy and $\epsilon$-greedy action-value methods, we compare them numerically on a suite of test problems. The tests are a set of 2000 randomly generated $k$-armed bandit problems with $k=10$. For each bandit problem, the action values, $q_*(a), \; a = 1,\dots,10,$ were selected according to a normal (Gaussian) distribution with a mean of 0 and variance 1. The probability distribution for the reward from each arm is then a gaussian with unit-variance and the appropriate mean corresponding to that arm. For any learning method, can we measure its performance and behavior as it improves with experience over 1000 time steps when applied to one of the bandit problems. This makes up one run. Experiments results are obtained by repeating this process for 2000 independent runs, each with a different bandit problem.

The following code recreates the 10-armed Testbed from section 2.3

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNH°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$083c721c-70dd-4ca3-8160-fc0b0531914f¹depends_on_disabled_cellsÂ§runtimeÎâ/µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$49e45202-b9ae-42ab-9575-a57edb626a20Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšThmD°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$49e45202-b9ae-42ab-9575-a57edb626a20¹depends_on_disabled_cellsÂ§runtimeÎ¡!µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$ff6598fa-3366-416c-88a1-6bfcefeb1719Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTie®°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$ff6598fa-3366-416c-88a1-6bfcefeb1719¹depends_on_disabled_cellsÂ§runtimeÎÛ^µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$647ab36b-641e-4024-ad2d-40ff33be28f4Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ;average_nonstationary_runs (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšSE›Z°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$647ab36b-641e-4024-ad2d-40ff33be28f4¹depends_on_disabled_cellsÂ§runtimeÎ„gµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$96566aad-6d5c-460c-a924-ae0bad5d8b2dŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ,updatecoef (generic function with 3 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšU?Ú8°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$96566aad-6d5c-460c-a924-ae0bad5d8b2d¹depends_on_disabled_cellsÂ§runtimeÎÔšµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$852df31d-18d8-466c-8225-e06ba7f05e96Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTc ¢°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$852df31d-18d8-466c-8225-e06ba7f05e96¹depends_on_disabled_cellsÂ§runtimeÎD©µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$a84f5393-bd7e-433e-b3d9-e9e7cfa1a329Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚí

In the case of a non-stationary problem, the sample-average method is not ideal because it weights samples from the past equally to the present. We can change the incremental implementation of the average to weight more recent rewards higher than past ones. A constant step-size parameter is one way of doing this. In this case we will change the update rule (2.3) to:

$$Q_{n+1} \doteq Q_n + \alpha [R_n - Q_n] \tag{2.5}$$

By writing this as an explicit sum over all rewards, one can observe that this update rule computes a weight average whose weights exponentially decay into the past. See a similar derivation in exercise 2.4. In order to guarantee that Q converges to the true expected value, the step size parameter must obey the following relationships:

$$\sum_{n=1}^\infty \alpha_n(a) = \infty \quad \quad \text{and} \quad \quad \sum_{n=1}^\infty \alpha_n^2(a) < \infty \tag{2.7}$$

The first condition ensures steps are large enough to overcome initial conditions and the second condition insures steps are small enough to converge. These are both met by the sample average step size of $\frac{1}{n}$ for not by the constant step size. That is desireable in a non-stationary environment where there is no stable value to converge to in the first place.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN`t°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$a84f5393-bd7e-433e-b3d9-e9e7cfa1a329¹depends_on_disabled_cellsÂ§runtimeÎh2µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$9bd99099-1dfa-477a-9896-3da94bcc0633Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ.exercise_2_11 (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšX°m]°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$9bd99099-1dfa-477a-9896-3da94bcc0633¹depends_on_disabled_cellsÂ§runtimeÎ"µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$3b9bb9f0-ba9d-4320-935a-58912afe34b6Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTfø¡°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$3b9bb9f0-ba9d-4320-935a-58912afe34b6¹depends_on_disabled_cellsÂ§runtimeÎ¾Ÿµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$14bd0549-747f-4513-809f-8bdb78027807Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ£

$\epsilon - Greedy$ Action Value Method

The functions below implement the sample-average method for estimating the value of each action with the Ïµ-greedy method of action selection. Note that the simple_algorithm uses the incremental implementation of calculating the sample average which is described in section 2.4.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN9°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$14bd0549-747f-4513-809f-8bdb78027807¹depends_on_disabled_cellsÂ§runtimeÎ¾½µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2eŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTjR°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2e¹depends_on_disabled_cellsÂ§runtimeÎÁ‰µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1fb1a518-e5ec-4777-80bc-bb55e8172100Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚMŒ ¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšPd N°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1fb1a518-e5ec-4777-80bc-bb55e8172100¹depends_on_disabled_cellsÂ§runtimeÎ6×í²µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1e1f6d10-1b31-4e0a-96f7-23207e913154Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTi¹ °persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1e1f6d10-1b31-4e0a-96f7-23207e913154¹depends_on_disabled_cellsÂ§runtimeÎÌ µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$44b9ff95-ea3d-41f5-8098-445a263738a9Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTc6‹°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$44b9ff95-ea3d-41f5-8098-445a263738a9¹depends_on_disabled_cellsÂ§runtimeÎäµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$6034cc0f-cbae-4d2f-a43f-1bb738c00f0bŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚñ

Testbed Params

The armed testbed with Ïµ-greedy 1 = and Ïµ-greedy 2 = over steps

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšR-§°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$6034cc0f-cbae-4d2f-a43f-1bb738c00f0b¹depends_on_disabled_cellsÂ§runtimeÎW¸µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$aa7c76b1-ac2d-4df5-88ef-b499b839181eŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ6

Update Functions

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN°°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$aa7c76b1-ac2d-4df5-88ef-b499b839181e¹depends_on_disabled_cellsÂ§runtimeÎæµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$0f6b4e2d-dc09-4e1c-834f-dd8aaa8743aeŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTg?°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$0f6b4e2d-dc09-4e1c-834f-dd8aaa8743ae¹depends_on_disabled_cellsÂ§runtimeÎ#¶µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$f7519adc-7dfb-4030-86f0-7445699dd3dbŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙEgradient_stationary_bandit_algorithm (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTtqz°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$f7519adc-7dfb-4030-86f0-7445699dd3db¹depends_on_disabled_cellsÂ§runtimeÎFÉhµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$181d7eef-24a0-4775-a535-8ef901b7e4ebŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ

The above plot shows the probability of selecting hte first action when its mean reward estimate is a given value above or below the alternative. The number of samples for each can be controlled above.

How easy is it to extend this to multiple actions? We would need to calculate the probability that a given action was best among all the alternatives:

$$P(a_i = a_{best}) = P(q_i > max(q_j \forall j \neq i))$$

Consider the case of 3 actions. If we have a distribution for the mean value of each, then there could be six possible rankings excluding the cases where the estimates are equal:

$$q_1 > q_2 > q_3$$

$$q_1 > q_3 > q_2$$

$$q_2 > q_3 > q_1$$

$$q_2 > q_1 > q_3$$

$$q_3 > q_1 > q_2$$

$$q_3 > q_2 > q_1$$

For the first ranking, we can consider two new normal random variable X = $q_2 - q_1$ and Y = $q_3 - q_2$. Now (X, Y) follows a multivariate distribution with means $\mu_2 - \mu_1$ and $\mu_3 - \mu_2$. The probability of this ordering can then be calculated by calculating the probability that both X The variances of X and Y are just equal to the sum of the individual variances for each pair of q's. However, since this is a multivariate distribution we also need the covariance. $\sigma(X, Y) = \sigma(q_2 - q_1, q_3 - q_2) = \sigma(q_2, q_3 - q_2) - \sigma(q_1, q_3 - q_2) = -\sigma^2(q_2)$ since all of the variables are independent. Now the probability follows simply from calculating the CDF of this distribution and evaluating it at (0, 0). This represents the probability that $q_1 > q_2$ and $q_2 > q_3$ simultaneously. We would then have to calculate this for each of the above orderings to get the true distribution of each q being maximum. In the case of even more actions this becomes extremely tedious although in the case of infinitely many actions it simplifies to an easier calculation in the limit. But we are concerned with cases in which the number of actions is quite small like 10.

One alternative to calculating this distribution is to simply sample from it. That is very easy because we need only generate a sample from each q distribution and then examine it to see which action produced the highest value. If we performed this sampling many times and kept track of which q appears on top, we would indeed recover the true distribution. But even better than this is that each sample is itself an unbiased estimator for this probability. Therefore we need only generate a single sample from this distribution and use that to select our action. Since we are interacting with the bandit for many time steps we will eventually collect enough samples to have a good distribution. Using this method of action selection is shown under the optimal distribution tag in the parameter study and it seems to perform quite well without any parameter tuning.

A further question could be which action selection is best for improving our accuracy of the distribution, but this requires more thought on how to formulate the problem. Intuitively we want to sample more around the part of the distribution where pairs of actions are hard to distinguish. We could rank all the pairs in terms of how indistinguished they are weighted by how likely they are to be the maximum and then sample from the action that appears the most.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN€Ü°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$5aa170d3-eb87-44dd-b4f4-3ac97476efd7¹depends_on_disabled_cellsÂ§runtimeÎ µgµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080bŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ+run_bandit (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšUN=K°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b¹depends_on_disabled_cellsÂ§runtimeÎpˆßµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$7cab41cc-2a39-45de-879d-f7f6105238d6Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ,^

Constant Step Size Î±: Max Samples: 1002

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšYBá°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$7cab41cc-2a39-45de-879d-f7f6105238d6¹depends_on_disabled_cellsÂ§runtimeÎ êµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$e50596ab-91db-42f0-a62c-77629a4e79c7Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTdå&°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$e50596ab-91db-42f0-a62c-77629a4e79c7¹depends_on_disabled_cellsÂ§runtimeÎwŸµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$fb3381b5-10e3-4307-b76d-672245fac9e7Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTi<°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$fb3381b5-10e3-4307-b76d-672245fac9e7¹depends_on_disabled_cellsÂ§runtimeÎ7µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$3118e102-aeac-42d9-98fc-ca29f40be4cdŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTd¨°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$3118e102-aeac-42d9-98fc-ca29f40be4cd¹depends_on_disabled_cellsÂ§runtimeÎØLµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$b3ec4673-af63-4d4a-8314-fa7e594f8a37Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ‹Ÿ ¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšSô*°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$b3ec4673-af63-4d4a-8314-fa7e594f8a37¹depends_on_disabled_cellsÂ§runtimeÎvOìµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$c2347999-5ade-420b-903f-30523b38eb0fŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTabÇ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$c2347999-5ade-420b-903f-30523b38eb0f¹depends_on_disabled_cellsÂ§runtimeÎêÓµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$1ac5588c-3c32-436e-8b40-41715223fba7Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTe±È°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$1ac5588c-3c32-436e-8b40-41715223fba7¹depends_on_disabled_cellsÂ§runtimeÎ÷íµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$34f65898-cbcc-4832-afac-0f7a284e7f0bŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ,exercise2_5 (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšUgqX°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$34f65898-cbcc-4832-afac-0f7a284e7f0b¹depends_on_disabled_cellsÂ§runtimeÎ@½Ìµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$4ebd4a5a-3bd1-48e2-b03e-a5a3b2ec18a1Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšThÁ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$4ebd4a5a-3bd1-48e2-b03e-a5a3b2ec18a1¹depends_on_disabled_cellsÂ§runtimeÎ¯Íµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$093f312b-d70d-4bf7-bd53-8a1c7b2bee31Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚJ

Figure 2.4 Parameters

Number of Actions: Confidence Bound c: Exploration parameter Ïµ:

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšT„s°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$093f312b-d70d-4bf7-bd53-8a1c7b2bee31¹depends_on_disabled_cellsÂ§runtimeÎ"Zeµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$7748ab8a-d186-49a4-b6ab-d1bd9ea34990Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙE

Bandit Arm Reward Distributions

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNÕN°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$7748ab8a-d186-49a4-b6ab-d1bd9ea34990¹depends_on_disabled_cellsÂ§runtimeÎâCµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$140f1e20-f86d-4a6f-9cff-99685e129e1cŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ=plot_stationary_param_search (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXQé–°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$140f1e20-f86d-4a6f-9cff-99685e129e1c¹depends_on_disabled_cellsÂ§runtimeÎàòGµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$965da91b-6a3f-456c-89ce-461c31e0fb7eŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

Exercise 2.2: Bandit example

Consider a k-armed bandit problem with k = 4 actions, denoted 1, 2, 3, and 4. Consider applying to this problem a bandit algorithm using $\epsilon$-greedy action selection, sample-average action-value estimates, and initial estimates of $Q_1(a) = 0$, for all a. Suppose the initial sequence of actions and rewards is $A_1 = 1,$ $R_1 = âˆ’1,$ $A_2 = 2,$ $R_2 = 1,$ $A_3 = 2,$ $R_3 = âˆ’2,$ $A_4 = 2,$ $R_4 = 2,$ $A_5 = 3,$ $R_5 = 0.$ On some of these time steps the $\epsilon$ case may have occurred, causing an action to be selected at random. On which time steps did this definitely occur? On which time steps could this possibly have occurred?

The table below summarizes the actions taken leading into every step and the Q estimate for each action at the end of each step. So step 0 shows the initial Q estimates of 0 for every action and the selected action 1 that generates the reward on step 1. For the row in step 1 it shows the Q estimates after receiving the reward on step 1 and thus what actions are demanded by a greedy choice leading into the next step. If the action selected is not in the set of greedy actions, then a random action must have occured. Since a random action choice can also select one of the greedy actions, such a random choice is possible at every step. Note that the answer in row 0 corresponds to action $A_1$, row 1 -> $A_2$ etc...

Step	Action Selected	Reward	$Q(1)$	$Q(2)$	Greedy Action Set	Greedy Selection	$\epsilon$ Case
1	1	-1	0	0	$\{1, 2, 3, 4\}$	True	possibly
2	2	1	-1	0	$\{2, 3, 4\}$	True	possibly
3	2	-2	-1	1	$\{2\}$	True	possibly
4	2	2	-1	$-\frac{1}{2}$	$\{3, 4\}$	False	definitely
5	3	0	-1	$\frac{1}{3}$	$\{2\}$	False	definitely

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN³°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$965da91b-6a3f-456c-89ce-461c31e0fb7e¹depends_on_disabled_cellsÂ§runtimeÎèfµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$9d7782f5-b530-40d5-9f75-280d3a762216Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ0

Exercise 2.9

Show that in the case of two actions, the soft-max distribution is the same as that given by the logistic, or sigmoid, function often used in statistics and artificial neural networks.

The sigmoid function is defined as: $S(x) = \frac{1}{1 + e^{-x}}.$ For two actions, let's denote them $a_1$ and $a_2.$ Now for the action probabilities we have.

$$\pi(a_1) = \frac{e^{H_t(a_1)}}{e^{H_t(a_1)} + e^{H_t(a_2)}}=\frac{1}{1+e^{-(H_t(a_1) - H_t(a_2))}}$$

This expression for $\pi(a_1)$ is equivalent to $S(x)$ with $x = H_t(a_1) - H_t(a_2)$ which is the degree of preference for action 1 over action 2. As expected, if the preferences are equal then it is equavalent to $x=0$ with a probability of 50%. The same analysis applies to action 2 with the actions reversed from this case.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNüô°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$9d7782f5-b530-40d5-9f75-280d3a762216¹depends_on_disabled_cellsÂ§runtimeÎ„µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$bc939df7-e457-496c-8977-5fbf9dfe3638Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ]

Alternative Distribution Method

The gradient bandit algorithm is motivated by maximizing an expected value which requires knowledge of the q values. However, the policy that maximizes that expected value will always be greedy with respect to whichever q value is thought to be maximal. In reality we do not have certainty over which q value is maximal and instead we have some knowledge of each q value based on samples collected. We know that the optimal choice only depends on the mean value of the rewards for each action. Our knowledge of the mean is from the reward samples collected, so we can use the distribution of the sample mean to represent our knowledge of the true q values. That would be a normal distribution with

$$\mu(a) = \sum_{A_t = a} R_t / N_{a}$$

and

$$\sigma(a)^2 = \frac{\sum_{A_t = a} (R_t - \mu(a))^2}{N_a^2}$$

If we seek to maximize the expected reward per step as before, all that matters for that is the mean value of each q or in our case the unbiased estimate we have of the mean value. This will also result in a greedy policy choosing which ever estimate is higher. We might explore actions but only in the case of the estimates fluctuating. If we ever consider a method that seeks to maximize the expected reward at a single step, we will always arrive at a greedy policy. One alternative we could consider is sampling actions based on the probability that what we've selected is optimal. In other words calculate $P(x = a_*) \quad \forall \quad x$

To simplify matters let's consider the case of only two actions and also assume that we already know the true variance of the reward for each action about the mean. That way our distribution for q simplifies to a normal distribution with a mean of $\mu(a) = \sum_{A_t = a}R_t / N_a$ and a variance of $\sigma(a) = 1/N_a$. Now we have two actions $1$ and $2$ each with their own samples. So our knowledge of $q_1$ and $q_2$ consists of two normal distributions: $N(\mu_1, \sigma_1^2)$ and $N(\mu_2, \sigma_2^2)$ with the mean and variance calculated as mentioned above. So what is the probability that action 1 is optimal? This is the same as asking the probability that a sample from the first distribution is larger than a sample from the second distribution.

$$P(X>Y)$$

with $X \sim N(\mu_1, \sigma_1^2)$ and $Y \sim N(\mu_2, \sigma_2^2)$

This is equivalent to asking $P(X-Y) > 0$. Since $X-Y$ is a sum of normal distributions, it is itself a normal distribution: $N(\mu_1 - \mu_2, \sigma_1^2 + \sigma_2^2)$. The cumulative distribution function will answer this question since it is the probability of a random variable being less than or equal to a given value. So $1-CDF(0)$ will be our answer where $CDF(0)$ is the cummulative distribution function of our normal variable evaluated at 0: $\frac{1}{2}\left[1+\text{erf}\left(\frac{-\mu_1 + \mu_2}{\sqrt{2(\sigma_1^2 + \sigma_2^2)}} \right ) \right]$. So this would be the probability of selecting action 1 and 1 minus this would be the probability of selecting action 2. The assumptions made here are that the rewards are generated by a stationary process with some unknown mean and unit variance. This method is different from any discussed earlier because we have a stochastic policy yet it is based on estimating the q values.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNQ °persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$bc939df7-e457-496c-8977-5fbf9dfe3638¹depends_on_disabled_cellsÂ§runtimeÎ nµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$464d43c0-cd59-49e6-88f6-12a767677418Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ ’

Exercise 2.3

In the comparison shown in Figure 2.2, which method will perform best in the long run in terms of cumulative reward and probability of selecting the best action? How much better will it be? Express your answer quantitatively.

In the long run both the $\epsilon = 0.1$ and $\epsilon = 0.01$ methods will have Q value estimates that converge to the true mean value of the reward distribution. Since both methods will necessarily take random actions 10% and 1% of the time respectively, we'd expect each method to take the optimal action with probability $(1-\epsilon) + \epsilon \times \frac{1}{10}=\frac{10 - 9 \times \epsilon}{10}$. So for each value of Ïµ we have.

$$Pr(a=a_{best}|\epsilon = 0.1) = 0.91$$

$$Pr(a=a_{best}|\epsilon = 0.01) = 0.991$$

For the $\epsilon = 0$ greedy case the expected reward and optimal action selection probability depends on the order of sampled actions and the likelihood of getting close to or on the optimal action enough to push its Q estimation to the top. From the plots in figure 2.2 in practice that seems to lead to an average reward of ~1.05 and an optimal action selection probability of 0.3825. For long term cummulative reward this case will have roughly $1.05 \times num\_steps$. For the other two cases, the long term cummulative reward is based on the expected value of the highest reward mean which is approximately ~1.539 (see below for calculation of bandit arm reward distributions). For a random action the expected reward should be 0 due to the normal distribution of the action mean rewards.

$$E(long\_term\_step\_reward|\epsilon=0.1) = 0.91 \times 1.539 = 1.4$$

$$E(long\_term\_step\_reward|\epsilon=0.01) = 0.991 \times 1.539 = 1.525$$

For each case the long run cumulative reward is just this long term expected reward per step times the number of steps. The statistical properties of the different arms of a generic bandit can be visualized below using the following function.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN×°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$464d43c0-cd59-49e6-88f6-12a767677418¹depends_on_disabled_cellsÂ§runtimeÎCµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$21e56374-35e6-4488-b8da-15e383017c77Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙE

Bandit Arm Reward Distributions

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNò®°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$21e56374-35e6-4488-b8da-15e383017c77¹depends_on_disabled_cellsÂ§runtimeÎçmµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$f33adafb-44b4-4e56-b3de-9717fe026288Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ8

Sampling Functions

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN•h°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$f33adafb-44b4-4e56-b3de-9717fe026288¹depends_on_disabled_cellsÂ§runtimeÎáIµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$7c2015dd-a786-49f5-9fe9-9199335ebd09Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ*maketrace (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXJ¢°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$7c2015dd-a786-49f5-9fe9-9199335ebd09¹depends_on_disabled_cellsÂ§runtimeÎ¨kµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$ecd8833d-441b-4673-b457-dc5109f575abŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚò

Why softmax?

For the gradient bandit algorithm we have a probability distribution over actions $\pi_t(x)$ and we seek to maximize the expected reward at each time step $\mathbb{E}[R_t]=\sum_x \pi_t(x) q_*(x)$. Since we do not know the true values of $q_*$ the algorithm simply replaces this with the sample reward collected at that time step. This is justifyable because $\mathbb{E}[R_t|A_t] = q_*(A_t)$. Using this method we only take one gradient step each sample so after a large number of steps we have sampled many rewards from each action and have better estimates for $q_*$.

$$\pi_t(x)$$

is a discrete distribution with one value for each action. We can represent that simply with a parameter for each action, let's call it $H(x)$ we seek to find the values $H(x)$ which maximize the expected value of reward by setting the partial derivatives with respect to those parameters equal to 0.

$$\frac{\partial{\mathbb{E}[R_t]}}{\partial{H_t(x)}} = \frac{\partial}{\partial H_t(a)} \left ( \sum_x H_t(x) q_*(x) \right ) = \sum_x q_*(x)=0$$

This is a problem because we do not have control over the $q_*$ values. So perhaps we should consider some other function of $H_t(x)$. In particular we know that the probabilities need to be positive so why not select a function that is always positive like $f(x) = x^2$. What expression for the derivative do we get in this case?

$$\frac{\partial{\mathbb{E}[R_t]}}{\partial{H_t(x)}} = \frac{\partial}{\partial H_t(a)} \left ( \sum_x H_t(x)^2 q_*(x) \right ) = \sum_x 2 H_t(x) q_*(x)=0$$

Consider the case of only two actions:

$$2 H_t(1) q_*(1) + 2 H_t(2) q_*(2) = 0 \implies H_t(1) = -H_t(2)\frac{q_*(2)}{q_*(1)}.$$

As a probability though, we know that $(H_t(1)^2 + H_t(2)^2 = 1)$, but substituting the expression for $H_t(1)$ we get

$$H_t(2)^2 \frac{q_*(2)}{q_*(1)} + H_t(2)^2 = 1 \implies H_t(2) = \sqrt{\frac{1}{1 + r}}$$

where r is the ratio of q values. This is a problem though because the q values can be any real number including negative ones so the expression in the square root could be negative. So this form of the probability will not have solutions for certian q values given the contraints of probability.

We could also consider a function that guarantees that our probabilities meet both contraints:

$$\sum_x p(x) = 1 \quad \text{and} \quad 0 \leq p(x) \leq 1 \forall x$$

In order to do this let's take the H values, subtract the minimum, and by the sum of the resulting numbers. That would certainly meet the constraints except in the case where all the values are 0 which could be a problem. But even ignoring that fact, this procedure is not a differentiable function so we have no convenient way to maximize it.

For all these cases it is clear that the only suitable function for the probabilities as a function of H is as follows:

$$f(H_t(x)) = \frac{u(H_t(x))}{\sum_x u(H_t(x))}$$

where $u(a)$ is some function defined over the entire real number line such that $u(a) \geq 0 \forall a$. We already saw that $u(a) = a^2$ does not work because certain q values do not lead to valid solutions. Another option for such a function is $u(x) = e^x$ and this has the additional property of being monotonic. Using this form for $u(x)$ leads to the softmax.

$$p(x) = \frac{e^{H_t(x)}}{\sum_x e^{H_t(x)}}$$

Using this form for the probability also means if we add some arbitrary constant to all the H values, it has no effect on the probability.

$$\frac{e^{c + H_t(x)}}{\sum_x e^{c + H_t(x)}} = \frac{e^c e^{H_t(x)}}{e^c \sum_x e^{H_t(x)}} = p(x)$$

This property is very important considering the fact that we could arbitarily choose to center the H values around 0 or subtract the minimum to make them all positive. We should be able to eliminate the H value of 1 action by setting it to 0 and considering all the others in relation to that. Making such a choice should not change the answer of which action is the maximizing one. Also, it means if our maximization algorithm starts moving in the direction of a certain magnitude of H values, there will always be some solution that works if we fix any particular value so it will be much easier to find the maximum.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN™·°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$ecd8833d-441b-4673-b457-dc5109f575ab¹depends_on_disabled_cellsÂ§runtimeÎ)Kµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$d42a346d-6af3-45df-9de5-7043f46c3da3Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTiñ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$d42a346d-6af3-45df-9de5-7043f46c3da3¹depends_on_disabled_cellsÂ§runtimeÎ×õµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$c66f1676-aec1-489d-96ff-99d748dac0feŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTjà7°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$c66f1676-aec1-489d-96ff-99d748dac0fe¹depends_on_disabled_cellsÂ§runtimeÎõðµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$d2ebd908-387d-4e40-bc00-61ce5f45ebddŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTi•Ñ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$d2ebd908-387d-4e40-bc00-61ce5f45ebdd¹depends_on_disabled_cellsÂ§runtimeÎ€µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$97f6221d-3289-4e54-a80d-26c5c81f2651Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚJ

By default, a plot will be loaded that matches this search criteria. Check the box below to recalculate the search and save a new plot whever the submit button is clicked.

Recompute Parameter Search:

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšXviä°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$97f6221d-3289-4e54-a80d-26c5c81f2651¹depends_on_disabled_cellsÂ§runtimeÎ.—Õµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8018edb9-853f-4a88-bef6-9535c2d774b3Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTfk©°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8018edb9-853f-4a88-bef6-9535c2d774b3¹depends_on_disabled_cellsÂ§runtimeÎ‚ûµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$672a91c0-aa77-4257-8c83-d857f47cab6cŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ/sample_action (generic function with 3 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTl%$°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$672a91c0-aa77-4257-8c83-d857f47cab6c¹depends_on_disabled_cellsÂ§runtimeÎÌµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$a04da367-3f8d-422d-a443-4e3e666e30efŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšThÀ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$a04da367-3f8d-422d-a443-4e3e666e30ef¹depends_on_disabled_cellsÂ§runtimeÎí†µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8f1c7b0d-121c-46bf-884d-729e3b593025Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ-param_search (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšX&G„°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8f1c7b0d-121c-46bf-884d-729e3b593025¹depends_on_disabled_cellsÂ§runtimeÎ<ñAµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$87f5ec29-623f-4666-824d-fad8c6448072Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ`¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšRØý°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$87f5ec29-623f-4666-824d-fad8c6448072¹depends_on_disabled_cellsÂ§runtimeÎ`Jµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$30aa1e1b-0b51-40c4-a093-ef92c3ad519aŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTgÉÚ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$30aa1e1b-0b51-40c4-a093-ef92c3ad519a¹depends_on_disabled_cellsÂ§runtimeÎ<Ìµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8a0462e9-fd65-4ccc-b795-446cf9ae7392Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ:make_or_lookup_param_plot (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXs"p°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8a0462e9-fd65-4ccc-b795-446cf9ae7392¹depends_on_disabled_cellsÂ§runtimeÎ1—µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$fb698663-e9d6-4368-bd53-d88da5e6f2c4Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXD ÿ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$fb698663-e9d6-4368-bd53-d88da5e6f2c4¹depends_on_disabled_cellsÂ§runtimeÎv„µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$13f0adab-7660-49df-b26d-5f89cd73192bŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTj×°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$13f0adab-7660-49df-b26d-5f89cd73192b¹depends_on_disabled_cellsÂ§runtimeÎ/þÁµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$c6a8a9d2-1f29-4082-9fd2-d409cf6227baŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÛ ÑÎ ¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšY _Ì°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$c6a8a9d2-1f29-4082-9fd2-d409cf6227ba¹depends_on_disabled_cellsÂ§runtimeÎ>zäµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$eed2a4f2-48b2-4684-846e-aa99bb6dafd9Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ—

Exercise 2.5 (programming)

Design and conduct an experiment to demonstrate the difficulties that sample-average methods have for nonstationary problems. Use a modified version of the 10-armed testbed in which all the $q_*(a)$ start out equal and then take independent random walks (say by adding a normally distributed increment with mean 0 and standard deviation 0.01 to all the $q_*(a)$ on each step). Prepare plots like Figure 2.2 for an action-value method using sample averages, incrementally computed, and another action-value method using a constant step-size parameter, $\alpha = 0.1$. Use $\epsilon = 0.1$ and longer runs, say of 10,000 steps.

See code and figures below for answer

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNŸ^°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$eed2a4f2-48b2-4684-846e-aa99bb6dafd9¹depends_on_disabled_cellsÂ§runtimeÎÙ>µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8c0f06f7-2ed0-4f3a-ab4e-90ac142f0cd9Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTd+ˆ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8c0f06f7-2ed0-4f3a-ab4e-90ac142f0cd9¹depends_on_disabled_cellsÂ§runtimeÎ‹µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$d53796f1-6fc4-4450-9d66-05ba752ff7ebŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ/sample_action (generic function with 7 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTm¦ °persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$d53796f1-6fc4-4450-9d66-05ba752ff7eb¹depends_on_disabled_cellsÂ§runtimeÎ-%uµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$0f244fa0-7591-4478-b172-d9c1de51f6e1Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

One natural update rule for the action preferences is to use stochastic gradient ascent. Using this technique we perform the following update on step $t+1$ after selecting action $A_t$ and receiving reward $R_t$ on step $t$.

$$\begin{flalign} H_{t+1}(A_t) &\doteq H_t(A_t) + \alpha (R_t - \overline R_t)(1-\pi_t (A_t)) \\ H_{t+1}(a) & \doteq H_t(a) - \alpha(R_t - \overline R_t)\pi_t(a) \quad \forall a \neq A_t \end{flalign} \tag{2.12}$$

where $\alpha > 0$ is a step-size parameter and $\overline R_t \in \mathbf{R}$ is the average rewards up to but not including time $t.$ This average can be computed by any of the techniques mentioned earlier.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNç°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$0f244fa0-7591-4478-b172-d9c1de51f6e1¹depends_on_disabled_cellsÂ§runtimeÎ¨ªµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$262952a7-280e-4af1-99a6-0899518484a2Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTfì°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$262952a7-280e-4af1-99a6-0899518484a2¹depends_on_disabled_cellsÂ§runtimeÎw¼µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$46478110-5ce5-4c72-bb3d-bcb5f516ffdcŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ^

Extra Notes on the Soft-Max and Gradient Bandit

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNT°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$46478110-5ce5-4c72-bb3d-bcb5f516ffdc¹depends_on_disabled_cellsÂ§runtimeÎx¾µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$f1466027-7f52-41df-a8ea-a0650981c9d4Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ-plot_softmax (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšXÛ·™°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$f1466027-7f52-41df-a8ea-a0650981c9d4¹depends_on_disabled_cellsÂ§runtimeÎàµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$33009d31-6d66-4aba-b44c-edd911c2f392Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ4average_simple_runs (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšQøV°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$33009d31-6d66-4aba-b44c-edd911c2f392¹depends_on_disabled_cellsÂ§runtimeÎÓ‘ˆµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$31d8fba9-cd28-4b2b-ba46-70a068b9ecadŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ

2.2: Action-value Methods

The true value is the mean reward when that action is selected. One way to estimate this is by averaging the rewards actually received:

$$Q_t(a) \doteq \frac{\text{sum of rewards when } a \text{ taken prior to } t}{\text{number of times } a \text{ taken prior to } t} = \frac{\sum_{i=1}^{t-1}R_i \cdot \mathbf{1}_{A_i=a}}{\sum_{i=1}^{t-1} \mathbf{1}_{A_i=a}} \tag{2.1}$$

If the denominator is zero then we instead define $Q_t(a)$ by some default value such as 0. We call this the sample-average method for estimating action values because each estmiate is an average of the sample of relevant rewards.

The simplest action selection rule is to select one of the actions with the higest estimated value. This is the greedy action.

$$A_t \doteq \operatorname*{argmax}_a Q_t(a)$$

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN×V°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$31d8fba9-cd28-4b2b-ba46-70a068b9ecad¹depends_on_disabled_cellsÂ§runtimeÎ µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$f78846a8-076a-4a87-b1d0-4f830e2bca27Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚü ¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšY$>°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$f78846a8-076a-4a87-b1d0-4f830e2bca27¹depends_on_disabled_cellsÂ§runtimeÎ Øvµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$73e5b719-8b91-41d2-b83d-471d981b027fŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšThDx°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$73e5b719-8b91-41d2-b83d-471d981b027f¹depends_on_disabled_cellsÂ§runtimeÎ;Rµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$51b7a645-269a-418c-b6d8-39c01d0609f1Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚJO

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšX™Ôú°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$51b7a645-269a-418c-b6d8-39c01d0609f1¹depends_on_disabled_cellsÂ§runtimeÍh¦µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$23b99305-c8d9-4129-85fb-a5e4aabc4a31Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ£

2.7 Upper-Confidence-Bound Action Selection

We can choose to explore non-greedy actions based on the probability that they are better than optimal. This probability always exists due to the uncertainty inherent in our value estimates. One way to implement this concept is to select actions according to:

$$A_t \doteq \operatorname*{argmax}_a \left [ Q_t(a) + c \sqrt{\frac{\ln{t}}{N_t(a)}}\right ] \tag{2.10}$$

where $N_t(a)$ is the number of times that action $a$ has been selected prior to time t and $c>0$ controls the degree of exploration. This idea is called upper confidence bound (UCB) action selection. Note that is $N_t(a)=0$ then that action will be selected or a random selection will be made among all actions with zero counts. See below for a comparison between the Ïµ-greedy exploration method and the UCB method with the ability to change parameters for both methods.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN–l°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$23b99305-c8d9-4129-85fb-a5e4aabc4a31¹depends_on_disabled_cellsÂ§runtimeÎxµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$4a89cdd9-c20f-40e2-bc84-c3ea9cbf00e7Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙÐ

Below are functions which mimic those for the stationary bandit parameter search. The final function produces a plot and saves it to disk or loads one that already exists.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN´°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$4a89cdd9-c20f-40e2-bc84-c3ea9cbf00e7¹depends_on_disabled_cellsÂ§runtimeÎb´µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$3a215c95-c595-4837-a842-1c1e1c6bfa3bŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTh•û°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$3a215c95-c595-4837-a842-1c1e1c6bfa3b¹depends_on_disabled_cellsÂ§runtimeÎÖeµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ+figure_2_5 (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšT„¥'°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9¹depends_on_disabled_cellsÂ§runtimeÎlå]µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$55f89ca1-cd57-44e0-95e5-63a4be31418fŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ.sample_bandit (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšQäŒ;°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$55f89ca1-cd57-44e0-95e5-63a4be31418f¹depends_on_disabled_cellsÂ§runtimeÎþSµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$d59126d7-5af0-4d06-a57b-e115eec32388Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙK

Non-stationary Parameter Study Figure

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN7L°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$d59126d7-5af0-4d06-a57b-e115eec32388¹depends_on_disabled_cellsÂ§runtimeÎÝrµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTd¹o°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65¹depends_on_disabled_cellsÂ§runtimeÎ›µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$f88029d6-3fc2-4552-8441-5ef37ac42638Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚâ

Exercise 2.8: UCB Spikes

In Figure 2.4 the UCB algorithm shows a distinct spike in performance on the 11th step. Why is this? Note that for your answer to be fully satisfactory it must explain both why the reward increases on the 11th step and why it decreases on the subsequent steps. Hint: If $c=1,$ then the spike is less prominent.

By definition, actions with zero visits are always considered maximizing. Therefore, for the first 10 steps, all 10 unique actions will be sampled once with each Q estimate updating from a single sample. On the 11th step, the exploration incentive for each action will be equal, so the action with the highest Q estimate will be selected. This is most likely to be the action with the highest $q^*$ value but there is a substantial probability it is the second best action and diminishing probabilities for the remaining actions ranked by true $q^*$. It is on this step though that we expect the selection to be substantially better than random chance although it is only using a single sample to validate the estimates. On step 12, that improved action will now have a visit count of 2 instead of 1 for every other action. In the calculation, the exploration bonus for that action will be $c\sqrt{\frac{\ln{12}}{2}}\approx 1.11465 \times c.$ Every other action will have an exploration bonus of $\approx 1.576 \times c.$ In order for the 2 visit action to be considered maximizing after this it must have a Q estimate that is $\approx 0.4617 \times c$ greater than any other action value estimate. In particular for $c = 2.0,$ the estimate must be $\approx 0.9234$ greater than the others. Since the q's are normally distributed, the difference in expected value between the best and second best action is only about 0.55. As a rough heuristic for the probability of the action selection remaining unchanged, we can consider the probability that the 2 sample best action estimate exceeds the single sample second best action estimate by 0.9234 with the following calculation: $1 - \operatorname*{cdf}(\operatorname*{Normal}(\mu = 1.54 - 1, \sigma = \sqrt{\frac{3}{2}}), x = 0.9234) \approx 0.377.$ Therefore, on step 12, the average run will change the action selection to something less optimal. The larger the value of c, the more likely the selection is to change due to the larger weight placed on exploration. Empirically, as c approaches $\infty$ the expected reward on step 12 approaches 0.744 vs 1.145 on step 11. That compares to the expected q value for the top 3 actions of approximately 0.656, 1, and 1.54.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN¶1°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$f88029d6-3fc2-4552-8441-5ef37ac42638¹depends_on_disabled_cellsÂ§runtimeÎ[³µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$9e644919-5833-46ff-8f6e-e755ba688329Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚw

One reason this formula is called the softmax is because it preserves the ordering of the original values including the maximum, and if a multiplicative parameter (usually called Î²) is applied to each value in the exponent, it can be made arbitrarily close to returning 1 for the maximum and zero for all other values. If that parameter is 0, then the function always returns the uniform distribution, so in this way the output can smoothly vary from completely uniform to the unique maximum. Below is a plot if the softmax of a given number of items where the Î² parameter can vary.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNµ-°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$9e644919-5833-46ff-8f6e-e755ba688329¹depends_on_disabled_cellsÂ§runtimeÎi«µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$3a798da5-c309-48f2-aab1-6602ded8a650Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ1simple_algorithm (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšQíe°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$3a798da5-c309-48f2-aab1-6602ded8a650¹depends_on_disabled_cellsÂ§runtimeÎsAµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$7950e06b-e8ce-4bd7-9681-ab7b66dfec69Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ4plot_bandit_testbed (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšPµy³°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$7950e06b-e8ce-4bd7-9681-ab7b66dfec69¹depends_on_disabled_cellsÂ§runtimeÎ=OŸµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$c9c2aa04-15a2-446a-839c-bcf724b0f57aŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙº ¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNâ&°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$c9c2aa04-15a2-446a-839c-bcf724b0f57a¹depends_on_disabled_cellsÂ§runtimeÎ°µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$d9265b98-cc3e-4a60-b16e-f54d9f78c9d3Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTkÐS°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$d9265b98-cc3e-4a60-b16e-f54d9f78c9d3¹depends_on_disabled_cellsÂ§runtimeÎ :µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$4afd6c2c-632e-4196-ab36-4d314ed9ec96Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚË

For $\alpha = 0.1$ this value is higher than for the sample average case from the second step onwards. Also it asymptotes to a finite value whereas the sample average variance trends towards zero. This asymptote can be calculated by inspecting the equation taking the limit for large N. If $\alpha < 1$ we have $\frac{\alpha}{2 - \alpha}$. For $\alpha = 0.1$ this is about 0.0526. If we draw an analogy to the sample average case, this would be equivalent to the variance after collecting 9.0 samples reflecting the fact that over time we are maintaining information from roughly a constant number of samples. The "optimal distribution" line in the non-stationary parameter study" figure shows the performance of this policy for different values of $\alpha$.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšY$ö°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$4afd6c2c-632e-4196-ab36-4d314ed9ec96¹depends_on_disabled_cellsÂ§runtimeÎ€÷nµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$d4ce45ae-613e-41ee-b626-69b0dbcf6452Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ …

Exercise 2.6: Mysterious Spikes

The results shown in Figure 2.3 should be quite reliable because they are averages over 2000 individual, randomly chosen 10-armed bandit tasks. Why, then, are there oscillations and spikes in the early part of the curve for the optimistic method? In other words, what might make this method perform particularly better or worse, on average, on particular early steps?

The spike occurs on step 11. Due to the initial Q values it is almost 100% likely that a given run will sample each of the 10 possible actions once before repeating any. For this not to be the case, one of the samples would have to exceed the initial value of 5.0 which has a probability near zero since the expected q value for the best arm is around 1.539 which unit variance. That would mean that at any given step only 10% of the runs would select the optimal action and indeed for the first 10 steps about 10% of the runs are selecting the optimal action as we'd expect from random chance.

On the 11th step, the Q value estimate for each action is $(0.9 \times 5) + (0.1 \times action\_reward)$. The optimal action will be selected on this step as long as the reward produced by the best action exceeded all the others. Empirically, that probability is ~44% which is similar to the probability calculated for the expected value of the best action of ~1.539 exceeding the rewards from the other 9 arms. For those 44% of the runs that do select the optimal action, they will obtain a reward with expected value 1.539. If they received that reward during both samples, then the Q value estimate will be $0.9 \times ((0.9 \times 5) + (0.1 \times 1.539)) + (0.1 \times 1.539) \approx 4.34$. Let's consider the second best arm which has an expected q value of ~1. The updated estimate for that arm after receiving a reward equal to the expected value is $0.9 \times 0.5 + 0.1 \times 1 \approx 4.6$. Following the same reasoning for the third best arm, the value is about 4.57. In fact even a reward of zero will produce an estimate of $0.9 \times 5 = 4.5$ which still exceeds the estimate for the optimal action in our scenario. That explains why the percentage of optimal actions drops in the 12th step because it is expected that the estimate of the action selected on step 11 will drop below at least one of the other arms, thus changing the maximizing action selection to a worse one.

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšN°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$d4ce45ae-613e-41ee-b626-69b0dbcf6452¹depends_on_disabled_cellsÂ§runtimeÎý&µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$33d201ba-52f2-44f0-8bc6-3930ec77f62fŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚ#I

Softmax Visualization

Number of Items: Î² Parameter: 1.0

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšXÕ!q°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$33d201ba-52f2-44f0-8bc6-3930ec77f62f¹depends_on_disabled_cellsÂ§runtimeÎâRRµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$0de99ee5-d94d-4d07-8cef-a6f9caf5e742Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚä" ¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšT3îÙ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$0de99ee5-d94d-4d07-8cef-a6f9caf5e742¹depends_on_disabled_cellsÂ§runtimeÎ#éEÛµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$a61c15eb-ed5f-4052-a3a3-3276940564a1Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTdV°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$a61c15eb-ed5f-4052-a3a3-3276940564a1¹depends_on_disabled_cellsÂ§runtimeÎ~¾µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$99945570-0b2a-432a-ae83-3a7abc39cac8Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ1normalize_Ï€vec! (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšT:`€°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$99945570-0b2a-432a-ae83-3a7abc39cac8¹depends_on_disabled_cellsÂ§runtimeÎßµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ/sample_action (generic function with 5 methods)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTl¥D°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18¹depends_on_disabled_cellsÂ§runtimeÎ9Rµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$2ceadc6d-5522-43f6-a803-a17a47a6048aŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÚš

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšXÄ=›°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$2ceadc6d-5522-43f6-a803-a17a47a6048a¹depends_on_disabled_cellsÂ§runtimeÍ`µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$9093ed39-f6e2-4c31-9fef-466bdd197423Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTf¢Â°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$9093ed39-f6e2-4c31-9fef-466bdd197423¹depends_on_disabled_cellsÂ§runtimeÎ ©Üµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ@nonstationary_bandit_statistics (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšSY4Ò°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2¹depends_on_disabled_cellsÂ§runtimeÎ‡Oµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ.update_Ï€vec! (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšT@´S°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5¹depends_on_disabled_cellsÂ§runtimeÎøÈµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$6292f449-8720-41f1-84de-1865fb5fddbfŠ¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ+figure_2_3 (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšSÌ¡Ó°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$6292f449-8720-41f1-84de-1865fb5fddbf¹depends_on_disabled_cellsÂ§runtimeÎ.øµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$0d8e4160-adf2-4b43-9914-942539339972Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÚu

Revisiting the Gradient Bandit

Consider what is the distribution $\pi_t(x)$ that maximizes $\mathbb{E}[R_t]=\sum_x \pi_t(x) q_*(x)$ using the softmax representation of the distribution. If we revisit the derivation in section 2.8, and keep the $q_*$ values instead of samples, we have

$$\frac{\partial \mathbb{E}[R_t]}{\partial H_t(a)} = \sum_x \left [ (q_*(x) - B_t) \left ( \mathbb{1}_{a = x} - \pi_t(a) \right ) \right ] = \sum_x \left [ (q_*(x) - B_t) \left ( \mathbb{1}_{a = x} - \frac{e^{H_t(a)}}{\sum_{b=1}^k e^{H_t(b)}} \right ) \right ]$$

At the maximum this must be 0 for all actions:

$$\sum_x \left [ (q_*(x) - B_t) \left ( \mathbb{1}_{a = x} - \frac{e^{H_t(a)}}{\sum_{b=1}^k e^{H_t(b)}} \right ) \right ] = 0$$

Consider the action that is the maximum. One solution that would cause that partial derivative to be 0 is if $\pi_t(a_{max}) = 1$ and all others are 0. To see this the term multiplying the q values would be $(0-0$ for $a \neq a_{max}$ and $(1-1)$ for $a = a_{max}$. But in this case the other terms are not zero. What are they? Consider an action $a \neq a_{max}$. The sum in this case is $q_a - q_{a_{max}}$

Since we know $q_*$, $B_t$ might as well be set to $\sum_x q_*(x) / [x]$

¤mime©text/html¬rootassigneeÀ²last_run_timestampËAÚšNäÁ°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$0d8e4160-adf2-4b43-9914-942539339972¹depends_on_disabled_cellsÂ§runtimeÎlÞµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$09227386-1620-4093-b913-5786205aad13Š¦queuedÂ¤logs§runningÂ¦output†¤bodyÙ2update_estimator! (generic function with 1 method)¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšUFÆ~°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$09227386-1620-4093-b913-5786205aad13¹depends_on_disabled_cellsÂ§runtimeÎî†µpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$f995d0af-50bc-4e33-9bbf-17a7ab06358aŠ¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTbã&°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$f995d0af-50bc-4e33-9bbf-17a7ab06358a¹depends_on_disabled_cellsÂ§runtimeÎ>Dµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂÙ$fbf7b108-dc68-4077-b63c-6f88161d2098Š¦queuedÂ¤logs§runningÂ¦output†¤body ¤mimeªtext/plain¬rootassigneeÀ²last_run_timestampËAÚšTfBW°persist_js_stateÂ·has_pluto_hook_featuresÂ§cell_idÙ$fbf7b108-dc68-4077-b63c-6f88161d2098¹depends_on_disabled_cellsÂ§runtimeÎ©Èµpublished_object_keys¸depends_on_skipped_cellsÂ§erroredÂ±cell_dependenciesÞ¸Ù$781fa565-398c-4c89-89f3-0595455afc85„´precedence_heuristic §cell_idÙ$781fa565-398c-4c89-89f3-0595455afc85´downstream_cells_map±update_estimator!‘Ù$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_mapŠ«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0fªupdatecoef”Ù$62eb0650-96bc-4fd6-bfe0-bf05a4137a03Ù$30f05bd9-e939-4810-b710-edc7f5975921Ù$96566aad-6d5c-460c-a924-ae0bad5d8b2dÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0ad¡-AverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1¡+¡*£one§IntegerAbstractFloatÙ$37446874-1c28-491b-b3cc-b4ad3282686e„´precedence_heuristic §cell_idÙ$37446874-1c28-491b-b3cc-b4ad3282686e´downstream_cells_map€²upstream_cells_mapƒ§@md_str¹nonstationarysearchparams‘Ù$3b88ec30-768b-44d0-88ee-b3ed989f22c3¨getindexÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872„´precedence_heuristic §cell_idÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872´downstream_cells_map·average_runs_cum_reward’Ù$74024d96-d0c7-43c8-8379-caf843cbe4b8Ù$22fa2b71-a98f-4b87-9e0b-9d373cd8915f²upstream_cells_mapÞ!¤zero®Base.Threads.*¤sqrt¦Random‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100»Base.Threads.threadpoolsize®Base.Threads.-«atomic_add!®Base.Threads.+create_bandit‘Ù$97c94391-397d-4cf2-88b1-3bea3af56ed3²Base.Threads.error®Base.Threads.>¡/¦Atomic¡^¯Base.Threads.!=£abs¨@threadsAbstractFloat¡:ºBase.Threads.threading_run¨Function³Base.Threads.divrem¤Base·Base.Threads.firstindex¡-ªrun_bandit‘Ù$1004eb4b-1fed-4328-a08b-6f5d9dd5080b¡*³Base.Threads.length®Base.Threads.:¯Base.Threads.<=¯Base.Threads.==¥ccall¬Random.seed!Ù$8fac4109-2e0d-4366-9118-018221e0b910„´precedence_heuristic §cell_idÙ$8fac4109-2e0d-4366-9118-018221e0b910´downstream_cells_mapªfigure_2_4‘Ù$0de99ee5-d94d-4d07-8cef-a6f9caf5e742²upstream_cells_map‰¡:§scatter¦Random‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100³average_simple_runs‘Ù$33009d31-6d66-4aba-b44c-edd911c2f392¤attr«PlutoPlotly‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100°PlutoPlotly.plot¦Layout¬Random.seed!Ù$98f4d5e6-7569-457e-851d-713c572ae400„´precedence_heuristic §cell_idÙ$98f4d5e6-7569-457e-851d-713c572ae400´downstream_cells_map¿plot_nonstationary_param_search‘Ù$9bd99099-1dfa-477a-9896-3da94bcc0633²upstream_cells_mapÞ¸LaTeXStrings.latexstring¤vcat¦colors‘Ù$fb698663-e9d6-4368-bd53-d88da5e6f2c4¹OptimalDistributionSample‘Ù$6d647790-a029-4579-86fa-9c7294aa94ea©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358a¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7£UCB‘Ù$60b2079e-0efa-427e-93cf-7f4646fe202e«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b©eachindex§scatter§Float32¡/¬LaTeXStrings‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¦unique¦@L_str©maketrace‘Ù$7c2015dd-a786-49f5-9fe9-9199335ebd09¤Plot¦reduce®GradientReward‘Ù$8a04adab-e97e-4ac4-a85e-5eae93b1c37b¥Int64¥floor¬preparetrace‘Ù$400e0de1-0101-4531-928f-08ca155da40c¬print_power2‘Ù$c61630b0-29c1-4183-90d9-57c999187b53¤sort¤attr¦Layoutºnonstationary_param_search‘Ù$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2Ù$4be9e81b-c3de-4c79-97b5-b41c03e0f187„´precedence_heuristic §cell_idÙ$4be9e81b-c3de-4c79-97b5-b41c03e0f187´downstream_cells_map€²upstream_cells_map‚µvisualize_bandit_dist‘Ù$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfaªn_arms_vis‘Ù$2bce1b80-2133-40a0-9367-fc2d491f6245Ù$78c45162-dc8e-4faf-b1a9-8c71be86dcee„´precedence_heuristic §cell_idÙ$78c45162-dc8e-4faf-b1a9-8c71be86dcee´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$30f05bd9-e939-4810-b710-edc7f5975921„´precedence_heuristic §cell_idÙ$30f05bd9-e939-4810-b710-edc7f5975921´downstream_cells_mapªupdatecoef’Ù$09227386-1620-4093-b913-5786205aad13Ù$781fa565-398c-4c89-89f3-0595455afc85²upstream_cells_map‡®GradientReward‘Ù$8a04adab-e97e-4ac4-a85e-5eae93b1c37b®GradientSample‘Ù$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65SampleAverage‘Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9¡/£one§IntegerAbstractFloatÙ$2bce1b80-2133-40a0-9367-fc2d491f6245„´precedence_heuristic §cell_idÙ$2bce1b80-2133-40a0-9367-fc2d491f6245´downstream_cells_mapªn_arms_vis‘Ù$4be9e81b-c3de-4c79-97b5-b41c03e0f187²upstream_cells_map‹§@md_str¤Core¡:¨Base.get¥@bind¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField¯Core.applicable¨getindexÙ$a15e5d05-a238-440b-9a43-d830c6ea2f4d„´precedence_heuristic §cell_idÙ$a15e5d05-a238-440b-9a43-d830c6ea2f4d´downstream_cells_map¬param_search’Ù$22fa2b71-a98f-4b87-9e0b-9d373cd8915fÙ$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2²upstream_cells_mapÞ¤vcat¡>¦islessÙ'Base.CoreLogging.Base.fixup_stdlib_path²Base.CoreLogging.!¡<¡/¡^¥@info±Base.invokelatest¢==¥roundAbstractFloat½Base.CoreLogging.invokelatest´Base.CoreLogging.===º#___this_pluto_module_name¤Base¥Int64´Base.CoreLogging.isa¢>=¡+³Base.CoreLogging.>=Ù$0e06d9cf-b995-40b1-86ee-1c6a56c63bd0„´precedence_heuristic §cell_idÙ$0e06d9cf-b995-40b1-86ee-1c6a56c63bd0´downstream_cells_map€²upstream_cells_map‚¤Type¹OptimalDistributionSample‘Ù$6d647790-a029-4579-86fa-9c7294aa94eaÙ$1cde4625-f8ed-4403-835c-95cc30206699„´precedence_heuristic §cell_idÙ$1cde4625-f8ed-4403-835c-95cc30206699´downstream_cells_map¨datapath’Ù$7c562867-55d3-4b4f-950d-c8efc4a9ff32Ù$8a0462e9-fd65-4ccc-b795-446cf9ae7392²upstream_cells_map‚¨joinpath¨@__DIR__Ù$6a00226c-cda1-49d8-92a8-514f2264469c„´precedence_heuristic §cell_idÙ$6a00226c-cda1-49d8-92a8-514f2264469c´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$c61630b0-29c1-4183-90d9-57c999187b53„´precedence_heuristic §cell_idÙ$c61630b0-29c1-4183-90d9-57c999187b53´downstream_cells_map¬print_power2’Ù$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_map‡¡<¡-¡>¡^¨latexify¦isless£absÙ$0e606680-dd65-444f-bc98-73de4abbcdd4„´precedence_heuristic §cell_idÙ$0e606680-dd65-444f-bc98-73de4abbcdd4´downstream_cells_map€²upstream_cells_map‚SampleAverage‘Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9¤TypeÙ$c695b7f9-76ca-419b-924d-8338a42c8188„´precedence_heuristic §cell_idÙ$c695b7f9-76ca-419b-924d-8338a42c8188´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$865610bb-ee82-4440-9f32-f00d0382783b„´precedence_heuristic §cell_idÙ$865610bb-ee82-4440-9f32-f00d0382783b´downstream_cells_map«run_or_load²upstream_cells_map‡§jldopen¨Function¡!§jldsave¦String¦isfile¤readÙ$50fbdc85-82f1-4c52-936b-84eb14951d71„´precedence_heuristic §cell_idÙ$50fbdc85-82f1-4c52-936b-84eb14951d71´downstream_cells_mapÙ average_gradient_stationary_runs‘Ù$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9²upstream_cells_mapÞ®Base.Threads.*»Base.Threads.threadpoolsize®Base.Threads.-¦Vector®Base.Threads.+©eachindexcreate_bandit‘Ù$97c94391-397d-4cf2-88b1-3bea3af56ed3²Base.Threads.error®Base.Threads.>¡/§Float32¯Base.Threads.!=¨@threads¡:ºBase.Threads.threading_runÙ$gradient_stationary_bandit_algorithm‘Ù$f7519adc-7dfb-4030-86f0-7445699dd3db³Base.Threads.divrem¤Base·Base.Threads.firstindex¡+¥undef®Base.Threads.:³Base.Threads.length¯Base.Threads.==¥ccall¯Base.Threads.<=Ù$2447c4ea-7752-457c-80da-ac0dd72a64c1„´precedence_heuristic §cell_idÙ$2447c4ea-7752-457c-80da-ac0dd72a64c1´downstream_cells_map©save_data²upstream_cells_map§jldsaveÙ$b5a2df21-4525-4320-b8dd-aea5ecdab832„´precedence_heuristic §cell_idÙ$b5a2df21-4525-4320-b8dd-aea5ecdab832´downstream_cells_mapªparams_2_5‘Ù$691aa77a-d6da-4fde-9024-c4195057179d²upstream_cells_mapŽ§@md_str¤Core¡:§PlutoUI‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¨Base.get¥@bind¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField§confirm¯Core.applicable¯PlutoUI.combine¨getindexÙ$c12dab6a-92f0-41b1-a6b7-c404c74d9a83„´precedence_heuristic §cell_idÙ$c12dab6a-92f0-41b1-a6b7-c404c74d9a83´downstream_cells_map¹action_value_testbed_plot‘Ù$79082409-3182-4e0b-9c8c-37a94543fee9²upstream_cells_map‹§@md_strcreate_bandit‘Ù$97c94391-397d-4cf2-88b1-3bea3af56ed3£map§scatter©enumerate³average_simple_runs‘Ù$33009d31-6d66-4aba-b44c-edd911c2f392¤attr«PlutoPlotly‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100°PlutoPlotly.plot¦Layout¨getindexÙ$36602c38-8b29-4158-b299-94015a333762„´precedence_heuristic §cell_idÙ$36602c38-8b29-4158-b299-94015a333762´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2„´precedence_heuristic §cell_idÙ$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2´downstream_cells_mapºnonstationary_param_search‘Ù$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_map‚¬param_search’Ù$8f1c7b0d-121c-46bf-884d-729e3b593025Ù$a15e5d05-a238-440b-9a43-d830c6ea2f4dÙ%average_nonstationary_runs_cum_reward‘Ù$74024d96-d0c7-43c8-8379-caf843cbe4b8Ù$68470b1d-3cc2-4cb1-8dc2-53227e6300e7„´precedence_heuristic §cell_idÙ$68470b1d-3cc2-4cb1-8dc2-53227e6300e7´downstream_cells_map€²upstream_cells_map„¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0f®GradientSample‘Ù$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65¤TypeAbstractFloatÙ$86350532-13f8-4035-bad8-f25f41c93163„´precedence_heuristic §cell_idÙ$86350532-13f8-4035-bad8-f25f41c93163´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$88e43fed-fcf3-4071-996a-63f63c3d49b4„´precedence_heuristic §cell_idÙ$88e43fed-fcf3-4071-996a-63f63c3d49b4´downstream_cells_map³stationary_numsteps‘Ù$51b7a645-269a-418c-b6d8-39c01d0609f1²upstream_cells_mapŒ§@md_str¤Core¡:¨Base.get¥@bind¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField§confirm¯Core.applicable¨getindexÙ$f00ab44e-0d84-40c7-aa23-358c77a013e3„´precedence_heuristic §cell_idÙ$f00ab44e-0d84-40c7-aa23-358c77a013e3´downstream_cells_map´UnbiasedConstantStep•Ù$0f6b4e2d-dc09-4e1c-834f-dd8aaa8743aeÙ$7a31d1c5-260b-42f4-b997-967150881e21Ù$5712b303-0aa3-4501-b1b5-020136d6e655Ù$d2ebd908-387d-4e40-bc00-61ce5f45ebddÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0ad²upstream_cells_map„¤zero£newAverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1AbstractFloatÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0ad„´precedence_heuristic §cell_idÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0ad´downstream_cells_mapªupdatecoef’Ù$09227386-1620-4093-b913-5786205aad13Ù$781fa565-398c-4c89-89f3-0595455afc85²upstream_cells_mapŒ¡T£one¡E§Integer¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0f¡-¡/´UnbiasedConstantStep‘Ù$f00ab44e-0d84-40c7-aa23-358c77a013e3¡+¡*¯BanditAlgorithm‘Ù$1e1f6d10-1b31-4e0a-96f7-23207e913154AbstractFloatÙ$46ce2b1f-02cf-4dae-bf02-f67543f38b91„´precedence_heuristic §cell_idÙ$46ce2b1f-02cf-4dae-bf02-f67543f38b91´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$211d9390-df44-4902-9e13-ad6744b9b7dd„´precedence_heuristic §cell_idÙ$211d9390-df44-4902-9e13-ad6744b9b7dd´downstream_cells_map¯getdistribution²upstream_cells_mapÞ¦argmax¡:£sum¡!¥zeros¤ones£one¤rand¦Vector¦length§findall¥Int64©eachindex¡/¡+§isempty¡*¢==AbstractFloat¦NormalÙ$6d647790-a029-4579-86fa-9c7294aa94ea„´precedence_heuristic §cell_idÙ$6d647790-a029-4579-86fa-9c7294aa94ea´downstream_cells_map¹OptimalDistributionSample–Ù$0e06d9cf-b995-40b1-86ee-1c6a56c63bd0Ù$d42a346d-6af3-45df-9de5-7043f46c3da3Ù$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0baÙ$d53796f1-6fc4-4450-9d66-05ba752ff7ebÙ$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_map‚¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0fAbstractFloatÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03„´precedence_heuristic §cell_idÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03´downstream_cells_mapªupdatecoef’Ù$09227386-1620-4093-b913-5786205aad13Ù$781fa565-398c-4c89-89f3-0595455afc85²upstream_cells_map‡«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0fSampleAverage‘Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9¡/£one§IntegerAbstractFloatÙ$51349e41-4696-4bd5-9bc1-cefbb82bea08„´precedence_heuristic §cell_idÙ$51349e41-4696-4bd5-9bc1-cefbb82bea08´downstream_cells_mapsample_action’Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_map‚¨pweights¦sampleÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4„´precedence_heuristic §cell_idÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4´downstream_cells_map€²upstream_cells_mapˆ«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0fSampleAverage‘Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9AverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1¤Type©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358a§IntegerAbstractFloatÙ$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0ba„´precedence_heuristic §cell_idÙ$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0ba´downstream_cells_mapsample_action’Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_mapÞSampleAverage‘Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9®AbstractVector¡!¡>¦isless£Inf¹OptimalDistributionSample‘Ù$6d647790-a029-4579-86fa-9c7294aa94ea£one¤rand§Integer«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b§findall¡<©eachindex¡-¡/§isempty¢==AbstractFloat¦NormalÙ$7678b06b-feef-4656-a801-33f630437bfb„´precedence_heuristic §cell_idÙ$7678b06b-feef-4656-a801-33f630437bfb´downstream_cells_map¬average_runs‘Ù$181d7eef-24a0-4775-a535-8ef901b7e4eb²upstream_cells_mapŠ¡:create_bandit‘Ù$97c94391-397d-4cf2-88b1-3bea3af56ed3©eachindex§Float32¡/ªrun_bandit‘Ù$1004eb4b-1fed-4328-a08b-6f5d9dd5080b¡+¥undef¨Function¦VectorÙ$3b88ec30-768b-44d0-88ee-b3ed989f22c3„´precedence_heuristic §cell_idÙ$3b88ec30-768b-44d0-88ee-b3ed989f22c3´downstream_cells_map¹nonstationarysearchparams’Ù$2ceadc6d-5522-43f6-a803-a17a47a6048aÙ$37446874-1c28-491b-b3cc-b4ad3282686e²upstream_cells_map§@md_str¤Core¡:§PlutoUI‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¨Base.get¥@bind¤Base«PlutoRunner¨CheckBox·PlutoRunner.create_bond«NumberField§confirm¯Core.applicable¯PlutoUI.combine¨getindexÙ$691aa77a-d6da-4fde-9024-c4195057179d„´precedence_heuristic §cell_idÙ$691aa77a-d6da-4fde-9024-c4195057179d´downstream_cells_map€²upstream_cells_map‚ªparams_2_5‘Ù$b5a2df21-4525-4320-b8dd-aea5ecdab832ªfigure_2_5‘Ù$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9Ù$276779a3-9332-46bd-b511-a33a2fea4b5f„´precedence_heuristic §cell_idÙ$276779a3-9332-46bd-b511-a33a2fea4b5f´downstream_cells_map³nonstationaryparams‘Ù$d24bd737-9e09-441a-aa94-9279c80f566d²upstream_cells_mapŽ§@md_str¤Core¡:§PlutoUI‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¨Base.get¥@bind¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField§confirm¯Core.applicable¯PlutoUI.combine¨getindexÙ$b26b3519-6ab4-4cc9-971d-de5e591cac86„´precedence_heuristic §cell_idÙ$b26b3519-6ab4-4cc9-971d-de5e591cac86´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$d0111453-9a66-411d-9966-fc386d1bdcb7„´precedence_heuristic §cell_idÙ$d0111453-9a66-411d-9966-fc386d1bdcb7´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$79082409-3182-4e0b-9c8c-37a94543fee9„´precedence_heuristic §cell_idÙ$79082409-3182-4e0b-9c8c-37a94543fee9´downstream_cells_map€²upstream_cells_map‚¶Ïµ_action_value_params‘Ù$6034cc0f-cbae-4d2f-a43f-1bb738c00f0b¹action_value_testbed_plot‘Ù$c12dab6a-92f0-41b1-a6b7-c404c74d9a83Ù$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802„´precedence_heuristic §cell_idÙ$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802´downstream_cells_map€²upstream_cells_map„§@md_str¥ktest‘Ù$3e5a226e-ecdb-43fb-a40a-a262da0ae542³plot_bandit_testbed‘Ù$7950e06b-e8ce-4bd7-9681-ab7b66dfec69¨getindexÙ$9b625fc0-89bd-4064-a379-225e6a940af7„´precedence_heuristic §cell_idÙ$9b625fc0-89bd-4064-a379-225e6a940af7´downstream_cells_map³ktest_nonstationary’Ù$8584ece7-badc-486a-9a57-b60e77f92673Ù$f9e60b35-84d2-4b4b-8832-6b0f08152396²upstream_cells_map‹§@md_str¤Core¡:¨Base.get¥@bind¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField¯Core.applicable¨getindexÙ$1f9a98fd-ea29-415c-9f35-add34b513a34„´precedence_heuristic §cell_idÙ$1f9a98fd-ea29-415c-9f35-add34b513a34´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5„´precedence_heuristic §cell_idÙ$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5´downstream_cells_map·nonstationary_algorithm‘Ù$647ab36b-641e-4024-ad2d-40ff33be28f4²upstream_cells_mapÞ¥randn¦argmax¡:§collect¥zeros¦isless¤ones¤rand§Integer¡<¡-¡/¡+¡*sample_bandit‘Ù$55f89ca1-cd57-44e0-95e5-63a4be31418f¢==¨shuffle!AbstractFloat¤fillÙ$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53„´precedence_heuristic §cell_idÙ$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53´downstream_cells_map§testvar‘Ù$c6a8a9d2-1f29-4082-9fd2-d409cf6227ba²upstream_cells_map§@md_str¤Core¡:§PlutoUI‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¨Base.get¥@bind¦Slider¤Base«PlutoRunner·PlutoRunner.create_bond¯Core.applicable¯PlutoUI.combine¨getindexÙ$649e3d20-e276-4f4b-aeb0-89150f180ef5„´precedence_heuristic §cell_idÙ$649e3d20-e276-4f4b-aeb0-89150f180ef5´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$32bff269-e893-4907-b589-7ba2ae1314bd„´precedence_heuristic §cell_idÙ$32bff269-e893-4907-b589-7ba2ae1314bd´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$7c562867-55d3-4b4f-950d-c8efc4a9ff32„´precedence_heuristic §cell_idÙ$7c562867-55d3-4b4f-950d-c8efc4a9ff32´downstream_cells_map©loadplots‘Ù$6ce00349-1cf0-4a80-bddd-c1d26b66d051²upstream_cells_map¥split¤Dict¨joinpath¦@r_str¥first¢|>¦filter¦String¢=>§readdir¨occursin¤HTML¨datapath‘Ù$1cde4625-f8ed-4403-835c-95cc30206699§isempty¤readÙ$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8e„´precedence_heuristic §cell_idÙ$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8e´downstream_cells_map€²upstream_cells_mapƒ¬plot_softmax‘Ù$f1466027-7f52-41df-a8ea-a0650981c9d4©softmax_k‘Ù$33d201ba-52f2-44f0-8bc6-3930ec77f62f¢Î²‘Ù$33d201ba-52f2-44f0-8bc6-3930ec77f62fÙ$60b2079e-0efa-427e-93cf-7f4646fe202e„´precedence_heuristic §cell_idÙ$60b2079e-0efa-427e-93cf-7f4646fe202e´downstream_cells_map£UCB™Ù$3118e102-aeac-42d9-98fc-ca29f40be4cdÙ$8c0f06f7-2ed0-4f3a-ab4e-90ac142f0cd9Ù$a61c15eb-ed5f-4052-a3a3-3276940564a1Ù$47be3ae6-20f7-47d0-aae3-b67154afc1a8Ù$3a215c95-c595-4837-a842-1c1e1c6bfa3bÙ$a04da367-3f8d-422d-a443-4e3e666e30efÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18Ù$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_map‚¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0fAbstractFloatÙ$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfa„´precedence_heuristic §cell_idÙ$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfa´downstream_cells_mapµvisualize_bandit_dist‘Ù$4be9e81b-c3de-4c79-97b5-b41c03e0f187²upstream_cells_mapÞ§@md_str¥randn£sum¨LinRange£bar§scatter¡/¥round¨getindex§extrema¤fmax‘Ù$85b3dbc4-2e02-4924-b3e5-887a1a557a2b¡:¢|>©mapreduce¡-¤plot¤sort¡+¡*«PlutoPlotly‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100°PlutoPlotly.plot§maximum¦LayoutÙ$c72d35fc-c8fd-450e-9b95-d12ece5c2291„´precedence_heuristic §cell_idÙ$c72d35fc-c8fd-450e-9b95-d12ece5c2291´downstream_cells_map®get_param_list‘Ù$8f1c7b0d-121c-46bf-884d-729e3b593025²upstream_cells_map…¡:§collect¡^§IntegerAbstractFloatÙ$97c94391-397d-4cf2-88b1-3bea3af56ed3„´precedence_heuristic §cell_idÙ$97c94391-397d-4cf2-88b1-3bea3af56ed3´downstream_cells_mapcreate_bandit–Ù$7950e06b-e8ce-4bd7-9681-ab7b66dfec69Ù$33009d31-6d66-4aba-b44c-edd911c2f392Ù$c12dab6a-92f0-41b1-a6b7-c404c74d9a83Ù$50fbdc85-82f1-4c52-936b-84eb14951d71Ù$7678b06b-feef-4656-a801-33f630437bfbÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872²upstream_cells_map„¥randn¡+§IntegerAbstractFloatÙ$839861db-676f-4544-a802-0abb5d0049e1„´precedence_heuristic §cell_idÙ$839861db-676f-4544-a802-0abb5d0049e1´downstream_cells_mapAverageMethodÙ$1ac5588c-3c32-436e-8b40-41715223fba7Ù$f00ab44e-0d84-40c7-aa23-358c77a013e3Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9Ù$fb3381b5-10e3-4307-b76d-672245fac9e7Ù$ff6598fa-3366-416c-88a1-6bfcefeb1719Ù$d2ebd908-387d-4e40-bc00-61ce5f45ebddÙ$13f0adab-7660-49df-b26d-5f89cd73192bÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2eÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4Ù$8a04adab-e97e-4ac4-a85e-5eae93b1c37bÙ$69b560c1-98ad-4cbf-89d2-e0516299bc69Ù$09227386-1620-4093-b913-5786205aad13Ù$781fa565-398c-4c89-89f3-0595455afc85²upstream_cells_mapAbstractFloatÙ$9d36934a-78cb-446b-b3db-1bbd88cf272d„´precedence_heuristic §cell_idÙ$9d36934a-78cb-446b-b3db-1bbd88cf272d´downstream_cells_map€²upstream_cells_map‚¤Type©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358aÙ$d24bd737-9e09-441a-aa94-9279c80f566d„´precedence_heuristic §cell_idÙ$d24bd737-9e09-441a-aa94-9279c80f566d´downstream_cells_map€²upstream_cells_map‚³nonstationaryparams‘Ù$276779a3-9332-46bd-b511-a33a2fea4b5f«exercise2_5‘Ù$34f65898-cbcc-4832-afac-0f7a284e7f0bÙ$69b560c1-98ad-4cbf-89d2-e0516299bc69„´precedence_heuristic §cell_idÙ$69b560c1-98ad-4cbf-89d2-e0516299bc69´downstream_cells_map€²upstream_cells_mapŠ®GradientReward‘Ù$8a04adab-e97e-4ac4-a85e-5eae93b1c37b¤zeroSampleAverage‘Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9¡/AverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1¤ones¥zeros¤Type§IntegerAbstractFloatÙ$24759e26-e670-4330-b6e4-b313620660f1„´precedence_heuristic §cell_idÙ$24759e26-e670-4330-b6e4-b313620660f1´downstream_cells_map¯calculate_Ï€vec‘Ù$f7519adc-7dfb-4030-86f0-7445699dd3db²upstream_cells_mapƒ£exp®AbstractVector°normalize_Ï€vec!‘Ù$99945570-0b2a-432a-ae83-3a7abc39cac8Ù$22fa2b71-a98f-4b87-9e0b-9d373cd8915f„´precedence_heuristic §cell_idÙ$22fa2b71-a98f-4b87-9e0b-9d373cd8915f´downstream_cells_map·stationary_param_search‘Ù$140f1e20-f86d-4a6f-9cff-99685e129e1c²upstream_cells_map‚·average_runs_cum_reward‘Ù$8a52acf7-5d57-490f-8bd9-6e1e0e322872¬param_search’Ù$8f1c7b0d-121c-46bf-884d-729e3b593025Ù$a15e5d05-a238-440b-9a43-d830c6ea2f4dÙ$f561b0a8-a086-4e1a-bc87-82c4205e89c9„´precedence_heuristic §cell_idÙ$f561b0a8-a086-4e1a-bc87-82c4205e89c9´downstream_cells_mapSampleAverage™Ù$30aa1e1b-0b51-40c4-a093-ef92c3ad519aÙ$0e606680-dd65-444f-bc98-73de4abbcdd4Ù$4ebd4a5a-3bd1-48e2-b03e-a5a3b2ec18a1Ù$fb3381b5-10e3-4307-b76d-672245fac9e7Ù$0fbbe455-79ce-44d6-b010-da0bb56adbb4Ù$69b560c1-98ad-4cbf-89d2-e0516299bc69Ù$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0baÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03Ù$30f05bd9-e939-4810-b710-edc7f5975921²upstream_cells_map‚AverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1AbstractFloatÙ$85b3dbc4-2e02-4924-b3e5-887a1a557a2b„´precedence_heuristic §cell_idÙ$85b3dbc4-2e02-4924-b3e5-887a1a557a2b´downstream_cells_map¤fmax‘Ù$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfa²upstream_cells_map†£pdf¡-¡^¡*£cdf¦NormalÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3e„´precedence_heuristic §cell_idÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3e´downstream_cells_mapsample_action’Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_mapŠ«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b¡<¦argmax®AbstractVector¦isless¤view©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358a¨shuffle!¤rand§IntegerÙ$cb93c588-3dfa-45f4-9d83-f2de26cb1cea„´precedence_heuristic §cell_idÙ$cb93c588-3dfa-45f4-9d83-f2de26cb1cea´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$5712b303-0aa3-4501-b1b5-020136d6e655„´precedence_heuristic §cell_idÙ$5712b303-0aa3-4501-b1b5-020136d6e655´downstream_cells_map€²upstream_cells_map‚´UnbiasedConstantStep‘Ù$f00ab44e-0d84-40c7-aa23-358c77a013e3¤TypeÙ$ca726a9d-364d-48e2-8882-20ddbc85b664„´precedence_heuristic §cell_idÙ$ca726a9d-364d-48e2-8882-20ddbc85b664´downstream_cells_map€²upstream_cells_mapƒ¤Type©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358aAbstractFloatÙ$1e4ac085-7b72-4bad-ad87-21635930a6f7„´precedence_heuristic §cell_idÙ$1e4ac085-7b72-4bad-ad87-21635930a6f7´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$bf3770ea-ee54-4296-ab33-340aea445670„´precedence_heuristic §cell_idÙ$bf3770ea-ee54-4296-ab33-340aea445670´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$db30e6c0-36bb-4602-a257-5768f3833525„´precedence_heuristic §cell_idÙ$db30e6c0-36bb-4602-a257-5768f3833525´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$4982b489-ca15-4188-90e5-565c45f02e01„´precedence_heuristic §cell_idÙ$4982b489-ca15-4188-90e5-565c45f02e01´downstream_cells_mapsample_action’Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_map„®GradientReward‘Ù$8a04adab-e97e-4ac4-a85e-5eae93b1c37b§weights¦sample§IntegerÙ$6ce00349-1cf0-4a80-bddd-c1d26b66d051„´precedence_heuristic §cell_idÙ$6ce00349-1cf0-4a80-bddd-c1d26b66d051´downstream_cells_map¨plotdict‘Ù$8a0462e9-fd65-4ccc-b795-446cf9ae7392²upstream_cells_map©loadplots‘Ù$7c562867-55d3-4b4f-950d-c8efc4a9ff32Ù$b29624b0-be1d-4cc2-964d-0a050f4c1bed„´precedence_heuristic §cell_idÙ$b29624b0-be1d-4cc2-964d-0a050f4c1bed´downstream_cells_map¡p‘Ù$c6a8a9d2-1f29-4082-9fd2-d409cf6227ba²upstream_cells_map‡£erf¡-¤sqrt¡^¡/¡+¡*Ù$400e0de1-0101-4531-928f-08ca155da40c„´precedence_heuristic §cell_idÙ$400e0de1-0101-4531-928f-08ca155da40c´downstream_cells_map¬preparetrace’Ù$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_map¨sortpermÙ$8a04adab-e97e-4ac4-a85e-5eae93b1c37b„´precedence_heuristic §cell_idÙ$8a04adab-e97e-4ac4-a85e-5eae93b1c37b´downstream_cells_map®GradientReward—Ù$69b560c1-98ad-4cbf-89d2-e0516299bc69Ù$d9265b98-cc3e-4a60-b16e-f54d9f78c9d3Ù$4982b489-ca15-4188-90e5-565c45f02e01Ù$30f05bd9-e939-4810-b710-edc7f5975921Ù$09227386-1620-4093-b913-5786205aad13Ù$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_map†®GradientSample‘Ù$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65£newAverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1¯BanditAlgorithm‘Ù$1e1f6d10-1b31-4e0a-96f7-23207e913154AbstractFloat¦VectorÙ$3e5a226e-ecdb-43fb-a40a-a262da0ae542„´precedence_heuristic §cell_idÙ$3e5a226e-ecdb-43fb-a40a-a262da0ae542´downstream_cells_map¥ktest‘Ù$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802²upstream_cells_mapŒ§@md_str¤Core¡:¨Base.get¥@bind¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField§confirm¯Core.applicable¨getindexÙ$7a31d1c5-260b-42f4-b997-967150881e21„´precedence_heuristic §cell_idÙ$7a31d1c5-260b-42f4-b997-967150881e21´downstream_cells_map€²upstream_cells_mapƒ´UnbiasedConstantStep‘Ù$f00ab44e-0d84-40c7-aa23-358c77a013e3¤TypeAbstractFloatÙ$8584ece7-badc-486a-9a57-b60e77f92673„´precedence_heuristic §cell_idÙ$8584ece7-badc-486a-9a57-b60e77f92673´downstream_cells_map€²upstream_cells_map‚³ktest_nonstationary‘Ù$9b625fc0-89bd-4064-a379-225e6a940af7¿nonstationary_bandit_statistics‘Ù$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2Ù$e07a27c5-0c9a-4893-a1cf-cf565ab78761„´precedence_heuristic §cell_idÙ$e07a27c5-0c9a-4893-a1cf-cf565ab78761´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$b24a92fc-f6c6-44e4-9afc-fa4249e4ab83„´precedence_heuristic §cell_idÙ$b24a92fc-f6c6-44e4-9afc-fa4249e4ab83´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$ea6d7cad-47ad-4472-a9e9-1ee33c81058d„´precedence_heuristic §cell_idÙ$ea6d7cad-47ad-4472-a9e9-1ee33c81058d´downstream_cells_map©update_H!‘Ù$f7519adc-7dfb-4030-86f0-7445699dd3db²upstream_cells_map…¡-®AbstractVector¡+¡*§IntegerÙ$74024d96-d0c7-43c8-8379-caf843cbe4b8„´precedence_heuristic §cell_idÙ$74024d96-d0c7-43c8-8379-caf843cbe4b8´downstream_cells_mapÙ%average_nonstationary_runs_cum_reward‘Ù$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2²upstream_cells_map‰¥Int64¥floor¡/·average_runs_cum_reward‘Ù$8a52acf7-5d57-490f-8bd9-6e1e0e322872¡+¤ones¡*¨FunctionAbstractFloatÙ$1c9b54cd-08dd-401e-9705-818741844e8d„´precedence_heuristic §cell_idÙ$1c9b54cd-08dd-401e-9705-818741844e8d´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$e2597cc6-a6f6-4260-887b-c587cacd3bc8„´precedence_heuristic §cell_idÙ$e2597cc6-a6f6-4260-887b-c587cacd3bc8´downstream_cells_map€²upstream_cells_map‚¤Type¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7Ù$47be3ae6-20f7-47d0-aae3-b67154afc1a8„´precedence_heuristic §cell_idÙ$47be3ae6-20f7-47d0-aae3-b67154afc1a8´downstream_cells_map€²upstream_cells_mapƒ¤Type£UCB‘Ù$60b2079e-0efa-427e-93cf-7f4646fe202eAbstractFloatÙ$70e40b75-e7d8-4009-af80-3bf4086a28df„´precedence_heuristic §cell_idÙ$70e40b75-e7d8-4009-af80-3bf4086a28df´downstream_cells_mapªparams_2_3‘Ù$b3ec4673-af63-4d4a-8314-fa7e594f8a37²upstream_cells_mapŽ§@md_str¤Core¡:§PlutoUI‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¨Base.get¥@bind¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField§confirm¯Core.applicable¯PlutoUI.combine¨getindexÙ$cb2bf56d-5c1d-4265-9480-11523f776a78„´precedence_heuristic §cell_idÙ$cb2bf56d-5c1d-4265-9480-11523f776a78´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$f9e60b35-84d2-4b4b-8832-6b0f08152396„´precedence_heuristic §cell_idÙ$f9e60b35-84d2-4b4b-8832-6b0f08152396´downstream_cells_map€²upstream_cells_mapƒ§@md_str³ktest_nonstationary‘Ù$9b625fc0-89bd-4064-a379-225e6a940af7¨getindexÙ$083c721c-70dd-4ca3-8160-fc0b0531914f„´precedence_heuristic §cell_idÙ$083c721c-70dd-4ca3-8160-fc0b0531914f´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$49e45202-b9ae-42ab-9575-a57edb626a20„´precedence_heuristic §cell_idÙ$49e45202-b9ae-42ab-9575-a57edb626a20´downstream_cells_map€²upstream_cells_map„¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0f¤Type©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358aAbstractFloatÙ$ff6598fa-3366-416c-88a1-6bfcefeb1719„´precedence_heuristic §cell_idÙ$ff6598fa-3366-416c-88a1-6bfcefeb1719´downstream_cells_map€²upstream_cells_map„AverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1¤Type¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7AbstractFloatÙ$647ab36b-641e-4024-ad2d-40ff33be28f4„´precedence_heuristic §cell_idÙ$647ab36b-641e-4024-ad2d-40ff33be28f4´downstream_cells_mapºaverage_nonstationary_runs‘Ù$34f65898-cbcc-4832-afac-0f7a284e7f0b²upstream_cells_mapÞ®Base.Threads.*·nonstationary_algorithm‘Ù$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5»Base.Threads.threadpoolsize®Base.Threads.-¦Vector®Base.Threads.+©eachindex¨deepcopy²Base.Threads.error®Base.Threads.>¡/§Float32¯Base.Threads.!=¨@threads¡:ºBase.Threads.threading_run³Base.Threads.divrem¤Base·Base.Threads.firstindex©enumerate¡+¥undef®Base.Threads.:³Base.Threads.length¯Base.Threads.==¥ccall¯Base.Threads.<=Ù$96566aad-6d5c-460c-a924-ae0bad5d8b2d„´precedence_heuristic §cell_idÙ$96566aad-6d5c-460c-a924-ae0bad5d8b2d´downstream_cells_mapªupdatecoef’Ù$09227386-1620-4093-b913-5786205aad13Ù$781fa565-398c-4c89-89f3-0595455afc85²upstream_cells_map‡¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0f¡T¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7¯BanditAlgorithm‘Ù$1e1f6d10-1b31-4e0a-96f7-23207e913154¡E§IntegerAbstractFloatÙ$852df31d-18d8-466c-8225-e06ba7f05e96„´precedence_heuristic §cell_idÙ$852df31d-18d8-466c-8225-e06ba7f05e96´downstream_cells_map€²upstream_cells_map‚¤Type©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358aÙ$a84f5393-bd7e-433e-b3d9-e9e7cfa1a329„´precedence_heuristic §cell_idÙ$a84f5393-bd7e-433e-b3d9-e9e7cfa1a329´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$9bd99099-1dfa-477a-9896-3da94bcc0633„´precedence_heuristic §cell_idÙ$9bd99099-1dfa-477a-9896-3da94bcc0633´downstream_cells_mapexercise_2_11‘Ù$2ceadc6d-5522-43f6-a803-a17a47a6048a²upstream_cells_map‚¿plot_nonstationary_param_search‘Ù$98f4d5e6-7569-457e-851d-713c572ae400¹make_or_lookup_param_plot‘Ù$8a0462e9-fd65-4ccc-b795-446cf9ae7392Ù$3b9bb9f0-ba9d-4320-935a-58912afe34b6„´precedence_heuristic §cell_idÙ$3b9bb9f0-ba9d-4320-935a-58912afe34b6´downstream_cells_map€²upstream_cells_mapƒ¤Type¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7AbstractFloatÙ$14bd0549-747f-4513-809f-8bdb78027807„´precedence_heuristic §cell_idÙ$14bd0549-747f-4513-809f-8bdb78027807´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2e„´precedence_heuristic §cell_idÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2e´downstream_cells_map€²upstream_cells_map‰«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0fAverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1¥zeros¤ones¡*¤Type§IntegerAbstractFloatÙ$1fb1a518-e5ec-4777-80bc-bb55e8172100„´precedence_heuristic§cell_idÙ$1fb1a518-e5ec-4777-80bc-bb55e8172100´downstream_cells_map‹ªStatistics©StatsBaseDistributions§PlutoUI˜Ù$6034cc0f-cbae-4d2f-a43f-1bb738c00f0bÙ$276779a3-9332-46bd-b511-a33a2fea4b5fÙ$70e40b75-e7d8-4009-af80-3bf4086a28dfÙ$093f312b-d70d-4bf7-bd53-8a1c7b2bee31Ù$b5a2df21-4525-4320-b8dd-aea5ecdab832Ù$3b88ec30-768b-44d0-88ee-b3ed989f22c3Ù$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53Ù$7cab41cc-2a39-45de-879d-f7f6105238d6§Threads¦Random“Ù$8fac4109-2e0d-4366-9118-018221e0b910Ù$8a52acf7-5d57-490f-8bd9-6e1e0e322872Ù$f1466027-7f52-41df-a8ea-a0650981c9d4¨Latexify°SpecialFunctions¬PlutoProfile¬LaTeXStrings“Ù$34f65898-cbcc-4832-afac-0f7a284e7f0bÙ$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400«PlutoPlotly—Ù$7950e06b-e8ce-4bd7-9681-ab7b66dfec69Ù$c12dab6a-92f0-41b1-a6b7-c404c74d9a83Ù$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfaÙ$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2Ù$6292f449-8720-41f1-84de-1865fb5fddbfÙ$8fac4109-2e0d-4366-9118-018221e0b910Ù$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9²upstream_cells_map¯TableOfContentsÙ$1e1f6d10-1b31-4e0a-96f7-23207e913154„´precedence_heuristic §cell_idÙ$1e1f6d10-1b31-4e0a-96f7-23207e913154´downstream_cells_map¯BanditAlgorithm•Ù$1004eb4b-1fed-4328-a08b-6f5d9dd5080bÙ$13f0adab-7660-49df-b26d-5f89cd73192bÙ$8a04adab-e97e-4ac4-a85e-5eae93b1c37bÙ$96566aad-6d5c-460c-a924-ae0bad5d8b2dÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0ad²upstream_cells_map€Ù$44b9ff95-ea3d-41f5-8098-445a263738a9„´precedence_heuristic §cell_idÙ$44b9ff95-ea3d-41f5-8098-445a263738a9´downstream_cells_map€²upstream_cells_mapƒ¤Type©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358aAbstractFloatÙ$6034cc0f-cbae-4d2f-a43f-1bb738c00f0b„´precedence_heuristic §cell_idÙ$6034cc0f-cbae-4d2f-a43f-1bb738c00f0b´downstream_cells_map¶Ïµ_action_value_params‘Ù$79082409-3182-4e0b-9c8c-37a94543fee9²upstream_cells_map§@md_str¤Core¡:§PlutoUI‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¨Base.get¥@bind¬testbedreset‘Ù$87f5ec29-623f-4666-824d-fad8c6448072¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField§confirm¯Core.applicable¯PlutoUI.combine¨getindexÙ$aa7c76b1-ac2d-4df5-88ef-b499b839181e„´precedence_heuristic §cell_idÙ$aa7c76b1-ac2d-4df5-88ef-b499b839181e´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$0f6b4e2d-dc09-4e1c-834f-dd8aaa8743ae„´precedence_heuristic §cell_idÙ$0f6b4e2d-dc09-4e1c-834f-dd8aaa8743ae´downstream_cells_map€²upstream_cells_map‚´UnbiasedConstantStep‘Ù$f00ab44e-0d84-40c7-aa23-358c77a013e3¤TypeÙ$f7519adc-7dfb-4030-86f0-7445699dd3db„´precedence_heuristic §cell_idÙ$f7519adc-7dfb-4030-86f0-7445699dd3db´downstream_cells_mapÙ$gradient_stationary_bandit_algorithm‘Ù$50fbdc85-82f1-4c52-936b-84eb14951d71²upstream_cells_mapÞ¦argmax¡:¯calculate_Ï€vec‘Ù$24759e26-e670-4330-b6e4-b313620660f1§collect¥zeros§Float64§Integer¦Vectorsample_action—Ù$51349e41-4696-4bd5-9bc1-cefbb82bea08Ù$4982b489-ca15-4188-90e5-565c45f02e01Ù$672a91c0-aa77-4257-8c83-d857f47cab6cÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3eÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18Ù$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0baÙ$d53796f1-6fc4-4450-9d66-05ba752ff7eb¡-update_Ï€vec!‘Ù$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5¡/¡+sample_bandit‘Ù$55f89ca1-cd57-44e0-95e5-63a4be31418f©update_H!‘Ù$ea6d7cad-47ad-4472-a9e9-1ee33c81058d¢==¤fill¡*Ù$181d7eef-24a0-4775-a535-8ef901b7e4eb„´precedence_heuristic §cell_idÙ$181d7eef-24a0-4775-a535-8ef901b7e4eb´downstream_cells_mapºaverage_nonstationary_runs‘Ù$34f65898-cbcc-4832-afac-0f7a284e7f0b²upstream_cells_map…¬average_runs‘Ù$7678b06b-feef-4656-a801-33f630437bfb¤ones¡*¨FunctionAbstractFloatÙ$5aa170d3-eb87-44dd-b4f4-3ac97476efd7„´precedence_heuristic §cell_idÙ$5aa170d3-eb87-44dd-b4f4-3ac97476efd7´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b„´precedence_heuristic §cell_idÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b´downstream_cells_mapªrun_bandit’Ù$7678b06b-feef-4656-a801-33f630437bfbÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872²upstream_cells_mapÞ¥randn¤zero§nothing¦Vectorsample_action—Ù$51349e41-4696-4bd5-9bc1-cefbb82bea08Ù$4982b489-ca15-4188-90e5-565c45f02e01Ù$672a91c0-aa77-4257-8c83-d857f47cab6cÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3eÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18Ù$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0baÙ$d53796f1-6fc4-4450-9d66-05ba752ff7eb©eachindex±update_estimator!’Ù$09227386-1620-4093-b913-5786205aad13Ù$781fa565-398c-4c89-89f3-0595455afc85¡/¢!=¤fill¢==AbstractFloat¦argmax¡:§collect¥zeros¡-¢>=¡+sample_bandit‘Ù$55f89ca1-cd57-44e0-95e5-63a4be31418f¡*¯BanditAlgorithm‘Ù$1e1f6d10-1b31-4e0a-96f7-23207e913154Ù$7cab41cc-2a39-45de-879d-f7f6105238d6„´precedence_heuristic §cell_idÙ$7cab41cc-2a39-45de-879d-f7f6105238d6´downstream_cells_mapªwsm_params‘Ù$f78846a8-076a-4a87-b1d0-4f830e2bca27²upstream_cells_mapŽ§@md_str¤Core¡:§PlutoUI‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¨Base.get¥@bind¦Slider¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField¯Core.applicable¯PlutoUI.combine¨getindexÙ$e50596ab-91db-42f0-a62c-77629a4e79c7„´precedence_heuristic §cell_idÙ$e50596ab-91db-42f0-a62c-77629a4e79c7´downstream_cells_map€²upstream_cells_map‚®GradientSample‘Ù$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65¤TypeÙ$fb3381b5-10e3-4307-b76d-672245fac9e7„´precedence_heuristic §cell_idÙ$fb3381b5-10e3-4307-b76d-672245fac9e7´downstream_cells_map€²upstream_cells_map„SampleAverage‘Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9AverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1¤TypeAbstractFloatÙ$3118e102-aeac-42d9-98fc-ca29f40be4cd„´precedence_heuristic §cell_idÙ$3118e102-aeac-42d9-98fc-ca29f40be4cd´downstream_cells_map€²upstream_cells_map‚¤Type£UCB‘Ù$60b2079e-0efa-427e-93cf-7f4646fe202eÙ$b3ec4673-af63-4d4a-8314-fa7e594f8a37„´precedence_heuristic §cell_idÙ$b3ec4673-af63-4d4a-8314-fa7e594f8a37´downstream_cells_map€²upstream_cells_map‚ªfigure_2_3‘Ù$6292f449-8720-41f1-84de-1865fb5fddbfªparams_2_3‘Ù$70e40b75-e7d8-4009-af80-3bf4086a28dfÙ$c2347999-5ade-420b-903f-30523b38eb0f„´precedence_heuristic §cell_idÙ$c2347999-5ade-420b-903f-30523b38eb0f´downstream_cells_map¨ExplorerÜÙ$f995d0af-50bc-4e33-9bbf-17a7ab06358aÙ$60b2079e-0efa-427e-93cf-7f4646fe202eÙ$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65Ù$6d647790-a029-4579-86fa-9c7294aa94eaÙ$73e5b719-8b91-41d2-b83d-471d981b027fÙ$49e45202-b9ae-42ab-9575-a57edb626a20Ù$3a215c95-c595-4837-a842-1c1e1c6bfa3bÙ$a04da367-3f8d-422d-a443-4e3e666e30efÙ$68470b1d-3cc2-4cb1-8dc2-53227e6300e7Ù$d42a346d-6af3-45df-9de5-7043f46c3da3Ù$13f0adab-7660-49df-b26d-5f89cd73192bÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2eÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4Ù$62eb0650-96bc-4fd6-bfe0-bf05a4137a03Ù$96566aad-6d5c-460c-a924-ae0bad5d8b2dÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0adÙ$781fa565-398c-4c89-89f3-0595455afc85²upstream_cells_mapAbstractFloatÙ$1ac5588c-3c32-436e-8b40-41715223fba7„´precedence_heuristic §cell_idÙ$1ac5588c-3c32-436e-8b40-41715223fba7´downstream_cells_map¬ConstantStep›Ù$262952a7-280e-4af1-99a6-0899518484a2Ù$fbf7b108-dc68-4077-b63c-6f88161d2098Ù$8018edb9-853f-4a88-bef6-9535c2d774b3Ù$9093ed39-f6e2-4c31-9fef-466bdd197423Ù$e2597cc6-a6f6-4260-887b-c587cacd3bc8Ù$3b9bb9f0-ba9d-4320-935a-58912afe34b6Ù$ff6598fa-3366-416c-88a1-6bfcefeb1719Ù$d53796f1-6fc4-4450-9d66-05ba752ff7ebÙ$96566aad-6d5c-460c-a924-ae0bad5d8b2dÙ$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_map†£new¡-¢>=AverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1AbstractFloat£invÙ$34f65898-cbcc-4832-afac-0f7a284e7f0b„´precedence_heuristic §cell_idÙ$34f65898-cbcc-4832-afac-0f7a284e7f0b´downstream_cells_map«exercise2_5‘Ù$d24bd737-9e09-441a-aa94-9279c80f566d²upstream_cells_mapŠ§@md_strBase.getindex¸LaTeXStrings.latexstringºaverage_nonstationary_runs’Ù$647ab36b-641e-4024-ad2d-40ff33be28f4Ù$181d7eef-24a0-4775-a535-8ef901b7e4eb¤Base§scatter¤plot¦@L_str¬LaTeXStrings‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¦LayoutÙ$4ebd4a5a-3bd1-48e2-b03e-a5a3b2ec18a1„´precedence_heuristic §cell_idÙ$4ebd4a5a-3bd1-48e2-b03e-a5a3b2ec18a1´downstream_cells_map€²upstream_cells_map‚SampleAverage‘Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9¤TypeÙ$093f312b-d70d-4bf7-bd53-8a1c7b2bee31„´precedence_heuristic §cell_idÙ$093f312b-d70d-4bf7-bd53-8a1c7b2bee31´downstream_cells_mapªparams_2_4‘Ù$0de99ee5-d94d-4d07-8cef-a6f9caf5e742²upstream_cells_mapŽ§@md_str¤Core¡:§PlutoUI‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¨Base.get¥@bind¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField§confirm¯Core.applicable¯PlutoUI.combine¨getindexÙ$7748ab8a-d186-49a4-b6ab-d1bd9ea34990„´precedence_heuristic §cell_idÙ$7748ab8a-d186-49a4-b6ab-d1bd9ea34990´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$140f1e20-f86d-4a6f-9cff-99685e129e1c„´precedence_heuristic §cell_idÙ$140f1e20-f86d-4a6f-9cff-99685e129e1c´downstream_cells_map¼plot_stationary_param_search‘Ù$8a0462e9-fd65-4ccc-b795-446cf9ae7392²upstream_cells_mapÞ¸LaTeXStrings.latexstring¤vcat¦colors‘Ù$fb698663-e9d6-4368-bd53-d88da5e6f2c4¹OptimalDistributionSample‘Ù$6d647790-a029-4579-86fa-9c7294aa94ea©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358a¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7£UCB‘Ù$60b2079e-0efa-427e-93cf-7f4646fe202e«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b©eachindex§scatter§Float32¬LaTeXStrings‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100¦unique¦@L_str©maketrace‘Ù$7c2015dd-a786-49f5-9fe9-9199335ebd09¤Plot¦reduce®GradientReward‘Ù$8a04adab-e97e-4ac4-a85e-5eae93b1c37b¬print_power2‘Ù$c61630b0-29c1-4183-90d9-57c999187b53¬preparetrace‘Ù$400e0de1-0101-4531-928f-08ca155da40c·stationary_param_search‘Ù$22fa2b71-a98f-4b87-9e0b-9d373cd8915f¤sort¤attr¦LayoutÙ$965da91b-6a3f-456c-89ce-461c31e0fb7e„´precedence_heuristic §cell_idÙ$965da91b-6a3f-456c-89ce-461c31e0fb7e´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$9d7782f5-b530-40d5-9f75-280d3a762216„´precedence_heuristic §cell_idÙ$9d7782f5-b530-40d5-9f75-280d3a762216´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$bc939df7-e457-496c-8977-5fbf9dfe3638„´precedence_heuristic §cell_idÙ$bc939df7-e457-496c-8977-5fbf9dfe3638´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$464d43c0-cd59-49e6-88f6-12a767677418„´precedence_heuristic §cell_idÙ$464d43c0-cd59-49e6-88f6-12a767677418´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$21e56374-35e6-4488-b8da-15e383017c77„´precedence_heuristic §cell_idÙ$21e56374-35e6-4488-b8da-15e383017c77´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$f33adafb-44b4-4e56-b3de-9717fe026288„´precedence_heuristic §cell_idÙ$f33adafb-44b4-4e56-b3de-9717fe026288´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$7c2015dd-a786-49f5-9fe9-9199335ebd09„´precedence_heuristic §cell_idÙ$7c2015dd-a786-49f5-9fe9-9199335ebd09´downstream_cells_map©maketrace’Ù$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_mapƒ¤vcat§scatter¤attrÙ$ecd8833d-441b-4673-b457-dc5109f575ab„´precedence_heuristic §cell_idÙ$ecd8833d-441b-4673-b457-dc5109f575ab´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$d42a346d-6af3-45df-9de5-7043f46c3da3„´precedence_heuristic §cell_idÙ$d42a346d-6af3-45df-9de5-7043f46c3da3´downstream_cells_map€²upstream_cells_map„¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0f¤Type¹OptimalDistributionSample‘Ù$6d647790-a029-4579-86fa-9c7294aa94eaAbstractFloatÙ$c66f1676-aec1-489d-96ff-99d748dac0fe„´precedence_heuristic §cell_idÙ$c66f1676-aec1-489d-96ff-99d748dac0fe´downstream_cells_map€²upstream_cells_map‚«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b¤TypeÙ$d2ebd908-387d-4e40-bc00-61ce5f45ebdd„´precedence_heuristic §cell_idÙ$d2ebd908-387d-4e40-bc00-61ce5f45ebdd´downstream_cells_map€²upstream_cells_map„AverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1´UnbiasedConstantStep‘Ù$f00ab44e-0d84-40c7-aa23-358c77a013e3¤TypeAbstractFloatÙ$97f6221d-3289-4e54-a80d-26c5c81f2651„´precedence_heuristic §cell_idÙ$97f6221d-3289-4e54-a80d-26c5c81f2651´downstream_cells_map²execute_stationary‘Ù$51b7a645-269a-418c-b6d8-39c01d0609f1²upstream_cells_mapŠ¤Core§@md_str¤Base·PlutoRunner.create_bond«PlutoRunner¨CheckBox¯Core.applicable¥@bind¨Base.get¨getindexÙ$8018edb9-853f-4a88-bef6-9535c2d774b3„´precedence_heuristic §cell_idÙ$8018edb9-853f-4a88-bef6-9535c2d774b3´downstream_cells_map€²upstream_cells_mapƒ¤Type¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7AbstractFloatÙ$672a91c0-aa77-4257-8c83-d857f47cab6c„´precedence_heuristic §cell_idÙ$672a91c0-aa77-4257-8c83-d857f47cab6c´downstream_cells_mapsample_action’Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_mapƒ«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b®AbstractVector§IntegerÙ$a04da367-3f8d-422d-a443-4e3e666e30ef„´precedence_heuristic §cell_idÙ$a04da367-3f8d-422d-a443-4e3e666e30ef´downstream_cells_map€²upstream_cells_map„¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0f¤Type£UCB‘Ù$60b2079e-0efa-427e-93cf-7f4646fe202eAbstractFloatÙ$8f1c7b0d-121c-46bf-884d-729e3b593025„´precedence_heuristic §cell_idÙ$8f1c7b0d-121c-46bf-884d-729e3b593025´downstream_cells_map¬param_search’Ù$22fa2b71-a98f-4b87-9e0b-9d373cd8915fÙ$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2²upstream_cells_map‡§similar¦length®get_param_list‘Ù$c72d35fc-c8fd-450e-9b95-d12ece5c2291¥undef£zipAbstractFloat¦VectorÙ$87f5ec29-623f-4666-824d-fad8c6448072„´precedence_heuristic §cell_idÙ$87f5ec29-623f-4666-824d-fad8c6448072´downstream_cells_map¬testbedreset‘Ù$6034cc0f-cbae-4d2f-a43f-1bb738c00f0b²upstream_cells_mapˆ¤Core¤Base·PlutoRunner.create_bond«PlutoRunner¯Core.applicable¥@bind¨Base.get¦ButtonÙ$30aa1e1b-0b51-40c4-a093-ef92c3ad519a„´precedence_heuristic §cell_idÙ$30aa1e1b-0b51-40c4-a093-ef92c3ad519a´downstream_cells_map€²upstream_cells_mapƒSampleAverage‘Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9¤Type§Float64Ù$8a0462e9-fd65-4ccc-b795-446cf9ae7392„´precedence_heuristic §cell_idÙ$8a0462e9-fd65-4ccc-b795-446cf9ae7392´downstream_cells_map¹make_or_lookup_param_plot’Ù$51b7a645-269a-418c-b6d8-39c01d0609f1Ù$9bd99099-1dfa-477a-9896-3da94bcc0633²upstream_cells_map¨plotdict‘Ù$6ce00349-1cf0-4a80-bddd-c1d26b66d051¨IOBuffer¡!¨joinpath¢|>¼plot_stationary_param_search‘Ù$140f1e20-f86d-4a6f-9cff-99685e129e1c¦haskey¦String²PlotlyBase.to_html¤HTML¤open¨datapath‘Ù$1cde4625-f8ed-4403-835c-95cc30206699ªPlotlyBaseÙ$fb698663-e9d6-4368-bd53-d88da5e6f2c4„´precedence_heuristic §cell_idÙ$fb698663-e9d6-4368-bd53-d88da5e6f2c4´downstream_cells_map¦colors’Ù$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_mapªPlotlyBaseÙ$13f0adab-7660-49df-b26d-5f89cd73192b„´precedence_heuristic §cell_idÙ$13f0adab-7660-49df-b26d-5f89cd73192b´downstream_cells_map«ActionValueœÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2eÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4Ù$c66f1676-aec1-489d-96ff-99d748dac0feÙ$672a91c0-aa77-4257-8c83-d857f47cab6cÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3eÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18Ù$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0baÙ$d53796f1-6fc4-4450-9d66-05ba752ff7ebÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03Ù$781fa565-398c-4c89-89f3-0595455afc85Ù$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_map…¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0fAverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1¯BanditAlgorithm‘Ù$1e1f6d10-1b31-4e0a-96f7-23207e913154AbstractFloat¦VectorÙ$c6a8a9d2-1f29-4082-9fd2-d409cf6227ba„´precedence_heuristic §cell_idÙ$c6a8a9d2-1f29-4082-9fd2-d409cf6227ba´downstream_cells_map€²upstream_cells_map‡§testvar‘Ù$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53¡:¡-¤plot¡/¦Layout¡p‘Ù$b29624b0-be1d-4cc2-964d-0a050f4c1bedÙ$eed2a4f2-48b2-4684-846e-aa99bb6dafd9„´precedence_heuristic §cell_idÙ$eed2a4f2-48b2-4684-846e-aa99bb6dafd9´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$8c0f06f7-2ed0-4f3a-ab4e-90ac142f0cd9„´precedence_heuristic §cell_idÙ$8c0f06f7-2ed0-4f3a-ab4e-90ac142f0cd9´downstream_cells_map€²upstream_cells_mapƒ¤Type£UCB‘Ù$60b2079e-0efa-427e-93cf-7f4646fe202eAbstractFloatÙ$d53796f1-6fc4-4450-9d66-05ba752ff7eb„´precedence_heuristic §cell_idÙ$d53796f1-6fc4-4450-9d66-05ba752ff7eb´downstream_cells_mapsample_action’Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_mapÞ¡!¡>¤sqrt¦isless¹OptimalDistributionSample‘Ù$6d647790-a029-4579-86fa-9c7294aa94ea¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b¡<©eachindex¡/¡^§isempty¢==AbstractFloat¦Normal®AbstractVector£Inf¤rand§Integer§findall¡-¡+¡*Ù$0f244fa0-7591-4478-b172-d9c1de51f6e1„´precedence_heuristic §cell_idÙ$0f244fa0-7591-4478-b172-d9c1de51f6e1´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$262952a7-280e-4af1-99a6-0899518484a2„´precedence_heuristic §cell_idÙ$262952a7-280e-4af1-99a6-0899518484a2´downstream_cells_map€²upstream_cells_mapƒ¤Type§Float64¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7Ù$46478110-5ce5-4c72-bb3d-bcb5f516ffdc„´precedence_heuristic §cell_idÙ$46478110-5ce5-4c72-bb3d-bcb5f516ffdc´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$f1466027-7f52-41df-a8ea-a0650981c9d4„´precedence_heuristic §cell_idÙ$f1466027-7f52-41df-a8ea-a0650981c9d4´downstream_cells_map¬plot_softmax‘Ù$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8e²upstream_cells_map§@md_str¥randn¬Random.seed!£exp¡:£sum¦Random‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100£bar¤plot¡/¡*¦Layout¨getindexÙ$33009d31-6d66-4aba-b44c-edd911c2f392„´precedence_heuristic §cell_idÙ$33009d31-6d66-4aba-b44c-edd911c2f392´downstream_cells_map³average_simple_runs“Ù$c12dab6a-92f0-41b1-a6b7-c404c74d9a83Ù$6292f449-8720-41f1-84de-1865fb5fddbfÙ$8fac4109-2e0d-4366-9118-018221e0b910²upstream_cells_mapÞ®Base.Threads.*»Base.Threads.threadpoolsize®Base.Threads.-¦Vector°simple_algorithm‘Ù$3a798da5-c309-48f2-aab1-6602ded8a650®Base.Threads.+©eachindex¨deepcopycreate_bandit‘Ù$97c94391-397d-4cf2-88b1-3bea3af56ed3²Base.Threads.error®Base.Threads.>¡/§Float32¯Base.Threads.!=¨@threads¡:ºBase.Threads.threading_run³Base.Threads.divrem¤Base·Base.Threads.firstindex¡+¥undef®Base.Threads.:³Base.Threads.length¯Base.Threads.==¥ccall¯Base.Threads.<=Ù$31d8fba9-cd28-4b2b-ba46-70a068b9ecad„´precedence_heuristic §cell_idÙ$31d8fba9-cd28-4b2b-ba46-70a068b9ecad´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$f78846a8-076a-4a87-b1d0-4f830e2bca27„´precedence_heuristic §cell_idÙ$f78846a8-076a-4a87-b1d0-4f830e2bca27´downstream_cells_map„ªsamplelist¯weightedmeanvar¢t2¢t1²upstream_cells_map‹¡:ªwsm_params‘Ù$7cab41cc-2a39-45de-879d-f7f6105238d6¡-§scatter¤plot¡/¡^¡+¤attr¡*¦LayoutÙ$73e5b719-8b91-41d2-b83d-471d981b027f„´precedence_heuristic §cell_idÙ$73e5b719-8b91-41d2-b83d-471d981b027f´downstream_cells_map€²upstream_cells_map„¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0f¤Type©Ïµ_Greedy‘Ù$f995d0af-50bc-4e33-9bbf-17a7ab06358aAbstractFloatÙ$51b7a645-269a-418c-b6d8-39c01d0609f1„´precedence_heuristic §cell_idÙ$51b7a645-269a-418c-b6d8-39c01d0609f1´downstream_cells_map€²upstream_cells_mapƒ³stationary_numsteps‘Ù$88e43fed-fcf3-4071-996a-63f63c3d49b4²execute_stationary‘Ù$97f6221d-3289-4e54-a80d-26c5c81f2651¹make_or_lookup_param_plot‘Ù$8a0462e9-fd65-4ccc-b795-446cf9ae7392Ù$23b99305-c8d9-4129-85fb-a5e4aabc4a31„´precedence_heuristic §cell_idÙ$23b99305-c8d9-4129-85fb-a5e4aabc4a31´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$4a89cdd9-c20f-40e2-bc84-c3ea9cbf00e7„´precedence_heuristic §cell_idÙ$4a89cdd9-c20f-40e2-bc84-c3ea9cbf00e7´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$3a215c95-c595-4837-a842-1c1e1c6bfa3b„´precedence_heuristic §cell_idÙ$3a215c95-c595-4837-a842-1c1e1c6bfa3b´downstream_cells_map€²upstream_cells_map„¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0f¤Type£UCB‘Ù$60b2079e-0efa-427e-93cf-7f4646fe202eAbstractFloatÙ$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9„´precedence_heuristic §cell_idÙ$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9´downstream_cells_mapªfigure_2_5‘Ù$691aa77a-d6da-4fde-9024-c4195057179d²upstream_cells_mapÞ§@md_str¦Layout¡:¡>¦isless¤Bool¦Vector¡<Ù average_gradient_stationary_runs‘Ù$50fbdc85-82f1-4c52-936b-84eb14951d71§scatter©enumerate¥push!¡*«PlutoPlotly‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100°PlutoPlotly.plotAbstractFloat¨getindexÙ$55f89ca1-cd57-44e0-95e5-63a4be31418f„´precedence_heuristic §cell_idÙ$55f89ca1-cd57-44e0-95e5-63a4be31418f´downstream_cells_mapsample_bandit”Ù$3a798da5-c309-48f2-aab1-6602ded8a650Ù$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_map…AbstractFloat¥randn¡+§Integer¦VectorÙ$d59126d7-5af0-4d06-a57b-e115eec32388„´precedence_heuristic §cell_idÙ$d59126d7-5af0-4d06-a57b-e115eec32388´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65„´precedence_heuristic §cell_idÙ$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65´downstream_cells_map®GradientSample•Ù$e50596ab-91db-42f0-a62c-77629a4e79c7Ù$68470b1d-3cc2-4cb1-8dc2-53227e6300e7Ù$8a04adab-e97e-4ac4-a85e-5eae93b1c37bÙ$30f05bd9-e939-4810-b710-edc7f5975921Ù$09227386-1620-4093-b913-5786205aad13²upstream_cells_map‚¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0fAbstractFloatÙ$f88029d6-3fc2-4552-8441-5ef37ac42638„´precedence_heuristic §cell_idÙ$f88029d6-3fc2-4552-8441-5ef37ac42638´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$9e644919-5833-46ff-8f6e-e755ba688329„´precedence_heuristic §cell_idÙ$9e644919-5833-46ff-8f6e-e755ba688329´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$3a798da5-c309-48f2-aab1-6602ded8a650„´precedence_heuristic §cell_idÙ$3a798da5-c309-48f2-aab1-6602ded8a650´downstream_cells_map°simple_algorithm‘Ù$33009d31-6d66-4aba-b44c-edd911c2f392²upstream_cells_mapÞ¤sqrt¦isless§Float64¤view¦Vector¡<¡/¤fill¢==AbstractFloat¦argmax¡:§collect¥zeros¤ones¤rand§Integer¡-£log¡+sample_bandit‘Ù$55f89ca1-cd57-44e0-95e5-63a4be31418f¡*¨shuffle!Ù$7950e06b-e8ce-4bd7-9681-ab7b66dfec69„´precedence_heuristic §cell_idÙ$7950e06b-e8ce-4bd7-9681-ab7b66dfec69´downstream_cells_map³plot_bandit_testbed‘Ù$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802²upstream_cells_mapŠ¥randn¦violincreate_bandit‘Ù$97c94391-397d-4cf2-88b1-3bea3af56ed3¡-¡:¡+¤attr«PlutoPlotly‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100°PlutoPlotly.plot¦LayoutÙ$c9c2aa04-15a2-446a-839c-bcf724b0f57a„´precedence_heuristic §cell_idÙ$c9c2aa04-15a2-446a-839c-bcf724b0f57a´downstream_cells_map€²upstream_cells_mapƒ¤Base®Base.Docs.HTML©@html_strÙ$d9265b98-cc3e-4a60-b16e-f54d9f78c9d3„´precedence_heuristic §cell_idÙ$d9265b98-cc3e-4a60-b16e-f54d9f78c9d3´downstream_cells_map€²upstream_cells_map‚®GradientReward‘Ù$8a04adab-e97e-4ac4-a85e-5eae93b1c37b¤TypeÙ$4afd6c2c-632e-4196-ab36-4d314ed9ec96„´precedence_heuristic §cell_idÙ$4afd6c2c-632e-4196-ab36-4d314ed9ec96´downstream_cells_map€²upstream_cells_map…§@md_str¡-¡/¥round¨getindexÙ$d4ce45ae-613e-41ee-b626-69b0dbcf6452„´precedence_heuristic §cell_idÙ$d4ce45ae-613e-41ee-b626-69b0dbcf6452´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$33d201ba-52f2-44f0-8bc6-3930ec77f62f„´precedence_heuristic §cell_idÙ$33d201ba-52f2-44f0-8bc6-3930ec77f62f´downstream_cells_map‚©softmax_k‘Ù$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8e¢Î²‘Ù$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8e²upstream_cells_mapŒ§@md_str¤Core¡:¨Base.get¥@bind¦Slider¤Base«PlutoRunner·PlutoRunner.create_bond«NumberField¯Core.applicable¨getindexÙ$0de99ee5-d94d-4d07-8cef-a6f9caf5e742„´precedence_heuristic §cell_idÙ$0de99ee5-d94d-4d07-8cef-a6f9caf5e742´downstream_cells_map€²upstream_cells_map‚ªparams_2_4‘Ù$093f312b-d70d-4bf7-bd53-8a1c7b2bee31ªfigure_2_4‘Ù$8fac4109-2e0d-4366-9118-018221e0b910Ù$a61c15eb-ed5f-4052-a3a3-3276940564a1„´precedence_heuristic §cell_idÙ$a61c15eb-ed5f-4052-a3a3-3276940564a1´downstream_cells_map€²upstream_cells_map‚¤Type£UCB‘Ù$60b2079e-0efa-427e-93cf-7f4646fe202eÙ$99945570-0b2a-432a-ae83-3a7abc39cac8„´precedence_heuristic §cell_idÙ$99945570-0b2a-432a-ae83-3a7abc39cac8´downstream_cells_map°normalize_Ï€vec!’Ù$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5Ù$24759e26-e670-4330-b6e4-b313620660f1²upstream_cells_map‰¥isinf¦length£sum¥isnan¡/£one¢==AbstractFloat¦VectorÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18„´precedence_heuristic §cell_idÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18´downstream_cells_mapsample_action’Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_map‹¦argmax®AbstractVector¤sqrt¤view§Integer£UCB‘Ù$60b2079e-0efa-427e-93cf-7f4646fe202e«ActionValue‘Ù$13f0adab-7660-49df-b26d-5f89cd73192b£log¡/¡+¡*Ù$2ceadc6d-5522-43f6-a803-a17a47a6048a„´precedence_heuristic §cell_idÙ$2ceadc6d-5522-43f6-a803-a17a47a6048a´downstream_cells_map€²upstream_cells_map‚exercise_2_11‘Ù$9bd99099-1dfa-477a-9896-3da94bcc0633¹nonstationarysearchparams‘Ù$3b88ec30-768b-44d0-88ee-b3ed989f22c3Ù$9093ed39-f6e2-4c31-9fef-466bdd197423„´precedence_heuristic §cell_idÙ$9093ed39-f6e2-4c31-9fef-466bdd197423´downstream_cells_map€²upstream_cells_map„¤Type¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7AbstractFloat¤RealÙ$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2„´precedence_heuristic §cell_idÙ$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2´downstream_cells_map¿nonstationary_bandit_statistics‘Ù$8584ece7-badc-486a-9a57-b60e77f92673²upstream_cells_mapÞ¥randn¡:£sum¤sqrt¤ones§Integer©eachindex¥push!§scatter¥sort!¡/¡+¤attr¡*«PlutoPlotly‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100°PlutoPlotly.plot¦LayoutÙ$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5„´precedence_heuristic §cell_idÙ$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5´downstream_cells_mapupdate_Ï€vec!’Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$09227386-1620-4093-b913-5786205aad13²upstream_cells_mapƒ£exp®AbstractVector°normalize_Ï€vec!‘Ù$99945570-0b2a-432a-ae83-3a7abc39cac8Ù$6292f449-8720-41f1-84de-1865fb5fddbf„´precedence_heuristic §cell_idÙ$6292f449-8720-41f1-84de-1865fb5fddbf´downstream_cells_mapªfigure_2_3‘Ù$b3ec4673-af63-4d4a-8314-fa7e594f8a37²upstream_cells_map‡¡:§scatter³average_simple_runs‘Ù$33009d31-6d66-4aba-b44c-edd911c2f392¤attr«PlutoPlotly‘Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100°PlutoPlotly.plot¦LayoutÙ$0d8e4160-adf2-4b43-9914-942539339972„´precedence_heuristic §cell_idÙ$0d8e4160-adf2-4b43-9914-942539339972´downstream_cells_map€²upstream_cells_map‚§@md_str¨getindexÙ$09227386-1620-4093-b913-5786205aad13„´precedence_heuristic §cell_idÙ$09227386-1620-4093-b913-5786205aad13´downstream_cells_map±update_estimator!‘Ù$1004eb4b-1fed-4328-a08b-6f5d9dd5080b²upstream_cells_mapŠ®GradientReward‘Ù$8a04adab-e97e-4ac4-a85e-5eae93b1c37b®GradientSample‘Ù$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65ªupdatecoef”Ù$62eb0650-96bc-4fd6-bfe0-bf05a4137a03Ù$30f05bd9-e939-4810-b710-edc7f5975921Ù$96566aad-6d5c-460c-a924-ae0bad5d8b2dÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0ad¡-update_Ï€vec!‘Ù$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5AverageMethod‘Ù$839861db-676f-4544-a802-0abb5d0049e1¡+¡*§IntegerAbstractFloatÙ$f995d0af-50bc-4e33-9bbf-17a7ab06358a„´precedence_heuristic §cell_idÙ$f995d0af-50bc-4e33-9bbf-17a7ab06358a´downstream_cells_map©Ïµ_GreedyšÙ$852df31d-18d8-466c-8225-e06ba7f05e96Ù$44b9ff95-ea3d-41f5-8098-445a263738a9Ù$ca726a9d-364d-48e2-8882-20ddbc85b664Ù$9d36934a-78cb-446b-b3db-1bbd88cf272dÙ$73e5b719-8b91-41d2-b83d-471d981b027fÙ$49e45202-b9ae-42ab-9575-a57edb626a20Ù$0fbbe455-79ce-44d6-b010-da0bb56adbb4Ù$638f99e6-1cdc-414c-9b67-fd626ec0be3eÙ$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$98f4d5e6-7569-457e-851d-713c572ae400²upstream_cells_map‚¨Explorer‘Ù$c2347999-5ade-420b-903f-30523b38eb0fAbstractFloatÙ$fbf7b108-dc68-4077-b63c-6f88161d2098„´precedence_heuristic §cell_idÙ$fbf7b108-dc68-4077-b63c-6f88161d2098´downstream_cells_map€²upstream_cells_mapƒ¤Type¬ConstantStep‘Ù$1ac5588c-3c32-436e-8b40-41715223fba7AbstractFloat´cell_execution_orderÜ¸Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100Ù$db30e6c0-36bb-4602-a257-5768f3833525Ù$31d8fba9-cd28-4b2b-ba46-70a068b9ecadÙ$1e4ac085-7b72-4bad-ad87-21635930a6f7Ù$083c721c-70dd-4ca3-8160-fc0b0531914fÙ$97c94391-397d-4cf2-88b1-3bea3af56ed3Ù$7950e06b-e8ce-4bd7-9681-ab7b66dfec69Ù$3e5a226e-ecdb-43fb-a40a-a262da0ae542Ù$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802Ù$14bd0549-747f-4513-809f-8bdb78027807Ù$55f89ca1-cd57-44e0-95e5-63a4be31418fÙ$3a798da5-c309-48f2-aab1-6602ded8a650Ù$33009d31-6d66-4aba-b44c-edd911c2f392Ù$c12dab6a-92f0-41b1-a6b7-c404c74d9a83Ù$87f5ec29-623f-4666-824d-fad8c6448072Ù$6034cc0f-cbae-4d2f-a43f-1bb738c00f0bÙ$79082409-3182-4e0b-9c8c-37a94543fee9Ù$965da91b-6a3f-456c-89ce-461c31e0fb7eÙ$464d43c0-cd59-49e6-88f6-12a767677418Ù$85b3dbc4-2e02-4924-b3e5-887a1a557a2bÙ$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfaÙ$21e56374-35e6-4488-b8da-15e383017c77Ù$2bce1b80-2133-40a0-9367-fc2d491f6245Ù$4be9e81b-c3de-4c79-97b5-b41c03e0f187Ù$e07a27c5-0c9a-4893-a1cf-cf565ab78761Ù$86350532-13f8-4035-bad8-f25f41c93163Ù$a84f5393-bd7e-433e-b3d9-e9e7cfa1a329Ù$78c45162-dc8e-4faf-b1a9-8c71be86dceeÙ$eed2a4f2-48b2-4684-846e-aa99bb6dafd9Ù$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5Ù$647ab36b-641e-4024-ad2d-40ff33be28f4Ù$276779a3-9332-46bd-b511-a33a2fea4b5fÙ$b26b3519-6ab4-4cc9-971d-de5e591cac86Ù$7748ab8a-d186-49a4-b6ab-d1bd9ea34990Ù$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2Ù$9b625fc0-89bd-4064-a379-225e6a940af7Ù$8584ece7-badc-486a-9a57-b60e77f92673Ù$f9e60b35-84d2-4b4b-8832-6b0f08152396Ù$32bff269-e893-4907-b589-7ba2ae1314bdÙ$6292f449-8720-41f1-84de-1865fb5fddbfÙ$70e40b75-e7d8-4009-af80-3bf4086a28dfÙ$b3ec4673-af63-4d4a-8314-fa7e594f8a37Ù$d4ce45ae-613e-41ee-b626-69b0dbcf6452Ù$cb93c588-3dfa-45f4-9d83-f2de26cb1ceaÙ$c695b7f9-76ca-419b-924d-8338a42c8188Ù$23b99305-c8d9-4129-85fb-a5e4aabc4a31Ù$8fac4109-2e0d-4366-9118-018221e0b910Ù$093f312b-d70d-4bf7-bd53-8a1c7b2bee31Ù$0de99ee5-d94d-4d07-8cef-a6f9caf5e742Ù$f88029d6-3fc2-4552-8441-5ef37ac42638Ù$b24a92fc-f6c6-44e4-9afc-fa4249e4ab83Ù$9d7782f5-b530-40d5-9f75-280d3a762216Ù$0f244fa0-7591-4478-b172-d9c1de51f6e1Ù$99945570-0b2a-432a-ae83-3a7abc39cac8Ù$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5Ù$24759e26-e670-4330-b6e4-b313620660f1Ù$ea6d7cad-47ad-4472-a9e9-1ee33c81058dÙ$51349e41-4696-4bd5-9bc1-cefbb82bea08Ù$b5a2df21-4525-4320-b8dd-aea5ecdab832Ù$649e3d20-e276-4f4b-aeb0-89150f180ef5Ù$1f9a98fd-ea29-415c-9f35-add34b513a34Ù$1c9b54cd-08dd-401e-9705-818741844e8dÙ$c2347999-5ade-420b-903f-30523b38eb0fÙ$f995d0af-50bc-4e33-9bbf-17a7ab06358aÙ$852df31d-18d8-466c-8225-e06ba7f05e96Ù$44b9ff95-ea3d-41f5-8098-445a263738a9Ù$ca726a9d-364d-48e2-8882-20ddbc85b664Ù$9d36934a-78cb-446b-b3db-1bbd88cf272dÙ$60b2079e-0efa-427e-93cf-7f4646fe202eÙ$3118e102-aeac-42d9-98fc-ca29f40be4cdÙ$8c0f06f7-2ed0-4f3a-ab4e-90ac142f0cd9Ù$a61c15eb-ed5f-4052-a3a3-3276940564a1Ù$47be3ae6-20f7-47d0-aae3-b67154afc1a8Ù$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65Ù$e50596ab-91db-42f0-a62c-77629a4e79c7Ù$6d647790-a029-4579-86fa-9c7294aa94eaÙ$0e06d9cf-b995-40b1-86ee-1c6a56c63bd0Ù$839861db-676f-4544-a802-0abb5d0049e1Ù$1ac5588c-3c32-436e-8b40-41715223fba7Ù$f00ab44e-0d84-40c7-aa23-358c77a013e3Ù$262952a7-280e-4af1-99a6-0899518484a2Ù$fbf7b108-dc68-4077-b63c-6f88161d2098Ù$8018edb9-853f-4a88-bef6-9535c2d774b3Ù$9093ed39-f6e2-4c31-9fef-466bdd197423Ù$e2597cc6-a6f6-4260-887b-c587cacd3bc8Ù$3b9bb9f0-ba9d-4320-935a-58912afe34b6Ù$0f6b4e2d-dc09-4e1c-834f-dd8aaa8743aeÙ$7a31d1c5-260b-42f4-b997-967150881e21Ù$5712b303-0aa3-4501-b1b5-020136d6e655Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9Ù$30aa1e1b-0b51-40c4-a093-ef92c3ad519aÙ$0e606680-dd65-444f-bc98-73de4abbcdd4Ù$4ebd4a5a-3bd1-48e2-b03e-a5a3b2ec18a1Ù$73e5b719-8b91-41d2-b83d-471d981b027fÙ$49e45202-b9ae-42ab-9575-a57edb626a20Ù$3a215c95-c595-4837-a842-1c1e1c6bfa3bÙ$a04da367-3f8d-422d-a443-4e3e666e30efÙ$68470b1d-3cc2-4cb1-8dc2-53227e6300e7Ù$d42a346d-6af3-45df-9de5-7043f46c3da3Ù$fb3381b5-10e3-4307-b76d-672245fac9e7Ù$ff6598fa-3366-416c-88a1-6bfcefeb1719Ù$d2ebd908-387d-4e40-bc00-61ce5f45ebddÙ$1e1f6d10-1b31-4e0a-96f7-23207e913154Ù$13f0adab-7660-49df-b26d-5f89cd73192bÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2eÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4Ù$c66f1676-aec1-489d-96ff-99d748dac0feÙ$8a04adab-e97e-4ac4-a85e-5eae93b1c37bÙ$69b560c1-98ad-4cbf-89d2-e0516299bc69Ù$d9265b98-cc3e-4a60-b16e-f54d9f78c9d3Ù$f33adafb-44b4-4e56-b3de-9717fe026288Ù$4982b489-ca15-4188-90e5-565c45f02e01Ù$672a91c0-aa77-4257-8c83-d857f47cab6cÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3eÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18Ù$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0baÙ$d53796f1-6fc4-4450-9d66-05ba752ff7ebÙ$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$50fbdc85-82f1-4c52-936b-84eb14951d71Ù$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9Ù$691aa77a-d6da-4fde-9024-c4195057179dÙ$aa7c76b1-ac2d-4df5-88ef-b499b839181eÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03Ù$30f05bd9-e939-4810-b710-edc7f5975921Ù$96566aad-6d5c-460c-a924-ae0bad5d8b2dÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0adÙ$09227386-1620-4093-b913-5786205aad13Ù$781fa565-398c-4c89-89f3-0595455afc85Ù$1004eb4b-1fed-4328-a08b-6f5d9dd5080bÙ$7678b06b-feef-4656-a801-33f630437bfbÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872Ù$181d7eef-24a0-4775-a535-8ef901b7e4ebÙ$34f65898-cbcc-4832-afac-0f7a284e7f0bÙ$d24bd737-9e09-441a-aa94-9279c80f566dÙ$74024d96-d0c7-43c8-8379-caf843cbe4b8Ù$46ce2b1f-02cf-4dae-bf02-f67543f38b91Ù$c61630b0-29c1-4183-90d9-57c999187b53Ù$c72d35fc-c8fd-450e-9b95-d12ece5c2291Ù$8f1c7b0d-121c-46bf-884d-729e3b593025Ù$a15e5d05-a238-440b-9a43-d830c6ea2f4dÙ$22fa2b71-a98f-4b87-9e0b-9d373cd8915fÙ$2447c4ea-7752-457c-80da-ac0dd72a64c1Ù$865610bb-ee82-4440-9f32-f00d0382783bÙ$400e0de1-0101-4531-928f-08ca155da40cÙ$fb698663-e9d6-4368-bd53-d88da5e6f2c4Ù$7c2015dd-a786-49f5-9fe9-9199335ebd09Ù$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$1cde4625-f8ed-4403-835c-95cc30206699Ù$7c562867-55d3-4b4f-950d-c8efc4a9ff32Ù$6ce00349-1cf0-4a80-bddd-c1d26b66d051Ù$8a0462e9-fd65-4ccc-b795-446cf9ae7392Ù$88e43fed-fcf3-4071-996a-63f63c3d49b4Ù$97f6221d-3289-4e54-a80d-26c5c81f2651Ù$bf3770ea-ee54-4296-ab33-340aea445670Ù$51b7a645-269a-418c-b6d8-39c01d0609f1Ù$d0111453-9a66-411d-9966-fc386d1bdcb7Ù$4a89cdd9-c20f-40e2-bc84-c3ea9cbf00e7Ù$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2Ù$98f4d5e6-7569-457e-851d-713c572ae400Ù$9bd99099-1dfa-477a-9896-3da94bcc0633Ù$3b88ec30-768b-44d0-88ee-b3ed989f22c3Ù$d59126d7-5af0-4d06-a57b-e115eec32388Ù$2ceadc6d-5522-43f6-a803-a17a47a6048aÙ$37446874-1c28-491b-b3cc-b4ad3282686eÙ$46478110-5ce5-4c72-bb3d-bcb5f516ffdcÙ$ecd8833d-441b-4673-b457-dc5109f575abÙ$9e644919-5833-46ff-8f6e-e755ba688329Ù$33d201ba-52f2-44f0-8bc6-3930ec77f62fÙ$f1466027-7f52-41df-a8ea-a0650981c9d4Ù$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8eÙ$0d8e4160-adf2-4b43-9914-942539339972Ù$cb2bf56d-5c1d-4265-9480-11523f776a78Ù$bc939df7-e457-496c-8977-5fbf9dfe3638Ù$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53Ù$b29624b0-be1d-4cc2-964d-0a050f4c1bedÙ$c6a8a9d2-1f29-4082-9fd2-d409cf6227baÙ$211d9390-df44-4902-9e13-ad6744b9b7ddÙ$5aa170d3-eb87-44dd-b4f4-3ac97476efd7Ù$6a00226c-cda1-49d8-92a8-514f2264469cÙ$7cab41cc-2a39-45de-879d-f7f6105238d6Ù$f78846a8-076a-4a87-b1d0-4f830e2bca27Ù$4afd6c2c-632e-4196-ab36-4d314ed9ec96Ù$36602c38-8b29-4158-b299-94015a333762Ù$c9c2aa04-15a2-446a-839c-bcf724b0f57a´last_hot_reload_timeË©shortpathÙ*Chapter_02_Multi-armed_Bandits_notebook.jl®process_status¥ready¤pathÙµ/home/runner/work/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Chapter-02/Chapter_02_Multi-armed_Bandits_notebook.jlpluto_version§v0.20.8®last_save_timeËAÚšMî#ªcell_orderÜ¸Ù$db30e6c0-36bb-4602-a257-5768f3833525Ù$31d8fba9-cd28-4b2b-ba46-70a068b9ecadÙ$1e4ac085-7b72-4bad-ad87-21635930a6f7Ù$083c721c-70dd-4ca3-8160-fc0b0531914fÙ$97c94391-397d-4cf2-88b1-3bea3af56ed3Ù$7950e06b-e8ce-4bd7-9681-ab7b66dfec69Ù$3e5a226e-ecdb-43fb-a40a-a262da0ae542Ù$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802Ù$14bd0549-747f-4513-809f-8bdb78027807Ù$55f89ca1-cd57-44e0-95e5-63a4be31418fÙ$3a798da5-c309-48f2-aab1-6602ded8a650Ù$33009d31-6d66-4aba-b44c-edd911c2f392Ù$c12dab6a-92f0-41b1-a6b7-c404c74d9a83Ù$6034cc0f-cbae-4d2f-a43f-1bb738c00f0bÙ$87f5ec29-623f-4666-824d-fad8c6448072Ù$79082409-3182-4e0b-9c8c-37a94543fee9Ù$965da91b-6a3f-456c-89ce-461c31e0fb7eÙ$464d43c0-cd59-49e6-88f6-12a767677418Ù$85b3dbc4-2e02-4924-b3e5-887a1a557a2bÙ$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfaÙ$21e56374-35e6-4488-b8da-15e383017c77Ù$2bce1b80-2133-40a0-9367-fc2d491f6245Ù$4be9e81b-c3de-4c79-97b5-b41c03e0f187Ù$e07a27c5-0c9a-4893-a1cf-cf565ab78761Ù$86350532-13f8-4035-bad8-f25f41c93163Ù$a84f5393-bd7e-433e-b3d9-e9e7cfa1a329Ù$78c45162-dc8e-4faf-b1a9-8c71be86dceeÙ$eed2a4f2-48b2-4684-846e-aa99bb6dafd9Ù$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5Ù$647ab36b-641e-4024-ad2d-40ff33be28f4Ù$34f65898-cbcc-4832-afac-0f7a284e7f0bÙ$276779a3-9332-46bd-b511-a33a2fea4b5fÙ$d24bd737-9e09-441a-aa94-9279c80f566dÙ$b26b3519-6ab4-4cc9-971d-de5e591cac86Ù$7748ab8a-d186-49a4-b6ab-d1bd9ea34990Ù$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2Ù$9b625fc0-89bd-4064-a379-225e6a940af7Ù$8584ece7-badc-486a-9a57-b60e77f92673Ù$f9e60b35-84d2-4b4b-8832-6b0f08152396Ù$32bff269-e893-4907-b589-7ba2ae1314bdÙ$6292f449-8720-41f1-84de-1865fb5fddbfÙ$70e40b75-e7d8-4009-af80-3bf4086a28dfÙ$b3ec4673-af63-4d4a-8314-fa7e594f8a37Ù$d4ce45ae-613e-41ee-b626-69b0dbcf6452Ù$cb93c588-3dfa-45f4-9d83-f2de26cb1ceaÙ$c695b7f9-76ca-419b-924d-8338a42c8188Ù$23b99305-c8d9-4129-85fb-a5e4aabc4a31Ù$8fac4109-2e0d-4366-9118-018221e0b910Ù$093f312b-d70d-4bf7-bd53-8a1c7b2bee31Ù$0de99ee5-d94d-4d07-8cef-a6f9caf5e742Ù$f88029d6-3fc2-4552-8441-5ef37ac42638Ù$b24a92fc-f6c6-44e4-9afc-fa4249e4ab83Ù$9d7782f5-b530-40d5-9f75-280d3a762216Ù$0f244fa0-7591-4478-b172-d9c1de51f6e1Ù$99945570-0b2a-432a-ae83-3a7abc39cac8Ù$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5Ù$24759e26-e670-4330-b6e4-b313620660f1Ù$ea6d7cad-47ad-4472-a9e9-1ee33c81058dÙ$51349e41-4696-4bd5-9bc1-cefbb82bea08Ù$f7519adc-7dfb-4030-86f0-7445699dd3dbÙ$50fbdc85-82f1-4c52-936b-84eb14951d71Ù$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9Ù$b5a2df21-4525-4320-b8dd-aea5ecdab832Ù$691aa77a-d6da-4fde-9024-c4195057179dÙ$649e3d20-e276-4f4b-aeb0-89150f180ef5Ù$1f9a98fd-ea29-415c-9f35-add34b513a34Ù$1c9b54cd-08dd-401e-9705-818741844e8dÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080bÙ$c2347999-5ade-420b-903f-30523b38eb0fÙ$f995d0af-50bc-4e33-9bbf-17a7ab06358aÙ$852df31d-18d8-466c-8225-e06ba7f05e96Ù$44b9ff95-ea3d-41f5-8098-445a263738a9Ù$ca726a9d-364d-48e2-8882-20ddbc85b664Ù$9d36934a-78cb-446b-b3db-1bbd88cf272dÙ$60b2079e-0efa-427e-93cf-7f4646fe202eÙ$3118e102-aeac-42d9-98fc-ca29f40be4cdÙ$8c0f06f7-2ed0-4f3a-ab4e-90ac142f0cd9Ù$a61c15eb-ed5f-4052-a3a3-3276940564a1Ù$47be3ae6-20f7-47d0-aae3-b67154afc1a8Ù$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65Ù$e50596ab-91db-42f0-a62c-77629a4e79c7Ù$6d647790-a029-4579-86fa-9c7294aa94eaÙ$0e06d9cf-b995-40b1-86ee-1c6a56c63bd0Ù$839861db-676f-4544-a802-0abb5d0049e1Ù$1ac5588c-3c32-436e-8b40-41715223fba7Ù$f00ab44e-0d84-40c7-aa23-358c77a013e3Ù$262952a7-280e-4af1-99a6-0899518484a2Ù$fbf7b108-dc68-4077-b63c-6f88161d2098Ù$8018edb9-853f-4a88-bef6-9535c2d774b3Ù$9093ed39-f6e2-4c31-9fef-466bdd197423Ù$e2597cc6-a6f6-4260-887b-c587cacd3bc8Ù$3b9bb9f0-ba9d-4320-935a-58912afe34b6Ù$0f6b4e2d-dc09-4e1c-834f-dd8aaa8743aeÙ$7a31d1c5-260b-42f4-b997-967150881e21Ù$5712b303-0aa3-4501-b1b5-020136d6e655Ù$f561b0a8-a086-4e1a-bc87-82c4205e89c9Ù$30aa1e1b-0b51-40c4-a093-ef92c3ad519aÙ$0e606680-dd65-444f-bc98-73de4abbcdd4Ù$4ebd4a5a-3bd1-48e2-b03e-a5a3b2ec18a1Ù$73e5b719-8b91-41d2-b83d-471d981b027fÙ$49e45202-b9ae-42ab-9575-a57edb626a20Ù$3a215c95-c595-4837-a842-1c1e1c6bfa3bÙ$a04da367-3f8d-422d-a443-4e3e666e30efÙ$68470b1d-3cc2-4cb1-8dc2-53227e6300e7Ù$d42a346d-6af3-45df-9de5-7043f46c3da3Ù$fb3381b5-10e3-4307-b76d-672245fac9e7Ù$ff6598fa-3366-416c-88a1-6bfcefeb1719Ù$d2ebd908-387d-4e40-bc00-61ce5f45ebddÙ$1e1f6d10-1b31-4e0a-96f7-23207e913154Ù$13f0adab-7660-49df-b26d-5f89cd73192bÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2eÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4Ù$c66f1676-aec1-489d-96ff-99d748dac0feÙ$8a04adab-e97e-4ac4-a85e-5eae93b1c37bÙ$69b560c1-98ad-4cbf-89d2-e0516299bc69Ù$d9265b98-cc3e-4a60-b16e-f54d9f78c9d3Ù$f33adafb-44b4-4e56-b3de-9717fe026288Ù$4982b489-ca15-4188-90e5-565c45f02e01Ù$672a91c0-aa77-4257-8c83-d857f47cab6cÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3eÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18Ù$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0baÙ$d53796f1-6fc4-4450-9d66-05ba752ff7ebÙ$aa7c76b1-ac2d-4df5-88ef-b499b839181eÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03Ù$30f05bd9-e939-4810-b710-edc7f5975921Ù$96566aad-6d5c-460c-a924-ae0bad5d8b2dÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0adÙ$09227386-1620-4093-b913-5786205aad13Ù$781fa565-398c-4c89-89f3-0595455afc85Ù$7678b06b-feef-4656-a801-33f630437bfbÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872Ù$181d7eef-24a0-4775-a535-8ef901b7e4ebÙ$74024d96-d0c7-43c8-8379-caf843cbe4b8Ù$46ce2b1f-02cf-4dae-bf02-f67543f38b91Ù$c61630b0-29c1-4183-90d9-57c999187b53Ù$c72d35fc-c8fd-450e-9b95-d12ece5c2291Ù$8f1c7b0d-121c-46bf-884d-729e3b593025Ù$a15e5d05-a238-440b-9a43-d830c6ea2f4dÙ$22fa2b71-a98f-4b87-9e0b-9d373cd8915fÙ$2447c4ea-7752-457c-80da-ac0dd72a64c1Ù$865610bb-ee82-4440-9f32-f00d0382783bÙ$400e0de1-0101-4531-928f-08ca155da40cÙ$fb698663-e9d6-4368-bd53-d88da5e6f2c4Ù$7c2015dd-a786-49f5-9fe9-9199335ebd09Ù$140f1e20-f86d-4a6f-9cff-99685e129e1cÙ$1cde4625-f8ed-4403-835c-95cc30206699Ù$7c562867-55d3-4b4f-950d-c8efc4a9ff32Ù$6ce00349-1cf0-4a80-bddd-c1d26b66d051Ù$8a0462e9-fd65-4ccc-b795-446cf9ae7392Ù$88e43fed-fcf3-4071-996a-63f63c3d49b4Ù$97f6221d-3289-4e54-a80d-26c5c81f2651Ù$bf3770ea-ee54-4296-ab33-340aea445670Ù$51b7a645-269a-418c-b6d8-39c01d0609f1Ù$d0111453-9a66-411d-9966-fc386d1bdcb7Ù$4a89cdd9-c20f-40e2-bc84-c3ea9cbf00e7Ù$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2Ù$98f4d5e6-7569-457e-851d-713c572ae400Ù$9bd99099-1dfa-477a-9896-3da94bcc0633Ù$3b88ec30-768b-44d0-88ee-b3ed989f22c3Ù$d59126d7-5af0-4d06-a57b-e115eec32388Ù$2ceadc6d-5522-43f6-a803-a17a47a6048aÙ$37446874-1c28-491b-b3cc-b4ad3282686eÙ$46478110-5ce5-4c72-bb3d-bcb5f516ffdcÙ$ecd8833d-441b-4673-b457-dc5109f575abÙ$9e644919-5833-46ff-8f6e-e755ba688329Ù$33d201ba-52f2-44f0-8bc6-3930ec77f62fÙ$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8eÙ$f1466027-7f52-41df-a8ea-a0650981c9d4Ù$0d8e4160-adf2-4b43-9914-942539339972Ù$cb2bf56d-5c1d-4265-9480-11523f776a78Ù$bc939df7-e457-496c-8977-5fbf9dfe3638Ù$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53Ù$c6a8a9d2-1f29-4082-9fd2-d409cf6227baÙ$b29624b0-be1d-4cc2-964d-0a050f4c1bedÙ$211d9390-df44-4902-9e13-ad6744b9b7ddÙ$5aa170d3-eb87-44dd-b4f4-3ac97476efd7Ù$6a00226c-cda1-49d8-92a8-514f2264469cÙ$7cab41cc-2a39-45de-879d-f7f6105238d6Ù$f78846a8-076a-4a87-b1d0-4f830e2bca27Ù$4afd6c2c-632e-4196-ab36-4d314ed9ec96Ù$36602c38-8b29-4158-b299-94015a333762Ù$1fb1a518-e5ec-4777-80bc-bb55e8172100Ù$c9c2aa04-15a2-446a-839c-bcf724b0f57a±published_objects€¥nbpkgŠ¯install_time_nsÎùl}Þ¬instantiatedÃ²installed_versionsŠªStatistics¦stdlib©StatsBase¦0.34.3Distributions¨0.25.113§PlutoUI¦0.7.60¨Latexify¦0.16.5¬PlutoProfile¥0.4.0¦Random¦stdlib°SpecialFunctions¥2.3.1¬LaTeXStrings¥1.3.1«PlutoPlotly¥0.3.9°terminal_outputsŒªStatisticsÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`DistributionsÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`§PlutoUIÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`¦RandomÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`¬PlutoProfileÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`¬LaTeXStringsÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`©StatsBaseÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`¨LatexifyÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`¤BaseÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`ªnbpkg_syncÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`°SpecialFunctionsÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`«PlutoPlotlyÚ@ [0m[1mResolving...[22m [90m===[39m [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Project.toml` [32m[1m No Changes[22m[39m to `/tmp/jl_zmbMTP/Manifest.toml` [0m[1mInstantiating...[22m [90m===[39m [0m[1mPrecompiling...[22m [90m===[39m [32m[1m Activating[22m[39m project at `/tmp/jl_zmbMTP`§enabledÃ·restart_recommended_msgÀ´restart_required_msgÀbusy_packages¶waiting_for_permissionÂÙ,waiting_for_permission_but_probably_disabledÂ«cell_inputsÞ¸Ù$781fa565-398c-4c89-89f3-0595455afc85„§cell_idÙ$781fa565-398c-4c89-89f3-0595455afc85¤codeÙëfunction update_estimator!(est::ActionValue{T, E, A}, a::Integer, r::T, step::Integer) where {T <: AbstractFloat, E <: Explorer{T}, A <: AverageMethod{T}} est.N[a] += one(T) est.Q[a] += updatecoef(est, a, step) * (r - est.Q[a]) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$37446874-1c28-491b-b3cc-b4ad3282686e„§cell_idÙ$37446874-1c28-491b-b3cc-b4ad3282686e¤codeÚòmd""" Recreation of Figure 2.6 for the non-stationary case over $(nonstationarysearchparams.steps) steps with a drift rate of Ïƒ = $(nonstationarysearchparams.Ïƒ). These parameters can be adjusted above, but by default this is performed over 200,000 steps with the accumulated reward only being measured on the final 100,000 steps. Unlike in the stationary case, the Ïµ-greedy method with Î±=0.1 for updating the Q values performs the best at a very small Ïµ value of $$2^{-7}.$$ The UCB method is the second best performer but requires a very large c value of 128 compared to ~1 for the stationary case in which it was the best performer. This UCB method also uses the sample average which is not ideal for a non-stationary distribution. That is one of the reasons why it was mentioned earlier in the chapter that it is difficult to adapt the UCB technique to the non-stationary problem. We can use the constant step size method but that doesn't help the fact that the variance estimates are wrong. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872„§cell_idÙ$8a52acf7-5d57-490f-8bd9-6e1e0e322872¤codeÚ$function average_runs_cum_reward(k, algorithm::Function; steps = 1000, n = 2000, offset::T = 0.0f0, make_bandit = create_bandit, kwargs...) where T <: AbstractFloat r_step = Atomic{T}(zero(T)) r_step_ideal = Atomic{T}(zero(T)) rdiff = Atomic{T}(zero(T)) rdiff2 = Atomic{T}(zero(T)) getvar(s, s2) = (s2 - (s*s/n))/(n-1) @threads for i in 1:n Random.seed!(i) qs = make_bandit(k, offset=offset) est = algorithm(k) Random.seed!(i) rewards = run_bandit(qs, est; steps = steps, saveall = false, kwargs...) atomic_add!(r_step, rewards[1]) x = abs(rewards[2] - rewards[1]) atomic_add!(rdiff, x) atomic_add!(rdiff2, x^2) atomic_add!(r_step_ideal, rewards[2]) end (means = (r_step[]/n, r_step_ideal[]/n), rstd = r_step[]/n - sqrt(getvar(rdiff[], rdiff2[]))) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$8fac4109-2e0d-4366-9118-018221e0b910„§cell_idÙ$8fac4109-2e0d-4366-9118-018221e0b910¤codeÚ²function figure_2_4(;k = 10, c = 2.0, Ïµ = 0.1) Random.seed!(1234) #ensure both techniques are using the same set of random bandits ucb_runs = average_simple_runs(k, 0.0, c = c) Random.seed!(1234) Ïµ_runs = average_simple_runs(k, 0.1) steps = 1:1000 t1 = scatter(x = steps, y = ucb_runs[1], name = "UCB c = $c") t2 = scatter(x = steps, y = Ïµ_runs[1], name = "Ïµ-greedy Ïµ = $Ïµ") t3 = scatter(x = steps, y = Ïµ_runs[2], name = "theoretical limit") PlutoPlotly.plot([t1, t2, t3], Layout(xaxis_title = "Step", yaxis_title = "Reward per Step Averaged Over Runs", title = "$k armed testbed average performance", legend = attr(orientation = "h", y = 1.1), hovermode = "x unified")) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$98f4d5e6-7569-457e-851d-713c572ae400„§cell_idÙ$98f4d5e6-7569-457e-851d-713c572ae400¤codeÚªfunction plot_nonstationary_param_search(;k = 10, steps = 1_000, kwargs...) algorithms = [ (p -> ActionValue(Qinit = 0.0f0, explorer = Ïµ_Greedy(p)), -10, -1), (p -> ActionValue(Qinit = 0.0f0, explorer = Ïµ_Greedy(p), update_average = ConstantStep()), -10, -1), (p -> GradientReward(Î±=p), -100, 100), (p -> GradientReward(Î±=p, update_average = ConstantStep()), -100, 100), (p -> ActionValue(Qinit = 0.0f0, explorer = UCB(p)), -100, 100), (p -> ActionValue(Qinit = 0.0f0, explorer = UCB(p), update_average = ConstantStep()), -100, 100), (p -> ActionValue(Qinit = p, explorer = Ïµ_Greedy(0.0), update_average = ConstantStep()), -10, 10), (p -> ActionValue(Qinit = 0.0f0, explorer = OptimalDistributionSample{Float32}(), update_average = ConstantStep(p)), -10, 10) ] names = [L"\epsilon\text{-greedy sample average}", L"\epsilon\text{-greedy constant step average } (\alpha = 0.1)", L"\text{gradient bandit sample average}", L"\text{gradient bandit constant step average}", "UCB sample average", "UCB constant step average", L"\text{greedy optimistic initialization } \alpha = 0.1", L"\text{Optimal Distribution}"] hovertemplates = [ "Ïµ = %{x:.2g}, reward = %{y:.3g} Ïµ-greedy Sample Average", "Ïµ = %{x:.2g}, reward = %{y:.3g} Ïµ-greedy Constant Step Average", "Î± = %{x:.2g}, reward = %{y:.3g} gradient bandit Sample Average", "Î± = %{x:.2g}, reward = %{y:.3g} gradient bandit Constant Step Average", "c = %{x:.2g}, reward = %{y:.3g} UCB Sample Average", "c = %{x:.2g}, reward = %{y:.3g} UCB Constant Step", "Q0 = %{x:.2g}, reward = %{y:.3g} greedy optimistic", "Î± = %{x:.2g}, reward = %{y:.3g} optimal distribution" ] results = [nonstationary_param_search(k, algo[1]; exmin = algo[2], exmax = algo[3], steps = steps) for algo in algorithms] extracts = [preparetrace(runlist) for runlist in results] idealx = reduce(vcat, a.x for a in extracts) idealy = reduce(vcat, a.ys[2] for a in extracts) nlist = sort(unique(reduce(vcat, a.nlist for a in extracts))) traces = reduce(vcat, [maketrace(extracts[i], names[i], hovertemplates[i], colors[i]) for i in eachindex(names)]) idealtrace = scatter(x = idealx, y = idealy, name = "ideal", hovertemplate = "ideal reward = %{y:.3g}", mode = "markers") # idealx = reduce(vcat, [results[i][1] for i in eachindex(names)]) # idealy = reduce(vcat, [results[i][3][2] for i in eachindex(names)]) # nlist = sort(unique(reduce(vcat, [results[i][2] for i in eachindex(names)]))) # traces = [scatter(x = results[i][1], y = results[i][3][1], name = names[i], hovertemplate = hovertemplates[i]) for i in eachindex(names)] # idealtrace = scatter(x = idealx, y = idealy, name = "ideal", hovertemplate = "ideal reward = %{y:.2g}", mode = "markers") Plot([idealtrace; traces], Layout(xaxis = attr(title = "Method Parameter (see hovertext)", type = "log", tickvals = sort(unique(idealx)), ticktext = print_power2.(nlist)), yaxis = attr(title = "Average Reward Final $(floor(Int64, steps / 2)) Steps"), legend = attr(orientation = "h", x = -0.1, y = -0.2), width = 700, height = 600)) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$4be9e81b-c3de-4c79-97b5-b41c03e0f187„§cell_idÙ$4be9e81b-c3de-4c79-97b5-b41c03e0f187¤codeÙ8visualize_bandit_dist(n = n_arms_vis, samples = 100_000)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$78c45162-dc8e-4faf-b1a9-8c71be86dcee„§cell_idÙ$78c45162-dc8e-4faf-b1a9-8c71be86dcee¤codeÚ®md""" > ### *Exercise 2.4* > If the step-size parameters, $$\alpha_n$$, are not constant, then the estimate $$Q_n$$ is a weighted average of previously received rewards with a weighting different from that given by (2.6). What is the weighting on each prior reward for the general case, analogous to (2.6), in terms of the sequence of step-size parameters? From (2.6): $$Q_{n+1} = Q_n + \alpha[R_n - Q_n]$$ so here we consider the case where $$\alpha$$ is not a constant but rather can have a unique value for each step n. $$\begin{flalign} Q_{n+1}&=Q_n + \alpha_n[R_n - Q_n]\\ &=\alpha_nR_n+(1-\alpha_n)Q_n\\ &=\alpha_nR_n+(1-\alpha_n)[\alpha_{n-1}R_{n-1}+(1-\alpha_{n-1})Q_{n-1}] \tag{using recursive formula from first step}\\ &=\alpha_nR_n+(1-\alpha_n)\alpha_{n-1}R_{n-1}+(1-\alpha_n)(1-\alpha_{n-1})Q_{n-1}\\ &=\alpha_nR_n+(1-\alpha_n)\alpha_{n-1}R_{n-1}+(1-\alpha_n)(1-\alpha_{n-1})[\alpha_{n-2}R_{n-2}+(1-\alpha_{n-2})Q_{n-2}]\\ &=\alpha_nR_n+(1-\alpha_n)\alpha_{n-1}R_{n-1}+(1-\alpha_n)(1-\alpha_{n-1})\alpha_{n-2}R_{n-2}+...\\ &=Q_1\prod_{i=1}^n \left( 1-\alpha_i \right)+\sum_{i=1}^{n} \left[ (R_i\alpha_i)\prod_{j=i+1}^n(1-\alpha_j) \right]\\ \end{flalign}$$ For example if $$\alpha_i=1/i$$ then the product in the first term is 0 and the formula becomes: $$\begin{flalign} Q_{n+1} &= \sum_{i=1}^{n} \left[ \frac{R_i}{i}\prod_{j=i+1}^n\frac{j-1}{j} \right]\\ &= \sum_{i=1}^{n} \left[ \frac{R_i}{i}\frac{i}{i+1}\frac{i+1}{i+2}...\frac{n-1}{n} \right]\\ &=\sum_{i=1}^{n} \frac{R_i}{n} \end{flalign}$$ from the expanded product we can see that all of the numerators and denominators cancel out leaving only $$\frac{R_i}{n}$$ which matches the sample-average as expected for this step-size. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$30f05bd9-e939-4810-b710-edc7f5975921„§cell_idÙ$30f05bd9-e939-4810-b710-edc7f5975921¤codeÙupdatecoef(est::GradientReward{T, GradientSample{T}, SampleAverage{T}}, a::Integer, step::Integer) where {T <: AbstractFloat} = one(T) / step¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$2bce1b80-2133-40a0-9367-fc2d491f6245„§cell_idÙ$2bce1b80-2133-40a0-9367-fc2d491f6245¤codeÙUmd""" Visualize $(@bind n_arms_vis NumberField(1:100, default = 10)) Armed Bandit """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$a15e5d05-a238-440b-9a43-d830c6ea2f4d„§cell_idÙ$a15e5d05-a238-440b-9a43-d830c6ea2f4d¤codeÚ.#parameter search that automatically searches to find a maximum starting from a power of -1 for base 2 function param_search(k, algorithm, f; base::T = 2.0f0, exmin = -10, exmax = 5, kwargs...) where T<:AbstractFloat @info "Starting parameter search with algorithm: $algorithm" exmean = round(Int64, (exmin + exmax) / 2) function makerun(ex) p = base^ex @info "Evaluating p = $base ^ $ex" return (param = p, ex = ex, rewards = f(k, algorithm(p); kwargs...)) end function step(count, incr, ex, maxreward, runlist) (ex < exmin) && return runlist (ex > exmax) && return step(0, -1, -2, maxreward, runlist) if (count >= 3) #in this case we've generated 3 points to the right or left of the maximum which means either we change direction and reset left of the maximum or terminate incr == -1 && return runlist return step(0, -1, -2, maxreward, runlist) else run = makerun(ex) #mean reward per step r = run.rewards.means[1] #if we have a new maximum reset the count to 0 otherwise increment by 1 if r > maxreward count = -1 maxreward = r end return step(count+1, incr, ex+incr, maxreward, vcat(runlist, run)) end end #initialize search starting from ex = -1 firstrun = makerun(-1) firstreward = firstrun.rewards.means[1] step(0, 1, 0, firstreward, [firstrun]) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$0e06d9cf-b995-40b1-86ee-1c6a56c63bd0„§cell_idÙ$0e06d9cf-b995-40b1-86ee-1c6a56c63bd0¤codeÙ:(::Type{T})(e::T) where T <: OptimalDistributionSample = e¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$1cde4625-f8ed-4403-835c-95cc30206699„§cell_idÙ$1cde4625-f8ed-4403-835c-95cc30206699¤codeÙ8const datapath = joinpath(@__DIR__, "parameter_studies")¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$6a00226c-cda1-49d8-92a8-514f2264469c„§cell_idÙ$6a00226c-cda1-49d8-92a8-514f2264469c¤codeÚ?md""" ### Ideas for Non-Stationary Problem In the non-stationary problem we can track each average with the constant step size method and this will be our estimate for the q values. Since the value is drifting over time we definitely cannot represent our knowledge of it with a probability distribution, however we can assume that the exponentially decaying average preserves a small enough amount of information that the samples over that period are roughly stationary. If we do that, then we can still use a normal distribution for each q value, but the standard deviation now takes a different form. For the sample average case we had: $q_{i} \sim N(\mu, \sigma)$ $N_i = \text{number of samples from action i}$ $\mu = \frac{1}{N_i}\sum_{j = 1}^N R_j$ $\sigma^2 = \frac{1}{N_i}$ For the constant step size case, our average can be represented as a weighted sum: $\mu = (1=\alpha)^N Q_1 + \sum_{i = j}^N \alpha(1-\alpha)^{N-i}R_j$ where $\alpha$ is the constant step size parameter and $Q_1$ is the initial value for the estimate. This weighted sum implies a different variance for the distribution: $\sigma^2 = \sum_j w_j^2$ where $w_j$ are the weights in the sum and there is still the assumption that each action produces iid rewards with unit variance. So what remains is to calculate what this squared sum of the weights is for this type of average. $\begin{flalign} \sigma^2 &= \sum_j w_j^2 \\ &= (1-\alpha)^{2N} + \sum_{j=1}^N \alpha^2(1-\alpha)^{2N-2j} \\ &= (1-\alpha)^{2N} + \alpha^2 \left [ 1 + (1-\alpha)^2 + \cdots + (1-\alpha)^{2N-2} \right ] \\ \end{flalign}$ If we take $a = (1-\alpha)^2$, then the expression in the brackets is just: $\sum_{j = 0}^{N-1} a^j = \frac{a^N - 1}{a - 1} = \frac{(1-\alpha)^{2N} - 1}{(1-\alpha)^2 - 1} = \frac{(1-\alpha)^{2N} - 1}{\alpha(\alpha - 2)}$ And our overall expression for the variance is: $\sigma^2 = (1-\alpha)^{2N} + \frac{\alpha((1-\alpha)^{2N} - 1)}{\alpha - 2}$ which depends on both $\alpha$ and $N$. The variance is plotted below compared to the variance for the sample mean with the option to change the $\alpha$ parameter. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$c61630b0-29c1-4183-90d9-57c999187b53„§cell_idÙ$c61630b0-29c1-4183-90d9-57c999187b53¤codeÙ„function print_power2(n) if abs(n) > 7 latexify("2^$n") elseif n < 0 latexify("1/$(2^-n)") else latexify("$(2^n)") end end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$0e606680-dd65-444f-bc98-73de4abbcdd4„§cell_idÙ$0e606680-dd65-444f-bc98-73de4abbcdd4¤codeÙ.(::Type{U})(a::U) where U <: SampleAverage = a¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$c695b7f9-76ca-419b-924d-8338a42c8188„§cell_idÙ$c695b7f9-76ca-419b-924d-8338a42c8188¤codeÚ Ômd""" $$Q_{n+1} = Q_n + \beta_n[R_n - Q_n]$$ where $$\beta_n \dot= \alpha / \bar{o}_n $$ and $$\bar{o}_n \dot= \bar{o}_{n-1}+\alpha(1-\bar{o}_{n-1})$$ $$\bar{o}_n = \bar{o}_{n-1} + \alpha(1-\bar{o}_{n-1})=\bar{o}_{n-1}(1-\alpha)+\alpha$$ We can expand $\bar{o}_n$ backwards to get an explicit formula. $$\begin{flalign} \bar{o}_n&=\bar{o}_{n-1}(1-\alpha)+\alpha\\ &=(\bar{o}_{n-2}(1-\alpha) + \alpha)(1-\alpha)+\alpha\\ &=\bar{o}_{n-2}(1-\alpha)^2 + \alpha((1-\alpha)+1)\\ &=(\bar{o}_{n-3}(1-\alpha)+\alpha)(1-\alpha)^2 + \alpha((1-\alpha)+1)\\ &=\bar{o}_{n-3}(1-\alpha)^3+\alpha((1-\alpha^2) + (1-\alpha)+1)\\ &\vdots \\ &=\bar{o}_0(1-\alpha)^n + \alpha\sum_{i=0}^{n-1}(1-\alpha)^i=\alpha\sum_{i=0}^{n-1}(1-\alpha)^i \end{flalign}$$ This sum has an explicit formula as can be seen by: $$\begin{flalign} S &= 1 + (1-\alpha) + (1-\alpha)^2 + \cdots + (1-\alpha)^{n-1} \\ S(1-\alpha) &= (1-\alpha)+\cdots+(1-\alpha)^n=S-1+(1-\alpha)^n \\ -S\alpha &=-1+(1-\alpha)^n \\ S&=\frac{1-(1-\alpha)^n}{\alpha} \\ \end{flalign}$$ Therefore, $$\bar{o}_n=\alpha\frac{1-(1-\alpha)^n}{\alpha}=1-(1-\alpha)^n$$, and since $$0<\alpha<1$$, then $$(1 - \alpha)^n \rightarrow 0 \text{ as } n \rightarrow \infty.$$ $$\beta_n=\frac{\alpha}{\bar{o}_n}=\frac{\alpha}{1-(1-\alpha)^n} \implies \beta_1=1$$ From exercise 2.4, we have the formula for $$Q_n$$ with a non-constant coefficient $$\alpha_n$$ which we can trivially replace here with $$\beta_n$$ $$Q_n=Q_1\prod_{i=1}^n \left( 1-\beta_i \right)+\sum_{i=1}^{n} \left[ R_i\beta_i\prod_{j=i+1}^n(1-\beta_j) \right]$$ Since $$\beta_1=1$$, the product associated with $$Q_1$$ will be 0. Since there is no dependency on the initial value of Q, we can say this formula for updating Q has *no initial bias*. If we then make the substitution $$\beta_n=\frac{\alpha}{1-(1-\alpha)^n}$$, we have $$\begin{flalign} Q_n&=\sum_{i=1}^{n} \left[ R_i\frac{\alpha}{1-(1-\alpha)^i}\prod_{j=i+1}^n \left( 1-\frac{\alpha}{1-(1-\alpha)^j} \right) \right] \\ &=\alpha\sum_{i=1}^n \left[ \frac{R_i}{1-(1-\alpha)^i}\prod_{j=i+1}^n \left( \frac{(1-\alpha)(1-(1-\alpha)^{j-1})}{1-(1-\alpha)^j} \right) \right] \\ &=\alpha\sum_{i=1}^n \left[ \frac{R_i(1-\alpha)^{n-i}}{1-(1-\alpha)^i}\prod_{j=i+1}^n \left( \frac{1-(1-\alpha)^{j-1}}{1-(1-\alpha)^j} \right) \right] \end{flalign}$$ Examining the product term on its own, we can see it simplifies. $$\prod_{j=i+1}^n \left( \frac{1-(1-\alpha)^{j-1}}{1-(1-\alpha)^j} \right)$$ $$\frac{1-(1-\alpha)^{i}}{1-(1-\alpha)^{i+1}}\frac{1-(1-\alpha)^{i+1}}{1-(1-\alpha)^{i+2}}\cdots\frac{1-(1-\alpha)^{n-1}}{1-(1-\alpha)^{n}}=\frac{1-(1-\alpha)^i}{1-(1-\alpha)^n} \text{ for iâ‰¤n}$$ Replacing this expression for the product in the expression for $$Q_n$$ we have: $$Q_n=\alpha\sum_{i=1}^n \left[ \frac{R_i(1-\alpha)^{n-i}}{1-(1-\alpha)^i}\frac{1-(1-\alpha)^i}{1-(1-\alpha)^n}\right]=\frac{\alpha}{1-(1-\alpha)^n}\sum_{i=1}^n R_i(1-\alpha)^{n-i}$$ If we expand this sum going backwards from $$i=n$$: $$Q_n=\frac{\alpha}{1-(1-\alpha)^n} \left[ R_n+R_{n-1}(1-\alpha)+R_{n-2}(1-\alpha)^2+\cdots+R_1(1-\alpha)^{n-1} \right]$$ The constant term starts off at $$1$$ for $$n=1$$ and approaches $$\alpha$$ in the limit of $$n \rightarrow \infty$$. If $$0<\alpha<1$$, then the coefficients in the sum section for $$R_i$$ decrease exponentially from 1 for $$i=n$$ to $$(1-\alpha)^{n-1}$$ for $$i=1.$$ So the average over rewards includes every reward back to $$R_1$$ like the simple average but the coefficients become exponentially smaller approaching 0 as $$n \rightarrow \infty$$. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$865610bb-ee82-4440-9f32-f00d0382783b„§cell_idÙ$865610bb-ee82-4440-9f32-f00d0382783b¤codeÙÙfunction run_or_load(varname::String, operation::Function) if !isfile("$varname.jld2") data = operation() jldsave("$varname.jld2"; data) else data = read(jldopen("$varname.jld2"), "data") end return data end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$50fbdc85-82f1-4c52-936b-84eb14951d71„§cell_idÙ$50fbdc85-82f1-4c52-936b-84eb14951d71¤codeÚ4function average_gradient_stationary_runs(k; steps = 1000, n = 2000, Î±=0.1, offset = 0.0, baseline = true) names = (:step_reward, :step_reward_ideal, :optimalstep, :cum_reward, :cum_reward_ideal, :optimalaction_pct) runs = Vector{Vector{Vector{Float32}}}(undef, n) qs = create_bandit(k, offset = offset) run1 = gradient_stationary_bandit_algorithm(qs, k, steps = steps, Î± = Î±, baseline = baseline) runs[1] = [Float32.(run1[name]) for name in names] @threads for i in 2:n qs = create_bandit(k, offset = offset) run = gradient_stationary_bandit_algorithm(qs, k, steps = steps, Î± = Î±, baseline = baseline) runs[i] = [Float32.(run[name]) for name in names] end for i in eachindex(names) for j in 2:n runs[1][i] .+= runs[j][i] end runs[1][i] ./= n end return runs[1] end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$2447c4ea-7752-457c-80da-ac0dd72a64c1„§cell_idÙ$2447c4ea-7752-457c-80da-ac0dd72a64c1¤codeÙ:save_data(varname, data) = jldsave("$varname.jld2"; data) ¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$b5a2df21-4525-4320-b8dd-aea5ecdab832„§cell_idÙ$b5a2df21-4525-4320-b8dd-aea5ecdab832¤codeÚ¼@bind params_2_5 confirm(PlutoUI.combine() do Child md""" ### Figure 2.5 Parameters Number of Actions: $(Child(:k, NumberField(1:100, default = 10))) Reward Offset: $(Child(:offset, NumberField(0.0:1.0:10.0, default = 4.0))) Step-Size Minimum With Baseline: $(Child(:Î±min1, NumberField(0.001:0.001:0.1, default = 0.025))) Step-Size Minimum Without Baseline: $(Child(:Î±min2, NumberField(0.001:0.0001:0.1, default = 0.0125))) """ end)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$c12dab6a-92f0-41b1-a6b7-c404c74d9a83„§cell_idÙ$c12dab6a-92f0-41b1-a6b7-c404c74d9a83¤codeÚfunction action_value_testbed_plot(; k = 10, Ïµ1 = 0.01, Ïµ2 = 0.1, steps = 1000) qs = create_bandit(k) Ïµ_list = [0.0, Ïµ1, Ïµ2] runs = [average_simple_runs(k, Ïµ; steps = steps) for Ïµ in Ïµ_list] labs = ["Ïµ=0 (greedy)", "Ïµ=$(Ïµ1)", "Ïµ=$(Ïµ2)"] fig22a = PlutoPlotly.plot([map(enumerate(runs)) do (i, a) scatter(y = a[1], name = labs[i]) end; scatter(y = runs[1][2], name = "Theoretical Limit")], Layout(xaxis_title = "Step", yaxis_title = "Reward Averaged Over Runs", legend = attr(orientation = "h", y = 1.2), height = 500)) fig22b = PlutoPlotly.plot(map(enumerate(runs)) do (i, a) scatter(y = a[3], name = labs[i]) end, Layout(xaxis_title="Step", yaxis_title = "% Runs Taking Optimal Action", height = 500, showlegend = false)) md""" ### Figure 2.2 $fig22a $fig22b Average performance of the $$\epsilon$$-greedy action-value methods on the $k-armed testbed. These data are averages over 2000 runs with different bandit problems. All methods use sample averages as their action-value estimates. """ end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$36602c38-8b29-4158-b299-94015a333762„§cell_idÙ$36602c38-8b29-4158-b299-94015a333762¤codeÙ%md""" # Dependencies and Settings """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2„§cell_idÙ$aa5acd7c-6a0b-454f-ab05-12a606dd9fc2¤codeÙxnonstationary_param_search(args...; kwargs...) = param_search(args..., average_nonstationary_runs_cum_reward; kwargs...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$68470b1d-3cc2-4cb1-8dc2-53227e6300e7„§cell_idÙ$68470b1d-3cc2-4cb1-8dc2-53227e6300e7¤codeÙU(::Type{Explorer{T}})(e::GradientSample) where T<:AbstractFloat = GradientSample{T}()¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$86350532-13f8-4035-bad8-f25f41c93163„§cell_idÙ$86350532-13f8-4035-bad8-f25f41c93163¤codeÙ1md""" ## 2.5 Tracking a Nonstationary Problem """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$88e43fed-fcf3-4071-996a-63f63c3d49b4„§cell_idÙ$88e43fed-fcf3-4071-996a-63f63c3d49b4¤codeÙ€md""" Number of Steps to Accumulate Reward: $(@bind stationary_numsteps confirm(NumberField(100:100:10000, default = 1000))) """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$f00ab44e-0d84-40c7-aa23-358c77a013e3„§cell_idÙ$f00ab44e-0d84-40c7-aa23-358c77a013e3¤codeÙ¢mutable struct UnbiasedConstantStep{T<:AbstractFloat}<:AverageMethod{T} Î±::T o::T UnbiasedConstantStep(Î±::T) where T<:AbstractFloat = new{T}(Î±, zero(T)) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0ad„§cell_idÙ$4191ff98-f4ab-4f18-a148-d3d3fff3d0ad¤codeÙïfunction updatecoef(est::BA, a::Integer, step::Integer) where {BA <: BanditAlgorithm{T, E, UnbiasedConstantStep{T}} where {T <: AbstractFloat, E <: Explorer{T}}} avg = est.update_average avg.o += avg.Î±*(one(T) - avg.o) avg.Î±/avg.o end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$46ce2b1f-02cf-4dae-bf02-f67543f38b91„§cell_idÙ$46ce2b1f-02cf-4dae-bf02-f67543f38b91¤code¾md""" ## Parameter Studies """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$211d9390-df44-4902-9e13-ad6744b9b7dd„§cell_idÙ$211d9390-df44-4902-9e13-ad6744b9b7dd¤codeÚfunction getdistribution(;Î¼s::Vector{T} = [1.0, 0.5, 0.0], ns = 2*ones(Int64, length(Î¼s)), samples = 1_000) where T <: AbstractFloat counts = zeros(T, length(Î¼s)) inds = findall(a -> a == 0, ns) if !isempty(inds) counts[inds] .= one(T) / length(inds) else values = zeros(T, length(Î¼s)) Ïƒs = one(T) ./ ns for i in 1:samples for i in eachindex(Î¼s) values[i] = T(rand(Normal(Î¼s[i], Ïƒs[i]))) end counts[argmax(values)] += one(T) end counts ./= sum(counts) end return counts end ;¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$6d647790-a029-4579-86fa-9c7294aa94ea„§cell_idÙ$6d647790-a029-4579-86fa-9c7294aa94ea¤codeÙGstruct OptimalDistributionSample{T <: AbstractFloat} <: Explorer{T} end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03„§cell_idÙ$62eb0650-96bc-4fd6-bfe0-bf05a4137a03¤codeÙupdatecoef(est::ActionValue{T, E, SampleAverage{T}}, a::Integer, step::Integer) where {T <: AbstractFloat, E <: Explorer{T}} = one(T) / est.N[a]¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$51349e41-4696-4bd5-9bc1-cefbb82bea08„§cell_idÙ$51349e41-4696-4bd5-9bc1-cefbb82bea08¤codeÙBsample_action(actions, Ï€_vec) = sample(actions, pweights(Ï€_vec))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4„§cell_idÙ$0fbbe455-79ce-44d6-b010-da0bb56adbb4¤codeÙî(::Type{ActionValue})(k::Integer; Qinit::T = 0.0, explorer::Explorer = Ïµ_Greedy(), update_average::AverageMethod = SampleAverage()) where T <: AbstractFloat = ActionValue(k, Qinit, Explorer{T}(explorer), AverageMethod{T}(update_average))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0ba„§cell_idÙ$6cfc6ec9-1c9c-4e90-ab3f-295ebf01b0ba¤codeÚQfunction sample_action(est::ActionValue{T, OptimalDistributionSample{T}, SampleAverage{T}}, explorer::OptimalDistributionSample{T}, i::Integer, actions::AbstractVector) where T <: AbstractFloat (Q, N) = (est.Q, est.N) inds = findall(a -> a == 0, N) if !isempty(inds) return rand(actions[inds]) else qmax = -T(Inf) amax = rand(eachindex(Q)) for i in eachindex(Q) qest = rand(Normal(Q[i], one(T)/N[i])) if qest > qmax qmax = qest amax = i end end return actions[amax] # w = weights(getdistribution(Î¼s = Q, ns = N, samples = 100)) # sample(actions, w) end end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$7678b06b-feef-4656-a801-33f630437bfb„§cell_idÙ$7678b06b-feef-4656-a801-33f630437bfb¤codeÚ©function average_runs(k, algorithm::Function; offset::Float32 = 0.0f0, steps = 1000, n = 2000, make_bandit = create_bandit, kwargs...) names = (:step_reward, :step_reward_ideal, :optimalstep, :cum_reward, :cum_reward_ideal, :optimalaction_pct) runs = Vector{Vector{Vector{Float32}}}(undef, n) function loopbody() qs = make_bandit(k, offset=offset) est = algorithm(k) run = run_bandit(qs, est; steps = steps, kwargs...) [Float32.(run[name]) for name in names] end runs[1] = loopbody() for i in 2:n runs[i] = loopbody() end for i in eachindex(names) for j in 2:n runs[1][i] .+= runs[j][i] end runs[1][i] ./= n end return runs[1] end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$3b88ec30-768b-44d0-88ee-b3ed989f22c3„§cell_idÙ$3b88ec30-768b-44d0-88ee-b3ed989f22c3¤codeÚ-@bind nonstationarysearchparams confirm(PlutoUI.combine() do Child md""" Number of Steps to Accumulate Reward (only measured on second half): $(Child(:steps, NumberField(1000:1000:1_000_000, default = 200_000))) Reward Drift Rate Per Step $$\sigma$$: $(Child(:Ïƒ, NumberField(0.001:0.001:0.1, default = 0.01))) By default, a plot will be loaded that matches this search criteria. Check the box below to recalculate the search and save a new plot whever the submit button is clicked. Recompute Parameter Search: $(Child(:remakeplot, CheckBox())) """ end)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$691aa77a-d6da-4fde-9024-c4195057179d„§cell_idÙ$691aa77a-d6da-4fde-9024-c4195057179d¤codeºfigure_2_5(;params_2_5...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$276779a3-9332-46bd-b511-a33a2fea4b5f„§cell_idÙ$276779a3-9332-46bd-b511-a33a2fea4b5f¤codeÚ@bind nonstationaryparams confirm(PlutoUI.combine() do Child md""" ### Figure Parameters for Exercise 2.5 Action Count: $(Child(:n, NumberField(1:100, default = 10))) Exploration Parameter Ïµ: $(Child(:Ïµ, NumberField(0.01:0.01:0.1, default = 0.1))) Constant Step-Size Parameter Î±: $(Child(:Î±, NumberField(0.01:0.01:0.1, default = 0.1))) Initial Reward Value: $(Child(:initR, NumberField(-10.0:0.1:10.0, default = 0.0))) Reward Drift Standard Deviation: $(Child(:Ïƒ, NumberField(0.001:0.001:0.1, default = 0.01))) """ end)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$b26b3519-6ab4-4cc9-971d-de5e591cac86„§cell_idÙ$b26b3519-6ab4-4cc9-971d-de5e591cac86¤codeÚ7md""" The sample average method fails to improve at finding the optimal action after about 4000 time steps. At this point in the learning process, it selects the optimal action about 45% of the time, while the constant step size averaging method continues to improve and approach the theoretical limit. It will take many more samples for the sample average method to learn the new rewards since each new sample is weighted progressively less. As the problem continues to change, the number of steps required to correct the estimates will grow infinitely large. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$d0111453-9a66-411d-9966-fc386d1bdcb7„§cell_idÙ$d0111453-9a66-411d-9966-fc386d1bdcb7¤codeÚamd""" > ### *Exercise 2.11 (programming)* > Make a figure analogous to Figure 2.6 for the nonstionary case outlined in Exercise 2.5. Include the constant-step-size Ïµ-greedy algorithm with Î±=0.1. Use runs of 200,000 steps and, as a performance measure for each algorithm and parameter setting, use the average reward over the last 100,000 steps. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$79082409-3182-4e0b-9c8c-37a94543fee9„§cell_idÙ$79082409-3182-4e0b-9c8c-37a94543fee9¤codeÙ5action_value_testbed_plot(;Ïµ_action_value_params...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802„§cell_idÙ$0b951e6e-4b97-4bb5-87d0-6be7f0fd4802¤codeÙ¸md""" ### Figure 2.1 Shows the reward distribution for each of the $ktest arms in the testbed. The mean value is marked with a dashed line for each. $(plot_bandit_testbed(ktest)) """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$9b625fc0-89bd-4064-a379-225e6a940af7„§cell_idÙ$9b625fc0-89bd-4064-a379-225e6a940af7¤codeÙWmd""" Number of Arms: $(@bind ktest_nonstationary NumberField(2:100, default = 10)) """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$1f9a98fd-ea29-415c-9f35-add34b513a34„§cell_idÙ$1f9a98fd-ea29-415c-9f35-add34b513a34¤codeÚmd""" > ### *Exercise 2.10* > Suppose you face a 2-armed bandit task whose true action values change randomly from time step to time step. Specifically, suppose that, for any time step, the true values of actions 1 and 2 are respectively 10 and 20 with probability 0.5 (case A), and 90 and 80 with probability 0.5 (case B). If you are not able to tell which case you face at any step, what is the best expected reward you can achieve and how should you behave to achieve it? Now suppose that on each step you are told whether you are facing case A or case B (although you still donâ€™t know the true action values). This is an associative search task. What is the best expected reward you can achieve in this task, and how should you behave to achieve it? When we do not know which case we are facing, we can calculate the expected reward for each action across all cases. $$E[R_1] = 0.5 \times 10 + 0.5 \times 90 = 50$$ $$E[R_2] = 0.5 \times 20 + 0.5 \times 80 = 50$$ Since the expected reward of each action is equal, the best we can do is pick randomly which will have an expected reward of 50. For the case in which we know if we are in case A or case B, we now can select the best action for each case which has a value of 20 (action 2) for case A and 90 (action 1) for case B. However, we have a 50% probability of facing each case so the best achievable expected reward is. $$E[R] = 20 \times 0.5 + 90 \times 0.5 = 55$$ To acheive this reward we could apply the action value estimate approach but separate our samples for case A and B. That way we would have 4 estimates representing the expected reward of each action in each case. We could perform any of the exploration strategies mentioned earlier such as Ïµ-greedy action selection but being careful to update the estimate for that case only. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5„§cell_idÙ$fbe3dc15-9171-4a7e-8eea-b8cc052c9ba5¤codeÚfunction nonstationary_algorithm(k::Integer, Ïµ::AbstractFloat; steps = 10000, Ïƒ = 0.01, Î± = 0.0, initR = 0.0) qs = initR .* ones(k) Q = initR .* ones(k) N = zeros(k) accum_reward = 0.0 step_reward = zeros(steps) accum_reward_ideal = 0.0 step_reward_ideal = zeros(steps) cum_reward_ideal = zeros(steps) cum_reward = zeros(steps) optimalcount = 0 optimalaction_pct = zeros(steps) optimalstep = fill(false, steps) actions = collect(1:k) for i = 1:steps shuffle!(actions) #so that ties are broken randomly with argmax a = if rand() < Ïµ rand(actions) else actions[argmax(Q[actions])] end optimalaction = argmax(qs) if a == optimalaction optimalcount += 1 optimalstep[i] = true end bandit(a) = sample_bandit(a, qs) step_reward[i] = bandit(a) step_reward_ideal[i] = bandit(optimalaction) accum_reward_ideal += step_reward_ideal[i] accum_reward += step_reward[i] cum_reward_ideal[i] = accum_reward_ideal cum_reward[i] = accum_reward optimalaction_pct[i] = optimalcount / i N[a] += 1.0 if Î± == 0.0 Q[a] += (1.0/N[a])*(step_reward[i] - Q[a]) else Q[a] += Î±*(step_reward[i] - Q[a]) end qs .+= randn(k) .*Ïƒ #update q values with random walk end return (;Q, step_reward, optimalstep, step_reward_ideal, cum_reward, cum_reward_ideal, optimalaction_pct) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53„§cell_idÙ$1aa7bc33-0ad6-41ab-8e2a-ba22cbfdab53¤codeÚ#@bind testvar PlutoUI.combine() do Child md""" N1: $(Child(:n1, Slider(1:1000, default = 100, show_value=true))) N2: $(Child(:n2, Slider(1:1000, default = 100, show_value=true))) Maximum Mean Difference : $(Child(:diff, Slider(0.0:0.1:10, default = 1.0, show_value = true))) """ end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$649e3d20-e276-4f4b-aeb0-89150f180ef5„§cell_idÙ$649e3d20-e276-4f4b-aeb0-89150f180ef5¤codeÙ6md""" ## 2.9 Associate Search (Contextual Bandits) """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$32bff269-e893-4907-b589-7ba2ae1314bd„§cell_idÙ$32bff269-e893-4907-b589-7ba2ae1314bd¤codeÚÎmd""" ## 2.6 Optmisitic Initial Values The averaging methods discussed above have some bias towards the initial value of the estimates. We can exploit this by initializing Q with a value much higher than we'd expect to receive as a reward from any action. That way every observed reward at first will be dissappointing thus encouraging the agent to try unvisited actions. See below an example whose performance can be observed under different conditions. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$7c562867-55d3-4b4f-950d-c8efc4a9ff32„§cell_idÙ$7c562867-55d3-4b4f-950d-c8efc4a9ff32¤codeÙôfunction loadplots() files = filter(f -> occursin(r"\.html$", f), readdir(datapath)) isempty(files) && return Dict{String, HTML{String}}() Dict(split(f, ".") |> first |> String => HTML(String(read(joinpath(datapath, f)))) for f in files) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8e„§cell_idÙ$a228c5ae-e2d0-40c8-9abc-527b8f6d2f8e¤code»plot_softmax(softmax_k, Î²)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$60b2079e-0efa-427e-93cf-7f4646fe202e„§cell_idÙ$60b2079e-0efa-427e-93cf-7f4646fe202e¤codeÙ7struct UCB{T <: AbstractFloat} <: Explorer{T} c::T end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfa„§cell_idÙ$8b8a9449-04b7-4901-9a2c-fbbdc33dfdfa¤codeÚfunction visualize_bandit_dist(;n = 10, samples = 100_000) maxdist = [maximum(randn(n)) for _ in 1:samples] (mn, mx) = extrema(maxdist) rval = LinRange(mn, mx, 100_00) Î” = rval[2] - rval[1] rpdf = fmax.(rval, n) rankdist = mapreduce(a -> sort(randn(n)), +, 1:samples) ./ samples expected_reward = sum(rval .* rpdf) * Î” # t1 = histogram(x = maxdist) t2 = scatter(x = rval, y = rpdf) p1 = plot(t2) p2 = bar(x = 1:n, y = rankdist) |> a -> PlutoPlotly.plot(a, Layout(xaxis_title = "Reward Rank", yaxis_title = "Mean Reward of Arm")) md""" #### Distribution of Best Action Reward for a Random $n Armed Bandit, Expected Value Estimate = $(round(expected_reward, sigdigits = 4)) $p1 #### Expected Value of Mean Reward for Arms Ranked from 1 to $n $p2 """ end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$c72d35fc-c8fd-450e-9b95-d12ece5c2291„§cell_idÙ$c72d35fc-c8fd-450e-9b95-d12ece5c2291¤codeÙÄfunction get_param_list(n1::Integer, n2::Integer; base::T = 2.0f0) where T<:AbstractFloat nlist = collect(n1:n2) plist = base .^nlist # namelist = print_power2.(nlist) return plist, nlist end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$97c94391-397d-4cf2-88b1-3bea3af56ed3„§cell_idÙ$97c94391-397d-4cf2-88b1-3bea3af56ed3¤codeÙ function create_bandit(k::Integer; offset::T = 0.0) where T<:AbstractFloat qs = randn(T, k) .+ offset #generate mean rewards for each arm of the bandit end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$839861db-676f-4544-a802-0abb5d0049e1„§cell_idÙ$839861db-676f-4544-a802-0abb5d0049e1¤codeÙ1abstract type AverageMethod{T<:AbstractFloat} end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$9d36934a-78cb-446b-b3db-1bbd88cf272d„§cell_idÙ$9d36934a-78cb-446b-b3db-1bbd88cf272d¤codeÙg#how to convert type of explorer when it is already the same (::Type{T})(e::T) where T <: Ïµ_Greedy = e¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$d24bd737-9e09-441a-aa94-9279c80f566d„§cell_idÙ$d24bd737-9e09-441a-aa94-9279c80f566d¤codeÙ$exercise2_5(;nonstationaryparams...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$69b560c1-98ad-4cbf-89d2-e0516299bc69„§cell_idÙ$69b560c1-98ad-4cbf-89d2-e0516299bc69¤codeÚfunction (::Type{GradientReward})(k::Integer; Î±::T=0.1, update_average::AverageMethod = SampleAverage()) where T <: AbstractFloat H = zeros(T, k) Ï€vec = ones(T, k) ./ k RÌ„ = zero(T) new_update = AverageMethod{T}(update_average) GradientReward(H, Ï€vec, Î±, RÌ„, new_update) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$24759e26-e670-4330-b6e4-b313620660f1„§cell_idÙ$24759e26-e670-4330-b6e4-b313620660f1¤codeÙYfunction calculate_Ï€vec(H::AbstractVector) Ï€vec = exp.(H) normalize_Ï€vec!(Ï€vec) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$22fa2b71-a98f-4b87-9e0b-9d373cd8915f„§cell_idÙ$22fa2b71-a98f-4b87-9e0b-9d373cd8915f¤codeÙgstationary_param_search(args...; kwargs...) = param_search(args..., average_runs_cum_reward; kwargs...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$f561b0a8-a086-4e1a-bc87-82c4205e89c9„§cell_idÙ$f561b0a8-a086-4e1a-bc87-82c4205e89c9¤codeÙ>struct SampleAverage{T<:AbstractFloat} <: AverageMethod{T} end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$85b3dbc4-2e02-4924-b3e5-887a1a557a2b„§cell_idÙ$85b3dbc4-2e02-4924-b3e5-887a1a557a2b¤codeÙe#pdf for the maximum of n iid standard normals fmax(x, n) = n*pdf(Normal(), x)*cdf(Normal(), x)^(n-1)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3e„§cell_idÙ$638f99e6-1cdc-414c-9b67-fd626ec0be3e¤codeÙéfunction sample_action(est::ActionValue, explorer::Ïµ_Greedy, i::Integer, actions::AbstractVector) shuffle!(actions) # Ïµ = explorer.Ïµ if rand() < explorer.Ïµ rand(actions) else actions[argmax(view(est.Q, actions))] end end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$cb93c588-3dfa-45f4-9d83-f2de26cb1cea„§cell_idÙ$cb93c588-3dfa-45f4-9d83-f2de26cb1cea¤codeÚâmd""" > ### *Exercise 2.7: Unbiased Constant-Step-Size Tick* > In most of this chapter we have used sample averages to estimate action values because sample averages do not produce the initial bias that constant step sizes do (see analysis leading to (2.6)). However, sample averages are not a completely satisfactory solution because they may perform poorly on nonstationary problems. Is it possible to avoid the bias of constant sample sizes while retaining their advantages on nonstationary problems? One way is to use a step size of $$\beta_n \dot= \alpha / \bar{o}_n,$$ >to process the nth reward for a particular action, where $$\alpha>0$$ is a conventional constant step size, and $$\bar{o}_n$$ is a trace of one that starts at 0: > $$\bar{o}_n \dot= \bar{o}_{n-1}+\alpha(1-\bar{o}_{n-1}), \text{ for } n \geq 0, \text{ with } \bar{o}_0 \dot= 0.$$ > Carry out an analysis like that in (2.6) to show that $$Q_n$$ is an exponential recency-weighted average *without initial bias*. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$5712b303-0aa3-4501-b1b5-020136d6e655„§cell_idÙ$5712b303-0aa3-4501-b1b5-020136d6e655¤codeÙ3(::Type{U})(a::U) where U<:UnbiasedConstantStep = a¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$ca726a9d-364d-48e2-8882-20ddbc85b664„§cell_idÙ$ca726a9d-364d-48e2-8882-20ddbc85b664¤codeÙP(::Type{Ïµ_Greedy{T}})(e::Ïµ_Greedy) where T<:AbstractFloat = Ïµ_Greedy(T(e.Ïµ))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$1e4ac085-7b72-4bad-ad87-21635930a6f7„§cell_idÙ$1e4ac085-7b72-4bad-ad87-21635930a6f7¤codeÚ¹md""" > ### *Exercise 2.1* > In Ïµ-greedy action selection, for the case of two actions and $$\epsilon = 0.5$$, what is the probability that the greedy action is selected? The greedy action could be selected in two cases, each of which has probability 0.5. For case 1 the greedy action is selected with probability 1. For case 2, we select an action randomly, so the probability of selecting the greedy action is $$\frac{\text{num greedy actions}}{\text{num total actions}}=0.5$$ Since both cases are independent, the probabilities can be summed after multiplying each by the probability of that case which is 0.5 for both. $$P(a = a_{greedy}) = 0.5 \times (1 + 0.5) = 0.5 + 0.25 = 0.75$$ """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$bf3770ea-ee54-4296-ab33-340aea445670„§cell_idÙ$bf3770ea-ee54-4296-ab33-340aea445670¤codeÙ‰md""" ### Figure 2.6 Parameter study of bandit algorithms on the 10-armed testbed for stationary normally distributed bandit rewards. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$db30e6c0-36bb-4602-a257-5768f3833525„§cell_idÙ$db30e6c0-36bb-4602-a257-5768f3833525¤codeÚmd""" # Chapter 2: Multi-armed Bandits Reinforcement learning is distinct from other types of learning in its use of training information to *evaluate* actions taken rather than simply being *instructed*. In supervised learning, correct actions would be given explicitely independent of the actual behavior. Evaluative feedback, on the other hand depends entirely on the action taken. In this chapter, we study evaluative feedback in a *nonassociative* setting, that is one in which the situation does not change. In doing so, we avoid must of the complexity of the full reinforcement learning problem while introducing a technique that will be important throughtout the rest of the book. ## 2.1: A *k*-armed Bandit Problem Consider a repeated choice among *k* different options. A numerical reward is chosen from a stationary probability distribution that depends only on the action selected. The objective is to maximize the accumulated reward over some time period, let's say 1000 action selections or *time steps*. The *bandit* described here can be specified by $\Pr \{ r \mid a \}$ for all $a$ where $\Pr$ is some probability distribution with a well defined mean value: $\mathbb{E}[r \vert a] = r_a \; \forall a$. We denote the action selected on time step *t* as $$A_t,$$ and the corresponding reward as $$R_t.$$ The value then of an arbitrary action $$a$$, denoted $$q_*(a),$$ is the expected reward given that $a$ is selected: $$q_*(a) \doteq \mathbb{E}[R_t \vert A_t=a]$$ If we know the expected values, then the problem is trivial, but we assume that we only have estimates of the values at a time step $a$ which we will denote $$Q_t(a).$$ At any given time step, the greedy action is the one with the highest value estimate. If we take non-greedy actions then we can improve our value estimate for other states. To solve the problem in general we must balance *exploiting* the action estimated to be the best with *exploring* the values of other candidate actions. What follows are various methods to balance these two choices. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$4982b489-ca15-4188-90e5-565c45f02e01„§cell_idÙ$4982b489-ca15-4188-90e5-565c45f02e01¤codeÙ]sample_action(est::GradientReward, i::Integer, actions) = sample(actions, weights(est.Ï€vec))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$6ce00349-1cf0-4a80-bddd-c1d26b66d051„§cell_idÙ$6ce00349-1cf0-4a80-bddd-c1d26b66d051¤code½const plotdict = loadplots();¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$b29624b0-be1d-4cc2-964d-0a050f4c1bed„§cell_idÙ$b29624b0-be1d-4cc2-964d-0a050f4c1bed¤codeÙNp(Î¼1, Î¼2, n1, n2) = 1 - 0.5*(1 + erf((-Î¼1+Î¼2)/sqrt(2*(1/n1^2 + 1/n2^2))));¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$400e0de1-0101-4531-928f-08ca155da40c„§cell_idÙ$400e0de1-0101-4531-928f-08ca155da40c¤codeÚl#makes vectors suitable for plotting the different lines function preparetrace(runlist) f(g) = [g(r) for r in runlist] plist = f(r -> r.param) nlist = f(r -> r.ex) inds = sortperm(plist) glist = [r -> r.rewards.means[1], r -> r.rewards.means[2], r -> r.rewards.rstd] yvecs = [f(g)[inds] for g in glist] (x = plist[inds], nlist = nlist[inds], ys = yvecs) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$8a04adab-e97e-4ac4-a85e-5eae93b1c37b„§cell_idÙ$8a04adab-e97e-4ac4-a85e-5eae93b1c37b¤codeÚ}mutable struct GradientReward{T<:AbstractFloat, E <: GradientSample{T}, A <: AverageMethod{T}} <: BanditAlgorithm{T, E, A} H::Vector{T} Ï€vec::Vector{T} Î±::T RÌ„::T update_average::A function GradientReward(H, Ï€vec, Î±::T, RÌ„, update_average::A) where {T <: AbstractFloat, A <: AverageMethod{T}} new{T, GradientSample{T}, A}(H, Ï€vec, Î±, RÌ„, update_average) end end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$3e5a226e-ecdb-43fb-a40a-a262da0ae542„§cell_idÙ$3e5a226e-ecdb-43fb-a40a-a262da0ae542¤codeÙ\md""" Number of Arms in Testbed $(@bind ktest confirm(NumberField(2:100, default = 10))) """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$7a31d1c5-260b-42f4-b997-967150881e21„§cell_idÙ$7a31d1c5-260b-42f4-b997-967150881e21¤codeÙo(::Type{UnbiasedConstantStep{T}})(a::UnbiasedConstantStep) where T<:AbstractFloat=UnbiasedConstantStep(T(a.Î±))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$8584ece7-badc-486a-9a57-b60e77f92673„§cell_idÙ$8584ece7-badc-486a-9a57-b60e77f92673¤codeÙ4nonstationary_bandit_statistics(ktest_nonstationary)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$e07a27c5-0c9a-4893-a1cf-cf565ab78761„§cell_idÙ$e07a27c5-0c9a-4893-a1cf-cf565ab78761¤codeÚ2md""" ## 2.4 Incremental Implementation The sample-average for the action-value estimate is defined as $$Q_n \dot = \frac{R_1+R_2+\cdots+R_{n-1}}{n-1}$$. To compute this we can maintain a record of every reward and every time we accumulate a new reward recompute the entire average. However that is inefficient in terms of computational and memory resources. It is possible to instead maintain a single value for the estimate at step n (and n itself) and update it incrementally every time we obtain a new reward. The update formula for $$Q_{n+1}$$ when we obtain a new reward sample is derived below: $$\begin{flalign} Q_{n+1} &= \frac{1}{n} \sum_{i=1}^n R_i \\ &= \frac{1}{n} \left ( R_n + \sum_{i=1}^{n-1} R_i \right ) \tag{separate final sum term}\\ &= \frac{1}{n} \left ( R_n + (n-1) \frac{1}{n-1} \sum_{i=1}^{n-1} R_i \right ) \tag{multiply by 1} \\ &= \frac{1}{n} \left ( R_n + (n-1) Q_n \right ) \tag{definition of Q} \\ &= \frac{1}{n} \left ( R_n + n Q_n - Q_n \right ) \\ &= Q_n + \frac{1}{n} \left [ R_n + Q_n \right ] \tag{2.3} \end{flalign}$$ The update rule (2.3) is of a form that occurs frequently whose general form is $$NewEstimate \leftarrow OldEstimate + StepSize \left [ Target - OldEstimate \right ] \tag{2.4}$$ The expression $[Target - OldEstimate]$ is an *error* in the estimate. It is reuced by taking a step toward the "Target." The target is presumed to indicate a desireable direction in which to move, though it may be noisy. In the case above, for example, the target is the nth reward. Notice that the step size parameter in this case is $$\frac{1}{n}$$ but in general it can be constant or depend on the step count and the action itself. In this case it is denoted $$\alpha_t (a)$$. The simple bandit algorithm that uses this incremental update rule is implemented above in section **2.3**. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$b24a92fc-f6c6-44e4-9afc-fa4249e4ab83„§cell_idÙ$b24a92fc-f6c6-44e4-9afc-fa4249e4ab83¤codeÚ2md""" ## 2.8 Gradient Bandit Algorithms As an alternative to estimating action values, we can attempt to learn a numerical *preference* for each action $a$ which we will denote $$H_t(a) \in \mathbf{R}.$$ This vector of preferences will be converted in a probability distribution using the *soft-max distribution*. $$\Pr\{A_t = a\} \doteq \frac{e^{H_t(a)}}{\sum_{b=1}^k e^{H_t(b)}} \doteq \pi_t(a) \tag{2.11}$$ $$\pi_t(a)$$ is the probability for this agent to select action $$a$$ at time $$t.$$ All action preferences are initialized at the same value. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$ea6d7cad-47ad-4472-a9e9-1ee33c81058d„§cell_idÙ$ea6d7cad-47ad-4472-a9e9-1ee33c81058d¤codeÙ‰function update_H!(a::Integer, H::AbstractVector, Ï€_vec::AbstractVector, Î±, R, RÌ„) v = Î±*(R - RÌ„) H .-= v .* Ï€_vec H[a] += v end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$74024d96-d0c7-43c8-8379-caf843cbe4b8„§cell_idÙ$74024d96-d0c7-43c8-8379-caf843cbe4b8¤codeÚ:average_nonstationary_runs_cum_reward(k, algorithm::Function; steps = 1_000, n = 2000, qinit::T=0.0f0) where T<:AbstractFloat = average_runs_cum_reward(k, algorithm; offset = qinit, steps = steps, n = n, make_bandit = (k; offset = qinit) -> ones(T, k) .* offset, Ïƒ = T(0.01), cumstart = floor(Int64, steps/2) + 1)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$1c9b54cd-08dd-401e-9705-818741844e8d„§cell_idÙ$1c9b54cd-08dd-401e-9705-818741844e8d¤codeÚîmd""" ## Code Refactoring Due to the variety of algorithms and parameters for the bandit, I have rewritten the test environment with types that represent the different algorithms. The run simulator will dispatch on the types to correctly simulate that method with its parameters. Some of the previous simluations and plots are generated again. Because of the style used, only one simulation function is needed with the flexibility to select any combination of techniques in the chapter. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$e2597cc6-a6f6-4260-887b-c587cacd3bc8„§cell_idÙ$e2597cc6-a6f6-4260-887b-c587cacd3bc8¤codeÙ-(::Type{U})(a::U) where U <: ConstantStep = a¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$47be3ae6-20f7-47d0-aae3-b67154afc1a8„§cell_idÙ$47be3ae6-20f7-47d0-aae3-b67154afc1a8¤codeÙ?(::Type{UCB{T}})(e::UCB) where {T<:AbstractFloat} = UCB(T(e.c))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$70e40b75-e7d8-4009-af80-3bf4086a28df„§cell_idÙ$70e40b75-e7d8-4009-af80-3bf4086a28df¤codeÚÄ@bind params_2_3 confirm(PlutoUI.combine() do Child md""" ### Figure 2.3 Parameters Default values match textbook Number of Actions: $(Child(:k, NumberField(1:100, default = 10))) Initial Optimistic Estimate: $(Child(:Qinit, NumberField(0.0:0.1:10.0, default = 5.0))) Exploration Parameter Ïµ: $(Child(:Ïµ, NumberField(0.01:0.01:1.0, default = 0.1))) Constant Step Size Î±: $(Child(:Î±, NumberField(0.01:0.01:1.0, default = 0.1))) """ end)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$cb2bf56d-5c1d-4265-9480-11523f776a78„§cell_idÙ$cb2bf56d-5c1d-4265-9480-11523f776a78¤codeÚÔmd""" This expression will look simpler if we treat the probabilities as variables themselves. $f(p_i) = \frac{\sum_i p_i q_i}{\sum_i p_i}$ $\frac{\partial{f(p_i)}}{\partial p_a} = \frac{q_a - \sum_i p_i q_i}{(\sum_i p_i)^2}$ Setting this equal to 0 for all the partials implies the following: $q_a = \sum_i p_i q_i \quad \forall \quad a$ Let's say we have 2 actions. Then we have 2 equations with two unknowns. $\begin{flalign} q_1 &= p_1 q_1 + p_2 q_2 \implies p_1 = \frac{q_1 - p_2 q_2}{q_1} \\ q_2 &= p_1 q_1 + p_2 q_2 \implies p_2 = \frac{q_2 - p_1 q_1}{q_2} \\ p_1 &= \frac{q_1 - q_2 + p_1 q_1}{q_1} \implies q_1 = q_2 \\ \therefore \\ q &= p_1 q + p_2 q \implies p_1 = p_2 \end{flalign}$ From the initial equations we see this necessitates all of the q's to be equal, and in this case the optimal solution is for the probabilities to also be equal. We can also simplify this case with two actions because $p_2 = 1-p_1$ so let's just call $p_1 = p$ and $p_2 = 1-p$. Then we are trying to maximize $(f(p) = pq_1 + (1-p)q_2$ and we can just take the derivative in terms of p and set that to 0. $q_1 - q_2 = 0$ which again gives us an unreasonable constraint on the q values. That is because the only solution with a finite maximum is one in which the probabilities are equal. In all other cases the extremum only occurs at one of the probabilities being infinite. We can fix this by sticking with one variable but forcing it to be a true probability. $p(x) = \frac{e^x}{e^x + 1}$ So now $f(x) = p(x)q_1 + (1-p(x))q_2 = p(x)q_1 + p(-x)q_2$ $\frac{d f(x)}{d x} = \frac{d p(x)}{d x} q_1 - \frac{d p(x)}{dx} q_2 = \frac{d p(x)}{d x}(q_1 - q_2)$ Now we can see there are two options for this derivative being 0. We first have the case where the q values are equal which causes the probabilities to be equal. Let's consider the case where $q_1 \neq q_2$. Now we must have: $\frac{d p(x)}{dx} = e^{-x}(1+e^{-x})^{-2} = 0$ Consider first the case of very large x so the exponential term is close to 0. In this case the limit as $x \rightarrow \infty$ approaches $\frac{0}{1}=0$. Now consider the second case of $x \rightarrow -\infty$. In this case the exponential term is much larger than 1 and the expression simplifies to $\frac{e^{-x}}{e^{-2x}} = e^x$. And the limit of $e^x$ as $x \rightarrow -\infty$ is 0. So we have shown that the x that solves this maximization can either trend towards positive or negative infinity which corresponds to the probability $p(x) = \{0, 1\}$. These two cases correspond to the case where $q_1 > q_2$ and $q_1 < q_2$ which can be shown by taking the second derivative. This same argument applies to the softmax as well in the limit such that the solution that maximizes the expected value is one in which the largest q value probability is 1 and all the rest are 0. So if we were to maximize the gradient fully after some set number of samples we'd always arrive at a distribution that only selects the greedy action. In this algorithm the only thing that controls whether we select other actions is the fact that at any given sample point we have yet to converge. This is not a very precise way to control exploration and it isn't surprising that simply adjusting the step size does not yield optimal results. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$f9e60b35-84d2-4b4b-8832-6b0f08152396„§cell_idÙ$f9e60b35-84d2-4b4b-8832-6b0f08152396¤codeÚomd""" Shows the reward mean for the arms at each ranking from 1 to $ktest_nonstationary. Each arm starts at 0 mean reward and is perturbed by a normal random variable with Ïƒ = 0.01 at each step. The rewards for an arm at a particular ranking seem to track the standard deviation of the overall distribution for the drift process. The functional form of this is $$\sqrt{n}$$ where $$n$$ is the number of steps so far. Depending on the number of arms, the multiplicative factor on the curve changes but for the 10 armed case, the second best arm seems to match the value for 1 standard deviation above the mean of 0. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$083c721c-70dd-4ca3-8160-fc0b0531914f„§cell_idÙ$083c721c-70dd-4ca3-8160-fc0b0531914f¤codeÚÚmd""" ## 2.3 The 10-armed Testbed To roughly assess the relative effectiveness of the greedy and $\epsilon$-greedy action-value methods, we compare them numerically on a suite of test problems. The tests are a set of 2000 randomly generated $k$-armed bandit problems with $k=10$. For each bandit problem, the action values, $q_*(a), \; a = 1,\dots,10,$ were selected according to a normal (Gaussian) distribution with a mean of 0 and variance 1. The probability distribution for the reward from each arm is then a gaussian with unit-variance and the appropriate mean corresponding to that arm. For any learning method, can we measure its performance and behavior as it improves with experience over 1000 time steps when applied to one of the bandit problems. This makes up one *run*. Experiments results are obtained by repeating this process for 2000 independent runs, each with a different bandit problem. The following code recreates the 10-armed Testbed from section 2.3 """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$49e45202-b9ae-42ab-9575-a57edb626a20„§cell_idÙ$49e45202-b9ae-42ab-9575-a57edb626a20¤codeÙA(::Type{Explorer{T}})(e::Ïµ_Greedy{T}) where T<:AbstractFloat = e¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$ff6598fa-3366-416c-88a1-6bfcefeb1719„§cell_idÙ$ff6598fa-3366-416c-88a1-6bfcefeb1719¤codeÙW(::Type{AverageMethod{T}})(a::ConstantStep) where T<:AbstractFloat = ConstantStep{T}(a)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$647ab36b-641e-4024-ad2d-40ff33be28f4„§cell_idÙ$647ab36b-641e-4024-ad2d-40ff33be28f4¤codeÚúfunction average_nonstationary_runs(k, Ïµ, Î±; n = 2000, kwargs...) names = (:step_reward, :optimalstep, :step_reward_ideal, :cum_reward, :cum_reward_ideal, :optimalaction_pct) run1 = nonstationary_algorithm(k, Ïµ; Î± = Î±, kwargs...) runs = Vector{Vector{Vector{Float32}}}(undef, n) runs[1] = [Float32.(run1[a]) for a in names] for i in 2:n runs[i] = deepcopy(runs[1]) end @threads for i in 2:n run = nonstationary_algorithm(k, Ïµ; Î± = Î±, kwargs...) for (j, a) in enumerate(names) runs[i][j] .= Float32.(run[a]) end end for j in eachindex(names) for i in 2:n runs[1][j] .+= runs[i][j] end runs[1][j] ./= n end return runs[1] # map(i -> mapreduce(a -> a[i], (a, b) -> a .+ b, runs)./n, eachindex(names)) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$96566aad-6d5c-460c-a924-ae0bad5d8b2d„§cell_idÙ$96566aad-6d5c-460c-a924-ae0bad5d8b2d¤codeÙ¨updatecoef(est::BA, a::Integer, step::Integer) where {BA <: BanditAlgorithm{T, E, ConstantStep{T}} where {T <: AbstractFloat, E <: Explorer{T}}} = est.update_average.Î±¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$852df31d-18d8-466c-8225-e06ba7f05e96„§cell_idÙ$852df31d-18d8-466c-8225-e06ba7f05e96¤codeÙY#creates additional constructor with default value (::Type{Ïµ_Greedy})() = Ïµ_Greedy(0.1)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$a84f5393-bd7e-433e-b3d9-e9e7cfa1a329„§cell_idÙ$a84f5393-bd7e-433e-b3d9-e9e7cfa1a329¤codeÚmd""" In the case of a non-stationary problem, the sample-average method is not ideal because it weights samples from the past equally to the present. We can change the incremental implementation of the average to weight more recent rewards higher than past ones. A constant step-size parameter is one way of doing this. In this case we will change the update rule (2.3) to: $$Q_{n+1} \doteq Q_n + \alpha [R_n - Q_n] \tag{2.5}$$ By writing this as an explicit sum over all rewards, one can observe that this update rule computes a weight average whose weights exponentially decay into the past. See a similar derivation in exercise 2.4. In order to guarantee that Q converges to the true expected value, the step size parameter must obey the following relationships: $$\sum_{n=1}^\infty \alpha_n(a) = \infty \quad \quad \text{and} \quad \quad \sum_{n=1}^\infty \alpha_n^2(a) < \infty \tag{2.7}$$ The first condition ensures steps are large enough to overcome initial conditions and the second condition insures steps are small enough to converge. These are both met by the sample average step size of $$\frac{1}{n}$$ for not by the constant step size. That is desireable in a non-stationary environment where there is no stable value to converge to in the first place. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$9bd99099-1dfa-477a-9896-3da94bcc0633„§cell_idÙ$9bd99099-1dfa-477a-9896-3da94bcc0633¤codeÙ“exercise_2_11(;kwargs...) = make_or_lookup_param_plot(;f = plot_nonstationary_param_search, basename = "nonstationary_parameter_search", kwargs...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$3b9bb9f0-ba9d-4320-935a-58912afe34b6„§cell_idÙ$3b9bb9f0-ba9d-4320-935a-58912afe34b6¤codeÙ\(::Type{ConstantStep{T}})(a::ConstantStep) where {T<:AbstractFloat} = ConstantStep{T}(a.Î±) ¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$14bd0549-747f-4513-809f-8bdb78027807„§cell_idÙ$14bd0549-747f-4513-809f-8bdb78027807¤codeÚUmd""" ### $$\epsilon - Greedy$$ Action Value Method The functions below implement the sample-average method for estimating the value of each action with the Ïµ-greedy method of action selection. Note that the `simple_algorithm` uses the incremental implementation of calculating the sample average which is described in section **2.4**. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2e„§cell_idÙ$45cc0a58-3534-4c67-bfd4-1c2b48d59a2e¤codeÚbfunction (::Type{ActionValue})(k::Integer, Qinit::T, explorer::Explorer, q_avg::AverageMethod) where T <: AbstractFloat #use the type of Qinit to initialize vectors and convert the other types if necessary N = zeros(T, k) Q = ones(T, k) .* Qinit new_e = Explorer{T}(explorer) new_avg = AverageMethod{T}(q_avg) ActionValue(N, Q, new_e, new_avg) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$1fb1a518-e5ec-4777-80bc-bb55e8172100„§cell_idÙ$1fb1a518-e5ec-4777-80bc-bb55e8172100¤codeÙ¬begin using Random, Base.Threads, PlutoPlotly, PlutoUI, PlutoProfile, Latexify, LaTeXStrings, SpecialFunctions, Distributions, Statistics, StatsBase TableOfContents() end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$1e1f6d10-1b31-4e0a-96f7-23207e913154„§cell_idÙ$1e1f6d10-1b31-4e0a-96f7-23207e913154¤codeÙ*abstract type BanditAlgorithm{T, E, A} end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$44b9ff95-ea3d-41f5-8098-445a263738a9„§cell_idÙ$44b9ff95-ea3d-41f5-8098-445a263738a9¤codeÙi#extends default value to other types (::Type{Ïµ_Greedy{T}})() where T<:AbstractFloat = Ïµ_Greedy(T(0.1))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$6034cc0f-cbae-4d2f-a43f-1bb738c00f0b„§cell_idÙ$6034cc0f-cbae-4d2f-a43f-1bb738c00f0b¤codeÚx@bind Ïµ_action_value_params confirm(PlutoUI.combine() do Child testbedreset md""" ### Testbed Params The $(Child(:k, NumberField(2:10, default = 10))) armed testbed with Ïµ-greedy 1 = $(Child(:Ïµ1, NumberField(0.01:0.01:0.1))) and Ïµ-greedy 2 = $(Child(:Ïµ2, NumberField(0.1:0.1:1.0))) over $(Child(:steps, NumberField(500:100:10000, default = 1000))) steps """ end)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$aa7c76b1-ac2d-4df5-88ef-b499b839181e„§cell_idÙ$aa7c76b1-ac2d-4df5-88ef-b499b839181e¤code¾md""" ### Update Functions """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$0f6b4e2d-dc09-4e1c-834f-dd8aaa8743ae„§cell_idÙ$0f6b4e2d-dc09-4e1c-834f-dd8aaa8743ae¤codeÙ<(::Type{UnbiasedConstantStep})() = UnbiasedConstantStep(0.1)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$f7519adc-7dfb-4030-86f0-7445699dd3db„§cell_idÙ$f7519adc-7dfb-4030-86f0-7445699dd3db¤codeÚKfunction gradient_stationary_bandit_algorithm(qs::Vector{Float64}, k::Integer; steps = 1000, Î± = 0.1, baseline = true) bandit(a) = sample_bandit(a, qs) H = zeros(k) Ï€vec = calculate_Ï€vec(H) RÌ„ = 0.0 accum_reward_ideal = 0.0 accum_reward = 0.0 cum_reward_ideal = zeros(steps) step_reward_ideal = zeros(steps) cum_reward = zeros(steps) step_reward = zeros(steps) bestaction = argmax(qs) optimalstep = fill(false, steps) optimalcount = 0 optimalaction_pct = zeros(steps) actions = collect(1:k) for i = 1:steps a = sample_action(actions, Ï€vec) if a == bestaction optimalstep[i] = true optimalcount += 1 end step_reward[i] = bandit(a) step_reward_ideal[i] = bandit(bestaction) accum_reward_ideal += step_reward_ideal[i] cum_reward_ideal[i] = accum_reward_ideal accum_reward += step_reward[i] cum_reward[i] = accum_reward optimalaction_pct[i] = optimalcount / i update_H!(a, H, Ï€vec, Î±, step_reward[i], RÌ„) #update RÌ„ with running average if baseline is true if baseline RÌ„ += (1.0/i)*(step_reward[i] - RÌ„) end #update Ï€_vec update_Ï€vec!(Ï€vec, H) end return (;step_reward, step_reward_ideal, cum_reward, cum_reward_ideal, optimalstep, optimalaction_pct) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$181d7eef-24a0-4775-a535-8ef901b7e4eb„§cell_idÙ$181d7eef-24a0-4775-a535-8ef901b7e4eb¤codeÙþaverage_nonstationary_runs(k, algorithm::Function; steps = 1_000, n = 2000, qinit::T=0.0f0) where T<:AbstractFloat = average_runs(k, algorithm; offset = qinit, steps = steps, n = n, make_bandit = (k; offset = qinit) -> ones(T, k) .* offset, Ïƒ = T(0.01))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$5aa170d3-eb87-44dd-b4f4-3ac97476efd7„§cell_idÙ$5aa170d3-eb87-44dd-b4f4-3ac97476efd7¤codeÚÒmd""" The above plot shows the probability of selecting hte first action when its mean reward estimate is a given value above or below the alternative. The number of samples for each can be controlled above. How easy is it to extend this to multiple actions? We would need to calculate the probability that a given action was best among all the alternatives: $P(a_i = a_{best}) = P(q_i > max(q_j \forall j \neq i))$ Consider the case of 3 actions. If we have a distribution for the mean value of each, then there could be six possible rankings excluding the cases where the estimates are equal: $q_1 > q_2 > q_3$ $q_1 > q_3 > q_2$ $q_2 > q_3 > q_1$ $q_2 > q_1 > q_3$ $q_3 > q_1 > q_2$ $q_3 > q_2 > q_1$ For the first ranking, we can consider two new normal random variable X = $q_2 - q_1$ and Y = $q_3 - q_2$. Now (X, Y) follows a multivariate distribution with means $\mu_2 - \mu_1$ and $\mu_3 - \mu_2$. The probability of this ordering can then be calculated by calculating the probability that both X The variances of X and Y are just equal to the sum of the individual variances for each pair of q's. However, since this is a multivariate distribution we also need the covariance. $\sigma(X, Y) = \sigma(q_2 - q_1, q_3 - q_2) = \sigma(q_2, q_3 - q_2) - \sigma(q_1, q_3 - q_2) = -\sigma^2(q_2)$ since all of the variables are independent. Now the probability follows simply from calculating the CDF of this distribution and evaluating it at (0, 0). This represents the probability that $q_1 > q_2$ and $q_2 > q_3$ simultaneously. We would then have to calculate this for each of the above orderings to get the true distribution of each q being maximum. In the case of even more actions this becomes extremely tedious although in the case of infinitely many actions it simplifies to an easier calculation in the limit. But we are concerned with cases in which the number of actions is quite small like 10. One alternative to calculating this distribution is to simply sample from it. That is very easy because we need only generate a sample from each q distribution and then examine it to see which action produced the highest value. If we performed this sampling many times and kept track of which q appears on top, we would indeed recover the true distribution. But even better than this is that each sample is itself an unbiased estimator for this probability. Therefore we need only generate a single sample from this distribution and use that to select our action. Since we are interacting with the bandit for many time steps we will eventually collect enough samples to have a good distribution. Using this method of action selection is shown under the *optimal distribution* tag in the parameter study and it seems to perform quite well without any parameter tuning. A further question could be which action selection is best for improving our accuracy of the distribution, but this requires more thought on how to formulate the problem. Intuitively we want to sample more around the part of the distribution where pairs of actions are hard to distinguish. We could rank all the pairs in terms of how indistinguished they are weighted by how likely they are to be the maximum and then sample from the action that appears the most. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b„§cell_idÙ$1004eb4b-1fed-4328-a08b-6f5d9dd5080b¤codeÚÊfunction run_bandit(qs::Vector{T}, algorithm::BanditAlgorithm{T}; steps = 1000, Î¼::T = zero(T), Ïƒ::T = zero(T), cumstart = 1, saveall = true) where T <: AbstractFloat #if saveall is false, then only saves the average cumulative reward per step, so it is faster #in this case the bandit is not stationary updateq = (Î¼ != 0) || (Ïƒ != 0) function qupdate!(qs) for i in eachindex(qs) qs[i] += (randn()*Ïƒ) + Î¼ end return nothing end #initialize values to keep track of actions = collect(eachindex(qs)) accum_reward_ideal = zero(T) accum_reward = zero(T) bestaction = argmax(qs) bandit(a) = sample_bandit(a, qs) if saveall cum_reward_ideal = zeros(T, steps) step_reward_ideal = zeros(T, steps) cum_reward = zeros(T, steps) step_reward = zeros(T, steps) optimalstep = fill(false, steps) optimalcount = 0 optimalaction_pct = zeros(T, steps) end for i = 1:steps a = sample_action(algorithm, i, actions) r = bandit(a) r_ideal = a == bestaction ? r : bandit(bestaction) if i >= cumstart accum_reward_ideal += r_ideal accum_reward += r end #update anything required before sampling the next action update_estimator!(algorithm, a, r, i) if saveall if a == bestaction optimalstep[i] = true optimalcount += 1 end step_reward[i] = r step_reward_ideal[i] = r_ideal cum_reward_ideal[i] = accum_reward_ideal cum_reward[i] = accum_reward optimalaction_pct[i] = optimalcount / i end #will only update qs in the non-stationary case and get a new bestaction if updateq qupdate!(qs) bestaction = argmax(qs) end end saveall && return (;step_reward, step_reward_ideal, cum_reward, cum_reward_ideal, optimalstep, optimalaction_pct) navg = steps - cumstart + 1 step_cum_reward = accum_reward / navg step_cum_reward_ideal = accum_reward_ideal / navg return (;step_cum_reward, step_cum_reward_ideal) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$7cab41cc-2a39-45de-879d-f7f6105238d6„§cell_idÙ$7cab41cc-2a39-45de-879d-f7f6105238d6¤codeÙÚ@bind wsm_params PlutoUI.combine() do Child md""" Constant Step Size Î±: $(Child(:Î±, NumberField(0.01:0.01:1.0, default = 0.1))) Max Samples: $(Child(:nmax, Slider(1:10000, default = 1000, show_value = true))) """ end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$e50596ab-91db-42f0-a62c-77629a4e79c7„§cell_idÙ$e50596ab-91db-42f0-a62c-77629a4e79c7¤codeÙ/(::Type{T})(e::T) where T <: GradientSample = e¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$fb3381b5-10e3-4307-b76d-672245fac9e7„§cell_idÙ$fb3381b5-10e3-4307-b76d-672245fac9e7¤codeÙX(::Type{AverageMethod{T}})(a::SampleAverage) where T<:AbstractFloat = SampleAverage{T}()¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$3118e102-aeac-42d9-98fc-ca29f40be4cd„§cell_idÙ$3118e102-aeac-42d9-98fc-ca29f40be4cd¤codeº(::Type{UCB})() = UCB(2.0)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$b3ec4673-af63-4d4a-8314-fa7e594f8a37„§cell_idÙ$b3ec4673-af63-4d4a-8314-fa7e594f8a37¤codeºfigure_2_3(;params_2_3...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$c2347999-5ade-420b-903f-30523b38eb0f„§cell_idÙ$c2347999-5ade-420b-903f-30523b38eb0f¤codeÙ.abstract type Explorer{T <: AbstractFloat} end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$1ac5588c-3c32-436e-8b40-41715223fba7„§cell_idÙ$1ac5588c-3c32-436e-8b40-41715223fba7¤codeÚstruct ConstantStep{T<:AbstractFloat} <: AverageMethod{T} Î±::T #ensures that if Î± exceeds 1 it gets closer to 2 without exceeding it function ConstantStep{T}(Î±::T) where T <: AbstractFloat if Î± >= 2 new{T}(2 - inv(Î±)) else new{T}(Î±) end end end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$34f65898-cbcc-4832-afac-0f7a284e7f0b„§cell_idÙ$34f65898-cbcc-4832-afac-0f7a284e7f0b¤codeÚßfunction exercise2_5(; n=10, Ïµ=0.1, Î±=0.1, kwargs...) sample_average_run = average_nonstationary_runs(n, Ïµ, 0.0; kwargs...) constant_step_update_run = average_nonstationary_runs(n, Ïµ, Î±; kwargs...) p1 = plot([scatter(y = sample_average_run[1], name = "Sample Average"), scatter(y = constant_step_update_run[1], name = L"\alpha = 0.1", hovertemplate = "%{y}Î±=0.1"), scatter(y=sample_average_run[3], name = "Theoretical Limit")], Layout(yaxis_title = "Reward Averaged Over Runs", xaxis_title = "Step", width = 700, height = 400, legend_orientation = "h", legend_y = 1.1, hovermode = "x unified")) p2 = plot([scatter(y = sample_average_run[2], name = "Sample Average"), scatter(y = constant_step_update_run[2], name = L"\alpha = 0.1", hovertemplate="%{y}Î±=0.1")], Layout(hovermode = "x", yaxis_title = "% Runs Taking Optimal Action", xaxis_title = "Step", width = 700, height = 400, legend_orientation = "h", legend_y = 1.1)) md""" $p1 $p2 """ end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$4ebd4a5a-3bd1-48e2-b03e-a5a3b2ec18a1„§cell_idÙ$4ebd4a5a-3bd1-48e2-b03e-a5a3b2ec18a1¤codeÙ=(::Type{U})(a::SampleAverage) where U <: SampleAverage = U() ¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$093f312b-d70d-4bf7-bd53-8a1c7b2bee31„§cell_idÙ$093f312b-d70d-4bf7-bd53-8a1c7b2bee31¤codeÚA@bind params_2_4 confirm(PlutoUI.combine() do Child md""" ### Figure 2.4 Parameters Number of Actions: $(Child(:k, NumberField(1:100, default = 10))) Confidence Bound c: $(Child(:c, NumberField(0.0:0.1:5.0, default = 2.0))) Exploration parameter Ïµ: $(Child(:Ïµ, NumberField(0.01:0.01:1.0, default = 0.1))) """ end)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$7748ab8a-d186-49a4-b6ab-d1bd9ea34990„§cell_idÙ$7748ab8a-d186-49a4-b6ab-d1bd9ea34990¤codeÙ-md""" ### Bandit Arm Reward Distributions """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$140f1e20-f86d-4a6f-9cff-99685e129e1c„§cell_idÙ$140f1e20-f86d-4a6f-9cff-99685e129e1c¤codeÚ›function plot_stationary_param_search(;k = 10, steps = 1000, kwargs...) algorithms = [ (p -> ActionValue(Qinit = 0.0f0, explorer = Ïµ_Greedy(p)), -100, -1), (p -> GradientReward(Î±=p), -100, 100), (p -> ActionValue(Qinit = 0.0f0, explorer = UCB(p)), -100, 100), (p -> ActionValue(Qinit = p, explorer = Ïµ_Greedy(0.0), update_average = ConstantStep()), -10, 10), (p -> ActionValue(Qinit = 0.0f0, explorer = OptimalDistributionSample{Float32}()), -10, 10) ] names = [L"\epsilon\text{-greedy }", L"\text{gradient bandit }", "UCB", L"\text{greedy optimistic initialization } \alpha = 0.1", L"\text{Optimal Distribution}"] hovertemplates = [ "Ïµ = %{x:.2g}, reward = %{y:.3g} Ïµ-greedy", "Î± = %{x:.2g}, reward = %{y:.3g} gradient bandit", "c = %{x:.2g}, reward = %{y:.3g} UCB", "Q0 = %{x:.2g}, reward = %{y:.3g} greedy optimistic", "Minimum Variance = %{x:.2g}, reward = %{y:.3g} Optimal Distribution Sample" ] results = [stationary_param_search(k, algo[1]; exmin = algo[2], exmax = algo[3], steps = steps, kwargs...) for algo in algorithms] extracts = [preparetrace(runlist) for runlist in results] idealx = reduce(vcat, a.x for a in extracts) idealy = reduce(vcat, a.ys[2] for a in extracts) nlist = sort(unique(reduce(vcat, a.nlist for a in extracts))) traces = reduce(vcat, [maketrace(extracts[i], names[i], hovertemplates[i], colors[i]) for i in eachindex(names)]) idealtrace = scatter(x = idealx, y = idealy, name = "ideal", hovertemplate = "ideal reward = %{y:.3g}", mode = "markers") Plot([idealtrace; traces], Layout(xaxis = attr(title = "Method Parameter (see hovertext)", type = "log", tickvals = sort(unique(idealx)), ticktext = print_power2.(nlist)), yaxis = attr(title = "Average Reward over first $steps steps"), legend = attr(orientation = "h", x = -.1, y = -.2), width = 700, height = 500)) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$965da91b-6a3f-456c-89ce-461c31e0fb7e„§cell_idÙ$965da91b-6a3f-456c-89ce-461c31e0fb7e¤codeÚ md""" > ### *Exercise 2.2:* *Bandit example* > Consider a k-armed bandit problem with k = 4 actions, denoted 1, 2, 3, and 4. Consider applying to this problem a bandit algorithm using $\epsilon$-greedy action selection, sample-average action-value estimates, and initial estimates of $$Q_1(a) = 0$$, for all a. Suppose the initial sequence of actions and rewards is $$A_1 = 1,$$ $$R_1 = âˆ’1,$$ $$A_2 = 2,$$ $$R_2 = 1,$$ $$A_3 = 2,$$ $$R_3 = âˆ’2,$$ $$A_4 = 2,$$ $$R_4 = 2,$$ $$A_5 = 3,$$ $$R_5 = 0.$$ On some of these time steps the $$\epsilon$$ case may have occurred, causing an action to be selected at random. On which time steps did this definitely occur? On which time steps could this possibly have occurred? The table below summarizes the actions taken leading into every step and the Q estimate for each action at the end of each step. So step 0 shows the initial Q estimates of 0 for every action and the selected action 1 that generates the reward on step 1. For the row in step 1 it shows the Q estimates after receiving the reward on step 1 and thus what actions are demanded by a greedy choice leading into the next step. If the action selected is not in the set of greedy actions, then a random action **must** have occured. Since a random action choice can also select one of the greedy actions, such a random choice is possible at every step. Note that the answer in row 0 corresponds to action $$A_1$$, row 1 -> $$A_2$$ etc... |Step| Action Selected | Reward | $$Q(1)$$ | $$Q(2)$$ | $$Q(3)$$ | $$Q(4)$$ | Greedy Action Set | Greedy Selection | $$\epsilon$$ Case | |----|---- |---- | ---- | ---- |---- | ---- | ---- | ---- | ---- | | 1 | 1| -1 | 0 | 0 | 0 | 0 | $$\{1, 2, 3, 4\}$$ | True | possibly | | 2 | 2 | 1 | -1 | 0 | 0 | 0 | $$\{2, 3, 4\}$$ | True | possibly | | 3 | 2 | -2 | -1 | 1 | 0 | 0 | $$\{2\}$$ | True | possibly | | 4 | 2 | 2 | -1 | $$-\frac{1}{2}$$ | 0 | 0 | $$\{3, 4\}$$ | False | definitely | | 5 | 3 | 0 | -1 | $$\frac{1}{3}$$ | 0 | 0 | $$\{2\}$$ | False | definitely | """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$9d7782f5-b530-40d5-9f75-280d3a762216„§cell_idÙ$9d7782f5-b530-40d5-9f75-280d3a762216¤codeÚ?md""" > ### *Exercise 2.9* > Show that in the case of two actions, the soft-max distribution is the same as that given by the logistic, or sigmoid, function often used in statistics and artificial neural networks. The sigmoid function is defined as: $$S(x) = \frac{1}{1 + e^{-x}}.$$ For two actions, let's denote them $$a_1$$ and $$a_2.$$ Now for the action probabilities we have. $$\pi(a_1) = \frac{e^{H_t(a_1)}}{e^{H_t(a_1)} + e^{H_t(a_2)}}=\frac{1}{1+e^{-(H_t(a_1) - H_t(a_2))}}$$ This expression for $$\pi(a_1)$$ is equivalent to $$S(x)$$ with $$x = H_t(a_1) - H_t(a_2)$$ which is the degree of preference for action 1 over action 2. As expected, if the preferences are equal then it is equavalent to $$x=0$$ with a probability of 50%. The same analysis applies to action 2 with the actions reversed from this case. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$bc939df7-e457-496c-8977-5fbf9dfe3638„§cell_idÙ$bc939df7-e457-496c-8977-5fbf9dfe3638¤codeÚímd""" ### Alternative Distribution Method The gradient bandit algorithm is motivated by maximizing an expected value which requires knowledge of the q values. However, the policy that maximizes that expected value will always be greedy with respect to whichever q value is thought to be maximal. In reality we do not have certainty over which q value is maximal and instead we have some knowledge of each q value based on samples collected. We know that the optimal choice only depends on the mean value of the rewards for each action. Our knowledge of the mean is from the reward samples collected, so we can use the distribution of the sample mean to represent our knowledge of the true q values. That would be a normal distribution with $\mu(a) = \sum_{A_t = a} R_t / N_{a}$ and $\sigma(a)^2 = \frac{\sum_{A_t = a} (R_t - \mu(a))^2}{N_a^2}$ If we seek to maximize the expected reward per step as before, all that matters for that is the mean value of each q or in our case the unbiased estimate we have of the mean value. This will also result in a greedy policy choosing which ever estimate is higher. We might explore actions but only in the case of the estimates fluctuating. If we ever consider a method that seeks to maximize the expected reward at a single step, we will always arrive at a greedy policy. One alternative we could consider is sampling actions based on the probability that what we've selected is optimal. In other words calculate $P(x = a_*) \quad \forall \quad x$ To simplify matters let's consider the case of only two actions and also assume that we already know the true variance of the reward for each action about the mean. That way our distribution for q simplifies to a normal distribution with a mean of $\mu(a) = \sum_{A_t = a}R_t / N_a$ and a variance of $\sigma(a) = 1/N_a$. Now we have two actions $1$ and $2$ each with their own samples. So our knowledge of $q_1$ and $q_2$ consists of two normal distributions: $N(\mu_1, \sigma_1^2)$ and $N(\mu_2, \sigma_2^2)$ with the mean and variance calculated as mentioned above. So what is the probability that action 1 is optimal? This is the same as asking the probability that a sample from the first distribution is larger than a sample from the second distribution. $P(X>Y)$ with $X \sim N(\mu_1, \sigma_1^2)$ and $Y \sim N(\mu_2, \sigma_2^2)$ This is equivalent to asking $P(X-Y) > 0$. Since $X-Y$ is a sum of normal distributions, it is itself a normal distribution: $N(\mu_1 - \mu_2, \sigma_1^2 + \sigma_2^2)$. The cumulative distribution function will answer this question since it is the probability of a random variable being less than or equal to a given value. So $1-CDF(0)$ will be our answer where $CDF(0)$ is the cummulative distribution function of our normal variable evaluated at 0: $\frac{1}{2}\left[1+\text{erf}\left(\frac{-\mu_1 + \mu_2}{\sqrt{2(\sigma_1^2 + \sigma_2^2)}} \right ) \right]$. So this would be the probability of selecting action 1 and 1 minus this would be the probability of selecting action 2. The assumptions made here are that the rewards are generated by a stationary process with some unknown mean and unit variance. This method is different from any discussed earlier because we have a stochastic policy yet it is based on estimating the q values. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$464d43c0-cd59-49e6-88f6-12a767677418„§cell_idÙ$464d43c0-cd59-49e6-88f6-12a767677418¤codeÚÌmd""" > ### *Exercise 2.3* > In the comparison shown in Figure 2.2, which method will perform best in the long run in terms of cumulative reward and probability of selecting the best action? How much better will it be? Express your answer quantitatively. In the long run both the $$\epsilon = 0.1$$ and $$\epsilon = 0.01$$ methods will have Q value estimates that converge to the true mean value of the reward distribution. Since both methods will necessarily take random actions 10% and 1% of the time respectively, we'd expect each method to take the optimal action with probability $$(1-\epsilon) + \epsilon \times \frac{1}{10}=\frac{10 - 9 \times \epsilon}{10}$$. So for each value of Ïµ we have. $$Pr(a=a_{best}|\epsilon = 0.1) = 0.91$$ $$Pr(a=a_{best}|\epsilon = 0.01) = 0.991$$ For the $$\epsilon = 0$$ greedy case the expected reward and optimal action selection probability depends on the order of sampled actions and the likelihood of getting close to or on the optimal action enough to push its Q estimation to the top. From the plots in figure 2.2 in practice that seems to lead to an average reward of ~1.05 and an optimal action selection probability of 0.3825. For long term cummulative reward this case will have roughly $$1.05 \times num\_steps$$. For the other two cases, the long term cummulative reward is based on the expected value of the highest reward mean which is approximately ~1.539 (see below for calculation of bandit arm reward distributions). For a random action the expected reward should be 0 due to the normal distribution of the action mean rewards. $$E(long\_term\_step\_reward|\epsilon=0.1) = 0.91 \times 1.539 = 1.4$$ $$E(long\_term\_step\_reward|\epsilon=0.01) = 0.991 \times 1.539 = 1.525$$ For each case the long run cumulative reward is just this long term expected reward per step times the number of steps. The statistical properties of the different arms of a generic bandit can be visualized below using the following function. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$21e56374-35e6-4488-b8da-15e383017c77„§cell_idÙ$21e56374-35e6-4488-b8da-15e383017c77¤codeÙ-md""" ### Bandit Arm Reward Distributions """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$f33adafb-44b4-4e56-b3de-9717fe026288„§cell_idÙ$f33adafb-44b4-4e56-b3de-9717fe026288¤codeÙ md""" ### Sampling Functions """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$7c2015dd-a786-49f5-9fe9-9199335ebd09„§cell_idÙ$7c2015dd-a786-49f5-9fe9-9199335ebd09¤codeÚ#prepare parameter scan trace with standard deviation function maketrace(prep, name, hovertemplate, color) hovertemplatelist = ["$hovertemplate, reward 1 std worse = $v" for v in prep.ys[3]] #mean line meantrace = scatter(x = prep.x, y = prep.ys[1], name = name, hovertemplate = hovertemplatelist, line_color = color, legendgroup = color) #dotted lines showing min and max extrematrace = scatter(x = prep.x, y = prep.ys[3], name = "", showlegend = false, legend = "legend2", line = attr(dash = "dot", width = 5, color = color), opacity = 0.25, hovertemplate = hovertemplatelist, legendgroup = color) vcat(meantrace, extrematrace) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$ecd8833d-441b-4673-b457-dc5109f575ab„§cell_idÙ$ecd8833d-441b-4673-b457-dc5109f575ab¤codeÚEmd""" ### Why softmax? For the gradient bandit algorithm we have a probability distribution over actions $\pi_t(x)$ and we seek to maximize the expected reward at each time step $\mathbb{E}[R_t]=\sum_x \pi_t(x) q_*(x)$. Since we do not know the true values of $q_*$ the algorithm simply replaces this with the sample reward collected at that time step. This is justifyable because $\mathbb{E}[R_t|A_t] = q_*(A_t)$. Using this method we only take one gradient step each sample so after a large number of steps we have sampled many rewards from each action and have better estimates for $q_*$. $\pi_t(x)$ is a discrete distribution with one value for each action. We can represent that simply with a parameter for each action, let's call it $H(x)$ we seek to find the values $H(x)$ which maximize the expected value of reward by setting the partial derivatives with respect to those parameters equal to 0. $\frac{\partial{\mathbb{E}[R_t]}}{\partial{H_t(x)}} = \frac{\partial}{\partial H_t(a)} \left ( \sum_x H_t(x) q_*(x) \right ) = \sum_x q_*(x)=0$ This is a problem because we do not have control over the $q_*$ values. So perhaps we should consider some other function of $H_t(x)$. In particular we know that the probabilities need to be positive so why not select a function that is always positive like $f(x) = x^2$. What expression for the derivative do we get in this case? $\frac{\partial{\mathbb{E}[R_t]}}{\partial{H_t(x)}} = \frac{\partial}{\partial H_t(a)} \left ( \sum_x H_t(x)^2 q_*(x) \right ) = \sum_x 2 H_t(x) q_*(x)=0$ Consider the case of only two actions: $2 H_t(1) q_*(1) + 2 H_t(2) q_*(2) = 0 \implies H_t(1) = -H_t(2)\frac{q_*(2)}{q_*(1)}.$ As a probability though, we know that $(H_t(1)^2 + H_t(2)^2 = 1)$, but substituting the expression for $H_t(1)$ we get $H_t(2)^2 \frac{q_*(2)}{q_*(1)} + H_t(2)^2 = 1 \implies H_t(2) = \sqrt{\frac{1}{1 + r}}$ where r is the ratio of q values. This is a problem though because the q values can be any real number including negative ones so the expression in the square root could be negative. So this form of the probability will not have solutions for certian q values given the contraints of probability. We could also consider a function that guarantees that our probabilities meet both contraints: $\sum_x p(x) = 1 \quad \text{and} \quad 0 \leq p(x) \leq 1 \forall x$ In order to do this let's take the H values, subtract the minimum, and by the sum of the resulting numbers. That would certainly meet the constraints except in the case where all the values are 0 which could be a problem. But even ignoring that fact, this procedure is not a differentiable function so we have no convenient way to maximize it. For all these cases it is clear that the only suitable function for the probabilities as a function of H is as follows: $f(H_t(x)) = \frac{u(H_t(x))}{\sum_x u(H_t(x))}$ where $u(a)$ is some function defined over the entire real number line such that $u(a) \geq 0 \forall a$. We already saw that $u(a) = a^2$ does not work because certain q values do not lead to valid solutions. Another option for such a function is $u(x) = e^x$ and this has the additional property of being monotonic. Using this form for $u(x)$ leads to the softmax. $p(x) = \frac{e^{H_t(x)}}{\sum_x e^{H_t(x)}}$ Using this form for the probability also means if we add some arbitrary constant to all the H values, it has no effect on the probability. $\frac{e^{c + H_t(x)}}{\sum_x e^{c + H_t(x)}} = \frac{e^c e^{H_t(x)}}{e^c \sum_x e^{H_t(x)}} = p(x)$ This property is very important considering the fact that we could arbitarily choose to center the H values around 0 or subtract the minimum to make them all positive. We should be able to eliminate the H value of 1 action by setting it to 0 and considering all the others in relation to that. Making such a choice should not change the answer of which action is the maximizing one. Also, it means if our maximization algorithm starts moving in the direction of a certain magnitude of H values, there will always be some solution that works if we fix any particular value so it will be much easier to find the maximum. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$d42a346d-6af3-45df-9de5-7043f46c3da3„§cell_idÙ$d42a346d-6af3-45df-9de5-7043f46c3da3¤codeÙk(::Type{Explorer{T}})(e::OptimalDistributionSample) where T<:AbstractFloat = OptimalDistributionSample{T}()¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$c66f1676-aec1-489d-96ff-99d748dac0fe„§cell_idÙ$c66f1676-aec1-489d-96ff-99d748dac0fe¤codeÙB(::Type{ActionValue})(;kwargs...) = k -> ActionValue(k; kwargs...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$d2ebd908-387d-4e40-bc00-61ce5f45ebdd„§cell_idÙ$d2ebd908-387d-4e40-bc00-61ce5f45ebdd¤codeÙg(::Type{AverageMethod{T}})(a::UnbiasedConstantStep) where T<:AbstractFloat = UnbiasedConstantStep{T}(a)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$97f6221d-3289-4e54-a80d-26c5c81f2651„§cell_idÙ$97f6221d-3289-4e54-a80d-26c5c81f2651¤codeÙúmd""" By default, a plot will be loaded that matches this search criteria. Check the box below to recalculate the search and save a new plot whever the submit button is clicked. Recompute Parameter Search: $(@bind execute_stationary CheckBox()) """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$8018edb9-853f-4a88-bef6-9535c2d774b3„§cell_idÙ$8018edb9-853f-4a88-bef6-9535c2d774b3¤codeÙL(::Type{ConstantStep})(Î±::T) where T <: AbstractFloat = ConstantStep{T}(Î±)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$672a91c0-aa77-4257-8c83-d857f47cab6c„§cell_idÙ$672a91c0-aa77-4257-8c83-d857f47cab6c¤codeÙ€sample_action(est::ActionValue, i::Integer, actions::AbstractVector) = sample_action(est::ActionValue, est.explorer, i, actions)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$a04da367-3f8d-422d-a443-4e3e666e30ef„§cell_idÙ$a04da367-3f8d-422d-a443-4e3e666e30ef¤codeÙ@(::Type{Explorer{T}})(e::UCB) where T<:AbstractFloat = UCB{T}(e)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$8f1c7b0d-121c-46bf-884d-729e3b593025„§cell_idÙ$8f1c7b0d-121c-46bf-884d-729e3b593025¤codeÚ#parameter search with fixed powers of 2 range function param_search(k, algorithm, n1, n2, f; base::T = 2.0f0, kwargs...) where T<:AbstractFloat plist, nlist = get_param_list(n1, n2, base = base) cum_rewards = Vector{T}(undef, length(plist)) cum_rewards_ideal = similar(cum_rewards) [(param = p, ex = n, rewards = f(k, algorithm(p); kwargs...)) for (p, n) in zip(plist, nlist)] end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$87f5ec29-623f-4666-824d-fad8c6448072„§cell_idÙ$87f5ec29-623f-4666-824d-fad8c6448072¤codeÙ"@bind testbedreset Button("reset")¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$30aa1e1b-0b51-40c4-a093-ef92c3ad519a„§cell_idÙ$30aa1e1b-0b51-40c4-a093-ef92c3ad519a¤codeÙ4(::Type{SampleAverage})() = SampleAverage{Float64}()¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$8a0462e9-fd65-4ccc-b795-446cf9ae7392„§cell_idÙ$8a0462e9-fd65-4ccc-b795-446cf9ae7392¤codeÚÖfunction make_or_lookup_param_plot(;f = plot_stationary_param_search, basename = "stationary_parameter_search", remakeplot = false, steps = 1000, kwargs...) corename = "$(basename)_$(steps)_steps" plotpath = joinpath(datapath, corename) # p = jldopen(jldpath, "a+") do f if !remakeplot && haskey(plotdict, corename) htmlplot = plotdict[corename] else io = IOBuffer() p = f(;steps = steps, kwargs...) PlotlyBase.to_html(io, p) htmlplot = io.data |> String |> HTML # delete!(jldfile, corename) # jldfile[corename] = p plotdict[corename] = htmlplot open("$plotpath.html", "w") do fplot PlotlyBase.to_html(fplot, p) end end # return p # end # return PlutoPlotly.plot(p) htmlplot end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$fb698663-e9d6-4368-bd53-d88da5e6f2c4„§cell_idÙ$fb698663-e9d6-4368-bd53-d88da5e6f2c4¤codeÙ?const colors = PlotlyBase.colors.cyclical[:tableau_colorblind];¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$13f0adab-7660-49df-b26d-5f89cd73192b„§cell_idÙ$13f0adab-7660-49df-b26d-5f89cd73192b¤codeÙ©struct ActionValue{T<:AbstractFloat, E <: Explorer{T}, A <: AverageMethod{T}} <: BanditAlgorithm{T, E, A} N::Vector{T} Q::Vector{T} explorer::E update_average::A end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$c6a8a9d2-1f29-4082-9fd2-d409cf6227ba„§cell_idÙ$c6a8a9d2-1f29-4082-9fd2-d409cf6227ba¤codeÙålet dvec = -testvar.diff:testvar.diff/20000:testvar.diff plot(dvec, [p(d, 0.0, testvar.n1, testvar.n2) for d in dvec], Layout(title = "Probability of Selecting First Action", xaxis_title = "Reward Mean Advantage Estimate")) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$eed2a4f2-48b2-4684-846e-aa99bb6dafd9„§cell_idÙ$eed2a4f2-48b2-4684-846e-aa99bb6dafd9¤codeÚÑmd""" > ### *Exercise 2.5 (programming)* > Design and conduct an experiment to demonstrate the difficulties that sample-average methods have for nonstationary problems. Use a modified version of the 10-armed testbed in which all the $$q_*(a)$$ start out equal and then take independent random walks (say by adding a normally distributed increment with mean 0 and standard deviation 0.01 to all the $$q_*(a)$$ on each step). Prepare plots like Figure 2.2 for an action-value method using sample averages, incrementally computed, and another action-value method using a constant step-size parameter, $$\alpha = 0.1$$. Use $$\epsilon = 0.1$$ and longer runs, say of 10,000 steps. See code and figures below for answer """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$8c0f06f7-2ed0-4f3a-ab4e-90ac142f0cd9„§cell_idÙ$8c0f06f7-2ed0-4f3a-ab4e-90ac142f0cd9¤codeÙ9(::Type{UCB{T}})() where T <: AbstractFloat = UCB(T(2.0))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$d53796f1-6fc4-4450-9d66-05ba752ff7eb„§cell_idÙ$d53796f1-6fc4-4450-9d66-05ba752ff7eb¤codeÚÀfunction sample_action(est::ActionValue{T, OptimalDistributionSample{T}, ConstantStep{T}}, explorer::OptimalDistributionSample{T}, i::Integer, actions::AbstractVector) where T <: AbstractFloat (Q, N) = (est.Q, est.N) Î± = est.update_average.Î± inds = findall(a -> a == 0, N) if !isempty(inds) return rand(actions[inds]) else qmax = -T(Inf) amax = rand(eachindex(Q)) for i in eachindex(Q) Î¼ = Q[i] n = N[i] ÏƒÂ² = (Î±*((1 - Î±)^(2*n) - 1)/(Î± - 2)) + (1-Î±)^(2*n) qest = rand(Normal(Q[i], sqrt(ÏƒÂ²))) if qest > qmax qmax = qest amax = i end end return actions[amax] # w = weights(getdistribution(Î¼s = Q, ns = N, samples = 100)) # sample(actions, w) end end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$0f244fa0-7591-4478-b172-d9c1de51f6e1„§cell_idÙ$0f244fa0-7591-4478-b172-d9c1de51f6e1¤codeÚ£md""" One natural update rule for the action preferences is to use stochastic gradient ascent. Using this technique we perform the following update on step $$t+1$$ after selecting action $$A_t$$ and receiving reward $$R_t$$ on step $$t$$. $$\begin{flalign} H_{t+1}(A_t) &\doteq H_t(A_t) + \alpha (R_t - \overline R_t)(1-\pi_t (A_t)) \\ H_{t+1}(a) & \doteq H_t(a) - \alpha(R_t - \overline R_t)\pi_t(a) \quad \forall a \neq A_t \end{flalign} \tag{2.12}$$ where $$\alpha > 0$$ is a step-size parameter and $$\overline R_t \in \mathbf{R}$$ is the average rewards up to but not including time $$t.$$ This average can be computed by any of the techniques mentioned earlier. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$262952a7-280e-4af1-99a6-0899518484a2„§cell_idÙ$262952a7-280e-4af1-99a6-0899518484a2¤codeÙ5(::Type{ConstantStep})() = ConstantStep{Float64}(0.1)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$46478110-5ce5-4c72-bb3d-bcb5f516ffdc„§cell_idÙ$46478110-5ce5-4c72-bb3d-bcb5f516ffdc¤codeÙ>md""" ## *Extra Notes on the Soft-Max and Gradient Bandit* """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$f1466027-7f52-41df-a8ea-a0650981c9d4„§cell_idÙ$f1466027-7f52-41df-a8ea-a0650981c9d4¤codeÚ`function plot_softmax(k, Î²) Random.seed!(1234) xs = randn(k) exs = exp.(Î² .*xs) ps = exs ./ sum(exs) md""" Original Values: $$h_i$$ $(plot(bar(x = 1:k, y = xs), Layout(height = 300))) Softmax: $$\frac{e^{(\beta h_i)}}{\sum_{i = 1} ^k e^{(\beta h_i)}}$$ $(plot(bar(x = 1:k, y = ps), Layout(xaxis_title = "Action (i)", height = 300))) """ end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$33009d31-6d66-4aba-b44c-edd911c2f392„§cell_idÙ$33009d31-6d66-4aba-b44c-edd911c2f392¤codeÚìfunction average_simple_runs(k, Ïµ; steps = 1000, n = 2000, Qinit = 0.0, Î±=0.0, c = 0.0) runs = Vector{Vector{Vector{Float32}}}(undef, n) outnames = (:step_reward, :step_reward_ideal, :optimalstep, :cum_reward, :cum_reward_ideal, :optimalaction_pct) qs = create_bandit(k) run1 = simple_algorithm(qs, k, Ïµ, steps = steps, Qinit = Qinit, Î± = Î±, c = c) out = [Float32.(run1[name]) for name in outnames] runs[1] = out @threads for i in 2:n runs[i] = deepcopy(out) qs = create_bandit(k) run = simple_algorithm(qs, k, Ïµ; steps = steps, Qinit = Qinit, Î± = Î±, c = c) # cum_reward_ideal = run1.cum_reward_ideal, # step_reward_ideal = run1.step_reward_ideal, # cum_reward = run1.cum_reward, # step_reward = run1.step_reward, # optimalstep = run1.optimalstep, optimalaction_pct = run1.optimalaction_pct) for j in eachindex(outnames) # out[i] .+= Float32.(run[outnames[i]]) runs[i][j] .= Float32.(run[outnames[j]]) end end for i in eachindex(out) for j in 2:n out[i] .+= runs[j][i] end out[i] ./= n end # map(i -> mapreduce(a -> a[i], (a, b) -> a .+ b, runs)./n, (:step_reward, :step_reward_ideal, :optimalstep, :cum_reward, :cum_reward_ideal, :optimalaction_pct)) return out end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$31d8fba9-cd28-4b2b-ba46-70a068b9ecad„§cell_idÙ$31d8fba9-cd28-4b2b-ba46-70a068b9ecad¤codeÚ0$$ controls the degree of exploration. This idea is called *upper confidence bound* (UCB) action selection. Note that is $$N_t(a)=0$$ then that action will be selected or a random selection will be made among all actions with zero counts. See below for a comparison between the Ïµ-greedy exploration method and the UCB method with the ability to change parameters for both methods. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$4a89cdd9-c20f-40e2-bc84-c3ea9cbf00e7„§cell_idÙ$4a89cdd9-c20f-40e2-bc84-c3ea9cbf00e7¤codeÙ¶md""" Below are functions which mimic those for the stationary bandit parameter search. The final function produces a plot and saves it to disk or loads one that already exists. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$3a215c95-c595-4837-a842-1c1e1c6bfa3b„§cell_idÙ$3a215c95-c595-4837-a842-1c1e1c6bfa3b¤codeÙ;(::Type{Explorer{T}})(e::UCB{T}) where T<:AbstractFloat = e¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9„§cell_idÙ$8a6f3f85-64e4-4c31-9e69-43f50f42bbc9¤codeÚÊfunction figure_2_5(;k = 10, offset = 4.0, Î±min1 = 0.025, Î±min2 = 0.025, Î±_list = [0.025, 0.05, 0.1, 0.2, 0.4]) ylabel = "% Runs Taking Optimal Action" steps = 1:1000 function make_Î±list(Î±::T, list = Vector{T}()) where T <: AbstractFloat Î± > 0.4 && return list make_Î±list(2*Î±, push!(list, Î±)) end function make_plot(baseline::Bool, Î±_list) results = [average_gradient_stationary_runs(k, Î±=Î±, offset = 4.0, baseline = baseline) for Î± in Î±_list] traces = [scatter(x = steps, y = a[3], name = "Î± = $(Î±_list[i])") for (i, a) in enumerate(results)] PlutoPlotly.plot(traces, Layout(xaxis_title = "Step", yaxis_title = ylabel, hovermode = "x unified", title = "Gradient Bandit $(baseline ? "With" : "Without") Baseline")) end md""" $(make_plot(true, make_Î±list(Î±min1))) $(make_plot(false, make_Î±list(Î±min2))) Average performance of the gradient bandit algorithm with and without a reward baseline on the $k-armed bandit testbed when the $$q_*(a)$$ are chosen to be near $offset rather than near 0. For the gradient bandit with a baseline, the offset doesn't affect the curves at all, but if the baseline is removed then the results are worse as seen in the second plot. However, if $$\alpha$$ is made smaller it seems like it will also converge to a similar success rate just over a longer time. The optimal value of $$\alpha$$ is much lower than when the baseline is removed which is consistent with slower convergence properties. """ end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$55f89ca1-cd57-44e0-95e5-63a4be31418f„§cell_idÙ$55f89ca1-cd57-44e0-95e5-63a4be31418f¤codeÙ’function sample_bandit(a::Integer, qs::Vector{T}) where T<:AbstractFloat randn(T) + qs[a] #generate a reward with mean q[a] and variance 1 end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$d59126d7-5af0-4d06-a57b-e115eec32388„§cell_idÙ$d59126d7-5af0-4d06-a57b-e115eec32388¤codeÙ3md""" ### Non-stationary Parameter Study Figure """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65„§cell_idÙ$4c6ccbfe-a3ce-4f2d-bcb1-7f1a4b735c65¤codeÙ ### *Exercise 2.8: UCB Spikes* > In Figure 2.4 the UCB algorithm shows a distinct spike in performance on the 11th step. Why is this? Note that for your answer to be fully satisfactory it must explain both why the reward increases on the 11th step and why it decreases on the subsequent steps. Hint: If $$c=1,$$ then the spike is less prominent. By definition, actions with zero visits are always considered maximizing. Therefore, for the first 10 steps, all 10 unique actions will be sampled once with each Q estimate updating from a single sample. On the 11th step, the exploration incentive for each action will be equal, so the action with the highest Q estimate will be selected. This is most likely to be the action with the highest $$q^*$$ value but there is a substantial probability it is the second best action and diminishing probabilities for the remaining actions ranked by true $$q^*$$. It is on this step though that we expect the selection to be substantially better than random chance although it is only using a single sample to validate the estimates. On step 12, that improved action will now have a visit count of 2 instead of 1 for every other action. In the calculation, the exploration bonus for that action will be $$c\sqrt{\frac{\ln{12}}{2}}\approx 1.11465 \times c.$$ Every other action will have an exploration bonus of $$\approx 1.576 \times c.$$ In order for the 2 visit action to be considered maximizing after this it must have a Q estimate that is $$\approx 0.4617 \times c$$ greater than any other action value estimate. In particular for $$c = 2.0,$$ the estimate must be $$\approx 0.9234$$ greater than the others. Since the q's are normally distributed, the difference in expected value between the best and second best action is only about 0.55. As a rough heuristic for the probability of the action selection remaining unchanged, we can consider the probability that the 2 sample best action estimate exceeds the single sample second best action estimate by 0.9234 with the following calculation: $$1 - \operatorname*{cdf}(\operatorname*{Normal}(\mu = 1.54 - 1, \sigma = \sqrt{\frac{3}{2}}), x = 0.9234) \approx 0.377.$$ Therefore, on step 12, the average run will change the action selection to something less optimal. The larger the value of c, the more likely the selection is to change due to the larger weight placed on exploration. Empirically, as c approaches $$\infty$$ the expected reward on step 12 approaches 0.744 vs 1.145 on step 11. That compares to the expected q value for the top 3 actions of approximately 0.656, 1, and 1.54. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$9e644919-5833-46ff-8f6e-e755ba688329„§cell_idÙ$9e644919-5833-46ff-8f6e-e755ba688329¤codeÚUmd""" One reason this formula is called the softmax is because it preserves the ordering of the original values including the maximum, and if a multiplicative parameter (usually called Î²) is applied to each value in the exponent, it can be made arbitrarily close to returning 1 for the maximum and zero for all other values. If that parameter is 0, then the function always returns the uniform distribution, so in this way the output can smoothly vary from completely uniform to the unique maximum. Below is a plot if the softmax of a given number of items where the Î² parameter can vary. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$3a798da5-c309-48f2-aab1-6602ded8a650„§cell_idÙ$3a798da5-c309-48f2-aab1-6602ded8a650¤codeÚ_function simple_algorithm(qs::Vector{Float64}, k::Integer, Ïµ::AbstractFloat; steps = 1000, Qinit = 0.0, Î± = 0.0, c = 0.0, #pre-allocated result vectors, can be provided to function to reduce allocations cum_reward_ideal = zeros(steps), step_reward_ideal = zeros(steps), cum_reward = zeros(steps), step_reward = zeros(steps), optimalstep = fill(false, steps), optimalaction_pct = zeros(steps)) #define bandit sampling function and initialize action value estimates bandit(a) = sample_bandit(a, qs) N = zeros(k) Q = ones(k) .* Qinit accum_reward_ideal = 0.0 accum_reward = 0.0 bestaction = argmax(qs) optimalcount = 0 actions = collect(1:k) for i = 1:steps shuffle!(actions) #so that ties are broken randomly with argmax a = if rand() < Ïµ rand(actions) elseif c == 0.0 #plain Ïµ-greedy action selection actions[argmax(view(Q, actions))] else #UCB action selection actions[argmax(view(Q, actions) .+ (c .* sqrt.(log(i) ./ view(N, actions))))] end if a == bestaction optimalstep[i] = true optimalcount += 1 else optimalstep[i] = false end #sample reward and step_reward[i] = bandit(a) step_reward_ideal[i] = bandit(bestaction) accum_reward_ideal += step_reward_ideal[i] cum_reward_ideal[i] = accum_reward_ideal accum_reward += step_reward[i] cum_reward[i] = accum_reward optimalaction_pct[i] = optimalcount / i N[a] += 1.0 if Î± == 0.0 #sample-average Q[a] += (1.0/N[a])*(step_reward[i] - Q[a]) else #exponential recency-weighted average Q[a] += Î±*(step_reward[i] - Q[a]) end end return (;Q, step_reward, step_reward_ideal, cum_reward, cum_reward_ideal, optimalstep, optimalaction_pct) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$7950e06b-e8ce-4bd7-9681-ab7b66dfec69„§cell_idÙ$7950e06b-e8ce-4bd7-9681-ab7b66dfec69¤codeÚÑfunction plot_bandit_testbed(n; npoints = 5_000) qs = create_bandit(n) spreads = randn(npoints) violins = [violin(y = spreads .+ qs[i], name = i, points=false, meanline_visible=true) for i in 1:n] PlutoPlotly.plot(violins, Layout(showlegend = false, yaxis = attr(title = "Reward distribution"), xaxis = attr(title = "Action"), annotations = [attr(x = i-1 + 0.5, y = qs[i], xref = "x", yref = "y", text = "q($i)", showhead=false, ax=0, ay=0) for i in 1:n])) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$c9c2aa04-15a2-446a-839c-bcf724b0f57a„§cell_idÙ$c9c2aa04-15a2-446a-839c-bcf724b0f57a¤codeÙÏhtml""" """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$d9265b98-cc3e-4a60-b16e-f54d9f78c9d3„§cell_idÙ$d9265b98-cc3e-4a60-b16e-f54d9f78c9d3¤codeÙH(::Type{GradientReward})(;kwargs...) = k -> GradientReward(k; kwargs...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$4afd6c2c-632e-4196-ab36-4d314ed9ec96„§cell_idÙ$4afd6c2c-632e-4196-ab36-4d314ed9ec96¤codeÚ;md""" For $$\alpha = 0.1$$ this value is higher than for the sample average case from the second step onwards. Also it asymptotes to a finite value whereas the sample average variance trends towards zero. This asymptote can be calculated by inspecting the equation taking the limit for large N. If $\alpha < 1$ we have $$\frac{\alpha}{2 - \alpha}$$. For $\alpha = 0.1$ this is about $(round((0.1 / (2 - 0.1)), sigdigits = 3)). If we draw an analogy to the sample average case, this would be equivalent to the variance after collecting $((1-0.1)/0.1) samples reflecting the fact that over time we are maintaining information from roughly a constant number of samples. The "optimal distribution" line in the non-stationary parameter study" figure shows the performance of this policy for different values of $$\alpha$$. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$d4ce45ae-613e-41ee-b626-69b0dbcf6452„§cell_idÙ$d4ce45ae-613e-41ee-b626-69b0dbcf6452¤codeÚ md""" > ### *Exercise 2.6: Mysterious Spikes* > The results shown in Figure 2.3 should be quite reliable because they are averages over 2000 individual, randomly chosen 10-armed bandit tasks. Why, then, are there oscillations and spikes in the early part of the curve for the optimistic method? In other words, what might make this method perform particularly better or worse, on average, on particular early steps? The spike occurs on step 11. Due to the initial Q values it is almost 100% likely that a given run will sample each of the 10 possible actions once before repeating any. For this not to be the case, one of the samples would have to exceed the initial value of 5.0 which has a probability near zero since the expected q value for the best arm is around 1.539 which unit variance. That would mean that at any given step only 10% of the runs would select the optimal action and indeed for the first 10 steps about 10% of the runs are selecting the optimal action as we'd expect from random chance. On the 11th step, the Q value estimate for each action is $$(0.9 \times 5) + (0.1 \times action\_reward)$$. The optimal action will be selected on this step as long as the reward produced by the best action exceeded all the others. Empirically, that probability is ~44% which is similar to the probability calculated for the expected value of the best action of ~1.539 exceeding the rewards from the other 9 arms. For those 44% of the runs that do select the optimal action, they will obtain a reward with expected value 1.539. If they received that reward during both samples, then the Q value estimate will be $$0.9 \times ((0.9 \times 5) + (0.1 \times 1.539)) + (0.1 \times 1.539) \approx 4.34$$. Let's consider the second best arm which has an expected q value of ~1. The updated estimate for that arm after receiving a reward equal to the expected value is $$0.9 \times 0.5 + 0.1 \times 1 \approx 4.6$$. Following the same reasoning for the third best arm, the value is about 4.57. In fact even a reward of zero will produce an estimate of $$0.9 \times 5 = 4.5$$ which still exceeds the estimate for the optimal action in our scenario. That explains why the percentage of optimal actions drops in the 12th step because it is expected that the estimate of the action selected on step 11 will drop below at least one of the other arms, thus changing the maximizing action selection to a worse one. """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$33d201ba-52f2-44f0-8bc6-3930ec77f62f„§cell_idÙ$33d201ba-52f2-44f0-8bc6-3930ec77f62f¤codeÙ·md""" ### Softmax Visualization Number of Items: $(@bind softmax_k NumberField(2:100, default = 10)) Î² Parameter: $(@bind Î² Slider(0.0:0.01:10, default = 1.0, show_value=true)) """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$0de99ee5-d94d-4d07-8cef-a6f9caf5e742„§cell_idÙ$0de99ee5-d94d-4d07-8cef-a6f9caf5e742¤codeºfigure_2_4(;params_2_4...)¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$a61c15eb-ed5f-4052-a3a3-3276940564a1„§cell_idÙ$a61c15eb-ed5f-4052-a3a3-3276940564a1¤codeÙ$(::Type{T})(e::T) where T <: UCB = e¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$99945570-0b2a-432a-ae83-3a7abc39cac8„§cell_idÙ$99945570-0b2a-432a-ae83-3a7abc39cac8¤codeÙºfunction normalize_Ï€vec!(Ï€vec::Vector{T}) where T <: AbstractFloat s = sum(Ï€vec) if (isinf(s) || isnan(s) || (s == 0)) Ï€vec .= one(T) / length(Ï€vec) else Ï€vec ./= s end end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18„§cell_idÙ$aa238ebc-8730-46c7-8ad9-41c7cac70b18¤codeÚfunction sample_action(est::ActionValue, explorer::UCB, i::Integer, actions::AbstractVector) (Q, N, c) = (est.Q, est.N, explorer.c) argmax(view(Q, actions) .+ c .* sqrt.(log(i) ./ view(N, actions))) # argmax(est.Q[actions] .+ (est.explorer.c .* sqrt.(log(i) ./ est.N[actions]))) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$2ceadc6d-5522-43f6-a803-a17a47a6048a„§cell_idÙ$2ceadc6d-5522-43f6-a803-a17a47a6048a¤codeÙ,exercise_2_11(;nonstationarysearchparams...)¨metadataƒ©show_logsÂ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$9093ed39-f6e2-4c31-9fef-466bdd197423„§cell_idÙ$9093ed39-f6e2-4c31-9fef-466bdd197423¤codeÙe(::Type{ConstantStep{T1}})(Î±::T2) where {T1 <: AbstractFloat, T2 <: Real} = ConstantStep{T1}(T1(Î±))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2„§cell_idÙ$bb16115a-a2d9-4b8d-9937-96ab1cdd1ce2¤codeÚÎfunction nonstationary_bandit_statistics(k::Integer; steps = 2_000, Ïƒ = 0.01, initR = 0.0, nruns = 1000) function run() qs = initR .* ones(k) q_history = [[q] for q in qs] for i = 1:steps qs .+= randn(k) .*Ïƒ #update q values with random walk sort!(qs, rev = true) for i in eachindex(qs) push!(q_history[i], qs[i]) end end return q_history end runs = [run() for _ in 1:nruns] q_avg = [sum(run[i] for run in runs) ./ nruns for i in 1:k] traces = [scatter(x = 1:steps, y = q_avg[i], name = "Arm Rank $i", showlegend = false) for i in 1:k] benchtrace = scatter(x = 1:steps, y = Ïƒ .* sqrt.(1:steps), name = "Expected Reward STD", line = attr(dash = "dot", width = 5, color = "rgba(100, 100, 100, 0.9)")) PlutoPlotly.plot([traces; benchtrace], Layout(xaxis_title = "Step", yaxis_title = "Mean Reward", legend = attr(orientation = "h", y = 1.), title = "Expected Value for Arm Mean Rewards Ranked from 1 to $k", height = 500)) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5„§cell_idÙ$54deaa09-8f87-4caf-b2a0-f15bcd5b40a5¤codeÙî#calculates a vector of probabilities for selecting each action given exponentiated "perferences" given in expH using the softmax distribution function update_Ï€vec!(Ï€vec, H::AbstractVector) Ï€vec .= exp.(H) normalize_Ï€vec!(Ï€vec) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$6292f449-8720-41f1-84de-1865fb5fddbf„§cell_idÙ$6292f449-8720-41f1-84de-1865fb5fddbf¤codeÚªfunction figure_2_3(;k = 10, Ïµ = 0.1, Qinit = 5.0, Î±=0.1) optimistic_greedy_runs = average_simple_runs(k, 0.0, Qinit = Qinit, Î± = Î±) realistic_Ïµ_runs = average_simple_runs(k, Ïµ, Î± = Î±) steps = 1:1000 t1 = scatter(x = steps, y = optimistic_greedy_runs[3], name = "Optimistic, greedy, Qinit = $Qinit") t2 = scatter(x = steps, y = realistic_Ïµ_runs[3], name = "Realistic, Ïµ-greedy, Qinit = 0.0. Ïµ = $Ïµ") PlutoPlotly.plot([t1, t2], Layout(xaxis_title = "Step", yaxis = attr(title = "% Runs Taking Optimal Action", range = [0, 1]), legend = attr(orientation = "h", y = 1.1), width = 700, height = 400, title = "$k Armed Testbed, Î± = $Î±", hovermode = "x unified")) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$0d8e4160-adf2-4b43-9914-942539339972„§cell_idÙ$0d8e4160-adf2-4b43-9914-942539339972¤codeÚËmd""" ### Revisiting the Gradient Bandit Consider what is the distribution $\pi_t(x)$ that maximizes $\mathbb{E}[R_t]=\sum_x \pi_t(x) q_*(x)$ using the softmax representation of the distribution. If we revisit the derivation in section 2.8, and keep the $q_*$ values instead of samples, we have $\frac{\partial \mathbb{E}[R_t]}{\partial H_t(a)} = \sum_x \left [ (q_*(x) - B_t) \left ( \mathbb{1}_{a = x} - \pi_t(a) \right ) \right ] = \sum_x \left [ (q_*(x) - B_t) \left ( \mathbb{1}_{a = x} - \frac{e^{H_t(a)}}{\sum_{b=1}^k e^{H_t(b)}} \right ) \right ]$ At the maximum this must be 0 for all actions: $\sum_x \left [ (q_*(x) - B_t) \left ( \mathbb{1}_{a = x} - \frac{e^{H_t(a)}}{\sum_{b=1}^k e^{H_t(b)}} \right ) \right ] = 0$ Consider the action that is the maximum. One solution that would cause that partial derivative to be 0 is if $\pi_t(a_{max}) = 1$ and all others are 0. To see this the term multiplying the q values would be $(0-0$ for $a \neq a_{max}$ and $(1-1)$ for $a = a_{max}$. But in this case the other terms are not zero. What are they? Consider an action $a \neq a_{max}$. The sum in this case is $q_a - q_{a_{max}}$ Since we know $q_*$, $B_t$ might as well be set to $\sum_x q_*(x) / [x]$ """¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÃÙ$09227386-1620-4093-b913-5786205aad13„§cell_idÙ$09227386-1620-4093-b913-5786205aad13¤codeÚGfunction update_estimator!(est::GradientReward{T, E, A}, a::Integer, r::T, step::Integer) where {T <: AbstractFloat, E <: GradientSample{T}, A <: AverageMethod{T}} rdiff = r - est.RÌ„ c = est.Î± * rdiff est.H .-= c .* est.Ï€vec est.H[a] += c est.RÌ„ += updatecoef(est, a, step) * rdiff update_Ï€vec!(est.Ï€vec,est.H) end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$f995d0af-50bc-4e33-9bbf-17a7ab06358a„§cell_idÙ$f995d0af-50bc-4e33-9bbf-17a7ab06358a¤codeÙ>struct Ïµ_Greedy{T <: AbstractFloat} <: Explorer{T} Ïµ::T end¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂÙ$fbf7b108-dc68-4077-b63c-6f88161d2098„§cell_idÙ$fbf7b108-dc68-4077-b63c-6f88161d2098¤codeÙI(::Type{ConstantStep{T}})() where T<:AbstractFloat = ConstantStep(T(0.1))¨metadataƒ©show_logsÃ¨disabledÂ®skip_as_scriptÂ«code_foldedÂ«notebook_idÙ$2f6e0b20-3902-11f0-352f-f90f1beb0d27«in_temp_dirÂ¨metadata€

Distribution of Best Action Reward for a Random 10 Armed Bandit, Expected Value Estimate = 1.539

Expected Value of Mean Reward for Arms Ranked from 1 to 10

Exercise 2.4

Ideas for Non-Stationary Problem

Figure 2.5 Parameters

Dependencies and Settings

2.5 Tracking a Nonstationary Problem

Parameter Studies

Figure Parameters for Exercise 2.5

Exercise 2.11 (programming)

Figure 2.2

Figure 2.1

Exercise 2.10

2.9 Associate Search (Contextual Bandits)

2.6 Optmisitic Initial Values

Exercise 2.7: Unbiased Constant-Step-Size Tick

Exercise 2.1

Figure 2.6

Chapter 2: Multi-armed Bandits

2.1: A k-armed Bandit Problem

2.4 Incremental Implementation

2.8 Gradient Bandit Algorithms

Code Refactoring

Figure 2.3 Parameters

2.3 The 10-armed Testbed

$\epsilon - Greedy$ Action Value Method

Testbed Params

Update Functions

Figure 2.4 Parameters

Bandit Arm Reward Distributions

Exercise 2.2: Bandit example

Exercise 2.9

Alternative Distribution Method

Exercise 2.3

Bandit Arm Reward Distributions

Sampling Functions

Why softmax?

Exercise 2.5 (programming)

Extra Notes on the Soft-Max and Gradient Bandit

2.2: Action-value Methods

2.7 Upper-Confidence-Bound Action Selection

Non-stationary Parameter Study Figure

Exercise 2.8: UCB Spikes

Exercise 2.6: Mysterious Spikes

Softmax Visualization

Revisiting the Gradient Bandit