mimetext/htmlrootassigneelast_run_timestampA Vpersist_js_state·has_pluto_hook_features§cell_id$19dfabda-7049-4050-8662-0385529c0c5adepends_on_disabled_cells§runtime Kpublished_object_keysdepends_on_skipped_cellsçerrored$b71145a4-2614-4f62-bfd2-7d5d1fecec56queued¤logsrunning¦outputbodyGactor_critic_with_eligibility_traces! (generic function with 3 methods)mimetext/plainrootassigneelast_run_timestampA +
)opersist_js_state·has_pluto_hook_features§cell_id$b71145a4-2614-4f62-bfd2-7d5d1fecec56depends_on_disabled_cells§runtime Xpublished_object_keysdepends_on_skipped_cells§errored$c0876a48-ea18-494d-8bfc-e2bceb73b417queued¤logsrunning¦outputbody F%
mimetext/htmlrootassigneelast_run_timestampA @!persist_js_state·has_pluto_hook_features§cell_id$c0876a48-ea18-494d-8bfc-e2bceb73b417depends_on_disabled_cells§runtime|published_object_keysdepends_on_skipped_cellsçerrored$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091queued¤logsrunning¦outputbodyDreinforce_monte_carlo_control_fcann (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA $0persist_js_state·has_pluto_hook_features§cell_id$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091depends_on_disabled_cells§runtime =xpublished_object_keysdepends_on_skipped_cells§errored$f4b6f10b-4cd0-4be6-98ec-4d4ffb696392queued¤logsrunning¦outputbodyS
One-step Actor-Critic Implementation
mimetext/htmlrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$f4b6f10b-4cd0-4be6-98ec-4d4ffb696392depends_on_disabled_cells§runtime mȵpublished_object_keysdepends_on_skipped_cells§errored$9db9ff71-bee9-4bea-a45b-748f8517fed1queued¤logsrunning¦outputbodyelementsaction_probabilitiesprefixFloat32elements0.0033094text/plain0.996691text/plaintypeArrayprefix_shortobjectidb337585f8106efc9!application/vnd.pluto.tree+objectstate_value_estimate-243.183text/plaintypeNamedTupleobjectid1ff9b53824d3895mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA *6persist_js_state·has_pluto_hook_features§cell_id$9db9ff71-bee9-4bea-a45b-748f8517fed1depends_on_disabled_cells§runtimeǵpublished_object_keysdepends_on_skipped_cellsçerrored$4634267b-5dea-4164-8bb2-1eb2fd4d7954queued¤logsrunning¦outputbodyBupdate_linear_eligibility_vector! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA ذpersist_js_state·has_pluto_hook_features§cell_id$4634267b-5dea-4164-8bb2-1eb2fd4d7954depends_on_disabled_cells§runtime "published_object_keysdepends_on_skipped_cells§errored$6c5f51bb-a6be-447e-b73d-4f9c2885e809queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestamp persist_js_state·has_pluto_hook_features§cell_id$6c5f51bb-a6be-447e-b73d-4f9c2885e809depends_on_disabled_cellsçruntimepublished_object_keysdepends_on_skipped_cells§errored$cc45091e-b889-4d5a-9eef-84d80f792046queued¤logsrunning¦outputbody
13.4 REINFORCE with Baseline
The policy gradient theorem (13.5) can be generalized to include a comparison of the action value to an arbitrary baseline b(s):
The baseline can be any function, even a random variable, as long as it does not vary with $a$; the euation remains valid because the subtracted quantity is zero:
The policy gradient theorem with baseline (13.10) can be used to derive an update rule using similar steps as in the previous section. The update rule that we end up with is a new version of REINFORCE that includes a general baseline:
Since the baseline could be uniformly zero, this is a strict generalization of REINFORCE. To have an effective baseline that depends on state we can use a state value estimate that is also updated with gradient steps: $\hat v(S_t, \mathbf{w})$. Using such an estimate we can revise the previous REINFORCE algorithm.
mimetext/htmlrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$cc45091e-b889-4d5a-9eef-84d80f792046depends_on_disabled_cells§runtime ]Apublished_object_keysdepends_on_skipped_cells§errored$5b15d91e-7119-4f85-a54a-7d4f1fdaf097queued¤logsrunning¦outputbodyIcreate_actor_critic_continuing_params_UI (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA rpersist_js_state·has_pluto_hook_features§cell_id$5b15d91e-7119-4f85-a54a-7d4f1fdaf097depends_on_disabled_cells§runtime >published_object_keysdepends_on_skipped_cellsçerrored$ba41f521-4ee2-42a6-bf18-078bfa4b875equeued¤logsrunning¦outputbodyAmake_n_param_dist_policy_params (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA ' persist_js_state·has_pluto_hook_features§cell_id$ba41f521-4ee2-42a6-bf18-078bfa4b875edepends_on_disabled_cells§runtime xpublished_object_keysdepends_on_skipped_cells§errored$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03queued¤logsrunning¦outputbodyNcartpole_tilecoding_reinforce_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 1<persist_js_state·has_pluto_hook_features§cell_id$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03depends_on_disabled_cells§runtime Y?published_object_keysdepends_on_skipped_cellsçerrored$3c695d54-c30f-4f04-bd40-f5da53be2a95queued¤logsrunning¦outputbodyN
mimetext/htmlrootassigneelast_run_timestampA A tKpersist_js_state·has_pluto_hook_features§cell_id$cd9c9eeb-c90d-4499-9503-7773d5250f47depends_on_disabled_cells§runtime -published_object_keysdepends_on_skipped_cellsçerrored$fd58402f-da65-44cf-b81a-e21192fd0e63queued¤logsrunning¦outputbody y
mimetext/htmlrootassigneelast_run_timestampA @`persist_js_state·has_pluto_hook_features§cell_id$fd58402f-da65-44cf-b81a-e21192fd0e63depends_on_disabled_cells§runtime\9published_object_keysdepends_on_skipped_cellsçerrored$8e39bd15-862e-4941-88f9-2794b861a523queued¤logsrunning¦outputbodyNreinforce_monte_carlo_control_linear_features (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA $ 9?persist_js_state·has_pluto_hook_features§cell_id$8e39bd15-862e-4941-88f9-2794b861a523depends_on_disabled_cells§runtime ;hpublished_object_keysdepends_on_skipped_cells§errored$64900586-ef92-48e4-839e-ff952a46671bqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestamp persist_js_state·has_pluto_hook_features§cell_id$64900586-ef92-48e4-839e-ff952a46671bdepends_on_disabled_cellsçruntimepublished_object_keysdepends_on_skipped_cells§errored$fddef10c-7695-4596-9e16-987fd45a57e6queued¤logsrunning¦outputbodyBsetup_cartpole_continuous_problem (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 0upersist_js_state·has_pluto_hook_features§cell_id$fddef10c-7695-4596-9e16-987fd45a57e6depends_on_disabled_cells§runtime Nvpublished_object_keysdepends_on_skipped_cells§errored$e2b09af1-0f22-4f7f-b806-54fa522adb20queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$e2b09af1-0f22-4f7f-b806-54fa522adb20depends_on_disabled_cells§runtime'Kpublished_object_keysdepends_on_skipped_cells§errored$2be8a812-4f21-4fe8-a2de-50497db0345aqueued¤logsrunning¦outputbodyg
Actor-Critic Implementation for Continuous Action Spaces
mimetext/htmlrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$2be8a812-4f21-4fe8-a2de-50497db0345adepends_on_disabled_cells§runtime upublished_object_keysdepends_on_skipped_cells§errored$68806899-9972-460a-9f11-daa708a9d610queued¤logsrunning¦outputbodyUactor_critic_with_eligibility_traces_linear_features (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA +NYpersist_js_state·has_pluto_hook_features§cell_id$68806899-9972-460a-9f11-daa708a9d610depends_on_disabled_cells§runtime KMpublished_object_keysdepends_on_skipped_cells§errored$189798b3-ec6b-48b9-918c-ee0f65935ab3queued¤logsrunning¦outputbody
Exercise 13.3
In Section 13.1 we considered policy parameterizations using the soft-max in action preferences (13.2) with linear action preferences (13.3). For this parameterization, prove that the eligibility vector is $\begin{flalign} \nabla \ln \pi(a|s, \boldsymbol{\theta}) = \mathbf{x}(s, a) - \sum_b \pi(b|s, \boldsymbol{\theta}) \mathbf{x}(s, b) \tag{13.9} \end{flalign}$ using the definitions and elementary calculus.
mimetext/htmlrootassigneelast_run_timestampA Epersist_js_state·has_pluto_hook_features§cell_id$189798b3-ec6b-48b9-918c-ee0f65935ab3depends_on_disabled_cells§runtime -published_object_keysdepends_on_skipped_cells§errored$00152954-dc98-4120-b94b-2ea4d987832bqueued¤logsrunning¦outputbodyBcreate_mountaincar_continuing_mdp (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA !'r persist_js_state·has_pluto_hook_features§cell_id$00152954-dc98-4120-b94b-2ea4d987832bdepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$42d4600a-bf3c-45ac-b7f5-d23917713ff5queued¤logsrunning¦outputbody
Layer Size: Num Layers:
mimetext/htmlrootassigneelast_run_timestampA !@ persist_js_state·has_pluto_hook_features§cell_id$42d4600a-bf3c-45ac-b7f5-d23917713ff5depends_on_disabled_cells§runtimé̵published_object_keysdepends_on_skipped_cellsçerrored$4e29c621-223e-4859-8e96-db04b967815aqueued¤logsrunning¦outputbodyPsetup_binary_squashed_gaussian_policy_arguments (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA '4Qpersist_js_state·has_pluto_hook_features§cell_id$4e29c621-223e-4859-8e96-db04b967815adepends_on_disabled_cells§runtime ӵpublished_object_keysdepends_on_skipped_cells§errored$5981f52b-d829-4c7d-b47b-33310f7d64a2queued¤logsrunning¦outputbodyprefixFloat32elements0.5text/plain0.5text/plaintypeArrayprefix_shortobjectid540650b0f89a7c43mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA >persist_js_state·has_pluto_hook_features§cell_id$5981f52b-d829-4c7d-b47b-33310f7d64a2depends_on_disabled_cells§runtimeipublished_object_keysdepends_on_skipped_cellsçerrored$0e9de19e-bcd4-40ac-9831-afb6cad38422queued¤logsrunning¦outputbody=setup_fcann_policy_arguments (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA ݅npersist_js_state·has_pluto_hook_features§cell_id$0e9de19e-bcd4-40ac-9831-afb6cad38422depends_on_disabled_cells§runtime Kpublished_object_keysdepends_on_skipped_cells§errored$ff3009eb-23f9-44fe-8e56-85dbc7b463d0queued¤logsrunning¦outputbody5show_squashed_policy (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA >^persist_js_state·has_pluto_hook_features§cell_id$ff3009eb-23f9-44fe-8e56-85dbc7b463d0depends_on_disabled_cells§runtime >published_object_keysdepends_on_skipped_cellsçerrored$4fb83451-b6f8-4e6e-a131-1accc8e10b08queued¤logsrunning¦outputbodyMreinforce_with_baseline_monte_carlo_control! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA #]R;persist_js_state·has_pluto_hook_features§cell_id$4fb83451-b6f8-4e6e-a131-1accc8e10b08depends_on_disabled_cells§runtimea:published_object_keysdepends_on_skipped_cells§errored$406638af-1e08-44d2-9ee4-97aa9294a94bqueued¤logsrunning¦outputbodyF
13.2 The Policy Gradient Theorem
mimetext/htmlrootassigneelast_run_timestampA vupersist_js_state·has_pluto_hook_features§cell_id$406638af-1e08-44d2-9ee4-97aa9294a94bdepends_on_disabled_cells§runtime õpublished_object_keysdepends_on_skipped_cells§errored$57e5e12a-b722-4ea3-ab3b-e5711029e640queued¤logsrunning¦outputbodyFone_step_actor_critic_linear_features (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA *persist_js_state·has_pluto_hook_features§cell_id$57e5e12a-b722-4ea3-ab3b-e5711029e640depends_on_disabled_cells§runtime :蘆published_object_keysdepends_on_skipped_cells§errored$374af774-3a97-49b5-a3bb-bc3f7f63a3faqueued¤logsrunning¦outputbodyԾ
mimetext/htmlrootassigneelast_run_timestampA !hHwpersist_js_state·has_pluto_hook_features§cell_id$7bf209c8-ef0a-46d1-937e-b1a6e45dc62edepends_on_disabled_cells§runtime2published_object_keysdepends_on_skipped_cellsçerrored$dd8e8cd2-7b41-46c4-8530-adefb7aea684queued¤logsrunning¦outputbodyRactor_critic_binary_episodic_beta_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA /吼persist_js_state·has_pluto_hook_features§cell_id$dd8e8cd2-7b41-46c4-8530-adefb7aea684depends_on_disabled_cells§runtime ;published_object_keysdepends_on_skipped_cellsçerrored$4fea7232-f286-4a8b-93f8-a0702818ab31queued¤logsrunning¦outputbodyO
The final expected value expression (13.5) can be sampled on a step by step basis during an episode since we would have access to both the step count and some unbiased sample of the state-action value.
Consider a new random variable $Y \sim \tanh(X)$ where $X \sim N(0, 1)$. Using the change of variables theorem from probability theory we can compute the density function of $Y$:
Consider the case of applying the techniques in this chapter to problems where we choose feature vectors and parameters to effectively compute the tabular case. That is we enumerate every state and state/action pair. Our parameters for each function will store a single value for each case. Let's consider the gradients for both the state-value estimate and the policy. We will use two sets of parameters: $\mathbf{w}$ and $\mathbf{\theta}$. $\mathbf{w}_s$ is the parameter for state s and $\mathbf{\theta}_{s, a}$ is the parameter for state/action pair $(s, a)$. Using this notation $\mathbf{w}$ is a vector and $\theta$ is a matrix.
Comparing to what we desire, $\mathbf{x} = \mathbf{\theta}_s$ which is the parameter vector for the state s and $\sigma = \pi$. So we can immediately write down the components of this gradient:
$$\begin{equation}
\nabla \ln{\pi(a|\theta_s)}_i =
\begin{cases}
-\pi(i|\theta_s) & i \neq a \\
1 - \pi(i|\theta_s) & i = a
\end{cases}
\end{equation}$$
This is a gradient vector which corresponds to the components of $\theta_s$ which is the parameter vector for each action at that state. We have a new vector update for each unique state/action pair observed, but once those two are fixed the number of components that need to be calculated is just a vector with a length equal to the number of actions.
mimetext/htmlrootassigneelast_run_timestampA ^persist_js_state·has_pluto_hook_features§cell_id$4c34640f-efa2-4e1d-8a70-0acd2ce45428depends_on_disabled_cells§runtime Rpublished_object_keysdepends_on_skipped_cells§errored$e7566274-5518-4e28-8738-d4b1747d0cfbqueued¤logsrunning¦outputbody:form_state_value_function (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA "persist_js_state·has_pluto_hook_features§cell_id$e7566274-5518-4e28-8738-d4b1747d0cfbdepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$6bf5ad39-1400-4e1f-a843-a1934b8aaa48queued¤logsrunning¦outputbodyNupdate_squashed_gaussian_eligibility_vector! (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA #SmHpersist_js_state·has_pluto_hook_features§cell_id$6bf5ad39-1400-4e1f-a843-a1934b8aaa48depends_on_disabled_cells§runtime ^published_object_keysdepends_on_skipped_cells§errored$17d07ef4-7c0a-47cc-a701-32c60336571bqueued¤logsrunning¦outputbody
Noticing this pattern, the kth term will be of the form $\gamma^k \sum_{x \in \mathcal{S}} \Pr(s \rightarrow x, k, \pi)f(x)$ and the total expression will just be a sum of all of these terms to infinity or the maximum length of an episode under the policy. Looking more closely at the probability term, we can equate it to some other probabilities regarding episode length.
mimetext/htmlrootassigneelast_run_timestampA Ұpersist_js_state·has_pluto_hook_features§cell_id$17d07ef4-7c0a-47cc-a701-32c60336571bdepends_on_disabled_cells§runtime Yܵpublished_object_keysdepends_on_skipped_cells§errored$76fd79a2-2bc8-45f8-a243-48415118898aqueued¤logsrunning¦outputbody'BinarySquashedGaussianEligibilityVectormimetext/plainrootassigneelast_run_timestampA !>persist_js_state·has_pluto_hook_features§cell_id$76fd79a2-2bc8-45f8-a243-48415118898adepends_on_disabled_cells§runtime Mypublished_object_keysdepends_on_skipped_cells§errored$0b01ba67-3921-4f3f-a7e8-235190bc84ebqueued¤logsrunning¦outputbody/make_beta_dist (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA !bpersist_js_state·has_pluto_hook_features§cell_id$0b01ba67-3921-4f3f-a7e8-235190bc84ebdepends_on_disabled_cells§runtime vߵpublished_object_keysdepends_on_skipped_cells§errored$9acdbf38-2e10-45ec-85a0-d0db8453a599queued¤logsrunning¦outputbody;fcann_feature_vector_setup (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 2|Ȱpersist_js_state·has_pluto_hook_features§cell_id$9acdbf38-2e10-45ec-85a0-d0db8453a599depends_on_disabled_cells§runtime M=/published_object_keysdepends_on_skipped_cells§errored$d4e87ac4-6008-43b2-aa06-e232ec2b2b5bqueued¤logsrunning¦outputbody
mimetext/htmlrootassigneelast_run_timestampA @tlpersist_js_state·has_pluto_hook_features§cell_id$d4e87ac4-6008-43b2-aa06-e232ec2b2b5bdepends_on_disabled_cells§runtime spublished_object_keysdepends_on_skipped_cellsçerrored$05f120be-9695-4824-82fd-142a0df13098queued¤logsrunning¦outputbodyoactor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA /Ipersist_js_state·has_pluto_hook_features§cell_id$05f120be-9695-4824-82fd-142a0df13098depends_on_disabled_cells§runtime NS+published_object_keysdepends_on_skipped_cells§errored$b2539398-fdbc-42a2-a8f3-d327358f3643queued¤logsrunning¦outputbodyB
As an initial test, consider the discrete action space originally used for the mountain car problem where there are three actions (-1, 0, 1) corresponding to full throttle reverse, idle, and full throttle forward. We can apply the same tile coding solution technique from before but with a policy gradient method instead of Sarsa.
mimetext/htmlrootassigneelast_run_timestampA >cCpersist_js_state·has_pluto_hook_features§cell_id$b7f77935-bcab-4ef1-8e1b-a7d059784ff3depends_on_disabled_cells§runtimeܵpublished_object_keysdepends_on_skipped_cellsçerrored$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5queued¤logsrunning¦outputbodymsg٦UndefVarError: `reinforce_test` not defined in `Main.var"workspace#8"`
Suggestion: add an appropriate import or assignment. This global was declared but not assigned.stacktracecall_shorttop-level scopeinlined£urlpath/home/runner/work/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Chapter-13/Chapter_13_Policy_Gradient_Methods.jl#==#f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefileMChapter_13_Policy_Gradient_Methods.jl#==#f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5functop-level scopeparent_modulefrom_c¤mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA 0 persist_js_state·has_pluto_hook_features§cell_id$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$00bd2835-b006-4244-9877-bc7e031e3ef8queued¤logsrunning¦outputbody7plot_squashed_gaussian (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA !@persist_js_state·has_pluto_hook_features§cell_id$00bd2835-b006-4244-9877-bc7e031e3ef8depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cellsçerrored$50ae94c4-70f3-4215-82bd-eb2227c2badfqueued¤logsrunning¦outputbodyB
Waiting to run parameter study
mimetext/htmlrootassigneelast_run_timestampA 2T*persist_js_state·has_pluto_hook_features§cell_id$50ae94c4-70f3-4215-82bd-eb2227c2badfdepends_on_disabled_cells§runtime >,published_object_keysdepends_on_skipped_cellsçerrored$cc3ac95e-a398-438a-ba3d-62b6733f6342queued¤logsrunning¦outputbodyAupdate_fcann_action_preferences! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$cc3ac95e-a398-438a-ba3d-62b6733f6342depends_on_disabled_cells§runtime Ppublished_object_keysdepends_on_skipped_cells§errored$c926b6df-c40b-4c4c-8a95-ce9e41feb100queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestamp persist_js_state·has_pluto_hook_features§cell_id$c926b6df-c40b-4c4c-8a95-ce9e41feb100depends_on_disabled_cellsçruntimepublished_object_keysdepends_on_skipped_cells§errored$740a3f41-9302-481d-b373-762c0dea8effqueued¤logsrunning¦outputbodyEupdate_gaussian_eligibility_vector! (generic function with 4 methods)mimetext/plainrootassigneelast_run_timestampA #]ֻpersist_js_state·has_pluto_hook_features§cell_id$740a3f41-9302-481d-b373-762c0dea8effdepends_on_disabled_cells§runtime %
published_object_keysdepends_on_skipped_cells§errored$ba642a22-6623-482a-ab4a-81585b83e457queued¤logsrunning¦outputbody8average_continuing_runs (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA _persist_js_state·has_pluto_hook_features§cell_id$ba642a22-6623-482a-ab4a-81585b83e457depends_on_disabled_cells§runtime/pSpublished_object_keysdepends_on_skipped_cells§errored$d17a4bd0-5992-4247-912d-73d51758d2f3queued¤logsrunning¦outputbodyJ
Continuing Cartpole Example
mimetext/htmlrootassigneelast_run_timestampA $persist_js_state·has_pluto_hook_features§cell_id$d17a4bd0-5992-4247-912d-73d51758d2f3depends_on_disabled_cells§runtime jpublished_object_keysdepends_on_skipped_cells§errored$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7efqueued¤logsrunning¦outputbody y
mimetext/htmlrootassigneelast_run_timestampA @Zpersist_js_state·has_pluto_hook_features§cell_id$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7efdepends_on_disabled_cells§runtime ZǮpublished_object_keysdepends_on_skipped_cellsçerrored$5ee4ce72-7740-4297-8d84-619e0708e4acqueued¤logsrunning¦outputbodyJcartpole_continuing_fcann_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 8epersist_js_state·has_pluto_hook_features§cell_id$5ee4ce72-7740-4297-8d84-619e0708e4acdepends_on_disabled_cells§runtime wwpublished_object_keysdepends_on_skipped_cellsçerrored$645e93e7-e92e-49c4-9757-8294fabf4e9bqueued¤logsrunning¦outputbodyz^
mimetext/htmlrootassigneelast_run_timestampA +{persist_js_state·has_pluto_hook_features§cell_id$645e93e7-e92e-49c4-9757-8294fabf4e9bdepends_on_disabled_cells§runtime'M}published_object_keysdepends_on_skipped_cellsçerrored$0c56b341-24eb-4c78-844e-182f44a7221aqueued¤logsrunning¦outputbody &V
mimetext/htmlrootassigneelast_run_timestampA #_ persist_js_state·has_pluto_hook_features§cell_id$0c56b341-24eb-4c78-844e-182f44a7221adepends_on_disabled_cells§runtime_bpublished_object_keysdepends_on_skipped_cellsçerrored$d34d22ad-89c2-423e-91dd-bfb895dc6540queued¤logsrunning¦outputbody?cartpole_fcann_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 3.ڰpersist_js_state·has_pluto_hook_features§cell_id$d34d22ad-89c2-423e-91dd-bfb895dc6540depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cellsçerrored$20776e09-7d9b-4db8-a060-7bceeec65b47queued¤logsrunning¦outputbodyfactor_critic_with_eligibility_traces_binary_features_gaussian_actions (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA /persist_js_state·has_pluto_hook_features§cell_id$20776e09-7d9b-4db8-a060-7bceeec65b47depends_on_disabled_cells§runtime >Jpublished_object_keysdepends_on_skipped_cells§errored$7856b8a0-565d-4c86-9b3c-4424ff9b86ddqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA d9persist_js_state·has_pluto_hook_features§cell_id$7856b8a0-565d-4c86-9b3c-4424ff9b86dddepends_on_disabled_cells§runtime&published_object_keysdepends_on_skipped_cells§errored$735b548a-88f5-4a30-ab8f-dfb3d6401b2bqueued¤logsrunning¦outputbody
13.7 Policy Parameterization for Continuous Actions
With a parameterized policy we are to learn statistics of the distribution that selects actions. As a foundation consider the normal distribution:
mimetext/htmlrootassigneelast_run_timestampA Xpersist_js_state·has_pluto_hook_features§cell_id$735b548a-88f5-4a30-ab8f-dfb3d6401b2bdepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$7cf26604-9c2b-4a77-9674-7d4dac2f99f0queued¤logslinemsgsReplacing docs for `Main.var"workspace#8".make_random_walk_mrp :: Tuple{Integer}` in module `Main.var"workspace#8"`text/plaincell_id$7cf26604-9c2b-4a77-9674-7d4dac2f99f0kwargsidBase_Docs_4352b6d8filedocs/Docs.jlgroupDocslevelWarnlinemsgsReplacing docs for `Main.var"workspace#8".make_random_walk_mrp :: Tuple{Integer}` in module `Main.var"workspace#8"`text/plaincell_id$7cf26604-9c2b-4a77-9674-7d4dac2f99f0kwargsidBase_Docs_4352b6d8filedocs/Docs.jlgroupDocslevelWarnlinemsgsReplacing docs for `Main.var"workspace#8".make_random_walk_mrp :: Tuple{Integer}` in module `Main.var"workspace#8"`text/plaincell_id$7cf26604-9c2b-4a77-9674-7d4dac2f99f0kwargsidBase_Docs_4352b6d8filedocs/Docs.jlgroupDocslevelWarnlinemsg}WARNING: could not import FCANN.cuda_allocate into workspace#8
WARNING: could not import FCANN.cuda_allocate into workspace#8
WARNING: replacing module MountainCarTask.
WARNING: could not import FCANN.cuda_allocate into workspace#8
WARNING: replacing module MountainCarTask.
WARNING: could not import FCANN.cuda_allocate into workspace#8
WARNING: replacing module MountainCarTask.
text/plaincell_id$7cf26604-9c2b-4a77-9674-7d4dac2f99f0kwargsidPlutoRunner_d1acb81efileP/home/runner/.julia/packages/Pluto/5ete1/src/runner/PlutoRunner/src/io/stdout.jlgroupstdoutlevelLogLevel(-555)running¦outputbody"# This file is machine-generated - editing it directly is not advised\n\njulia_version = \"1.11.5\"\nmanifest_format = \"2.0\"\nproject_hash = \"52f0e08d74c26001471ce64a62da0627b2421990\"\n\n[[deps.AbstractPlutoDingetjes]]\ndeps = [\"Pkg\"]\ngit-tree-sha1 = \"6e1d2a35f2f90a4bc7c2ed98079b2ba09c35b83a\"\nuuid = \"6e696" ⋯ 22302 bytes ⋯ " \"8e850b90-86db-534c-a0d3-1478176c7d93\"\nversion = \"5.11.0+0\"\n\n[[deps.nghttp2_jll]]\ndeps = [\"Artifacts\", \"Libdl\"]\nuuid = \"8e850ede-7688-5339-a07c-302acd2aaf8d\"\nversion = \"1.59.0+0\"\n\n[[deps.p7zip_jll]]\ndeps = [\"Artifacts\", \"Libdl\"]\nuuid = \"3f19e933-33d8-53b3-aaab-bd5110c3b7a0\"\nversion = \"17.4.0+2\"\n"mimetext/plainrootassigneelast_run_timestampA 5persist_js_state·has_pluto_hook_features§cell_id$7cf26604-9c2b-4a77-9674-7d4dac2f99f0depends_on_disabled_cells§runtime \published_object_keysdepends_on_skipped_cells§errored$87ee21f3-16ca-4c8c-a0b9-f9e2fd258a91queued¤logsrunning¦outputbodyd
REINFORCE Implementation for Continuous Action Spaces
mimetext/htmlrootassigneelast_run_timestampA Zpersist_js_state·has_pluto_hook_features§cell_id$54f1546d-87ae-49d2-92ed-6fcc9b66e027depends_on_disabled_cells§runtime qpublished_object_keysdepends_on_skipped_cells§errored$63fbf8f4-e4e2-4893-be09-67450e92dbd7queued¤logsrunning¦outputbody*plot_cart (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA @{!persist_js_state·has_pluto_hook_features§cell_id$63fbf8f4-e4e2-4893-be09-67450e92dbd7depends_on_disabled_cells§runtime Ikpublished_object_keysdepends_on_skipped_cellsçerrored$d5020a8d-1dd7-403c-9d1f-665b95543943queued¤logsrunning¦outputbodymreinforce_with_baseline_monte_carlo_control_linear_features_gaussian_actions (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 'Apersist_js_state·has_pluto_hook_features§cell_id$d5020a8d-1dd7-403c-9d1f-665b95543943depends_on_disabled_cells§runtime @0Spublished_object_keysdepends_on_skipped_cells§errored$37a8ef7e-e859-4ef0-81e2-76c02a324031queued¤logsrunning¦outputbody
Policy Gradient Theorem Proof
In all cases below when a sum over states is taken, it is assumed to be over the set of non-terminal states: $\sum_s \implies \sum_{s \in \mathcal{S}}$ Note that for the case of the value function this is identical to the sum over $\mathcal{S}^+$ because the state-action values are always zero for terminal states.
$$\begin{flalign}
\nabla v_\pi(s) &= \nabla \left [ \sum_a \pi(a \vert s) q_\pi(s, a) \right ] \text{, } \forall s \in \mathcal{S} \tag{definitiong of value functions and expected value} \\
&= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \nabla q_\pi(s, a) \right ] \tag{product rule} \\
&= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \nabla \sum_{s^\prime, r} p(s^\prime, r \vert s, a)(r + \gamma v_\pi(s^\prime) \right ] \tag{relationship between action and state values} \\
&= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \gamma \sum_{s^\prime} p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) \right ] \tag{gradient independence}\\
\end{flalign}$$
Note that the final term in the sum is the original expression evaluated at $s^\prime$ instead of $s$, so we have derived a recurssive expression which can be applied repeatedly:
The probability transition function is normalized over all possible transition states $\sum_{s^\prime \in \mathcal{S}^+} p(s^\prime \vert s, a) = 1$. If we only take the sum of $\mathcal{S}$ then we instead get the probability that after a single transition we have NOT reached a terminal state. Let's say we also have a policy function $\pi(a \vert s)$ which is normalized over actions: $\sum_a \pi(a \vert s) = 1$. Now if we combine the two, we can arrive at a new distribution over transition states: $p(s^\prime \vert s, \pi) = \sum_a \pi(a \vert s) p(s^\prime \vert s, a)$ which is the probability of transitioning from $s$ to $s^\prime$ under the policy. We can see that this distribution is normalized over the transition states as well as long as we include the terminal state: $\sum_{s^\prime \in \mathcal{S}^+} p(s^\prime \vert s, \pi) = \sum_{s^\prime \in \mathcal{S}^+, a} \pi(a \vert s) p(s^\prime \vert s, a) = \sum_a \pi(a \vert s) \sum_{s^\prime \in \mathcal{S}^+} p(s^\prime \vert s, a) = 1 \times 1 = 1$. If instead we take the sum over $\mathcal{S}$ we simply get the probability of NOT terminating in one step.
What if we consider two steps into the future though? Now we have $\sum_{s^\prime}\sum_{a^\prime}\pi(a^\prime \vert s^\prime)p(s^{\prime \prime} \vert s^\prime, a^\prime)\sum_a \pi(a \vert s) p(s^\prime \vert s, a) = \sum_{s^\prime}p(s^{\prime \prime} \vert s^\prime, \pi) p(s^\prime \vert s, \pi)$. It would appear as though we can just put the two probabilities together and consider a new distribution over $s^{\prime \prime}$ which is $p(s^{\prime \prime} \vert s, \pi, 2)$ where instead of one step this now occurs over two steps, but how is this distribution normalized? In the case of the one step, transition, we saw that its sum over all transition states is 1 as expected. If we sum both transition states over only $\mathcal{S}$ rather than $\mathcal{S}^+$ what is the result? We already know that $\sum_{s^{\prime \prime} \in \mathcal{S}^+} p(s^{\prime \prime} \vert s^\prime , \pi) = \Pr \{ S_1 \neq S_T \ \vert S_0 = s^\prime, \pi \}$ that is the probability that after transitioning out of $s^\prime$ under the policy $\pi$ we have not reached a terminal state.
which is to say the probability that after two transitions from $s$ we are not in a terminal state under the policy $\pi$.
For the derivations that follow, we always take sums of these distributions over $\mathcal{S}$. For episodic problems, the on policy distribution $\mu_\pi(s)$ which is the probability of being in a state $s$ during an episode always excludes the terminal state. That is because if there is a non-zero probability of reaching a terminal state under a policy, then considering all possible episodes we may have an infinite number of visits to the terminal state. Technically the episodes have infinite length but we are only interested in the portion of the episode that preceeds the terminal state for the purpose of calculating probabilities. The more careful statement about the on policy distribution is that it measures the probability of being in a state during the non-terminal part of an episode. If we try to include the terminal states, then we cannot have a proper normalized definition of the on-policy distribution. Moreover, we have no need to measure the value of a terminal state accurately, since we always know it to be 0. The on policy distribution is used to formulate the value error objective function and it should only include states for which the value estimation is non-trivial.
mimetext/htmlrootassigneelast_run_timestampA =lpersist_js_state·has_pluto_hook_features§cell_id$7dbb42a3-aa8c-47e5-b668-18e6325d4038depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$192b9f82-8d3a-408f-91c2-829cfcd32572queued¤logsrunning¦outputbody8cartpole_vector_update! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 2Xpersist_js_state·has_pluto_hook_features§cell_id$192b9f82-8d3a-408f-91c2-829cfcd32572depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$b5319d8b-0420-4ebf-b603-ea0b93365ac1queued¤logsrunning¦outputbodyGshow_mountaincar_continuous_trajectory (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA AXpersist_js_state·has_pluto_hook_features§cell_id$b5319d8b-0420-4ebf-b603-ea0b93365ac1depends_on_disabled_cells§runtime PPpublished_object_keysdepends_on_skipped_cellsçerrored$4cbdb082-22ba-49e9-a6ed-4380917625acqueued¤logsrunning¦outputbodyb
Actor-Critic with Eligibility Traces Implementation
mimetext/htmlrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$4cbdb082-22ba-49e9-a6ed-4380917625acdepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$cc80848a-6834-4272-9152-e17b45448814queued¤logsrunning¦outputbody,wind_speeds (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA ?/persist_js_state·has_pluto_hook_features§cell_id$cc80848a-6834-4272-9152-e17b45448814depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cellsçerrored$05bfd818-bf4e-4bda-baa9-5ba647867097queued¤logsrunning¦outputbodyUactor_critic_with_eligibility_traces_binary_features (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA +~persist_js_state·has_pluto_hook_features§cell_id$05bfd818-bf4e-4bda-baa9-5ba647867097depends_on_disabled_cells§runtime 8 Rpublished_object_keysdepends_on_skipped_cells§errored$f0962801-0dfa-421f-8ffc-e64068e49913queued¤logsrunning¦outputbodyelementsfeature_vectorprefixFloat32elements0.0text/plain0.0text/plaintypeArrayprefix_shortobjectid7a7b475772429c5a!application/vnd.pluto.tree+objectnum_features2text/plainupdate_feature_vector!update_feature_vector!text/plaintypeNamedTupleobjectidcf73078e0a2e34b3mime!application/vnd.pluto.tree+objectrootassignee%const mountaincar_fcann_feature_setuplast_run_timestampA 2uhpersist_js_state·has_pluto_hook_features§cell_id$f0962801-0dfa-421f-8ffc-e64068e49913depends_on_disabled_cells§runtime^published_object_keysdepends_on_skipped_cells§errored$11a55af7-5301-4507-bb26-88e1e11236dbqueued¤logsrunning¦outputbody.
mimetext/htmlrootassigneelast_run_timestampA 2lpersist_js_state·has_pluto_hook_features§cell_id$11a55af7-5301-4507-bb26-88e1e11236dbdepends_on_disabled_cells§runtime o)gpublished_object_keysdepends_on_skipped_cellsçerrored$ddbca73f-c692-46f2-95f3-a7dd849d33f7queued¤logsrunning¦outputbodyڒ'Total Reward: -113.0
mimetext/htmlrootassigneelast_run_timestampA @icpersist_js_state·has_pluto_hook_features§cell_id$af144759-fe66-4ad0-b378-e9eb4e859db4depends_on_disabled_cells§runtime /published_object_keysdepends_on_skipped_cellsçerrored$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2queued¤logsrunning¦outputbodyprefix"ContinuousMDP{Float32, Tuple{Float32, Float32}, Float32, ContinuousMDPTransitionSampler{Float32, Tuple{Float32, Float32}, Float32, var"#step#1603"{Float32}}, typeof(Main.var"workspace#8".MountainCarTask.initialize_state), typeof(Main.var"workspace#8".MountainCarTask.isterm), Returns{Bool}}elementsptfprefixcContinuousMDPTransitionSampler{Float32, Tuple{Float32, Float32}, Float32, var"#step#1603"{Float32}}elementsstepS(::Main.var"workspace#8".var"#step#1603"{Float32}) (generic function with 1 method)text/plaintypestructprefix_shortContinuousMDPTransitionSamplerobjectid10d85fa8!application/vnd.pluto.tree+objectinitialize_state1initialize_state (generic function with 1 method)text/plainisterm'isterm (generic function with 1 method)text/plainis_valid_actionReturns{Bool}(true)text/plaintypestructprefix_shortContinuousMDPobjectid81156278aaa90f92mime!application/vnd.pluto.tree+objectrootassignee const mountaincar_continuous_mdplast_run_timestampA =sPpersist_js_state·has_pluto_hook_features§cell_id$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2depends_on_disabled_cells§runtime+published_object_keysdepends_on_skipped_cells§errored$fb8904a9-ae64-41cc-93b6-5a25855edad0queued¤logsrunning¦outputbody;get_corridor_episode_stats (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA !fpersist_js_state·has_pluto_hook_features§cell_id$fb8904a9-ae64-41cc-93b6-5a25855edad0depends_on_disabled_cells§runtime 3vpublished_object_keysdepends_on_skipped_cellsçerrored$a5b002c9-5e11-462a-9da0-6e060c7963f8queued¤logsrunning¦outputbodyelementsprefix,Main.var"workspace#8".CartPoleState{Float32}elementsprefixCartPoleState{Float32}elementsx30.0text/plainθ0.8text/plainẋ0.0text/plainθ̇-0.0text/plaint0.0text/plaintypestructprefix_shortCartPoleStateobjectida843943c04b3a7e!application/vnd.pluto.tree+objectprefixCartPoleState{Float32}elementsx30.0221text/plainθ0.795094text/plainẋ1.10655text/plainθ̇-0.245739text/plaint0.04text/plaintypestructprefix_shortCartPoleStateobjectidce8725f349c35742!application/vnd.pluto.tree+objectprefixCartPoleState{Float32}elementsx30.0885text/plainθ0.780267text/plainẋ2.21393text/plainθ̇-0.497006text/plaint0.08text/plaintypestructprefix_shortCartPoleStateobjectid56323732a2cd6c6!application/vnd.pluto.tree+objectprefixCartPoleState{Float32}elementsx30.1993text/plainθ0.755186text/plainẋ3.323text/plainθ̇-0.759398text/plaint0.12text/plaintypestructprefix_shortCartPoleStateobjectide9d1d7d09422679d!application/vnd.pluto.tree+objectprefixCartPoleState{Float32}elementsx30.3544text/plainθ0.719291text/plainẋ4.43467text/plainθ̇-1.03864text/plaint0.16text/plaintypestructprefix_shortCartPoleStateobjectid587ca329eacee254!application/vnd.pluto.tree+objectprefixCartPoleState{Float32}elementsx30.5541text/plainθ0.671792text/plainẋ5.54994text/plainθ̇-1.34061text/plaint0.2text/plaintypestructprefix_shortCartPoleStateobjectidf23d0775120eae84!application/vnd.pluto.tree+objectprefixCartPoleState{Float32}elementsx30.7985text/plainθ0.61166text/plainẋ6.66984text/plainθ̇-1.67126text/plaint0.24text/plaintypestructprefix_shortCartPoleStateobjectid937ae5d5b0481894!application/vnd.pluto.tree+objectprefixCartPoleState{Float32}elementsx31.0878text/plainθ0.53763text/plainẋ7.79538text/plainθ̇-2.03646text/plaint0.28text/plaintypestructprefix_shortCartPoleStateobjectidac7ee607274c7714!application/vnd.pluto.tree+object prefixCartPoleState{Float32}elementsx31.4222text/plainθ0.448211text/plainẋ8.92731text/plainθ̇-2.44158text/plaint0.32text/plaintypestructprefix_shortCartPoleStateobjectidd7442e47daa49912!application/vnd.pluto.tree+objectmoreprefixCartPoleState{Float32}elementsx36.3249text/plainθ-1.21277text/plainẋ-1.05244text/plainθ̇-1.20747text/plaint1.0text/plaintypestructprefix_shortCartPoleStateobjectidf9c13b3b4eec42a5!application/vnd.pluto.tree+objecttypeArrayprefix_shortobjectid1e33aa8b879d9c18!application/vnd.pluto.tree+objectprefixInt64elements3text/plain3text/plain3text/plain3text/plain3text/plain3text/plain3text/plain3text/plain 3text/plainmore1text/plaintypeArrayprefix_shortobjectidaf7504f82a205193!application/vnd.pluto.tree+objectprefixFloat32elements1.0text/plain1.0text/plain1.0text/plain1.0text/plain1.0text/plain1.0text/plain1.0text/plain1.0text/plain 1.0text/plainmore1.0text/plaintypeArrayprefix_shortobjectid5288029df0465058!application/vnd.pluto.tree+objectprefixCartPoleState{Float32}elementsx36.2609text/plainθ-1.26108text/plainẋ-2.1482text/plainθ̇-1.21309text/plaint1.04text/plaintypestructprefix_shortCartPoleStateobjectid54ad65ee320ed994!application/vnd.pluto.tree+object26text/plaintypeTupleobjectid77b4b6114cbab8a0mime!application/vnd.pluto.tree+objectrootassigneeconst ep2last_run_timestampA :?persist_js_state·has_pluto_hook_features§cell_id$a5b002c9-5e11-462a-9da0-6e060c7963f8depends_on_disabled_cells§runtimeY)published_object_keysdepends_on_skipped_cells§errored$83640f5b-fe13-4ec1-98a0-67a56c189ba1queued¤logsrunning¦outputbodyGactor_critic_with_eligibility_traces! (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA + qapersist_js_state·has_pluto_hook_features§cell_id$83640f5b-fe13-4ec1-98a0-67a56c189ba1depends_on_disabled_cells§runtime ppublished_object_keysdepends_on_skipped_cells§errored$61650a97-b353-4a85-b50b-93fee296ac7bqueued¤logsrunning¦outputbodyelementsfeature_vectorprefixFloat32elements0.0text/plain0.0text/plain0.0text/plain0.0text/plaintypeArrayprefix_shortobjectidaa97f30b6a1c9855!application/vnd.pluto.tree+objectnum_features4text/plainupdate_feature_vector!update_feature_vector!text/plaintypeNamedTupleobjectid18316424d13c4613mime!application/vnd.pluto.tree+objectrootassignee"const cartpole_fcann_feature_setuplast_run_timestampA 2I0persist_js_state·has_pluto_hook_features§cell_id$61650a97-b353-4a85-b50b-93fee296ac7bdepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$602a07dd-8928-4b44-97e5-01c5cbf38351queued¤logsrunning¦outputbody5plot_cartpole_policy (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA @6gpersist_js_state·has_pluto_hook_features§cell_id$602a07dd-8928-4b44-97e5-01c5cbf38351depends_on_disabled_cells§runtime u published_object_keysdepends_on_skipped_cellsçerrored$f7433324-acc3-49a5-b5b3-ada0c8f09d52queued¤logsrunning¦outputbodyelementsprefixInt64elements1text/plain2text/plain3text/plaintypeArrayprefix_shortobjectidb9d5c490997071d7!application/vnd.pluto.tree+objectprefixInt64elements2text/plain1text/plain2text/plaintypeArrayprefix_shortobjectid45f86e4ab1b5cc81!application/vnd.pluto.tree+objectprefixFloat32elements-1.0text/plain-1.0text/plain-1.0text/plaintypeArrayprefix_shortobjectide320a1e36c0632f8!application/vnd.pluto.tree+object4text/plain3text/plaintypeTupleobjectid2f266a154ddd7c23mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA ! persist_js_state·has_pluto_hook_features§cell_id$f7433324-acc3-49a5-b5b3-ada0c8f09d52depends_on_disabled_cells§runtime2]published_object_keysdepends_on_skipped_cellsçerrored$0c9986bb-54c0-4b08-9c29-4bfb0b68b54equeued¤logsrunning¦outputbody
Exercise 13.2
Generalize the proof of the policy gradient theorem and the steps leading to the REINFORCE update equation (13.8), so that (13.8) ends up with a factor of $\gamma^t$ and thus aligns with the general algorithm given in the pseudocode.
See proof above in the section on proving the policy gradient theorem.
So, in order to maximize the value at state 1, we have $p_{\text{left}} \approx 0.414$ and $p_{\text{right}} \approx 0.586$. That also implies that $v_1 = -2\frac{1+p}{p(1-p)} = -2\frac{\sqrt{2}}{(\sqrt{2}-1)(2 - \sqrt{2})}= \frac{-2\sqrt{2}}{2 \sqrt{2} - 2 - 2 + \sqrt{2}} = \frac{-2 \sqrt{2}}{3\sqrt{2} - 4} \approx -11.657$
mimetext/htmlrootassigneelast_run_timestampA -persist_js_state·has_pluto_hook_features§cell_id$38e5d800-4d43-40d2-87ea-f7d4b4283dabdepends_on_disabled_cells§runtime %published_object_keysdepends_on_skipped_cells§errored$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2queued¤logsrunning¦outputbodyFone_step_actor_critic_binary_features (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 'U[persist_js_state·has_pluto_hook_features§cell_id$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2depends_on_disabled_cells§runtime 7published_object_keysdepends_on_skipped_cells§errored$73b90260-d57a-449a-8db6-47f91e6a4e4fqueued¤logsrunning¦outputbodyM
Eligibility Vector with Binary Features
mimetext/htmlrootassigneelast_run_timestampA װpersist_js_state·has_pluto_hook_features§cell_id$73b90260-d57a-449a-8db6-47f91e6a4e4fdepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$5aba4f96-e877-457e-8e95-18737348f99fqueued¤logsrunning¦outputbodyCactor_critic_fcann_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA l۰persist_js_state·has_pluto_hook_features§cell_id$5aba4f96-e877-457e-8e95-18737348f99fdepends_on_disabled_cells§runtimehzpublished_object_keysdepends_on_skipped_cells§errored$fed4dc4c-0d1c-4ee3-9d0e-8ef2a7db7486queued¤logsrunning¦outputbodyd<
mimetext/htmlrootassigneelast_run_timestampA Wipersist_js_state·has_pluto_hook_features§cell_id$966ef17c-23be-49dc-bc37-4cb52b34c049depends_on_disabled_cells§runtime Ƞpublished_object_keysdepends_on_skipped_cells§errored$e7e49ff8-32df-48a4-afb2-462859592e92queued¤logsrunning¦outputbodyGform_state_and_policy_function_outputs (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 5?persist_js_state·has_pluto_hook_features§cell_id$e7e49ff8-32df-48a4-afb2-462859592e92depends_on_disabled_cells§runtime ܱpublished_object_keysdepends_on_skipped_cells§errored$78c83673-2117-4542-b4d8-1c243e8f610bqueued¤logsrunning¦outputbody
Eligibility Vector
Recall for the gaussian case and linear approximation we had:
mimetext/htmlrootassigneelast_run_timestampA A,persist_js_state·has_pluto_hook_features§cell_id$a6be9a4c-d43b-4867-b7a2-07a46a9d0d8fdepends_on_disabled_cells§runtime$Spublished_object_keysdepends_on_skipped_cellsçerrored$396e0047-d848-462f-a769-0cc2829abc78queued¤logsrunning¦outputbodyelementsaction_probabilitiesprefixFloat32elements0.00209303text/plain0.997907text/plaintypeArrayprefix_shortobjectid2bca965e70e9689c!application/vnd.pluto.tree+objectstate_value_estimate-160.412text/plaintypeNamedTupleobjectid860e53ebb42149aemime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA +&`ٰpersist_js_state·has_pluto_hook_features§cell_id$396e0047-d848-462f-a769-0cc2829abc78depends_on_disabled_cells§runtime&0published_object_keysdepends_on_skipped_cellsçerrored$ff4f977e-48df-4c12-845c-c245b4d39d6dqueued¤logsrunning¦outputbodyEactor_critic_linear_parameter_study (generic function with 3 methods)mimetext/plainrootassigneelast_run_timestampA +d
persist_js_state·has_pluto_hook_features§cell_id$ff4f977e-48df-4c12-845c-c245b4d39d6ddepends_on_disabled_cells§runtime |$published_object_keysdepends_on_skipped_cells§errored$aa450da4-fe84-4eea-b6c4-9820b7982437queued¤logsrunning¦outputbody'
With continuous policy parametrization, we can smoothly very action selection probabilities by arbitrarily small amounts, something that was not possible with ϵ-greedy action selection. Therefore stronger convergence guarantees are possible for policy-gradient methods than for action-value methods.
In the episodic case, assuming some particular non-random starting state $s_0$, we define the performance of a policy parametrized by θ as:
where $v_{\pi_\mathbf{\theta}}$ is the true value function for $\pi_\mathbf{\theta}$, the policy determined by $\mathbf{\theta}$.
The policy gradient theorem provides an analytic expression for the gradient of performance with respect to the policy parameter that does not involve the derivative of the state distribution:
$$\begin{align}
\nabla J(\mathbf{\theta}) \propto \sum_s \mu (s) \sum_a q_\pi (s, a) \nabla \pi (a|s,\mathbf{\theta}) \tag{13.5}
\end{align}$$
where the gradients are column vectors of partial derivatives with respect to the components of $\mathbf{\theta}$. In the episodic case, the constant of proportionality is the average length of an episode, and in the continuing case it is 1. The distribution here $\mu$ is the on-policy distribution under $\pi$.
REINFORCE on short-corridor gridworld (Example 13.1). Performance varies with step size but can approach the ideal. Feature vector encodes every state identically.
Now consider $\mathbf{h} = \theta ^ \top \mathbf{x}$ with $h_a = \mathbf{h}_a$. Since the parameters are now represented as a matrix, we can also index the gradient partial derivatives such that $\nabla \left ( f(\theta) \right )_{i, j} = \frac{\partial f(\theta)}{\theta_{i, j}}$
mimetext/htmlrootassigneelast_run_timestampA jpersist_js_state·has_pluto_hook_features§cell_id$3cfd63ad-b1a2-4b99-ae97-2ff10351e4f5depends_on_disabled_cells§runtime ǵpublished_object_keysdepends_on_skipped_cells§errored$31db0f58-28e4-454f-9394-25565687266fqueued¤logsrunning¦outputbody b
mimetext/htmlrootassigneelast_run_timestampA 0 persist_js_state·has_pluto_hook_features§cell_id$31db0f58-28e4-454f-9394-25565687266fdepends_on_disabled_cells§runtime
aGpublished_object_keysdepends_on_skipped_cellsçerrored$822e4d69-2582-4956-858e-06ecb091e76aqueued¤logsrunning¦outputbody9display_cartpole_episode (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 0`Ұpersist_js_state·has_pluto_hook_features§cell_id$822e4d69-2582-4956-858e-06ecb091e76adepends_on_disabled_cells§runtime HFpublished_object_keysdepends_on_skipped_cellsçerrored$d7f6ff79-3c0f-4f16-aa1c-3bc534ce580aqueued¤logsrunning¦outputbody
mimetext/htmlrootassigneelast_run_timestampA `persist_js_state·has_pluto_hook_features§cell_id$05b0fcad-628b-48d2-aa24-f6f562dbb660depends_on_disabled_cells§runtime {published_object_keysdepends_on_skipped_cells§errored$d2729657-d0bf-4d39-8ec7-f242a1ad48d6queued¤logsrunning¦outputbodyJcreate_continuous_action_mountaincar_beta (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA >tpersist_js_state·has_pluto_hook_features§cell_id$d2729657-d0bf-4d39-8ec7-f242a1ad48d6depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$5c11a92d-7496-4aba-af15-2537eac49dd7queued¤logsrunning¦outputbody!Array{Vector{T}, 1} where T<:Realmimetext/plainrootassigneelast_run_timestampA NQpersist_js_state·has_pluto_hook_features§cell_id$5c11a92d-7496-4aba-af15-2537eac49dd7depends_on_disabled_cells§runtime >cpublished_object_keysdepends_on_skipped_cells§errored$1753b5ed-c00b-4b60-b492-822180778e8cqueued¤logsrunning¦outputbody>update_linear_value_gradient! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$1753b5ed-c00b-4b60-b492-822180778e8cdepends_on_disabled_cells§runtime qpublished_object_keysdepends_on_skipped_cells§errored$f7ede764-5ad8-426b-a805-cc21b622d977queued¤logsrunning¦outputbody5
mimetext/htmlrootassigneelast_run_timestampA ? persist_js_state·has_pluto_hook_features§cell_id$36d514fa-b27a-4c6b-8399-9d108377b9b5depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cellsçerrored$6b1acb57-159a-4b7f-99fe-5f996522243bqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestamp persist_js_state·has_pluto_hook_features§cell_id$6b1acb57-159a-4b7f-99fe-5f996522243bdepends_on_disabled_cellsçruntimepublished_object_keysdepends_on_skipped_cells§errored$45f0a385-6465-4acc-8637-1b007a0fe215queued¤logsrunning¦outputbodyAupdate_fcann_eligibility_vector! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$45f0a385-6465-4acc-8637-1b007a0fe215depends_on_disabled_cells§runtime fpublished_object_keysdepends_on_skipped_cells§errored$c52c4cec-0ea8-4af3-831a-d284f0e086eequeued¤logsrunning¦outputbody-
mimetext/htmlrootassigneelast_run_timestampA @-Upersist_js_state·has_pluto_hook_features§cell_id$c52c4cec-0ea8-4af3-831a-d284f0e086eedepends_on_disabled_cells§runtimeM published_object_keysdepends_on_skipped_cellsçerrored$f8614042-7c94-4d47-a1b6-4e96676b4e8bqueued¤logsrunning¦outputbodyLactor_critic_fcann_episodic_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA /qްpersist_js_state·has_pluto_hook_features§cell_id$f8614042-7c94-4d47-a1b6-4e96676b4e8bdepends_on_disabled_cells§runtime ~published_object_keysdepends_on_skipped_cellsçerrored$76eb6743-cac0-4174-9ba3-a0691c200b54queued¤logsrunning¦outputbody:make_n_param_dist_params (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA '$:persist_js_state·has_pluto_hook_features§cell_id$76eb6743-cac0-4174-9ba3-a0691c200b54depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$94517664-6988-44dc-a297-e9d5873ee540queued¤logsrunning¦outputbodyV
mimetext/htmlrootassigneelast_run_timestampA 2zpersist_js_state·has_pluto_hook_features§cell_id$cb70d400-3e9c-441c-b17c-e727e8c928f3depends_on_disabled_cells§runtime Xpublished_object_keysdepends_on_skipped_cellsçerrored$e034b9cb-f4ee-46f4-bea6-72c93c75d966queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA %zpersist_js_state·has_pluto_hook_features§cell_id$e034b9cb-f4ee-46f4-bea6-72c93c75d966depends_on_disabled_cells§runtime `?9published_object_keysdepends_on_skipped_cells§errored$e6cf9550-2e69-4b82-92cf-5e07a35490aaqueued¤logsrunning¦outputbody.zero_params! (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA ;persist_js_state·has_pluto_hook_features§cell_id$e6cf9550-2e69-4b82-92cf-5e07a35490aadepends_on_disabled_cells§runtime ˵published_object_keysdepends_on_skipped_cells§errored$717e4c69-59d5-4929-923f-dd35a97fb160queued¤logsrunning¦outputbodypactor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA /ͳհpersist_js_state·has_pluto_hook_features§cell_id$717e4c69-59d5-4929-923f-dd35a97fb160depends_on_disabled_cells§runtime .ŵpublished_object_keysdepends_on_skipped_cells§errored$1386ffdb-940d-4f1b-a872-4e38647b5335queued¤logsrunning¦outputbody
Test One-step Actor-Critic
The following function calls execute the One-step Actor-Critic algorithm on Example 13.1. The output displayed is the policy function acting on the single state representation for the problem. The two values represent the probability of taking the left and right action respectively. If converged properly, the right action probability should be higher, approaching a value of about 60%.
Consider $\mathbf{x}(s)$ and $\mathbf{h}(s, \boldsymbol{\theta})$ which produces a vector of action preferences. We would like to derive an expression for $\nabla \ln \pi (a \vert s, \boldsymbol{\theta})$ in the case of $\mathbf{\pi}(s, \boldsymbol{\theta}) = \sigma(\mathbf{h}(s, \boldsymbol{\theta}))$ where $\sigma(\mathbf{x})$ is the softmax function defined in section 13.1. Here I'm using the notation $\mathbf{\pi}(s, \boldsymbol{\theta})$ to refer to the vector of action probabilities at a given state. The subscript on the vector refers to selecting that element from the vector. To shorten expressions, the following terms are equivalent:
$$\begin{flalign}
\mathbf{\pi} &\doteq \mathbf{\pi}(s, \boldsymbol{\theta}) \\
\mathbf{h} &\doteq \mathbf{h}(s, \boldsymbol{\theta}) \\
x_i &\doteq \mathbf{x}_i \text{ for all vectors} \\
\end{flalign}$$
Using these conventions, we previously had an expression for the ith component of the gradient of the policy:
Classification problems involve training a function to predict the class label of an input. The function returns a vector of class preferences which can be converted to a probability distribution by the soft-max function. The cross-entropy loss is a way of comparing this distribution with the desired output label to generate an error value.
Let's denote $\mathbf{p}(s)$ as the vector of true probabilities for an example $s$ and keep our output function as $\pi(s,\theta) = \sigma(\mathbf{h}(s, \boldsymbol{\theta}))$. The cross entropy loss is defined as:
In a typical situation with a dataset, $\mathbf{p}(s)$ will be a one-hot vector representing the index of label of the example in the dataset. Let's call that index $a$ such that $p_a = 1$ and $p_i = 0 \: \forall i \neq a$. The loss then simplifies to $\mathcal{L}(a, \mathbf{\pi}) = -\ln \mathbf{\pi}_a$. When we train with gradient descent on such a dataset, we must compute the gradient of this loss with respect to the parameters or $-\nabla \ln \pi_a$ which is just negative one times the eligibility vector for general paramaterized approximation. So if we have a function that computes the gradient of the cross entropy loss of the soft-max output for a vector function and a label index, we can replace the label index of the dataset with the desired action index $a$ and then that gradient will match our desired gradient after multiplying by negative one.
mimetext/htmlrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$58403c8e-0ee4-4466-ba25-ee0c86fb0b47depends_on_disabled_cells§runtime
published_object_keysdepends_on_skipped_cells§errored$e1aec891-d95a-47d1-97d7-d2a4cfb16e64queued¤logsrunning¦outputbodyGsetup_fcann_policy_and_value_arguments (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA -persist_js_state·has_pluto_hook_features§cell_id$e1aec891-d95a-47d1-97d7-d2a4cfb16e64depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$3d065608-eef2-4caa-b17d-ec60714e3d58queued¤logsrunning¦outputbodySactor_critic_binary_episodic_beta_parameter_study (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA /persist_js_state·has_pluto_hook_features§cell_id$3d065608-eef2-4caa-b17d-ec60714e3d58depends_on_disabled_cells§runtime 0Tpublished_object_keysdepends_on_skipped_cells§errored$b87ff1a9-abff-40f7-a1d8-f751a1c8b060queued¤logsrunning¦outputbodyS
In the episodic case, we provided a reward of -1 per step and then considered an episode finished when a failure state was reached. In the continuing case, the step function will provide a reward of 0 unless a failure occurs in which case it will provide a reward of -1 and then initialize a new state.
mimetext/htmlrootassigneelast_run_timestampA #persist_js_state·has_pluto_hook_features§cell_id$b87ff1a9-abff-40f7-a1d8-f751a1c8b060depends_on_disabled_cells§runtime #published_object_keysdepends_on_skipped_cells§errored$e89bdc84-dbb5-4c73-a39c-6392e5f79704queued¤logsrunning¦outputbody W
Note that for the corridor problem, the state-value learning rates have very little impact and learning is most effective when $\lambda_{\boldsymbol{\theta}}$ is close to 1 which mimics REINFORCE with baseline.
A Bernoulli-logistic unit is a stochastic neuron-like unit used in some ANNs. Its input at time t is a feature vector $\mathbf{x}(S_t)$; its output, $A_t$, is a random variable having two values, 0 and 1, with $\Pr \{A_t=1 \}=P_t$ and $\Pr\{A_t=0\}=1-P_t$ (the Bernoulli distribution). Let $h(s, 0, \mathbf{\theta})$ and $h(s, 1, \mathbf{\theta})$ be the preferences in state $s$ for the unit's two actions given by policy parameter $\mathbf{\theta}$. Assume that the difference between the action preferences is given by a weights sum of teh unit's input vector, that is, assume that $h(s, 1, \mathbf{\theta})-h(s,0, \mathbf{\theta}) = \mathbf{\theta}^\top \mathbf{x}(s)$, where $\mathbf{\theta}$ is the unit's weight vector.
Show that if the exponential soft-max distribution (13.2) is used to convert action preferences to policies, then ${P_t = \pi(1|S_t, \theta_t)=1/(1+\exp(-\theta_t^\top\mathbf{x}(S_t)))}$ (the logistic function).
What is the Monte-Carlo REINFORCE update of $\theta_t$ to $\theta_{t+1}$ upon receipt of return $G_t$?
Express the eligility $\nabla \ln \pi(a|s, \theta)$ for a Bernoulli-logistic unit, in terms of $a$, $\mathbf{x}(s)$, and $\pi(a|s, \theta)$ by calculating the gradient.
Hint for part (c): Define $P=\pi(1|s,\theta)$ and compute the derivative of the logarithm, for each action, using the chain rule on $P$. Combine the two results into one expression that depends on $a$ and $P$, and then use the chain rule again, this time on $\theta^\top\mathbf{x}(s)$, noting that the derivative of the logistic function $f(x)=1/(1+e^{-x})$ is $f(x)(1-f(x))$.
In order to prove the policy gradient theorem, we must manipulate terms that are probability distributions over states and visit steps. In order to build intuition for these distributions, we can visualize how data is being averaged with the sort corridor example. The following function simulates many episodes in the environment with a stochastic policy that has some probability of moving left regardless of the state. The simulation keeps track of the visit count for a given state and the visit step. The result of the accumulation is a matrix who's columns contain the number of times each state was visited on every step of an episode across all of the simulated episodes. If we divide each count by the number of episodes simulated, then we have an unbiased sample of the probability of visiting a state on each step $k$ of an episode: $\Pr \{ S_k = s \mid \pi \}$ such that $\sum_{s \in \mathcal{S}^+} \Pr \{ S_k = s \mid \pi \} = 1$.
Note that this distribution is only normalized over the sum of all states including terminal states which is denoted in episodic problems by the notation $\mathcal{S}^+$. The notation $\mathcal{S}$ excludes all terminal states, so if we sum the above probabilities over that set on a given step $k$ we calculate the probability that we are NOT in a terminal state by the time we reach step $k$: $\sum_\mathcal{S} \Pr \{ S_k = s \mid \pi \} = \Pr \{ T \gt k \mid \pi \}$ where we use the notation that $T$ is the step of termination for a particular episode.
mimetext/htmlrootassigneelast_run_timestampA qzpersist_js_state·has_pluto_hook_features§cell_id$6c5e9bb2-4c38-4613-9652-dec99e97b512depends_on_disabled_cells§runtime Opublished_object_keysdepends_on_skipped_cells§errored$b0a66a19-ee76-463b-a704-8fcee85444d0queued¤logsrunning¦outputbody>update_params_with_gradient! (generic function with 4 methods)mimetext/plainrootassigneelast_run_timestampA spersist_js_state·has_pluto_hook_features§cell_id$b0a66a19-ee76-463b-a704-8fcee85444d0depends_on_disabled_cells§runtime ^ߞpublished_object_keysdepends_on_skipped_cells§errored$13ebc12f-ff6f-4266-88d3-28d6df5fcf59queued¤logsrunning¦outputbodyWactor_critic_binary_episodic_gaussian_parameter_study (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA /persist_js_state·has_pluto_hook_features§cell_id$13ebc12f-ff6f-4266-88d3-28d6df5fcf59depends_on_disabled_cells§runtime 2xpublished_object_keysdepends_on_skipped_cells§errored$7a6fb1f0-fc3c-4c29-a6d9-769d32ca98a9queued¤logsrunning¦outputbodyK
Example 13.1 Short corridor gridworld
mimetext/htmlrootassigneelast_run_timestampA (Rpersist_js_state·has_pluto_hook_features§cell_id$7a6fb1f0-fc3c-4c29-a6d9-769d32ca98a9depends_on_disabled_cells§runtime µpublished_object_keysdepends_on_skipped_cells§errored$f2f2dd1d-180c-4d36-b515-5079d129f93aqueued¤logsrunning¦outputbody $
mimetext/htmlrootassigneelast_run_timestampA ڰpersist_js_state·has_pluto_hook_features§cell_id$f2f2dd1d-180c-4d36-b515-5079d129f93adepends_on_disabled_cells§runtimeΜ published_object_keysdepends_on_skipped_cellsçerrored$553b0ceb-f2ca-41ee-99bc-9f53a4487b49queued¤logsrunning¦outputbody11.67104mimetext/plainrootassigneelast_run_timestampA &persist_js_state·has_pluto_hook_features§cell_id$553b0ceb-f2ca-41ee-99bc-9f53a4487b49depends_on_disabled_cells§runtime apublished_object_keysdepends_on_skipped_cellsçerrored$f9facbba-39d4-483e-9066-275603156db0queued¤logsrunning¦outputbody8plot_mountaincar_values (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA @K
persist_js_state·has_pluto_hook_features§cell_id$f9facbba-39d4-483e-9066-275603156db0depends_on_disabled_cells§runtime <ɵpublished_object_keysdepends_on_skipped_cellsçerrored$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0queued¤logsrunning¦outputbodyprefixFloat32elements0.999969text/plain3.11996f-5text/plaintypeArrayprefix_shortobjectidb2757275efe1e0camime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA +5persist_js_state·has_pluto_hook_features§cell_id$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0depends_on_disabled_cells§runtime mpublished_object_keysdepends_on_skipped_cellsçerrored$d41f1dd1-45fe-4456-9a01-ed47fd6704a7queued¤logsrunning¦outputbodyAupdate_beta_eligibility_vector! (generic function with 4 methods)mimetext/plainrootassigneelast_run_timestampA #^V persist_js_state·has_pluto_hook_features§cell_id$d41f1dd1-45fe-4456-9a01-ed47fd6704a7depends_on_disabled_cells§runtime $epublished_object_keysdepends_on_skipped_cells§errored$ba5d6311-daee-4abc-b2fb-fae2184ef3ebqueued¤logsrunning¦outputbodyGsetup_binary_gaussian_policy_arguments (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA ''Epersist_js_state·has_pluto_hook_features§cell_id$ba5d6311-daee-4abc-b2fb-fae2184ef3ebdepends_on_disabled_cells§runtime epublished_object_keysdepends_on_skipped_cells§errored$8e742d32-c074-4981-b35b-b596b64c869bqueued¤logsrunning¦outputbodydD
$\lambda_\theta$:
$\lambda_\mathbf{w}$:
$\alpha_{\overline{r}}$:
$\log_2 \alpha_\theta$ min:
$\log_2 \alpha_{\mathbf{w}}$ min:
mimetext/htmlrootassigneelast_run_timestampA [}persist_js_state·has_pluto_hook_features§cell_id$8e742d32-c074-4981-b35b-b596b64c869bdepends_on_disabled_cells§runtime lpublished_object_keysdepends_on_skipped_cellsçerrored$03a218cb-aa83-4000-85b5-c6f247087053queued¤logsrunning¦outputbody>update_binary_value_gradient! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA Opersist_js_state·has_pluto_hook_features§cell_id$03a218cb-aa83-4000-85b5-c6f247087053depends_on_disabled_cells§runtime
Upublished_object_keysdepends_on_skipped_cells§errored$1ec1acf1-f833-4478-9b3c-88029340a629queued¤logsrunning¦outputbodyV
Non-linear Features
This version of REINFORCE uses non-linear features in a fully connected neural network. The number of parameters no longer matches the size of the input feature vector, but a mapping from state to feature vector is still required. One must specify the size of the feature vector, a function that updates the values in a feature vector given a state, and the size of each hidden layer in the neural network. Additional keyword arguments are available to change the construction of the neural network such as adding residual layers.
If the action space is small enough, then it may be convenient to create a function that simply outputs the preferences for all of the actions at a given state. Let's call $N_a$ to be the number of available actions. We would then consider the vector function $\mathbf{h}(s, \boldsymbol{\theta}) \in \mathbb{R}^{N_a}$ and its components $h_1, h_2, h_3, \dots, h_{N_a}$. To be the action preferences at each state. With this style of paramaterization, we need only compute state feature vectors $\mathbf{x}(s) \in \mathbb{R}^d$.
Similarly, the policy function would also be a vector function. In order to compute the softmax, we must evaluate the denominator of (13.2) which requires knowing all of the action preferences. Practically, it is only defined as a function on vectors, so consider the following notation to simplify expressions where we use the symbol $\mathbf{\sigma}$ to denote the soft-max vector function.
$$\sigma(\mathbf{x}) = \frac{e^{\mathbf{x}}}{\sum_j{e^{x_j}}} \text{ where we abuse the notation } e^{\mathbf{x}} = \begin{pmatrix}
e^{x_1} \\
e^{x_2} \\
\vdots \\
e^{x_n}
\end{pmatrix}$$
Using this notation, we can write down the policy function under this new parameterization: $\mathbf{\pi}(s, \boldsymbol{\theta}) = \mathbf{\sigma}(\mathbf{h}(s, \boldsymbol{\theta}))$. What do linear preferences look like with this parameterization? Instead of a parameter vector $\boldsymbol{\theta} \in \mathbb{R}^{d^\prime}$, we have a parameter matrix $\boldsymbol{\theta} \in \mathbb{R}^{d \times N_a}$ and the vector of preferences is the result of a matrix vector multiplication: $\mathbf{h}(s, \boldsymbol{\theta}) = \theta^\top \mathbf{x}(s) \in \mathbb{R}^{N_a}$. Subscript notation is used to refer to single preference values so $\mathbf{h}_i$ would be the $ith$ index of $\mathbf{h}$ for the $ith$ action preference equivalent to $h_i$.
In the continuing case we need to define the average reward per time step as discussed in Section 10.3. In the update procedure the δ is calculated differently in terms of the reward compared to this long running average. The value functions in this case will also learn the reward difference from the average which is assumed to have a well defined expected value under the stationary state distribution for the policy. This shift in the value function will not affect performance since shifting the value function up and down by a constant does not affect the learned policy. To implement this we need a new learning rate $α_{\overline{R}}$ which controls how quickly the reward average updates. This replaces $γ$ in a sense since we no longer discount rewards of future time steps.
This policy requires μ and σ to be parameterized by the parameter vector. To make a linear model for both parameters we can use the following formulas:
where $\mathbf{x}_\mu(s)$ and $\mathbf{x}_\sigma(s)$ are state feature vectors. With these formulas we can apply the previous algorithms to solve environments with real-valued actions.
Adding a baseline to REINFORCE can make it learn much faster as illustrated here on the short-corridor gridworld (Example 13.1). Here the approximate state-value function used in the baseline is $\hat v(s, \mathbf{w}) = w$. There is only one component of the feature vector and the state value approximation parameters.
This version of REINFORCE uses binary feature vectors for which one needs to specify the total number of features as well as a function that returns the active features for a given state.
mimetext/htmlrootassigneelast_run_timestampA ڰpersist_js_state·has_pluto_hook_features§cell_id$48b342f2-e48f-457a-9bd3-b3504a79f3a6depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$5d50a5d0-8fe2-4c6e-b76c-d5614e4fd884queued¤logsrunning¦outputbody4show_or_lookup_plot (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA >ѿpersist_js_state·has_pluto_hook_features§cell_id$5d50a5d0-8fe2-4c6e-b76c-d5614e4fd884depends_on_disabled_cells§runtime :澵published_object_keysdepends_on_skipped_cells§errored$ba645f6b-143f-4e83-9003-707770ae308dqueued¤logsrunning¦outputbody
Probability distributions for short corridor gridworld example with probability of left action selected below
To use policy gradient methods, we must be able to take the gradient of the policy function for every state-action pair. Using the above notation and treating the policy as a vector function, we must know the gradient of the soft-max applied to a vector function at a particular index. Each gradient is a column vector of length $d$ where $d$ is the number of parameters. There is a separate gradient available for every index in the vector output which is one for each action or a total of $N_a$. To simplify expressions, $\mathbf{h}(s, \boldsymbol{\theta})$ will we written as $\mathbf{h}$ and $\mathbf{\pi} = \mathbf{\sigma}(\mathbf{h})$. Our desired gradient is with respect to a particular component of $\mathbf{\sigma}(\mathbf{h})$ denoted $\mathbf{\sigma}(\mathbf{h})_a$ where $a$ represents the action index. The gradient itself is the vector of partial derivatives with respect to the parameters $\theta$. The $ith$ component of the gradient $\nabla(f(\theta))_i = \frac{\partial f(\theta)}{\partial \theta_i}$. When we compute the gradient we need all the components whose expression is derived below.
The final step results form the fact that the policy function is a probability distribution so the sum over it is always 1.
mimetext/htmlrootassigneelast_run_timestampA tpersist_js_state·has_pluto_hook_features§cell_id$d95f75b5-21d8-4862-baa7-50b58d9725b8depends_on_disabled_cells§runtime ̦published_object_keysdepends_on_skipped_cells§errored$65be0e58-24be-4932-92a9-9e4825b14144queued¤logsrunning¦outputbodybactor_critic_binary_continuing_squashed_gaussian_parameter_study (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA /[persist_js_state·has_pluto_hook_features§cell_id$65be0e58-24be-4932-92a9-9e4825b14144depends_on_disabled_cells§runtime &spublished_object_keysdepends_on_skipped_cells§errored$60c21e9c-e42d-4f0b-a910-3b318440fbc8queued¤logsrunning¦outputbody:
Normal Distribution Plot with
$\mu$:
$\sigma$:
mimetext/htmlrootassigneelast_run_timestampA !Nepersist_js_state·has_pluto_hook_features§cell_id$60c21e9c-e42d-4f0b-a910-3b318440fbc8depends_on_disabled_cells§runtimeƵpublished_object_keysdepends_on_skipped_cellsçerrored$da2d3186-a778-41cc-9b49-759bf1e9b8faqueued¤logsrunning¦outputbodyٚUnion{AbstractVector{I} where I, C1, C2, C3} where {I<:Integer, C1<:AbstractVector{I}, N, C2<:NTuple{N, I}, T<:AbstractVector{I}, C3<:(Base.Generator{T})}mimetext/plainrootassigneelast_run_timestampA ՙpersist_js_state·has_pluto_hook_features§cell_id$da2d3186-a778-41cc-9b49-759bf1e9b8fadepends_on_disabled_cells§runtime [published_object_keysdepends_on_skipped_cells§errored$b695ef21-a1ac-4d1f-a0e1-71cd81cede18queued¤logsrunning¦outputbody ݽ
mimetext/htmlrootassigneelast_run_timestampA @㍵persist_js_state·has_pluto_hook_features§cell_id$b695ef21-a1ac-4d1f-a0e1-71cd81cede18depends_on_disabled_cells§runtime J=Եpublished_object_keysdepends_on_skipped_cellsçerrored$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00queued¤logsrunning¦outputbodymreinforce_with_baseline_monte_carlo_control_binary_features_gaussian_actions (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA ';persist_js_state·has_pluto_hook_features§cell_id$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00depends_on_disabled_cells§runtime =>published_object_keysdepends_on_skipped_cells§errored$dcb306ae-a1b1-43d6-ba6e-e38668838689queued¤logsrunning¦outputbodyF
where $\eta$ is the average length of an episode. The quantity inside the brackets is the probability that an episode has not terminated by step k and follows from the fact that the sum over states in $\mathcal{S}$ is over the set of non-terminal states. If the sum was over $\mathcal{S}^+$ instead then it would be infinite since the first sum term would be 1 for every k. Normally to calculate $\eta$, we would use the expected value with the probability of an episode lasting exactly $k$ steps, but the probability we have access to here is actually the distribution function, not the density function. That is $\Pr \{s_0 \rightarrow S_T, k, \pi \} = \sum_{t = 0}^k \Pr \{ T = t \} = \Pr \{ T \leq k \}$ where $T$ is the length of an episode. Using these probabilities, we can write $\eta = \mathbb{E}_\pi [T] = \sum_{k = 0}^\infty k \Pr \{ T = k \} = \Pr \{T = 1 \} + 2 \Pr \{T = 2 \} + \cdots$.
Earlier we had the expression $\eta = \sum_{k = 0}^\infty \left [ 1 - \Pr \{s_0 \rightarrow S_T, k, \pi \} \right ] = \sum_{k = 0}^\infty \Pr \{T \gt k \} = \sum_{k = 0}^\infty \sum_{t = k + 1}^\infty \Pr \{T = t \}$
We can stack up the terms of this double sum to see that it is equivalent to the expected value calcuation from before:
$$\begin{flalign}
\Pr \{ T = 1 \} + \Pr \{ T = 2 \} + &\Pr \{ T = 3 \} +\cdots \\
\Pr \{ T = 2 \} + &\Pr \{ T = 3 \} + \cdots \\
&\Pr \{ T = 3 \} + \cdots \\
\vdots
\end{flalign}$$
If we count terms along the diagonal, we see that each value of $k$ has exactly $k$ terms, matching the expected value calculation.
What if we wanted to calculate the bivariate distribution over states and steps where we ignore the terminal states $\mu_\pi(s, k)$ such that $\sum_{s \in \mathcal{S}} \sum_k \mu_\pi(s, k) = 1$. This probability represents the chance of sampling a particular step and state simultaneously from a unbiased sample of non-terminal states in an episode. Luckily we can break down this probability into two components: 1) the probability of reaching a step k without terminating 2) the probability of being in a non-terminal state on step k. We saw already that 1) is just $\sum_{s \in \mathcal{S}} \Pr \{ s_0 \rightarrow s, k, \pi \}$ and 2) we can calculate by normalizing those probabilities over only the non-terminal states: $\frac{\Pr \{ s_0 \rightarrow s, k, \pi \}}{\sum_{s \in \mathcal{S}} \Pr \{ s_0 \rightarrow s, k, \pi \} }$. By multiplying these two together we see that the probability is just the original distribution but where the domain of possible input values is $s \in \mathcal{S}$ and all possible steps $k$. Therefore, we can transform this into a normalized bivariate distribution by dividing by its sum over those two sets:
Now that we have established the relationship between the on-policy distribution function and the probability expression we have, we can use it to complete the proof below.
This version of REINFORCE uses linear feature vectors for which one needs to specify the total number of features as well as a function that updates the values in a feature vector given a state.
mimetext/htmlrootassigneelast_run_timestampA 8
fpersist_js_state·has_pluto_hook_features§cell_id$87feff3e-e510-4916-91a9-db3a2cd12225depends_on_disabled_cells§runtime jpublished_object_keysdepends_on_skipped_cellsçerrored$5261651e-a51e-4e80-8e23-83a4c10e5259queued¤logsrunning¦outputbodyEupdate_gaussian_eligibility_vector! (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA #Eհpersist_js_state·has_pluto_hook_features§cell_id$5261651e-a51e-4e80-8e23-83a4c10e5259depends_on_disabled_cells§runtime wpublished_object_keysdepends_on_skipped_cells§errored$dddc4a2f-34b2-41dc-85b3-55aba4880fa6queued¤logsrunning¦outputbodymsg٦UndefVarError: `reinforce_test` not defined in `Main.var"workspace#8"`
Suggestion: add an appropriate import or assignment. This global was declared but not assigned.stacktracecall_shorttop-level scopeinlined£urlpath/home/runner/work/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Chapter-13/Chapter_13_Policy_Gradient_Methods.jl#==#dddc4a2f-34b2-41dc-85b3-55aba4880fa6source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefileMChapter_13_Policy_Gradient_Methods.jl#==#dddc4a2f-34b2-41dc-85b3-55aba4880fa6functop-level scopeparent_modulefrom_c¤mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA 0Mpersist_js_state·has_pluto_hook_features§cell_id$dddc4a2f-34b2-41dc-85b3-55aba4880fa6depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cellsçerrored$54fff14b-cf53-47b0-9cfa-8b9ee33df54equeued¤logsrunning¦outputbodyBinaryBetaEligibilityVectormimetext/plainrootassigneelast_run_timestampA !persist_js_state·has_pluto_hook_features§cell_id$54fff14b-cf53-47b0-9cfa-8b9ee33df54edepends_on_disabled_cells§runtime KCǵpublished_object_keysdepends_on_skipped_cells§errored$023f67b8-8f38-470a-9766-ac60a75678aaqueued¤logsrunning¦outputbodyelementsfeature_vectorprefixFloat32elements0.0text/plain0.0text/plaintypeArrayprefix_shortobjectid96151aba818dc130!application/vnd.pluto.tree+objectnum_features2text/plainupdate_feature_vector!update_feature_vector!text/plaintypeNamedTupleobjectid534e5799b20b5f5bmime!application/vnd.pluto.tree+objectrootassigneeconst mountaincar_fcann_setuplast_run_timestampA : jpersist_js_state·has_pluto_hook_features§cell_id$023f67b8-8f38-470a-9766-ac60a75678aadepends_on_disabled_cells§runtime S~published_object_keysdepends_on_skipped_cells§errored$1558cec1-c4fd-4bc0-85ed-ae22c6067d41queued¤logsrunning¦outputbody:
We can also repeat this derivation for the alternative linear parameterization where we only have state feature vectors and a parameter matrix with components $\boldsymbol{\theta}_{i, j}$:
Notice now that all of the parameters associated with the state-value estimate are irrelevent since they always cancel out in the parameter update. Even though we have added a parameter, this method effectively removes two from the analysis. Also, we seem to actually benefit from an intermediate value of $\lambda_{\boldsymbol{\theta}}$ unlike in the episodic case where using the Monte Carlo method was always the best.
where $\mu$ is the steady-state distribution under $\pi$, $\mu(s) \doteq \lim_{t \rightarrow \infty} \Pr \{ S_t = s \vert A_{0:t} \sim \pi \}$, which is assumed to exist and to be independent of $S_0$ (an ergodicity assumption). Remember that this is the special distribution under which, if you select actions according to $\pi$, you remain the same distribution:
Naturally, in the continuing case, we define values, $v_\pi(s) \doteq \mathbb{E}_\pi [G_t \vert S_t = s]$ and $q_\pi(s, a) \doteq \mathbb{E}_\pi[G_t \vert S_t = s, A_t = a]$, with respect to the differential return:
With these alternate definitions, the policy gradient theorem as given for the episodic case (13.5) remains true for the continuing case. See proof below:
So, in order to maximize the value at state 2, we have $p_{\text{left}} \approx 0.4495$ and $p_{\text{right}} \approx 0.5505$. Which is different from the value we got for state 1. So There is a different optimal policy depending on the starting state. It should be obvious for example that starting in the third state results in an optimial policy of choosing the right action every time. The value functions for each state are plotted below. The behavior of $v_3$ is not well defined at $p=0$ because for any finite $v_2$ it should be 0 but the limit approaching from the right side is -3. This is because for $p=0$ both $v_1$ and $v_2$ are not finite and the episode never terminates.
The value of the state at this probability is: $v_2 = - \frac{2+p}{p(1-p)} = -\frac{\sqrt{6}}{(\sqrt{6}-2)(3 - \sqrt{6})} = - \frac{\sqrt{6}}{3 \sqrt{6} - 6 - 6 + 2 \sqrt{6}} = - \frac{\sqrt{6}}{5 \sqrt{6} - 12} \approx -9.9$
The plots below visualize these distributions for the corridor problem starting with the normalized distributions per step which include the terminal states. If we continued to create these plots for larger values of $k$, then the distribution would collapse to a value of 1 for being in a terminal state. In order to calculate other distributions such as the stationary state distribution, it is necessary to renormalize these probabilities by excluding the terminal states:
On-policy Distributions
$$\begin{flalign}
&\mu_{k, \pi}(s) = \Pr\{S_k = s \mid \pi \} \; \forall s \in \mathcal{S}^+ \tag{state visits per step}\\
&\Pr \{ T \leq k \vert \pi \} = 1 - \sum_{s \in \mathcal{S}} \Pr\{S_k = s \mid \pi \} \; \forall k \tag{Chance of terminating already (distribution function not density)}\\
&\mu_\pi(s) = \frac{\sum_k \Pr \{ S_k = s \mid \pi \}}{\sum_{k} \sum_{s \in \mathcal{S}} \Pr \{ S_k = s \mid \pi \}} \; \forall s \in \mathcal{S} \tag{non-terminal state visits}\\
&\mu_\pi(s, k) = \frac{\Pr \{ S_k = s \mid \pi \}}{\sum_{k} \sum_{s \in \mathcal{S}} \Pr \{ S_k = s \mid \pi \}} \; \forall s \in \mathcal{S} \tag{non-terminal state and step visits}\\
\end{flalign}$$
Note that final two distributions are only defined for non-terminal states. If we tried to include terminal states we would be unable to normalize the distribution since $\lim_{k \rightarrow \infty} \Pr \{ S_k = S_T \mid \pi \} = 1$ and we would have a diverging sum in the denominator. The only reason these calculation is possible is that the probabilities reach zero quickly enough at higher $k$ for the non-terminal states.
The plots below visualize the four expressions above. The second expression notably is not a probability density but a cummulative distribution function since it includes a sum of all probabilities that meet the condition.
One common form of linear feature vector is one that selects active features per state. Tile coding is an example of this where a state is assigned a tile in each tiling used and the number of tilings control how many active features a given state will have. Because the only possible feature vector values are 1 or 0, this style of encoding need not be as complex as other methods. We can see by the form of the gradients an abbreviated algorithm that need not compute the eligibility vector explicitely.
We can define a binary feature encoding by the function $\mathcal{F}(s)$ which returns the indices of active features for a state $s$ as well as the knowledge of how many total features there are, $d$. All of the values of $\mathbf{x}(s)$ are zero except for the indices in $\mathcal{F}(s)$ whose values are 1. That simplifies the expression we have before for the linear feature eligibility vector:
$$\begin{flalign}
\nabla \left ( \ln \pi_a \right )_{i, j} &= \frac{\nabla \left ( \pi_a \right )_{i, j}}{\pi_a} \\
&= \begin{cases}
\mathbf{x}(s)_i (1 - \pi_j), & \text{ if } j = a \\
-\pi_j \mathbf{x}(s)_i, & \text{ else }\\
\end{cases} \\
&= \begin{cases}
(1 - \pi_j), & \text{ if } j = a \text{, } i \in \mathcal{F}(s) \\
-\pi_j, & \text{ if } j \neq a \text{, } i \in \mathcal{F}(s) \\
0, & \text{ otherwise}
\end{cases}
\end{flalign}$$
We can see from this form of the eligibility vector that it need not be computed explicitely and we do not need to instantiate a feature vector either. Rather we can simply go through the active feature indices and subtract the policy output for the column index at each row and then add 1 to the column corresponding to the selected action:
Loop for each step of the episode $t = 0, 1, \cdots, T-1$
Specialized versions of REINFORCE that use binary features and linear features can be found below as well as the general case that works for any type of parameterized function approximation.
Note that if we try to apply this algorithm to the short corridor gridworld it fails because a terminal state is encountered. This condition is checked inside the algorithm because there is nothing about an MDP the way it is defined which tells you in advance if it is a continuing task or not. In the tabular case you can always check to see if a terminal state exists since every state is available, but for the non-tabular case, all we can do is note the problem if a terminal state is encountered.
It is implicit in all expressions below that $\pi$ is a function of $\boldsymbol{\theta}$ and that the gradients are with respect to $\boldsymbol{\theta}$. The performance measure for the continuing case is $J(\boldsymbol{\theta}) = r(\boldsymbol{\theta})$ (13.15) and all value functions use the definition of the differential return. We begin by expressing the gradient of the state value function in terms of the state-action value function, the policy, the average return and gradients thereof:
The purpose of this expression is to isolate the term which is the gradient of the average return since this is the performance metric gradient we originally sought. Note that if we separate the terms inside the sum, the one with the gradient of $r$ is $\sum_a \pi(a\vert s) [- \nabla r(\boldsymbol{\theta})] = -\nabla r(\boldsymbol{\theta}) \sum_a \pi(a \vert s)$. But the policy function is a probability distribution so its sum over actions is just 1. Therefore, this term simplifies to just $-\nabla r(\boldsymbol{\theta})$ which we can simply move to the other side of the expression swapping its place with the state value function:
Now the left hand side is $\nabla J(\boldsymbol{\theta})$ and does not depend on $s$. As such, the right hand side as a whole must be independent of $s$ as well so we are free to take a weighted sum of it over some probability distribution on $s$ since all the terms sum to 1. That is, if $f$ is independent of $s$, then $f = \sum_s \mu(s) f = f \sum_s \mu(s) = f \times 1 = f$:
$$\begin{flalign}
\nabla J(\boldsymbol{\theta}) &= \sum_s \mu(s) \left ( \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) \right ] - \nabla v_\pi(s) \right ) \\
&= \sum_s \mu(s) \sum_a \nabla \pi(a \vert s) q_\pi(s, a) + \sum_s \mu(s) \sum_a \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) - \sum_s \mu(s) \nabla v_\pi(s) \tag{separating sum terms}\\
&= \sum_s \mu(s) \sum_a \nabla \pi(a \vert s) q_\pi(s, a) + \sum_{s^\prime} \sum_s \mu(s) \sum_a \pi(a \vert s) p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) - \sum_s \mu(s) \nabla v_\pi(s) \tag{swapping sum order in second term}\\
&= \sum_s \mu(s) \sum_a \nabla \pi(a \vert s) q_\pi(s, a) + \sum_{s^\prime} \mu(s^\prime) \nabla v_\pi(s^\prime) - \sum_s \mu(s) \nabla v_\pi(s) \tag{stationary state distribution definition}\\
&= \sum_s \mu(s) \sum_a \nabla \pi(a \vert s) q_\pi(s, a) \tag{cancelling equivalent sum terms}\\
&= \mathbb{E}_\pi \left [ \sum_a \nabla \pi(a \vert S_t) q_\pi(S_t, a) \right ] \tag{expected value definition}\\
&= \mathbb{E}_\pi \left [ \sum_a \pi(a \vert S_t) \frac{\nabla \pi(a \vert S_t)}{\pi(a \vert S_t)} q_\pi(S_t, a) \right ] \tag{multiplying and dividing by the policy}\\
&= \mathbb{E}_\pi \left [\frac{\nabla \pi(A_t \vert S_t)}{\pi(A_t \vert S_t)} q_\pi(S_t, A_t) \right ] \tag{expected value definition}\\
&= \mathbb{E}_\pi \left [\frac{\nabla \pi(A_t \vert S_t)}{\pi(A_t \vert S_t)} G_t \right ] \tag{differential return definition}\\
&= \mathbb{E}_\pi \left [G_t \nabla \ln \pi(A_t \vert S_t) \right ] \tag{chain rule}\\
\end{flalign}$$
The expression inside the expected value can be sampled on every time step and the gradient is only in terms of the policy function which we have selected as something differentiable with respect to the parameters. Since this method will only be used for continuing problems, we cannot rely on Monte Carlo sampling for the differential return. Instead, our only option is to use a bootstrap value estimate in combination with a running estimate of the average reward and the immediate sample reward: $R - \overline{R} + \hat v^\prime$ where $\hat v^\prime$ is the differential value function estimate at the transition state and $\overline{R}$ is an estimate of the average reward. We can apply the existing actor-critic algorithms to these continuing problems as long as we track that additional information and use an additional step size parameter to update the average reward estimate. This step size parameter replaces the discount rate. See a full implementation below:
mimetext/htmlrootassigneelast_run_timestampA wWpersist_js_state·has_pluto_hook_features§cell_id$4915b1ed-ad53-4ece-9b00-bc136d47d8dcdepends_on_disabled_cells§runtime 3published_object_keysdepends_on_skipped_cells§errored$f924eb30-d1cc-4941-8fb5-ff70ad425ab9queued¤logsrunning¦outputbody E
13.3 REINFORCE: Monte Carlo Policy Gradient
If we replace the true action-value function in (13.5) with a learned approximation $\hat q_\pi$, then we have a method called the all-actions method because the update involves the sum over all actions. For the REINFORCE algorithm, we instead sample this value using the actual return and the policy distribution.
We can re-write (13.5) using an expected value under the policy and continue from there:
$$\begin{flalign}
\nabla J(\boldsymbol{\theta}) & \propto \mathbb{E}_\pi \left [ \gamma^t \sum_a q_\pi (S_t, a) \nabla \pi(a|S_t, \boldsymbol{\theta}) \right ] \tag{13.6}\\
&= \mathbb{E}_\pi \left [\gamma^t \sum_a \pi(a|S_t, \boldsymbol{\theta}) q_\pi (S_t, a) \frac{\nabla \pi(a|S_t, \boldsymbol{\theta})}{\pi(a|S_t, \boldsymbol{\theta})} \right ] \tag{multiply and divide by policy} \\
&= \mathbb{E}_\pi \left [ \gamma^t q_\pi (S_t, A_t) \frac{\nabla \pi(A_t|S_t, \boldsymbol{\theta})}{\pi(A_t|S_t, \boldsymbol{\theta})} \right ] \tag{replace a with sample under policy} \\
&= \mathbb{E}_\pi \left [ \gamma^t G_t \frac{\nabla \pi(A_t|S_t, \boldsymbol{\theta})}{\pi(A_t|S_t, \boldsymbol{\theta})} \right ] \tag{replace value with sample return} \\
\end{flalign}$$
Using the expression in the brackets we can write down an update rule for the parameters that can be sampled on each time step. This is the REINFORCE update:
Because it uses all future returns after step t, REINFORCE is a Monte Carlo algorithm and is well defined only for the episodic case. For implementation purposes we can replace $\frac{\nabla \pi(A_t|S_t, \boldsymbol{\theta})}{\pi(A_t|S_t, \boldsymbol{\theta})}$ with $\nabla \ln \pi(A_t|S_t, \boldsymbol{\theta}_t)$ which is usually refered to as the eligibility vector.
With the alternative parameterization, the eligibility vector is $\nabla \ln \pi(S_t, \theta_t)_{A_t}$ where $\pi$ is a vector and the $A_t$ subscript takes the value of that vector at the index corresponding to the action $A_t$.
These functions use two sets of parameters, one to calculate the policy function and another to calculate the state value function. The state representation vector is shared between the two functions, but the policy function will return a distribution of preferences over actions while the value function will return a single value. If linear approximation is used to estimate both functions, the the policy parameters $\boldsymbol{\theta}$ will be a $d \times N_a$ matrix where $d$ is the length of the state feature vector representation and the value function parameters $\mathbf{w}$ will be a length $d$ vector. It is also possible to mix linear and non-linear approximation with this method.
mimetext/htmlrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$d83dc659-dce7-41dd-a8e7-2933ab39d15cdepends_on_disabled_cells§runtime Yrpublished_object_keysdepends_on_skipped_cells§errored$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fcequeued¤logsrunning¦outputbodyXh
mimetext/htmlrootassigneelast_run_timestampA 3!M'persist_js_state·has_pluto_hook_features§cell_id$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fcedepends_on_disabled_cells§runtime ޘpublished_object_keysdepends_on_skipped_cellsçerrored$83ca0577-15d7-4448-b597-c77810b812bfqueued¤logsrunning¦outputbody1figure_13_2_test (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA %Wj8persist_js_state·has_pluto_hook_features§cell_id$83ca0577-15d7-4448-b597-c77810b812bfdepends_on_disabled_cells§runtime _published_object_keysdepends_on_skipped_cellsçerrored$a7c9ae69-f4b8-471c-ab97-90642f3c2bdbqueued¤logsrunning¦outputbody\reinforce_with_baseline_monte_carlo_control_binary_features (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA %H persist_js_state·has_pluto_hook_features§cell_id$a7c9ae69-f4b8-471c-ab97-90642f3c2bdbdepends_on_disabled_cells§runtime @_published_object_keysdepends_on_skipped_cells§errored$a7dcc8cd-04ec-48f2-a387-116330eaffb2queued¤logsrunning¦outputbody 5
mimetext/htmlrootassigneelast_run_timestampA &ðpersist_js_state·has_pluto_hook_features§cell_id$a7dcc8cd-04ec-48f2-a387-116330eaffb2depends_on_disabled_cells§runtime Gpublished_object_keysdepends_on_skipped_cellsçerrored$0ab70fc3-6188-42eb-aba2-d808f319be9fqueued¤logsrunning¦outputbody2
The above matrix represents an estimate of $\Pr \{ S_k = s \mid \pi \}$; however note that the terminal states are excluded from the rows. This corridor problem only has three non-terminal states. If we sum across each row, then we have the probability of reaching that step prior to terminating. The vector defined below measures the probability of an episode terminating prior to each step. Notably, this probablity is 0 for the first three steps since no policy starting from the left can terminate that quickly. As expected, the probability of terminating under the random policy grows with time approaching 1.
mimetext/htmlrootassigneelast_run_timestampA @/̰persist_js_state·has_pluto_hook_features§cell_id$8eb42403-1234-4e59-993e-057cc3a6d5c9depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cellsçerrored$bbc8864a-1545-433f-bc7c-0ddf6e907138queued¤logsrunning¦outputbody?plot_mountaincar_policy_values (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA @-0persist_js_state·has_pluto_hook_features§cell_id$bbc8864a-1545-433f-bc7c-0ddf6e907138depends_on_disabled_cells§runtime ipublished_object_keysdepends_on_skipped_cellsçerrored$a12b92d1-e045-4f92-b8cd-eee5d56fa67dqueued¤logsrunning¦outputbodyelementsepisode_rewardsprefixFloat32elements-6.0text/plain-5.0text/plain-9.0text/plain-7.0text/plain-4.0text/plain-22.0text/plain-34.0text/plain-6.0text/plain -27.0text/plainmored-12.0text/plaintypeArrayprefix_shortobjectid561caa86399f4303!application/vnd.pluto.tree+objectepisode_stepsprefixInt64elements6text/plain5text/plain9text/plain7text/plain4text/plain22text/plain34text/plain6text/plain 27text/plainmored12text/plaintypeArrayprefix_shortobjectid74a251baf6aa9110!application/vnd.pluto.tree+objectpolicy_functionπ2text/plainpolicy_sample_actionπ_sample2text/plainpolicy_parameters*1×2 Matrix{Float32}:
-0.199834 0.199834text/plainestimate_state_valueestimate_state_valuetext/plainvalue_parametersprefixFloat32elements-9.63535text/plaintypeArrayprefix_shortobjectid35fd187379601766!application/vnd.pluto.tree+objectpolicy_and_valuepolicy_and_valuetext/plaintypeNamedTupleobjectid9c86eb7ef1a9a622mime!application/vnd.pluto.tree+objectrootassigneeconst best_mc_corridorlast_run_timestampA & persist_js_state·has_pluto_hook_features§cell_id$a12b92d1-e045-4f92-b8cd-eee5d56fa67ddepends_on_disabled_cells§runtime GEpublished_object_keysdepends_on_skipped_cellsçerrored$ce33f710-fd9d-4dfa-acda-40204e54d518queued¤logsrunning¦outputbodyx
13.5 Actor-Critic Methods
Here we also use the value function estimator to calculate the the return estimate using the one step bootstrap return. When the state value function is used in this way we call it the critic. In general we can use this function with n-step returns and eligibility traces. Recall from the subject of TD learning of value functions that the one-step return is often superior to the actual return regarding variance and ease of computation, although it does introduce bias to the estimate. With the use of eligibility traces we can smoothly vary arbitrarily close to the Monte Carlo return. Note that the bias in the gradient estimate is n due to the bootstrapping as such; the actor would be biased even if the critic was learned by a Monte Carlo method.
The one-step actor-critic method is the analog of the one step methods such as TD$(0)$, Sarsa$(0)$, and Q learning. These methods replace the full return of REINFORCE with the one step return as follows:
This can be implemented as a fully online algorithm because we do not have to wait until the end of an episode to calculate return estimates. The natural state-value-function learning method to pair with this is semi-gradient TD(0). See a full implementation below.
This expression repeats terms of the form $\nabla \pi(a \vert s) q_\pi(s, a)$ summed over different probabilities. The first appearance of this term is just a sum over all actions at the state $s$ which is the state we are using for the gradient expression. The next appearance of the expression is a sum over actions at state $s^\prime$. Let's define a new expressions:
So the second term can be written as $\gamma g(s)$
where the final expression uses the probability that the agent transitions from state $s$ to $s^\prime$ in one step under the policy $\pi$. Using this same logic, we can rewrite the third expression as well.
mimetext/htmlrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$339b4d2b-2237-46a3-9867-ecc3332856c1depends_on_disabled_cells§runtime !published_object_keysdepends_on_skipped_cells§errored$a8349352-3242-46d5-b0d5-1b6eb5d77e90queued¤logsrunning¦outputbodymimetext/htmlrootassigneelast_run_timestampA :ؾpersist_js_state·has_pluto_hook_features§cell_id$a8349352-3242-46d5-b0d5-1b6eb5d77e90depends_on_disabled_cells§runtimeUpublished_object_keysdepends_on_skipped_cellsçerrored$7d63b960-3998-4f7b-8cbb-ccd49db9aeacqueued¤logsrunning¦outputbodyelementsaction_probabilitiesprefixFloat32elements0.000138965text/plain0.999861text/plaintypeArrayprefix_shortobjectidd09c2324e4b17111!application/vnd.pluto.tree+objectstate_value_estimate-96.5535text/plaintypeNamedTupleobjectid26386ed69de54735mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA 'b~persist_js_state·has_pluto_hook_features§cell_id$7d63b960-3998-4f7b-8cbb-ccd49db9aeacdepends_on_disabled_cells§runtime7published_object_keysdepends_on_skipped_cellsçerrored$65d2add6-fd6f-456c-92ed-3cd9d1862ef6queued¤logsrunning¦outputbody=update_binary_policy_params! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA @persist_js_state·has_pluto_hook_features§cell_id$65d2add6-fd6f-456c-92ed-3cd9d1862ef6depends_on_disabled_cells§runtime !*published_object_keysdepends_on_skipped_cells§errored$f55afa58-962d-4551-8d95-a5b467d61adfqueued¤logsrunning¦outputbody?update_params_with_gradient! (generic function with 10 methods)mimetext/plainrootassigneelast_run_timestampA #Upersist_js_state·has_pluto_hook_features§cell_id$f55afa58-962d-4551-8d95-a5b467d61adfdepends_on_disabled_cells§runtimeLjpublished_object_keysdepends_on_skipped_cells§errored$d9d11d69-bc16-400a-8f46-f9a8ecb8516aqueued¤logsrunning¦outputbodyNactor_critic_binary_episodic_parameter_study (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA =l
persist_js_state·has_pluto_hook_features§cell_id$d9d11d69-bc16-400a-8f46-f9a8ecb8516adepends_on_disabled_cells§runtime 2xpublished_object_keysdepends_on_skipped_cells§errored$ed93259c-7b8b-46d7-97fb-f194e0e04b3aqueued¤logsrunning¦outputbodyCsetup_binary_beta_policy_arguments (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA '-Ŭpersist_js_state·has_pluto_hook_features§cell_id$ed93259c-7b8b-46d7-97fb-f194e0e04b3adepends_on_disabled_cells§runtime ɵpublished_object_keysdepends_on_skipped_cells§errored$d1ed25e6-60c6-411f-a541-99986e5da2c5queued¤logsrunning¦outputbody\reinforce_with_baseline_monte_carlo_control_linear_features (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA &8persist_js_state·has_pluto_hook_features§cell_id$d1ed25e6-60c6-411f-a541-99986e5da2c5depends_on_disabled_cells§runtime 9published_object_keysdepends_on_skipped_cells§errored$b966b248-fb4d-457d-90f6-114370846242queued¤logsrunning¦outputbody7bad_continuous_action (generic function with 3 methods)mimetext/plainrootassigneelast_run_timestampA !ǰpersist_js_state·has_pluto_hook_features§cell_id$b966b248-fb4d-457d-90f6-114370846242depends_on_disabled_cells§runtime bpublished_object_keysdepends_on_skipped_cells§errored$4156d955-9daf-4429-b152-e8332980fb9equeued¤logsrunning¦outputbodyelementsstep_rewardsprefixFloat32elementstypeArrayprefix_shortobjectid3c0520dc1d635d04!application/vnd.pluto.tree+objectepisode_stepsprefixInt64elements30971text/plain33158text/plain36744text/plain39697text/plain42025text/plain44282text/plain45403text/plain47954text/plain 49838text/plainmoreE99724text/plaintypeArrayprefix_shortobjectid402d7f32a5b41291!application/vnd.pluto.tree+objectepisode_rewardsprefixFloat32elements-30970.0text/plain-2187.0text/plain-3586.0text/plain-2953.0text/plain-2328.0text/plain-2257.0text/plain-1121.0text/plain-2551.0text/plain -1884.0text/plainmoreE-532.0text/plaintypeArrayprefix_shortobjectid68896aa3766bea26!application/vnd.pluto.tree+objectpolicy_parameters$1452×2 Matrix{Float32}:
0.0 0.0
0.0 0.0
0.0 0.0
0.0 0.0
0.0 0.0
0.0 0.0
0.0 0.0
⋮
0.0437175 -0.0569757
0.0326978 -0.0276905
0.00138512 0.0006306
0.0 0.0
0.0 0.0
0.0 0.0text/plainvalue_parametersprefixFloat32elements0.0text/plain0.0text/plain0.0text/plain0.0text/plain0.0text/plain0.0text/plain0.0text/plain0.0text/plain 0.0text/plainmore0.0text/plaintypeArrayprefix_shortobjectidf01afe6ea747a0b0!application/vnd.pluto.tree+objectpolicy_functionπtext/plainpolicy_sample_actionπ_sampletext/plainestimate_state_valueestimate_state_valuetext/plainpolicy_and_valuepolicy_and_valuetext/plaintypeNamedTupleobjectid7aefd39e1e4696e7mime!application/vnd.pluto.tree+objectrootassignee,const mountaincar_continuous_test_train_betalast_run_timestampA >Xpersist_js_state·has_pluto_hook_features§cell_id$4156d955-9daf-4429-b152-e8332980fb9edepends_on_disabled_cells§runtime*<ŵpublished_object_keysdepends_on_skipped_cells§errored$b09e1e48-494e-4967-826a-6e70199acad4queued¤logsrunning¦outputbodyC
Squashed Gaussian Alternative
mimetext/htmlrootassigneelast_run_timestampA Epersist_js_state·has_pluto_hook_features§cell_id$b09e1e48-494e-4967-826a-6e70199acad4depends_on_disabled_cells§runtime (published_object_keysdepends_on_skipped_cells§errored$734573e5-547b-4dcc-89bb-412aa6cc42d6queued¤logsrunning¦outputbodyEactor_critic_linear_parameter_study (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA +{persist_js_state·has_pluto_hook_features§cell_id$734573e5-547b-4dcc-89bb-412aa6cc42d6depends_on_disabled_cells§runtime x",published_object_keysdepends_on_skipped_cellsçerrored$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54queued¤logsrunning¦outputbodyKactor_critic_with_eligibility_traces_fcann (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA /Bpersist_js_state·has_pluto_hook_features§cell_id$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54depends_on_disabled_cells§runtime Em$published_object_keysdepends_on_skipped_cells§errored$692c1043-4eaf-491e-b8fe-368618867f99queued¤logsrunning¦outputbody
The soft-max distribution is:
$$\sigma(a|s, \theta) = \frac{e^{h(s, a, \theta)}}{\sum_b e^{h(s, b, \theta)}}$$
We only have two possible actions in each state so the policy for action 1 would be given by:
Given the assumption that $h(s, 1, \theta)-h(s, 0, \theta) = \theta^\top\mathbf{x}(s)$, we replace the expression in the exponent resulting in the final expression of:
Using the notation $f(x) = 1/(1+e^{-x})$ we can write $\pi(1|S_t, \theta_t) = f(\theta_t^\top \mathbf{x}(S_t))$ where $f$ is the logistic function. Consider this notation for the rest of the exercises.
The REINFORCE update is given by: $\theta_{t+1} = \theta_t + \alpha G_t \frac{\nabla\pi(A_t|S_t, \theta_t)}{\pi(A_t|S_t, \theta_t)}$, so we need to compute the gradient of the policy in terms of the parameters for this action selection: $\nabla \pi(1|S_t, \theta_t)$. Luckily, the derivative of the logistic function is simply given by: $f(x)(1-f(x))$ where $f(x)$ is the logistic function itself. In our case $x = \theta_t^\top \mathbf{x}_t$ so after applying the chain rule we have:
$$\nabla\pi(1|S_t, \theta_t) = f(x)(1-f(x))\nabla x = f(x)(1-f(x)) \mathbf{x_t}$$
since $x$ is just a linear function of the parameters. So for the parameter update step we have:
Where $\chi (a)$ is a function that returns 1 for $a=1$ and -1 for $a=0$. There are many ways to achieve this but the following expression is simple and works: $\chi(a) = 2a - 1$. Dividing by the policy yields a unified expression for the eligibility vector:
Instead of selection actions based on action-value estimates we learn a parameterized policy with parameters $\boldsymbol{θ}$. $\pi(a|s, \boldsymbol{\theta}) = \text{Pr}\{A_t=a|S_t=s, \boldsymbol{\theta}_t=\boldsymbol{\theta\}}$ denotes the probability that action a is taken at time t given that the environment is in state s at time t with parameter $\boldsymbol{θ}$.
We consider methods that improve the policy parameter using the gradient of some scalar performance measure $J(\boldsymbol{\theta})$ with respect to the policy parameters. We follow gradient ascent since we are trying to maximize this value and methods that use this approach are called policy gradient methods. Methods that learn approximations to both policy and value functions are often called actor-critic methods, where 'actor' is a reference to the learned policy, and 'critic' refers to the learned value function, usually a state-value function.
In order to test this algorithm we need to use a continuing task which is lacking a terminal state. We could simply modify the corridor MDP to be a continuing task by altering the reward structure so a reward of 1 is received upon moving to the right from state 3 after which the state is reset to 1. Se below for a version of this MDP updated to be a continuing problem.
mimetext/htmlrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$436c52d2-280b-4ca4-9360-d6587b8254c7depends_on_disabled_cells§runtime 2rpublished_object_keysdepends_on_skipped_cells§errored$e96d592d-1e54-486d-8ad9-b857f85476e8queued¤logsrunning¦outputbodyDactor_critic_linear_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA e|1persist_js_state·has_pluto_hook_features§cell_id$e96d592d-1e54-486d-8ad9-b857f85476e8depends_on_disabled_cells§runtime 2 published_object_keysdepends_on_skipped_cells§errored$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6cqueued¤logsrunning¦outputbody/b
mimetext/htmlrootassigneelast_run_timestampA *1
persist_js_state·has_pluto_hook_features§cell_id$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6cdepends_on_disabled_cells§runtimeQ published_object_keysdepends_on_skipped_cellsçerrored$4da20fd7-b897-4f26-bf2a-f08d66ddf90fqueued¤logsrunning¦outputbodyGactor_critic_with_eligibility_traces! (generic function with 4 methods)mimetext/plainrootassigneelast_run_timestampA +
Npersist_js_state·has_pluto_hook_features§cell_id$4da20fd7-b897-4f26-bf2a-f08d66ddf90fdepends_on_disabled_cells§runtime }lpublished_object_keysdepends_on_skipped_cells§errored$11ea640c-3981-404d-87c6-4d3d0708a2b8queued¤logsrunning¦outputbodyMactor_critic_linear_episodic_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA +ǖpersist_js_state·has_pluto_hook_features§cell_id$11ea640c-3981-404d-87c6-4d3d0708a2b8depends_on_disabled_cells§runtime E.published_object_keysdepends_on_skipped_cellsçerrored$281360af-46bf-4c73-bf11-3cb1153ad3e2queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestamp persist_js_state·has_pluto_hook_features§cell_id$281360af-46bf-4c73-bf11-3cb1153ad3e2depends_on_disabled_cellsçruntimepublished_object_keysdepends_on_skipped_cells§errored$9ae58dd6-3cde-4943-9ac1-bd9d4f7d690cqueued¤logsrunning¦outputbodyNupdate_squashed_gaussian_eligibility_vector! (generic function with 4 methods)mimetext/plainrootassigneelast_run_timestampA #^persist_js_state·has_pluto_hook_features§cell_id$9ae58dd6-3cde-4943-9ac1-bd9d4f7d690cdepends_on_disabled_cells§runtime #published_object_keysdepends_on_skipped_cells§errored$da3cb392-78f2-48b2-b0dc-5f016664798cqueued¤logsrunning¦outputbodyڗ%Total Reward: -142.0
mimetext/htmlrootassigneelast_run_timestampA ANpersist_js_state·has_pluto_hook_features§cell_id$da3cb392-78f2-48b2-b0dc-5f016664798cdepends_on_disabled_cells§runtime Vpublished_object_keysdepends_on_skipped_cellsçerrored$dca2f8e2-76af-4679-bf81-3824c15fc76dqueued¤logsrunning¦outputbodyelementsstep_rewardsprefixFloat32elementstypeArrayprefix_shortobjectid9f06ece3690faedd!application/vnd.pluto.tree+objectepisode_stepsprefixInt64elements19text/plain34text/plain61text/plain75text/plain90text/plain117text/plain135text/plain178text/plain 204text/plainmore 99994text/plaintypeArrayprefix_shortobjectid245ae93c73b7a831!application/vnd.pluto.tree+objectepisode_rewardsprefixFloat32elements18.0text/plain15.0text/plain27.0text/plain14.0text/plain15.0text/plain27.0text/plain18.0text/plain43.0text/plain 26.0text/plainmore 11.0text/plaintypeArrayprefix_shortobjectid1b64855ae6e6478!application/vnd.pluto.tree+objectpolicy_parameters52488×3 Matrix{Float32}:
NaN NaN NaN
NaN NaN NaN
NaN NaN NaN
NaN NaN NaN
NaN NaN NaN
NaN NaN NaN
NaN NaN NaN
⋮
NaN NaN NaN
NaN NaN NaN
NaN NaN NaN
NaN NaN NaN
NaN NaN NaN
NaN NaN NaNtext/plainvalue_parametersprefixFloat32elementsNaNtext/plainNaNtext/plainNaNtext/plainNaNtext/plainNaNtext/plainNaNtext/plainNaNtext/plainNaNtext/plain NaNtext/plainmoreNaNtext/plaintypeArrayprefix_shortobjectidd26d759cd1b52e3f!application/vnd.pluto.tree+objectpolicy_functionπtext/plainpolicy_sample_actionπ_sampletext/plainestimate_state_valueestimate_state_valuetext/plainpolicy_and_valuepolicy_and_valuetext/plaintypeNamedTupleobjectide492fba9453c9690mime!application/vnd.pluto.tree+objectrootassigneeconst reinforce_test3last_run_timestampA 2k;Űpersist_js_state·has_pluto_hook_features§cell_id$dca2f8e2-76af-4679-bf81-3824c15fc76ddepends_on_disabled_cells§runtime G>ŵpublished_object_keysdepends_on_skipped_cells§errored$8019bec9-1228-407b-9199-2fe29f26a981queued¤logsrunning¦outputbody
Exercise 13.1
Use your knowledge of the gridworld and its dynamics to determine an exact symbolic expression for the optimal probability of selecting the right action in Example 13.1
Example 13.1 is a gridworld with 3 non-terminal states and a terminal state at the far right. The reward is -1 per step. States 1 and 3 have actions left/right that move in the expected directions but state 2 reverses the directions. We use a performance measure $J(\mathbf{\theta}) = v_{\pi_\theta}(S)$. Given our feature representations of $\mathbf{x}(s, \text{right}) = [1, 0]^{\top}$ and $\mathbf{x}(s, \text{left}) = [0, 1]^{\top}$, we can only learn policies that are stochastic in terms of left/right action selection but do not vary between states. Also observe that due to probability constraints $p_{\text{right}} = 1 - p_{\text{left}}$. For simplicity, we will use the notation $p \doteq p_{\text{left}}$ and the following for the three state values: $v1, v2, v3$.
The beta distribution has two parameters like the normal distribution but is only defined from 0 to 1. The two parameters $\alpha$ and $\beta$ are positive real numbers and control the shape of the distribution. The density function is given below:
where $\text{B}(\alpha, \beta) = \frac{\Gamma(\alpha) \Gamma(\beta)}{\Gamma(\alpha + \beta)}$ and $\Gamma(z) = \int_0^\infty t^{z-1}e^{-t} \text{d} t$
We saw earlier from the treatment of the gaussian distribution that we need to find the gradient of a function of each distribution parameter with respect to the parameters of the function approximation. Luckily, the maximum likelihood estimator already computes the gradient we are interested in for this distribution. Note that the likelihood function for a single sample of the random variable $x$ which follows the beta distribution is given by $\mathcal{L}(\alpha, \beta \vert X) = \ln(f(X_i; \alpha, \beta))$ and the partial derivative of this function with respect to each parameter $\alpha$ and $\beta$ is given by:
where $\frac{\partial \ln \text{B}(\alpha, \beta)}{\partial \alpha} = -\psi(\alpha + \beta) + \psi(\alpha)$ and $\frac{\partial \ln \text{B}(\alpha, \beta)}{\partial \beta} = -\psi(\alpha + \beta) + \psi(\beta)$ and $\phi(\alpha)$ is the digamma function which is just the derivative of the logarithm of the gamma function.
Since both $\alpha$ and $\beta$ must be greater than zero, we can use for an estimate for each one the exponential function applied to a dot product of the parameter vector with the feature vector: $\alpha(s, \boldsymbol{\theta}) \doteq \exp \left (\boldsymbol{\theta}_\alpha^\top \mathbf{x}(s) \right )$ and $\beta(s, \boldsymbol{\theta}) \doteq \exp \left (\boldsymbol{\theta}_\beta^\top \mathbf{x}(s) \right )$.
The eligibility vector for this distribution is then:
where $\alpha$ is a function of its parameters and $\beta$ is a function of the other parameter vector. The gradient components corresponding to each vector is only a function of a partial derivative of the distribution with respect to $\alpha$ and $\beta$. That is, since $\frac{\partial \alpha}{\partial \theta_{\beta_i}} = 0 \forall i$ and vice versa, then we can treat each part of the gradient separately.
mimetext/htmlrootassigneelast_run_timestampA +Zpersist_js_state·has_pluto_hook_features§cell_id$fd964539-2baf-4ff1-b286-5a0bb1b222c4depends_on_disabled_cells§runtime
Rpublished_object_keysdepends_on_skipped_cells§errored$5720e942-d3f8-4329-83a8-8bcedf078b6aqueued¤logsrunning¦outputbodyprefixFloat32elements0.47295text/plain0.52705text/plaintypeArrayprefix_shortobjectid81035272d8cf80fdmime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA $)persist_js_state·has_pluto_hook_features§cell_id$5720e942-d3f8-4329-83a8-8bcedf078b6adepends_on_disabled_cells§runtime!*published_object_keysdepends_on_skipped_cellsçerrored$62e677ac-2070-4f6b-9df2-90849d89fa9fqueued¤logsrunning¦outputbodyٯ152×1 Matrix{Float64}:
0.0
0.0
0.0
0.12513599999999991
0.1874460000000001
0.28129099999999996
0.343742
⋮
0.999999
0.999999
0.999999
0.999999
0.999999
0.999999mimetext/plainrootassignee%const corridor_terminal_probabilitieslast_run_timestampA #'Gpersist_js_state·has_pluto_hook_features§cell_id$62e677ac-2070-4f6b-9df2-90849d89fa9fdepends_on_disabled_cells§runtimePEpublished_object_keysdepends_on_skipped_cellsçerrored$11b9beea-b0cd-45eb-84c6-151728894df0queued¤logsrunning¦outputbodyHform_state_and_policy_function_outputs (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA 'Epersist_js_state·has_pluto_hook_features§cell_id$11b9beea-b0cd-45eb-84c6-151728894df0depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290queued¤logsrunning¦outputbodyNreinforce_monte_carlo_control_binary_features (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA #umPpersist_js_state·has_pluto_hook_features§cell_id$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290depends_on_disabled_cells§runtime /published_object_keysdepends_on_skipped_cells§errored$55ba8725-0ddf-4196-a41d-3f3c490a8d84queued¤logsrunning¦outputbodyVactor_critic_binary_episodic_gaussian_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA /ɰpersist_js_state·has_pluto_hook_features§cell_id$55ba8725-0ddf-4196-a41d-3f3c490a8d84depends_on_disabled_cells§runtime Xɵpublished_object_keysdepends_on_skipped_cellsçerrored$a540814a-57a1-4b98-9443-59e401425444queued¤logsrunning¦outputbody6binary_value_function (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$a540814a-57a1-4b98-9443-59e401425444depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$1b102220-6d78-480d-a77f-0e57bad23dcaqueued¤logsrunning¦outputbodyKcartpole_binary_continuing_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA / ְpersist_js_state·has_pluto_hook_features§cell_id$1b102220-6d78-480d-a77f-0e57bad23dcadepends_on_disabled_cells§runtime "published_object_keysdepends_on_skipped_cellsçerrored$4d4ae57b-afc3-44f9-b6fc-892f59f82921queued¤logsrunning¦outputbody7one_step_actor_critic! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA 'O) persist_js_state·has_pluto_hook_features§cell_id$4d4ae57b-afc3-44f9-b6fc-892f59f82921depends_on_disabled_cells§runtime [Bpublished_object_keysdepends_on_skipped_cells§errored$61949faa-8174-4b7b-8fbc-01d5f850b419queued¤logsrunning¦outputbodyXactor_critic_binary_continuing_gaussian_parameter_study (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA /*persist_js_state·has_pluto_hook_features§cell_id$61949faa-8174-4b7b-8fbc-01d5f850b419depends_on_disabled_cells§runtime %;published_object_keysdepends_on_skipped_cellsçerrored$5b15f5c9-80bf-47f0-898a-f8dead5b927cqueued¤logsrunning¦outputbody'
Continuing Case Actor-Critic Implementation
Note that this function has the same name as the episodic version. The only difference other than keyword arguments is that the max_episodes argument is missing. Since we already defined the versions of the algorithm for linear and non-linear cases in a generic manner, we only need to define the core version of this algorithm and the other functions will dispatch to it if they are called without the max_episodes argument.
mimetext/htmlrootassigneelast_run_timestampA ٰpersist_js_state·has_pluto_hook_features§cell_id$5b15f5c9-80bf-47f0-898a-f8dead5b927cdepends_on_disabled_cells§runtime ͵published_object_keysdepends_on_skipped_cells§errored$266d2234-26c8-43f1-9e75-49440a230ed6queued¤logsrunning¦outputbodyFactor_critic_with_eligibility_traces! (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA + pPpersist_js_state·has_pluto_hook_features§cell_id$266d2234-26c8-43f1-9e75-49440a230ed6depends_on_disabled_cells§runtime n2published_object_keysdepends_on_skipped_cells§errored$aa69e4ea-91e0-496a-a7be-529e67f4dbecqueued¤logsrunning¦outputbodyprefixFloat32elements0.508736text/plain0.491264text/plaintypeArrayprefix_shortobjectid7c15309e849c4f78mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA '&persist_js_state·has_pluto_hook_features§cell_id$aa69e4ea-91e0-496a-a7be-529e67f4dbecdepends_on_disabled_cells§runtime#X:published_object_keysdepends_on_skipped_cellsçerrored$10ee7709-0816-48d2-abe0-9be3dd04700fqueued¤logsrunning¦outputbody|
mimetext/htmlrootassigneelast_run_timestampA =%persist_js_state·has_pluto_hook_features§cell_id$10ee7709-0816-48d2-abe0-9be3dd04700fdepends_on_disabled_cells§runtime %published_object_keysdepends_on_skipped_cellsçerrored$7d94922e-dc9f-4953-b539-24aaa2c85b12queued¤logsrunning¦outputbodyd1
$\lambda_\theta$:
$\lambda_\mathbf{w}$:
$\alpha_{\overline{r}}$:
$\log_2 \alpha_\theta$ min:
$\log_2 \alpha_{\mathbf{w}}$ min:
mimetext/htmlrootassigneelast_run_timestampA 9persist_js_state·has_pluto_hook_features§cell_id$7d94922e-dc9f-4953-b539-24aaa2c85b12depends_on_disabled_cells§runtimelGpublished_object_keysdepends_on_skipped_cellsçerrored$df7f84e8-b42a-4001-9dbf-6bc3ced94207queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA (persist_js_state·has_pluto_hook_features§cell_id$df7f84e8-b42a-4001-9dbf-6bc3ced94207depends_on_disabled_cells§runtime?]Apublished_object_keysdepends_on_skipped_cells§errored$352d2952-cb83-47d3-9078-2b2ef9927443queued¤logsrunning¦outputbody:create_cartpole_functions (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA $persist_js_state·has_pluto_hook_features§cell_id$352d2952-cb83-47d3-9078-2b2ef9927443depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$0964133c-3a5b-433b-a8c4-a97813c37583queued¤logsrunning¦outputbody=plot_continuing_step_rewards (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA ! persist_js_state·has_pluto_hook_features§cell_id$0964133c-3a5b-433b-a8c4-a97813c37583depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cellsçerrored$349631b2-4686-49a9-9f3a-1e4ad588b568queued¤logsrunning¦outputbodyprefix"ContinuousMDP{Float32, Tuple{Float32, Float32}, Float32, ContinuousMDPTransitionSampler{Float32, Tuple{Float32, Float32}, Float32, var"#step#1603"{Float32}}, typeof(Main.var"workspace#8".MountainCarTask.initialize_state), typeof(Main.var"workspace#8".MountainCarTask.isterm), Returns{Bool}}elementsptfprefixcContinuousMDPTransitionSampler{Float32, Tuple{Float32, Float32}, Float32, var"#step#1603"{Float32}}elementsstepS(::Main.var"workspace#8".var"#step#1603"{Float32}) (generic function with 1 method)text/plaintypestructprefix_shortContinuousMDPTransitionSamplerobjectid5f924a83!application/vnd.pluto.tree+objectinitialize_state1initialize_state (generic function with 1 method)text/plainisterm'isterm (generic function with 1 method)text/plainis_valid_actionReturns{Bool}(true)text/plaintypestructprefix_shortContinuousMDPobjectidd13aca4d611093b3mime!application/vnd.pluto.tree+objectrootassignee!const mountaincar_continuous_mdp2last_run_timestampA =#persist_js_state·has_pluto_hook_features§cell_id$349631b2-4686-49a9-9f3a-1e4ad588b568depends_on_disabled_cells§runtime c|Opublished_object_keysdepends_on_skipped_cells§errored$8544eddb-2095-4a3c-82e0-920123a88e6dqueued¤logsrunning¦outputbody
Test REINFORCE With and Without Baseline
The following function calls execute the REINFORCE algorithm on Example 13.1. The output displayed is the policy function acting on the single state representation for the problem. The two values represent the probability of taking the left and right action respectively. If converged properly, the right action probability should be higher, approaching a value of about 60%.
In policy gradient methods, the policy can be parameterized in any way, as long as $\pi(a \vert s, \boldsymbol{\theta})$ is differentiable with respect to its parameters, that is, as long as $\nabla \pi(a \vert s, \boldsymbol{\theta})$ exists and is finite for all $s \in \mathcal{S}, a \in \mathcal{A}(s)$, and $\boldsymbol{\theta} \in \mathbb{R}^{d^\prime}$ where $d^\prime$ is the number of parameters.
If the action space is discrete and not too large then we can have numerical preferences for each state/action pair parameterized by $\boldsymbol{\theta}$. $h(s, a, \boldsymbol{\theta})$ and the corresponding policy can be to select actions according to the probability distribution generated by the soft-max. $\pi(a|s, \boldsymbol{\theta}) \doteq \frac{\exp{h(s, a, \boldsymbol{\theta})}}{\sum_b \exp{h(s, b, \boldsymbol{\theta})}}$. One advantage of using the soft-max is that the optimal policy can be stochastic or we can approach a deterministic policy by selecting the action with the highest probability. If we include a temperature parameter in the soft-max then we can vary the same policy to be more or less stochastic as needed.
If we calculate preferences with linear features, then we would have feature vectors $\mathbf{x}(s, a) \in \mathbb{R}^{d^\prime}$ to match with the parameter vector $\boldsymbol{\theta} \in \mathbb{R}^{d^\prime}$. Then the preferences would be calculated:
$$h(s, a, \boldsymbol{\theta}) = \boldsymbol{\theta}^\top \mathbf{x}(s, a)$$
Another advantage is that for some problems the policy may be easier to approximate than the action-value function. We can also inject some prior knowledge of the environment into how the policy is parametrized.
The first application of the quotient rule is trivial, I will omit the input arguments to μ and σ keeping in mind that these are functions of the parameters. Also let $\left ( - \frac{(a-\mu)^2}{2\sigma^2} \right ) = f(\mu, \sigma)$ which results in $\pi(a|s, \mathbf{\theta}) = \frac{1}{\sigma \sqrt{2\pi}} \exp{(f(\mu, \sigma))}$. Therefore:
There are two components to the sum, one for $\mu$ and one for $\sigma$. If we think of the paramters and feature vectors as concatenated, then this sum would be an element by element sum where $\mathbf{x}_\mu$ has a zero value for all the feature indices corresponding to $\sigma$ and vice-versa. This way doing the sum will form one complete vector that has gradient components for all the parameters $\mathbf{\theta}_\mu$ and $\mathbf{\theta}_\sigma$. Alternatively, the sum can be separated and each gradient can be treated separately with only those components keeping them separated throughout the calculation.
mimetext/htmlrootassigneelast_run_timestampA persist_js_state·has_pluto_hook_features§cell_id$beb01fb8-c77d-4b5c-a66d-3812415e04a3depends_on_disabled_cells§runtime 4Qpublished_object_keysdepends_on_skipped_cells§errored$8bc280db-e57d-4e40-be46-1790f4f7d9e7queued¤logsrunning¦outputbodyDactor_critic_fcann_parameter_study (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA /Rzpersist_js_state·has_pluto_hook_features§cell_id$8bc280db-e57d-4e40-be46-1790f4f7d9e7depends_on_disabled_cells§runtime z
published_object_keysdepends_on_skipped_cellsçerrored$89901156-b874-416b-89c1-6dc434a4eb17queued¤logsrunning¦outputbodyG
Now that we have verified the success of policy gradient methods on this problem, we can consider using a continuous action space where the policy can output a distribution over throttles. In the original problem, the maximum throttle value is 1, but the velocity of the car is already capped at 0.07. We can see if a policy attempts to use much higher throttle values to end the episode faster even if the physics is unrealistic. That observation would confirm a successful use of continuous actions where the throttle is an unbounded continuous value. The optimal policy would likely try to use the highest throttle possible to reach the maximum speed in either direction faster. We could apply friction to the problem so that the car would actually slip if it attempts to accelerate too quickly.
State Distribution Per Step Including Terminal State
mimetext/htmlrootassigneelast_run_timestampA ?Fpersist_js_state·has_pluto_hook_features§cell_id$9cf3dc5f-8a25-479f-93db-06e34f0d37a0depends_on_disabled_cells§runtimet_published_object_keysdepends_on_skipped_cellsçerrored$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70queued¤logsrunning¦outputbodyM
mimetext/htmlrootassigneelast_run_timestampA 0Upersist_js_state·has_pluto_hook_features§cell_id$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70depends_on_disabled_cells§runtime 9[published_object_keysdepends_on_skipped_cellsçerrored$bd6a7c16-6c25-4fc2-8e1b-4dab693ce19fqueued¤logsrunning¦outputbody`actor_critic_binary_episodic_squashed_gaussian_parameter_study (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA /D,persist_js_state·has_pluto_hook_features§cell_id$bd6a7c16-6c25-4fc2-8e1b-4dab693ce19fdepends_on_disabled_cells§runtime 0
published_object_keysdepends_on_skipped_cells§errored$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72cqueued¤logsrunning¦outputbodyelementsvalue_functionvalue_functiontext/plaingreedy_policygreedy_policytext/plainhistoryelementsepisode_stepsprefixFloat32elementstypeArrayprefix_shortobjectidc648579b1e24c833!application/vnd.pluto.tree+objectstep_rewardsprefixFloat32elementstypeArrayprefix_shortobjectid3ce720620b625923!application/vnd.pluto.tree+objecttypeNamedTupleobjectidad66844189f9d9c6!application/vnd.pluto.tree+objecttypeNamedTupleobjectid5dcda75341102be0mime!application/vnd.pluto.tree+objectrootassigneecorridor_trainlast_run_timestampA lpersist_js_state·has_pluto_hook_features§cell_id$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72cdepends_on_disabled_cells§runtime@hpublished_object_keysdepends_on_skipped_cellsçerrored±cell_dependencies$4f96be72-ef3e-4e08-ac4c-be4271dcd14cprecedence_heuristic cell_id$4f96be72-ef3e-4e08-ac4c-be4271dcd14cdownstream_cells_mapupstream_cells_map$19dfabda-7049-4050-8662-0385529c0c5aprecedence_heuristic cell_id$19dfabda-7049-4050-8662-0385529c0c5adownstream_cells_mapsref_cartpole_binary$0574f5a0-72e7-4aa2-80ac-f4ce4f0fe7c2upstream_cells_map@md_strCore:PlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70|>Base.get@bindSliderBasePlutoRunnerPlutoRunner.create_bondconfirmCore.applicablePlutoUI.combinegetindex$b71145a4-2614-4f62-bfd2-7d5d1fecec56precedence_heuristic cell_id$b71145a4-2614-4f62-bfd2-7d5d1fecec56downstream_cells_map%actor_critic_with_eligibility_traces!$05bfd818-bf4e-4bda-baa9-5ba647867097$68806899-9972-460a-9f11-daa708a9d610$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098upstream_cells_map zerotypeminzero_params!$e6cf9550-2e69-4b82-92cf-5e07a35490aaupdate_traces_with_gradient!$25be5dcf-be63-46c4-b6de-6cf79fa28fd0$056a8adc-92f4-4b33-90d9-4b3b4026bbbconeContinuousMDP$537270ba-122b-4f2b-880b-31d086766295Base.CoreLogging.!VectorReal'Base.CoreLogging.Base.fixup_stdlib_pathdeepcopy/@infoBase.invokelatestupdate_params_with_gradient!$b0a66a19-ee76-463b-a704-8fcee85444d0$a893a87b-2d07-4db5-9d1a-9da8646216f4$f55afa58-962d-4551-8d95-a5b467d61adfBase.CoreLogging.invokelatestBase.CoreLogging.===&form_state_and_policy_function_outputs$e7e49ff8-32df-48a4-afb2-462859592e92$11b9beea-b0cd-45eb-84c6-151728894df0#___this_pluto_module_nameIntegerFunctionBase<=Int64push!Base.CoreLogging.isa-bad_continuous_action$b966b248-fb4d-457d-90f6-114370846242+*Base.CoreLogging.>=$c0876a48-ea18-494d-8bfc-e2bceb73b417precedence_heuristic cell_id$c0876a48-ea18-494d-8bfc-e2bceb73b417downstream_cells_mapupstream_cells_mapplot_mountaincar_values$f9facbba-39d4-483e-9066-275603156db0!mountaincar_continuing_fcann_test$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091precedence_heuristic cell_id$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091downstream_cells_map#reinforce_monte_carlo_control_fcann$07ad517a-c2ac-4377-99fb-adb13d0f1d0cupstream_cells_mapFCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANNParamsIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANN.initializeparams_saxeInt64VectorRealFunctionlengthreinforce_monte_carlo_control!$0ac7ea44-14f6-4e80-80f9-d6df8059bb38fillsetup_fcann_policy_arguments$0e9de19e-bcd4-40ac-9831-afb6cad38422$f4b6f10b-4cd0-4be6-98ec-4d4ffb696392precedence_heuristic cell_id$f4b6f10b-4cd0-4be6-98ec-4d4ffb696392downstream_cells_mapupstream_cells_map@md_strgetindex$9db9ff71-bee9-4bea-a45b-748f8517fed1precedence_heuristic cell_id$9db9ff71-bee9-4bea-a45b-748f8517fed1downstream_cells_mapupstream_cells_mapInt64corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabe^%one_step_actor_critic_linear_features$57e5e12a-b722-4ea3-ab3b-e5711029e640update_corridor_features!$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811typemax$4634267b-5dea-4164-8bb2-1eb2fd4d7954precedence_heuristic cell_id$4634267b-5dea-4164-8bb2-1eb2fd4d7954downstream_cells_map!update_linear_eligibility_vector!$8e39bd15-862e-4941-88f9-2794b861a523$d1ed25e6-60c6-411f-a541-99986e5da2c5$57e5e12a-b722-4ea3-ab3b-e5711029e640$68806899-9972-460a-9f11-daa708a9d610upstream_cells_map zeroBLASisless@inboundsonenothingVectorPlutoUI.combineconfirmSlidergetindex$ba41f521-4ee2-42a6-bf18-078bfa4b875eprecedence_heuristic cell_id$ba41f521-4ee2-42a6-bf18-078bfa4b875edownstream_cells_mapmake_n_param_dist_policy_params$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$d5020a8d-1dd7-403c-9d1f-665b95543943$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098$55ba8725-0ddf-4196-a41d-3f3c490a8d84$61949faa-8174-4b7b-8fbc-01d5f850b419$dd8e8cd2-7b41-46c4-8530-adefb7aea684$08505e88-9c23-4e95-91e3-d18bf5133dbc$87482ea5-5265-4e02-92c0-1a8bb44ff0f4upstream_cells_mapzerosNTuple*IntegerReal$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03precedence_heuristic cell_id$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03downstream_cells_map-cartpole_tilecoding_reinforce_parameter_studyupstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6:|>setup_cartpole_problemmax_steps;reinforce_with_baseline_monte_carlo_control_binary_features$a7c9ae69-f4b8-471c-ab97-90642f3c2bdbscatterplot/foldxt+isemptyMapmeanLayout$3c695d54-c30f-4f04-bd40-f5da53be2a95precedence_heuristic cell_id$3c695d54-c30f-4f04-bd40-f5da53be2a95downstream_cells_mapupstream_cells_map@md_strgetindex$0d45ae72-572f-4d17-83cf-9814f2854131precedence_heuristic cell_id$0d45ae72-572f-4d17-83cf-9814f2854131downstream_cells_map%mountaincar_binary_continuous_params2$0d93132d-5819-47dc-8cf2-462d480d9c3dupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCore.applicable@bindBase.getcreate_actor_critic_params_UI$a8b40b8f-051a-4e6f-a079-ece4f32873de$cd9c9eeb-c90d-4499-9503-7773d5250f47precedence_heuristic cell_id$cd9c9eeb-c90d-4499-9503-7773d5250f47downstream_cells_mapupstream_cells_mapmountaincar_continuous_mdp2$349631b2-4686-49a9-9f3a-1e4ad588b568&show_mountaincar_continuous_trajectory$b5319d8b-0420-4ebf-b603-ea0b93365ac1"mountaincar_continuous_test_train2$fee14dfe-c5ca-4126-a830-cc9d7eda5433$fd58402f-da65-44cf-b81a-e21192fd0e63precedence_heuristic cell_id$fd58402f-da65-44cf-b81a-e21192fd0e63downstream_cells_mapupstream_cells_map$cartpole_fcann_continuing_test_state$28ce6e60-59cf-408a-8081-b978507b3c72CartPoleStateplot_cartpole_policy$602a07dd-8928-4b44-97e5-01c5cbf38351cartpole_continuing_fcann_test$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27$8e39bd15-862e-4941-88f9-2794b861a523precedence_heuristic cell_id$8e39bd15-862e-4941-88f9-2794b861a523downstream_cells_map-reinforce_monte_carlo_control_linear_features$5720e942-d3f8-4329-83a8-8bcedf078b6aupstream_cells_mapzerosIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84copyVectorReal!update_linear_eligibility_vector!$4634267b-5dea-4164-8bb2-1eb2fd4d7954Functionlength!update_linear_action_preferences!$581f7e9b-a5c2-4841-9605-85f9585b0274Matrixreinforce_monte_carlo_control!$0ac7ea44-14f6-4e80-80f9-d6df8059bb38$64900586-ef92-48e4-839e-ff952a46671bprecedence_heuristic cell_id$64900586-ef92-48e4-839e-ff952a46671bdownstream_cells_mapupstream_cells_map$fddef10c-7695-4596-9e16-987fd45a57e6precedence_heuristic cell_id$fddef10c-7695-4596-9e16-987fd45a57e6downstream_cells_map!setup_cartpole_continuous_problem$26880577-d267-4950-8725-7afe0d0402b6upstream_cells_maptile_coding_setup-deg2radTuple/create_cartpole_mdps$3c316495-bb6c-41e2-a38f-ba867a319fbbrand$e2b09af1-0f22-4f7f-b806-54fa522adb20precedence_heuristic cell_id$e2b09af1-0f22-4f7f-b806-54fa522adb20downstream_cells_mapupstream_cells_map$2be8a812-4f21-4fe8-a2de-50497db0345aprecedence_heuristic cell_id$2be8a812-4f21-4fe8-a2de-50497db0345adownstream_cells_mapupstream_cells_map@md_strgetindex$68806899-9972-460a-9f11-daa708a9d610precedence_heuristic cell_id$68806899-9972-460a-9f11-daa708a9d610downstream_cells_map4actor_critic_with_eligibility_traces_linear_features$11ea640c-3981-404d-87c6-4d3d0708a2b8$734573e5-547b-4dcc-89bb-412aa6cc42d6$ff4f977e-48df-4c12-845c-c245b4d39d6dupstream_cells_mapzerosIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84update_linear_value_gradient!$1753b5ed-c00b-4b60-b492-822180778e8ccopyVector!update_linear_eligibility_vector!$4634267b-5dea-4164-8bb2-1eb2fd4d7954RealFunction!update_linear_action_preferences!$581f7e9b-a5c2-4841-9605-85f9585b0274Matrixlengthlinear_value_function$0bf3b988-b3fb-49d5-8dde-b25766596363%actor_critic_with_eligibility_traces!$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90f$189798b3-ec6b-48b9-918c-ee0f65935ab3precedence_heuristic cell_id$189798b3-ec6b-48b9-918c-ee0f65935ab3downstream_cells_mapupstream_cells_map@md_strgetindex$00152954-dc98-4120-b94b-2ea4d987832bprecedence_heuristic cell_id$00152954-dc98-4120-b94b-2ea4d987832bdownstream_cells_map!create_mountaincar_continuing_mdp$46fea69b-599e-46ab-8455-d2da865d9a8eupstream_cells_mapStateMDPTransitionSampler$d963ff6d-f1b6-4799-aa0e-1ae100310d84mountaincar_continuing_step$a9db3f85-ff56-4bbc-be87-47b893ef3b7bMountainCarTaskStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84$42d4600a-bf3c-45ac-b7f5-d23917713ff5precedence_heuristic cell_id$42d4600a-bf3c-45ac-b7f5-d23917713ff5downstream_cells_map(cartpole_continuing_fcann_network_params$50ae94c4-70f3-4215-82bd-eb2227c2badfupstream_cells_map@md_strCore:PlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70|>Base.get@bindBasePlutoRunnerPlutoRunner.create_bondNumberFieldconfirmCore.applicablePlutoUI.combinegetindex$4e29c621-223e-4859-8e96-db04b967815aprecedence_heuristic cell_id$4e29c621-223e-4859-8e96-db04b967815adownstream_cells_map/setup_binary_squashed_gaussian_policy_arguments$05f120be-9695-4824-82fd-142a0df13098upstream_cells_map'BinarySquashedGaussianEligibilityVector$76fd79a2-2bc8-45f8-a243-48415118898aBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41randIntegerContinuousMDP$537270ba-122b-4f2b-880b-31d086766295RealFunctionupdate_binary_feature_vector!$8eab55a5-41b7-4f5e-a02f-4c19388bc9eamake_n_param_dist_params$76eb6743-cac0-4174-9ba3-a0691c200b54UnionNTuple$5981f52b-d829-4c7d-b47b-33310f7d64a2precedence_heuristic cell_id$5981f52b-d829-4c7d-b47b-33310f7d64a2downstream_cells_mapupstream_cells_mapcorridor_train.value_functioncorridor_train$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72cmake_ϵ_greedy_policy!$d963ff6d-f1b6-4799-aa0e-1ae100310d84$0e9de19e-bcd4-40ac-9831-afb6cad38422precedence_heuristic cell_id$0e9de19e-bcd4-40ac-9831-afb6cad38422downstream_cells_mapsetup_fcann_policy_arguments$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091$e1aec891-d95a-47d1-97d7-d2a4cfb16e64upstream_cells_map FCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84zerosBoolsizeFCANNParamsoneIntegerVectorRealInt64FCANN.form_activationseachindexdeepcopy update_fcann_action_preferences!$cc3ac95e-a398-438a-ba3d-62b6733f6342length/+ update_fcann_eligibility_vector!$45f0a385-6465-4acc-8637-1b007a0fe215fill$ff3009eb-23f9-44fe-8e56-85dbc7b463d0precedence_heuristic cell_id$ff3009eb-23f9-44fe-8e56-85dbc7b463d0downstream_cells_mapshow_squashed_policy$f8215517-b18f-4a03-9421-8edab4ca8089upstream_cells_mapexpplot_squashed_gaussian$00bd2835-b006-4244-9877-bc7e031e3ef8Function$4fb83451-b6f8-4e6e-a131-1accc8e10b08precedence_heuristic cell_id$4fb83451-b6f8-4e6e-a131-1accc8e10b08downstream_cells_map,reinforce_with_baseline_monte_carlo_control!$0ac7ea44-14f6-4e80-80f9-d6df8059bb38$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$d1ed25e6-60c6-411f-a541-99986e5da2c5$697b2310-9d96-4f7f-be62-c3bd6bf736f3$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$d5020a8d-1dd7-403c-9d1f-665b95543943upstream_cells_map zerooneStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84lengthsample_action$d963ff6d-f1b6-4799-aa0e-1ae100310d84copyeachindexrunepisode!$d963ff6d-f1b6-4799-aa0e-1ae100310d84deepcopyReal/^form_state_policy_function$37ec6802-d4c2-4470-ad69-439d5a732f77update_params_with_gradient!$b0a66a19-ee76-463b-a704-8fcee85444d0$a893a87b-2d07-4db5-9d1a-9da8646216f4$f55afa58-962d-4551-8d95-a5b467d61adfrunepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84:zerosIntegerFunctionInt64-+*form_state_value_function$e7566274-5518-4e28-8738-d4b1747d0cfb$406638af-1e08-44d2-9ee4-97aa9294a94bprecedence_heuristic cell_id$406638af-1e08-44d2-9ee4-97aa9294a94bdownstream_cells_mapupstream_cells_map@md_strgetindex$57e5e12a-b722-4ea3-ab3b-e5711029e640precedence_heuristic cell_id$57e5e12a-b722-4ea3-ab3b-e5711029e640downstream_cells_map%one_step_actor_critic_linear_features$9db9ff71-bee9-4bea-a45b-748f8517fed1upstream_cells_mapzerosIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84update_linear_value_gradient!$1753b5ed-c00b-4b60-b492-822180778e8ccopyVector!update_linear_eligibility_vector!$4634267b-5dea-4164-8bb2-1eb2fd4d7954Realone_step_actor_critic!$4d4ae57b-afc3-44f9-b6fc-892f59f82921FunctionMatrix!update_linear_action_preferences!$581f7e9b-a5c2-4841-9605-85f9585b0274lengthlinear_value_function$0bf3b988-b3fb-49d5-8dde-b25766596363$374af774-3a97-49b5-a3bb-bc3f7f63a3faprecedence_heuristic cell_id$374af774-3a97-49b5-a3bb-bc3f7f63a3fadownstream_cells_mapupstream_cells_mapep$e1274f57-75cb-4659-a82f-e5870c5367e2ep_step$a4eec4d3-5a75-4b52-ab9c-9d9e83d5547dplot_cart$63fbf8f4-e4e2-4893-be09-67450e92dbd7$7bf209c8-ef0a-46d1-937e-b1a6e45dc62eprecedence_heuristic cell_id$7bf209c8-ef0a-46d1-937e-b1a6e45dc62edownstream_cells_mapbeta_params$ad0009af-2cfc-4820-bd4a-698ad391f459upstream_cells_map@md_strCore:PlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70Base.get@bindSliderBasePlutoRunnerPlutoRunner.create_bondCore.applicablePlutoUI.combinegetindex$dd8e8cd2-7b41-46c4-8530-adefb7aea684precedence_heuristic cell_id$dd8e8cd2-7b41-46c4-8530-adefb7aea684downstream_cells_map1actor_critic_binary_episodic_beta_parameter_studyupstream_cells_map Random$df7f84e8-b42a-4001-9dbf-6bc3ced94207ContinuousMDP$537270ba-122b-4f2b-880b-31d086766295copyAactor_critic_with_eligibility_traces_binary_features_beta_actions$3e3c5897-809f-46e3-bb58-f115b082443eVectorRealscatter/Matrixisemptymean:AbstractVector|>Infmake_n_param_dist_policy_params$ba41f521-4ee2-42a6-bf18-078bfa4b875ezerosrandIntegerFunctionUInt64-plotfoldxt+MapLayoutRandom.seed!$4fea7232-f286-4a8b-93f8-a0702818ab31precedence_heuristic cell_id$4fea7232-f286-4a8b-93f8-a0702818ab31downstream_cells_mapupstream_cells_map@md_strgetindex$26880577-d267-4950-8725-7afe0d0402b6precedence_heuristic cell_id$26880577-d267-4950-8725-7afe0d0402b6downstream_cells_mapcartpole_setup $0cd96c44-cae6-421f-9fae-26141600bef4$64b38d1f-ecf9-4843-89a1-4c8953048265$24fa139c-ad4b-49db-ac8f-23c476ed8608$dddc4a2f-34b2-41dc-85b3-55aba4880fa6$d3b56fca-5b79-4465-8987-8d0005f854d8$5859ca11-90f8-4fd6-88ed-c56efe796fe8$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03$8aa16866-bfda-48df-9cf1-cf3d2e203ccb$dca2f8e2-76af-4679-bf81-3824c15fc76d$11a55af7-5301-4507-bb26-88e1e11236db$61650a97-b353-4a85-b50b-93fee296ac7b$d34d22ad-89c2-423e-91dd-bfb895dc6540$407a0724-4bb6-4c83-ab2d-17a0e19c4072$27487ad0-4779-42ce-8def-e660ef04bee0$07ba9fe4-aaa7-4123-9865-cbfa79d0d44a$e1274f57-75cb-4659-a82f-e5870c5367e2$5ee4ce72-7740-4297-8d84-619e0708e4ac$82e0e9a0-9662-429a-87e3-e6bdae02709a$daf35bfe-8f9c-4f55-971d-4d443be8f8bf$a5b002c9-5e11-462a-9da0-6e060c7963f8$d21617aa-6f38-4a90-8586-4b32022497adupstream_cells_map!setup_cartpole_continuous_problem$fddef10c-7695-4596-9e16-987fd45a57e6$a7891c63-18d6-4c1f-ba67-adf7c547d334precedence_heuristic cell_id$a7891c63-18d6-4c1f-ba67-adf7c547d334downstream_cells_mapupstream_cells_map$44f14d4f-7414-4c6f-883a-042ca261a403precedence_heuristic cell_id$44f14d4f-7414-4c6f-883a-042ca261a403downstream_cells_mapupstream_cells_map$94354552-9920-4b90-98d9-f75286d1f53eprecedence_heuristic cell_id$94354552-9920-4b90-98d9-f75286d1f53edownstream_cells_mapupstream_cells_map:corridor_parameter_studies$e5c1aca8-7575-4835-8273-e69ca0a55fe8$646bc853-b7fc-49fa-a201-ff98e8f952d4^$e5faaa1b-88cb-43e2-8d04-8972b58b4bdaprecedence_heuristic cell_id$e5faaa1b-88cb-43e2-8d04-8972b58b4bdadownstream_cells_mapplistv2v1v3tracesupstream_cells_map:-scatterplot/+*zipLayoutbgcolor$9c342958-1971-48ec-b919-5dfdcbc915a4$70096b14-beab-4f71-9886-6355c749bb8aprecedence_heuristic cell_id$70096b14-beab-4f71-9886-6355c749bb8adownstream_cells_mapupstream_cells_map@md_strgetindex$90d3b96b-ad2b-405c-951b-f48ec7ccf24aprecedence_heuristic cell_id$90d3b96b-ad2b-405c-951b-f48ec7ccf24adownstream_cells_mapupstream_cells_map@md_strgetindex$700dcbc4-c94c-4287-8cf0-0b2c7a320a3aprecedence_heuristic cell_id$700dcbc4-c94c-4287-8cf0-0b2c7a320a3adownstream_cells_mapupstream_cells_map reinforce_test5.policy_and_valueCartPoleStatereinforce_test5$82e0e9a0-9662-429a-87e3-e6bdae02709a$f59a5dcd-9f4a-4336-a391-e64af35ef799precedence_heuristic cell_id$f59a5dcd-9f4a-4336-a391-e64af35ef799downstream_cells_mapupstream_cells_mapBaseBase.Docs.HTML@html_str$5864a5a3-a5a5-43c2-9cb4-7d13b2d20bedprecedence_heuristic cell_id$5864a5a3-a5a5-43c2-9cb4-7d13b2d20beddownstream_cells_mapupstream_cells_map@md_strgetindex$e3a2fb12-37ce-4c23-ad93-5fc89991aabbprecedence_heuristic cell_id$e3a2fb12-37ce-4c23-ad93-5fc89991aabbdownstream_cells_mapupstream_cells_map@md_strgetindex$e5c1aca8-7575-4835-8273-e69ca0a55fe8precedence_heuristic cell_id$e5c1aca8-7575-4835-8273-e69ca0a55fe8downstream_cells_mapcorridor_parameter_studies$94354552-9920-4b90-98d9-f75286d1f53e$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6cupstream_cells_map Layout:sumRandom$df7f84e8-b42a-4001-9dbf-6bc3ced94207|>Int64corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512scatterplot/log2+foldxt-reinforce_monte_carlo_control_binary_features$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290Mapround;reinforce_with_baseline_monte_carlo_control_binary_features$a7c9ae69-f4b8-471c-ab97-90642f3c2bdbRandom.seed!$44b32cc0-36a8-41fd-89bc-ce894536926cprecedence_heuristic cell_id$44b32cc0-36a8-41fd-89bc-ce894536926cdownstream_cells_mapupstream_cells_mapbest_mc_corridor$a12b92d1-e045-4f92-b8cd-eee5d56fa67d!best_mc_corridor.policy_and_value$646bc853-b7fc-49fa-a201-ff98e8f952d4precedence_heuristic cell_id$646bc853-b7fc-49fa-a201-ff98e8f952d4downstream_cells_mapcorridor_parameter_studies$94354552-9920-4b90-98d9-f75286d1f53e$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6cupstream_cells_map Layout%one_step_actor_critic_binary_features$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2:sum|>lengthInt64corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512scatter-/plot+Inf32log2Maproundfoldxtisempty$25be5dcf-be63-46c4-b6de-6cf79fa28fd0precedence_heuristic cell_id$25be5dcf-be63-46c4-b6de-6cf79fa28fd0downstream_cells_mapupdate_traces_with_gradient!$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90fupstream_cells_map zeroisless@inboundsFCANNParamsoneBinaryEligibilityVector$41dc149d-c6f3-4b0d-a856-06f3aaae3049nothingVectorFunctionlengthcorridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabe/foldxt+Maprunepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84$4c4e643b-d4b9-44f0-8d30-dc521bcc55acprecedence_heuristic cell_id$4c4e643b-d4b9-44f0-8d30-dc521bcc55acdownstream_cells_mapcartpole_continuing_mdp$1b102220-6d78-480d-a77f-0e57bad23dca$3c89209c-9202-4d5d-841c-ea34be369616$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27upstream_cells_map#cartpole_functions.initialize_stateStateMDPTransitionSampler$d963ff6d-f1b6-4799-aa0e-1ae100310d84cartpole_functions$f27f2bcd-05b6-44fe-bf9e-a3e51556db7ccartpole_continuing_step$5d434c83-c9ca-499f-8695-c7733031c2deStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84$738ada7f-edc7-4ed3-a15e-e92113468738precedence_heuristic cell_id$738ada7f-edc7-4ed3-a15e-e92113468738downstream_cells_mapupstream_cells_map$cacaaca6-6e01-464f-a2ee-cbf62737a426precedence_heuristic cell_id$cacaaca6-6e01-464f-a2ee-cbf62737a426downstream_cells_mapupstream_cells_mapcorridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabe;reinforce_with_baseline_monte_carlo_control_linear_features$d1ed25e6-60c6-411f-a541-99986e5da2c5^update_corridor_features!$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811$daf35bfe-8f9c-4f55-971d-4d443be8f8bfprecedence_heuristic cell_id$daf35bfe-8f9c-4f55-971d-4d443be8f8bfdownstream_cells_mapupstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6display_cartpole_episode$822e4d69-2582-4956-858e-06ecb091e76a|>reinforce_test5$82e0e9a0-9662-429a-87e3-e6bdae02709arunepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84$8e096fae-9941-49d8-ae87-c68b02f68da5precedence_heuristic cell_id$8e096fae-9941-49d8-ae87-c68b02f68da5downstream_cells_mapmountaincar_continuous_beta_mdp$4156d955-9daf-4429-b152-e8332980fb9e$a6be9a4c-d43b-4867-b7a2-07a46a9d0d8fupstream_cells_map)create_continuous_action_mountaincar_beta$d2729657-d0bf-4d39-8ec7-f242a1ad48d6$666a4e89-306b-4fb2-bdc4-3dda2c63153fprecedence_heuristiccell_id$666a4e89-306b-4fb2-bdc4-3dda2c63153fdownstream_cells_mapSpecialFunctionsupstream_cells_map$5d35e515-e2d3-443e-becf-eb28c25db346precedence_heuristic cell_id$5d35e515-e2d3-443e-becf-eb28c25db346downstream_cells_map#mountaincar_continuing_fcann_params$cb70d400-3e9c-441c-b17c-e727e8c928f3upstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunner(create_actor_critic_continuing_params_UI$5b15d91e-7119-4f85-a54a-7d4f1fdaf097Core.applicable@bindBase.get$4c34640f-efa2-4e1d-8a70-0acd2ce45428precedence_heuristic cell_id$4c34640f-efa2-4e1d-8a70-0acd2ce45428downstream_cells_mapupstream_cells_map@md_strgetindex$e7566274-5518-4e28-8738-d4b1747d0cfbprecedence_heuristic cell_id$e7566274-5518-4e28-8738-d4b1747d0cfbdownstream_cells_mapform_state_value_function$4fb83451-b6f8-4e6e-a131-1accc8e10b08$e7e49ff8-32df-48a4-afb2-462859592e92$5b868eba-c1af-49f6-8f93-79b78c319a6f$11b9beea-b0cd-45eb-84c6-151728894df0upstream_cells_mapFunction$6bf5ad39-1400-4e1f-a843-a1934b8aaa48precedence_heuristic cell_id$6bf5ad39-1400-4e1f-a843-a1934b8aaa48downstream_cells_map,update_squashed_gaussian_eligibility_vector!$05f120be-9695-4824-82fd-142a0df13098upstream_cells_map zeroisless@inboundsonenothingVector*cartpole_binary_continuing_parameter_study$1b102220-6d78-480d-a77f-0e57bad23dcaislessgetindex$c5dd7e99-57e0-4bc7-97d2-2c780b23bcffprecedence_heuristic cell_id$c5dd7e99-57e0-4bc7-97d2-2c780b23bcffdownstream_cells_mapupstream_cells_map@md_strgetindex$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9precedence_heuristic cell_id$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9downstream_cells_map!mountaincar_continuing_fcann_test$10ee7709-0816-48d2-abe0-9be3dd04700f$c0876a48-ea18-494d-8bfc-e2bceb73b417$3a37b53d-9174-4faa-9404-74a40c385b0aupstream_cells_map*actor_critic_with_eligibility_traces_fcann$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54mountaincar_fcann_setup$023f67b8-8f38-470a-9766-ac60a75678aamountaincar_continuing_mdp$46fea69b-599e-46ab-8455-d2da865d9a8e$042fbafe-2401-4fb7-ac13-4531e0782c79precedence_heuristic cell_id$042fbafe-2401-4fb7-ac13-4531e0782c79downstream_cells_map!update_binary_eligibility_vector!$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2$05bfd818-bf4e-4bda-baa9-5ba647867097upstream_cells_mapRealMatrix!update_binary_action_preferences!$a361f4c9-47ce-42ad-899c-87b611c0d471BinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41soft_max!$33c99850-67cd-4754-94b9-6df97b238e27BinaryEligibilityVector$41dc149d-c6f3-4b0d-a856-06f3aaae3049IntegerVector$d57375a5-b9e0-4742-b5f7-6a7da891604aprecedence_heuristic cell_id$d57375a5-b9e0-4742-b5f7-6a7da891604adownstream_cells_map-mountaincar_binary_continuing_parameter_study$04f42c09-8ab5-4233-b196-51c4aa2dcedbupstream_cells_mapmountaincar_tilecoding_setup$7c592385-e8d3-4efe-962c-d39debb64405mountaincar_continuing_mdp$46fea69b-599e-46ab-8455-d2da865d9a8e#actor_critic_linear_parameter_study$734573e5-547b-4dcc-89bb-412aa6cc42d6$e96d592d-1e54-486d-8ad9-b857f85476e8$ff4f977e-48df-4c12-845c-c245b4d39d6d$07ad517a-c2ac-4377-99fb-adb13d0f1d0cprecedence_heuristic cell_id$07ad517a-c2ac-4377-99fb-adb13d0f1d0cdownstream_cells_mapupstream_cells_map#reinforce_monte_carlo_control_fcann$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabe^update_corridor_features!$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811$71a5fce8-6d9a-4625-bad1-a951d61bff28precedence_heuristic cell_id$71a5fce8-6d9a-4625-bad1-a951d61bff28downstream_cells_map$mountaincar_binary_continuous_params$b53dba81-a9e9-41da-8fc2-7736bf25f2dcupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCore.applicable@bindBase.getcreate_actor_critic_params_UI$a8b40b8f-051a-4e6f-a079-ece4f32873de$77906355-08f8-4b08-b051-84697199b519precedence_heuristic cell_id$77906355-08f8-4b08-b051-84697199b519downstream_cells_mapmountaincar_max_vals$023f67b8-8f38-470a-9766-ac60a75678aa$7c592385-e8d3-4efe-962c-d39debb64405upstream_cells_map$5207308e-f636-4d47-b135-036a6e7b8ecdprecedence_heuristic cell_id$5207308e-f636-4d47-b135-036a6e7b8ecddownstream_cells_mapupstream_cells_map&show_mountaincar_continuous_trajectory$b5319d8b-0420-4ebf-b603-ea0b93365ac1"mountaincar_continuous_test_train3$5eb8d9f9-8512-4e00-8cb5-cec68d73cc7d$16113560-e911-47b4-abc4-641bbd246454precedence_heuristic cell_id$16113560-e911-47b4-abc4-641bbd246454downstream_cells_mapupstream_cells_map&mountaincar_continuous_test_train_beta$4156d955-9daf-4429-b152-e8332980fb9eplotLayout$b7f77935-bcab-4ef1-8e1b-a7d059784ff3precedence_heuristic cell_id$b7f77935-bcab-4ef1-8e1b-a7d059784ff3downstream_cells_maptest_mountaincar_state$f8215517-b18f-4a03-9421-8edab4ca8089upstream_cells_map@md_strCore:PlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70Base.get@bindSliderBasePlutoRunnerPlutoRunner.create_bondCore.applicablePlutoUI.combinegetindex$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5precedence_heuristic cell_id$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5downstream_cells_mapupstream_cells_mapreinforce_test.policy_and_valuecartpole_mdps$024dcd1a-8eaa-4a95-8037-2f578828309creinforce_test$24fa139c-ad4b-49db-ac8f-23c476ed86082cartpole_mdps.episodic.continuous.initialize_state$00bd2835-b006-4244-9877-bc7e031e3ef8precedence_heuristic cell_id$00bd2835-b006-4244-9877-bc7e031e3ef8downstream_cells_mapplot_squashed_gaussian$3e7cecec-eb77-4862-8e3c-b510422e06db$ff3009eb-23f9-44fe-8e56-85dbc7b463d0upstream_cells_mapsquashed_gaussian_pdf$b16899b7-36bf-4a5e-8e2f-4496b8450687-plotLinRange*oneLayoutReal$50ae94c4-70f3-4215-82bd-eb2227c2badfprecedence_heuristic cell_id$50ae94c4-70f3-4215-82bd-eb2227c2badfdownstream_cells_mapupstream_cells_map@md_str<(cartpole_continuing_fcann_network_params$42d4600a-bf3c-45ac-b7f5-d23917713ff5)cartpole_fcann_continuing_parameter_study$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62>+start_cartpole_continuing_fcann_param_study$2c5d221a-2469-49e1-9249-dfdc2457f2faisless&cartpole_continuing_fcann_study_params$5ffc271f-c73f-494a-9727-8d7516af2191getindex$cc3ac95e-a398-438a-ba3d-62b6733f6342precedence_heuristic cell_id$cc3ac95e-a398-438a-ba3d-62b6733f6342downstream_cells_map update_fcann_action_preferences!$0e9de19e-bcd4-40ac-9831-afb6cad38422upstream_cells_mapFCANN.forwardNOGRAD_base!endFCANNActivations$5c11a92d-7496-4aba-af15-2537eac49dd7Float32FCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANNParamsIntegerVector$c926b6df-c40b-4c4c-8a95-ce9e41feb100precedence_heuristic cell_id$c926b6df-c40b-4c4c-8a95-ce9e41feb100downstream_cells_mapupstream_cells_map$740a3f41-9302-481d-b373-762c0dea8effprecedence_heuristic cell_id$740a3f41-9302-481d-b373-762c0dea8effdownstream_cells_map#update_gaussian_eligibility_vector!$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$d5020a8d-1dd7-403c-9d1f-665b95543943$20776e09-7d9b-4db8-a060-7bceeec65b47upstream_cells_mapexp:kfirstBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41VectorRealBinaryGaussianEligibilityVector$10cdd16e-a337-4421-a7a0-6de4e4b60c0fMatrix+NTuplelast$ba642a22-6623-482a-ab4a-81585b83e457precedence_heuristic cell_id$ba642a22-6623-482a-ab4a-81585b83e457downstream_cells_map$##average_continuing_runs_unmemoizedaverage_continuing_runs$734573e5-547b-4dcc-89bb-412aa6cc42d6$ff4f977e-48df-4c12-845c-c245b4d39d6d$8bc280db-e57d-4e40-be46-1790f4f7d9e7$11063fff-4d36-46d5-828f-dbed0f46b9cfupstream_cells_map:@memoizeRandom$df7f84e8-b42a-4001-9dbf-6bc3ced94207|>empty!IntegerRealdeepcopy/foldxt+Mapget!Random.seed!$d17a4bd0-5992-4247-912d-73d51758d2f3precedence_heuristic cell_id$d17a4bd0-5992-4247-912d-73d51758d2f3downstream_cells_mapupstream_cells_map@md_strgetindex$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7efprecedence_heuristic cell_id$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7efdownstream_cells_mapupstream_cells_map&cartpole_fcann_continuing_test_episode$64b38d1f-ecf9-4843-89a1-4c8953048265plot_cartpole_policy$602a07dd-8928-4b44-97e5-01c5cbf38351-cartpole_fcann_continuing_episode_step_select$6acb549a-5d90-4457-a347-d22448ad8071cartpole_continuing_fcann_test$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27$5ee4ce72-7740-4297-8d84-619e0708e4acprecedence_heuristic cell_id$5ee4ce72-7740-4297-8d84-619e0708e4acdownstream_cells_map)cartpole_continuing_fcann_parameter_studyupstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6:*actor_critic_with_eligibility_traces_fcann$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54|>setup_cartpole_problemscatterplot/foldxt+cartpole_fcann_feature_setup$61650a97-b353-4a85-b50b-93fee296ac7bMap3cartpole_fcann_feature_setup.update_feature_vector!Layout$645e93e7-e92e-49c4-9757-8294fabf4e9bprecedence_heuristic cell_id$645e93e7-e92e-49c4-9757-8294fabf4e9bdownstream_cells_mapupstream_cells_mapcartpole_continuing_test$3c89209c-9202-4d5d-841c-ea34be369616plot_continuing_step_rewards$0964133c-3a5b-433b-a8c4-a97813c37583$0c56b341-24eb-4c78-844e-182f44a7221aprecedence_heuristic cell_id$0c56b341-24eb-4c78-844e-182f44a7221adownstream_cells_mapupstream_cells_map^figure_13_1$d037ea92-915c-4dc7-97c6-d006d92e088a$d34d22ad-89c2-423e-91dd-bfb895dc6540precedence_heuristic cell_id$d34d22ad-89c2-423e-91dd-bfb895dc6540downstream_cells_mapcartpole_fcann_parameter_studyupstream_cells_map+actor_critic_fcann_episodic_parameter_study$f8614042-7c94-4d47-a1b6-4e96676b4e8bcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6cartpole_vector_update!$192b9f82-8d3a-408f-91c2-829cfcd32572cartpole_fcann_feature_setup$61650a97-b353-4a85-b50b-93fee296ac7b$20776e09-7d9b-4db8-a060-7bceeec65b47precedence_heuristic cell_id$20776e09-7d9b-4db8-a060-7bceeec65b47downstream_cells_mapEactor_critic_with_eligibility_traces_binary_features_gaussian_actions$55ba8725-0ddf-4196-a41d-3f3c490a8d84$61949faa-8174-4b7b-8fbc-01d5f850b419$b8532822-179b-4cd5-a279-4b71dafb544a$fee14dfe-c5ca-4126-a830-cc9d7eda5433upstream_cells_map binary_value_function$a540814a-57a1-4b98-9443-59e401425444make_n_param_dist_policy_params$ba41f521-4ee2-42a6-bf18-078bfa4b875ezerosBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41randIntegermake_gaussian_sampler$bba13634-ff0e-47f7-a23b-8d56098f4ac6VectorContinuousMDP$537270ba-122b-4f2b-880b-31d086766295#update_gaussian_eligibility_vector!$5261651e-a51e-4e80-8e23-83a4c10e5259$740a3f41-9302-481d-b373-762c0dea8effRealFunction&setup_binary_gaussian_policy_arguments$ba5d6311-daee-4abc-b2fb-fae2184ef3ebupdate_binary_value_gradient!$03a218cb-aa83-4000-85b5-c6f247087053MatrixNTupleUnion!update_binary_action_preferences!$a361f4c9-47ce-42ad-899c-87b611c0d471%actor_critic_with_eligibility_traces!$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90f$7856b8a0-565d-4c86-9b3c-4424ff9b86ddprecedence_heuristic cell_id$7856b8a0-565d-4c86-9b3c-4424ff9b86dddownstream_cells_mapupstream_cells_map$735b548a-88f5-4a30-ab8f-dfb3d6401b2bprecedence_heuristic cell_id$735b548a-88f5-4a30-ab8f-dfb3d6401b2bdownstream_cells_mapupstream_cells_map@md_strgetindex$7cf26604-9c2b-4a77-9674-7d4dac2f99f0precedence_heuristiccell_id$7cf26604-9c2b-4a77-9674-7d4dac2f99f0downstream_cells_mapupstream_cells_mapjoinpath@__DIR__include$87ee21f3-16ca-4c8c-a0b9-f9e2fd258a91precedence_heuristic cell_id$87ee21f3-16ca-4c8c-a0b9-f9e2fd258a91downstream_cells_mapupstream_cells_map@md_strgetindex$54f1546d-87ae-49d2-92ed-6fcc9b66e027precedence_heuristic cell_id$54f1546d-87ae-49d2-92ed-6fcc9b66e027downstream_cells_mapupstream_cells_map@md_strgetindex$63fbf8f4-e4e2-4893-be09-67450e92dbd7precedence_heuristic cell_id$63fbf8f4-e4e2-4893-be09-67450e92dbd7downstream_cells_mapplot_cart$fad02876-efba-46a7-9cb7-43820528779f$374af774-3a97-49b5-a3bb-bc3f7f63a3fa$1ce4bc6c-7cde-48e9-8ff1-7281697fd121upstream_cells_mapCartPoleStateHypertextLiteral.BypassindicatorHypertextLiteral.content@htl$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebInt64-scatterplotHypertextLiteral.ResultHypertextLiteral$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70attrcosLayoutsin$d5020a8d-1dd7-403c-9d1f-665b95543943precedence_heuristic cell_id$d5020a8d-1dd7-403c-9d1f-665b95543943downstream_cells_mapLreinforce_with_baseline_monte_carlo_control_linear_features_gaussian_actionsupstream_cells_map ,reinforce_with_baseline_monte_carlo_control!$4fb83451-b6f8-4e6e-a131-1accc8e10b08$5b868eba-c1af-49f6-8f93-79b78c319a6fmake_n_param_dist_policy_params$ba41f521-4ee2-42a6-bf18-078bfa4b875ezerosrandIntegermake_gaussian_sampler$bba13634-ff0e-47f7-a23b-8d56098f4ac6update_linear_value_gradient!$1753b5ed-c00b-4b60-b492-822180778e8ccopy#update_gaussian_eligibility_vector!$5261651e-a51e-4e80-8e23-83a4c10e5259$740a3f41-9302-481d-b373-762c0dea8effVectorContinuousMDP$537270ba-122b-4f2b-880b-31d086766295RealFunctionMatrix!update_linear_action_preferences!$581f7e9b-a5c2-4841-9605-85f9585b0274NTupleUnionlinear_value_function$0bf3b988-b3fb-49d5-8dde-b25766596363make_gaussian_params$37a8ef7e-e859-4ef0-81e2-76c02a324031precedence_heuristic cell_id$37a8ef7e-e859-4ef0-81e2-76c02a324031downstream_cells_mapupstream_cells_map@md_strgetindex$98229733-a71e-44ca-a52a-b7229cf8b422precedence_heuristic cell_id$98229733-a71e-44ca-a52a-b7229cf8b422downstream_cells_mapupstream_cells_map@md_strgetindex$42775fd1-5b27-48e0-abf1-9b22bb775e6dprecedence_heuristic cell_id$42775fd1-5b27-48e0-abf1-9b22bb775e6ddownstream_cells_mapupstream_cells_map#corridor_continuing_parameter_study$7afb6fb0-248a-4518-b94f-9876f81eca64continuing_study_params$7d94922e-dc9f-4953-b539-24aaa2c85b12$7dbb42a3-aa8c-47e5-b668-18e6325d4038precedence_heuristic cell_id$7dbb42a3-aa8c-47e5-b668-18e6325d4038downstream_cells_mapupstream_cells_map@md_strgetindex$192b9f82-8d3a-408f-91c2-829cfcd32572precedence_heuristic cell_id$192b9f82-8d3a-408f-91c2-829cfcd32572downstream_cells_mapcartpole_vector_update!$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27$d34d22ad-89c2-423e-91dd-bfb895dc6540upstream_cells_mapRealCartPoleStatecartpole_fcann_feature_setup$61650a97-b353-4a85-b50b-93fee296ac7b3cartpole_fcann_feature_setup.update_feature_vector!Vector$b5319d8b-0420-4ebf-b603-ea0b93365ac1precedence_heuristic cell_id$b5319d8b-0420-4ebf-b603-ea0b93365ac1downstream_cells_map&show_mountaincar_continuous_trajectory$c87dba8c-9a96-41b3-9dc7-a6c088ec1eaf$cd9c9eeb-c90d-4499-9503-7773d5250f47$5207308e-f636-4d47-b135-036a6e7b8ecd$a6be9a4c-d43b-4867-b7a2-07a46a9d0d8fupstream_cells_mapsumHypertextLiteral.BypassHypertextLiteral.content@htl$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebIntegerFunctionmountaincar_continuous_mdp$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2scatterplotHypertextLiteral.ResultHypertextLiteral$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70Layoutrunepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84$4cbdb082-22ba-49e9-a6ed-4380917625acprecedence_heuristic cell_id$4cbdb082-22ba-49e9-a6ed-4380917625acdownstream_cells_mapupstream_cells_map@md_strgetindex$cc80848a-6834-4272-9152-e17b45448814precedence_heuristic cell_id$cc80848a-6834-4272-9152-e17b45448814downstream_cells_mapwind_speedsupstream_cells_map:HypertextLiteral.BypassHypertextLiteral.ResultHypertextLiteral$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70PlutoUI.combinePlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70HypertextLiteral.content@htl$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebSlider$05bfd818-bf4e-4bda-baa9-5ba647867097precedence_heuristic cell_id$05bfd818-bf4e-4bda-baa9-5ba647867097downstream_cells_map4actor_critic_with_eligibility_traces_binary_features$3bccf6fc-6e5e-4f62-ad40-1ff0a3740728$396e0047-d848-462f-a769-0cc2829abc78$1f041cb3-618c-4380-a1ec-d7bbe4a80f62$72273f27-d0b9-4645-a609-cb65cc9332ee$734573e5-547b-4dcc-89bb-412aa6cc42d6$ff4f977e-48df-4c12-845c-c245b4d39d6d$8b35661b-5075-4d63-bc31-044407f99acf$3c89209c-9202-4d5d-841c-ea34be369616$b02ba928-5b9f-4695-b980-07988c788bb9$dca2f8e2-76af-4679-bf81-3824c15fc76d$6d0925d3-af96-4b94-8e2e-4941cce39e51upstream_cells_mapbinary_value_function$a540814a-57a1-4b98-9443-59e401425444setup_binary_policy_arguments$96506201-6b66-49e6-8179-06952e2394e1zerosBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41IntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84VectorRealFunction!update_binary_eligibility_vector!$042fbafe-2401-4fb7-ac13-4531e0782c79lengthupdate_binary_value_gradient!$03a218cb-aa83-4000-85b5-c6f247087053Matrix!update_binary_action_preferences!$a361f4c9-47ce-42ad-899c-87b611c0d471%actor_critic_with_eligibility_traces!$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90f$f0962801-0dfa-421f-8ffc-e64068e49913precedence_heuristic cell_id$f0962801-0dfa-421f-8ffc-e64068e49913downstream_cells_mapmountaincar_fcann_feature_setup$c251a630-7114-4188-9323-8d8feb5c32e0upstream_cells_mapfcann_feature_vector_setup$9acdbf38-2e10-45ec-85a0-d0db8453a599$11a55af7-5301-4507-bb26-88e1e11236dbprecedence_heuristic cell_id$11a55af7-5301-4507-bb26-88e1e11236dbdownstream_cells_mapupstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6display_cartpole_episode$822e4d69-2582-4956-858e-06ecb091e76a|>reinforce_test3$dca2f8e2-76af-4679-bf81-3824c15fc76drunepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84$ddbca73f-c692-46f2-95f3-a7dd849d33f7precedence_heuristic cell_id$ddbca73f-c692-46f2-95f3-a7dd849d33f7downstream_cells_mapupstream_cells_mapshow_mountaincar_trajectory$ba645f6b-143f-4e83-9003-707770ae308dmountaincar_test_train$6d0925d3-af96-4b94-8e2e-4941cce39e51$b4875f2b-5487-429f-80a3-d1032bbccfc1precedence_heuristic cell_id$b4875f2b-5487-429f-80a3-d1032bbccfc1downstream_cells_mapupstream_cells_map@md_strgetindex$0cd96c44-cae6-421f-9fae-26141600bef4precedence_heuristic cell_id$0cd96c44-cae6-421f-9fae-26141600bef4downstream_cells_mapupstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6display_cartpole_episode$822e4d69-2582-4956-858e-06ecb091e76a|>cartpole_continuing_test$3c89209c-9202-4d5d-841c-ea34be369616runepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84$51d6337d-c0bd-40a9-9129-7d88e41e4093precedence_heuristic cell_id$51d6337d-c0bd-40a9-9129-7d88e41e4093downstream_cells_mapupstream_cells_map$5859ca11-90f8-4fd6-88ed-c56efe796fe8precedence_heuristic cell_id$5859ca11-90f8-4fd6-88ed-c56efe796fe8downstream_cells_mapupstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6display_cartpole_episode$822e4d69-2582-4956-858e-06ecb091e76a|>runepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84reinforce_test2$d3b56fca-5b79-4465-8987-8d0005f854d8$3ea08816-705e-4be7-a175-dbd3f3e4c17dprecedence_heuristic cell_id$3ea08816-705e-4be7-a175-dbd3f3e4c17ddownstream_cells_mapupstream_cells_map@md_strgetindex$f3e2db06-9cb7-464a-96b8-938175efd26bprecedence_heuristic cell_id$f3e2db06-9cb7-464a-96b8-938175efd26bdownstream_cells_mapsetup_fcann_value_arguments$e1aec891-d95a-47d1-97d7-d2a4cfb16e64upstream_cells_map fcann_value_function$635abb34-2c97-4f04-a74c-22fbec32f408onescale_fcann_params!$77cf3a74-899f-4ade-99f2-5aaf7a98c02dFCANN.makeorthonormalrandVectorlengthNamedTupleRealeachindexdeepcopy/^last==:FCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84zerosBoolIntegerendInt64FCANN.form_activations+*update_fcann_value_gradient!$5c4a383f-fcf2-4f2b-819f-6d84471dda00$b2082ab0-73a4-45a6-8772-a2e6e22b519aprecedence_heuristic cell_id$b2082ab0-73a4-45a6-8772-a2e6e22b519adownstream_cells_mapmake_beta_n_samplermake_beta_sampler$3e3c5897-809f-46e3-bb58-f115b082443ebeta_action_samplerupstream_cells_mapzeroexpmaxisnanislessrandIntegerVectorepsRealBetaVal+NTuplentuple$a361f4c9-47ce-42ad-899c-87b611c0d471precedence_heuristic cell_id$a361f4c9-47ce-42ad-899c-87b611c0d471downstream_cells_map!update_binary_action_preferences!$042fbafe-2401-4fb7-ac13-4531e0782c79$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2$05bfd818-bf4e-4bda-baa9-5ba647867097$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098upstream_cells_map zero:islessBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41julia.simdloop@inboundsnothingVectorrandReallengthisless@inboundsoneviewnothingcopyzerosjulia.simdloopsizerandIntegerBase<=Base.simd_outer_rangeBase.simd_inner_lengthfoldxt+ArrayMap$6d0925d3-af96-4b94-8e2e-4941cce39e51precedence_heuristic cell_id$6d0925d3-af96-4b94-8e2e-4941cce39e51downstream_cells_mapmountaincar_test_train$dc2efc6c-8da8-425b-aa5f-290949109565$ddbca73f-c692-46f2-95f3-a7dd849d33f7upstream_cells_mapInt644actor_critic_with_eligibility_traces_binary_features$05bfd818-bf4e-4bda-baa9-5ba647867097mountaincar_tilecoding_setup$7c592385-e8d3-4efe-962c-d39debb64405MountainCarTasktypemax$6bb0263e-368e-462a-948c-baf9cfa82512precedence_heuristic cell_id$6bb0263e-368e-462a-948c-baf9cfa82512downstream_cells_mapget_corridor_features $f2f2dd1d-180c-4d36-b515-5079d129f93a$e1493cea-19c4-475d-98a0-86d27fb04af1$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72c$d037ea92-915c-4dc7-97c6-d006d92e088a$f2ed56c9-c2b7-42cb-a083-e12aeaa126ef$cbea5840-49d2-4e91-be9c-f5f15666d78a$83ca0577-15d7-4448-b597-c77810b812bf$e5c1aca8-7575-4835-8273-e69ca0a55fe8$7d63b960-3998-4f7b-8cbb-ccd49db9aeac$646bc853-b7fc-49fa-a201-ff98e8f952d4$3bccf6fc-6e5e-4f62-ad40-1ff0a3740728$396e0047-d848-462f-a769-0cc2829abc78$bc8a399b-8864-4473-89d2-e3b0a03d15b5$72273f27-d0b9-4645-a609-cb65cc9332ee$7afb6fb0-248a-4518-b94f-9876f81eca64$8b35661b-5075-4d63-bc31-044407f99acfupstream_cells_map:$72273f27-d0b9-4645-a609-cb65cc9332eeprecedence_heuristic cell_id$72273f27-d0b9-4645-a609-cb65cc9332eedownstream_cells_mapupstream_cells_mapcorridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512^4actor_critic_with_eligibility_traces_binary_features$05bfd818-bf4e-4bda-baa9-5ba647867097$87482ea5-5265-4e02-92c0-1a8bb44ff0f4precedence_heuristic cell_id$87482ea5-5265-4e02-92c0-1a8bb44ff0f4downstream_cells_map@actor_critic_binary_continuing_squashed_gaussian_parameter_studyupstream_cells_map Random$df7f84e8-b42a-4001-9dbf-6bc3ced94207ContinuousMDP$537270ba-122b-4f2b-880b-31d086766295copyVectorRealscatter/MatrixNactor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions$05f120be-9695-4824-82fd-142a0df13098$717e4c69-59d5-4929-923f-dd35a97fb160:AbstractVector|>make_n_param_dist_policy_params$ba41f521-4ee2-42a6-bf18-078bfa4b875ezerosrandIntegerFunctionUInt64plotfoldxt+MapLayoutRandom.seed!$3bafd7df-9bc0-4d13-874d-739590cf3ad9precedence_heuristic cell_id$3bafd7df-9bc0-4d13-874d-739590cf3ad9downstream_cells_mapupstream_cells_map@md_strgetindex$f27f2bcd-05b6-44fe-bf9e-a3e51556db7cprecedence_heuristic cell_id$f27f2bcd-05b6-44fe-bf9e-a3e51556db7cdownstream_cells_mapcartpole_functions$5d434c83-c9ca-499f-8695-c7733031c2de$4c4e643b-d4b9-44f0-8d30-dc521bcc55ac$de3cba34-9842-44d1-9b79-47126c0a0751upstream_cells_mapcreate_cartpole_functions$352d2952-cb83-47d3-9078-2b2ef9927443$41dc149d-c6f3-4b0d-a856-06f3aaae3049precedence_heuristic cell_id$41dc149d-c6f3-4b0d-a856-06f3aaae3049downstream_cells_mapBinaryEligibilityVector$b0a66a19-ee76-463b-a704-8fcee85444d0$042fbafe-2401-4fb7-ac13-4531e0782c79$96506201-6b66-49e6-8179-06952e2394e1$25be5dcf-be63-46c4-b6de-6cf79fa28fd0upstream_cells_mapInt64BinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41Vector$38e5d800-4d43-40d2-87ea-f7d4b4283dabprecedence_heuristic cell_id$38e5d800-4d43-40d2-87ea-f7d4b4283dabdownstream_cells_mapupstream_cells_map@md_strgetindex$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2precedence_heuristic cell_id$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2downstream_cells_map%one_step_actor_critic_binary_features$7d63b960-3998-4f7b-8cbb-ccd49db9aeac$646bc853-b7fc-49fa-a201-ff98e8f952d4upstream_cells_mapbinary_value_function$a540814a-57a1-4b98-9443-59e401425444setup_binary_policy_arguments$96506201-6b66-49e6-8179-06952e2394e1zerosBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41IntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84VectorRealone_step_actor_critic!$4d4ae57b-afc3-44f9-b6fc-892f59f82921Function!update_binary_eligibility_vector!$042fbafe-2401-4fb7-ac13-4531e0782c79lengthupdate_binary_value_gradient!$03a218cb-aa83-4000-85b5-c6f247087053Matrix!update_binary_action_preferences!$a361f4c9-47ce-42ad-899c-87b611c0d471$73b90260-d57a-449a-8db6-47f91e6a4e4fprecedence_heuristic cell_id$73b90260-d57a-449a-8db6-47f91e6a4e4fdownstream_cells_mapupstream_cells_map@md_strgetindex$5aba4f96-e877-457e-8e95-18737348f99fprecedence_heuristic cell_id$5aba4f96-e877-457e-8e95-18737348f99fdownstream_cells_map"actor_critic_fcann_parameter_study$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62$c251a630-7114-4188-9323-8d8feb5c32e0upstream_cells_map@NamedTuple:IntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84VectorInt64RealFunction-Base^+$fed4dc4c-0d1c-4ee3-9d0e-8ef2a7db7486precedence_heuristic cell_id$fed4dc4c-0d1c-4ee3-9d0e-8ef2a7db7486downstream_cells_map$mountaincar_continuing_binary_params$04f42c09-8ab5-4233-b196-51c4aa2dcedbupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunner(create_actor_critic_continuing_params_UI$5b15d91e-7119-4f85-a54a-7d4f1fdaf097Core.applicable@bindBase.get$27487ad0-4779-42ce-8def-e660ef04bee0precedence_heuristic cell_id$27487ad0-4779-42ce-8def-e660ef04bee0downstream_cells_mapupstream_cells_mapreinforce_test4$407a0724-4bb6-4c83-ab2d-17a0e19c4072cartpole_setup$26880577-d267-4950-8725-7afe0d0402b6 reinforce_test4.policy_and_value6cartpole_setup.mdps.episodic.discrete.initialize_state$0d93132d-5819-47dc-8cf2-462d480d9c3dprecedence_heuristic cell_id$0d93132d-5819-47dc-8cf2-462d480d9c3ddownstream_cells_mapupstream_cells_mapmountaincar_continuous_mdp$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2@md_str<>islessmountaincar_tilecoding_setup$7c592385-e8d3-4efe-962c-d39debb64405%mountaincar_binary_continuous_params2$0d45ae72-572f-4d17-83cf-9814f2854131>actor_critic_binary_episodic_squashed_gaussian_parameter_study$08505e88-9c23-4e95-91e3-d18bf5133dbc$bd6a7c16-6c25-4fc2-8e1b-4dab693ce19f$c5a2879c-e89b-47f7-bbd6-48200d7e89e38run_mountaincar_binary_episodic_countinuous_param_study2$e524f8cc-ab69-4f8b-a59f-28156696a104getindex$9978d537-49ff-4014-a971-b42704c50a6bprecedence_heuristic cell_id$9978d537-49ff-4014-a971-b42704c50a6bdownstream_cells_mapfcann_cartpole_study_paramsupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCore.applicable@bindBase.get#create_actor_critic_fcann_params_UI$5eebf3da-bfe7-46eb-81a3-f87f334ee270$f8215517-b18f-4a03-9421-8edab4ca8089precedence_heuristic cell_id$f8215517-b18f-4a03-9421-8edab4ca8089downstream_cells_mapupstream_cells_mapshow_squashed_policy$ff3009eb-23f9-44fe-8e56-85dbc7b463d0test_mountaincar_state$b7f77935-bcab-4ef1-8e1b-a7d059784ff3"mountaincar_continuous_test_train3$5eb8d9f9-8512-4e00-8cb5-cec68d73cc7d$1ac9296f-047b-4051-ba5c-0c23d5f9cde9precedence_heuristic cell_id$1ac9296f-047b-4051-ba5c-0c23d5f9cde9downstream_cells_mapcorridor_continuing_mdp$7afb6fb0-248a-4518-b94f-9876f81eca64$8b35661b-5075-4d63-bc31-044407f99acfupstream_cells_mapmake_corridor_continuing_mdp$f0104778-81a6-417b-8501-f916e5e7f3af$c87dba8c-9a96-41b3-9dc7-a6c088ec1eafprecedence_heuristic cell_id$c87dba8c-9a96-41b3-9dc7-a6c088ec1eafdownstream_cells_mapupstream_cells_map!mountaincar_continuous_test_train$b8532822-179b-4cd5-a279-4b71dafb544a&show_mountaincar_continuous_trajectory$b5319d8b-0420-4ebf-b603-ea0b93365ac1$5cc4d12d-b537-47e2-8109-4e7a234fdf25precedence_heuristic cell_id$5cc4d12d-b537-47e2-8109-4e7a234fdf25downstream_cells_mapmake_corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeupstream_cells_mapmaxStateMDPTransitionSampler$d963ff6d-f1b6-4799-aa0e-1ae100310d84-isless+*iseven==IntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84$5334064b-5a16-4135-afa0-86a48291725bprecedence_heuristic cell_id$5334064b-5a16-4135-afa0-86a48291725bdownstream_cells_mapupstream_cells_mapcorridor_train.value_functioncorridor_train$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72c$9c342958-1971-48ec-b919-5dfdcbc915a4precedence_heuristic cell_id$9c342958-1971-48ec-b919-5dfdcbc915a4downstream_cells_mapbgcolor$e5faaa1b-88cb-43e2-8d04-8972b58b4bdaupstream_cells_map@md_strBaseColorStringPickerPlutoRunner.create_bondPlutoRunnerCoreCore.applicableBase.get@bindgetindex$966ef17c-23be-49dc-bc37-4cb52b34c049precedence_heuristic cell_id$966ef17c-23be-49dc-bc37-4cb52b34c049downstream_cells_mapupstream_cells_map@md_strgetindex$e7e49ff8-32df-48a4-afb2-462859592e92precedence_heuristic cell_id$e7e49ff8-32df-48a4-afb2-462859592e92downstream_cells_map&form_state_and_policy_function_outputs$4d4ae57b-afc3-44f9-b6fc-892f59f82921$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90fupstream_cells_mapVectorsample_action$d963ff6d-f1b6-4799-aa0e-1ae100310d84deepcopyform_state_policy_function$37ec6802-d4c2-4470-ad69-439d5a732f77form_state_value_function$e7566274-5518-4e28-8738-d4b1747d0cfbFunctioncopy$78c83673-2117-4542-b4d8-1c243e8f610bprecedence_heuristic cell_id$78c83673-2117-4542-b4d8-1c243e8f610bdownstream_cells_mapupstream_cells_map@md_strgetindex$a6be9a4c-d43b-4867-b7a2-07a46a9d0d8fprecedence_heuristic cell_id$a6be9a4c-d43b-4867-b7a2-07a46a9d0d8fdownstream_cells_mapupstream_cells_map&mountaincar_continuous_test_train_beta$4156d955-9daf-4429-b152-e8332980fb9e&show_mountaincar_continuous_trajectory$b5319d8b-0420-4ebf-b603-ea0b93365ac1mountaincar_continuous_beta_mdp$8e096fae-9941-49d8-ae87-c68b02f68da5$396e0047-d848-462f-a769-0cc2829abc78precedence_heuristic cell_id$396e0047-d848-462f-a769-0cc2829abc78downstream_cells_mapupstream_cells_mapInt64corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512^4actor_critic_with_eligibility_traces_binary_features$05bfd818-bf4e-4bda-baa9-5ba647867097typemax$ff4f977e-48df-4c12-845c-c245b4d39d6dprecedence_heuristic cell_id$ff4f977e-48df-4c12-845c-c245b4d39d6ddownstream_cells_map#actor_critic_linear_parameter_study$7afb6fb0-248a-4518-b94f-9876f81eca64$1b102220-6d78-480d-a77f-0e57bad23dca$d57375a5-b9e0-4742-b5f7-6a7da891604aupstream_cells_map:AbstractVectorzerosrandIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84UInt64RealFunctionlength4actor_critic_with_eligibility_traces_binary_features$05bfd818-bf4e-4bda-baa9-5ba647867097Matrixaverage_continuing_runs$ba642a22-6623-482a-ab4a-81585b83e457DataFrame4actor_critic_with_eligibility_traces_linear_features$68806899-9972-460a-9f11-daa708a9d610$aa450da4-fe84-4eea-b6c4-9820b7982437precedence_heuristic cell_id$aa450da4-fe84-4eea-b6c4-9820b7982437downstream_cells_mapupstream_cells_map@md_strgetindex$bb1ef180-39ac-475f-beea-ef573e71a3bfprecedence_heuristic cell_id$bb1ef180-39ac-475f-beea-ef573e71a3bfdownstream_cells_mapupstream_cells_mapdisplay_cartpole_episode$822e4d69-2582-4956-858e-06ecb091e76a|>ep2$a5b002c9-5e11-462a-9da0-6e060c7963f8$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27precedence_heuristic cell_id$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27downstream_cells_mapcartpole_continuing_fcann_test$04b5929a-2058-49c9-963a-96c752a1d67d$64b38d1f-ecf9-4843-89a1-4c8953048265$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7ef$fd58402f-da65-44cf-b81a-e21192fd0e63upstream_cells_map*actor_critic_with_eligibility_traces_fcann$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54cartpole_vector_update!$192b9f82-8d3a-408f-91c2-829cfcd32572cartpole_continuing_mdp$4c4e643b-d4b9-44f0-8d30-dc521bcc55accartpole_fcann_feature_setup$61650a97-b353-4a85-b50b-93fee296ac7b$5b868eba-c1af-49f6-8f93-79b78c319a6fprecedence_heuristic cell_id$5b868eba-c1af-49f6-8f93-79b78c319a6fdownstream_cells_map,reinforce_with_baseline_monte_carlo_control!$0ac7ea44-14f6-4e80-80f9-d6df8059bb38$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$d1ed25e6-60c6-411f-a541-99986e5da2c5$697b2310-9d96-4f7f-be62-c3bd6bf736f3$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$d5020a8d-1dd7-403c-9d1f-665b95543943upstream_cells_map zerooneContinuousMDP$537270ba-122b-4f2b-880b-31d086766295copyVector%form_state_continuous_policy_function$f545c800-0bf3-491f-9d7d-42341cfdb573Realeachindexrunepisode!$d963ff6d-f1b6-4799-aa0e-1ae100310d84deepcopy/^update_params_with_gradient!$b0a66a19-ee76-463b-a704-8fcee85444d0$a893a87b-2d07-4db5-9d1a-9da8646216f4$f55afa58-962d-4551-8d95-a5b467d61adfrunepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84:zerosIntegerFunctionInt64-+*form_state_value_function$e7566274-5518-4e28-8738-d4b1747d0cfb$68469a40-7976-48b7-b7a1-eaa4c5f33a18precedence_heuristic cell_id$68469a40-7976-48b7-b7a1-eaa4c5f33a18downstream_cells_map"plot_mountaincar_continuous_values$d7f6ff79-3c0f-4f16-aa1c-3bc534ce580a$b695ef21-a1ac-4d1f-a0e1-71cd81cede18$a0ca7a5e-0089-4a45-9278-c0f27cd096a0upstream_cells_mapHypertextLiteral.BypassLinRangezerosHypertextLiteral.content@htl$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebFunctionenumerateFloat32plotHypertextLiteral$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70heatmapHypertextLiteral.ResultLayout$2a586e46-66e4-461a-85c8-5817e4d1aa43precedence_heuristic cell_id$2a586e46-66e4-461a-85c8-5817e4d1aa43downstream_cells_mapupstream_cells_map@md_strgetindex$a206c759-3f6e-4003-8cba-5f6ce6742646precedence_heuristic cell_id$a206c759-3f6e-4003-8cba-5f6ce6742646downstream_cells_mapupstream_cells_map@md_strgetindex$fc3dcd26-c5cf-4141-bf6c-eaed5fc9bb1dprecedence_heuristic cell_id$fc3dcd26-c5cf-4141-bf6c-eaed5fc9bb1ddownstream_cells_mapupstream_cells_map@md_strgetindex$3cfd63ad-b1a2-4b99-ae97-2ff10351e4f5precedence_heuristic cell_id$3cfd63ad-b1a2-4b99-ae97-2ff10351e4f5downstream_cells_mapupstream_cells_map@md_strgetindex$31db0f58-28e4-454f-9394-25565687266fprecedence_heuristic cell_id$31db0f58-28e4-454f-9394-25565687266fdownstream_cells_mapupstream_cells_maprandndisplay_cartpole_episode$822e4d69-2582-4956-858e-06ecb091e76aFloat32|>cartpole_mdps$024dcd1a-8eaa-4a95-8037-2f578828309crunepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84$822e4d69-2582-4956-858e-06ecb091e76aprecedence_heuristic cell_id$822e4d69-2582-4956-858e-06ecb091e76adownstream_cells_mapdisplay_cartpole_episode$0cd96c44-cae6-421f-9fae-26141600bef4$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fce$31db0f58-28e4-454f-9394-25565687266f$dddc4a2f-34b2-41dc-85b3-55aba4880fa6$5859ca11-90f8-4fd6-88ed-c56efe796fe8$11a55af7-5301-4507-bb26-88e1e11236db$07ba9fe4-aaa7-4123-9865-cbfa79d0d44a$daf35bfe-8f9c-4f55-971d-4d443be8f8bf$bb1ef180-39ac-475f-beea-ef573e71a3bfupstream_cells_mapgetfieldCartPoleStateenumeratescatterplotattrLayoutVector$d7f6ff79-3c0f-4f16-aa1c-3bc534ce580aprecedence_heuristic cell_id$d7f6ff79-3c0f-4f16-aa1c-3bc534ce580adownstream_cells_mapupstream_cells_map"plot_mountaincar_continuous_values$68469a40-7976-48b7-b7a1-eaa4c5f33a18!mountaincar_continuous_test_train$b8532822-179b-4cd5-a279-4b71dafb544a$05b0fcad-628b-48d2-aa24-f6f562dbb660precedence_heuristic cell_id$05b0fcad-628b-48d2-aa24-f6f562dbb660downstream_cells_mapupstream_cells_map@md_strgetindex$d2729657-d0bf-4d39-8ec7-f242a1ad48d6precedence_heuristic cell_id$d2729657-d0bf-4d39-8ec7-f242a1ad48d6downstream_cells_map)create_continuous_action_mountaincar_beta$8e096fae-9941-49d8-ae87-c68b02f68da5upstream_cells_mapMountainCarTask.step-*ContinuousMDP$537270ba-122b-4f2b-880b-31d086766295MountainCarTask$5c11a92d-7496-4aba-af15-2537eac49dd7precedence_heuristic cell_id$5c11a92d-7496-4aba-af15-2537eac49dd7downstream_cells_mapFCANNActivations$cc3ac95e-a398-438a-ba3d-62b6733f6342$5c4a383f-fcf2-4f2b-819f-6d84471dda00$635abb34-2c97-4f04-a74c-22fbec32f408upstream_cells_mapRealVector$1753b5ed-c00b-4b60-b492-822180778e8cprecedence_heuristic cell_id$1753b5ed-c00b-4b60-b492-822180778e8cdownstream_cells_mapupdate_linear_value_gradient!$d1ed25e6-60c6-411f-a541-99986e5da2c5$57e5e12a-b722-4ea3-ab3b-e5711029e640$68806899-9972-460a-9f11-daa708a9d610$d5020a8d-1dd7-403c-9d1f-665b95543943upstream_cells_mapRealVector$f7ede764-5ad8-426b-a805-cc21b622d977precedence_heuristic cell_id$f7ede764-5ad8-426b-a805-cc21b622d977downstream_cells_mapupstream_cells_map@md_strgetindex$36d514fa-b27a-4c6b-8399-9d108377b9b5precedence_heuristic cell_id$36d514fa-b27a-4c6b-8399-9d108377b9b5downstream_cells_mapstudy_params$c52c4cec-0ea8-4af3-831a-d284f0e086eeupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCore.applicable@bindBase.getcreate_actor_critic_params_UI$a8b40b8f-051a-4e6f-a079-ece4f32873de$6b1acb57-159a-4b7f-99fe-5f996522243bprecedence_heuristic cell_id$6b1acb57-159a-4b7f-99fe-5f996522243bdownstream_cells_mapupstream_cells_map$45f0a385-6465-4acc-8637-1b007a0fe215precedence_heuristic cell_id$45f0a385-6465-4acc-8637-1b007a0fe215downstream_cells_map update_fcann_eligibility_vector!$0e9de19e-bcd4-40ac-9831-afb6cad38422upstream_cells_map:CrossEntropyLoss$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84@inboundsFCANNParamsIntegerVectoreachindexFloat32*FCANN.nnCostFunction$c52c4cec-0ea8-4af3-831a-d284f0e086eeprecedence_heuristic cell_id$c52c4cec-0ea8-4af3-831a-d284f0e086eedownstream_cells_mapupstream_cells_map:^+study_params$36d514fa-b27a-4c6b-8399-9d108377b9b5corridor_parameter_study$bc8a399b-8864-4473-89d2-e3b0a03d15b5$f8614042-7c94-4d47-a1b6-4e96676b4e8bprecedence_heuristic cell_id$f8614042-7c94-4d47-a1b6-4e96676b4e8bdownstream_cells_map+actor_critic_fcann_episodic_parameter_study$d34d22ad-89c2-423e-91dd-bfb895dc6540upstream_cells_map !Random$df7f84e8-b42a-4001-9dbf-6bc3ced94207FilterStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84VectorRealscatterisemptymeanmissing:AbstractVector*actor_critic_with_eligibility_traces_fcann$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54|>ismissingrandIntegerFunctionUInt64tcollectInt64plotMapLayoutRandom.seed!$76eb6743-cac0-4174-9ba3-a0691c200b54precedence_heuristic cell_id$76eb6743-cac0-4174-9ba3-a0691c200b54downstream_cells_mapmake_n_param_dist_params$ba5d6311-daee-4abc-b2fb-fae2184ef3eb$ed93259c-7b8b-46d7-97fb-f194e0e04b3a$4e29c621-223e-4859-8e96-db04b967815aupstream_cells_mapzerosNTuple*IntegerReal$94517664-6988-44dc-a297-e9d5873ee540precedence_heuristic cell_id$94517664-6988-44dc-a297-e9d5873ee540downstream_cells_mapsquashed_gaussian_plot_params$3e7cecec-eb77-4862-8e3c-b510422e06dbupstream_cells_map@md_strCore:PlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70Base.get@bindSliderBasePlutoRunnerPlutoRunner.create_bondCore.applicablePlutoUI.combinegetindex$d037ea92-915c-4dc7-97c6-d006d92e088aprecedence_heuristic cell_id$d037ea92-915c-4dc7-97c6-d006d92e088adownstream_cells_mapfigure_13_1$0c56b341-24eb-4c78-844e-182f44a7221aupstream_cells_map foldxtLayout:*sqrtRandom$df7f84e8-b42a-4001-9dbf-6bc3ced94207|>Int64corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512scatter-/plot+log2-reinforce_monte_carlo_control_binary_features$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290fillMaproundRandom.seed!$24fa139c-ad4b-49db-ac8f-23c476ed8608precedence_heuristic cell_id$24fa139c-ad4b-49db-ac8f-23c476ed8608downstream_cells_mapreinforce_test$dddc4a2f-34b2-41dc-85b3-55aba4880fa6$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5upstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6^Lreinforce_with_baseline_monte_carlo_control_binary_features_gaussian_actions$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$2025ff38-f2ec-4224-b771-ff72ffe1af28precedence_heuristic cell_id$2025ff38-f2ec-4224-b771-ff72ffe1af28downstream_cells_mapmountaincar_min_vals$023f67b8-8f38-470a-9766-ac60a75678aa$7c592385-e8d3-4efe-962c-d39debb64405upstream_cells_map$cb70d400-3e9c-441c-b17c-e727e8c928f3precedence_heuristic cell_id$cb70d400-3e9c-441c-b17c-e727e8c928f3downstream_cells_mapupstream_cells_map@md_str<.start_mountaincar_continuing_fcann_param_study$f487f2dd-ad09-48ac-ae34-bf50cfa6ac7d>#mountaincar_continuing_fcann_params$5d35e515-e2d3-443e-becf-eb28c25db346,mountaincar_fcann_continuing_parameter_study$c251a630-7114-4188-9323-8d8feb5c32e0islessgetindex$e034b9cb-f4ee-46f4-bea6-72c93c75d966precedence_heuristiccell_id$e034b9cb-f4ee-46f4-bea6-72c93c75d966downstream_cells_mapDataFramesupstream_cells_map$e6cf9550-2e69-4b82-92cf-5e07a35490aaprecedence_heuristic cell_id$e6cf9550-2e69-4b82-92cf-5e07a35490aadownstream_cells_mapzero_params!$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90fupstream_cells_mapzeroeachindex:ArrayFCANNParamsReal$717e4c69-59d5-4929-923f-dd35a97fb160precedence_heuristic cell_id$717e4c69-59d5-4929-923f-dd35a97fb160downstream_cells_mapNactor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions$08505e88-9c23-4e95-91e3-d18bf5133dbc$87482ea5-5265-4e02-92c0-1a8bb44ff0f4$5eb8d9f9-8512-4e00-8cb5-cec68d73cc7dupstream_cells_mapRealFunctionNTupleUniononeIntegerContinuousMDP$537270ba-122b-4f2b-880b-31d086766295$1386ffdb-940d-4f1b-a872-4e38647b5335precedence_heuristic cell_id$1386ffdb-940d-4f1b-a872-4e38647b5335downstream_cells_mapupstream_cells_map@md_strgetindex$a893a87b-2d07-4db5-9d1a-9da8646216f4precedence_heuristic cell_id$a893a87b-2d07-4db5-9d1a-9da8646216f4downstream_cells_mapupdate_params_with_gradient!$4fb83451-b6f8-4e6e-a131-1accc8e10b08$4d4ae57b-afc3-44f9-b6fc-892f59f82921$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$5b868eba-c1af-49f6-8f93-79b78c319a6f$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90fupstream_cells_map zero:islessBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41julia.simdloop@inboundsnothingVectorreinforce_test5$82e0e9a0-9662-429a-87e3-e6bdae02709aplotcumsumround$fac138d9-3c5d-44b0-a87c-b13872f19450precedence_heuristiccell_id$fac138d9-3c5d-44b0-a87c-b13872f19450downstream_cells_mapMemoizeupstream_cells_map$82e0e9a0-9662-429a-87e3-e6bdae02709aprecedence_heuristic cell_id$82e0e9a0-9662-429a-87e3-e6bdae02709adownstream_cells_mapreinforce_test5$27441783-d3c6-40be-9c36-4941613e6ae9$daf35bfe-8f9c-4f55-971d-4d443be8f8bf$a5b002c9-5e11-462a-9da0-6e060c7963f8$d4e87ac4-6008-43b2-aa06-e232ec2b2b5b$700dcbc4-c94c-4287-8cf0-0b2c7a320a3aupstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6*actor_critic_with_eligibility_traces_fcann$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54cartpole_fcann_feature_setup$61650a97-b353-4a85-b50b-93fee296ac7b3cartpole_fcann_feature_setup.update_feature_vector!$d3c1379f-acd6-4e15-be7e-a5dbe46a4f62precedence_heuristic cell_id$d3c1379f-acd6-4e15-be7e-a5dbe46a4f62downstream_cells_map(start_mountaincar_continuing_param_study$04f42c09-8ab5-4233-b196-51c4aa2dcedbupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCounterButtonCore.applicable@bindBase.get$fad02876-efba-46a7-9cb7-43820528779fprecedence_heuristic cell_id$fad02876-efba-46a7-9cb7-43820528779fdownstream_cells_mapupstream_cells_map&cartpole_fcann_continuing_test_episode$64b38d1f-ecf9-4843-89a1-4c8953048265plot_cart$63fbf8f4-e4e2-4893-be09-67450e92dbd7-cartpole_fcann_continuing_episode_step_select$6acb549a-5d90-4457-a347-d22448ad8071$1ce4bc6c-7cde-48e9-8ff1-7281697fd121precedence_heuristic cell_id$1ce4bc6c-7cde-48e9-8ff1-7281697fd121downstream_cells_mapupstream_cells_mapep2_step$9bce6fdb-2cbc-4758-9a8b-794e490c973dep2$a5b002c9-5e11-462a-9da0-6e060c7963f8plot_cart$63fbf8f4-e4e2-4893-be09-67450e92dbd7$024dcd1a-8eaa-4a95-8037-2f578828309cprecedence_heuristic cell_id$024dcd1a-8eaa-4a95-8037-2f578828309cdownstream_cells_mapcartpole_mdps$cf1859d6-f889-4923-8c87-2d7c039f26c3$31db0f58-28e4-454f-9394-25565687266f$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5upstream_cells_mapcreate_cartpole_mdps$3c316495-bb6c-41e2-a38f-ba867a319fbb$e1274f57-75cb-4659-a82f-e5870c5367e2precedence_heuristic cell_id$e1274f57-75cb-4659-a82f-e5870c5367e2downstream_cells_mapep$a4eec4d3-5a75-4b52-ab9c-9d9e83d5547d$374af774-3a97-49b5-a3bb-bc3f7f63a3fa$af144759-fe66-4ad0-b378-e9eb4e859db4upstream_cells_mapreinforce_test4$407a0724-4bb6-4c83-ab2d-17a0e19c4072cartpole_setup$26880577-d267-4950-8725-7afe0d0402b6runepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84$fdd3f4fd-4706-4d6b-b150-6ee6b4b370cbprecedence_heuristic cell_id$fdd3f4fd-4706-4d6b-b150-6ee6b4b370cbdownstream_cells_mapupstream_cells_map@md_strgetindex$b02ba928-5b9f-4695-b980-07988c788bb9precedence_heuristic cell_id$b02ba928-5b9f-4695-b980-07988c788bb9downstream_cells_map mountaincar_continuing_tile_test$98222fcd-b456-477c-90dd-844df36877e5$0ce66c9d-6d1c-4c2d-8178-5bcdfa247cd6$e89bdc84-dbb5-4c73-a39c-6392e5f79704$da3cb392-78f2-48b2-b0dc-5f016664798cupstream_cells_map4actor_critic_with_eligibility_traces_binary_features$05bfd818-bf4e-4bda-baa9-5ba647867097mountaincar_tilecoding_setup$7c592385-e8d3-4efe-962c-d39debb64405mountaincar_continuing_mdp$46fea69b-599e-46ab-8455-d2da865d9a8e$f946c886-6246-4f98-a96f-f06984691ad8precedence_heuristic cell_id$f946c886-6246-4f98-a96f-f06984691ad8downstream_cells_mapApproximationUtils.runepisode!ApproximationUtils.runepisodeupstream_cells_map $@assertTuple>isless!ContinuousMDP$537270ba-122b-4f2b-880b-31d086766295Base.CoreLogging.!length=$3c316495-bb6c-41e2-a38f-ba867a319fbbprecedence_heuristic cell_id$3c316495-bb6c-41e2-a38f-ba867a319fbbdownstream_cells_mapcreate_cartpole_mdps$024dcd1a-8eaa-4a95-8037-2f578828309c$fddef10c-7695-4596-9e16-987fd45a57e6upstream_cells_map zeroTabularRL$d963ff6d-f1b6-4799-aa0e-1ae100310d84>islessoneContinuousMDP$537270ba-122b-4f2b-880b-31d086766295Realtypemaxsarsa_λ$553b0ceb-f2ca-41ee-99bc-9f53a4487b49precedence_heuristic cell_id$553b0ceb-f2ca-41ee-99bc-9f53a4487b49downstream_cells_mapupstream_cells_mapbest_mc_corridor$a12b92d1-e045-4f92-b8cd-eee5d56fa67dget_corridor_episode_stats$fb8904a9-ae64-41cc-93b6-5a25855edad0$cecc2a35-3850-4f66-84e8-e29da4f3d4b0$f9facbba-39d4-483e-9066-275603156db0precedence_heuristic cell_id$f9facbba-39d4-483e-9066-275603156db0downstream_cells_mapplot_mountaincar_values$e89bdc84-dbb5-4c73-a39c-6392e5f79704$c0876a48-ea18-494d-8bfc-e2bceb73b417$d82e7ab8-c372-4462-afb5-1617560cdb56upstream_cells_mapHypertextLiteral.BypassLinRangezerosHypertextLiteral.content@htl$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebenumerateFloat32plotHypertextLiteral$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70heatmapHypertextLiteral.ResultLayout$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0precedence_heuristic cell_id$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0downstream_cells_mapupstream_cells_mapone_step_actor_critic_fcann$57bbdb10-bed8-459d-8f67-9ea637cf12baInt64corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabe^update_corridor_features!$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811typemax$d41f1dd1-45fe-4456-9a01-ed47fd6704a7precedence_heuristic cell_id$d41f1dd1-45fe-4456-9a01-ed47fd6704a7downstream_cells_mapupdate_beta_eligibility_vector!$3e3c5897-809f-46e3-bb58-f115b082443eupstream_cells_mapexp:kfirstBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41VectorRealBinaryBetaEligibilityVector$54fff14b-cf53-47b0-9cfa-8b9ee33df54eMatrix+NTuplelast$ba5d6311-daee-4abc-b2fb-fae2184ef3ebprecedence_heuristic cell_id$ba5d6311-daee-4abc-b2fb-fae2184ef3ebdownstream_cells_map&setup_binary_gaussian_policy_arguments$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$20776e09-7d9b-4db8-a060-7bceeec65b47upstream_cells_mapBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41randIntegerContinuousMDP$537270ba-122b-4f2b-880b-31d086766295RealFunctionBinaryGaussianEligibilityVector$10cdd16e-a337-4421-a7a0-6de4e4b60c0fupdate_binary_feature_vector!$8eab55a5-41b7-4f5e-a02f-4c19388bc9eamake_n_param_dist_params$76eb6743-cac0-4174-9ba3-a0691c200b54UnionNTuple$8e742d32-c074-4981-b35b-b596b64c869bprecedence_heuristic cell_id$8e742d32-c074-4981-b35b-b596b64c869bdownstream_cells_map'cartpole_continuing_binary_study_params$b2539398-fdbc-42a2-a8f3-d327358f3643upstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunner(create_actor_critic_continuing_params_UI$5b15d91e-7119-4f85-a54a-7d4f1fdaf097Core.applicable@bindBase.get$03a218cb-aa83-4000-85b5-c6f247087053precedence_heuristic cell_id$03a218cb-aa83-4000-85b5-c6f247087053downstream_cells_mapupdate_binary_value_gradient!$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2$05bfd818-bf4e-4bda-baa9-5ba647867097$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098upstream_cells_mapRealBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41Vector$1ec1acf1-f833-4478-9b3c-88029340a629precedence_heuristic cell_id$1ec1acf1-f833-4478-9b3c-88029340a629downstream_cells_mapupstream_cells_map@md_strgetindex$de3cba34-9842-44d1-9b79-47126c0a0751precedence_heuristic cell_id$de3cba34-9842-44d1-9b79-47126c0a0751downstream_cells_mapcartpole_tilecoding_setup$1b102220-6d78-480d-a77f-0e57bad23dca$3c89209c-9202-4d5d-841c-ea34be369616upstream_cells_maptile_coding_setup/cartpole_functions$f27f2bcd-05b6-44fe-bf9e-a3e51556db7c$04f42c09-8ab5-4233-b196-51c4aa2dcedbprecedence_heuristic cell_id$04f42c09-8ab5-4233-b196-51c4aa2dcedbdownstream_cells_mapupstream_cells_map@md_str<$mountaincar_continuing_binary_params$fed4dc4c-0d1c-4ee3-9d0e-8ef2a7db7486>isless-mountaincar_binary_continuing_parameter_study$d57375a5-b9e0-4742-b5f7-6a7da891604a(start_mountaincar_continuing_param_study$d3c1379f-acd6-4e15-be7e-a5dbe46a4f62getindex$54ff46a2-489a-4dd2-bc30-df70c780cc42precedence_heuristic cell_id$54ff46a2-489a-4dd2-bc30-df70c780cc42downstream_cells_mapupstream_cells_map$7126aefd-b847-497a-9545-514e9b9afa71precedence_heuristic cell_id$7126aefd-b847-497a-9545-514e9b9afa71downstream_cells_mapupstream_cells_map$48dcd2d0-a940-41da-a097-90c780f2ec4dprecedence_heuristic cell_id$48dcd2d0-a940-41da-a097-90c780f2ec4ddownstream_cells_mapupstream_cells_map@md_strgetindex$e1493cea-19c4-475d-98a0-86d27fb04af1precedence_heuristic cell_id$e1493cea-19c4-475d-98a0-86d27fb04af1downstream_cells_mapupstream_cells_mapsarsa_λInt64corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512|>typemaxget_corridor_episode_stats$fb8904a9-ae64-41cc-93b6-5a25855edad0$cecc2a35-3850-4f66-84e8-e29da4f3d4b0$511a847f-234c-465e-8f4a-688e79d9b975precedence_heuristic cell_id$511a847f-234c-465e-8f4a-688e79d9b975downstream_cells_mapupstream_cells_map@md_strgetindex$697b2310-9d96-4f7f-be62-c3bd6bf736f3precedence_heuristic cell_id$697b2310-9d96-4f7f-be62-c3bd6bf736f3downstream_cells_map1reinforce_with_baseline_monte_carlo_control_fcann$aa69e4ea-91e0-496a-a7be-529e67f4dbecupstream_cells_map,reinforce_with_baseline_monte_carlo_control!$4fb83451-b6f8-4e6e-a131-1accc8e10b08$5b868eba-c1af-49f6-8f93-79b78c319a6fFCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANNParamsIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANN.initializeparams_saxeInt64VectorRealFunction&setup_fcann_policy_and_value_arguments$e1aec891-d95a-47d1-97d7-d2a4cfb16e64lengthfill$056a8adc-92f4-4b33-90d9-4b3b4026bbbcprecedence_heuristic cell_id$056a8adc-92f4-4b33-90d9-4b3b4026bbbcdownstream_cells_mapupdate_traces_with_gradient!$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90fupstream_cells_map *zero'BinarySquashedGaussianEligibilityVector$76fd79a2-2bc8-45f8-a243-48415118898aisnanislessdigamma@inboundsone'Base.CoreLogging.Base.fixup_stdlib_pathBase.CoreLogging.!nothing=Base.simd_inner_lengthBinaryGaussianEligibilityVector$10cdd16e-a337-4421-a7a0-6de4e4b60c0fθ$bc8a399b-8864-4473-89d2-e3b0a03d15b5precedence_heuristic cell_id$bc8a399b-8864-4473-89d2-e3b0a03d15b5downstream_cells_mapcorridor_parameter_study$c52c4cec-0ea8-4af3-831a-d284f0e086eeupstream_cells_mapcorridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512,actor_critic_binary_episodic_parameter_study$1f041cb3-618c-4380-a1ec-d7bbe4a80f62$d9d11d69-bc16-400a-8f46-f9a8ecb8516a$bba13634-ff0e-47f7-a23b-8d56098f4ac6precedence_heuristic cell_id$bba13634-ff0e-47f7-a23b-8d56098f4ac6downstream_cells_mapgaussian_action_samplermake_gaussian_n_samplermake_gaussian_sampler$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$d5020a8d-1dd7-403c-9d1f-665b95543943$20776e09-7d9b-4db8-a060-7bceeec65b47upstream_cells_mapisapproxzeroexpisnanrandIntegerVectorRealisinfVal+NTuplentupleNormal$407a0724-4bb6-4c83-ab2d-17a0e19c4072precedence_heuristic cell_id$407a0724-4bb6-4c83-ab2d-17a0e19c4072downstream_cells_mapreinforce_test4$27487ad0-4779-42ce-8def-e660ef04bee0$9d264543-33ab-498a-90f5-5f913c252484$07ba9fe4-aaa7-4123-9865-cbfa79d0d44a$af144759-fe66-4ad0-b378-e9eb4e859db4$e1274f57-75cb-4659-a82f-e5870c5367e2upstream_cells_mapInt64cartpole_setup$26880577-d267-4950-8725-7afe0d0402b6*actor_critic_with_eligibility_traces_fcann$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54cartpole_fcann_feature_setup$61650a97-b353-4a85-b50b-93fee296ac7b3cartpole_fcann_feature_setup.update_feature_vector!typemax$77cf3a74-899f-4ade-99f2-5aaf7a98c02dprecedence_heuristic cell_id$77cf3a74-899f-4ade-99f2-5aaf7a98c02ddownstream_cells_mapscale_fcann_params!$f3e2db06-9cb7-464a-96b8-938175efd26bupstream_cells_mapReal:eachindex/@inboundsFCANNParamsVector$28ce6e60-59cf-408a-8081-b978507b3c72precedence_heuristic cell_id$28ce6e60-59cf-408a-8081-b978507b3c72downstream_cells_map$cartpole_fcann_continuing_test_state$fd58402f-da65-44cf-b81a-e21192fd0e63upstream_cells_map @md_strCore:deg2radLinRangePlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70|>Base.get@bindSliderBasePlutoRunner-PlutoRunner.create_bondconfirmCore.applicablePlutoUI.combinegetindex$7ccadf01-fbba-4dfd-a5ad-770dab9946f9precedence_heuristic cell_id$7ccadf01-fbba-4dfd-a5ad-770dab9946f9downstream_cells_mapupstream_cells_map@md_strgetindex$b72e030f-7d52-481f-b4f7-2b16b227e547precedence_heuristic cell_id$b72e030f-7d52-481f-b4f7-2b16b227e547downstream_cells_mapupstream_cells_map@md_strgetindex$4c5cb75e-79b5-4502-b1eb-6246e002feafprecedence_heuristic cell_id$4c5cb75e-79b5-4502-b1eb-6246e002feafdownstream_cells_mapmountaincar_binary_params$8eb42403-1234-4e59-993e-057cc3a6d5c9upstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCore.applicable@bindBase.getcreate_actor_critic_params_UI$a8b40b8f-051a-4e6f-a079-ece4f32873de$48b342f2-e48f-457a-9bd3-b3504a79f3a6precedence_heuristic cell_id$48b342f2-e48f-457a-9bd3-b3504a79f3a6downstream_cells_mapupstream_cells_map@md_strgetindex$5d50a5d0-8fe2-4c6e-b76c-d5614e4fd884precedence_heuristic cell_id$5d50a5d0-8fe2-4c6e-b76c-d5614e4fd884downstream_cells_mapshow_or_lookup_plotupstream_cells_mapDictNamedTuple@md_strFunctionAbstractStringTuplehaskey==Integergetindex$ba645f6b-143f-4e83-9003-707770ae308dprecedence_heuristic cell_id$ba645f6b-143f-4e83-9003-707770ae308ddownstream_cells_mapshow_mountaincar_trajectory$da3cb392-78f2-48b2-b0dc-5f016664798c$3a37b53d-9174-4faa-9404-74a40c385b0a$ddbca73f-c692-46f2-95f3-a7dd849d33f7upstream_cells_mapsumHypertextLiteral.BypassHypertextLiteral.content@htl$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebIntegerFunctionscatterplotHypertextLiteral.ResultHypertextLiteral$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70Layoutrunepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84MountainCarTask$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811precedence_heuristic cell_id$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811downstream_cells_mapupdate_corridor_features!$5720e942-d3f8-4329-83a8-8bcedf078b6a$cacaaca6-6e01-464f-a2ee-cbf62737a426$07ad517a-c2ac-4377-99fb-adb13d0f1d0c$aa69e4ea-91e0-496a-a7be-529e67f4dbec$a12b92d1-e045-4f92-b8cd-eee5d56fa67d$9db9ff71-bee9-4bea-a45b-748f8517fed1$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0upstream_cells_mapRealoneVector$8f1b2db4-ed35-44fc-a3d5-e06deae16d48precedence_heuristic cell_id$8f1b2db4-ed35-44fc-a3d5-e06deae16d48downstream_cells_mapupstream_cells_map$57bbdb10-bed8-459d-8f67-9ea637cf12baprecedence_heuristic cell_id$57bbdb10-bed8-459d-8f67-9ea637cf12badownstream_cells_mapone_step_actor_critic_fcann$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0upstream_cells_mapFCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANNParamsIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANN.initializeparams_saxeInt64VectorRealone_step_actor_critic!$4d4ae57b-afc3-44f9-b6fc-892f59f82921Function&setup_fcann_policy_and_value_arguments$e1aec891-d95a-47d1-97d7-d2a4cfb16e64lengthfill$ca360680-afc9-4dd9-9351-493643f91575precedence_heuristic cell_id$ca360680-afc9-4dd9-9351-493643f91575downstream_cells_mapupstream_cells_map@md_strgetindex$d95f75b5-21d8-4862-baa7-50b58d9725b8precedence_heuristic cell_id$d95f75b5-21d8-4862-baa7-50b58d9725b8downstream_cells_mapupstream_cells_map@md_strgetindex$65be0e58-24be-4932-92a9-9e4825b14144precedence_heuristic cell_id$65be0e58-24be-4932-92a9-9e4825b14144downstream_cells_map@actor_critic_binary_continuing_squashed_gaussian_parameter_studyupstream_cells_mapRealFunctiononeIntegerContinuousMDP$537270ba-122b-4f2b-880b-31d086766295$60c21e9c-e42d-4f0b-a910-3b318440fbc8precedence_heuristic cell_id$60c21e9c-e42d-4f0b-a910-3b318440fbc8downstream_cells_mapgaussian_plot_params$09dd1440-5d09-421f-addc-b1ede43ff517upstream_cells_map@md_strCore:PlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70Base.get@bindSliderBasePlutoRunnerPlutoRunner.create_bondCore.applicablePlutoUI.combinegetindex$da2d3186-a778-41cc-9b49-759bf1e9b8faprecedence_heuristic cell_id$da2d3186-a778-41cc-9b49-759bf1e9b8fadownstream_cells_mapBinaryFeatures$65d2add6-fd6f-456c-92ed-3cd9d1862ef6$2cbc972b-c685-4c1c-8a8d-9d58b197ad90upstream_cells_mapC2C1C3TAbstractVectorNBaseNTupleUnionInteger$b695ef21-a1ac-4d1f-a0e1-71cd81cede18precedence_heuristic cell_id$b695ef21-a1ac-4d1f-a0e1-71cd81cede18downstream_cells_mapupstream_cells_map"plot_mountaincar_continuous_values$68469a40-7976-48b7-b7a1-eaa4c5f33a18"mountaincar_continuous_test_train2$fee14dfe-c5ca-4126-a830-cc9d7eda5433$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00precedence_heuristic cell_id$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00downstream_cells_mapLreinforce_with_baseline_monte_carlo_control_binary_features_gaussian_actions$24fa139c-ad4b-49db-ac8f-23c476ed8608$8aa16866-bfda-48df-9cf1-cf3d2e203ccbupstream_cells_map ,reinforce_with_baseline_monte_carlo_control!$4fb83451-b6f8-4e6e-a131-1accc8e10b08$5b868eba-c1af-49f6-8f93-79b78c319a6fbinary_value_function$a540814a-57a1-4b98-9443-59e401425444make_n_param_dist_policy_params$ba41f521-4ee2-42a6-bf18-078bfa4b875ezerosBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41randIntegermake_gaussian_sampler$bba13634-ff0e-47f7-a23b-8d56098f4ac6VectorContinuousMDP$537270ba-122b-4f2b-880b-31d086766295#update_gaussian_eligibility_vector!$5261651e-a51e-4e80-8e23-83a4c10e5259$740a3f41-9302-481d-b373-762c0dea8effRealFunction&setup_binary_gaussian_policy_arguments$ba5d6311-daee-4abc-b2fb-fae2184ef3ebupdate_binary_value_gradient!$03a218cb-aa83-4000-85b5-c6f247087053MatrixNTupleUnion!update_binary_action_preferences!$a361f4c9-47ce-42ad-899c-87b611c0d471$dcb306ae-a1b1-43d6-ba6e-e38668838689precedence_heuristic cell_id$dcb306ae-a1b1-43d6-ba6e-e38668838689downstream_cells_mapupstream_cells_map@md_strgetindex$54f559b6-8a62-4a42-894d-c56e41d5ebefprecedence_heuristic cell_id$54f559b6-8a62-4a42-894d-c56e41d5ebefdownstream_cells_mapcorridor_state_counts$62e677ac-2070-4f6b-9df2-90849d89fa9fupstream_cells_mapcollect_state_distributions$0c9986bb-54c0-4b08-9c29-4bfb0b68b54e$f545c800-0bf3-491f-9d7d-42341cfdb573precedence_heuristic cell_id$f545c800-0bf3-491f-9d7d-42341cfdb573downstream_cells_map%form_state_continuous_policy_function$5b868eba-c1af-49f6-8f93-79b78c319a6f$11b9beea-b0cd-45eb-84c6-151728894df0upstream_cells_mapFunction$8b35661b-5075-4d63-bc31-044407f99acfprecedence_heuristic cell_id$8b35661b-5075-4d63-bc31-044407f99acfdownstream_cells_mapupstream_cells_mapget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa825124actor_critic_with_eligibility_traces_binary_features$05bfd818-bf4e-4bda-baa9-5ba647867097corridor_continuing_mdp$1ac9296f-047b-4051-ba5c-0c23d5f9cde9$09dd1440-5d09-421f-addc-b1ede43ff517precedence_heuristic cell_id$09dd1440-5d09-421f-addc-b1ede43ff517downstream_cells_mapupstream_cells_mapNormalpdfscatterplotLinRangeLayoutgaussian_plot_params$60c21e9c-e42d-4f0b-a910-3b318440fbc8$a0ca7a5e-0089-4a45-9278-c0f27cd096a0precedence_heuristic cell_id$a0ca7a5e-0089-4a45-9278-c0f27cd096a0downstream_cells_mapupstream_cells_map"plot_mountaincar_continuous_values$68469a40-7976-48b7-b7a1-eaa4c5f33a18"mountaincar_continuous_test_train3$5eb8d9f9-8512-4e00-8cb5-cec68d73cc7d$64b38d1f-ecf9-4843-89a1-4c8953048265precedence_heuristic cell_id$64b38d1f-ecf9-4843-89a1-4c8953048265downstream_cells_map&cartpole_fcann_continuing_test_episode$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fce$6acb549a-5d90-4457-a347-d22448ad8071$fad02876-efba-46a7-9cb7-43820528779f$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7efupstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6runepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84cartpole_continuing_fcann_test$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27$d963ff6d-f1b6-4799-aa0e-1ae100310d84precedence_heuristiccell_id$d963ff6d-f1b6-4799-aa0e-1ae100310d84downstream_cells_map spzerosdroptol!requestCostFunctions#monte_carlo_off_policy_prediction_qdouble_expected_sarsaAbstractAveragingMethodpolicy_iteration_vTailRecsample_rolloutautoTuneParamsq_learninggetBackendfullTrainsparse_vcatmonte_carlo_control_ϵ_softreadBinParamsmake_greedy_policy!policy_iteration!smartTuneR$monte_carlo_control_exploring_startssparseinitialize_state_action_valuemultiTrainStateMRPTransitionDistribution!monte_carlo_off_policy_predictionrook_actionstd0_policy_prediction_vtd0_policy_prediction_qFCANN$cc3ac95e-a398-438a-ba3d-62b6733f6342$45f0a385-6465-4acc-8637-1b007a0fe215$0e9de19e-bcd4-40ac-9831-afb6cad38422$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091$5c4a383f-fcf2-4f2b-819f-6d84471dda00$635abb34-2c97-4f04-a74c-22fbec32f408$f3e2db06-9cb7-464a-96b8-938175efd26b$697b2310-9d96-4f7f-be62-c3bd6bf736f3$57bbdb10-bed8-459d-8f67-9ea637cf12ba$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54$8bc280db-e57d-4e40-be46-1790f4f7d9e7$11063fff-4d36-46d5-828f-dbed0f46b9cf@tailrecsimulate!TabularMRPTransitionSamplermultiTrainAutoRegget_cuda_toolkit_versionsStateMRPTransitionSamplerautoTuneRwriteArrayAbstractSparseMatrix@using_nvidialib_settingsbellman_state_valuedropoutRegsmartEvalLayersdistribution_rolloutcalcfeatureimpactSparseMatrixCSCADAMAXTrainNNCPUAbstractSparseVectorAbstractMDP$537270ba-122b-4f2b-880b-31d086766295policy_evaluation_qmonte_carlo_policy_predictionsample_action$4fb83451-b6f8-4e6e-a131-1accc8e10b08$e7e49ff8-32df-48a4-afb2-462859592e92$4d4ae57b-afc3-44f9-b6fc-892f59f82921$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1traintrialsrunepisode!$4fb83451-b6f8-4e6e-a131-1accc8e10b08$5b868eba-c1af-49f6-8f93-79b78c319a6fdouble_q_learningmaxNormRegmonte_carlo_policy_prediction_vNVIDIALibrariesAbstractAfterstateMDPsparse_hvcatAbstractMPpolicy_evaluation!ApproximationUtils$f946c886-6246-4f98-a96f-f06984691ad8sparse_hcatvalue_iteration_qinitializeParamsissparsemonte_carlo_tree_search_Previous_Controller_archEvalmonte_carlo_predictionAbstractSparseArraybenchmarkCPUThreadsnonzerosgeneralized_sarsa!policy_evaluation_vevalLayersftranspose!StateMRPnzrangeAbstractTabularTransitionTabularRL$3c316495-bb6c-41e2-a38f-ba867a319fbbConstantStepAveragingarchEvalSampletuneAlphamake_stochastic_gridworldwriteParamsbellman_afterstate_valueStateMDP $5cc4d12d-b537-47e2-8109-4e7a234fdf25$0ac7ea44-14f6-4e80-80f9-d6df8059bb38$96506201-6b66-49e6-8179-06952e2394e1$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290$8e39bd15-862e-4941-88f9-2794b861a523$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091$4fb83451-b6f8-4e6e-a131-1accc8e10b08$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$d1ed25e6-60c6-411f-a541-99986e5da2c5$697b2310-9d96-4f7f-be62-c3bd6bf736f3$4d4ae57b-afc3-44f9-b6fc-892f59f82921$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2$57e5e12a-b722-4ea3-ab3b-e5711029e640$57bbdb10-bed8-459d-8f67-9ea637cf12ba$266d2234-26c8-43f1-9e75-49440a230ed6$05bfd818-bf4e-4bda-baa9-5ba647867097$68806899-9972-460a-9f11-daa708a9d610$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54$1f041cb3-618c-4380-a1ec-d7bbe4a80f62$11ea640c-3981-404d-87c6-4d3d0708a2b8$f8614042-7c94-4d47-a1b6-4e96676b4e8b$83640f5b-fe13-4ec1-98a0-67a56c189ba1$f0104778-81a6-417b-8501-f916e5e7f3af$734573e5-547b-4dcc-89bb-412aa6cc42d6$e96d592d-1e54-486d-8ad9-b857f85476e8$ff4f977e-48df-4c12-845c-c245b4d39d6d$8bc280db-e57d-4e40-be46-1790f4f7d9e7$5aba4f96-e877-457e-8e95-18737348f99f$11063fff-4d36-46d5-828f-dbed0f46b9cf$4c4e643b-d4b9-44f0-8d30-dc521bcc55ac$00152954-dc98-4120-b94b-2ea4d987832b$d9d11d69-bc16-400a-8f46-f9a8ecb8516amake_random_policyspdiagmdropzerospreptrainingbackendListsprandnLossTypemrp_evaluationAbstractStateTransitionexpected_sarsaevalMultiSampleAveragingmonte_carlo_off_policy_controltuneRsetBackendTabularMRPuctmakelookupapply_uct!L2RegGridworldActiontd0_predictionsparsevecTabularDeterministicTransitionvalue_iteration!blockdiagTabularAfterstateMDPtd0_policy_predictionGridworldStatemake_ϵ_greedy_policy!$5981f52b-d829-4c7d-b47b-33310f7d64a2value_iteration_vTabularMDPAbstractTransition$c8b47eac-2d45-419a-bec6-2ae0cdc59393mrp_evaluation!AbstractMRPsprandfind_terminal_statessarsaTabularMDPTransitionSamplerswitch_devicedropzeros!TabularTransitionDistributionOutputIndex$5c4a383f-fcf2-4f2b-819f-6d84471dda00monte_carlo_controlStateMDPTransitionDistributionpolicy_evaluationreadBinInputfkeep!bellman_policy_update!SparseVectorvalue_iterationtestTrainrunepisode $f7433324-acc3-49a5-b5b3-ada0c8f09d52$fb8904a9-ae64-41cc-93b6-5a25855edad0$cecc2a35-3850-4f66-84e8-e29da4f3d4b0$0c9986bb-54c0-4b08-9c29-4bfb0b68b54e$4fb83451-b6f8-4e6e-a131-1accc8e10b08$0cd96c44-cae6-421f-9fae-26141600bef4$64b38d1f-ecf9-4843-89a1-4c8953048265$0ce66c9d-6d1c-4c2d-8178-5bcdfa247cd6$5b868eba-c1af-49f6-8f93-79b78c319a6f$cf1859d6-f889-4923-8c87-2d7c039f26c3$31db0f58-28e4-454f-9394-25565687266f$dddc4a2f-34b2-41dc-85b3-55aba4880fa6$5859ca11-90f8-4fd6-88ed-c56efe796fe8$11a55af7-5301-4507-bb26-88e1e11236db$07ba9fe4-aaa7-4123-9865-cbfa79d0d44a$e1274f57-75cb-4659-a82f-e5870c5367e2$daf35bfe-8f9c-4f55-971d-4d443be8f8bf$a5b002c9-5e11-462a-9da0-6e060c7963f8$ba645f6b-143f-4e83-9003-707770ae308d$b5319d8b-0420-4ebf-b603-ea0b93365ac1make_deterministic_gridworldCrossEntropyLoss$45f0a385-6465-4acc-8637-1b007a0fe215findnzSparseArrayscheckNumGradTabularStochasticTransitioninitialize_afterstate_valuennzbellman_state_action_valueStateMDPTransitionSampler$5cc4d12d-b537-47e2-8109-4e7a234fdf25$f0104778-81a6-417b-8501-f916e5e7f3af$4c4e643b-d4b9-44f0-8d30-dc521bcc55ac$00152954-dc98-4120-b94b-2ea4d987832b$3c316495-bb6c-41e2-a38f-ba867a319fbbbenchmarkDevicefind_available_actionsinitialize_state_valuemonte_carlo_policy_prediction_qpolicy_iterationpermuterowvalsupstream_cells_mapBase.CoreLogging.invokelatestBase.CoreLogging.===@raw_str#___this_pluto_module_namerethrowBase.CoreLogging.!'Base.CoreLogging.Base.fixup_stdlib_pathBaseBase.CoreLogging.isaBase.invokelatestBase.CoreLogging.>=@__DIR__PlutoDevMacros.@frompackage$b16899b7-36bf-4a5e-8e2f-4496b8450687precedence_heuristic cell_id$b16899b7-36bf-4a5e-8e2f-4496b8450687downstream_cells_mapsquashed_gaussian_pdf$00bd2835-b006-4244-9877-bc7e031e3ef8upstream_cells_mapexpsqrtAbstractArrayReal-atanh/^πUnion*absinv$10cdd16e-a337-4421-a7a0-6de4e4b60c0fprecedence_heuristic cell_id$10cdd16e-a337-4421-a7a0-6de4e4b60c0fdownstream_cells_mapBinaryGaussianEligibilityVector$f55afa58-962d-4551-8d95-a5b467d61adf$740a3f41-9302-481d-b373-762c0dea8eff$ba5d6311-daee-4abc-b2fb-fae2184ef3eb$056a8adc-92f4-4b33-90d9-4b3b4026bbbcupstream_cells_mapRealzeroNzerosBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41UnionNTupleonesoneVector$a8b40b8f-051a-4e6f-a079-ece4f32873deprecedence_heuristic cell_id$a8b40b8f-051a-4e6f-a079-ece4f32873dedownstream_cells_mapcreate_actor_critic_params_UI$36d514fa-b27a-4c6b-8399-9d108377b9b5$4c5cb75e-79b5-4502-b1eb-6246e002feaf$71a5fce8-6d9a-4625-bad1-a951d61bff28$0d45ae72-572f-4d17-83cf-9814f2854131upstream_cells_map@md_strBase.getindex:PlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70|>HypertextLiteral.contentSlider@htl$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebBaseNumberFieldHypertextLiteral.ResultHypertextLiteral$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70PlutoUI.combineconfirm$5eebf3da-bfe7-46eb-81a3-f87f334ee270precedence_heuristic cell_id$5eebf3da-bfe7-46eb-81a3-f87f334ee270downstream_cells_map#create_actor_critic_fcann_params_UI$9978d537-49ff-4014-a971-b42704c50a6bupstream_cells_map@md_str:PlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70NumberField|>PlutoUI.combineconfirmSlidergetindex$9bce6fdb-2cbc-4758-9a8b-794e490c973dprecedence_heuristic cell_id$9bce6fdb-2cbc-4758-9a8b-794e490c973ddownstream_cells_mapep2_step$1ce4bc6c-7cde-48e9-8ff1-7281697fd121upstream_cells_mapCore:Base.get@bindep2$a5b002c9-5e11-462a-9da0-6e060c7963f8SliderlengthBasePlutoRunnerPlutoRunner.create_bondCore.applicable$b86ee9d3-b6b5-4ea0-8f55-1927571cdfbfprecedence_heuristic cell_id$b86ee9d3-b6b5-4ea0-8f55-1927571cdfbfdownstream_cells_map$create_continuous_action_mountaincar$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2$349631b2-4686-49a9-9f3a-1e4ad588b568upstream_cells_mapabsisless*ContinuousMDP$537270ba-122b-4f2b-880b-31d086766295MountainCarTask$0ce66c9d-6d1c-4c2d-8178-5bcdfa247cd6precedence_heuristic cell_id$0ce66c9d-6d1c-4c2d-8178-5bcdfa247cd6downstream_cells_map#mountaincar_continuing_test_episodeupstream_cells_map mountaincar_continuing_tile_test$b02ba928-5b9f-4695-b980-07988c788bb9runepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84MountainCarTask$7afb6fb0-248a-4518-b94f-9876f81eca64precedence_heuristic cell_id$7afb6fb0-248a-4518-b94f-9876f81eca64downstream_cells_map#corridor_continuing_parameter_study$42775fd1-5b27-48e0-abf1-9b22bb775e6dupstream_cells_mapget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512corridor_continuing_mdp$1ac9296f-047b-4051-ba5c-0c23d5f9cde9#actor_critic_linear_parameter_study$734573e5-547b-4dcc-89bb-412aa6cc42d6$e96d592d-1e54-486d-8ad9-b857f85476e8$ff4f977e-48df-4c12-845c-c245b4d39d6d$37a273b6-b104-46f0-987a-401dc1c97327precedence_heuristic cell_id$37a273b6-b104-46f0-987a-401dc1c97327downstream_cells_map,start_cartpole_continuing_binary_param_study$b2539398-fdbc-42a2-a8f3-d327358f3643upstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCounterButtonCore.applicable@bindBase.get$7a6f3f79-ea06-4994-8b62-90b2056e4034precedence_heuristic cell_id$7a6f3f79-ea06-4994-8b62-90b2056e4034downstream_cells_map squashed_gaussian_action_sampler make_squashed_gaussian_n_samplermake_squashed_gaussian_sampler$05f120be-9695-4824-82fd-142a0df13098upstream_cells_map isapproxzeroexpisnanrandIntegerVectorRealisinfsignVal+NTuplentupletanh*Normal$f2ed56c9-c2b7-42cb-a083-e12aeaa126efprecedence_heuristic cell_id$f2ed56c9-c2b7-42cb-a083-e12aeaa126efdownstream_cells_mapupstream_cells_mapcorridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512^-reinforce_monte_carlo_control_binary_features$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290$cbea5840-49d2-4e91-be9c-f5f15666d78aprecedence_heuristic cell_id$cbea5840-49d2-4e91-be9c-f5f15666d78adownstream_cells_mapupstream_cells_mapcorridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512;reinforce_with_baseline_monte_carlo_control_binary_features$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb^$1f041cb3-618c-4380-a1ec-d7bbe4a80f62precedence_heuristic cell_id$1f041cb3-618c-4380-a1ec-d7bbe4a80f62downstream_cells_map,actor_critic_binary_episodic_parameter_study$bc8a399b-8864-4473-89d2-e3b0a03d15b5$8eb42403-1234-4e59-993e-057cc3a6d5c9upstream_cells_map Random$df7f84e8-b42a-4001-9dbf-6bc3ced94207StateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84lengthcopyVectorRealscatter/Matrix4actor_critic_with_eligibility_traces_binary_features$05bfd818-bf4e-4bda-baa9-5ba647867097isemptymean:AbstractVector|>InfzerosrandIntegerFunctionUInt64-plotfoldxt+MapLayoutRandom.seed!$96506201-6b66-49e6-8179-06952e2394e1precedence_heuristic cell_id$96506201-6b66-49e6-8179-06952e2394e1downstream_cells_mapsetup_binary_policy_arguments$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2$05bfd818-bf4e-4bda-baa9-5ba647867097upstream_cells_mapcopyRealFunctionlengthupdate_binary_feature_vector!$8eab55a5-41b7-4f5e-a02f-4c19388bc9eazerosBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41BinaryEligibilityVector$41dc149d-c6f3-4b0d-a856-06f3aaae3049IntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84$76b03e72-da04-4530-8534-6d6468268cbdprecedence_heuristic cell_id$76b03e72-da04-4530-8534-6d6468268cbddownstream_cells_mapupstream_cells_map@md_strgetindex$fd89433e-643c-474b-b3c4-a997678421a6precedence_heuristic cell_id$fd89433e-643c-474b-b3c4-a997678421a6downstream_cells_mapupstream_cells_map@md_strgetindex$87feff3e-e510-4916-91a9-db3a2cd12225precedence_heuristic cell_id$87feff3e-e510-4916-91a9-db3a2cd12225downstream_cells_map&fcann_continuing_cartpole_study_paramsupstream_cells_map @md_strCore:PlutoUI$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70|>Base.get@bindSliderBasePlutoRunnerPlutoRunner.create_bondNumberFieldconfirmCore.applicablePlutoUI.combinegetindex$5261651e-a51e-4e80-8e23-83a4c10e5259precedence_heuristic cell_id$5261651e-a51e-4e80-8e23-83a4c10e5259downstream_cells_map#update_gaussian_eligibility_vector!$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$d5020a8d-1dd7-403c-9d1f-665b95543943$20776e09-7d9b-4db8-a060-7bceeec65b47upstream_cells_map zeroisless@inboundsonenothingVectorreinforce_test$24fa139c-ad4b-49db-ac8f-23c476ed8608runepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84$54fff14b-cf53-47b0-9cfa-8b9ee33df54eprecedence_heuristic cell_id$54fff14b-cf53-47b0-9cfa-8b9ee33df54edownstream_cells_mapBinaryBetaEligibilityVector$f55afa58-962d-4551-8d95-a5b467d61adf$d41f1dd1-45fe-4456-9a01-ed47fd6704a7$ed93259c-7b8b-46d7-97fb-f194e0e04b3a$056a8adc-92f4-4b33-90d9-4b3b4026bbbcupstream_cells_mapRealNBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41UnionNTupleonesoneVector$023f67b8-8f38-470a-9766-ac60a75678aaprecedence_heuristic cell_id$023f67b8-8f38-470a-9766-ac60a75678aadownstream_cells_mapmountaincar_fcann_setup$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9upstream_cells_mapfcann_feature_vector_setup$9acdbf38-2e10-45ec-85a0-d0db8453a599mountaincar_min_vals$2025ff38-f2ec-4224-b771-ff72ffe1af28mountaincar_max_vals$77906355-08f8-4b08-b051-84697199b519$1558cec1-c4fd-4bc0-85ed-ae22c6067d41precedence_heuristic cell_id$1558cec1-c4fd-4bc0-85ed-ae22c6067d41downstream_cells_mapupstream_cells_map@md_strgetindex$da8d0bca-105b-4d0b-a73d-ee5c9059aeafprecedence_heuristic cell_id$da8d0bca-105b-4d0b-a73d-ee5c9059aeafdownstream_cells_mapupstream_cells_map@md_strgetindex$3e7cecec-eb77-4862-8e3c-b510422e06dbprecedence_heuristic cell_id$3e7cecec-eb77-4862-8e3c-b510422e06dbdownstream_cells_mapupstream_cells_mapsquashed_gaussian_plot_params$94517664-6988-44dc-a297-e9d5873ee540plot_squashed_gaussian$00bd2835-b006-4244-9877-bc7e031e3ef8$0284f0d7-b8a9-4ae6-add0-ac1078571d9bprecedence_heuristic cell_id$0284f0d7-b8a9-4ae6-add0-ac1078571d9bdownstream_cells_mapupstream_cells_map@md_strgetindex$b94fc99c-f439-4df2-8da3-c01718a136c4precedence_heuristic cell_id$b94fc99c-f439-4df2-8da3-c01718a136c4downstream_cells_mapupstream_cells_map@md_strgetindex$b8532822-179b-4cd5-a279-4b71dafb544aprecedence_heuristic cell_id$b8532822-179b-4cd5-a279-4b71dafb544adownstream_cells_map!mountaincar_continuous_test_train$d7f6ff79-3c0f-4f16-aa1c-3bc534ce580a$c87dba8c-9a96-41b3-9dc7-a6c088ec1eafupstream_cells_mapmountaincar_continuous_mdp$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2Int64mountaincar_tilecoding_setup$7c592385-e8d3-4efe-962c-d39debb64405Eactor_critic_with_eligibility_traces_binary_features_gaussian_actions$20776e09-7d9b-4db8-a060-7bceeec65b47typemax$07ba9fe4-aaa7-4123-9865-cbfa79d0d44aprecedence_heuristic cell_id$07ba9fe4-aaa7-4123-9865-cbfa79d0d44adownstream_cells_mapupstream_cells_mapreinforce_test4$407a0724-4bb6-4c83-ab2d-17a0e19c4072cartpole_setup$26880577-d267-4950-8725-7afe0d0402b6display_cartpole_episode$822e4d69-2582-4956-858e-06ecb091e76a|>runepisode$d963ff6d-f1b6-4799-aa0e-1ae100310d84$f487f2dd-ad09-48ac-ae34-bf50cfa6ac7dprecedence_heuristic cell_id$f487f2dd-ad09-48ac-ae34-bf50cfa6ac7ddownstream_cells_map.start_mountaincar_continuing_fcann_param_study$cb70d400-3e9c-441c-b17c-e727e8c928f3upstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCounterButtonCore.applicable@bindBase.get$5c4a383f-fcf2-4f2b-819f-6d84471dda00precedence_heuristic cell_id$5c4a383f-fcf2-4f2b-819f-6d84471dda00downstream_cells_mapupdate_fcann_value_gradient!$f3e2db06-9cb7-464a-96b8-938175efd26bupstream_cells_map:AbstractVectorFCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84BoolFCANNParams@inboundsOutputIndex$d963ff6d-f1b6-4799-aa0e-1ae100310d84IntegerVectorInt64FCANNActivations$5c11a92d-7496-4aba-af15-2537eac49dd7eachindexFloat32*FCANN.nnCostFunction$135f205a-f87e-4691-8e87-d317d6312c84precedence_heuristic cell_id$135f205a-f87e-4691-8e87-d317d6312c84downstream_cells_mapupstream_cells_map@md_strgetindex$4a39f9a7-72d4-44ad-895a-742cd1291f92precedence_heuristic cell_id$4a39f9a7-72d4-44ad-895a-742cd1291f92downstream_cells_mapdist_plot_p$9cf3dc5f-8a25-479f-93db-06e34f0d37a0upstream_cells_mapCore:|>Base.get@bindSliderBasePlutoRunnerPlutoRunner.create_bondconfirmCore.applicable$ee72af8d-3cb8-4314-82df-580f068e1252precedence_heuristic cell_id$ee72af8d-3cb8-4314-82df-580f068e1252downstream_cells_mapupstream_cells_map@md_strgetindex$e524f8cc-ab69-4f8b-a59f-28156696a104precedence_heuristic cell_id$e524f8cc-ab69-4f8b-a59f-28156696a104downstream_cells_map8run_mountaincar_binary_episodic_countinuous_param_study2$0d93132d-5819-47dc-8cf2-462d480d9c3dupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCounterButtonCore.applicable@bindBase.get$1894ae1a-bb68-4de0-a4d2-ac5d02c49f09precedence_heuristic cell_id$1894ae1a-bb68-4de0-a4d2-ac5d02c49f09downstream_cells_mapupstream_cells_map$f3bc47b5-03fc-4bd9-a890-26f9608a730bprecedence_heuristic cell_id$f3bc47b5-03fc-4bd9-a890-26f9608a730bdownstream_cells_mapupstream_cells_map@md_strgetindex$4915b1ed-ad53-4ece-9b00-bc136d47d8dcprecedence_heuristic cell_id$4915b1ed-ad53-4ece-9b00-bc136d47d8dcdownstream_cells_mapupstream_cells_map@md_strgetindex$f924eb30-d1cc-4941-8fb5-ff70ad425ab9precedence_heuristic cell_id$f924eb30-d1cc-4941-8fb5-ff70ad425ab9downstream_cells_mapupstream_cells_map@md_strgetindex$d83dc659-dce7-41dd-a8e7-2933ab39d15cprecedence_heuristic cell_id$d83dc659-dce7-41dd-a8e7-2933ab39d15cdownstream_cells_mapupstream_cells_map@md_strgetindex$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fceprecedence_heuristic cell_id$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fcedownstream_cells_mapupstream_cells_mapdisplay_cartpole_episode$822e4d69-2582-4956-858e-06ecb091e76a&cartpole_fcann_continuing_test_episode$64b38d1f-ecf9-4843-89a1-4c8953048265$83ca0577-15d7-4448-b597-c77810b812bfprecedence_heuristic cell_id$83ca0577-15d7-4448-b597-c77810b812bfdownstream_cells_mapfigure_13_2_test$a7dcc8cd-04ec-48f2-a387-116330eaffb2upstream_cells_map sqrtRandom$df7f84e8-b42a-4001-9dbf-6bc3ced94207corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabescatter/fillround:|>Int64get_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512-log2foldxtplot+*-reinforce_monte_carlo_control_binary_features$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290Map;reinforce_with_baseline_monte_carlo_control_binary_features$a7c9ae69-f4b8-471c-ab97-90642f3c2bdbLayoutRandom.seed!$a7c9ae69-f4b8-471c-ab97-90642f3c2bdbprecedence_heuristic cell_id$a7c9ae69-f4b8-471c-ab97-90642f3c2bdbdownstream_cells_map;reinforce_with_baseline_monte_carlo_control_binary_features$cbea5840-49d2-4e91-be9c-f5f15666d78a$83ca0577-15d7-4448-b597-c77810b812bf$e5c1aca8-7575-4835-8273-e69ca0a55fe8$d3b56fca-5b79-4465-8987-8d0005f854d8$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03upstream_cells_map,reinforce_with_baseline_monte_carlo_control!$4fb83451-b6f8-4e6e-a131-1accc8e10b08$5b868eba-c1af-49f6-8f93-79b78c319a6fbinary_value_function$a540814a-57a1-4b98-9443-59e401425444setup_binary_policy_arguments$96506201-6b66-49e6-8179-06952e2394e1zerosBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41IntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84VectorRealFunction!update_binary_eligibility_vector!$042fbafe-2401-4fb7-ac13-4531e0782c79lengthupdate_binary_value_gradient!$03a218cb-aa83-4000-85b5-c6f247087053Matrix!update_binary_action_preferences!$a361f4c9-47ce-42ad-899c-87b611c0d471$a7dcc8cd-04ec-48f2-a387-116330eaffb2precedence_heuristic cell_id$a7dcc8cd-04ec-48f2-a387-116330eaffb2downstream_cells_mapupstream_cells_map:figure_13_2_test$83ca0577-15d7-4448-b597-c77810b812bfvcat^$0ab70fc3-6188-42eb-aba2-d808f319be9fprecedence_heuristic cell_id$0ab70fc3-6188-42eb-aba2-d808f319be9fdownstream_cells_mapupstream_cells_map@md_strgetindex$047656d1-2921-40f2-b75b-ce4a87098007precedence_heuristic cell_id$047656d1-2921-40f2-b75b-ce4a87098007downstream_cells_mapupstream_cells_map@md_strgetindex$5d434c83-c9ca-499f-8695-c7733031c2deprecedence_heuristic cell_id$5d434c83-c9ca-499f-8695-c7733031c2dedownstream_cells_mapcartpole_continuing_step$4c4e643b-d4b9-44f0-8d30-dc521bcc55acupstream_cells_mapCartPoleStatecartpole_functions.step#cartpole_functions.initialize_statecartpole_functions.failure+cartpole_functions$f27f2bcd-05b6-44fe-bf9e-a3e51556db7cInteger$3a37b53d-9174-4faa-9404-74a40c385b0aprecedence_heuristic cell_id$3a37b53d-9174-4faa-9404-74a40c385b0adownstream_cells_mapupstream_cells_mapshow_mountaincar_trajectory$ba645f6b-143f-4e83-9003-707770ae308d!mountaincar_continuing_fcann_test$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9$820752af-8966-4ee8-82f7-a40934522de5precedence_heuristic cell_id$820752af-8966-4ee8-82f7-a40934522de5downstream_cells_mapupstream_cells_map$6acb549a-5d90-4457-a347-d22448ad8071precedence_heuristic cell_id$6acb549a-5d90-4457-a347-d22448ad8071downstream_cells_map-cartpole_fcann_continuing_episode_step_select$fad02876-efba-46a7-9cb7-43820528779f$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7efupstream_cells_mapCore:Base.get@bindSliderlengthBasePlutoRunnerPlutoRunner.create_bond&cartpole_fcann_continuing_test_episode$64b38d1f-ecf9-4843-89a1-4c8953048265Core.applicable$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62precedence_heuristic cell_id$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62downstream_cells_map)cartpole_fcann_continuing_parameter_study$50ae94c4-70f3-4215-82bd-eb2227c2badfupstream_cells_mapcartpole_vector_update!$192b9f82-8d3a-408f-91c2-829cfcd32572cartpole_continuing_mdp$4c4e643b-d4b9-44f0-8d30-dc521bcc55accartpole_fcann_feature_setup$61650a97-b353-4a85-b50b-93fee296ac7bfill"actor_critic_fcann_parameter_study$8bc280db-e57d-4e40-be46-1790f4f7d9e7$5aba4f96-e877-457e-8e95-18737348f99f$11063fff-4d36-46d5-828f-dbed0f46b9cfInteger$3bccf6fc-6e5e-4f62-ad40-1ff0a3740728precedence_heuristic cell_id$3bccf6fc-6e5e-4f62-ad40-1ff0a3740728downstream_cells_mapupstream_cells_mapInt64corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512^4actor_critic_with_eligibility_traces_binary_features$05bfd818-bf4e-4bda-baa9-5ba647867097typemax$ae0f5a96-7a4b-47f9-be1e-e803a238a071precedence_heuristic cell_id$ae0f5a96-7a4b-47f9-be1e-e803a238a071downstream_cells_mapupstream_cells_map@md_strgetindex$41d62de1-2c92-41ee-9430-b9ca3007afd9precedence_heuristic cell_id$41d62de1-2c92-41ee-9430-b9ca3007afd9downstream_cells_mapupstream_cells_map@md_strgetindex$8eb42403-1234-4e59-993e-057cc3a6d5c9precedence_heuristic cell_id$8eb42403-1234-4e59-993e-057cc3a6d5c9downstream_cells_mapupstream_cells_map@md_str<,actor_critic_binary_episodic_parameter_study$1f041cb3-618c-4380-a1ec-d7bbe4a80f62$d9d11d69-bc16-400a-8f46-f9a8ecb8516a>mountaincar_binary_params$4c5cb75e-79b5-4502-b1eb-6246e002feafislessmountaincar_tilecoding_setup$7c592385-e8d3-4efe-962c-d39debb64405+run_mountaincar_binary_episodic_param_study$192cc1cf-9ea1-492d-baa7-f2e197abecd4MountainCarTaskgetindex$bbc8864a-1545-433f-bc7c-0ddf6e907138precedence_heuristic cell_id$bbc8864a-1545-433f-bc7c-0ddf6e907138downstream_cells_mapplot_mountaincar_policy_values$dc2efc6c-8da8-425b-aa5f-290949109565upstream_cells_map:HypertextLiteral.BypassLinRangezerosHypertextLiteral.content@htl$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebFunctionenumerateFloat32plotHypertextLiteral$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70heatmapHypertextLiteral.ResultLayout$a12b92d1-e045-4f92-b8cd-eee5d56fa67dprecedence_heuristic cell_id$a12b92d1-e045-4f92-b8cd-eee5d56fa67ddownstream_cells_mapbest_mc_corridor$44b32cc0-36a8-41fd-89bc-ce894536926c$553b0ceb-f2ca-41ee-99bc-9f53a4487b49upstream_cells_mapcorridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabe;reinforce_with_baseline_monte_carlo_control_linear_features$d1ed25e6-60c6-411f-a541-99986e5da2c5^update_corridor_features!$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811$ce33f710-fd9d-4dfa-acda-40204e54d518precedence_heuristic cell_id$ce33f710-fd9d-4dfa-acda-40204e54d518downstream_cells_mapupstream_cells_map@md_strgetindex$339b4d2b-2237-46a3-9867-ecc3332856c1precedence_heuristic cell_id$339b4d2b-2237-46a3-9867-ecc3332856c1downstream_cells_mapupstream_cells_map@md_strgetindex$a8349352-3242-46d5-b0d5-1b6eb5d77e90precedence_heuristic cell_id$a8349352-3242-46d5-b0d5-1b6eb5d77e90downstream_cells_mapx$d4e87ac4-6008-43b2-aa06-e232ec2b2b5bupstream_cells_mapCoreBase:PlutoRunner.create_bondPlutoRunnerCore.applicable@bindBase.getSlider$7d63b960-3998-4f7b-8cbb-ccd49db9aeacprecedence_heuristic cell_id$7d63b960-3998-4f7b-8cbb-ccd49db9aeacdownstream_cells_mapupstream_cells_mapInt64%one_step_actor_critic_binary_features$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512^typemax$65d2add6-fd6f-456c-92ed-3cd9d1862ef6precedence_heuristic cell_id$65d2add6-fd6f-456c-92ed-3cd9d1862ef6downstream_cells_mapupdate_binary_policy_params!upstream_cells_mapRealeachindex-BinaryFeatures$da2d3186-a778-41cc-9b49-759bf1e9b8faMatrix+@inbounds*IntegerVector$f55afa58-962d-4551-8d95-a5b467d61adfprecedence_heuristic cell_id$f55afa58-962d-4551-8d95-a5b467d61adfdownstream_cells_mapupdate_params_with_gradient!$4fb83451-b6f8-4e6e-a131-1accc8e10b08$4d4ae57b-afc3-44f9-b6fc-892f59f82921$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$5b868eba-c1af-49f6-8f93-79b78c319a6f$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90fupstream_cells_map zero'BinarySquashedGaussianEligibilityVector$76fd79a2-2bc8-45f8-a243-48415118898aislessdigamma@inboundsonenothingVectorisless+BinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41Function$0ac7ea44-14f6-4e80-80f9-d6df8059bb38precedence_heuristic cell_id$0ac7ea44-14f6-4e80-80f9-d6df8059bb38downstream_cells_mapreinforce_monte_carlo_control!$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290$8e39bd15-862e-4941-88f9-2794b861a523$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091upstream_cells_mapRealnothingFunction,reinforce_with_baseline_monte_carlo_control!$4fb83451-b6f8-4e6e-a131-1accc8e10b08$5b868eba-c1af-49f6-8f93-79b78c319a6fReturnszero/oneIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84$5ffc271f-c73f-494a-9727-8d7516af2191precedence_heuristic cell_id$5ffc271f-c73f-494a-9727-8d7516af2191downstream_cells_map&cartpole_continuing_fcann_study_params$50ae94c4-70f3-4215-82bd-eb2227c2badfupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunner(create_actor_critic_continuing_params_UI$5b15d91e-7119-4f85-a54a-7d4f1fdaf097Core.applicable@bindBase.get$c5a2879c-e89b-47f7-bbd6-48200d7e89e3precedence_heuristic cell_id$c5a2879c-e89b-47f7-bbd6-48200d7e89e3downstream_cells_map>actor_critic_binary_episodic_squashed_gaussian_parameter_study$0d93132d-5819-47dc-8cf2-462d480d9c3dupstream_cells_mapRealFunctiononeIntegerContinuousMDP$537270ba-122b-4f2b-880b-31d086766295$537270ba-122b-4f2b-880b-31d086766295precedence_heuristic cell_id$537270ba-122b-4f2b-880b-31d086766295downstream_cells_mapContinuousMDP $f946c886-6246-4f98-a96f-f06984691ad8$5b868eba-c1af-49f6-8f93-79b78c319a6f$ba5d6311-daee-4abc-b2fb-fae2184ef3eb$ed93259c-7b8b-46d7-97fb-f194e0e04b3a$4e29c621-223e-4859-8e96-db04b967815a$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$d5020a8d-1dd7-403c-9d1f-665b95543943$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90f$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098$717e4c69-59d5-4929-923f-dd35a97fb160$55ba8725-0ddf-4196-a41d-3f3c490a8d84$61949faa-8174-4b7b-8fbc-01d5f850b419$dd8e8cd2-7b41-46c4-8530-adefb7aea684$08505e88-9c23-4e95-91e3-d18bf5133dbc$87482ea5-5265-4e02-92c0-1a8bb44ff0f4$13ebc12f-ff6f-4266-88d3-28d6df5fcf59$3d065608-eef2-4caa-b17d-ec60714e3d58$bd6a7c16-6c25-4fc2-8e1b-4dab693ce19f$c5a2879c-e89b-47f7-bbd6-48200d7e89e3$65be0e58-24be-4932-92a9-9e4825b14144$3c316495-bb6c-41e2-a38f-ba867a319fbb$b86ee9d3-b6b5-4ea0-8f55-1927571cdfbf$d2729657-d0bf-4d39-8ec7-f242a1ad48d6upstream_cells_mapFunctionnewContinuousMDPTransitionSampler$c8b47eac-2d45-419a-bec6-2ae0cdc59393ReturnsAbstractContinuousTransition$c8b47eac-2d45-419a-bec6-2ae0cdc59393AbstractMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84FReal$dc2efc6c-8da8-425b-aa5f-290949109565precedence_heuristic cell_id$dc2efc6c-8da8-425b-aa5f-290949109565downstream_cells_mapupstream_cells_mapplot_mountaincar_policy_values$bbc8864a-1545-433f-bc7c-0ddf6e907138mountaincar_test_train$6d0925d3-af96-4b94-8e2e-4941cce39e51$a019925a-460a-410e-a54b-50a4cfe0e90eprecedence_heuristic cell_id$a019925a-460a-410e-a54b-50a4cfe0e90edownstream_cells_mapupstream_cells_map-scatterplotLinRangeLayoutget_corridor_episode_stats$fb8904a9-ae64-41cc-93b6-5a25855edad0$cecc2a35-3850-4f66-84e8-e29da4f3d4b0$f92bb265-4b19-4f0e-a698-d7547bb6dd41precedence_heuristic cell_id$f92bb265-4b19-4f0e-a698-d7547bb6dd41downstream_cells_mapBinaryFeatureVector $b0a66a19-ee76-463b-a704-8fcee85444d0$8eab55a5-41b7-4f5e-a02f-4c19388bc9ea$a361f4c9-47ce-42ad-899c-87b611c0d471$41dc149d-c6f3-4b0d-a856-06f3aaae3049$042fbafe-2401-4fb7-ac13-4531e0782c79$96506201-6b66-49e6-8179-06952e2394e1$03a218cb-aa83-4000-85b5-c6f247087053$a893a87b-2d07-4db5-9d1a-9da8646216f4$a540814a-57a1-4b98-9443-59e401425444$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2$25be5dcf-be63-46c4-b6de-6cf79fa28fd0$05bfd818-bf4e-4bda-baa9-5ba647867097$10cdd16e-a337-4421-a7a0-6de4e4b60c0f$54fff14b-cf53-47b0-9cfa-8b9ee33df54e$76fd79a2-2bc8-45f8-a243-48415118898a$f55afa58-962d-4551-8d95-a5b467d61adf$740a3f41-9302-481d-b373-762c0dea8eff$d41f1dd1-45fe-4456-9a01-ed47fd6704a7$9ae58dd6-3cde-4943-9ac1-bd9d4f7d690c$ba5d6311-daee-4abc-b2fb-fae2184ef3eb$ed93259c-7b8b-46d7-97fb-f194e0e04b3a$4e29c621-223e-4859-8e96-db04b967815a$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$056a8adc-92f4-4b33-90d9-4b3b4026bbbc$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098upstream_cells_mapInt64newIntegerVector$ac9c8845-284d-4c21-b05d-d930f86598a3precedence_heuristic cell_id$ac9c8845-284d-4c21-b05d-d930f86598a3downstream_cells_map7run_mountaincar_binary_episodic_countinuous_param_study$b53dba81-a9e9-41da-8fc2-7736bf25f2dcupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCounterButtonCore.applicable@bindBase.get$192cc1cf-9ea1-492d-baa7-f2e197abecd4precedence_heuristic cell_id$192cc1cf-9ea1-492d-baa7-f2e197abecd4downstream_cells_map+run_mountaincar_binary_episodic_param_study$8eb42403-1234-4e59-993e-057cc3a6d5c9upstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunnerCounterButtonCore.applicable@bindBase.get$a4eec4d3-5a75-4b52-ab9c-9d9e83d5547dprecedence_heuristic cell_id$a4eec4d3-5a75-4b52-ab9c-9d9e83d5547ddownstream_cells_mapep_step$374af774-3a97-49b5-a3bb-bc3f7f63a3fa$af144759-fe66-4ad0-b378-e9eb4e859db4upstream_cells_mapCore:Base.get@bindSliderlengthep$e1274f57-75cb-4659-a82f-e5870c5367e2BasePlutoRunnerPlutoRunner.create_bondCore.applicable$c8b47eac-2d45-419a-bec6-2ae0cdc59393precedence_heuristic cell_id$c8b47eac-2d45-419a-bec6-2ae0cdc59393downstream_cells_mapContinuousMDPTransitionSampler$c8b47eac-2d45-419a-bec6-2ae0cdc59393$537270ba-122b-4f2b-880b-31d086766295$3c316495-bb6c-41e2-a38f-ba867a319fbbAbstractContinuousTransition$c8b47eac-2d45-419a-bec6-2ae0cdc59393$537270ba-122b-4f2b-880b-31d086766295upstream_cells_map AbstractTransition$d963ff6d-f1b6-4799-aa0e-1ae100310d84Main.Base.inferencebarrier@assertAbstractContinuousTransition$c8b47eac-2d45-419a-bec6-2ae0cdc59393typeofpromote_typeAnyFunctionRealMainnewContinuousMDPTransitionSampler$c8b47eac-2d45-419a-bec6-2ae0cdc59393throwAssertionError!===$36a6e43f-6bcf-4c27-bfbb-047760e77adaprecedence_heuristic cell_id$36a6e43f-6bcf-4c27-bfbb-047760e77adadownstream_cells_mapupstream_cells_map@md_strgetindex$436c52d2-280b-4ca4-9360-d6587b8254c7precedence_heuristic cell_id$436c52d2-280b-4ca4-9360-d6587b8254c7downstream_cells_mapupstream_cells_map@md_strgetindex$e96d592d-1e54-486d-8ad9-b857f85476e8precedence_heuristic cell_id$e96d592d-1e54-486d-8ad9-b857f85476e8downstream_cells_map#actor_critic_linear_parameter_study$7afb6fb0-248a-4518-b94f-9876f81eca64$1b102220-6d78-480d-a77f-0e57bad23dca$d57375a5-b9e0-4742-b5f7-6a7da891604aupstream_cells_map@NamedTuple:IntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84RealInt64FunctionBase-^+$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6cprecedence_heuristic cell_id$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6cdownstream_cells_mapupstream_cells_map:corridor_parameter_studies$e5c1aca8-7575-4835-8273-e69ca0a55fe8$646bc853-b7fc-49fa-a201-ff98e8f952d4^$4da20fd7-b897-4f26-bf2a-f08d66ddf90fprecedence_heuristic cell_id$4da20fd7-b897-4f26-bf2a-f08d66ddf90fdownstream_cells_map%actor_critic_with_eligibility_traces!$05bfd818-bf4e-4bda-baa9-5ba647867097$68806899-9972-460a-9f11-daa708a9d610$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098upstream_cells_map "zerotypeminzero_params!$e6cf9550-2e69-4b82-92cf-5e07a35490aaupdate_traces_with_gradient!$25be5dcf-be63-46c4-b6de-6cf79fa28fd0$056a8adc-92f4-4b33-90d9-4b3b4026bbbconeContinuousMDP$537270ba-122b-4f2b-880b-31d086766295Base.CoreLogging.!VectorReal'Base.CoreLogging.Base.fixup_stdlib_pathepisode_stepsdeepcopy/@infoBase.invokelatestupdate_params_with_gradient!$b0a66a19-ee76-463b-a704-8fcee85444d0$a893a87b-2d07-4db5-9d1a-9da8646216f4$f55afa58-962d-4551-8d95-a5b467d61adfBase.CoreLogging.invokelatestBase.CoreLogging.===error&form_state_and_policy_function_outputs$e7e49ff8-32df-48a4-afb2-462859592e92$11b9beea-b0cd-45eb-84c6-151728894df0#___this_pluto_module_nameIntegerFunctionBase<=γpush!Base.CoreLogging.isa-bad_continuous_action$b966b248-fb4d-457d-90f6-114370846242+*Base.CoreLogging.>=episode_rewards$11ea640c-3981-404d-87c6-4d3d0708a2b8precedence_heuristic cell_id$11ea640c-3981-404d-87c6-4d3d0708a2b8downstream_cells_map,actor_critic_linear_episodic_parameter_studyupstream_cells_map sumRandom$df7f84e8-b42a-4001-9dbf-6bc3ced94207StateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84lengthRealscatter/isempty4actor_critic_with_eligibility_traces_linear_features$68806899-9972-460a-9f11-daa708a9d610:AbstractVector|>InfrandIntegerFunctionUInt64-plotfoldxt+MapLayoutRandom.seed!$281360af-46bf-4c73-bf11-3cb1153ad3e2precedence_heuristic cell_id$281360af-46bf-4c73-bf11-3cb1153ad3e2downstream_cells_mapupstream_cells_map$9ae58dd6-3cde-4943-9ac1-bd9d4f7d690cprecedence_heuristic cell_id$9ae58dd6-3cde-4943-9ac1-bd9d4f7d690cdownstream_cells_map,update_squashed_gaussian_eligibility_vector!$05f120be-9695-4824-82fd-142a0df13098upstream_cells_mapexp:'BinarySquashedGaussianEligibilityVector$76fd79a2-2bc8-45f8-a243-48415118898akfirstBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41VectorRealMatrix+NTuplelast$da3cb392-78f2-48b2-b0dc-5f016664798cprecedence_heuristic cell_id$da3cb392-78f2-48b2-b0dc-5f016664798cdownstream_cells_mapupstream_cells_map mountaincar_continuing_tile_test$b02ba928-5b9f-4695-b980-07988c788bb9show_mountaincar_trajectory$ba645f6b-143f-4e83-9003-707770ae308d$dca2f8e2-76af-4679-bf81-3824c15fc76dprecedence_heuristic cell_id$dca2f8e2-76af-4679-bf81-3824c15fc76ddownstream_cells_mapreinforce_test3$11a55af7-5301-4507-bb26-88e1e11236dbupstream_cells_mapInt64cartpole_setup$26880577-d267-4950-8725-7afe0d0402b6^4actor_critic_with_eligibility_traces_binary_features$05bfd818-bf4e-4bda-baa9-5ba647867097typemax$8019bec9-1228-407b-9199-2fe29f26a981precedence_heuristic cell_id$8019bec9-1228-407b-9199-2fe29f26a981downstream_cells_mapupstream_cells_map@md_strgetindex$fd964539-2baf-4ff1-b286-5a0bb1b222c4precedence_heuristic cell_id$fd964539-2baf-4ff1-b286-5a0bb1b222c4downstream_cells_mapupstream_cells_map@md_strgetindex$5720e942-d3f8-4329-83a8-8bcedf078b6aprecedence_heuristic cell_id$5720e942-d3f8-4329-83a8-8bcedf078b6adownstream_cells_mapupstream_cells_mapcorridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabe-reinforce_monte_carlo_control_linear_features$8e39bd15-862e-4941-88f9-2794b861a523^update_corridor_features!$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811$62e677ac-2070-4f6b-9df2-90849d89fa9fprecedence_heuristic cell_id$62e677ac-2070-4f6b-9df2-90849d89fa9fdownstream_cells_mapcorridor_terminal_probabilitiesupstream_cells_mapcorridor_state_counts$54f559b6-8a62-4a42-894d-c56e41d5ebefsum-$11b9beea-b0cd-45eb-84c6-151728894df0precedence_heuristic cell_id$11b9beea-b0cd-45eb-84c6-151728894df0downstream_cells_map&form_state_and_policy_function_outputs$4d4ae57b-afc3-44f9-b6fc-892f59f82921$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90fupstream_cells_mapVector%form_state_continuous_policy_function$f545c800-0bf3-491f-9d7d-42341cfdb573Realdeepcopyform_state_value_function$e7566274-5518-4e28-8738-d4b1747d0cfbFunctioncopy$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290precedence_heuristic cell_id$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290downstream_cells_map-reinforce_monte_carlo_control_binary_features$d037ea92-915c-4dc7-97c6-d006d92e088a$f2ed56c9-c2b7-42cb-a083-e12aeaa126ef$83ca0577-15d7-4448-b597-c77810b812bf$e5c1aca8-7575-4835-8273-e69ca0a55fe8upstream_cells_mapsetup_binary_policy_arguments$96506201-6b66-49e6-8179-06952e2394e1zerosIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84RealFunctionlength!update_binary_eligibility_vector!$042fbafe-2401-4fb7-ac13-4531e0782c79Matrix!update_binary_action_preferences!$a361f4c9-47ce-42ad-899c-87b611c0d471reinforce_monte_carlo_control!$0ac7ea44-14f6-4e80-80f9-d6df8059bb38$55ba8725-0ddf-4196-a41d-3f3c490a8d84precedence_heuristic cell_id$55ba8725-0ddf-4196-a41d-3f3c490a8d84downstream_cells_map5actor_critic_binary_episodic_gaussian_parameter_study$b53dba81-a9e9-41da-8fc2-7736bf25f2dcupstream_cells_map Random$df7f84e8-b42a-4001-9dbf-6bc3ced94207ContinuousMDP$537270ba-122b-4f2b-880b-31d086766295copyVectorRealscatter/Matrixisemptymean:AbstractVector|>Infmake_n_param_dist_policy_params$ba41f521-4ee2-42a6-bf18-078bfa4b875ezerosrandIntegerFunctionUInt64-plotfoldxt+MapEactor_critic_with_eligibility_traces_binary_features_gaussian_actions$20776e09-7d9b-4db8-a060-7bceeec65b47LayoutRandom.seed!$a540814a-57a1-4b98-9443-59e401425444precedence_heuristic cell_id$a540814a-57a1-4b98-9443-59e401425444downstream_cells_mapbinary_value_function$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2$05bfd818-bf4e-4bda-baa9-5ba647867097$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098upstream_cells_map zero:islessBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41julia.simdloop@inboundsnothingVectormake_n_param_dist_policy_params$ba41f521-4ee2-42a6-bf18-078bfa4b875ezerosrandIntegerFunctionUInt64plotfoldxt+MapEactor_critic_with_eligibility_traces_binary_features_gaussian_actions$20776e09-7d9b-4db8-a060-7bceeec65b47LayoutRandom.seed!$5b15f5c9-80bf-47f0-898a-f8dead5b927cprecedence_heuristic cell_id$5b15f5c9-80bf-47f0-898a-f8dead5b927cdownstream_cells_mapupstream_cells_map@md_strgetindex$266d2234-26c8-43f1-9e75-49440a230ed6precedence_heuristic cell_id$266d2234-26c8-43f1-9e75-49440a230ed6downstream_cells_map%actor_critic_with_eligibility_traces!$05bfd818-bf4e-4bda-baa9-5ba647867097$68806899-9972-460a-9f11-daa708a9d610$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098upstream_cells_map zerozero_params!$e6cf9550-2e69-4b82-92cf-5e07a35490aaupdate_traces_with_gradient!$25be5dcf-be63-46c4-b6de-6cf79fa28fd0$056a8adc-92f4-4b33-90d9-4b3b4026bbbconeStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84lengthsample_action$d963ff6d-f1b6-4799-aa0e-1ae100310d84VectorRealdeepcopy/update_params_with_gradient!$b0a66a19-ee76-463b-a704-8fcee85444d0$a893a87b-2d07-4db5-9d1a-9da8646216f4$f55afa58-962d-4551-8d95-a5b467d61adfzerossoft_max!$33c99850-67cd-4754-94b9-6df97b238e27&form_state_and_policy_function_outputs$e7e49ff8-32df-48a4-afb2-462859592e92$11b9beea-b0cd-45eb-84c6-151728894df0IntegerFunction<=Int64push!-+*$aa69e4ea-91e0-496a-a7be-529e67f4dbecprecedence_heuristic cell_id$aa69e4ea-91e0-496a-a7be-529e67f4dbecdownstream_cells_mapupstream_cells_map1reinforce_with_baseline_monte_carlo_control_fcann$697b2310-9d96-4f7f-be62-c3bd6bf736f3corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabe^update_corridor_features!$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811$10ee7709-0816-48d2-abe0-9be3dd04700fprecedence_heuristic cell_id$10ee7709-0816-48d2-abe0-9be3dd04700fdownstream_cells_mapupstream_cells_mapplot_continuing_step_rewards$0964133c-3a5b-433b-a8c4-a97813c37583!mountaincar_continuing_fcann_test$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9$7d94922e-dc9f-4953-b539-24aaa2c85b12precedence_heuristic cell_id$7d94922e-dc9f-4953-b539-24aaa2c85b12downstream_cells_mapcontinuing_study_params$42775fd1-5b27-48e0-abf1-9b22bb775e6dupstream_cells_mapCoreBasePlutoRunner.create_bondPlutoRunner(create_actor_critic_continuing_params_UI$5b15d91e-7119-4f85-a54a-7d4f1fdaf097Core.applicable@bindBase.get$df7f84e8-b42a-4001-9dbf-6bc3ced94207precedence_heuristiccell_id$df7f84e8-b42a-4001-9dbf-6bc3ced94207downstream_cells_mapStatisticsStaticArraysStatsBaseTransducersLinearAlgebraDistributionsPlutoDevMacrosRandom$d037ea92-915c-4dc7-97c6-d006d92e088a$83ca0577-15d7-4448-b597-c77810b812bf$e5c1aca8-7575-4835-8273-e69ca0a55fe8$1f041cb3-618c-4380-a1ec-d7bbe4a80f62$11ea640c-3981-404d-87c6-4d3d0708a2b8$f8614042-7c94-4d47-a1b6-4e96676b4e8b$ba642a22-6623-482a-ab4a-81585b83e457$8bc280db-e57d-4e40-be46-1790f4f7d9e7$11063fff-4d36-46d5-828f-dbed0f46b9cf$55ba8725-0ddf-4196-a41d-3f3c490a8d84$61949faa-8174-4b7b-8fbc-01d5f850b419$dd8e8cd2-7b41-46c4-8530-adefb7aea684$08505e88-9c23-4e95-91e3-d18bf5133dbc$87482ea5-5265-4e02-92c0-1a8bb44ff0f4Threadsupstream_cells_map$352d2952-cb83-47d3-9078-2b2ef9927443precedence_heuristic cell_id$352d2952-cb83-47d3-9078-2b2ef9927443downstream_cells_mapcreate_cartpole_functions$f27f2bcd-05b6-44fe-bf9e-a3e51556db7cupstream_cells_mapCartPoleStatezerodeg2rad>islessrandcartpole_runge_kutta_stepRealFunction<-Float32clampCartPoleVehicleabs$0964133c-3a5b-433b-a8c4-a97813c37583precedence_heuristic cell_id$0964133c-3a5b-433b-a8c4-a97813c37583downstream_cells_mapplot_continuing_step_rewards$645e93e7-e92e-49c4-9757-8294fabf4e9b$04b5929a-2058-49c9-963a-96c752a1d67d$98222fcd-b456-477c-90dd-844df36877e5$10ee7709-0816-48d2-abe0-9be3dd04700fupstream_cells_map:LinRangecumsumVectorRealInt64lengthscatterplot/Layoutround$349631b2-4686-49a9-9f3a-1e4ad588b568precedence_heuristic cell_id$349631b2-4686-49a9-9f3a-1e4ad588b568downstream_cells_mapmountaincar_continuous_mdp2$fee14dfe-c5ca-4126-a830-cc9d7eda5433$cd9c9eeb-c90d-4499-9503-7773d5250f47upstream_cells_map$create_continuous_action_mountaincar$b86ee9d3-b6b5-4ea0-8f55-1927571cdfbf$8544eddb-2095-4a3c-82e0-920123a88e6dprecedence_heuristic cell_id$8544eddb-2095-4a3c-82e0-920123a88e6ddownstream_cells_mapupstream_cells_map@md_strgetindex$31f7e903-30b6-4193-9174-88093e004de4precedence_heuristic cell_id$31f7e903-30b6-4193-9174-88093e004de4downstream_cells_mapupstream_cells_map@md_strgetindex$fee14dfe-c5ca-4126-a830-cc9d7eda5433precedence_heuristic cell_id$fee14dfe-c5ca-4126-a830-cc9d7eda5433downstream_cells_map"mountaincar_continuous_test_train2$cd9c9eeb-c90d-4499-9503-7773d5250f47$b695ef21-a1ac-4d1f-a0e1-71cd81cede18upstream_cells_mapInt64mountaincar_continuous_mdp2$349631b2-4686-49a9-9f3a-1e4ad588b568mountaincar_tilecoding_setup$7c592385-e8d3-4efe-962c-d39debb64405Eactor_critic_with_eligibility_traces_binary_features_gaussian_actions$20776e09-7d9b-4db8-a060-7bceeec65b47typemax$b53dba81-a9e9-41da-8fc2-7736bf25f2dcprecedence_heuristic cell_id$b53dba81-a9e9-41da-8fc2-7736bf25f2dcdownstream_cells_mapupstream_cells_mapmountaincar_continuous_mdp$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2@md_str$mountaincar_binary_continuous_params$71a5fce8-6d9a-4625-bad1-a951d61bff28<>isless7run_mountaincar_binary_episodic_countinuous_param_study$ac9c8845-284d-4c21-b05d-d930f86598a3mountaincar_tilecoding_setup$7c592385-e8d3-4efe-962c-d39debb644055actor_critic_binary_episodic_gaussian_parameter_study$55ba8725-0ddf-4196-a41d-3f3c490a8d84$13ebc12f-ff6f-4266-88d3-28d6df5fcf59getindex$beb01fb8-c77d-4b5c-a66d-3812415e04a3precedence_heuristic cell_id$beb01fb8-c77d-4b5c-a66d-3812415e04a3downstream_cells_mapupstream_cells_map@md_strgetindex$8bc280db-e57d-4e40-be46-1790f4f7d9e7precedence_heuristic cell_id$8bc280db-e57d-4e40-be46-1790f4f7d9e7downstream_cells_map"actor_critic_fcann_parameter_study$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62$c251a630-7114-4188-9323-8d8feb5c32e0upstream_cells_map AbstractVector*actor_critic_with_eligibility_traces_fcann$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54FCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84Random$df7f84e8-b42a-4001-9dbf-6bc3ced94207randIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANN.initializeparams_saxeInt64VectorRealFunctionscatterplotUInt64lengthaverage_continuing_runs$ba642a22-6623-482a-ab4a-81585b83e457LayoutRandom.seed!$89901156-b874-416b-89c1-6dc434a4eb17precedence_heuristic cell_id$89901156-b874-416b-89c1-6dc434a4eb17downstream_cells_mapupstream_cells_map@md_strgetindex$ff76ef94-fdf5-41f3-a31a-21c4629efabeprecedence_heuristic cell_id$ff76ef94-fdf5-41f3-a31a-21c4629efabedownstream_cells_mapcorridor_mdp $f7433324-acc3-49a5-b5b3-ada0c8f09d52$fb8904a9-ae64-41cc-93b6-5a25855edad0$cecc2a35-3850-4f66-84e8-e29da4f3d4b0$f2f2dd1d-180c-4d36-b515-5079d129f93a$e1493cea-19c4-475d-98a0-86d27fb04af1$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72c$0c9986bb-54c0-4b08-9c29-4bfb0b68b54e$d037ea92-915c-4dc7-97c6-d006d92e088a$f2ed56c9-c2b7-42cb-a083-e12aeaa126ef$cbea5840-49d2-4e91-be9c-f5f15666d78a$5720e942-d3f8-4329-83a8-8bcedf078b6a$cacaaca6-6e01-464f-a2ee-cbf62737a426$07ad517a-c2ac-4377-99fb-adb13d0f1d0c$aa69e4ea-91e0-496a-a7be-529e67f4dbec$83ca0577-15d7-4448-b597-c77810b812bf$a12b92d1-e045-4f92-b8cd-eee5d56fa67d$e5c1aca8-7575-4835-8273-e69ca0a55fe8$7d63b960-3998-4f7b-8cbb-ccd49db9aeac$9db9ff71-bee9-4bea-a45b-748f8517fed1$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0$646bc853-b7fc-49fa-a201-ff98e8f952d4$3bccf6fc-6e5e-4f62-ad40-1ff0a3740728$396e0047-d848-462f-a769-0cc2829abc78$bc8a399b-8864-4473-89d2-e3b0a03d15b5$72273f27-d0b9-4645-a609-cb65cc9332eeupstream_cells_mapmake_corridor_mdp$5cc4d12d-b537-47e2-8109-4e7a234fdf25$581f7e9b-a5c2-4841-9605-85f9585b0274precedence_heuristic cell_id$581f7e9b-a5c2-4841-9605-85f9585b0274downstream_cells_map!update_linear_action_preferences!$4634267b-5dea-4164-8bb2-1eb2fd4d7954$8e39bd15-862e-4941-88f9-2794b861a523$d1ed25e6-60c6-411f-a541-99986e5da2c5$57e5e12a-b722-4ea3-ab3b-e5711029e640$68806899-9972-460a-9f11-daa708a9d610$d5020a8d-1dd7-403c-9d1f-665b95543943upstream_cells_mapzeroBLASBLAS.gemv!MatrixoneAbstractFloatVector$8aa16866-bfda-48df-9cf1-cf3d2e203ccbprecedence_heuristic cell_id$8aa16866-bfda-48df-9cf1-cf3d2e203ccbdownstream_cells_map8cartpole_tilecoding_reinforce_continuous_parameter_studyupstream_cells_mapcartpole_setup$26880577-d267-4950-8725-7afe0d0402b6:|>setup_cartpole_problemscatterplot/Lreinforce_with_baseline_monte_carlo_control_binary_features_gaussian_actions$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00+foldxtisemptyMapmeanLayout$04b5929a-2058-49c9-963a-96c752a1d67dprecedence_heuristic cell_id$04b5929a-2058-49c9-963a-96c752a1d67ddownstream_cells_mapupstream_cells_mapplot_continuing_step_rewards$0964133c-3a5b-433b-a8c4-a97813c37583cartpole_continuing_fcann_test$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27$f0104778-81a6-417b-8501-f916e5e7f3afprecedence_heuristic cell_id$f0104778-81a6-417b-8501-f916e5e7f3afdownstream_cells_mapmake_corridor_continuing_mdp$1ac9296f-047b-4051-ba5c-0c23d5f9cde9upstream_cells_mapifelseIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84Returns-StateMDPTransitionSampler$d963ff6d-f1b6-4799-aa0e-1ae100310d84Float32+*iseven==$3e3c5897-809f-46e3-bb58-f115b082443eprecedence_heuristic cell_id$3e3c5897-809f-46e3-bb58-f115b082443edownstream_cells_mapAactor_critic_with_eligibility_traces_binary_features_beta_actions$dd8e8cd2-7b41-46c4-8530-adefb7aea684$4156d955-9daf-4429-b152-e8332980fb9eupstream_cells_map update_beta_eligibility_vector!$bfe7e41d-6318-4bd4-b892-287831876abc$d41f1dd1-45fe-4456-9a01-ed47fd6704a7binary_value_function$a540814a-57a1-4b98-9443-59e401425444make_beta_sampler$b2082ab0-73a4-45a6-8772-a2e6e22b519amake_n_param_dist_policy_params$ba41f521-4ee2-42a6-bf18-078bfa4b875ezerosBinaryFeatureVector$f92bb265-4b19-4f0e-a698-d7547bb6dd41randIntegerVectorContinuousMDP$537270ba-122b-4f2b-880b-31d086766295RealFunctionupdate_binary_value_gradient!$03a218cb-aa83-4000-85b5-c6f247087053"setup_binary_beta_policy_arguments$ed93259c-7b8b-46d7-97fb-f194e0e04b3aNTupleUnionMatrix!update_binary_action_preferences!$a361f4c9-47ce-42ad-899c-87b611c0d471%actor_critic_with_eligibility_traces!$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90f$a9db3f85-ff56-4bbc-be87-47b893ef3b7bprecedence_heuristic cell_id$a9db3f85-ff56-4bbc-be87-47b893ef3b7bdownstream_cells_mapmountaincar_continuing_step$00152954-dc98-4120-b94b-2ea4d987832bupstream_cells_mapMountainCarTask.step MountainCarTask.initialize_state==IntegerMountainCarTask$08505e88-9c23-4e95-91e3-d18bf5133dbcprecedence_heuristic cell_id$08505e88-9c23-4e95-91e3-d18bf5133dbcdownstream_cells_map>actor_critic_binary_episodic_squashed_gaussian_parameter_study$0d93132d-5819-47dc-8cf2-462d480d9c3dupstream_cells_map Random$df7f84e8-b42a-4001-9dbf-6bc3ced94207ContinuousMDP$537270ba-122b-4f2b-880b-31d086766295copyVectorRealscatter/MatrixisemptymeanNactor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions$05f120be-9695-4824-82fd-142a0df13098$717e4c69-59d5-4929-923f-dd35a97fb160:AbstractVector|>Infmake_n_param_dist_policy_params$ba41f521-4ee2-42a6-bf18-078bfa4b875ezerosrandIntegerFunctionUInt64-plotfoldxt+MapLayoutRandom.seed!$ad0009af-2cfc-4820-bd4a-698ad391f459precedence_heuristic cell_id$ad0009af-2cfc-4820-bd4a-698ad391f459downstream_cells_mapupstream_cells_mapbeta_params$7bf209c8-ef0a-46d1-937e-b1a6e45dc62escatterplotLinRangemake_beta_dist$0b01ba67-3921-4f3f-a7e8-235190bc84eb$16fcc2d0-9f2f-4226-9dcc-6d86248cab26precedence_heuristic cell_id$16fcc2d0-9f2f-4226-9dcc-6d86248cab26downstream_cells_mapplot_state_distributions$9cf3dc5f-8a25-479f-93db-06e34f0d37a0upstream_cells_map :sumvcatHypertextLiteral.BypassHypertextLiteral.contentsizeadjoint@htl$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebbar-plot/HypertextLiteral$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70attrheatmapHypertextLiteral.ResultLayoutcollect_state_distributions$0c9986bb-54c0-4b08-9c29-4bfb0b68b54econj$11063fff-4d36-46d5-828f-dbed0f46b9cfprecedence_heuristic cell_id$11063fff-4d36-46d5-828f-dbed0f46b9cfdownstream_cells_map"actor_critic_fcann_parameter_study$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62$c251a630-7114-4188-9323-8d8feb5c32e0upstream_cells_map :AbstractVector*actor_critic_with_eligibility_traces_fcann$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54FCANN$d963ff6d-f1b6-4799-aa0e-1ae100310d84Random$df7f84e8-b42a-4001-9dbf-6bc3ced94207randIntegerStateMDP$d963ff6d-f1b6-4799-aa0e-1ae100310d84FCANN.initializeparams_saxeInt64VectorRealFunctionUInt64lengthaverage_continuing_runs$ba642a22-6623-482a-ab4a-81585b83e457DataFrameRandom.seed!$8fcdca63-01a0-4d4b-933c-06a7621d980aprecedence_heuristic cell_id$8fcdca63-01a0-4d4b-933c-06a7621d980adownstream_cells_mapupstream_cells_map$33c99850-67cd-4754-94b9-6df97b238e27precedence_heuristic cell_id$33c99850-67cd-4754-94b9-6df97b238e27downstream_cells_mapsoft_max!$4634267b-5dea-4164-8bb2-1eb2fd4d7954$042fbafe-2401-4fb7-ac13-4531e0782c79$37ec6802-d4c2-4470-ad69-439d5a732f77$4d4ae57b-afc3-44f9-b6fc-892f59f82921$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1upstream_cells_map zeroisless@inboundsonenothinglengthactor_critic_binary_episodic_squashed_gaussian_parameter_study$0d93132d-5819-47dc-8cf2-462d480d9c3dupstream_cells_map@NamedTuple:IntegerContinuousMDP$537270ba-122b-4f2b-880b-31d086766295RealInt64FunctionBase-^+$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72cprecedence_heuristic cell_id$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72cdownstream_cells_mapcorridor_train$5334064b-5a16-4135-afa0-86a48291725b$5981f52b-d829-4c7d-b47b-33310f7d64a2$573878bb-020d-40f6-9329-3d5f91843010upstream_cells_mapInt64corridor_mdp$ff76ef94-fdf5-41f3-a31a-21c4629efabeget_corridor_features$6bb0263e-368e-462a-948c-baf9cfa82512typemaxsarsa_λcell_execution_order$fac138d9-3c5d-44b0-a87c-b13872f19450$e034b9cb-f4ee-46f4-bea6-72c93c75d966$666a4e89-306b-4fb2-bdc4-3dda2c63153f$df7f84e8-b42a-4001-9dbf-6bc3ced94207$d963ff6d-f1b6-4799-aa0e-1ae100310d84$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70$7cf26604-9c2b-4a77-9674-7d4dac2f99f0$36a6e43f-6bcf-4c27-bfbb-047760e77ada$31f7e903-30b6-4193-9174-88093e004de4$48dcd2d0-a940-41da-a097-90c780f2ec4d$d95f75b5-21d8-4862-baa7-50b58d9725b8$fc3dcd26-c5cf-4141-bf6c-eaed5fc9bb1d$dcb306ae-a1b1-43d6-ba6e-e38668838689$33c99850-67cd-4754-94b9-6df97b238e27$7a6fb1f0-fc3c-4c29-a6d9-769d32ca98a9$5cc4d12d-b537-47e2-8109-4e7a234fdf25$ff76ef94-fdf5-41f3-a31a-21c4629efabe$6bb0263e-368e-462a-948c-baf9cfa82512$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811$f2f2dd1d-180c-4d36-b515-5079d129f93a$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72c$5334064b-5a16-4135-afa0-86a48291725b$5981f52b-d829-4c7d-b47b-33310f7d64a2$8019bec9-1228-407b-9199-2fe29f26a981$38e5d800-4d43-40d2-87ea-f7d4b4283dab$b94fc99c-f439-4df2-8da3-c01718a136c4$9c342958-1971-48ec-b919-5dfdcbc915a4$e5faaa1b-88cb-43e2-8d04-8972b58b4bda$406638af-1e08-44d2-9ee4-97aa9294a94b$aa450da4-fe84-4eea-b6c4-9820b7982437$fdd3f4fd-4706-4d6b-b150-6ee6b4b370cb$41d62de1-2c92-41ee-9430-b9ca3007afd9$135f205a-f87e-4691-8e87-d317d6312c84$ca360680-afc9-4dd9-9351-493643f91575$4a39f9a7-72d4-44ad-895a-742cd1291f92$98229733-a71e-44ca-a52a-b7229cf8b422$37a8ef7e-e859-4ef0-81e2-76c02a324031$339b4d2b-2237-46a3-9867-ecc3332856c1$05b0fcad-628b-48d2-aa24-f6f562dbb660$17d07ef4-7c0a-47cc-a701-32c60336571b$76b03e72-da04-4530-8534-6d6468268cbd$2a586e46-66e4-461a-85c8-5817e4d1aa43$90d3b96b-ad2b-405c-951b-f48ec7ccf24a$f924eb30-d1cc-4941-8fb5-ff70ad425ab9$189798b3-ec6b-48b9-918c-ee0f65935ab3$70096b14-beab-4f71-9886-6355c749bb8a$1558cec1-c4fd-4bc0-85ed-ae22c6067d41$e3a2fb12-37ce-4c23-ad93-5fc89991aabb$58403c8e-0ee4-4466-ba25-ee0c86fb0b47$73b90260-d57a-449a-8db6-47f91e6a4e4f$ee72af8d-3cb8-4314-82df-580f068e1252$89901156-b874-416b-89c1-6dc434a4eb17$5c11a92d-7496-4aba-af15-2537eac49dd7$581f7e9b-a5c2-4841-9605-85f9585b0274$da2d3186-a778-41cc-9b49-759bf1e9b8fa$f92bb265-4b19-4f0e-a698-d7547bb6dd41$8eab55a5-41b7-4f5e-a02f-4c19388bc9ea$a361f4c9-47ce-42ad-899c-87b611c0d471$cc3ac95e-a398-438a-ba3d-62b6733f6342$4634267b-5dea-4164-8bb2-1eb2fd4d7954$45f0a385-6465-4acc-8637-1b007a0fe215$41dc149d-c6f3-4b0d-a856-06f3aaae3049$b0a66a19-ee76-463b-a704-8fcee85444d0$042fbafe-2401-4fb7-ac13-4531e0782c79$65d2add6-fd6f-456c-92ed-3cd9d1862ef6$96506201-6b66-49e6-8179-06952e2394e1$0e9de19e-bcd4-40ac-9831-afb6cad38422$a206c759-3f6e-4003-8cba-5f6ce6742646$3bafd7df-9bc0-4d13-874d-739590cf3ad9$cc45091e-b889-4d5a-9eef-84d80f792046$d83dc659-dce7-41dd-a8e7-2933ab39d15c$1753b5ed-c00b-4b60-b492-822180778e8c$03a218cb-aa83-4000-85b5-c6f247087053$a893a87b-2d07-4db5-9d1a-9da8646216f4$5c4a383f-fcf2-4f2b-819f-6d84471dda00$2cbc972b-c685-4c1c-8a8d-9d58b197ad90$77cf3a74-899f-4ade-99f2-5aaf7a98c02d$0bf3b988-b3fb-49d5-8dde-b25766596363$a540814a-57a1-4b98-9443-59e401425444$635abb34-2c97-4f04-a74c-22fbec32f408$37ec6802-d4c2-4470-ad69-439d5a732f77$e7566274-5518-4e28-8738-d4b1747d0cfb$f3e2db06-9cb7-464a-96b8-938175efd26b$e1aec891-d95a-47d1-97d7-d2a4cfb16e64$8544eddb-2095-4a3c-82e0-920123a88e6d$48b342f2-e48f-457a-9bd3-b3504a79f3a6$fd89433e-643c-474b-b3c4-a997678421a6$1ec1acf1-f833-4478-9b3c-88029340a629$b72e030f-7d52-481f-b4f7-2b16b227e547$047656d1-2921-40f2-b75b-ce4a87098007$738ada7f-edc7-4ed3-a15e-e92113468738$ce33f710-fd9d-4dfa-acda-40204e54d518$f4b6f10b-4cd0-4be6-98ec-4d4ffb696392$e7e49ff8-32df-48a4-afb2-462859592e92$1386ffdb-940d-4f1b-a872-4e38647b5335$e2b09af1-0f22-4f7f-b806-54fa522adb20$4cbdb082-22ba-49e9-a6ed-4380917625ac$e6cf9550-2e69-4b82-92cf-5e07a35490aa$25be5dcf-be63-46c4-b6de-6cf79fa28fd0$4fea7232-f286-4a8b-93f8-a0702818ab31$d8222abf-139c-4220-8e92-cc987ec6900c$511a847f-234c-465e-8f4a-688e79d9b975$0284f0d7-b8a9-4ae6-add0-ac1078571d9b$b4875f2b-5487-429f-80a3-d1032bbccfc1$4915b1ed-ad53-4ece-9b00-bc136d47d8dc$5b15f5c9-80bf-47f0-898a-f8dead5b927c$f3bc47b5-03fc-4bd9-a890-26f9608a730b$436c52d2-280b-4ca4-9360-d6587b8254c7$f0104778-81a6-417b-8501-f916e5e7f3af$1ac9296f-047b-4051-ba5c-0c23d5f9cde9$ba642a22-6623-482a-ab4a-81585b83e457$e96d592d-1e54-486d-8ad9-b857f85476e8$5aba4f96-e877-457e-8e95-18737348f99f$5b15d91e-7119-4f85-a54a-7d4f1fdaf097$7d94922e-dc9f-4953-b539-24aaa2c85b12$da8d0bca-105b-4d0b-a73d-ee5c9059aeaf$d17a4bd0-5992-4247-912d-73d51758d2f3$352d2952-cb83-47d3-9078-2b2ef9927443$f27f2bcd-05b6-44fe-bf9e-a3e51556db7c$b87ff1a9-abff-40f7-a1d8-f751a1c8b060$5d434c83-c9ca-499f-8695-c7733031c2de$4c4e643b-d4b9-44f0-8d30-dc521bcc55ac$7dbb42a3-aa8c-47e5-b668-18e6325d4038$de3cba34-9842-44d1-9b79-47126c0a0751$37a273b6-b104-46f0-987a-401dc1c97327$8e742d32-c074-4981-b35b-b596b64c869b$64900586-ef92-48e4-839e-ff952a46671b$19dfabda-7049-4050-8662-0385529c0c5a$966ef17c-23be-49dc-bc37-4cb52b34c049$2c5d221a-2469-49e1-9249-dfdc2457f2fa$5ffc271f-c73f-494a-9727-8d7516af2191$42d4600a-bf3c-45ac-b7f5-d23917713ff5$820752af-8966-4ee8-82f7-a40934522de5$0964133c-3a5b-433b-a8c4-a97813c37583$28ce6e60-59cf-408a-8081-b978507b3c72$5500fd8e-64cb-4af7-808d-230440746319$a9db3f85-ff56-4bbc-be87-47b893ef3b7b$00152954-dc98-4120-b94b-2ea4d987832b$46fea69b-599e-46ab-8455-d2da865d9a8e$d3c1379f-acd6-4e15-be7e-a5dbe46a4f62$fed4dc4c-0d1c-4ee3-9d0e-8ef2a7db7486$c926b6df-c40b-4c4c-8a95-ce9e41feb100$f487f2dd-ad09-48ac-ae34-bf50cfa6ac7d$5d35e515-e2d3-443e-becf-eb28c25db346$735b548a-88f5-4a30-ab8f-dfb3d6401b2b$60c21e9c-e42d-4f0b-a910-3b318440fbc8$09dd1440-5d09-421f-addc-b1ede43ff517$7ccadf01-fbba-4dfd-a5ad-770dab9946f9$beb01fb8-c77d-4b5c-a66d-3812415e04a3$68e6f17e-8c87-40f0-a673-1115ecd1b71d$692c1043-4eaf-491e-b8fe-368618867f99$3cfd63ad-b1a2-4b99-ae97-2ff10351e4f5$fd964539-2baf-4ff1-b286-5a0bb1b222c4$0b01ba67-3921-4f3f-a7e8-235190bc84eb$7bf209c8-ef0a-46d1-937e-b1a6e45dc62e$ad0009af-2cfc-4820-bd4a-698ad391f459$b09e1e48-494e-4967-826a-6e70199acad4$5864a5a3-a5a5-43c2-9cb4-7d13b2d20bed$94517664-6988-44dc-a297-e9d5873ee540$b16899b7-36bf-4a5e-8e2f-4496b8450687$00bd2835-b006-4244-9877-bc7e031e3ef8$3e7cecec-eb77-4862-8e3c-b510422e06db$78c83673-2117-4542-b4d8-1c243e8f610b$ae0f5a96-7a4b-47f9-be1e-e803a238a071$c8b47eac-2d45-419a-bec6-2ae0cdc59393$537270ba-122b-4f2b-880b-31d086766295$10cdd16e-a337-4421-a7a0-6de4e4b60c0f$54fff14b-cf53-47b0-9cfa-8b9ee33df54e$76fd79a2-2bc8-45f8-a243-48415118898a$87ee21f3-16ca-4c8c-a0b9-f9e2fd258a91$b966b248-fb4d-457d-90f6-114370846242$f946c886-6246-4f98-a96f-f06984691ad8$f7433324-acc3-49a5-b5b3-ada0c8f09d52$fb8904a9-ae64-41cc-93b6-5a25855edad0$cecc2a35-3850-4f66-84e8-e29da4f3d4b0$a019925a-460a-410e-a54b-50a4cfe0e90e$e1493cea-19c4-475d-98a0-86d27fb04af1$573878bb-020d-40f6-9329-3d5f91843010$0c9986bb-54c0-4b08-9c29-4bfb0b68b54e$54f559b6-8a62-4a42-894d-c56e41d5ebef$62e677ac-2070-4f6b-9df2-90849d89fa9f$bba13634-ff0e-47f7-a23b-8d56098f4ac6$b2082ab0-73a4-45a6-8772-a2e6e22b519a$7a6f3f79-ea06-4994-8b62-90b2056e4034$5261651e-a51e-4e80-8e23-83a4c10e5259$bfe7e41d-6318-4bd4-b892-287831876abc$6bf5ad39-1400-4e1f-a843-a1934b8aaa48$f55afa58-962d-4551-8d95-a5b467d61adf$4fb83451-b6f8-4e6e-a131-1accc8e10b08$740a3f41-9302-481d-b373-762c0dea8eff$d41f1dd1-45fe-4456-9a01-ed47fd6704a7$9ae58dd6-3cde-4943-9ac1-bd9d4f7d690c$f545c800-0bf3-491f-9d7d-42341cfdb573$5b868eba-c1af-49f6-8f93-79b78c319a6f$0ac7ea44-14f6-4e80-80f9-d6df8059bb38$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290$d037ea92-915c-4dc7-97c6-d006d92e088a$0c56b341-24eb-4c78-844e-182f44a7221a$f2ed56c9-c2b7-42cb-a083-e12aeaa126ef$8e39bd15-862e-4941-88f9-2794b861a523$5720e942-d3f8-4329-83a8-8bcedf078b6a$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091$07ad517a-c2ac-4377-99fb-adb13d0f1d0c$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$cbea5840-49d2-4e91-be9c-f5f15666d78a$83ca0577-15d7-4448-b597-c77810b812bf$a7dcc8cd-04ec-48f2-a387-116330eaffb2$e5c1aca8-7575-4835-8273-e69ca0a55fe8$d1ed25e6-60c6-411f-a541-99986e5da2c5$cacaaca6-6e01-464f-a2ee-cbf62737a426$a12b92d1-e045-4f92-b8cd-eee5d56fa67d$44b32cc0-36a8-41fd-89bc-ce894536926c$553b0ceb-f2ca-41ee-99bc-9f53a4487b49$697b2310-9d96-4f7f-be62-c3bd6bf736f3$aa69e4ea-91e0-496a-a7be-529e67f4dbec$76eb6743-cac0-4174-9ba3-a0691c200b54$ba41f521-4ee2-42a6-bf18-078bfa4b875e$ba5d6311-daee-4abc-b2fb-fae2184ef3eb$ed93259c-7b8b-46d7-97fb-f194e0e04b3a$4e29c621-223e-4859-8e96-db04b967815a$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$d5020a8d-1dd7-403c-9d1f-665b95543943$2be8a812-4f21-4fe8-a2de-50497db0345a$056a8adc-92f4-4b33-90d9-4b3b4026bbbc$11b9beea-b0cd-45eb-84c6-151728894df0$4d4ae57b-afc3-44f9-b6fc-892f59f82921$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2$7d63b960-3998-4f7b-8cbb-ccd49db9aeac$646bc853-b7fc-49fa-a201-ff98e8f952d4$94354552-9920-4b90-98d9-f75286d1f53e$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6c$57e5e12a-b722-4ea3-ab3b-e5711029e640$9db9ff71-bee9-4bea-a45b-748f8517fed1$57bbdb10-bed8-459d-8f67-9ea637cf12ba$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0$266d2234-26c8-43f1-9e75-49440a230ed6$83640f5b-fe13-4ec1-98a0-67a56c189ba1$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90f$05bfd818-bf4e-4bda-baa9-5ba647867097$3bccf6fc-6e5e-4f62-ad40-1ff0a3740728$396e0047-d848-462f-a769-0cc2829abc78$1f041cb3-618c-4380-a1ec-d7bbe4a80f62$72273f27-d0b9-4645-a609-cb65cc9332ee$8b35661b-5075-4d63-bc31-044407f99acf$3c89209c-9202-4d5d-841c-ea34be369616$645e93e7-e92e-49c4-9757-8294fabf4e9b$68806899-9972-460a-9f11-daa708a9d610$11ea640c-3981-404d-87c6-4d3d0708a2b8$734573e5-547b-4dcc-89bb-412aa6cc42d6$ff4f977e-48df-4c12-845c-c245b4d39d6d$7afb6fb0-248a-4518-b94f-9876f81eca64$42775fd1-5b27-48e0-abf1-9b22bb775e6d$1b102220-6d78-480d-a77f-0e57bad23dca$b2539398-fdbc-42a2-a8f3-d327358f3643$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54$f8614042-7c94-4d47-a1b6-4e96676b4e8b$8bc280db-e57d-4e40-be46-1790f4f7d9e7$11063fff-4d36-46d5-828f-dbed0f46b9cf$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098$717e4c69-59d5-4929-923f-dd35a97fb160$55ba8725-0ddf-4196-a41d-3f3c490a8d84$61949faa-8174-4b7b-8fbc-01d5f850b419$dd8e8cd2-7b41-46c4-8530-adefb7aea684$08505e88-9c23-4e95-91e3-d18bf5133dbc$87482ea5-5265-4e02-92c0-1a8bb44ff0f4$13ebc12f-ff6f-4266-88d3-28d6df5fcf59$3d065608-eef2-4caa-b17d-ec60714e3d58$bd6a7c16-6c25-4fc2-8e1b-4dab693ce19f$c5a2879c-e89b-47f7-bbd6-48200d7e89e3$65be0e58-24be-4932-92a9-9e4825b14144$3c695d54-c30f-4f04-bd40-f5da53be2a95$3c316495-bb6c-41e2-a38f-ba867a319fbb$024dcd1a-8eaa-4a95-8037-2f578828309c$822e4d69-2582-4956-858e-06ecb091e76a$cf1859d6-f889-4923-8c87-2d7c039f26c3$31db0f58-28e4-454f-9394-25565687266f$fddef10c-7695-4596-9e16-987fd45a57e6$26880577-d267-4950-8725-7afe0d0402b6$0cd96c44-cae6-421f-9fae-26141600bef4$24fa139c-ad4b-49db-ac8f-23c476ed8608$dddc4a2f-34b2-41dc-85b3-55aba4880fa6$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5$d3b56fca-5b79-4465-8987-8d0005f854d8$5859ca11-90f8-4fd6-88ed-c56efe796fe8$281360af-46bf-4c73-bf11-3cb1153ad3e2$8f1b2db4-ed35-44fc-a3d5-e06deae16d48$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03$8aa16866-bfda-48df-9cf1-cf3d2e203ccb$dca2f8e2-76af-4679-bf81-3824c15fc76d$11a55af7-5301-4507-bb26-88e1e11236db$7856b8a0-565d-4c86-9b3c-4424ff9b86dd$8fcdca63-01a0-4d4b-933c-06a7621d980a$76d54520-baa3-44bf-b303-4cdcb8b87080$9acdbf38-2e10-45ec-85a0-d0db8453a599$f0962801-0dfa-421f-8ffc-e64068e49913$c251a630-7114-4188-9323-8d8feb5c32e0$cb70d400-3e9c-441c-b17c-e727e8c928f3$61650a97-b353-4a85-b50b-93fee296ac7b$192b9f82-8d3a-408f-91c2-829cfcd32572$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62$50ae94c4-70f3-4215-82bd-eb2227c2badf$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27$04b5929a-2058-49c9-963a-96c752a1d67d$64b38d1f-ecf9-4843-89a1-4c8953048265$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fce$6acb549a-5d90-4457-a347-d22448ad8071$d34d22ad-89c2-423e-91dd-bfb895dc6540$5eebf3da-bfe7-46eb-81a3-f87f334ee270$9978d537-49ff-4014-a971-b42704c50a6b$54ff46a2-489a-4dd2-bc30-df70c780cc42$407a0724-4bb6-4c83-ab2d-17a0e19c4072$27487ad0-4779-42ce-8def-e660ef04bee0$9d264543-33ab-498a-90f5-5f913c252484$07ba9fe4-aaa7-4123-9865-cbfa79d0d44a$e1274f57-75cb-4659-a82f-e5870c5367e2$a4eec4d3-5a75-4b52-ab9c-9d9e83d5547d$5ee4ce72-7740-4297-8d84-619e0708e4ac$87feff3e-e510-4916-91a9-db3a2cd12225$6b1acb57-159a-4b7f-99fe-5f996522243b$82e0e9a0-9662-429a-87e3-e6bdae02709a$27441783-d3c6-40be-9c36-4941613e6ae9$daf35bfe-8f9c-4f55-971d-4d443be8f8bf$51d6337d-c0bd-40a9-9129-7d88e41e4093$a5b002c9-5e11-462a-9da0-6e060c7963f8$9bce6fdb-2cbc-4758-9a8b-794e490c973d$bb1ef180-39ac-475f-beea-ef573e71a3bf$a8349352-3242-46d5-b0d5-1b6eb5d77e90$2e7c737c-c798-4442-a7e1-d74ccfd73119$f7f58fd2-facc-4b87-9172-5e911677c8f4$d21617aa-6f38-4a90-8586-4b32022497ad$700dcbc4-c94c-4287-8cf0-0b2c7a320a3a$4f96be72-ef3e-4e08-ac4c-be4271dcd14c$54f1546d-87ae-49d2-92ed-6fcc9b66e027$c5dd7e99-57e0-4bc7-97d2-2c780b23bcff$2025ff38-f2ec-4224-b771-ff72ffe1af28$77906355-08f8-4b08-b051-84697199b519$023f67b8-8f38-470a-9766-ac60a75678aa$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9$10ee7709-0816-48d2-abe0-9be3dd04700f$7c592385-e8d3-4efe-962c-d39debb64405$d57375a5-b9e0-4742-b5f7-6a7da891604a$04f42c09-8ab5-4233-b196-51c4aa2dcedb$b02ba928-5b9f-4695-b980-07988c788bb9$98222fcd-b456-477c-90dd-844df36877e5$0ce66c9d-6d1c-4c2d-8178-5bcdfa247cd6$d9d11d69-bc16-400a-8f46-f9a8ecb8516a$bc8a399b-8864-4473-89d2-e3b0a03d15b5$192cc1cf-9ea1-492d-baa7-f2e197abecd4$6d0925d3-af96-4b94-8e2e-4941cce39e51$786a5385-b648-4fc3-8e19-bf6582828136$b86ee9d3-b6b5-4ea0-8f55-1927571cdfbf$38acd032-1d18-4760-9111-67c9cdd2e892$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2$349631b2-4686-49a9-9f3a-1e4ad588b568$ac9c8845-284d-4c21-b05d-d930f86598a3$b8532822-179b-4cd5-a279-4b71dafb544a$fee14dfe-c5ca-4126-a830-cc9d7eda5433$e524f8cc-ab69-4f8b-a59f-28156696a104$5eb8d9f9-8512-4e00-8cb5-cec68d73cc7d$ff3009eb-23f9-44fe-8e56-85dbc7b463d0$b7f77935-bcab-4ef1-8e1b-a7d059784ff3$6c5e9bb2-4c38-4613-9652-dec99e97b512$f8215517-b18f-4a03-9421-8edab4ca8089$d2729657-d0bf-4d39-8ec7-f242a1ad48d6$8e096fae-9941-49d8-ae87-c68b02f68da5$44f14d4f-7414-4c6f-883a-042ca261a403$6c5f51bb-a6be-447e-b73d-4f9c2885e809$4156d955-9daf-4429-b152-e8332980fb9e$16113560-e911-47b4-abc4-641bbd246454$a7891c63-18d6-4c1f-ba67-adf7c547d334$7126aefd-b847-497a-9545-514e9b9afa71$1894ae1a-bb68-4de0-a4d2-ac5d02c49f09$4c34640f-efa2-4e1d-8a70-0acd2ce45428$f7ede764-5ad8-426b-a805-cc21b622d977$3ea08816-705e-4be7-a175-dbd3f3e4c17d$5d50a5d0-8fe2-4c6e-b76c-d5614e4fd884$0ab70fc3-6188-42eb-aba2-d808f319be9f$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaeb$16fcc2d0-9f2f-4226-9dcc-6d86248cab26$9cf3dc5f-8a25-479f-93db-06e34f0d37a0$cc80848a-6834-4272-9152-e17b45448814$a8b40b8f-051a-4e6f-a079-ece4f32873de$36d514fa-b27a-4c6b-8399-9d108377b9b5$c52c4cec-0ea8-4af3-831a-d284f0e086ee$4c5cb75e-79b5-4502-b1eb-6246e002feaf$8eb42403-1234-4e59-993e-057cc3a6d5c9$71a5fce8-6d9a-4625-bad1-a951d61bff28$b53dba81-a9e9-41da-8fc2-7736bf25f2dc$0d45ae72-572f-4d17-83cf-9814f2854131$0d93132d-5819-47dc-8cf2-462d480d9c3d$602a07dd-8928-4b44-97e5-01c5cbf38351$0574f5a0-72e7-4aa2-80ac-f4ce4f0fe7c2$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7ef$fd58402f-da65-44cf-b81a-e21192fd0e63$af144759-fe66-4ad0-b378-e9eb4e859db4$d4e87ac4-6008-43b2-aa06-e232ec2b2b5b$63fbf8f4-e4e2-4893-be09-67450e92dbd7$fad02876-efba-46a7-9cb7-43820528779f$374af774-3a97-49b5-a3bb-bc3f7f63a3fa$1ce4bc6c-7cde-48e9-8ff1-7281697fd121$f9facbba-39d4-483e-9066-275603156db0$e89bdc84-dbb5-4c73-a39c-6392e5f79704$c0876a48-ea18-494d-8bfc-e2bceb73b417$d82e7ab8-c372-4462-afb5-1617560cdb56$bbc8864a-1545-433f-bc7c-0ddf6e907138$dc2efc6c-8da8-425b-aa5f-290949109565$68469a40-7976-48b7-b7a1-eaa4c5f33a18$d7f6ff79-3c0f-4f16-aa1c-3bc534ce580a$b695ef21-a1ac-4d1f-a0e1-71cd81cede18$a0ca7a5e-0089-4a45-9278-c0f27cd096a0$ba645f6b-143f-4e83-9003-707770ae308d$da3cb392-78f2-48b2-b0dc-5f016664798c$3a37b53d-9174-4faa-9404-74a40c385b0a$ddbca73f-c692-46f2-95f3-a7dd849d33f7$b5319d8b-0420-4ebf-b603-ea0b93365ac1$c87dba8c-9a96-41b3-9dc7-a6c088ec1eaf$cd9c9eeb-c90d-4499-9503-7773d5250f47$5207308e-f636-4d47-b135-036a6e7b8ecd$a6be9a4c-d43b-4867-b7a2-07a46a9d0d8f$f59a5dcd-9f4a-4336-a391-e64af35ef799last_hot_reload_time shortpath%Chapter_13_Policy_Gradient_Methods.jlprocess_statusreadypathٰ/home/runner/work/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions/Chapter-13/Chapter_13_Policy_Gradient_Methods.jlpluto_versionv0.20.8last_save_timeA ycell_order$36a6e43f-6bcf-4c27-bfbb-047760e77ada$31f7e903-30b6-4193-9174-88093e004de4$48dcd2d0-a940-41da-a097-90c780f2ec4d$d95f75b5-21d8-4862-baa7-50b58d9725b8$fc3dcd26-c5cf-4141-bf6c-eaed5fc9bb1d$dcb306ae-a1b1-43d6-ba6e-e38668838689$33c99850-67cd-4754-94b9-6df97b238e27$7a6fb1f0-fc3c-4c29-a6d9-769d32ca98a9$5cc4d12d-b537-47e2-8109-4e7a234fdf25$ff76ef94-fdf5-41f3-a31a-21c4629efabe$f7433324-acc3-49a5-b5b3-ada0c8f09d52$fb8904a9-ae64-41cc-93b6-5a25855edad0$cecc2a35-3850-4f66-84e8-e29da4f3d4b0$6bb0263e-368e-462a-948c-baf9cfa82512$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811$a019925a-460a-410e-a54b-50a4cfe0e90e$f2f2dd1d-180c-4d36-b515-5079d129f93a$e1493cea-19c4-475d-98a0-86d27fb04af1$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72c$5334064b-5a16-4135-afa0-86a48291725b$5981f52b-d829-4c7d-b47b-33310f7d64a2$573878bb-020d-40f6-9329-3d5f91843010$8019bec9-1228-407b-9199-2fe29f26a981$38e5d800-4d43-40d2-87ea-f7d4b4283dab$b94fc99c-f439-4df2-8da3-c01718a136c4$9c342958-1971-48ec-b919-5dfdcbc915a4$e5faaa1b-88cb-43e2-8d04-8972b58b4bda$406638af-1e08-44d2-9ee4-97aa9294a94b$aa450da4-fe84-4eea-b6c4-9820b7982437$fdd3f4fd-4706-4d6b-b150-6ee6b4b370cb$0c9986bb-54c0-4b08-9c29-4bfb0b68b54e$54f559b6-8a62-4a42-894d-c56e41d5ebef$41d62de1-2c92-41ee-9430-b9ca3007afd9$62e677ac-2070-4f6b-9df2-90849d89fa9f$135f205a-f87e-4691-8e87-d317d6312c84$ca360680-afc9-4dd9-9351-493643f91575$4a39f9a7-72d4-44ad-895a-742cd1291f92$9cf3dc5f-8a25-479f-93db-06e34f0d37a0$16fcc2d0-9f2f-4226-9dcc-6d86248cab26$98229733-a71e-44ca-a52a-b7229cf8b422$37a8ef7e-e859-4ef0-81e2-76c02a324031$339b4d2b-2237-46a3-9867-ecc3332856c1$05b0fcad-628b-48d2-aa24-f6f562dbb660$17d07ef4-7c0a-47cc-a701-32c60336571b$76b03e72-da04-4530-8534-6d6468268cbd$2a586e46-66e4-461a-85c8-5817e4d1aa43$90d3b96b-ad2b-405c-951b-f48ec7ccf24a$f924eb30-d1cc-4941-8fb5-ff70ad425ab9$189798b3-ec6b-48b9-918c-ee0f65935ab3$70096b14-beab-4f71-9886-6355c749bb8a$1558cec1-c4fd-4bc0-85ed-ae22c6067d41$e3a2fb12-37ce-4c23-ad93-5fc89991aabb$58403c8e-0ee4-4466-ba25-ee0c86fb0b47$73b90260-d57a-449a-8db6-47f91e6a4e4f$ee72af8d-3cb8-4314-82df-580f068e1252$89901156-b874-416b-89c1-6dc434a4eb17$5c11a92d-7496-4aba-af15-2537eac49dd7$b0a66a19-ee76-463b-a704-8fcee85444d0$581f7e9b-a5c2-4841-9605-85f9585b0274$da2d3186-a778-41cc-9b49-759bf1e9b8fa$f92bb265-4b19-4f0e-a698-d7547bb6dd41$8eab55a5-41b7-4f5e-a02f-4c19388bc9ea$a361f4c9-47ce-42ad-899c-87b611c0d471$cc3ac95e-a398-438a-ba3d-62b6733f6342$4634267b-5dea-4164-8bb2-1eb2fd4d7954$45f0a385-6465-4acc-8637-1b007a0fe215$41dc149d-c6f3-4b0d-a856-06f3aaae3049$042fbafe-2401-4fb7-ac13-4531e0782c79$65d2add6-fd6f-456c-92ed-3cd9d1862ef6$0ac7ea44-14f6-4e80-80f9-d6df8059bb38$96506201-6b66-49e6-8179-06952e2394e1$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290$8e39bd15-862e-4941-88f9-2794b861a523$0e9de19e-bcd4-40ac-9831-afb6cad38422$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091$d037ea92-915c-4dc7-97c6-d006d92e088a$a206c759-3f6e-4003-8cba-5f6ce6742646$0c56b341-24eb-4c78-844e-182f44a7221a$3bafd7df-9bc0-4d13-874d-739590cf3ad9$cc45091e-b889-4d5a-9eef-84d80f792046$d83dc659-dce7-41dd-a8e7-2933ab39d15c$1753b5ed-c00b-4b60-b492-822180778e8c$03a218cb-aa83-4000-85b5-c6f247087053$a893a87b-2d07-4db5-9d1a-9da8646216f4$5c4a383f-fcf2-4f2b-819f-6d84471dda00$2cbc972b-c685-4c1c-8a8d-9d58b197ad90$77cf3a74-899f-4ade-99f2-5aaf7a98c02d$0bf3b988-b3fb-49d5-8dde-b25766596363$a540814a-57a1-4b98-9443-59e401425444$635abb34-2c97-4f04-a74c-22fbec32f408$37ec6802-d4c2-4470-ad69-439d5a732f77$e7566274-5518-4e28-8738-d4b1747d0cfb$4fb83451-b6f8-4e6e-a131-1accc8e10b08$a7c9ae69-f4b8-471c-ab97-90642f3c2bdb$d1ed25e6-60c6-411f-a541-99986e5da2c5$f3e2db06-9cb7-464a-96b8-938175efd26b$e1aec891-d95a-47d1-97d7-d2a4cfb16e64$697b2310-9d96-4f7f-be62-c3bd6bf736f3$8544eddb-2095-4a3c-82e0-920123a88e6d$48b342f2-e48f-457a-9bd3-b3504a79f3a6$f2ed56c9-c2b7-42cb-a083-e12aeaa126ef$cbea5840-49d2-4e91-be9c-f5f15666d78a$fd89433e-643c-474b-b3c4-a997678421a6$5720e942-d3f8-4329-83a8-8bcedf078b6a$cacaaca6-6e01-464f-a2ee-cbf62737a426$1ec1acf1-f833-4478-9b3c-88029340a629$07ad517a-c2ac-4377-99fb-adb13d0f1d0c$aa69e4ea-91e0-496a-a7be-529e67f4dbec$83ca0577-15d7-4448-b597-c77810b812bf$b72e030f-7d52-481f-b4f7-2b16b227e547$a7dcc8cd-04ec-48f2-a387-116330eaffb2$047656d1-2921-40f2-b75b-ce4a87098007$94354552-9920-4b90-98d9-f75286d1f53e$a12b92d1-e045-4f92-b8cd-eee5d56fa67d$44b32cc0-36a8-41fd-89bc-ce894536926c$553b0ceb-f2ca-41ee-99bc-9f53a4487b49$738ada7f-edc7-4ed3-a15e-e92113468738$e5c1aca8-7575-4835-8273-e69ca0a55fe8$ce33f710-fd9d-4dfa-acda-40204e54d518$f4b6f10b-4cd0-4be6-98ec-4d4ffb696392$e7e49ff8-32df-48a4-afb2-462859592e92$4d4ae57b-afc3-44f9-b6fc-892f59f82921$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2$57e5e12a-b722-4ea3-ab3b-e5711029e640$57bbdb10-bed8-459d-8f67-9ea637cf12ba$1386ffdb-940d-4f1b-a872-4e38647b5335$7d63b960-3998-4f7b-8cbb-ccd49db9aeac$9db9ff71-bee9-4bea-a45b-748f8517fed1$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0$646bc853-b7fc-49fa-a201-ff98e8f952d4$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6c$e2b09af1-0f22-4f7f-b806-54fa522adb20$4cbdb082-22ba-49e9-a6ed-4380917625ac$e6cf9550-2e69-4b82-92cf-5e07a35490aa$25be5dcf-be63-46c4-b6de-6cf79fa28fd0$266d2234-26c8-43f1-9e75-49440a230ed6$05bfd818-bf4e-4bda-baa9-5ba647867097$68806899-9972-460a-9f11-daa708a9d610$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54$4fea7232-f286-4a8b-93f8-a0702818ab31$3bccf6fc-6e5e-4f62-ad40-1ff0a3740728$396e0047-d848-462f-a769-0cc2829abc78$1f041cb3-618c-4380-a1ec-d7bbe4a80f62$11ea640c-3981-404d-87c6-4d3d0708a2b8$f8614042-7c94-4d47-a1b6-4e96676b4e8b$bc8a399b-8864-4473-89d2-e3b0a03d15b5$cc80848a-6834-4272-9152-e17b45448814$a8b40b8f-051a-4e6f-a079-ece4f32873de$36d514fa-b27a-4c6b-8399-9d108377b9b5$c52c4cec-0ea8-4af3-831a-d284f0e086ee$d8222abf-139c-4220-8e92-cc987ec6900c$511a847f-234c-465e-8f4a-688e79d9b975$0284f0d7-b8a9-4ae6-add0-ac1078571d9b$b4875f2b-5487-429f-80a3-d1032bbccfc1$4915b1ed-ad53-4ece-9b00-bc136d47d8dc$5b15f5c9-80bf-47f0-898a-f8dead5b927c$83640f5b-fe13-4ec1-98a0-67a56c189ba1$f3bc47b5-03fc-4bd9-a890-26f9608a730b$72273f27-d0b9-4645-a609-cb65cc9332ee$436c52d2-280b-4ca4-9360-d6587b8254c7$f0104778-81a6-417b-8501-f916e5e7f3af$1ac9296f-047b-4051-ba5c-0c23d5f9cde9$fac138d9-3c5d-44b0-a87c-b13872f19450$ba642a22-6623-482a-ab4a-81585b83e457$734573e5-547b-4dcc-89bb-412aa6cc42d6$e96d592d-1e54-486d-8ad9-b857f85476e8$ff4f977e-48df-4c12-845c-c245b4d39d6d$8bc280db-e57d-4e40-be46-1790f4f7d9e7$5aba4f96-e877-457e-8e95-18737348f99f$11063fff-4d36-46d5-828f-dbed0f46b9cf$7afb6fb0-248a-4518-b94f-9876f81eca64$5b15d91e-7119-4f85-a54a-7d4f1fdaf097$7d94922e-dc9f-4953-b539-24aaa2c85b12$42775fd1-5b27-48e0-abf1-9b22bb775e6d$da8d0bca-105b-4d0b-a73d-ee5c9059aeaf$8b35661b-5075-4d63-bc31-044407f99acf$d17a4bd0-5992-4247-912d-73d51758d2f3$352d2952-cb83-47d3-9078-2b2ef9927443$f27f2bcd-05b6-44fe-bf9e-a3e51556db7c$b87ff1a9-abff-40f7-a1d8-f751a1c8b060$5d434c83-c9ca-499f-8695-c7733031c2de$4c4e643b-d4b9-44f0-8d30-dc521bcc55ac$602a07dd-8928-4b44-97e5-01c5cbf38351$7dbb42a3-aa8c-47e5-b668-18e6325d4038$de3cba34-9842-44d1-9b79-47126c0a0751$1b102220-6d78-480d-a77f-0e57bad23dca$37a273b6-b104-46f0-987a-401dc1c97327$8e742d32-c074-4981-b35b-b596b64c869b$b2539398-fdbc-42a2-a8f3-d327358f3643$e034b9cb-f4ee-46f4-bea6-72c93c75d966$64900586-ef92-48e4-839e-ff952a46671b$3c89209c-9202-4d5d-841c-ea34be369616$645e93e7-e92e-49c4-9757-8294fabf4e9b$0cd96c44-cae6-421f-9fae-26141600bef4$19dfabda-7049-4050-8662-0385529c0c5a$0574f5a0-72e7-4aa2-80ac-f4ce4f0fe7c2$966ef17c-23be-49dc-bc37-4cb52b34c049$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62$2c5d221a-2469-49e1-9249-dfdc2457f2fa$5ffc271f-c73f-494a-9727-8d7516af2191$42d4600a-bf3c-45ac-b7f5-d23917713ff5$50ae94c4-70f3-4215-82bd-eb2227c2badf$820752af-8966-4ee8-82f7-a40934522de5$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27$0964133c-3a5b-433b-a8c4-a97813c37583$04b5929a-2058-49c9-963a-96c752a1d67d$64b38d1f-ecf9-4843-89a1-4c8953048265$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fce$6acb549a-5d90-4457-a347-d22448ad8071$fad02876-efba-46a7-9cb7-43820528779f$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7ef$28ce6e60-59cf-408a-8081-b978507b3c72$fd58402f-da65-44cf-b81a-e21192fd0e63$5500fd8e-64cb-4af7-808d-230440746319$a9db3f85-ff56-4bbc-be87-47b893ef3b7b$00152954-dc98-4120-b94b-2ea4d987832b$46fea69b-599e-46ab-8455-d2da865d9a8e$d57375a5-b9e0-4742-b5f7-6a7da891604a$d3c1379f-acd6-4e15-be7e-a5dbe46a4f62$fed4dc4c-0d1c-4ee3-9d0e-8ef2a7db7486$04f42c09-8ab5-4233-b196-51c4aa2dcedb$b02ba928-5b9f-4695-b980-07988c788bb9$98222fcd-b456-477c-90dd-844df36877e5$0ce66c9d-6d1c-4c2d-8178-5bcdfa247cd6$e89bdc84-dbb5-4c73-a39c-6392e5f79704$da3cb392-78f2-48b2-b0dc-5f016664798c$f0962801-0dfa-421f-8ffc-e64068e49913$c251a630-7114-4188-9323-8d8feb5c32e0$c926b6df-c40b-4c4c-8a95-ce9e41feb100$f487f2dd-ad09-48ac-ae34-bf50cfa6ac7d$5d35e515-e2d3-443e-becf-eb28c25db346$cb70d400-3e9c-441c-b17c-e727e8c928f3$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9$10ee7709-0816-48d2-abe0-9be3dd04700f$c0876a48-ea18-494d-8bfc-e2bceb73b417$3a37b53d-9174-4faa-9404-74a40c385b0a$735b548a-88f5-4a30-ab8f-dfb3d6401b2b$60c21e9c-e42d-4f0b-a910-3b318440fbc8$09dd1440-5d09-421f-addc-b1ede43ff517$7ccadf01-fbba-4dfd-a5ad-770dab9946f9$beb01fb8-c77d-4b5c-a66d-3812415e04a3$68e6f17e-8c87-40f0-a673-1115ecd1b71d$692c1043-4eaf-491e-b8fe-368618867f99$3cfd63ad-b1a2-4b99-ae97-2ff10351e4f5$fd964539-2baf-4ff1-b286-5a0bb1b222c4$666a4e89-306b-4fb2-bdc4-3dda2c63153f$0b01ba67-3921-4f3f-a7e8-235190bc84eb$7bf209c8-ef0a-46d1-937e-b1a6e45dc62e$ad0009af-2cfc-4820-bd4a-698ad391f459$b09e1e48-494e-4967-826a-6e70199acad4$5864a5a3-a5a5-43c2-9cb4-7d13b2d20bed$94517664-6988-44dc-a297-e9d5873ee540$b16899b7-36bf-4a5e-8e2f-4496b8450687$00bd2835-b006-4244-9877-bc7e031e3ef8$3e7cecec-eb77-4862-8e3c-b510422e06db$78c83673-2117-4542-b4d8-1c243e8f610b$ae0f5a96-7a4b-47f9-be1e-e803a238a071$c8b47eac-2d45-419a-bec6-2ae0cdc59393$537270ba-122b-4f2b-880b-31d086766295$10cdd16e-a337-4421-a7a0-6de4e4b60c0f$54fff14b-cf53-47b0-9cfa-8b9ee33df54e$76fd79a2-2bc8-45f8-a243-48415118898a$87ee21f3-16ca-4c8c-a0b9-f9e2fd258a91$b966b248-fb4d-457d-90f6-114370846242$f946c886-6246-4f98-a96f-f06984691ad8$bba13634-ff0e-47f7-a23b-8d56098f4ac6$b2082ab0-73a4-45a6-8772-a2e6e22b519a$7a6f3f79-ea06-4994-8b62-90b2056e4034$5261651e-a51e-4e80-8e23-83a4c10e5259$bfe7e41d-6318-4bd4-b892-287831876abc$6bf5ad39-1400-4e1f-a843-a1934b8aaa48$f55afa58-962d-4551-8d95-a5b467d61adf$740a3f41-9302-481d-b373-762c0dea8eff$d41f1dd1-45fe-4456-9a01-ed47fd6704a7$9ae58dd6-3cde-4943-9ac1-bd9d4f7d690c$f545c800-0bf3-491f-9d7d-42341cfdb573$5b868eba-c1af-49f6-8f93-79b78c319a6f$76eb6743-cac0-4174-9ba3-a0691c200b54$ba41f521-4ee2-42a6-bf18-078bfa4b875e$ba5d6311-daee-4abc-b2fb-fae2184ef3eb$ed93259c-7b8b-46d7-97fb-f194e0e04b3a$4e29c621-223e-4859-8e96-db04b967815a$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00$d5020a8d-1dd7-403c-9d1f-665b95543943$2be8a812-4f21-4fe8-a2de-50497db0345a$056a8adc-92f4-4b33-90d9-4b3b4026bbbc$11b9beea-b0cd-45eb-84c6-151728894df0$b71145a4-2614-4f62-bfd2-7d5d1fecec56$4da20fd7-b897-4f26-bf2a-f08d66ddf90f$20776e09-7d9b-4db8-a060-7bceeec65b47$3e3c5897-809f-46e3-bb58-f115b082443e$05f120be-9695-4824-82fd-142a0df13098$717e4c69-59d5-4929-923f-dd35a97fb160$55ba8725-0ddf-4196-a41d-3f3c490a8d84$61949faa-8174-4b7b-8fbc-01d5f850b419$dd8e8cd2-7b41-46c4-8530-adefb7aea684$08505e88-9c23-4e95-91e3-d18bf5133dbc$87482ea5-5265-4e02-92c0-1a8bb44ff0f4$13ebc12f-ff6f-4266-88d3-28d6df5fcf59$3d065608-eef2-4caa-b17d-ec60714e3d58$bd6a7c16-6c25-4fc2-8e1b-4dab693ce19f$c5a2879c-e89b-47f7-bbd6-48200d7e89e3$65be0e58-24be-4932-92a9-9e4825b14144$3c695d54-c30f-4f04-bd40-f5da53be2a95$3c316495-bb6c-41e2-a38f-ba867a319fbb$024dcd1a-8eaa-4a95-8037-2f578828309c$822e4d69-2582-4956-858e-06ecb091e76a$cf1859d6-f889-4923-8c87-2d7c039f26c3$31db0f58-28e4-454f-9394-25565687266f$fddef10c-7695-4596-9e16-987fd45a57e6$26880577-d267-4950-8725-7afe0d0402b6$24fa139c-ad4b-49db-ac8f-23c476ed8608$dddc4a2f-34b2-41dc-85b3-55aba4880fa6$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5$d3b56fca-5b79-4465-8987-8d0005f854d8$5859ca11-90f8-4fd6-88ed-c56efe796fe8$281360af-46bf-4c73-bf11-3cb1153ad3e2$8f1b2db4-ed35-44fc-a3d5-e06deae16d48$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03$8aa16866-bfda-48df-9cf1-cf3d2e203ccb$dca2f8e2-76af-4679-bf81-3824c15fc76d$11a55af7-5301-4507-bb26-88e1e11236db$7856b8a0-565d-4c86-9b3c-4424ff9b86dd$8fcdca63-01a0-4d4b-933c-06a7621d980a$76d54520-baa3-44bf-b303-4cdcb8b87080$9acdbf38-2e10-45ec-85a0-d0db8453a599$61650a97-b353-4a85-b50b-93fee296ac7b$192b9f82-8d3a-408f-91c2-829cfcd32572$d34d22ad-89c2-423e-91dd-bfb895dc6540$5eebf3da-bfe7-46eb-81a3-f87f334ee270$9978d537-49ff-4014-a971-b42704c50a6b$54ff46a2-489a-4dd2-bc30-df70c780cc42$407a0724-4bb6-4c83-ab2d-17a0e19c4072$27487ad0-4779-42ce-8def-e660ef04bee0$9d264543-33ab-498a-90f5-5f913c252484$07ba9fe4-aaa7-4123-9865-cbfa79d0d44a$a4eec4d3-5a75-4b52-ab9c-9d9e83d5547d$374af774-3a97-49b5-a3bb-bc3f7f63a3fa$af144759-fe66-4ad0-b378-e9eb4e859db4$e1274f57-75cb-4659-a82f-e5870c5367e2$63fbf8f4-e4e2-4893-be09-67450e92dbd7$5ee4ce72-7740-4297-8d84-619e0708e4ac$87feff3e-e510-4916-91a9-db3a2cd12225$6b1acb57-159a-4b7f-99fe-5f996522243b$82e0e9a0-9662-429a-87e3-e6bdae02709a$27441783-d3c6-40be-9c36-4941613e6ae9$daf35bfe-8f9c-4f55-971d-4d443be8f8bf$51d6337d-c0bd-40a9-9129-7d88e41e4093$a5b002c9-5e11-462a-9da0-6e060c7963f8$9bce6fdb-2cbc-4758-9a8b-794e490c973d$1ce4bc6c-7cde-48e9-8ff1-7281697fd121$bb1ef180-39ac-475f-beea-ef573e71a3bf$a8349352-3242-46d5-b0d5-1b6eb5d77e90$2e7c737c-c798-4442-a7e1-d74ccfd73119$d4e87ac4-6008-43b2-aa06-e232ec2b2b5b$f7f58fd2-facc-4b87-9172-5e911677c8f4$d21617aa-6f38-4a90-8586-4b32022497ad$700dcbc4-c94c-4287-8cf0-0b2c7a320a3a$4f96be72-ef3e-4e08-ac4c-be4271dcd14c$54f1546d-87ae-49d2-92ed-6fcc9b66e027$c5dd7e99-57e0-4bc7-97d2-2c780b23bcff$2025ff38-f2ec-4224-b771-ff72ffe1af28$77906355-08f8-4b08-b051-84697199b519$023f67b8-8f38-470a-9766-ac60a75678aa$7c592385-e8d3-4efe-962c-d39debb64405$d9d11d69-bc16-400a-8f46-f9a8ecb8516a$192cc1cf-9ea1-492d-baa7-f2e197abecd4$4c5cb75e-79b5-4502-b1eb-6246e002feaf$8eb42403-1234-4e59-993e-057cc3a6d5c9$6d0925d3-af96-4b94-8e2e-4941cce39e51$dc2efc6c-8da8-425b-aa5f-290949109565$ddbca73f-c692-46f2-95f3-a7dd849d33f7$786a5385-b648-4fc3-8e19-bf6582828136$b86ee9d3-b6b5-4ea0-8f55-1927571cdfbf$38acd032-1d18-4760-9111-67c9cdd2e892$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2$349631b2-4686-49a9-9f3a-1e4ad588b568$ac9c8845-284d-4c21-b05d-d930f86598a3$71a5fce8-6d9a-4625-bad1-a951d61bff28$b53dba81-a9e9-41da-8fc2-7736bf25f2dc$b8532822-179b-4cd5-a279-4b71dafb544a$d7f6ff79-3c0f-4f16-aa1c-3bc534ce580a$c87dba8c-9a96-41b3-9dc7-a6c088ec1eaf$fee14dfe-c5ca-4126-a830-cc9d7eda5433$cd9c9eeb-c90d-4499-9503-7773d5250f47$b695ef21-a1ac-4d1f-a0e1-71cd81cede18$e524f8cc-ab69-4f8b-a59f-28156696a104$0d45ae72-572f-4d17-83cf-9814f2854131$0d93132d-5819-47dc-8cf2-462d480d9c3d$5eb8d9f9-8512-4e00-8cb5-cec68d73cc7d$a0ca7a5e-0089-4a45-9278-c0f27cd096a0$5207308e-f636-4d47-b135-036a6e7b8ecd$ff3009eb-23f9-44fe-8e56-85dbc7b463d0$b7f77935-bcab-4ef1-8e1b-a7d059784ff3$6c5e9bb2-4c38-4613-9652-dec99e97b512$f8215517-b18f-4a03-9421-8edab4ca8089$d2729657-d0bf-4d39-8ec7-f242a1ad48d6$8e096fae-9941-49d8-ae87-c68b02f68da5$44f14d4f-7414-4c6f-883a-042ca261a403$6c5f51bb-a6be-447e-b73d-4f9c2885e809$4156d955-9daf-4429-b152-e8332980fb9e$d82e7ab8-c372-4462-afb5-1617560cdb56$a6be9a4c-d43b-4867-b7a2-07a46a9d0d8f$16113560-e911-47b4-abc4-641bbd246454$a7891c63-18d6-4c1f-ba67-adf7c547d334$7126aefd-b847-497a-9545-514e9b9afa71$1894ae1a-bb68-4de0-a4d2-ac5d02c49f09$f9facbba-39d4-483e-9066-275603156db0$bbc8864a-1545-433f-bc7c-0ddf6e907138$68469a40-7976-48b7-b7a1-eaa4c5f33a18$ba645f6b-143f-4e83-9003-707770ae308d$b5319d8b-0420-4ebf-b603-ea0b93365ac1$4c34640f-efa2-4e1d-8a70-0acd2ce45428$f7ede764-5ad8-426b-a805-cc21b622d977$3ea08816-705e-4be7-a175-dbd3f3e4c17d$5d50a5d0-8fe2-4c6e-b76c-d5614e4fd884$0ab70fc3-6188-42eb-aba2-d808f319be9f$df7f84e8-b42a-4001-9dbf-6bc3ced94207$d963ff6d-f1b6-4799-aa0e-1ae100310d84$7cf26604-9c2b-4a77-9674-7d4dac2f99f0$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaeb$f59a5dcd-9f4a-4336-a391-e64af35ef799published_objectsnbpkginstall_time_ns instantiatedòinstalled_versions StatisticsstdlibTransducers0.4.84StatsBase0.33.21Memoize0.4.4Distributions0.25.117PlutoUI0.7.61ProgressLogging0.1.4RandomstdlibBenchmarkTools1.3.2StaticArrays1.5.26LinearAlgebrastdlibPlutoDevMacros0.9.0DataFrames1.7.0PlutoProfile0.4.0HypertextLiteral0.9.5SpecialFunctions2.5.0LaTeXStrings1.3.1PlutoPlotly0.3.9terminal_outputs Statistics
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mTransducers
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mMemoize
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mProgressLogging
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mDistributions
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mPlutoUI
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mRandom
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mBenchmarkTools
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mLinearAlgebra
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mPlutoProfile
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mLaTeXStrings
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mStatsBase
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mApproximationUtils
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mStaticArrays
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mBase
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mnbpkg_sync
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mPlutoDevMacros
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mSpecialFunctions
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mHypertextLiteral
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mDataFrames
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39mPlutoPlotly
[0m[1mResolving...[22m
[90m===[39m
[32m[1m Installed[22m[39m PDMats ────────────────── v0.11.32
Installed Crayons ───────────────── v4.1.1 [32m[1m[22m[39m
Installed HypergeometricFunctions ─ v0.3.27 [32m[1m[22m[39m
Installed Accessors ─────────────── v0.1.41 [32m[1m[22m[39m
Installed PlotlyBase ────────────── v0.8.20 [32m[1m[22m[39m
Installed SpecialFunctions ──────── v2.5.0 [32m[1m[22m[39m
Installed PrettyTables ──────────── v2.4.0 [32m[1m[22m[39m
Installed MIMEs ─────────────────── v1.0.0 [32m[1m[22m[39m
Installed JuliaInterpreter ──────── v0.9.41 [32m[1m[22m[39m
Installed InvertedIndices ───────── v1.3.1 [32m[1m[22m[39m
Installed PlutoUI ───────────────── v0.7.61 [32m[1m[22m[39m
Installed StaticArrays ──────────── v1.5.26 [32m[1m[22m[39m
Installed DataFrames ────────────── v1.7.0 [32m[1m[22m[39m
Installed Memoize ───────────────── v0.4.4 [32m[1m[22m[39m
Installed Distributions ─────────── v0.25.117 [32m[1m[22m[39m
Installed StringManipulation ────── v0.4.1 [32m[1m[22m[39m
Installed MacroTools ────────────── v0.5.15 [32m[1m[22m[39m
No Changes to `/tmp/jl_E3c5X0/Project.toml` [32m[1m[22m[39m
[32m[1m No Changes[22m[39m to `/tmp/jl_E3c5X0/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1m Activating[22m[39m project at `/tmp/jl_E3c5X0`
[92m[1mPrecompiling[22m[39m project...
47 dependencies successfully precompiled in 90 seconds. 108 already precompiled.
[33m1[39m dependency had output during precompilation:[33m
┌ [39mPlutoPlotly[33m
│ [39m┌ Warning: You are trying to show a PlutoPlot outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly.[33m
│ [39m│ NOTE: If you receive this warning during pre-compilation or sysimage creation, you can ignore this warning.[33m
│ [39m└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/5DpMg/notebooks/wrapper.jl:43[33m
└ [39menabled÷restart_recommended_msgrestart_required_msgbusy_packageswaiting_for_permission,waiting_for_permission_but_probably_disabled«cell_inputs$4f96be72-ef3e-4e08-ac4c-be4271dcd14ccell_id$4f96be72-ef3e-4e08-ac4c-be4271dcd14ccodemetadatashow_logsèdisabled®skip_as_script«code_folded$19dfabda-7049-4050-8662-0385529c0c5acell_id$19dfabda-7049-4050-8662-0385529c0c5acode@bind sref_cartpole_binary PlutoUI.combine() do Child
md"""
x position: $(Child(:x, Slider(-50f0:50f0, default = 0f0, show_value=true)))
x velocity: $(Child(:ẋ, Slider(-50f0:50f0, default = 0f0, show_value=true)))
"""
end |> confirmmetadatashow_logsèdisabled®skip_as_script«code_folded$b71145a4-2614-4f62-bfd2-7d5d1fecec56cell_id$b71145a4-2614-4f62-bfd2-7d5d1fecec56code
#version of reinforce for general function approximation
function actor_critic_with_eligibility_traces!(policy_params::P1, ∇lnπ, value_params::P2, ∇v̂, mdp::ContinuousMDP{T, S, A, PTF, F1, F2, F3}, λ_θ::T, λ_w::T, update_action_distribution!::Function, action_dist_params::Vector{T}, action_sampler::Function, update_eligibility_vector!::Function, x, update_feature_vector!::Function, value_function::Function, update_value_gradient!::Function, max_episodes::Integer, max_steps::Integer; α_w::T = one(T)/10, α_θ::T = one(T)/10, γ::T = one(T), z_θ::P1 = deepcopy(policy_params), z_w::P2 = deepcopy(value_params), save_step_rewards = false) where {P1, P2, T<:Real, S, A, PTF, F1, F2, F3}
step_rewards = Vector{T}()
episode_steps = Vector{Int64}()
episode_rewards = Vector{T}()
#initialize variables
ep = 1
step = 1
rtot = zero(T)
c = one(T)
zero_params!(z_θ)
zero_params!(z_w)
s = mdp.initialize_state()
update_feature_vector!(x, s)
while (ep <= max_episodes) && (step <= max_steps)
update_value_gradient!(∇v̂, x, value_params)
v̂ = value_function(x, value_params)
update_action_distribution!(action_dist_params, x, policy_params)
a = action_sampler(action_dist_params)
if bad_continuous_action(a)
@info "terminating after $step steps and episode $ep due to invalid continuous action $a taken in state $s with action distribution parameters $action_dist_params"
push!(episode_steps, max_steps)
push!(episode_rewards, typemin(T))
break
end
update_eligibility_vector!(∇lnπ, action_dist_params, x, a, policy_params)
(r, s′) = mdp.ptf(s, a)
rtot += r
save_step_rewards && push!(step_rewards, r)
step += 1
if mdp.isterm(s′)
push!(episode_steps, step)
push!(episode_rewards, rtot)
v̂′ = zero(T)
rtot = zero(T)
zero_params!(z_θ)
zero_params!(z_w)
ep += 1
c = one(T)
s = mdp.initialize_state()
update_feature_vector!(x, s)
else
update_feature_vector!(x, s′)
v̂′ = value_function(x, value_params)
s = s′
c *= γ
end
δ = r + γ*v̂′ - v̂
update_traces_with_gradient!(γ*λ_w, z_w, ∇v̂)
update_traces_with_gradient!(γ*λ_θ, z_θ, c, ∇lnπ)
update_params_with_gradient!(value_params, α_w*δ, z_w)
update_params_with_gradient!(policy_params, α_θ*c*δ, z_θ)
end
function_outputs = form_state_and_policy_function_outputs(update_feature_vector!, update_action_distribution!, action_dist_params, action_sampler, value_function, x, policy_params, value_params)
return (;step_rewards = step_rewards, episode_steps = episode_steps, episode_rewards = episode_rewards, policy_parameters = policy_params, value_parameters = value_params, function_outputs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$c0876a48-ea18-494d-8bfc-e2bceb73b417cell_id$c0876a48-ea18-494d-8bfc-e2bceb73b417codeهplot_mountaincar_values(mountaincar_continuing_fcann_test.estimate_state_value, mountaincar_continuing_fcann_test.policy_sample_action)metadatashow_logsèdisabled®skip_as_script«code_folded$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091cell_id$1d36ae81-d3da-45c0-bbcf-0b6e0e80b091codefunction reinforce_monte_carlo_control_fcann(mdp::StateMDP{T, S, A, P, F1, F2, F3}, input_length::Integer, hidden_layers::Vector{Int64}, update_feature_vector!::Function,max_episodes::Integer; params::FCANNParams = FCANN.initializeparams_saxe(input_length, hidden_layers, length(mdp.actions)), reslayers = 0, l2 = 0f0, dropout = 0f0, use_μP = true, activation_list = fill(true, length(hidden_layers)), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
setup = setup_fcann_policy_arguments(params, input_length, hidden_layers, reslayers, l2, dropout, use_μP, activation_list)
reinforce_monte_carlo_control!(params, setup.eligibility_vector, mdp, setup.update_action_preferences!, setup.update_eligibility_vector!, setup.feature_vector, update_feature_vector!, max_episodes; kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$f4b6f10b-4cd0-4be6-98ec-4d4ffb696392cell_id$f4b6f10b-4cd0-4be6-98ec-4d4ffb696392code4md"""
### *One-step Actor-Critic Implementation*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$9db9ff71-bee9-4bea-a45b-748f8517fed1cell_id$9db9ff71-bee9-4bea-a45b-748f8517fed1codeٺone_step_actor_critic_linear_features(corridor_mdp, update_corridor_features!, 1, typemax(Int64), 100_000, α_θ = 2f0^-8, α_w = 2f0^-8, policy_params = [0f0 3.7f0]).policy_and_value(1)metadatashow_logsèdisabled®skip_as_script«code_folded$4634267b-5dea-4164-8bb2-1eb2fd4d7954cell_id$4634267b-5dea-4164-8bb2-1eb2fd4d7954codefunction update_linear_eligibility_vector!(∇lnπ::Matrix{T}, action_preferences::Vector{T}, x::Vector{T}, i_a::Integer, params::Matrix{T}) where T<:AbstractFloat
update_linear_action_preferences!(action_preferences, x, params)
soft_max!(action_preferences)
BLAS.gemm!('N', 'T', -one(T), x, action_preferences, zero(T), ∇lnπ)
@inbounds @simd for i in eachindex(x)
∇lnπ[i, i_a] += x[i]
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$6c5f51bb-a6be-447e-b73d-4f9c2885e809cell_id$6c5f51bb-a6be-447e-b73d-4f9c2885e809codeactor_critic_binary_episodic_beta_parameter_study(mountaincar_continuous_mdp, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, mountaincar_binary_continuous_params2, 5, 3, 10000; max_steps = 100_000)metadatashow_logsèdisabledîskip_as_script«code_folded$cc45091e-b889-4d5a-9eef-84d80f792046cell_id$cc45091e-b889-4d5a-9eef-84d80f792046code md"""
## 13.4 REINFORCE with Baseline
The policy gradient theorem (13.5) can be generalized to include a comparison of the action value to an arbitrary *baseline* b(s):
$\nabla J(\boldsymbol{\theta}) \propto \sum_s \mu(s)\sum_a\left( q_\pi(s,a)-b(s) \right ) \nabla\pi(a|s,\boldsymbol{\theta}) \tag{13.10}$
The baseline can be any function, even a random variable, as long as it does not vary with $a$; the euation remains valid because the subtracted quantity is zero:
$\sum_ab(s)\nabla\pi(a|s,\boldsymbol{\theta})=b(s)\nabla\sum_a\pi(a|s,\boldsymbol{\theta})=b(s)\nabla1=0$
The policy gradient theorem with baseline (13.10) can be used to derive an update rule using similar steps as in the previous section. The update rule that we end up with is a new version of REINFORCE that includes a general baseline:
$\boldsymbol{\theta}_{t+1} \doteq \boldsymbol{\theta}_t+\alpha(G_t-b(S_t))\frac{\nabla\pi(A_t|S_t,\boldsymbol{\theta}_t)}{\pi(A_t|S_t,\boldsymbol{\theta}_t)} \tag{13.11}$
Since the baseline could be uniformly zero, this is a strict generalization of REINFORCE. To have an effective baseline that depends on state we can use a state value estimate that is also updated with gradient steps: $\hat v(S_t, \mathbf{w})$. Using such an estimate we can revise the previous REINFORCE algorithm.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$5b15d91e-7119-4f85-a54a-7d4f1fdaf097cell_id$5b15d91e-7119-4f85-a54a-7d4f1fdaf097codefunction create_actor_critic_continuing_params_UI(;λ_θ = 0.5f0, λ_w = 0.5f0, log2α_θ = -10, log2α_w = -10, α_r̄ = 0.005f0)
PlutoUI.combine() do Child
md"""
$$\lambda_\theta$$: $(Child(:λ_θ, Slider(0.00f0:0.001f0:.999f0, default = λ_θ, show_value=true)))
$$\lambda_\mathbf{w}$$: $(Child(:λ_w, Slider(0.00f0:0.001f0:.999f0, default = λ_w, show_value=true)))
$$\alpha_{\overline{r}}$$: $(Child(:α_r̄, NumberField(0.00f0:0.001f0:1f0, default = α_r̄)))
$$\log_2 \alpha_\theta$$ min: $(Child(:α_θ_min, NumberField(-100:0, default = log2α_θ)))
$$\log_2 \alpha_{\mathbf{w}}$$ min: $(Child(:α_w_min, NumberField(-100:0, default = log2α_w)))
"""
end |> confirm
endmetadatashow_logsèdisabled®skip_as_script«code_folded$ba41f521-4ee2-42a6-bf18-078bfa4b875ecell_id$ba41f521-4ee2-42a6-bf18-078bfa4b875ecodebegin
make_n_param_dist_policy_params(n::Integer, num_features::Integer, ::T) where T<:Real = zeros(T, num_features, n)
make_n_param_dist_policy_params(n::Integer, num_features::Integer, ::NTuple{N, T}) where {N, T<:Real} = zeros(T, num_features, n*N)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03cell_id$d41f0dc4-15ac-4f8f-acb5-a7ccd8d48f03codeYfunction cartpole_tilecoding_reinforce_parameter_study(α1_list, α2_list, max_episodes; num_trials = 100, kwargs...)
setup = setup_cartpole_problem(;kwargs...)
traces = [begin
steps = [begin
1:num_trials |> Map() do i
solution = reinforce_with_baseline_monte_carlo_control_binary_features(cartpole_setup.mdps.episodic.discrete, cartpole_setup.get_active_features, cartpole_setup.num_features, max_episodes; α_θ = α1, α_w = α2)
steps = solution.episode_steps
isempty(steps) && return max_steps
mean(steps)
end |> foldxt(+) |> x -> x / num_trials
end
for α1 in α1_list]
scatter(x = α1_list, y = steps, name = "α_w = $α2")
end
for α2 in α2_list]
plot(traces, Layout(xaxis_title = "Policy Learning Rate α_θ", yaxis_title = "Average Episode Duration Over First $max_episodes Episodes", xaxis_type = "log"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$3c695d54-c30f-4f04-bd40-f5da53be2a95cell_id$3c695d54-c30f-4f04-bd40-f5da53be2a95code/md"""
### *Cart Pole Continuous Action MDP*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$0d45ae72-572f-4d17-83cf-9814f2854131cell_id$0d45ae72-572f-4d17-83cf-9814f2854131codeg@bind mountaincar_binary_continuous_params2 create_actor_critic_params_UI(λ_θ = 0.05f0, λ_w = 0.8f0)metadatashow_logsèdisabled®skip_as_script«code_folded$cd9c9eeb-c90d-4499-9503-7773d5250f47cell_id$cd9c9eeb-c90d-4499-9503-7773d5250f47codeىshow_mountaincar_continuous_trajectory(mountaincar_continuous_test_train2.policy_sample_action, 1_000; mdp = mountaincar_continuous_mdp2)metadatashow_logsèdisabled®skip_as_script«code_folded$fd58402f-da65-44cf-b81a-e21192fd0e63cell_id$fd58402f-da65-44cf-b81a-e21192fd0e63codeمplot_cartpole_policy(cartpole_continuing_fcann_test.policy_and_value; s_ref = CartPoleState(cartpole_fcann_continuing_test_state...))metadatashow_logsèdisabled®skip_as_script«code_folded$8e39bd15-862e-4941-88f9-2794b861a523cell_id$8e39bd15-862e-4941-88f9-2794b861a523codereinforce_monte_carlo_control_linear_features(mdp::StateMDP{T, S, A, P, F1, F2, F3}, update_feature_vector!::Function, num_features::Integer, max_episodes::Integer; params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), x::Vector{T} = zeros(T, num_features), ∇lnπ::Matrix{T} = copy(params), kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = reinforce_monte_carlo_control!(params, ∇lnπ, mdp, update_linear_action_preferences!, update_linear_eligibility_vector!, x, update_feature_vector!, max_episodes; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$64900586-ef92-48e4-839e-ff952a46671bcell_id$64900586-ef92-48e4-839e-ff952a46671bcodetest_study = actor_critic_linear_parameter_study(cartpole_continuing_mdp, s -> cartpole_tilecoding_setup.get_active_features((s.x, s.θ, s.ẋ, s.θ̇)), cartpole_tilecoding_setup.num_features, LinRange(.5f0, .95f0, 10), LinRange(0.5f0, .95f0, 10), [0.005f0, 0.01f0, 0.05f0], 2f0 .^ (-5:-1), 2f0 .^ (-10:-5), 100, 10_000; nruns = 40, seed = 45, binary_features = true) |> df -> sort(df, :output; rev=true)metadatashow_logsèdisabledîskip_as_script«code_folded$fddef10c-7695-4596-9e16-987fd45a57e6cell_id$fddef10c-7695-4596-9e16-987fd45a57e6codefunction setup_cartpole_continuous_problem(;h = 4f-2, f = 300f0, x_max = 50f0, θ_max = deg2rad(70f0), ẋ_max = 50f0, θ̇_max = 10f0, num_tiles = (8, 8, 8, 8), num_tilings = 8, kwargs...)
tile_size = Tuple(1f0 / n for n in num_tiles)
min_vals = (-x_max, -θ_max, -ẋ_max, -θ̇_max)
max_vals = (x_max, θ_max, ẋ_max, θ̇_max)
setup = tile_coding_setup(min_vals, max_vals, tile_size, num_tilings, (1, 3, 5, 7))
init_θ() = rand([-0.05f0, 0.05f0])
mdps = create_cartpole_mdps(h = h, fmax = f, x_max = x_max, θ_max = θ_max, ẋ_max = ẋ_max, θ̇_max = θ̇_max, init_θ = init_θ, kwargs...)
(mdps = mdps, get_active_features = s -> setup.get_active_features((s.x, s.θ, s.ẋ, s.θ̇)), num_features = setup.num_features, min_vals = min_vals, max_vals = max_vals)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$e2b09af1-0f22-4f7f-b806-54fa522adb20cell_id$e2b09af1-0f22-4f7f-b806-54fa522adb20code#note that due to the feature construction in this problem, the bootstrapping estimate is worthless so we'd expect this to do poorly at this task. Imagine that the value function starts out perfectly accurate for the initial policy with bad parameterization which we know finishes episodes in 90 to 100 steps. Then the δ on each step is just -1. Given the policy initialization, this is very poor progress since the improvement is only towards something still with worse performance than the completely random policymetadatashow_logsèdisabled®skip_as_script«code_folded$2be8a812-4f21-4fe8-a2de-50497db0345acell_id$2be8a812-4f21-4fe8-a2de-50497db0345acodeHmd"""
### *Actor-Critic Implementation for Continuous Action Spaces*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$68806899-9972-460a-9f11-daa708a9d610cell_id$68806899-9972-460a-9f11-daa708a9d610codeactor_critic_with_eligibility_traces_linear_features(mdp::StateMDP{T, S, A, P, F1, F2, F3}, λ_θ::T, λ_w::T, update_feature_vector!::Function, num_features::Integer, args...; policy_params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), value_params::Vector{T} = zeros(T, num_features), x = zeros(T, num_features), action_preferences = zeros(T, length(mdp.actions)), kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = actor_critic_with_eligibility_traces!(policy_params, copy(policy_params), value_params, copy(value_params), mdp, λ_θ, λ_w, update_linear_action_preferences!, update_linear_eligibility_vector!, x, update_feature_vector!, linear_value_function, update_linear_value_gradient!, args...; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$189798b3-ec6b-48b9-918c-ee0f65935ab3cell_id$189798b3-ec6b-48b9-918c-ee0f65935ab3codemd"""
> ### *Exercise 13.3*
> In Section 13.1 we considered policy parameterizations using the soft-max in action preferences (13.2) with linear action preferences (13.3). For this parameterization, prove that the eligibility vector is
> $\begin{flalign}
> \nabla \ln \pi(a|s, \boldsymbol{\theta}) = \mathbf{x}(s, a) - \sum_b \pi(b|s, \boldsymbol{\theta}) \mathbf{x}(s, b) \tag{13.9}
> \end{flalign}$
> using the definitions and elementary calculus.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$00152954-dc98-4120-b94b-2ea4d987832bcell_id$00152954-dc98-4120-b94b-2ea4d987832bcodefunction create_mountaincar_continuing_mdp()
ptf = StateMDPTransitionSampler(mountaincar_continuing_step, (0f0, 0f0))
StateMDP(MountainCarTask.actions, ptf, MountainCarTask.initialize_state)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$42d4600a-bf3c-45ac-b7f5-d23917713ff5cell_id$42d4600a-bf3c-45ac-b7f5-d23917713ff5code@bind cartpole_continuing_fcann_network_params PlutoUI.combine() do Child
md"""
Layer Size: $(Child(NumberField(1:128, default = 4)))
Num Layers: $(Child(NumberField(1:10, default = 2)))
"""
end |> confirmmetadatashow_logsèdisabled®skip_as_script«code_folded$4e29c621-223e-4859-8e96-db04b967815acell_id$4e29c621-223e-4859-8e96-db04b967815acodefunction setup_binary_squashed_gaussian_policy_arguments(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, amax::A, get_active_features::Function, num_features::Integer) where {T<:Real, S, N, A<:Union{T, NTuple{N, T}}, P, F1, F2, F3}
x = BinaryFeatureVector()
update_feature_vector!(x::BinaryFeatureVector, s) = update_binary_feature_vector!(x, s, get_active_features)
sample_action = rand(A)
action_dist_params = make_n_param_dist_params(2, sample_action)
∇lnπ = BinarySquashedGaussianEligibilityVector(sample_action, amax)
return (feature_vector = x, update_feature_vector! = update_feature_vector!, action_distribution_parameters = action_dist_params, eligibility_vector = ∇lnπ)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$5981f52b-d829-4c7d-b47b-33310f7d64a2cell_id$5981f52b-d829-4c7d-b47b-33310f7d64a2codeRmake_ϵ_greedy_policy!(corridor_train.value_function(1).action_values; ϵ = 0.0f0)metadatashow_logsèdisabled®skip_as_script«code_folded$0e9de19e-bcd4-40ac-9831-afb6cad38422cell_id$0e9de19e-bcd4-40ac-9831-afb6cad38422codefunction setup_fcann_policy_arguments(params::FCANNParams{T}, input_length::Integer, hidden_layers::Vector{Int64}, reslayers::Integer, l2::T, dropout::T, use_μP::Bool, activation_list) where {T<:Real}
x = zeros(T, input_length)
activations = FCANN.form_activations(params[1])
tanh_grad_z = deepcopy(activations)
deltas = deepcopy(activations)
scales = fill(one(T), length(params[1]))
if use_μP
for i in eachindex(hidden_layers)
i′ = i + 1
scales[i′] /= size(params[1][i′], 2)
end
end
∇lnπ = deepcopy(params)
update_eligibility_vector!(∇lnπ::FCANNParams, action_preferences::Vector{T}, x, i_a, params::FCANNParams) = update_fcann_eligibility_vector!(∇lnπ, action_preferences, x, i_a, params, hidden_layers, l2, tanh_grad_z, activations, deltas, dropout, reslayers, activation_list, scales)
update_action_preferences!(action_preferences::Vector{T}, x::Vector{T}, params::FCANNParams) = update_fcann_action_preferences!(action_preferences, x, params, activations, reslayers)
return (feature_vector = x, params = params, eligibility_vector = ∇lnπ, update_eligibility_vector! = update_eligibility_vector!, update_action_preferences! = update_action_preferences!, scales = scales)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$ff3009eb-23f9-44fe-8e56-85dbc7b463d0cell_id$ff3009eb-23f9-44fe-8e56-85dbc7b463d0codeufunction show_squashed_policy(π::Function, s)
pdist=π(s)
plot_squashed_gaussian(pdist[1], exp(pdist[2]), 1f0)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$4fb83451-b6f8-4e6e-a131-1accc8e10b08cell_id$4fb83451-b6f8-4e6e-a131-1accc8e10b08code
B#version of reinforce for general function approximation
function reinforce_with_baseline_monte_carlo_control!(policy_params, ∇lnπ, value_params, ∇v̂, mdp::StateMDP{T, S, A, PTF, F1, F2, F3}, update_action_preferences!::Function, update_eligibility_vector!::Function, x, update_feature_vector!::Function, value_function::Function, update_value_gradient!::Function, max_episodes::Integer; α_w::T = one(T)/10, α_θ::T = one(T)/10, γ::T = one(T), action_preferences = zeros(T, length(mdp.actions)), epkwargs...) where {T<:Real, S, A, PTF, F1, F2, F3}
rewards = zeros(T, max_episodes)
steps = zeros(Int64, max_episodes)
π! = form_state_policy_function(update_feature_vector!, update_action_preferences!)
π(s) = π!(x, action_preferences, s, policy_params)
π_sample(s) = sample_action(π(s))
v! = form_state_value_function(update_feature_vector!, value_function)
estimate_state_value(s) = v!(x, s, value_params)
state_history, action_history, reward_history, _, _ = runepisode(mdp; π = π_sample, max_steps = 0) #initialize variables to update episodes
for i in eachindex(rewards)
# @info "On episode $i of $max_episodes"
state_history, action_history, reward_history, sterm, nsteps = runepisode!((state_history, action_history, reward_history), mdp; π = π_sample, epkwargs...)
g = zero(T)
rtotal = zero(T)
#iterate through episode beginning at the end
for i in nsteps:-1:1
g = (γ * g) + reward_history[i]
update_feature_vector!(x, state_history[i])
v̂ = value_function(x, value_params)
δ = g - v̂
update_value_gradient!(∇v̂, x, value_params)
c = α_w*δ
update_params_with_gradient!(value_params, c, ∇v̂)
update_eligibility_vector!(∇lnπ, action_preferences, x, action_history[i], policy_params)
c = α_θ * γ^(i-1) * δ
update_params_with_gradient!(policy_params, c, ∇lnπ)
rtotal += reward_history[i]
end
rewards[i] = rtotal
steps[i] = nsteps
end
π2(s; feature_vector = deepcopy(x), action_preferences = copy(action_preferences)) = π!(feature_vector, action_preferences, s, policy_params)
π_sample2(s; kwargs...) = sample_action(π2(s; kwargs...))
function policy_and_value(s::S)
π!(x, action_preferences, s, policy_params)
v̂ = value_function(x, value_params)
return (action_probabilities = action_preferences, state_value_estimate = v̂)
end
return (episode_rewards = rewards, episode_steps = steps, policy_function = π2, policy_sample_action = π_sample2, policy_parameters = policy_params, estimate_state_value = estimate_state_value, value_parameters = value_params, policy_and_value = policy_and_value)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$406638af-1e08-44d2-9ee4-97aa9294a94bcell_id$406638af-1e08-44d2-9ee4-97aa9294a94bcode-md"""
## 13.2 The Policy Gradient Theorem
"""metadatashow_logsèdisabled®skip_as_script«code_folded$57e5e12a-b722-4ea3-ab3b-e5711029e640cell_id$57e5e12a-b722-4ea3-ab3b-e5711029e640codeone_step_actor_critic_linear_features(mdp::StateMDP{T, S, A, P, F1, F2, F3}, update_feature_vector!::Function, num_features::Integer, max_episodes::Integer, max_steps::Integer; policy_params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), value_params::Vector{T} = zeros(T, num_features), x = zeros(T, num_features), action_preferences = zeros(T, length(mdp.actions)), kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = one_step_actor_critic!(policy_params, copy(policy_params), value_params, copy(value_params), mdp, update_linear_action_preferences!, update_linear_eligibility_vector!, x, update_feature_vector!, linear_value_function, update_linear_value_gradient!, max_episodes, max_steps; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$374af774-3a97-49b5-a3bb-bc3f7f63a3facell_id$374af774-3a97-49b5-a3bb-bc3f7f63a3facode)plot_cart(ep[1][ep_step], ep[2][ep_step])metadatashow_logsèdisabled®skip_as_script«code_folded$7bf209c8-ef0a-46d1-937e-b1a6e45dc62ecell_id$7bf209c8-ef0a-46d1-937e-b1a6e45dc62ecode٨@bind beta_params PlutoUI.combine() do Child
md"""
α: $(Child(Slider(0.01:0.1:100; show_value=true)))
β: $(Child(Slider(0.01:0.1:100, show_value=true)))
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$dd8e8cd2-7b41-46c4-8530-adefb7aea684cell_id$dd8e8cd2-7b41-46c4-8530-adefb7aea684codegfunction actor_critic_binary_episodic_beta_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, λ_θ::T, λ_w::T, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, max_episodes::Integer; nruns::Integer = 100, max_steps::Integer = 10_000, seed = rand(UInt64), init_policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), init_value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
Random.seed!(seed)
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> actor_critic_with_eligibility_traces_binary_features_beta_actions(mdp, λ_θ, λ_w, get_active_features, num_features, max_episodes, max_steps; α_θ = α_θ, α_w = α_w, policy_params = copy(init_policy_params), value_params = copy(init_value_params), kwargs...) |> x -> isempty(x.episode_rewards) ? -T(Inf) : mean(x.episode_rewards)) |> foldxt(+) |> x -> x / nruns
end
traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Episode in the First $max_episodes Episodes Averaged Over $nruns Runs", xaxis_type = "log", title = "Binary Feature Encoding with $num_features Features, λ_θ = $λ_θ, λ_w = $λ_w"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$4fea7232-f286-4a8b-93f8-a0702818ab31cell_id$4fea7232-f286-4a8b-93f8-a0702818ab31code8md"""
#### Test Actor-Critic with Eligibility Traces
"""metadatashow_logsèdisabled®skip_as_script«code_folded$26880577-d267-4950-8725-7afe0d0402b6cell_id$26880577-d267-4950-8725-7afe0d0402b6code:const cartpole_setup = setup_cartpole_continuous_problem()metadatashow_logsèdisabled®skip_as_script«code_folded$a7891c63-18d6-4c1f-ba67-adf7c547d334cell_id$a7891c63-18d6-4c1f-ba67-adf7c547d334codeُ@bind fcann_mountaincar_study_params create_actor_critic_fcann_params_UI(;λ_θ = 0.5f0, λ_w = 0.5f0, h = 16, log2α_θ = -10, log2α_w = -11)metadatashow_logsèdisabledîskip_as_script«code_folded$44f14d4f-7414-4c6f-883a-042ca261a403cell_id$44f14d4f-7414-4c6f-883a-042ca261a403codeK@bind mountaincar_binary_continuous_params2 create_actor_critic_params_UI()metadatashow_logsèdisabledîskip_as_script«code_folded$94354552-9920-4b90-98d9-f75286d1f53ecell_id$94354552-9920-4b90-98d9-f75286d1f53ecode`corridor_parameter_studies(1.5f0 .^(-24:-20), 1.25f0 .^ (-27:-20), 2f0 .^(-3:-1); nruns = 1_000)metadatashow_logsèdisabled®skip_as_script«code_folded$e5faaa1b-88cb-43e2-8d04-8972b58b4bdacell_id$e5faaa1b-88cb-43e2-8d04-8972b58b4bdacodebegin
v1(p) = -2*(1+p)/((1-p)*p)
v2(p) = -(p+2)/((1-p)*p)
v3(p) = -3/(1-p)
plist = 0.:0.001:1.
traces = [scatter(x = plist, y = f.(1 .- plist), name = n) for (f, n) in zip([v1, v2, v3], ["V(S1)", "V(S2)", "V(S3)"])]
plot(traces, Layout(font_color = "LightGray", plot_bgcolor = bgcolor, paper_bgcolor = "rgb(40, 40, 40)", yaxis_range = [-100, 0], xaxis_title = "probability of right action", yaxis_title = "State Value", width = 900, height = 600))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$70096b14-beab-4f71-9886-6355c749bb8acell_id$70096b14-beab-4f71-9886-6355c749bb8acodemd"""
We previously derived an expression for the gradient of the policy itself in the case of linear action preferences:
$\begin{flalign}
h_a &= \boldsymbol{\theta}^\top \mathbf{x}(s, a) \\
\pi_a &= \frac{e^{h_a}}{\sum_k e^{h_k}} \\
\nabla(\pi_a)_i &= \pi_a \left ( \mathbf{x}(s, a)_i - \sum_k \pi_k \mathbf{x}(s, k)_i \right)
\end{flalign}$
Applying the chain rule to the natural logarithm produces:
$\nabla \left ( \ln f(\theta) \right) = \frac{\nabla f(\theta)}{f(\theta)} \implies \nabla \left ( \ln f(\theta) \right )_i = \frac{\nabla \left ( f(\theta) \right )_i}{f(\theta)}$
Applying this to the above expression yields:
$\begin{flalign}
\nabla \left ( \ln \pi_a \right )_i &= \frac{\nabla \left ( \pi_a \right )_i}{\pi_a} \\
&= \frac{\pi_a \left ( \mathbf{x}(s, a)_i - \sum_k \pi_k \mathbf{x}(s, k)_i \right)}{\pi_a} \\
&= \mathbf{x}(s, a)_i - \sum_k \pi_k \mathbf{x}(s, k)_i
\end{flalign}$
which is the per component version of the desired vector expression.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$90d3b96b-ad2b-405c-951b-f48ec7ccf24acell_id$90d3b96b-ad2b-405c-951b-f48ec7ccf24acodemd"""
The final expected value expression (13.5) can be sampled on a step by step basis during an episode since we would have access to both the step count and some unbiased sample of the state-action value.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$700dcbc4-c94c-4287-8cf0-0b2c7a320a3acell_id$700dcbc4-c94c-4287-8cf0-0b2c7a320a3acode1reinforce_test5.policy_and_value(CartPoleState())metadatashow_logsèdisabled®skip_as_script«code_folded$f59a5dcd-9f4a-4336-a391-e64af35ef799cell_id$f59a5dcd-9f4a-4336-a391-e64af35ef799codehtml"""
"""metadatashow_logsèdisabled®skip_as_scriptëcode_folded$5864a5a3-a5a5-43c2-9cb4-7d13b2d20bedcell_id$5864a5a3-a5a5-43c2-9cb4-7d13b2d20bedcodeHmd"""
Normal Distribution: $f(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{- \frac{(x - \mu)^2}{2 \sigma^2}}$
Consider a new random variable $Y \sim \tanh(X)$ where $X \sim N(0, 1)$. Using the change of variables theorem from probability theory we can compute the density function of $Y$:
$f_Y(y) = f_X (g^{-1}(y)) \cdot \left \vert \frac{d}{dy} g^{-1}(y) \right \vert$
where $g(x) = \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$ so $f_Y(y) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{- \frac{\left (\tanh^{-1}(y) - \mu \right )^2}{2 \sigma^2}} \left \vert \frac{1}{1 - y^2} \right \vert$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$e3a2fb12-37ce-4c23-ad93-5fc89991aabbcell_id$e3a2fb12-37ce-4c23-ad93-5fc89991aabbcodeNmd"""
### Eligibility Vector for General Soft-Max and State Feature Vector
"""metadatashow_logsèdisabled®skip_as_script«code_folded$e5c1aca8-7575-4835-8273-e69ca0a55fe8cell_id$e5c1aca8-7575-4835-8273-e69ca0a55fe8codefunction corridor_parameter_studies(α_list, α_θ_list, α_w_list; nruns = 100, num_episodes = 100, max_steps = 1_000)
Random.seed!(45)
function average_runs(α)
1:nruns |> Map(_ -> reinforce_monte_carlo_control_binary_features(corridor_mdp, get_corridor_features, 1, num_episodes, params = [0f0 3.7f0], α = α, max_steps = max_steps).episode_rewards |> sum) |> foldxt(+) |> x -> x / nruns / num_episodes
end
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> reinforce_with_baseline_monte_carlo_control_binary_features(corridor_mdp, get_corridor_features, 1, num_episodes, policy_params = [0f0 3.7f0], α_θ = α_θ, α_w = α_w, max_steps = max_steps).episode_rewards |> sum) |> foldxt(+) |> x -> x / nruns / num_episodes
end
trace1 = scatter(x = α_list, y = average_runs.(α_list), name = "REINFORCE")
with_baseline_traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "REINFORCE with Baseline: α_w = 2^$(round(Int64, log2(α_w)))")
end
for α_w in α_w_list]
plot([trace1; with_baseline_traces], Layout(xaxis_title = "Policy Parameters Learning Rate", yaxis_title = "Average Reward Per Episode Over First $num_episodes Episodes", xaxis_type = "log", yaxis_range = [-70, -10]))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$44b32cc0-36a8-41fd-89bc-ce894536926ccell_id$44b32cc0-36a8-41fd-89bc-ce894536926ccode$best_mc_corridor.policy_and_value(1)metadatashow_logsèdisabled®skip_as_script«code_folded$646bc853-b7fc-49fa-a201-ff98e8f952d4cell_id$646bc853-b7fc-49fa-a201-ff98e8f952d4codeQfunction corridor_parameter_studies(α_θ_list, α_w_list; nruns = 100, max_episodes = 100, max_steps = 1_000_000)
# Random.seed!(45)
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> one_step_actor_critic_binary_features(corridor_mdp, get_corridor_features, 1, max_episodes, max_steps, policy_params = [0f0 3.7f0], α_θ = α_θ, α_w = α_w) |> x -> isempty(x.episode_rewards) ? -Inf32 : (sum(x.episode_rewards) / length(x.episode_rewards))) |> foldxt(+) |> x -> x / nruns
end
traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "α_w = 2^$(round(Int64, log2(α_w)))")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "Policy Parameters Learning Rate", yaxis_title = "Average Reward Per Episode In First $max_episodes Episodes Averaged over $nruns Runs", xaxis_type = "log"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$25be5dcf-be63-46c4-b6de-6cf79fa28fd0cell_id$25be5dcf-be63-46c4-b6de-6cf79fa28fd0code begin
function update_traces_with_gradient!(c1::T, z_θ::Matrix{T}, c2::T, ∇θ::BinaryEligibilityVector{T, B}) where {T<:Real, B<:BinaryFeatureVector}
z_θ .*= c1
@inbounds for i in eachindex(∇θ.π_dist)
@simd for j in 1:∇θ.binary_features.num_features
k = ∇θ.binary_features.active_features[j]
z_θ[k, i] -= c2*∇θ.π_dist[i]
end
end
@inbounds @simd for i in 1:∇θ.binary_features.num_features
j = ∇θ.binary_features.active_features[i]
z_θ[j, ∇θ.i_a] += c2
end
return z_θ
end
function update_traces_with_gradient!(c1::T, z_w::Vector{T}, ∇w::BinaryFeatureVector) where {T<:Real}
z_w .*= c1
@inbounds @simd for i in 1:∇w.num_features
j = ∇w.active_features[i]
z_w[j] += one(T)
end
return z_w
end
function update_traces_with_gradient!(c1::T, z_θ::Array{T, N}, ∇θ::Array{T, N}) where {T<:Real, N}
z_θ .= c1 .* z_θ .+ ∇θ
end
function update_traces_with_gradient!(c1::T, z_θ::Array{T, N}, c2::T, ∇θ::Array{T, N}) where {T<:Real, N}
z_θ .= c1 .* z_θ .+ c2 .* ∇θ
end
function update_traces_with_gradient!(c1::Float32, z_θ::FCANNParams, ∇θ::FCANNParams)
for i in eachindex(first(z_θ))
for j in 1:2
update_traces_with_gradient!(c1, z_θ[j][i], ∇θ[j][i])
end
end
end
function update_traces_with_gradient!(c1::Float32, z_θ::FCANNParams, c2::Float32, ∇θ::FCANNParams)
for i in eachindex(first(z_θ))
for j in 1:2
update_traces_with_gradient!(c1, z_θ[j][i], c2, ∇θ[j][i])
end
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$38acd032-1d18-4760-9111-67c9cdd2e892cell_id$38acd032-1d18-4760-9111-67c9cdd2e892codeq#without limiting the force in this way, the learned policy just applies so much force to go up the hill directlymetadatashow_logsèdisabled®skip_as_script«code_folded$cecc2a35-3850-4f66-84e8-e29da4f3d4b0cell_id$cecc2a35-3850-4f66-84e8-e29da4f3d4b0codefunction get_corridor_episode_stats(π::Function; ntrials=10_000, kwargs...)
1:ntrials |> Map(_ -> runepisode(corridor_mdp; π = π, kwargs...) |> first |> length) |> foldxt(+) |> a -> a / ntrials
endmetadatashow_logsèdisabled®skip_as_script«code_folded$4c4e643b-d4b9-44f0-8d30-dc521bcc55accell_id$4c4e643b-d4b9-44f0-8d30-dc521bcc55accodeconst cartpole_continuing_mdp = StateMDP(cartpole_functions.discrete_actions, StateMDPTransitionSampler(cartpole_continuing_step, cartpole_functions.initialize_state()), cartpole_functions.initialize_state)metadatashow_logsèdisabled®skip_as_script«code_folded$738ada7f-edc7-4ed3-a15e-e92113468738cell_id$738ada7f-edc7-4ed3-a15e-e92113468738codet#note that the random policy i.e. p = 0.5 has an expected episode length of 12 which is very close to ideal already.metadatashow_logsèdisabled®skip_as_script«code_folded$cacaaca6-6e01-464f-a2ee-cbf62737a426cell_id$cacaaca6-6e01-464f-a2ee-cbf62737a426codeٵreinforce_with_baseline_monte_carlo_control_linear_features(corridor_mdp, update_corridor_features!, 1, 1_000; α_θ = 2f0^-12, α_w = 2f0^-6, max_steps = 1_000).policy_function(1)metadatashow_logsèdisabled®skip_as_script«code_folded$daf35bfe-8f9c-4f55-971d-4d443be8f8bfcell_id$daf35bfe-8f9c-4f55-971d-4d443be8f8bfcode٣display_cartpole_episode((runepisode(cartpole_setup.mdps.episodic.discrete; π = reinforce_test5.policy_sample_action, max_steps = 1_000) |> x -> (x[1], x[2]))...)metadatashow_logsèdisabled®skip_as_script«code_folded$8e096fae-9941-49d8-ae87-c68b02f68da5cell_id$8e096fae-9941-49d8-ae87-c68b02f68da5codeSconst mountaincar_continuous_beta_mdp = create_continuous_action_mountaincar_beta()metadatashow_logsèdisabled®skip_as_script«code_folded$666a4e89-306b-4fb2-bdc4-3dda2c63153fcell_id$666a4e89-306b-4fb2-bdc4-3dda2c63153fcodeusing SpecialFunctionsmetadatashow_logsèdisabled®skip_as_script«code_folded$5d35e515-e2d3-443e-becf-eb28c25db346cell_id$5d35e515-e2d3-443e-becf-eb28c25db346codes@bind mountaincar_continuing_fcann_params create_actor_critic_continuing_params_UI(; λ_θ = 0.85f0, λ_w = 0.95f0)metadatashow_logsèdisabled®skip_as_script«code_folded$4c34640f-efa2-4e1d-8a70-0acd2ce45428cell_id$4c34640f-efa2-4e1d-8a70-0acd2ce45428code md"""
# Bonus Problems: Comparing Techniques
Consider the case of applying the techniques in this chapter to problems where we choose feature vectors and parameters to effectively compute the tabular case. That is we enumerate every state and state/action pair. Our parameters for each function will store a single value for each case. Let's consider the gradients for both the state-value estimate and the policy. We will use two sets of parameters: $\mathbf{w}$ and $\mathbf{\theta}$. $\mathbf{w}_s$ is the parameter for state s and $\mathbf{\theta}_{s, a}$ is the parameter for state/action pair $(s, a)$. Using this notation $\mathbf{w}$ is a vector and $\theta$ is a matrix.
Starting with the state-value function:
$\begin{align}
\hat v(s, \mathbf{w}) &= \mathbf{w}_s \\
\nabla v(s, \mathbf{w}) &= \nabla \mathbf{w}_s \\
&= \mathbf{e}_s
\end{align}$
where $\mathbf{e}_s$ is the one-hot vector for index s and length equal to the number of states.
Now moving on to the policy, we will use a soft-max function to convert action preferences into probabilities.
$\begin{align}
\pi(a|s, \theta) &= \frac{\exp{\theta_{s, a}}}{\sum_{i = 1}^{n_A}{\exp{\theta_{s, i}}}} \\
\nabla \pi(a|s, \theta) &= \nabla \frac{\exp{\theta_{s, a}}}{\sum_{i = 1}^{n_A}{\exp{\theta_{s, i}}}} \\
\end{align}$
But we already calculated the gradient of the soft-max function of a vector $\mathbf{x}$.
$\nabla\sigma(\mathbf{x})_{i, j} = \sigma(\mathbf{x})_i \left ( \delta_{i, j} - \sigma(\mathbf{x})_j \right )$
Comparing to what we desire, $\mathbf{x} = \mathbf{\theta}_s$ which is the parameter vector for the state s and $\sigma = \pi$. So we can immediately write down the components of this gradient:
$\begin{align}
\nabla \pi(a|\theta_s)_i &= \pi(a|\theta_s) \left (\delta_{a, i} - \pi(i|\theta_s) \right ) \\
\frac{\nabla \pi(a|\theta_s)_i}{\pi(a|\theta_s)} = \nabla \ln \pi(a|\theta_s)_i &= \left (\delta_{a, i} - \pi(i|\theta_s) \right ) \\
\end{align}$
$\begin{equation}
\nabla \ln{\pi(a|\theta_s)}_i =
\begin{cases}
-\pi(i|\theta_s) & i \neq a \\
1 - \pi(i|\theta_s) & i = a
\end{cases}
\end{equation}$
This is a gradient vector which corresponds to the components of $\theta_s$ which is the parameter vector for each action at that state. We have a new vector update for each unique state/action pair observed, but once those two are fixed the number of components that need to be calculated is just a vector with a length equal to the number of actions.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$e7566274-5518-4e28-8738-d4b1747d0cfbcell_id$e7566274-5518-4e28-8738-d4b1747d0cfbcodefunction form_state_value_function(update_feature_vector!::Function, value_function::Function)
function v!(x, s, value_params)
update_feature_vector!(x, s)
value_function(x, value_params)
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$6bf5ad39-1400-4e1f-a843-a1934b8aaa48cell_id$6bf5ad39-1400-4e1f-a843-a1934b8aaa48codebegin
function update_squashed_gaussian_eligibility_vector!(∇lnπ::Matrix{T}, action_dist_params::Vector{T}, x::Vector{T}, action::T, policy_params::Matrix{T}, amax::T) where T<:Real
c1 = atanh(action/amax) - first(action_dist_params)
σ = exp(last(action_dist_params))
c2 = σ^-2
c3 = c2*c1
c4 = c3*c1 - one(T)
@inbounds @simd for i in eachindex(x)
∇lnπ[i, 1] = x[i]*c3
end
@inbounds @simd for i in eachindex(x)
∇lnπ[i, 2] = x[i]*c4
end
end
function update_squashed_gaussian_eligibility_vector!(∇lnπ::Matrix{T}, action_dist_params::Vector{T}, x::Vector{T}, action::NTuple{N, T}, policy_params::Matrix{T}, amax::NTuple{N, T}) where {N, T <: Real}
for k = 1:N
c1 = atanh(action/amax[k]) - action_dist_params[k]
σ = exp(action_dist_params[k+N])
c2 = σ^-2
c3 = c2*c1
c4 = c3*c1 - one(T)
@inbounds @simd for i in eachindex(x)
∇lnπ[i, k] = x[i]*c3
end
@inbounds @simd for i in eachindex(x)
∇lnπ[i, k+N] = x[i]*c4
end
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$17d07ef4-7c0a-47cc-a701-32c60336571bcell_id$17d07ef4-7c0a-47cc-a701-32c60336571bcodemd"""
Noticing this pattern, the kth term will be of the form $\gamma^k \sum_{x \in \mathcal{S}} \Pr(s \rightarrow x, k, \pi)f(x)$ and the total expression will just be a sum of all of these terms to infinity or the maximum length of an episode under the policy. Looking more closely at the probability term, we can equate it to some other probabilities regarding episode length.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$76fd79a2-2bc8-45f8-a243-48415118898acell_id$76fd79a2-2bc8-45f8-a243-48415118898acode+begin
mutable struct BinarySquashedGaussianEligibilityVector{T<:Real, A<:Union{T, NTuple{N, T} where N}, P<:Union{T, Vector{T}}, B <: BinaryFeatureVector}
binary_features::B
a::A
μ::P
σ::P
amax::A
end
BinarySquashedGaussianEligibilityVector(a::T, amax::T) where T<:Real = BinarySquashedGaussianEligibilityVector(BinaryFeatureVector(), a, zero(T), one(T), amax)
BinarySquashedGaussianEligibilityVector(a::NTuple{N, T}) where {T<:Real, N} = BinarySquashedGaussianEligibilityVector(BinaryFeatureVector(), a, zeros(T, N), ones(T, N), amax)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$0b01ba67-3921-4f3f-a7e8-235190bc84ebcell_id$0b01ba67-3921-4f3f-a7e8-235190bc84ebcodeRfunction make_beta_dist(α, β)
f(x) = x^(α-1) * (1-x)^(β-1) / beta(α, β)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$9acdbf38-2e10-45ec-85a0-d0db8453a599cell_id$9acdbf38-2e10-45ec-85a0-d0db8453a599code#this version of tile coding setup just produces a function that returns the active indices as a generator rather than actually update the feature vector
function fcann_feature_vector_setup(min_value::S, max_value::S) where {T<:Real, N, S <: Union{T, NTuple{N, T}}}
#states must be tuples with k elements or some number value
k = S == T ? 1 : N
s_range = if k == 1
max_value - min_value
else
Tuple(max_value[i] - min_value[i] for i in 1:k)
end
sample_vector = make_sample_vector(min_value)
function update_feature_vector!(x::Vector{T}, s::Real)
x[1] = scale_state(s, min_value, s_range)
return x
end
function update_feature_vector!(x::Vector{T}, s::NTuple{N, T})
for i in 1:N
x[i] = scale_state(s[i], min_value[i], s_range[i])
end
return x
end
(feature_vector = sample_vector, num_features = length(sample_vector), update_feature_vector! = update_feature_vector!)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$d4e87ac4-6008-43b2-aa06-e232ec2b2b5bcell_id$d4e87ac4-6008-43b2-aa06-e232ec2b2b5bcodeqplot_cartpole_policy(reinforce_test5.policy_and_value; s_ref = CartPoleState(Float32(x), 0f0, Float32(ẋ), 0f0))metadatashow_logsèdisabled®skip_as_script«code_folded$05f120be-9695-4824-82fd-142a0df13098cell_id$05f120be-9695-4824-82fd-142a0df13098codefunction actor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, amax::A, λ_θ::T, λ_w::T, get_active_features::Function, num_features::Integer, args...; policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, N, A <: Union{T, NTuple{N, T}}, P, F1, F2, F3}
setup = setup_binary_squashed_gaussian_policy_arguments(mdp, amax, get_active_features, num_features)
actor_critic_with_eligibility_traces!(policy_params, setup.eligibility_vector, value_params, BinaryFeatureVector(), mdp, λ_θ, λ_w, update_binary_action_preferences!, setup.action_distribution_parameters, make_squashed_gaussian_sampler(rand(A), amax), update_squashed_gaussian_eligibility_vector!, setup.feature_vector, setup.update_feature_vector!, binary_value_function, update_binary_value_gradient!, args...; kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$b2539398-fdbc-42a2-a8f3-d327358f3643cell_id$b2539398-fdbc-42a2-a8f3-d327358f3643codeif start_cartpole_continuing_binary_param_study > 0
cartpole_binary_continuing_parameter_study(cartpole_continuing_binary_study_params, 5, 3, 10_000; seed = 45)
else
md"""
Waiting to run parameter study
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$c5dd7e99-57e0-4bc7-97d2-2c780b23bcffcell_id$c5dd7e99-57e0-4bc7-97d2-2c780b23bcffcodeqmd"""
#### Discrete Action Space
As an initial test, consider the discrete action space originally used for the mountain car problem where there are three actions (-1, 0, 1) corresponding to full throttle reverse, idle, and full throttle forward. We can apply the same tile coding solution technique from before but with a policy gradient method instead of Sarsa.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9cell_id$d5ab6d24-dd4e-4410-a50e-fe3584b21cf9code9const mountaincar_continuing_fcann_test = actor_critic_with_eligibility_traces_fcann(mountaincar_continuing_mdp, 0.85f0, 0.95f0, mountaincar_fcann_setup.num_features, [32, 32, 32], mountaincar_fcann_setup.update_feature_vector!, 1_000_000, α_θ = 0.002f0, α_w = 0.002f0, α_r̄ = 0.01f0; save_step_rewards=true)metadatashow_logsèdisabled®skip_as_script«code_folded$042fbafe-2401-4fb7-ac13-4531e0782c79cell_id$042fbafe-2401-4fb7-ac13-4531e0782c79codefunction update_binary_eligibility_vector!(∇lnπ::BinaryEligibilityVector{T, B}, action_preferences::Vector{T}, binary_features::B, i_a::Integer, params::Matrix{T}) where {T<:Real, B<:BinaryFeatureVector}
update_binary_action_preferences!(action_preferences, binary_features, params)
soft_max!(action_preferences)
∇lnπ.binary_features = binary_features
∇lnπ.i_a = i_a
∇lnπ.π_dist .= action_preferences
return ∇lnπ
endmetadatashow_logsèdisabled®skip_as_script«code_folded$d57375a5-b9e0-4742-b5f7-6a7da891604acell_id$d57375a5-b9e0-4742-b5f7-6a7da891604acode
mountaincar_binary_continuing_parameter_study(args...; kwargs...) = actor_critic_linear_parameter_study(mountaincar_continuing_mdp, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, args...; binary_features=true, kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$07ad517a-c2ac-4377-99fb-adb13d0f1d0ccell_id$07ad517a-c2ac-4377-99fb-adb13d0f1d0ccodeٓreinforce_monte_carlo_control_fcann(corridor_mdp, 1, [10, 10], update_corridor_features!, 100; α = 2f0^-14, max_steps = 10_000).policy_function(1)metadatashow_logsèdisabled®skip_as_script«code_folded$71a5fce8-6d9a-4625-bad1-a951d61bff28cell_id$71a5fce8-6d9a-4625-bad1-a951d61bff28codef@bind mountaincar_binary_continuous_params create_actor_critic_params_UI(λ_θ = 0.05f0, λ_w = 0.8f0)metadatashow_logsèdisabled®skip_as_script«code_folded$77906355-08f8-4b08-b051-84697199b519cell_id$77906355-08f8-4b08-b051-84697199b519code,const mountaincar_max_vals = (0.5f0, 0.07f0)metadatashow_logsèdisabled®skip_as_script«code_folded$5207308e-f636-4d47-b135-036a6e7b8ecdcell_id$5207308e-f636-4d47-b135-036a6e7b8ecdcodefshow_mountaincar_continuous_trajectory(mountaincar_continuous_test_train3.policy_sample_action, 1_000)metadatashow_logsèdisabled®skip_as_script«code_folded$16113560-e911-47b4-abc4-641bbd246454cell_id$16113560-e911-47b4-abc4-641bbd246454code_plot(mountaincar_continuous_test_train_beta.episode_rewards, Layout(yaxis_range = [-10000, 0]))metadatashow_logsèdisabled®skip_as_script«code_folded$b7f77935-bcab-4ef1-8e1b-a7d059784ff3cell_id$b7f77935-bcab-4ef1-8e1b-a7d059784ff3code@bind test_mountaincar_state PlutoUI.combine() do Child
md"""
#### Evaluation State for Policy Function
x position: $(Child(Slider(-1.2f0:0.1f0:0.5f0, default = 0f0, show_value=true)))
velocity: $(Child(Slider(-0.07f0:0.01f0:0.07f0, default = 0f0, show_value=true)))
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5cell_id$f9ac1bf0-55ee-4c71-bdaa-a00f9d779bf5codeUreinforce_test.policy_and_value(cartpole_mdps.episodic.continuous.initialize_state())metadatashow_logsèdisabled®skip_as_script«code_folded$00bd2835-b006-4244-9877-bc7e031e3ef8cell_id$00bd2835-b006-4244-9877-bc7e031e3ef8codefunction plot_squashed_gaussian(μ::T, σ::T, xmax::T; npoints = 1000) where T<:Real
x = LinRange(-one(T)*xmax, one(T)*xmax, npoints)
y = squashed_gaussian_pdf.(x, μ, σ, xmax)
plot(x, y, Layout(xaxis_range = [-2, 2]))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$50ae94c4-70f3-4215-82bd-eb2227c2badfcell_id$50ae94c4-70f3-4215-82bd-eb2227c2badfcodeif start_cartpole_continuing_fcann_param_study > 0
cartpole_fcann_continuing_parameter_study(cartpole_continuing_fcann_network_params..., cartpole_continuing_fcann_study_params, 4, 3, 100_000; seed = 45)
else
md"""
Waiting to run parameter study
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$cc3ac95e-a398-438a-ba3d-62b6733f6342cell_id$cc3ac95e-a398-438a-ba3d-62b6733f6342code function update_fcann_action_preferences!(action_preferences::Vector{T}, x::Vector{T}, params::FCANNParams, activations::FCANNActivations{T}, reslayers::Integer) where T<:Float32
FCANN.forwardNOGRAD_base!(activations, params..., x, reslayers)
action_preferences .= activations[end]
endmetadatashow_logsèdisabled®skip_as_script«code_folded$c926b6df-c40b-4c4c-8a95-ce9e41feb100cell_id$c926b6df-c40b-4c4c-8a95-ce9e41feb100codeMactor_critic_fcann_parameter_study(mountaincar_continuing_mdp, mountaincar_fcann_feature_setup.update_feature_vector!, mountaincar_fcann_feature_setup.num_features, [4, 4], 0.0f0:0.05f0:0.95f0, 0.0f0:0.05f0:0.95f0, [0.01f0, 0.005f0], 2f0 .^ (-20:-1), 2f0 .^ (-20:-1), 1_000, 1_000_000; seed = 45) |> df -> sort(df, :output; rev=true)metadatashow_logsèdisabledîskip_as_script«code_folded$740a3f41-9302-481d-b373-762c0dea8effcell_id$740a3f41-9302-481d-b373-762c0dea8effcodeVbegin
function update_gaussian_eligibility_vector!(∇lnπ::BinaryGaussianEligibilityVector{T, T, T, B}, dist_params::Vector{T}, x::B, action::T, policy_params::Matrix{T}) where {T<:Real, B<:BinaryFeatureVector}
∇lnπ.binary_features = x
∇lnπ.a = action
∇lnπ.μ = first(dist_params)
∇lnπ.σ = exp(last(dist_params))
# isapprox(∇lnπ.σ, 0f0) && @info "with distribution params $dist_params having 0 result for σ of $∇lnπ.σ"
# isinf(∇lnπ.σ) && @info "with distribution params $dist_params having inf result for σ of $∇lnπ.σ"
# isnan(∇lnπ.σ) && @info "with distribution params $dist_params having nan result for σ of $∇lnπ.σ"
return ∇lnπ
end
function update_gaussian_eligibility_vector!(∇lnπ::BinaryGaussianEligibilityVector{T, NTuple{N, T}, Vector{T}, B}, dist_params::Vector{T}, x::B, action::NTuple{N, T}, policy_params::Matrix{T}) where {T<:Real, N, B<:BinaryFeatureVector}
∇lnπ.binary_features = x
∇lnπ.a = action
for i in 1:N
∇lnπ.μ[k] = dist_params[k]
∇lnπ.σ[k] = exp(dist_params[k+N])
end
return ∇lnπ
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$ba642a22-6623-482a-ab4a-81585b83e457cell_id$ba642a22-6623-482a-ab4a-81585b83e457code-@memoize Dict function average_continuing_runs(nruns::Integer, seed::Integer, α_θ::T, α_w::T, α_r̄::T, policy_params, algo, args...; kwargs...) where T<:Real
# @info "Running trials for continuing actor critic with binary encoding: $nruns $seed $α_θ $α_w $α_r̄ $mdp $λ_θ $λ_w $get_active_features $num_features"
Random.seed!(seed)
1:nruns |> Map() do _
x =algo(args...; α_θ = α_θ, α_w = α_w, α_r̄ = α_r̄, policy_params = deepcopy(policy_params), kwargs...)
x.total_reward / x.total_steps
end |> foldxt(+) |> a -> a / nruns
endmetadatashow_logsèdisabled®skip_as_script«code_folded$d17a4bd0-5992-4247-912d-73d51758d2f3cell_id$d17a4bd0-5992-4247-912d-73d51758d2f3code+md"""
### *Continuing Cartpole Example*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7efcell_id$db6ed0ea-c26b-4ea1-b4a1-7641f0f9c7efcode٧plot_cartpole_policy(cartpole_continuing_fcann_test.policy_and_value; s_ref = cartpole_fcann_continuing_test_episode[1][cartpole_fcann_continuing_episode_step_select])metadatashow_logsèdisabled®skip_as_script«code_folded$5ee4ce72-7740-4297-8d84-619e0708e4accell_id$5ee4ce72-7740-4297-8d84-619e0708e4accodefunction cartpole_continuing_fcann_parameter_study(α1_list, α2_list, α_r̄, λ_θ, λ_w, hidden_layers, max_steps; num_trials = 100, kwargs...)
setup = setup_cartpole_problem(;kwargs...)
traces = [begin
steps = [begin
1:num_trials |> Map() do i
solution = actor_critic_with_eligibility_traces_fcann(cartpole_setup.mdps.continuing.discrete, λ_θ, λ_w, cartpole_fcann_feature_setup.num_features, hidden_layers, (x, s) -> cartpole_fcann_feature_setup.update_feature_vector!(x, (s.x, s.θ, s.ẋ, s.θ̇)), max_steps; α_θ = α1, α_w = α2, α_r̄ = α_r̄)
solution.total_reward / max_steps
end |> foldxt(+) |> x -> x / num_trials
end
for α1 in α1_list]
scatter(x = α1_list, y = steps, name = "α_w = $α2")
end
for α2 in α2_list]
plot(traces, Layout(xaxis_title = "Policy Learning Rate α_θ", yaxis_title = "Average Failure Rate Over First $max_steps Steps", xaxis_type = "log", title = "Hiden Layers = $hidden_layers, λ_θ = $λ_θ, λ_w = $λ_w"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$645e93e7-e92e-49c4-9757-8294fabf4e9bcell_id$645e93e7-e92e-49c4-9757-8294fabf4e9bcodeCplot_continuing_step_rewards(cartpole_continuing_test.step_rewards)metadatashow_logsèdisabled®skip_as_script«code_folded$0c56b341-24eb-4c78-844e-182f44a7221acell_id$0c56b341-24eb-4c78-844e-182f44a7221acode#in the source code used to generate this for the book found here: http://incompleteideas.net/book/code/figure_13_1.py - graphs look as they do because of poor parameter initialization since the random policy is fairly close to ideal already
figure_13_1(2f0 .^ [-12, -13, -14])metadatashow_logsèdisabled®skip_as_script«code_folded$d34d22ad-89c2-423e-91dd-bfb895dc6540cell_id$d34d22ad-89c2-423e-91dd-bfb895dc6540codecartpole_fcann_parameter_study(args...; kwargs...) = actor_critic_fcann_episodic_parameter_study(cartpole_setup.mdps.episodic.discrete, cartpole_vector_update!, cartpole_fcann_feature_setup.num_features, args...; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$20776e09-7d9b-4db8-a060-7bceeec65b47cell_id$20776e09-7d9b-4db8-a060-7bceeec65b47codefunction actor_critic_with_eligibility_traces_binary_features_gaussian_actions(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, λ_θ::T, λ_w::T, get_active_features::Function, num_features::Integer, args...; policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, N, A <: Union{T, NTuple{N, T}}, P, F1, F2, F3}
setup = setup_binary_gaussian_policy_arguments(mdp, get_active_features, num_features)
actor_critic_with_eligibility_traces!(policy_params, setup.eligibility_vector, value_params, BinaryFeatureVector(), mdp, λ_θ, λ_w, update_binary_action_preferences!, setup.action_distribution_parameters, make_gaussian_sampler(rand(A)), update_gaussian_eligibility_vector!, setup.feature_vector, setup.update_feature_vector!, binary_value_function, update_binary_value_gradient!, args...; kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$7856b8a0-565d-4c86-9b3c-4424ff9b86ddcell_id$7856b8a0-565d-4c86-9b3c-4424ff9b86ddcodeb#add policy gradient example on cartpole without continuous actions and parameter studies for bothmetadatashow_logsèdisabled®skip_as_script«code_folded$735b548a-88f5-4a30-ab8f-dfb3d6401b2bcell_id$735b548a-88f5-4a30-ab8f-dfb3d6401b2bcodeAmd"""
## 13.7 Policy Parameterization for Continuous Actions
With a parameterized policy we are to learn statistics of the distribution that selects actions. As a foundation consider the normal distribution:
$p(x) \doteq \frac{1}{\sigma \sqrt{2\pi}} \exp \left ( - \frac{(x-\mu)^2}{2\sigma^2} \right ) \tag{13.18}$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$7cf26604-9c2b-4a77-9674-7d4dac2f99f0cell_id$7cf26604-9c2b-4a77-9674-7d4dac2f99f0codebegin
include(joinpath(@__DIR__, "..", "Chapter-09", "Chapter_09_On-policy_Prediction_with_Approximation.jl"))
include(joinpath(@__DIR__, "..", "Chapter-10", "Chapter_10_On_policy_Control_with_Approximation.jl"))
include(joinpath(@__DIR__, "..", "Chapter-11", "Chapter_11_Off_policy_Methods_with_Approximation.jl"))
include(joinpath(@__DIR__, "..", "Chapter-12", "Chapter_12_Eligibility_Traces.jl"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$87ee21f3-16ca-4c8c-a0b9-f9e2fd258a91cell_id$87ee21f3-16ca-4c8c-a0b9-f9e2fd258a91codeEmd"""
### *REINFORCE Implementation for Continuous Action Spaces*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$54f1546d-87ae-49d2-92ed-6fcc9b66e027cell_id$54f1546d-87ae-49d2-92ed-6fcc9b66e027code md"""
### *Mountain Car MDP*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$63fbf8f4-e4e2-4893-be09-67450e92dbd7cell_id$63fbf8f4-e4e2-4893-be09-67450e92dbd7code,function plot_cart(s::CartPoleState, a::Int64; xmin = -50, xmax = 50, θ̇_min = -10, θ̇_max = 10)
s.x
s.θ
t1 = scatter(x = [0, sin(s.θ)], y = [0, cos(s.θ)], mode = "lines", color = "black")
t2 = scatter(x = [sin(s.θ)], y = [cos(s.θ)], mode = "markers", color = "black")
p1 = plot([t1, t2], Layout(yaxis_range = [-.1, 1.2], xaxis_range = [-1.2, 1.2], xaxis_scaleanchor = "y", width = 250, height = 230, showlegend = false, title = "Pole Angle"))
p2 = plot(scatter(x = [s.x], y = [0]), Layout(xaxis_range = [xmin, xmax], width = 250, height = 230, title = "X Location"))
p3 = plot(indicator(mode = "gauge+number+delta", value = s.θ̇, title_text = "Angular Speed in Radians per Second", delta_reference = 0, gauge_axis_range = [-10, 10]), Layout(width = 250, height = 230))
p4 = plot(indicator(mode = "gauge+number+delta", value = s.ẋ, title_text = "Horizontal Speed in Meters per Second", delta_reference = 0, gauge_axis_range = [-50, 50]), Layout(width = 250, height = 230))
p5 = plot(indicator(mode = "gaugue+number", gauge=attr(shape="bullet"), value = a - 2, title_text = "Action", delta_reference = 0, gauge_axis_range = [-1, 1]), Layout(width = 250, height = 230))
@htl("""
$p1
$p2
$p3
$p4
$p5
""")
endmetadatashow_logsèdisabled®skip_as_script«code_folded$d5020a8d-1dd7-403c-9d1f-665b95543943cell_id$d5020a8d-1dd7-403c-9d1f-665b95543943codeVreinforce_with_baseline_monte_carlo_control_linear_features_gaussian_actions(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, update_feature_vector!::Function, num_features::Integer, max_episodes::Integer; policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), value_params::Vector{T} = zeros(T, num_features), x = zeros(T, num_features), action_dist_params::Vector{T} = make_gaussian_params(rand(A)), kwargs...) where {T<:Real, S, N, A<:Union{T, NTuple{N, T}}, P, F1, F2, F3} = reinforce_with_baseline_monte_carlo_control!(policy_params, copy(policy_params), value_params, copy(value_params), mdp, update_linear_action_preferences!, action_dist_params, make_gaussian_sampler(rand(A)), update_gaussian_eligibility_vector!, x, update_feature_vector!, linear_value_function, update_linear_value_gradient!, max_episodes; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$37a8ef7e-e859-4ef0-81e2-76c02a324031cell_id$37a8ef7e-e859-4ef0-81e2-76c02a324031code md"""
### Policy Gradient Theorem Proof
In all cases below when a sum over states is taken, it is assumed to be over the set of non-terminal states: $\sum_s \implies \sum_{s \in \mathcal{S}}$ Note that for the case of the value function this is identical to the sum over $\mathcal{S}^+$ because the state-action values are always zero for terminal states.
$\begin{flalign}
\nabla v_\pi(s) &= \nabla \left [ \sum_a \pi(a \vert s) q_\pi(s, a) \right ] \text{, } \forall s \in \mathcal{S} \tag{definitiong of value functions and expected value} \\
&= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \nabla q_\pi(s, a) \right ] \tag{product rule} \\
&= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \nabla \sum_{s^\prime, r} p(s^\prime, r \vert s, a)(r + \gamma v_\pi(s^\prime) \right ] \tag{relationship between action and state values} \\
&= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \gamma \sum_{s^\prime} p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) \right ] \tag{gradient independence}\\
\end{flalign}$
Note that the final term in the sum is the original expression evaluated at $s^\prime$ instead of $s$, so we have derived a recurssive expression which can be applied repeatedly:
$\begin{flalign}
\nabla v_\pi(s) &= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \gamma \sum_{s^\prime} p(s^\prime \vert s, a) \sum_{a^\prime} \left [ \nabla \pi(a^\prime \vert s^\prime) q_\pi(s^\prime, a^\prime) + \pi(a^\prime \vert s^\prime) \gamma \sum_{s^{\prime \prime}} p(s^{\prime \prime} \vert s^\prime, a^\prime) \nabla v_\pi(s^{\prime \prime}) \right ] \right ] \tag{recur once}\\
&= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) \right ] + \gamma \sum_a \left [ \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \sum_{a^\prime} \left [ \nabla \pi(a^\prime \vert s^\prime) q_\pi(s^\prime, a^\prime) \right ] \right ] + \\ &\hspace{50px} \gamma^2 \sum_a \left [ \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \sum_{a^\prime} \pi(a^\prime \vert s^\prime) \sum_{s^{\prime \prime}} p(s^{\prime \prime} \vert s^\prime, a^\prime) \nabla v_\pi(s^{\prime \prime}) \right ] \tag{grouping terms}\\
&= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) \right ] + \gamma \sum_a \left [ \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \sum_{a^\prime} \left [ \nabla \pi(a^\prime \vert s^\prime) q_\pi(s^\prime, a^\prime) \right ] \right ] + \\ &\hspace{50px} \gamma^2 \sum_a \left [ \pi(a \vert s)\sum_{s^\prime} p(s^\prime \vert s, a) \sum_{a^\prime} \pi(a^\prime \vert s^\prime) \sum_{s^{\prime \prime}} p(s^{\prime \prime} \vert s^\prime, a^\prime) \sum_{a^{\prime \prime}} [ \nabla \pi(a^{\prime \prime} \vert s^{\prime \prime}) q_\pi(s^{\prime \prime}, a^{\prime \prime})\right ] + \cdots \tag{extend recursion}\\
\end{flalign}$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$98229733-a71e-44ca-a52a-b7229cf8b422cell_id$98229733-a71e-44ca-a52a-b7229cf8b422code1md"""
The probability transition function is normalized over all possible transition states $\sum_{s^\prime \in \mathcal{S}^+} p(s^\prime \vert s, a) = 1$. If we only take the sum of $\mathcal{S}$ then we instead get the probability that after a single transition we have NOT reached a terminal state. Let's say we also have a policy function $\pi(a \vert s)$ which is normalized over actions: $\sum_a \pi(a \vert s) = 1$. Now if we combine the two, we can arrive at a new distribution over transition states: $p(s^\prime \vert s, \pi) = \sum_a \pi(a \vert s) p(s^\prime \vert s, a)$ which is the probability of transitioning from $s$ to $s^\prime$ under the policy. We can see that this distribution is normalized over the transition states as well as long as we include the terminal state: $\sum_{s^\prime \in \mathcal{S}^+} p(s^\prime \vert s, \pi) = \sum_{s^\prime \in \mathcal{S}^+, a} \pi(a \vert s) p(s^\prime \vert s, a) = \sum_a \pi(a \vert s) \sum_{s^\prime \in \mathcal{S}^+} p(s^\prime \vert s, a) = 1 \times 1 = 1$. If instead we take the sum over $\mathcal{S}$ we simply get the probability of NOT terminating in one step.
What if we consider two steps into the future though? Now we have $\sum_{s^\prime}\sum_{a^\prime}\pi(a^\prime \vert s^\prime)p(s^{\prime \prime} \vert s^\prime, a^\prime)\sum_a \pi(a \vert s) p(s^\prime \vert s, a) = \sum_{s^\prime}p(s^{\prime \prime} \vert s^\prime, \pi) p(s^\prime \vert s, \pi)$. It would appear as though we can just put the two probabilities together and consider a new distribution over $s^{\prime \prime}$ which is $p(s^{\prime \prime} \vert s, \pi, 2)$ where instead of one step this now occurs over two steps, but how is this distribution normalized? In the case of the one step, transition, we saw that its sum over all transition states is 1 as expected. If we sum both transition states over only $\mathcal{S}$ rather than $\mathcal{S}^+$ what is the result? We already know that $\sum_{s^{\prime \prime} \in \mathcal{S}^+} p(s^{\prime \prime} \vert s^\prime , \pi) = \Pr \{ S_1 \neq S_T \ \vert S_0 = s^\prime, \pi \}$ that is the probability that after transitioning out of $s^\prime$ under the policy $\pi$ we have not reached a terminal state.
$\sum_{s^{\prime \prime} \in \mathcal{S}} \sum_{s^\prime \in \mathcal{S}} p(s^{\prime \prime} \vert s^\prime, \pi) p(s^\prime \vert s, \pi) = \sum_{s^\prime \in \mathcal{S}} p(s^\prime \vert s, \pi) \sum_{s^{\prime \prime} \in \mathcal{S}} p(s^{\prime \prime} \vert s^\prime, \pi) = \Pr \{ S_2 \neq S_T \vert S_0 = s, \pi \}$ which is to say the probability that after two transitions from $s$ we are not in a terminal state under the policy $\pi$.
For the derivations that follow, we always take sums of these distributions over $\mathcal{S}$. For episodic problems, the on policy distribution $\mu_\pi(s)$ which is the probability of being in a state $s$ during an episode always excludes the terminal state. That is because if there is a non-zero probability of reaching a terminal state under a policy, then considering all possible episodes we may have an infinite number of visits to the terminal state. Technically the episodes have infinite length but we are only interested in the portion of the episode that preceeds the terminal state for the purpose of calculating probabilities. The more careful statement about the on policy distribution is that it measures the probability of being in a state during the non-terminal part of an episode. If we try to include the terminal states, then we cannot have a proper normalized definition of the on-policy distribution. Moreover, we have no need to measure the value of a terminal state accurately, since we always know it to be 0. The on policy distribution is used to formulate the value error objective function and it should only include states for which the value estimation is non-trivial.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$42775fd1-5b27-48e0-abf1-9b22bb775e6dcell_id$42775fd1-5b27-48e0-abf1-9b22bb775e6dcodeKcorridor_continuing_parameter_study(continuing_study_params, 5, 3, 100_000)metadatashow_logsèdisabled®skip_as_script«code_folded$7dbb42a3-aa8c-47e5-b668-18e6325d4038cell_id$7dbb42a3-aa8c-47e5-b668-18e6325d4038code!md"""
#### Tile Coding Method
"""metadatashow_logsèdisabled®skip_as_script«code_folded$192b9f82-8d3a-408f-91c2-829cfcd32572cell_id$192b9f82-8d3a-408f-91c2-829cfcd32572codeٝcartpole_vector_update!(x::Vector{T}, s::CartPoleState{T}) where T<:Real = cartpole_fcann_feature_setup.update_feature_vector!(x, (s.x, s.θ, s.ẋ, s.θ̇))metadatashow_logsèdisabled®skip_as_script«code_folded$b5319d8b-0420-4ebf-b603-ea0b93365ac1cell_id$b5319d8b-0420-4ebf-b603-ea0b93365ac1codenfunction show_mountaincar_continuous_trajectory(π::Function, max_steps::Integer; mdp = mountaincar_continuous_mdp)
states, actions, rewards, sterm, nsteps = runepisode(mdp, π; max_steps = max_steps)
positions = [s[1] for s in states]
velocities = [s[2] for s in states]
tr1 = scatter(x = positions, y = velocities, mode = "markers", showlegend = false)
tr2 = scatter(y = positions, showlegend = false)
tr3 = scatter(y = actions, showlegend = false)
p1 = plot(tr1, Layout(xaxis_title = "position", yaxis_title = "velocity", xaxis_range = [-1.2, 0.5], yaxis_range = [-0.07, 0.07], height = 400))
p2 = plot(tr2, Layout(xaxis_title = "time", yaxis_title = "position", height = 400))
p3 = plot(tr3, Layout(xaxis_title = "time", yaxis_title = "action", height = 400))
@htl("""
Total Reward: $(sum(rewards))
$([p1 p2 p3])
""")
endmetadatashow_logsèdisabled®skip_as_script«code_folded$4cbdb082-22ba-49e9-a6ed-4380917625accell_id$4cbdb082-22ba-49e9-a6ed-4380917625accodeCmd"""
### *Actor-Critic with Eligibility Traces Implementation*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$cc80848a-6834-4272-9152-e17b45448814cell_id$cc80848a-6834-4272-9152-e17b45448814codefunction wind_speeds(directions)
PlutoUI.combine() do Child
@htl("""
Wind speeds
$([
@htl("
$(name): $(Child(name, Slider(1:100)))
")
for name in directions
])
""")
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$05bfd818-bf4e-4bda-baa9-5ba647867097cell_id$05bfd818-bf4e-4bda-baa9-5ba647867097code.function actor_critic_with_eligibility_traces_binary_features(mdp::StateMDP{T, S, A, P, F1, F2, F3}, λ_θ::T, λ_w::T, get_active_features::Function, num_features::Integer, args...; policy_params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
setup = setup_binary_policy_arguments(mdp, get_active_features, num_features)
actor_critic_with_eligibility_traces!(policy_params, setup.eligibility_vector, value_params, BinaryFeatureVector(), mdp, λ_θ, λ_w, update_binary_action_preferences!, update_binary_eligibility_vector!, setup.feature_vector, setup.update_feature_vector!, binary_value_function, update_binary_value_gradient!, args...; action_preferences = setup.action_preferences, kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$f0962801-0dfa-421f-8ffc-e64068e49913cell_id$f0962801-0dfa-421f-8ffc-e64068e49913codefconst mountaincar_fcann_feature_setup = fcann_feature_vector_setup((-1.2f0, -0.07f0), (0.5f0, 0.07f0))metadatashow_logsèdisabled®skip_as_script«code_folded$11a55af7-5301-4507-bb26-88e1e11236dbcell_id$11a55af7-5301-4507-bb26-88e1e11236dbcode٥display_cartpole_episode((runepisode(cartpole_setup.mdps.episodic.discrete; π = reinforce_test3.policy_sample_action, max_steps = 100_000) |> x -> (x[1], x[2]))...)metadatashow_logsèdisabled®skip_as_script«code_folded$ddbca73f-c692-46f2-95f3-a7dd849d33f7cell_id$ddbca73f-c692-46f2-95f3-a7dd849d33f7codeOshow_mountaincar_trajectory(mountaincar_test_train.policy_sample_action, 1_000)metadatashow_logsèdisabled®skip_as_script«code_folded$b4875f2b-5487-429f-80a3-d1032bbccfc1cell_id$b4875f2b-5487-429f-80a3-d1032bbccfc1codeCmd"""
### Policy Gradient Theorem Proof for Continuing Problems
"""metadatashow_logsèdisabled®skip_as_script«code_folded$0cd96c44-cae6-421f-9fae-26141600bef4cell_id$0cd96c44-cae6-421f-9fae-26141600bef4code٬display_cartpole_episode((runepisode(cartpole_setup.mdps.episodic.discrete; π = cartpole_continuing_test.policy_sample_action, max_steps = 1_000) |> x -> (x[1], x[2]))...)metadatashow_logsèdisabled®skip_as_script«code_folded$51d6337d-c0bd-40a9-9129-7d88e41e4093cell_id$51d6337d-c0bd-40a9-9129-7d88e41e4093codeR#add plot under this to show the action selection or force being applied over timemetadatashow_logsèdisabled®skip_as_script«code_folded$5859ca11-90f8-4fd6-88ed-c56efe796fe8cell_id$5859ca11-90f8-4fd6-88ed-c56efe796fe8code٥display_cartpole_episode((runepisode(cartpole_setup.mdps.episodic.discrete; π = reinforce_test2.policy_sample_action, max_steps = 100_000) |> x -> (x[1], x[2]))...)metadatashow_logsèdisabled®skip_as_script«code_folded$3ea08816-705e-4be7-a175-dbd3f3e4c17dcell_id$3ea08816-705e-4be7-a175-dbd3f3e4c17dcode$md"""
# Misc Utilities/Functions
"""metadatashow_logsèdisabled®skip_as_script«code_folded$f3e2db06-9cb7-464a-96b8-938175efd26bcell_id$f3e2db06-9cb7-464a-96b8-938175efd26bcodefunction setup_fcann_value_arguments(policy_setup::NamedTuple, input_length::Integer, hidden_layers::Vector{Int64}, reslayers::Integer, l2::T, dropout::T, use_μP::Bool, activation_list, scales) where {T<:Real}
scale = (reslayers == 0) ? 1 : length(hidden_layers)/(reslayers + 1) + 1
c = scale*last(hidden_layers)
f = use_μP ? one(T) / c : c^T(-0.5)
w_θ_out = T.(FCANN.makeorthonormalrand(1, last(hidden_layers)) .* f)
w_β_out = zeros(T, 1)
#value function shares its params with the policy function
value_params = deepcopy(policy_setup.params)
for i in eachindex(hidden_layers)
for j in 1:2
value_params[j][i] = policy_setup.params[j][i]
end
end
#replace the final layer of the value network with something that outputs a single value
value_params[1][end] = w_θ_out
value_params[2][end] = w_β_out
# value_params = FCANN.initializeparams_saxe(input_length, hidden_layers, 1)
#form activations for value network
value_activations = FCANN.form_activations(value_params[1])
value_activations[end] = zeros(T, 1)
value_tanh_grad_z = deepcopy(value_activations)
value_deltas = deepcopy(value_activations)
value_function(x, params) = fcann_value_function(x, params, value_activations, reslayers)
function update_value_gradient!(∇v̂, x, value_params)
update_fcann_value_gradient!(∇v̂, x, value_params, hidden_layers, l2, value_tanh_grad_z, value_activations, value_deltas, dropout, reslayers, activation_list, scales)
use_μP && scale_fcann_params!(∇v̂, policy_setup.scales)
end
return (value_params = value_params, value_gradient = deepcopy(value_params), value_function = value_function, gradient_update = update_value_gradient!)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$b2082ab0-73a4-45a6-8772-a2e6e22b519acell_id$b2082ab0-73a4-45a6-8772-a2e6e22b519acodebegin
function beta_action_sampler(p1::T, p2::T) where T<:Real
isnan(p1) && return p1
isnan(p2) && return p2
ϵ = eps(zero(T))
α = max(ϵ, exp(p1))
β = max(ϵ, exp(p2))
T(rand(Beta(α, β)))
end
beta_action_sampler(params::Vector{T}) where T<:Real = beta_action_sampler(params[1], params[2])
make_beta_n_sampler(::Val{1}) = beta_action_sampler
function make_beta_n_sampler(::Val{N}) where N
function f(params::Vector{T}) where T<:Real
ntuple(i -> beta_action_sampler(params[i], params[i+N]), N)
end
end
make_beta_n_sampler(n::Integer) = make_beta_n_sampler(Val(n))
make_beta_sampler(::T) where T<:Real = beta_action_sampler
make_beta_sampler(::NTuple{N, T}) where {N, T<:Real} = make_beta_n_sampler(N)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a361f4c9-47ce-42ad-899c-87b611c0d471cell_id$a361f4c9-47ce-42ad-899c-87b611c0d471codefunction update_binary_action_preferences!(action_preferences::Vector{T}, binary_features::BinaryFeatureVector, params::Matrix{T}) where T<:Real
@inbounds for i_a in eachindex(action_preferences)
action_preferences[i_a] = zero(T)
@simd for i in 1:binary_features.num_features
j = binary_features.active_features[i]
action_preferences[i_a] += params[j, i_a]
end
end
return action_preferences
endmetadatashow_logsèdisabled®skip_as_script«code_folded$46fea69b-599e-46ab-8455-d2da865d9a8ecell_id$46fea69b-599e-46ab-8455-d2da865d9a8ecodeFconst mountaincar_continuing_mdp = create_mountaincar_continuing_mdp()metadatashow_logsèdisabled®skip_as_scriptëcode_folded$bfe7e41d-6318-4bd4-b892-287831876abccell_id$bfe7e41d-6318-4bd4-b892-287831876abccode7begin
function update_beta_eligibility_vector!(∇lnπ::Matrix{T}, action_dist_params::Vector{T}, x::Vector{T}, action::T, policy_params::Matrix{T}) where T<:Real
α = exp(first(action_dist_params))
β = exp(last(action_dist_params))
c1 = digamma(α + β)
δ1 = (log(action) + c1 - digamma(α))*α
@inbounds @simd for i in eachindex(x)
∇lnπ[i, 1] = x[i]*δ1
end
δ2 = (log(one(T) - action) + c1 - digamma(β))*β
@inbounds @simd for i in eachindex(x)
∇lnπ[i, 2] = x[i]*δ2
end
end
function update_beta_eligibility_vector!(∇lnπ::Matrix{T}, action_dist_params::Vector{T}, x::Vector{T}, action::NTuple{N, T}, policy_params::Matrix{T}) where {N, T <: Real}
for k = 1:N
α = exp(action_dist_params[k])
β = exp(action_dist_params[k+N])
c1 = digamma(α + β)
δ1 = (log(action[k]) + c1 - digamma(α))*α
@inbounds @simd for i in eachindex(x)
∇lnπ[i, k] = x[i]*δ1
end
δ2 = (log(one(T) - action[k]) + c1 - digamma(β))*β
@inbounds @simd for i in eachindex(x)
∇lnπ[i, k+N] = x[i]*δ2
end
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$c251a630-7114-4188-9323-8d8feb5c32e0cell_id$c251a630-7114-4188-9323-8d8feb5c32e0codeCmountaincar_fcann_continuing_parameter_study(layer_size::Integer, num_layers::Integer, args...; kwargs...) = actor_critic_fcann_parameter_study(mountaincar_continuing_mdp, mountaincar_fcann_feature_setup.update_feature_vector!, mountaincar_fcann_feature_setup.num_features, fill(layer_size, num_layers), args...; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$af144759-fe66-4ad0-b378-e9eb4e859db4cell_id$af144759-fe66-4ad0-b378-e9eb4e859db4codeNplot_cartpole_policy(reinforce_test4.policy_and_value; s_ref = ep[1][ep_step])metadatashow_logsèdisabled®skip_as_script«code_folded$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2cell_id$d560b2a0-c571-4ad7-b1c9-83ec03fc8cc2codeIconst mountaincar_continuous_mdp = create_continuous_action_mountaincar()metadatashow_logsèdisabled®skip_as_script«code_folded$fb8904a9-ae64-41cc-93b6-5a25855edad0cell_id$fb8904a9-ae64-41cc-93b6-5a25855edad0codefunction get_corridor_episode_stats(p::Real; ntrials=10_000)
1:ntrials |> Map(_ -> runepisode(corridor_mdp; π = s -> (rand() < p) + 1) |> first |> length) |> foldxt(+) |> a -> a / ntrials
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a5b002c9-5e11-462a-9da0-6e060c7963f8cell_id$a5b002c9-5e11-462a-9da0-6e060c7963f8code٦const ep2 = runepisode(cartpole_setup.mdps.episodic.discrete; π = reinforce_test5.policy_sample_action, max_steps = 1000, s0 = CartPoleState(30f0, 0.8f0, 0f0, -0f0))metadatashow_logsèdisabled®skip_as_script«code_folded$83640f5b-fe13-4ec1-98a0-67a56c189ba1cell_id$83640f5b-fe13-4ec1-98a0-67a56c189ba1code0function actor_critic_with_eligibility_traces!(policy_params::P1, ∇lnπ, value_params::P2, ∇v̂, mdp::StateMDP{T, S, A, PTF, F1, F2, F3}, λ_θ::T, λ_w::T, update_action_preferences!::Function, update_eligibility_vector!::Function, x, update_feature_vector!::Function, value_function::Function, update_value_gradient!::Function, max_steps::Integer; α_w::T = one(T)/10, α_θ::T = one(T)/10, α_r̄ = one(T)/10, action_preferences = zeros(T, length(mdp.actions)), z_θ::P1 = deepcopy(policy_params), z_w::P2 = deepcopy(value_params), save_step_rewards = false) where {P1, P2, T<:Real, S, A, PTF, F1, F2, F3}
step_rewards = Vector{T}()
#initialize variables
step = 1
r̄ = zero(T)
zero_params!(z_θ)
zero_params!(z_w)
rtot = zero(T)
s = mdp.initialize_state()
update_feature_vector!(x, s)
while step <= max_steps
update_value_gradient!(∇v̂, x, value_params)
v̂ = value_function(x, value_params)
update_action_preferences!(action_preferences, x, policy_params)
soft_max!(action_preferences)
i_a = sample_action(action_preferences)
update_eligibility_vector!(∇lnπ, action_preferences, x, i_a, policy_params)
(r, s′) = mdp.ptf(s, i_a)
rtot += r
save_step_rewards && push!(step_rewards, r)
step += 1
mdp.isterm(s′) && error("$s′ is a terminal state and this method only applies to continuing tasks")
update_feature_vector!(x, s′)
v̂′ = value_function(x, value_params)
δ = r - r̄ + v̂′ - v̂
r̄ += α_r̄*δ
update_traces_with_gradient!(λ_w, z_w, ∇v̂)
update_traces_with_gradient!(λ_θ, z_θ, one(T), ∇lnπ)
update_params_with_gradient!(value_params, α_w*δ, z_w)
update_params_with_gradient!(policy_params, α_θ*δ, z_θ)
s = s′
end
function_outputs = form_state_and_policy_function_outputs(update_feature_vector!, update_action_preferences!, value_function, x, action_preferences, policy_params, value_params)
return (; step_rewards = step_rewards, total_reward = rtot, total_steps = step - 1, policy_parameters = policy_params, value_parameters = value_params, function_outputs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$61650a97-b353-4a85-b50b-93fee296ac7bcell_id$61650a97-b353-4a85-b50b-93fee296ac7bcodeqconst cartpole_fcann_feature_setup = fcann_feature_vector_setup(cartpole_setup.min_vals, cartpole_setup.max_vals)metadatashow_logsèdisabled®skip_as_script«code_folded$602a07dd-8928-4b44-97e5-01c5cbf38351cell_id$602a07dd-8928-4b44-97e5-01c5cbf38351code function plot_cartpole_policy(policy_and_value::Function; θ̇_range = 1, npoints = 100, s_ref::CartPoleState = CartPoleState())
θs = LinRange(-1.2f0, 1.2f0, npoints)
θ̇s = LinRange(-10f0, 10f0, npoints)
value_output = zeros(Float32, npoints, npoints)
policy_outputs = [zeros(Float32, npoints, npoints) for _ in 1:3]
x = s_ref.x
ẋ = s_ref.ẋ
policy_output = policy_and_value(s_ref)
policy_plot = plot(bar(x = [-1, 0, 1], y = policy_output.action_probabilities), Layout(height = 350, xaxis_title = "Policy Action", yaxis_title = "Action Probability", title = "Policy Distribution Function"))
for i in 1:npoints
for j in 1:npoints
s = CartPoleState(x, θs[i], ẋ, θ̇s[j])
output = policy_and_value(s)
value_output[i, j] = output.state_value_estimate
for i_a in 1:3
policy_outputs[i_a][i, j] = output.action_probabilities[i_a]
end
end
end
reference_trace = scatter(x = [s_ref.θ], y = [s_ref.θ̇], name = "reference state", marker_color = "black", marker_symbol = "x")
value_plot = plot([heatmap(x = θs, y = θ̇s, z = value_output, name = "value function"), reference_trace], Layout(xaxis_title = "Pole Angle in Radians", yaxis_title = "Pole Angular Velocity", title = "Value Estimate for x = $x and ẋ = $ẋ", height = 350))
policy_plots = [plot([heatmap(x = θs, y = θ̇s, z = policy_outputs[i_a], zmin = 0, zmax = 1), reference_trace], Layout(title = "Action $i_a", xaxis_title = "Pole Angle in Radians", yaxis_title = "Pole Angular Velocity", height = 350)) for i_a in 1:3]
@htl("""
$(vcat(value_plot, policy_plot))
$policy_plots
""")
# value_traces = [begin
# states = [CartPoleState(0f0, θ, 0f0, θ̇) for θ in θs]
# output = [policy_and_value(s) for s in states]
# scatter(x = θs, y = [a.state_value_estimate for a in output], name = "θ̇ = $θ̇")
# end
# for θ̇ in θ̇s]
# plot(value_traces, Layout(xaxis_title = "Pole Angle in Radians", yaxis_title = "State Value Estimate"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$f7433324-acc3-49a5-b5b3-ada0c8f09d52cell_id$f7433324-acc3-49a5-b5b3-ada0c8f09d52coderunepisode(corridor_mdp)metadatashow_logsèdisabled®skip_as_script«code_folded$0c9986bb-54c0-4b08-9c29-4bfb0b68b54ecell_id$0c9986bb-54c0-4b08-9c29-4bfb0b68b54ecode2function collect_state_distributions(;num_episodes::Integer = 1_000_000, p::T = 0.5) where T<:Real
function add_vecs(x::Array{T, N}, y::Array{T, N}) where {T<:Real, N}
l1 = size(x, 1)
l2 = size(y, 1)
(l1 == l2) && return x .+ y
if l1 > l2
out = copy(x)
for i in 1:l2
view(out, i, :) .+= view(y, i, :)
end
else
out = copy(y)
for i in 1:l1
view(out, i, :) .+= view(x, i, :)
end
end
return out
end
function π(s)
rand(T) <= p && return 1
return 2
end
counts = 1:num_episodes |> Map() do _
(states, actions, rewards, _, l) = runepisode(corridor_mdp; π = π)
state_visits = zeros(T, l, 3)
@inbounds @simd for i in eachindex(states)
s = states[i]
state_visits[i, s] += one(T)
end
return state_visits
end |> foldxt(add_vecs)
counts ./ num_episodes
endmetadatashow_logsèdisabled®skip_as_script«code_folded$6d0925d3-af96-4b94-8e2e-4941cce39e51cell_id$6d0925d3-af96-4b94-8e2e-4941cce39e51code
const mountaincar_test_train = actor_critic_with_eligibility_traces_binary_features(MountainCarTask.mdp, 0.1f0, 0.9f0, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, typemax(Int64), 100_000; α_θ = 0.008f0, α_w = 0.004f0)metadatashow_logsèdisabled®skip_as_script«code_folded$6bb0263e-368e-462a-948c-baf9cfa82512cell_id$6bb0263e-368e-462a-948c-baf9cfa82512codeget_corridor_features(s) = 1:1metadatashow_logsèdisabled®skip_as_script«code_folded$72273f27-d0b9-4645-a609-cb65cc9332eecell_id$72273f27-d0b9-4645-a609-cb65cc9332eecodeactor_critic_with_eligibility_traces_binary_features(corridor_mdp, 0f0, 0f0, get_corridor_features, 1, 100_000, α_θ = 2f0 ^ -4, α_w = 2f0 ^ -10, policy_params = [0f0 3.7f0]).policy_and_value(1)metadatashow_logsèdisabled®skip_as_script«code_folded$87482ea5-5265-4e02-92c0-1a8bb44ff0f4cell_id$87482ea5-5265-4e02-92c0-1a8bb44ff0f4codeUfunction actor_critic_binary_continuing_squashed_gaussian_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, amax::A, get_active_features::Function, num_features::Integer, λ_θ::T, λ_w::T, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, α_r̄::T, max_steps::Integer; nruns::Integer = 100, seed = rand(UInt64), init_policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), init_value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
Random.seed!(seed)
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> actor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions(mdp, λ_θ, λ_w, get_active_features, num_features, max_steps; α_θ = α_θ, α_w = α_w, α_r̄ = α_r̄, policy_params = copy(init_policy_params), value_params = copy(init_value_params), kwargs...).total_reward) |> foldxt(+) |> x -> x / nruns / max_steps
end
traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Step in the First $max_steps Steps Averaged Over $nruns Runs", xaxis_type = "log", title = "Binary Feature Encoding with $num_features Features, λ_θ = $λ_θ, λ_w = $λ_w, α_r̄ = $α_r̄"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$3bafd7df-9bc0-4d13-874d-739590cf3ad9cell_id$3bafd7df-9bc0-4d13-874d-739590cf3ad9codeSmd"""
> ### *Exercise 13.2*
> Generalize the proof of the policy gradient theorem and the steps leading to the REINFORCE update equation (13.8), so that (13.8) ends up with a factor of $\gamma^t$ and thus aligns with the general algorithm given in the pseudocode.
See proof above in the section on proving the policy gradient theorem.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$f27f2bcd-05b6-44fe-bf9e-a3e51556db7ccell_id$f27f2bcd-05b6-44fe-bf9e-a3e51556db7ccode6const cartpole_functions = create_cartpole_functions()metadatashow_logsèdisabled®skip_as_scriptëcode_folded$41dc149d-c6f3-4b0d-a856-06f3aaae3049cell_id$41dc149d-c6f3-4b0d-a856-06f3aaae3049code{mutable struct BinaryEligibilityVector{T, B <: BinaryFeatureVector}
binary_features::B
i_a::Int64
π_dist::Vector{T}
endmetadatashow_logsèdisabled®skip_as_script«code_folded$38e5d800-4d43-40d2-87ea-f7d4b4283dabcell_id$38e5d800-4d43-40d2-87ea-f7d4b4283dabcodemd"""
In order to find the p that maximizes the expected value for state 1, we should differentiate by p and set the result to 0
$\frac{\partial v_1}{\partial p} = -\frac{2p(1-p) - 2(1+p)(1 - 2p)}{p^2(1-p)^2}$
Setting this equal to 0 implies
$\begin{flalign}
p-p^2 &= 1 - 2p + p - 2p^2\\
p^2 + 2p - 1 &= 0 \\
\end{flalign}$
Using the quadratic equation, there are two solutions but since we know p has to be positive we only take that one.
$p = \frac{-2 \pm \sqrt{4 + 4}}{2} = \frac{-2 \pm 2\sqrt{2}}{2} = -1 \pm \sqrt{2} \implies p = \sqrt{2} - 1 \approx 0.41421$
So, in order to maximize the value at state 1, we have $p_{\text{left}} \approx 0.414$ and $p_{\text{right}} \approx 0.586$. That also implies that $v_1 = -2\frac{1+p}{p(1-p)} = -2\frac{\sqrt{2}}{(\sqrt{2}-1)(2 - \sqrt{2})}= \frac{-2\sqrt{2}}{2 \sqrt{2} - 2 - 2 + \sqrt{2}} = \frac{-2 \sqrt{2}}{3\sqrt{2} - 4} \approx -11.657$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2cell_id$aa797ac6-5c79-4bc2-942f-7e2c6cdfaaa2code"function one_step_actor_critic_binary_features(mdp::StateMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, max_episodes::Integer, max_steps::Integer; policy_params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
setup = setup_binary_policy_arguments(mdp, get_active_features, num_features)
one_step_actor_critic!(policy_params, setup.eligibility_vector, value_params, BinaryFeatureVector(), mdp, update_binary_action_preferences!, update_binary_eligibility_vector!, setup.feature_vector, setup.update_feature_vector!, binary_value_function, update_binary_value_gradient!, max_episodes, max_steps; action_preferences = setup.action_preferences, kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$73b90260-d57a-449a-8db6-47f91e6a4e4fcell_id$73b90260-d57a-449a-8db6-47f91e6a4e4fcode5md"""
### Eligibility Vector with Binary Features
"""metadatashow_logsèdisabled®skip_as_script«code_folded$5aba4f96-e877-457e-8e95-18737348f99fcell_id$5aba4f96-e877-457e-8e95-18737348f99fcode_actor_critic_fcann_parameter_study(mdp::StateMDP{T, S, A, P, F1, F2, F3}, update_feature_vector!::Function, num_features::Integer, hidden_layers::Vector{Int64}, params::@NamedTuple{λ_θ::T, λ_w::T, α_r̄::T, α_θ_min::Int64, α_w_min::Int64}, num_θ::Integer, num_w::Integer, max_steps::Integer; kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = actor_critic_fcann_parameter_study(mdp, update_feature_vector!, num_features, hidden_layers, params.λ_θ, params.λ_w, params.α_r̄, 2f0 .^(params.α_θ_min:params.α_θ_min+num_θ-1), 2f0 .^(params.α_w_min:params.α_w_min+num_w-1), max_steps; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$fed4dc4c-0d1c-4ee3-9d0e-8ef2a7db7486cell_id$fed4dc4c-0d1c-4ee3-9d0e-8ef2a7db7486codeِ@bind mountaincar_continuing_binary_params create_actor_critic_continuing_params_UI(λ_θ = 0.1f0, λ_w = 0.98f0, log2α_θ = -5, log2α_w = -8)metadatashow_logsèdisabled®skip_as_script«code_folded$27487ad0-4779-42ce-8def-e660ef04bee0cell_id$27487ad0-4779-42ce-8def-e660ef04bee0codeZreinforce_test4.policy_and_value(cartpole_setup.mdps.episodic.discrete.initialize_state())metadatashow_logsèdisabled®skip_as_script«code_folded$0d93132d-5819-47dc-8cf2-462d480d9c3dcell_id$0d93132d-5819-47dc-8cf2-462d480d9c3dcodeif run_mountaincar_binary_episodic_countinuous_param_study2 > 0
actor_critic_binary_episodic_squashed_gaussian_parameter_study(mountaincar_continuous_mdp, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, mountaincar_binary_continuous_params2, 4, 3, 1000; max_steps = 100_000, seed = 45)
else
md"""
Waiting to run parameter study
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$9978d537-49ff-4014-a971-b42704c50a6bcell_id$9978d537-49ff-4014-a971-b42704c50a6bcodeٍ@bind fcann_cartpole_study_params create_actor_critic_fcann_params_UI(;λ_θ = 0.95f0, λ_w = 0.2f0, h = 16, log2α_θ = -10, log2α_w = -11)metadatashow_logsèdisabled®skip_as_script«code_folded$f8215517-b18f-4a03-9421-8edab4ca8089cell_id$f8215517-b18f-4a03-9421-8edab4ca8089code`show_squashed_policy(mountaincar_continuous_test_train3.policy_function, test_mountaincar_state)metadatashow_logsèdisabled®skip_as_script«code_folded$1ac9296f-047b-4051-ba5c-0c23d5f9cde9cell_id$1ac9296f-047b-4051-ba5c-0c23d5f9cde9code>const corridor_continuing_mdp = make_corridor_continuing_mdp()metadatashow_logsèdisabled®skip_as_script«code_folded$c87dba8c-9a96-41b3-9dc7-a6c088ec1eafcell_id$c87dba8c-9a96-41b3-9dc7-a6c088ec1eafcodefshow_mountaincar_continuous_trajectory(mountaincar_continuous_test_train.policy_sample_action, 10_000)metadatashow_logsèdisabled®skip_as_script«code_folded$5cc4d12d-b537-47e2-8109-4e7a234fdf25cell_id$5cc4d12d-b537-47e2-8109-4e7a234fdf25codefunction make_corridor_mdp()
function step(s::Integer, i_a::Integer)
δ = 2*i_a - 3 #calculates the s change -1 for left (1) and 1 for right (2)
switch = iseven(s) #returns true in state 2 which is where actions are switched, when switch is true, multiply δ by -1, otherwise by 1
c = 1 - 2*switch
s′ = max(1, s + c*δ)
(-1f0, s′)
end
actions = [:left, :right]
ptf = StateMDPTransitionSampler(step, 1)
StateMDP(actions, ptf, () -> 1, s -> s == 4)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$5334064b-5a16-4135-afa0-86a48291725bcell_id$5334064b-5a16-4135-afa0-86a48291725bcode corridor_train.value_function(1)metadatashow_logsèdisabled®skip_as_script«code_folded$9c342958-1971-48ec-b919-5dfdcbc915a4cell_id$9c342958-1971-48ec-b919-5dfdcbc915a4codecmd"""
#### Change Plot Background Color $(@bind bgcolor ColorStringPicker(default = "#121212"))
"""metadatashow_logsèdisabled®skip_as_script«code_folded$966ef17c-23be-49dc-bc37-4cb52b34c049cell_id$966ef17c-23be-49dc-bc37-4cb52b34c049code$md"""
#### Neural Network Method
"""metadatashow_logsèdisabled®skip_as_script«code_folded$e7e49ff8-32df-48a4-afb2-462859592e92cell_id$e7e49ff8-32df-48a4-afb2-462859592e92code1function form_state_and_policy_function_outputs(update_feature_vector!::Function, update_action_preferences!::Function, value_function::Function, feature_vector, action_preferences::Vector, policy_params, value_params)
π! = form_state_policy_function(update_feature_vector!, update_action_preferences!)
π(s; x = deepcopy(feature_vector), action_preferences = copy(action_preferences)) = π!(x, action_preferences, s, policy_params)
π_sample(s; kwargs...) = sample_action(π(s; kwargs...))
v! = form_state_value_function(update_feature_vector!, value_function)
estimate_state_value(s; x = deepcopy(feature_vector)) = v!(x, s, value_params)
function policy_and_value(s; x = deepcopy(feature_vector), action_preferences = copy(action_preferences))
π!(x, action_preferences, s, policy_params)
v̂ = value_function(x, value_params)
return (action_probabilities = action_preferences, state_value_estimate = v̂)
end
(policy_function = π, policy_sample_action = π_sample, estimate_state_value = estimate_state_value, policy_and_value = policy_and_value)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$78c83673-2117-4542-b4d8-1c243e8f610bcell_id$78c83673-2117-4542-b4d8-1c243e8f610bcodemd"""
#### Eligibility Vector
Recall for the gaussian case and linear approximation we had:
$\begin{flalign}
\pi(a \vert s, \boldsymbol{\theta}) &= \frac{1}{\sqrt{2 \pi \sigma(s, \boldsymbol{\theta})^2}} \exp \left ( - \frac{(a - \mu(s, \boldsymbol{\theta}))^2}{2 \sigma(s, \boldsymbol{\theta})^2} \right )\\
\mu(s, \boldsymbol{\theta}) & \doteq \boldsymbol{\theta}_\mu ^ \top \mathbf{x}_\mu(s) \\
\sigma(s, \boldsymbol{\theta}) & \doteq \exp \left ( \boldsymbol{\theta}_\sigma ^ \top \mathbf{x}_\sigma(s) \right ) \\
\nabla \ln \pi(a \vert s, \boldsymbol{\theta}_\mu) &= \frac{1}{\sigma(s, \boldsymbol{\theta})^2} \left ( a - \mu(s, \boldsymbol{\theta}) \right ) \mathbf{x}_\mu(s) \\
\nabla \ln \pi(a \vert s, \boldsymbol{\theta}_\sigma) &= \left (\frac{(a - \mu(s, \boldsymbol{\theta}))^2}{\sigma(s, \boldsymbol{\theta})^2} \right )\mathbf{x}_\sigma(s) \\
\end{flalign}$
For the squashed gaussian we can apply the previous results to the new pdf:
$\begin{flalign}
\pi(a \vert s, \boldsymbol{\theta}) &= \frac{1}{\sqrt{2 \pi \sigma(s, \boldsymbol{\theta})^2}} \exp \left ( - \frac{(\tanh^{-1}(a) - \mu(s, \boldsymbol{\theta}))^2}{2 \sigma(s, \boldsymbol{\theta})^2} \right ) \left \vert \frac{1}{1 - a^2} \right \vert\\
\mu(s, \boldsymbol{\theta}) & \doteq \boldsymbol{\theta}_\mu ^ \top \mathbf{x}_\mu(s) \\
\sigma(s, \boldsymbol{\theta}) & \doteq \exp \left ( \boldsymbol{\theta}_\sigma ^ \top \mathbf{x}_\sigma(s) \right ) \\
\nabla \ln \pi(a \vert s, \boldsymbol{\theta}_\mu) &= \frac{1}{\sigma(s, \boldsymbol{\theta})^2} \left ( \tanh^{-1}(a) - \mu(s, \boldsymbol{\theta}) \right ) \mathbf{x}_\mu(s) \\
\nabla \ln \pi(a \vert s, \boldsymbol{\theta}_\sigma) &= \left (\frac{(\tanh^{-1}(a) - \mu(s, \boldsymbol{\theta}))^2}{\sigma(s, \boldsymbol{\theta})^2} \right )\mathbf{x}_\sigma(s) \\
\end{flalign}$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$a6be9a4c-d43b-4867-b7a2-07a46a9d0d8fcell_id$a6be9a4c-d43b-4867-b7a2-07a46a9d0d8fcodeّshow_mountaincar_continuous_trajectory(mountaincar_continuous_test_train_beta.policy_sample_action, 1_000; mdp = mountaincar_continuous_beta_mdp)metadatashow_logsèdisabled®skip_as_script«code_folded$396e0047-d848-462f-a769-0cc2829abc78cell_id$396e0047-d848-462f-a769-0cc2829abc78codeactor_critic_with_eligibility_traces_binary_features(corridor_mdp, .5f0, .5f0, get_corridor_features, 1, typemax(Int64), 100_000, α_θ = 2f0 ^ -4, α_w = 2f0 ^ -10, policy_params = [0f0 3.7f0]).policy_and_value(1)metadatashow_logsèdisabled®skip_as_script«code_folded$ff4f977e-48df-4c12-845c-c245b4d39d6dcell_id$ff4f977e-48df-4c12-845c-c245b4d39d6dcodefunction actor_critic_linear_parameter_study(mdp::StateMDP{T, S, A, P, F1, F2, F3}, feature_function::Function, num_features::Integer, λ_θ_list::AbstractVector{T}, λ_w_list::AbstractVector{T}, α_r̄_list::AbstractVector{T}, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, num_tests::Integer, max_steps::Integer; nruns::Integer = 100, seed = rand(UInt64), init_policy_params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), binary_features = false, kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
if binary_features
algo = actor_critic_with_eligibility_traces_binary_features
title_prefix = "Binary Feature Encoding"
else
algo = actor_critic_with_eligibility_traces_linear_features
title_prefix = "Linear Encoding"
end
run_test(α_θ, α_w, α_r̄, λ_θ, λ_w) = average_continuing_runs(nruns, seed, α_θ, α_w, α_r̄, init_policy_params, algo, mdp, λ_θ, λ_w, feature_function, num_features, max_steps; kwargs...)
test_params = [(α_θ = rand(α_θ_list), α_w = rand(α_w_list), α_r̄ = rand(α_r̄_list), λ_θ = rand(λ_θ_list), λ_w = rand(λ_w_list)) for _ in 1:num_tests]
DataFrame([begin
output = run_test(params...)
(;params..., output = output)
end
for params in test_params])
endmetadatashow_logsèdisabled®skip_as_script«code_folded$aa450da4-fe84-4eea-b6c4-9820b7982437cell_id$aa450da4-fe84-4eea-b6c4-9820b7982437code:md"""
With continuous policy parametrization, we can smoothly very action selection probabilities by arbitrarily small amounts, something that was not possible with ϵ-greedy action selection. Therefore stronger convergence guarantees are possible for policy-gradient methods than for action-value methods.
In the episodic case, assuming some particular non-random starting state $s_0$, we define the performance of a policy parametrized by *θ* as:
$\begin{align}
J(\mathbf{\theta}) \doteq v_{\pi_\mathbf{\theta}}(s_0) \tag{13.4}
\end{align}$
where $v_{\pi_\mathbf{\theta}}$ is the true value function for $\pi_\mathbf{\theta}$, the policy determined by $\mathbf{\theta}$.
The *policy gradient theorem* provides an analytic expression for the gradient of performance with respect to the policy parameter that does *not* involve the derivative of the state distribution:
$\begin{align}
\nabla J(\mathbf{\theta}) \propto \sum_s \mu (s) \sum_a q_\pi (s, a) \nabla \pi (a|s,\mathbf{\theta}) \tag{13.5}
\end{align}$
where the gradients are column vectors of partial derivatives with respect to the components of $\mathbf{\theta}$. In the episodic case, the constant of proportionality is the average length of an episode, and in the continuing case it is 1. The distribution here $\mu$ is the on-policy distribution under $\pi$.
"""
metadatashow_logsèdisabled®skip_as_script«code_folded$bb1ef180-39ac-475f-beea-ef573e71a3bfcell_id$bb1ef180-39ac-475f-beea-ef573e71a3bfcode7display_cartpole_episode((ep2 |> x -> (x[1], x[2]))...)metadatashow_logsèdisabled®skip_as_script«code_folded$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27cell_id$eae6493e-81b6-4d99-a9c6-6e75d3b3dc27codeconst cartpole_continuing_fcann_test = actor_critic_with_eligibility_traces_fcann(cartpole_continuing_mdp, 0.25f0, 0.1f0, cartpole_fcann_feature_setup.num_features, [4, 4], cartpole_vector_update!, 300_000, α_θ = 0.015f0, α_w = 0.125f0, α_r̄ = 0.01f0; save_step_rewards=true)metadatashow_logsèdisabled®skip_as_script«code_folded$5b868eba-c1af-49f6-8f93-79b78c319a6fcell_id$5b868eba-c1af-49f6-8f93-79b78c319a6fcode
#version of reinforce for general function approximation
function reinforce_with_baseline_monte_carlo_control!(policy_params, ∇lnπ, value_params, ∇v̂, mdp::ContinuousMDP{T, S, A, PTF, F1, F2, F3}, update_action_distribution!::Function, action_dist_params::Vector{T}, action_sampler::Function, update_eligibility_vector!::Function, x, update_feature_vector!::Function, value_function::Function, update_value_gradient!::Function, max_episodes::Integer; α_w::T = one(T)/10, α_θ::T = one(T)/10, γ::T = one(T), epkwargs...) where {T<:Real, S, A, PTF, F1, F2, F3}
rewards = zeros(T, max_episodes)
steps = zeros(Int64, max_episodes)
π! = form_state_continuous_policy_function(update_feature_vector!, update_action_distribution!)
π(s) = π!(x, action_dist_params, s, policy_params)
π_sample(s) = action_sampler(π(s))
v! = form_state_value_function(update_feature_vector!, value_function)
estimate_state_value(s) = v!(x, s, value_params)
state_history, action_history, reward_history, _, _ = runepisode(mdp, π_sample, max_steps = 0) #initialize variables to update episodes
for i in eachindex(rewards)
# @info "On episode $i of $max_episodes"
state_history, action_history, reward_history, sterm, nsteps = runepisode!((state_history, action_history, reward_history), mdp, π_sample, epkwargs...)
g = zero(T)
rtotal = zero(T)
#iterate through episode beginning at the end
for i in nsteps:-1:1
g = (γ * g) + reward_history[i]
update_feature_vector!(x, state_history[i])
v̂ = value_function(x, value_params)
δ = g - v̂
update_value_gradient!(∇v̂, x, value_params)
c = α_w*δ
update_params_with_gradient!(value_params, c, ∇v̂)
update_eligibility_vector!(∇lnπ, action_dist_params, x, action_history[i], policy_params)
c = α_θ * γ^(i-1) * δ
update_params_with_gradient!(policy_params, c, ∇lnπ)
rtotal += reward_history[i]
end
rewards[i] = rtotal
steps[i] = nsteps
end
π2(s; feature_vector = deepcopy(x), action_dist_params = copy(action_dist_params)) = π!(feature_vector, action_dist_params, s, policy_params)
π_sample2(s; kwargs...) = action_sampler(π2(s; kwargs...))
function policy_and_value(s::S)
π!(x, action_dist_params, s, policy_params)
v̂ = value_function(x, value_params)
return (action_distribution_parameters = action_dist_params, sampler_function = () -> action_sampler(action_dist_params), state_value_estimate = v̂)
end
return (episode_rewards = rewards, episode_steps = steps, policy_function = π2, policy_sample_action = π_sample2, policy_parameters = policy_params, estimate_state_value = estimate_state_value, value_parameters = value_params, policy_and_value = policy_and_value)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$68469a40-7976-48b7-b7a1-eaa4c5f33a18cell_id$68469a40-7976-48b7-b7a1-eaa4c5f33a18code)function plot_mountaincar_continuous_values(policy_and_value::Function; n1 = 100, n2 = 100)
xvals = LinRange(-1.2f0, 0.5f0, n1)
vvals = LinRange(-0.07f0, 0.07f0, n2)
values = zeros(Float32, n1, n2)
action_p1 = zeros(Float32, n1, n2)
action_p2 = zeros(Float32, n1, n2)
for (i, x) in enumerate(xvals)
for (j, v) in enumerate(vvals)
dist, v̂ = policy_and_value((x, v))
values[j, i] = v̂
action_p1[j, i] = dist[1]
action_p2[j, i] = dist[2]
end
end
p1 = plot(heatmap(x = xvals, y = vvals, z = values), Layout(xaxis_title = "position", yaxis_title = "velocity", title = "Learned Value Function", height = 400))
p2 = plot(heatmap(x = xvals, y = vvals, z = action_p1, colorscale = "rb"), Layout(xaxis_title = "position", yaxis_title = "velocity", title = "Policy Parameter 1", height = 400))
p3 = plot(heatmap(x = xvals, y = vvals, z = action_p2, colorscale = "rb"), Layout(xaxis_title = "position", yaxis_title = "velocity", title = "Policy Parameter 2", height = 400))
@htl("""
$p1
$p2
$p3
""")
endmetadatashow_logsèdisabled®skip_as_script«code_folded$2a586e46-66e4-461a-85c8-5817e4d1aa43cell_id$2a586e46-66e4-461a-85c8-5817e4d1aa43codemd"""
$\begin{flalign}
\nabla J(\boldsymbol{\theta}) &= \nabla v_\pi(s_0) \\
&= \sum_s \sum_k \gamma^k \Pr \{ s_0 \rightarrow s, k, \pi \} f(s) \\
&= \sum_s \sum_k \gamma^k \frac{\sum_{x \in \mathcal{S}} \sum_{t = 0}^\infty \Pr \{ s_0 \rightarrow x, t, \pi \}}{\sum_{x \in \mathcal{S}} \sum_{t = 0}^\infty \Pr \{ s_0 \rightarrow x, t, \pi \}} \Pr \{ s_0 \rightarrow s, k, \pi \} f(s) \tag{multiply by 1}\\
&= \eta \sum_s \sum_k \gamma^k \frac{\Pr \{ s_0 \rightarrow s, k, \pi \}}{\sum_{x \in \mathcal{S}} \sum_{t = 0}^\infty \Pr \{ s_0 \rightarrow x, t, \pi \}} f(s) \tag{average episode length}\\
&= \eta \sum_s \sum_k \gamma^k \mu_\pi(s, k) f(s) \tag{on policy distribution over states and steps}\\
&= \eta \mathbb{E}_\pi[ \gamma^k f(s) \mid S_0 = s_0, S_k = s] \tag{definition of expected value}\\
&\propto \mathbb{E}_\pi \left [ \gamma^k \sum_a \nabla \pi(a \vert s) q_\pi(s, a) \mid S_0 = s_0, S_k = s \right ] \tag{13.5}\\
\end{flalign}$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$a206c759-3f6e-4003-8cba-5f6ce6742646cell_id$a206c759-3f6e-4003-8cba-5f6ce6742646codeٿmd"""
### Figure 13.1
REINFORCE on short-corridor gridworld (Example 13.1). Performance varies with step size but can approach the ideal. Feature vector encodes every state identically.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$fc3dcd26-c5cf-4141-bf6c-eaed5fc9bb1dcell_id$fc3dcd26-c5cf-4141-bf6c-eaed5fc9bb1dcodemd"""
Consider the linear parameterization proposed with $h_a = \boldsymbol{\theta}^\top \mathbf{x}(s, a)$:
$\frac{\partial{h_a}}{\partial{\theta_i}} = \mathbf{x}(s, a)_i \implies \nabla(\pi(a \vert s, \boldsymbol{\theta}))_i = \pi_a \left ( \mathbf{x}(s, a)_i - \sum_k \pi_k \mathbf{x}(s, k)_i \right)$
Now consider $\mathbf{h} = \theta ^ \top \mathbf{x}$ with $h_a = \mathbf{h}_a$. Since the parameters are now represented as a matrix, we can also index the gradient partial derivatives such that $\nabla \left ( f(\theta) \right )_{i, j} = \frac{\partial f(\theta)}{\theta_{i, j}}$
$\frac{\partial{h_a}}{\partial{\theta_{i, j}}} = \begin{cases}
\mathbf{x}(s)_i, & \text{ if } j = a \\
0, & \text{ else }
\end{cases}
\implies \nabla(\pi(s, \boldsymbol{\theta})_a)_{i, j} = \pi_a \left ( \frac{\partial h_a}{\partial \theta_{i, j}} - \sum_k \pi_k \frac{\partial h_k}{\partial \theta_{i, j}} \right)=\pi_a \begin{cases}
\mathbf{x}(s)_i (1 - \pi_j), & \text{ if } j = a \\
-\pi_j \mathbf{x}(s)_i, & \text{ else }\\
\end{cases}$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$3cfd63ad-b1a2-4b99-ae97-2ff10351e4f5cell_id$3cfd63ad-b1a2-4b99-ae97-2ff10351e4f5code+md"""
### Beta Distribution Alternative
"""metadatashow_logsèdisabled®skip_as_script«code_folded$31db0f58-28e4-454f-9394-25565687266fcell_id$31db0f58-28e4-454f-9394-25565687266fcodexdisplay_cartpole_episode((runepisode(cartpole_mdps.episodic.continuous, s -> Float32(randn())) |> x -> (x[1], x[2]))...)metadatashow_logsèdisabled®skip_as_script«code_folded$822e4d69-2582-4956-858e-06ecb091e76acell_id$822e4d69-2582-4956-858e-06ecb091e76acode^function display_cartpole_episode(states::Vector{S}, actions::Vector) where S<:CartPoleState
fields = [:x, :θ, :ẋ, :θ̇]
names = ["x", "θ", "ẋ", "θ̇"]
yaxes = ["y", "y2", "y", "y2"]
x = [s.t for s in states] #time history in seconds
state_traces = [begin
y = [getfield(s, f) for s in states]
scatter(x = x, y = y, name = names[i], yaxis = yaxes[i])
end
for (i, f) in enumerate(fields)]
plot(state_traces, Layout(xaxis_title = "Time(s)", yaxis_title = "Horizontal Position", yaxis2 = attr(title = "Pole Angle (Radians)", overlaying = "y", side = "right"), legend_orientation = "h"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$d7f6ff79-3c0f-4f16-aa1c-3bc534ce580acell_id$d7f6ff79-3c0f-4f16-aa1c-3bc534ce580acodeVplot_mountaincar_continuous_values(mountaincar_continuous_test_train.policy_and_value)metadatashow_logsèdisabled®skip_as_script«code_folded$05b0fcad-628b-48d2-aa24-f6f562dbb660cell_id$05b0fcad-628b-48d2-aa24-f6f562dbb660code_md"""
$\begin{flalign}
&\gamma^2 \sum_a \left [ \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \sum_{a^\prime} \pi(a^\prime \vert s^\prime) \sum_{s^{\prime \prime}} p(s^{\prime \prime} \vert s^\prime, a^\prime) \sum_{a^{\prime \prime}} [ \nabla \pi(a^{\prime \prime} \vert s^{\prime \prime}) q_\pi(s^{\prime \prime}, a^{\prime \prime})\right ] \\
&\gamma^2 \sum_a \left [ \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \sum_{a^\prime} \pi(a^\prime \vert s^\prime) \sum_{s^{\prime \prime}} p(s^{\prime \prime} \vert s^\prime, a^\prime) f(s^{\prime \prime}) \right ] \\
&\gamma^2 \sum_a \left [ \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \sum_{s^{\prime \prime}} f(s^{\prime \prime}) \sum_{a^\prime} \pi(a^\prime \vert s^\prime) p(s^{\prime \prime} \vert s^\prime, a^\prime) \right ] \\
&\gamma^2 \sum_a \left [ \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \mathbb{E}_\pi[f(s^{\prime \prime}) \vert s^\prime] \right ] \\
&\gamma^2 \mathbb{E}_\pi[f(s^{\prime \prime}) \vert s] \\
&\gamma^2 \sum_a \left [ \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) g(s^\prime) \right ] \\
&\gamma^2 \sum_a \left [ \pi(a \vert s) \mathbb{E}[g(s^\prime) \vert s, a] \right ] \\
&\gamma^2 \mathbb{E}_\pi[g(s^\prime) \vert s]\\
&\gamma^2 \sum_{s^{\prime \prime}} \Pr(s \rightarrow s^{\prime \prime}, 2, \pi) f(s^{\prime \prime})
\end{flalign}$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$d2729657-d0bf-4d39-8ec7-f242a1ad48d6cell_id$d2729657-d0bf-4d39-8ec7-f242a1ad48d6codefunction create_continuous_action_mountaincar_beta()
#if we sample actions from a beta distribution then the action will always be bounded between 0 and 1. this step function rescales it to -1 to 1
mdp = MountainCarTask.mdp
function step(s, a)
f = 2f0*(a - 0.5f0)
(-1f0, MountainCarTask.step(s, f))
end
ContinuousMDP(step, mdp.initialize_state, 0f0; isterm = mdp.isterm)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$5c11a92d-7496-4aba-af15-2537eac49dd7cell_id$5c11a92d-7496-4aba-af15-2537eac49dd7code Map(_ -> actor_critic_with_eligibility_traces_fcann(mdp, λ_θ, λ_w, num_features, hidden_layers, update_feature_vector!, max_episodes, max_steps; α_θ = α_θ, α_w = α_w, kwargs...) |> x -> isempty(x.episode_rewards) ? missing : mean(x.episode_rewards)) |> Filter(!ismissing) |> tcollect |> x -> isempty(x) ? missing : mean(x)
end
traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Episode in the First $max_episodes Episodes Averaged Over $nruns Runs", xaxis_type = "log", title = "$num_features Inputs, $hidden_layers Hidden Non Linear, λ_θ = $λ_θ, λ_w = $λ_w"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$76eb6743-cac0-4174-9ba3-a0691c200b54cell_id$76eb6743-cac0-4174-9ba3-a0691c200b54code٩begin
make_n_param_dist_params(n::Integer, ::T) where T<:Real = zeros(T, n)
make_n_param_dist_params(n::Integer, ::NTuple{N, T}) where {N, T<:Real} = zeros(T, n*N)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$94517664-6988-44dc-a297-e9d5873ee540cell_id$94517664-6988-44dc-a297-e9d5873ee540codeN@bind squashed_gaussian_plot_params PlutoUI.combine() do Child
md"""
### Squashed Gaussian Plot Parameters
$$\mu$$: $(Child(Slider(-4:.1:4, default = 0, show_value=true)))
$$\sigma$$: $(Child(Slider(0.1:0.1:2, default = .5, show_value=true)))
maximum value: $(Child(Slider(.1:0.1:2., default = 1, show_value=true)))
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$d037ea92-915c-4dc7-97c6-d006d92e088acell_id$d037ea92-915c-4dc7-97c6-d006d92e088acodeefunction figure_13_1(α_list; nruns = 100, num_episodes = 1_000, max_steps = 1_000)
Random.seed!(45)
function average_runs(α)
1:nruns |> Map(_ -> reinforce_monte_carlo_control_binary_features(corridor_mdp, get_corridor_features, 1, num_episodes; params = [0f0 3.7f0], α = α, max_steps = max_steps).episode_rewards) |> foldxt((a, b) -> a .+ b) |> v -> v ./ nruns
end
traces = [begin
out = average_runs(α)
scatter(x = 1:num_episodes, y = out, name = "α = 2^$(round(Int64, log2(α)))")
end
for α in α_list]
baselinetrace = scatter(x = 1:num_episodes, y = fill(-2*sqrt(2) / (3*sqrt(2) - 4), num_episodes), name = "ideal value", line_dash = "dash", line_color = "gray")
plot([baselinetrace; traces], Layout(yaxis_range = [-90, -10], yaxis_title = "Total reward on episode (averaged over $nruns runs)", xaxis_title = "Episode", width = 800))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$24fa139c-ad4b-49db-ac8f-23c476ed8608cell_id$24fa139c-ad4b-49db-ac8f-23c476ed8608codeconst reinforce_test = reinforce_with_baseline_monte_carlo_control_binary_features_gaussian_actions(cartpole_setup.mdps.episodic.continuous, cartpole_setup.get_active_features, cartpole_setup.num_features, 10_000; α_θ = 2f0 ^-14, α_w = 2f0 ^-6)metadatashow_logsèdisabled®skip_as_script«code_folded$2025ff38-f2ec-4224-b771-ff72ffe1af28cell_id$2025ff38-f2ec-4224-b771-ff72ffe1af28code.const mountaincar_min_vals = (-1.2f0, -0.07f0)metadatashow_logsèdisabled®skip_as_script«code_folded$cb70d400-3e9c-441c-b17c-e727e8c928f3cell_id$cb70d400-3e9c-441c-b17c-e727e8c928f3codeif start_mountaincar_continuing_fcann_param_study > 0
mountaincar_fcann_continuing_parameter_study(32, 3, mountaincar_continuing_fcann_params, 5, 3, 1_000_000; seed = 45)
else
md"""
Waiting to run parameter study
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$e034b9cb-f4ee-46f4-bea6-72c93c75d966cell_id$e034b9cb-f4ee-46f4-bea6-72c93c75d966codeusing DataFramesmetadatashow_logsèdisabled®skip_as_script«code_folded$e6cf9550-2e69-4b82-92cf-5e07a35490aacell_id$e6cf9550-2e69-4b82-92cf-5e07a35490aacodebegin
zero_params!(params::Array{T, N}) where {N, T<:Real} = params .= zero(T)
function zero_params!(params::FCANNParams)
for i = 1:2
for j in eachindex(params[i])
zero_params!(params[i][j])
end
end
end
end metadatashow_logsèdisabled®skip_as_script«code_folded$717e4c69-59d5-4929-923f-dd35a97fb160cell_id$717e4c69-59d5-4929-923f-dd35a97fb160codeactor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, λ_θ::T, λ_w::T, get_active_features::Function, num_features::Integer, args...; kwargs...) where {T<:Real, S, N, A <: Union{T, NTuple{N, T}}, P, F1, F2, F3} = actor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions(mdp, one(T), λ_θ, λ_w, get_active_features, num_features, args...; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$1386ffdb-940d-4f1b-a872-4e38647b5335cell_id$1386ffdb-940d-4f1b-a872-4e38647b5335codemd"""
#### Test One-step Actor-Critic
The following function calls execute the One-step Actor-Critic algorithm on Example 13.1. The output displayed is the policy function acting on the single state representation for the problem. The two values represent the probability of taking the left and right action respectively. If converged properly, the right action probability should be higher, approaching a value of about 60%.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$a893a87b-2d07-4db5-9d1a-9da8646216f4cell_id$a893a87b-2d07-4db5-9d1a-9da8646216f4codefunction update_params_with_gradient!(w::Vector{T}, α::T, ∇w::BinaryFeatureVector) where {T<:Real}
@inbounds @simd for i in 1:∇w.num_features
j = ∇w.active_features[i]
w[j] += α
end
return w
endmetadatashow_logsèdisabled®skip_as_script«code_folded$2cbc972b-c685-4c1c-8a8d-9d58b197ad90cell_id$2cbc972b-c685-4c1c-8a8d-9d58b197ad90codeٹfunction update_binary_value_params!(params::Vector{T}, active_features::BinaryFeatures, c::T) where T<:Real
@inbounds for i in active_features
params[i] += c
end
return params
endmetadatashow_logsèdisabled®skip_as_script«code_folded$37ec6802-d4c2-4470-ad69-439d5a732f77cell_id$37ec6802-d4c2-4470-ad69-439d5a732f77code function form_state_policy_function(update_feature_vector!::Function, update_action_preferences!::Function)
function π!(x, action_preferences, s, params)
update_feature_vector!(x, s)
update_action_preferences!(action_preferences, x, params)
soft_max!(action_preferences)
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$98222fcd-b456-477c-90dd-844df36877e5cell_id$98222fcd-b456-477c-90dd-844df36877e5codeKplot_continuing_step_rewards(mountaincar_continuing_tile_test.step_rewards)metadatashow_logsèdisabled®skip_as_script«code_folded$f7f58fd2-facc-4b87-9172-5e911677c8f4cell_id$f7f58fd2-facc-4b87-9172-5e911677c8f4codez#for an episode progressing, show the point in the state space that the cart exsits and use the value of x and ẋ in thatmetadatashow_logsèdisabled®skip_as_script«code_folded$58403c8e-0ee4-4466-ba25-ee0c86fb0b47cell_id$58403c8e-0ee4-4466-ba25-ee0c86fb0b47code md"""
Consider $\mathbf{x}(s)$ and $\mathbf{h}(s, \boldsymbol{\theta})$ which produces a vector of action preferences. We would like to derive an expression for $\nabla \ln \pi (a \vert s, \boldsymbol{\theta})$ in the case of $\mathbf{\pi}(s, \boldsymbol{\theta}) = \sigma(\mathbf{h}(s, \boldsymbol{\theta}))$ where $\sigma(\mathbf{x})$ is the softmax function defined in section 13.1. Here I'm using the notation $\mathbf{\pi}(s, \boldsymbol{\theta})$ to refer to the vector of action probabilities at a given state. The subscript on the vector refers to selecting that element from the vector. To shorten expressions, the following terms are equivalent:
$\begin{flalign}
\mathbf{\pi} &\doteq \mathbf{\pi}(s, \boldsymbol{\theta}) \\
\mathbf{h} &\doteq \mathbf{h}(s, \boldsymbol{\theta}) \\
x_i &\doteq \mathbf{x}_i \text{ for all vectors} \\
\end{flalign}$
Using these conventions, we previously had an expression for the ith component of the gradient of the policy:
$\nabla \left( \pi_a \right )_i = \pi_a \left ( \frac{\partial{h_a}}{\partial{\theta_i}} - \sum_k{\pi_k \frac{\partial{h_k}}{\partial{\theta_i}}} \right )$
We can use this expression to derive the components of the eligibility vector in general:
$\begin{flalign}
\nabla \left( \ln \mathbf{\pi}_a \right)_i &= \frac{\nabla \left( \pi_a \right )_i}{\pi_a}\\
&=\frac{\partial{h_a}}{\partial{\theta_i}} - \sum_k{\pi_k \frac{\partial{h_k}}{\partial{\theta_i}}} \\
\end{flalign}$
### Connection to Cross-Entropy Loss
Classification problems involve training a function to predict the class label of an input. The function returns a vector of class preferences which can be converted to a probability distribution by the soft-max function. The cross-entropy loss is a way of comparing this distribution with the desired output label to generate an error value.
Let's denote $\mathbf{p}(s)$ as the vector of true probabilities for an example $s$ and keep our output function as $\pi(s,\theta) = \sigma(\mathbf{h}(s, \boldsymbol{\theta}))$. The cross entropy loss is defined as:
$\mathcal{L}(\mathbf{p}, \mathbf{\pi}) = -\sum_i \mathbf{p}_i \ln \mathbf{\pi}_i$ omitting $s$ and $\boldsymbol{\theta}$.
In a typical situation with a dataset, $\mathbf{p}(s)$ will be a one-hot vector representing the index of label of the example in the dataset. Let's call that index $a$ such that $p_a = 1$ and $p_i = 0 \: \forall i \neq a$. The loss then simplifies to $\mathcal{L}(a, \mathbf{\pi}) = -\ln \mathbf{\pi}_a$. When we train with gradient descent on such a dataset, we must compute the gradient of this loss with respect to the parameters or $-\nabla \ln \pi_a$ which is just negative one times the eligibility vector for general paramaterized approximation. So if we have a function that computes the gradient of the cross entropy loss of the soft-max output for a vector function and a label index, we can replace the label index of the dataset with the desired action index $a$ and then that gradient will match our desired gradient after multiplying by negative one.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$e1aec891-d95a-47d1-97d7-d2a4cfb16e64cell_id$e1aec891-d95a-47d1-97d7-d2a4cfb16e64codefunction setup_fcann_policy_and_value_arguments(policy_params::FCANNParams{T}, input_length::Integer, hidden_layers::Vector{Int64}, reslayers::Integer, l2::T, dropout::T, use_μP::Bool, activation_list) where {T<:Real}
policy_setup = setup_fcann_policy_arguments(policy_params::FCANNParams{T}, input_length::Integer, hidden_layers::Vector{Int64}, reslayers::Integer, l2::T, dropout::T, use_μP::Bool, activation_list)
value_setup = setup_fcann_value_arguments(policy_setup, input_length::Integer, hidden_layers::Vector{Int64}, reslayers::Integer, l2::T, dropout::T, use_μP::Bool, activation_list, policy_setup.scales)
(;policy_setup..., value_setup...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$3d065608-eef2-4caa-b17d-ec60714e3d58cell_id$3d065608-eef2-4caa-b17d-ec60714e3d58code;actor_critic_binary_episodic_beta_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, params::@NamedTuple{λ_θ::T, λ_w::T, α_θ_min::Int64, α_w_min::Int64}, num_θ::Integer, num_w::Integer, num_episodes::Integer; kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = actor_critic_binary_episodic_beta_parameter_study(mdp, get_active_features, num_features, params.λ_θ, params.λ_w, 2f0 .^(params.α_θ_min:params.α_θ_min+num_θ-1), 2f0 .^(params.α_w_min:params.α_w_min+num_w-1), num_episodes; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$b87ff1a9-abff-40f7-a1d8-f751a1c8b060cell_id$b87ff1a9-abff-40f7-a1d8-f751a1c8b060code9md"""
In the episodic case, we provided a reward of -1 per step and then considered an episode finished when a failure state was reached. In the continuing case, the step function will provide a reward of 0 unless a failure occurs in which case it will provide a reward of -1 and then initialize a new state.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$e89bdc84-dbb5-4c73-a39c-6392e5f79704cell_id$e89bdc84-dbb5-4c73-a39c-6392e5f79704codeمplot_mountaincar_values(mountaincar_continuing_tile_test.estimate_state_value, mountaincar_continuing_tile_test.policy_sample_action)metadatashow_logsèdisabled®skip_as_script«code_folded$d3b56fca-5b79-4465-8987-8d0005f854d8cell_id$d3b56fca-5b79-4465-8987-8d0005f854d8codeconst reinforce_test2 = reinforce_with_baseline_monte_carlo_control_binary_features(cartpole_setup.mdps.episodic.discrete, cartpole_setup.get_active_features, cartpole_setup.num_features, 10_000; α_θ = 2f0 ^-14, α_w = 2f0 ^-8)metadatashow_logsèdisabled®skip_as_script«code_folded$d21617aa-6f38-4a90-8586-4b32022497adcell_id$d21617aa-6f38-4a90-8586-4b32022497adcode'cartpole_setup.mdps.continuing.discretemetadatashow_logsèdisabled®skip_as_script«code_folded$0574f5a0-72e7-4aa2-80ac-f4ce4f0fe7c2cell_id$0574f5a0-72e7-4aa2-80ac-f4ce4f0fe7c2codeْplot_cartpole_policy(cartpole_continuing_test.policy_and_value; s_ref = CartPoleState(sref_cartpole_binary.x, 0f0, sref_cartpole_binary.ẋ, 0f0))metadatashow_logsèdisabled®skip_as_script«code_folded$5eb8d9f9-8512-4e00-8cb5-cec68d73cc7dcell_id$5eb8d9f9-8512-4e00-8cb5-cec68d73cc7dcode;const mountaincar_continuous_test_train3 = actor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions(mountaincar_continuous_mdp, 0.2f0, 0.99f0, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, typemax(Int64), 1_000_000; α_θ = 1f-5, α_w = 0.0001f0)metadatashow_logsèdisabled®skip_as_script«code_folded$d82e7ab8-c372-4462-afb5-1617560cdb56cell_id$d82e7ab8-c372-4462-afb5-1617560cdb56codeّplot_mountaincar_values(mountaincar_continuous_test_train_beta.estimate_state_value, mountaincar_continuous_test_train_beta.policy_sample_action)metadatashow_logsèdisabled®skip_as_script«code_folded$3c89209c-9202-4d5d-841c-ea34be369616cell_id$3c89209c-9202-4d5d-841c-ea34be369616codeHconst cartpole_continuing_test = actor_critic_with_eligibility_traces_binary_features(cartpole_continuing_mdp, 0.95f0, 0.8f0, s -> cartpole_tilecoding_setup.get_active_features((s.x, s.θ, s.ẋ, s.θ̇)), cartpole_tilecoding_setup.num_features, 30_000, α_θ = .125f0, α_w = 0.006f0, α_r̄ = 0.01f0, save_step_rewards = true)metadatashow_logsèdisabled®skip_as_script«code_folded$635abb34-2c97-4f04-a74c-22fbec32f408cell_id$635abb34-2c97-4f04-a74c-22fbec32f408codefunction fcann_value_function(x::Vector{T}, params::FCANNParams, activations::FCANNActivations{T}, reslayers::Integer) where T<:Float32
FCANN.forwardNOGRAD_base!(activations, params..., x, reslayers)
return first(last(activations))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$0bf3b988-b3fb-49d5-8dde-b25766596363cell_id$0bf3b988-b3fb-49d5-8dde-b25766596363codeMlinear_value_function(x::Vector{T}, w::Vector{T}) where {T<:Real} = dot(x, w)metadatashow_logsèdisabled®skip_as_script«code_folded$d8222abf-139c-4220-8e92-cc987ec6900ccell_id$d8222abf-139c-4220-8e92-cc987ec6900ccodemd"""
Note that for the corridor problem, the state-value learning rates have very little impact and learning is most effective when $\lambda_{\boldsymbol{\theta}}$ is close to 1 which mimics REINFORCE with baseline.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$68e6f17e-8c87-40f0-a673-1115ecd1b71dcell_id$68e6f17e-8c87-40f0-a673-1115ecd1b71dcodepmd"""
> ### *Exercise 13.5*
> A *Bernoulli-logistic unit* is a stochastic neuron-like unit used in some ANNs. Its input at time *t* is a feature vector $\mathbf{x}(S_t)$; its output, $A_t$, is a random variable having two values, 0 and 1, with $\Pr \{A_t=1 \}=P_t$ and $\Pr\{A_t=0\}=1-P_t$ (the Bernoulli distribution). Let $h(s, 0, \mathbf{\theta})$ and $h(s, 1, \mathbf{\theta})$ be the preferences in state $s$ for the unit's two actions given by policy parameter $\mathbf{\theta}$. Assume that the difference between the action preferences is given by a weights sum of teh unit's input vector, that is, assume that $h(s, 1, \mathbf{\theta})-h(s,0, \mathbf{\theta}) = \mathbf{\theta}^\top \mathbf{x}(s)$, where $\mathbf{\theta}$ is the unit's weight vector.
> 1. Show that if the exponential soft-max distribution (13.2) is used to convert action preferences to policies, then ${P_t = \pi(1|S_t, \theta_t)=1/(1+\exp(-\theta_t^\top\mathbf{x}(S_t)))}$ (the logistic function).
> 2. What is the Monte-Carlo REINFORCE update of $\theta_t$ to $\theta_{t+1}$ upon receipt of return $G_t$?
> 3. Express the eligility $\nabla \ln \pi(a|s, \theta)$ for a Bernoulli-logistic unit, in terms of $a$, $\mathbf{x}(s)$, and $\pi(a|s, \theta)$ by calculating the gradient.
> Hint for part (c): Define $P=\pi(1|s,\theta)$ and compute the derivative of the logarithm, for each action, using the chain rule on $P$. Combine the two results into one expression that depends on $a$ and $P$, and then use the chain rule again, this time on $\theta^\top\mathbf{x}(s)$, noting that the derivative of the logistic function $f(x)=1/(1+e^{-x})$ is $f(x)(1-f(x))$.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$cf1859d6-f889-4923-8c87-2d7c039f26c3cell_id$cf1859d6-f889-4923-8c87-2d7c039f26c3codeDrunepisode(cartpole_mdps.episodic.continuous, s -> Float32(randn()))metadatashow_logsèdisabled®skip_as_script«code_folded$5500fd8e-64cb-4af7-808d-230440746319cell_id$5500fd8e-64cb-4af7-808d-230440746319code/md"""
### *Continuing Mountain Car Example*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$76d54520-baa3-44bf-b303-4cdcb8b87080cell_id$76d54520-baa3-44bf-b303-4cdcb8b87080codeكbegin
make_sample_vector(::T) where T<:Real = zeros(T, 1)
make_sample_vector(::NTuple{N, T}) where {N, T<:Real} = zeros(T, N)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$27441783-d3c6-40be-9c36-4941613e6ae9cell_id$27441783-d3c6-40be-9c36-4941613e6ae9codezplot(reinforce_test5.step_rewards |> cumsum |> x -> x ./ length(x) |> x -> x[round.(Int64, LinRange(1, length(x), 1000))])metadatashow_logsèdisabled®skip_as_script«code_folded$fac138d9-3c5d-44b0-a87c-b13872f19450cell_id$fac138d9-3c5d-44b0-a87c-b13872f19450codeusing Memoizemetadatashow_logsèdisabled®skip_as_script«code_folded$82e0e9a0-9662-429a-87e3-e6bdae02709acell_id$82e0e9a0-9662-429a-87e3-e6bdae02709acodefconst reinforce_test5 = actor_critic_with_eligibility_traces_fcann(cartpole_setup.mdps.continuing.discrete, 0.90f0, 0.1f0, cartpole_fcann_feature_setup.num_features, [32, 32], (x, s) -> cartpole_fcann_feature_setup.update_feature_vector!(x, (s.x, s.θ, s.ẋ, s.θ̇)), 1_000_000; α_θ = 0.0625f0, α_w = 0.0625f0, α_r̄ = 0.01f0, save_step_rewards = true)metadatashow_logsèdisabled®skip_as_script«code_folded$d3c1379f-acd6-4e15-be7e-a5dbe46a4f62cell_id$d3c1379f-acd6-4e15-be7e-a5dbe46a4f62code_@bind start_mountaincar_continuing_param_study CounterButton("Run Mountaincar Parameter Study")metadatashow_logsèdisabled®skip_as_script«code_folded$fad02876-efba-46a7-9cb7-43820528779fcell_id$fad02876-efba-46a7-9cb7-43820528779fcodeٽplot_cart(cartpole_fcann_continuing_test_episode[1][cartpole_fcann_continuing_episode_step_select], cartpole_fcann_continuing_test_episode[2][cartpole_fcann_continuing_episode_step_select])metadatashow_logsèdisabled®skip_as_script«code_folded$1ce4bc6c-7cde-48e9-8ff1-7281697fd121cell_id$1ce4bc6c-7cde-48e9-8ff1-7281697fd121code-plot_cart(ep2[1][ep2_step], ep2[2][ep2_step])metadatashow_logsèdisabled®skip_as_script«code_folded$024dcd1a-8eaa-4a95-8037-2f578828309ccell_id$024dcd1a-8eaa-4a95-8037-2f578828309ccode-const cartpole_mdps = create_cartpole_mdps()metadatashow_logsèdisabled®skip_as_script«code_folded$e1274f57-75cb-4659-a82f-e5870c5367e2cell_id$e1274f57-75cb-4659-a82f-e5870c5367e2codeyconst ep = runepisode(cartpole_setup.mdps.episodic.discrete; π = reinforce_test4.policy_sample_action, max_steps = 1000)metadatashow_logsèdisabled®skip_as_script«code_folded$fdd3f4fd-4706-4d6b-b150-6ee6b4b370cbcell_id$fdd3f4fd-4706-4d6b-b150-6ee6b4b370cbcodemd"""
### Notes on Probability Distributions
In order to prove the policy gradient theorem, we must manipulate terms that are probability distributions over states and visit steps. In order to build intuition for these distributions, we can visualize how data is being averaged with the sort corridor example. The following function simulates many episodes in the environment with a stochastic policy that has some probability of moving left regardless of the state. The simulation keeps track of the visit count for a given state and the visit step. The result of the accumulation is a matrix who's columns contain the number of times each state was visited on every step of an episode across all of the simulated episodes. If we divide each count by the number of episodes simulated, then we have an unbiased sample of the probability of visiting a state on each step $k$ of an episode: $\Pr \{ S_k = s \mid \pi \}$ such that $\sum_{s \in \mathcal{S}^+} \Pr \{ S_k = s \mid \pi \} = 1$.
Note that this distribution is only normalized over the sum of all states including terminal states which is denoted in episodic problems by the notation $\mathcal{S}^+$. The notation $\mathcal{S}$ excludes all terminal states, so if we sum the above probabilities over that set on a given step $k$ we calculate the probability that we are NOT in a terminal state by the time we reach step $k$: $\sum_\mathcal{S} \Pr \{ S_k = s \mid \pi \} = \Pr \{ T \gt k \mid \pi \}$ where we use the notation that $T$ is the step of termination for a particular episode.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$b02ba928-5b9f-4695-b980-07988c788bb9cell_id$b02ba928-5b9f-4695-b980-07988c788bb9code8const mountaincar_continuing_tile_test = actor_critic_with_eligibility_traces_binary_features(mountaincar_continuing_mdp, 0.1f0, 0.98f0, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, 200_000, α_θ = 0.5f0, α_w = 0.0025f0, α_r̄ = 0.005f0; save_step_rewards=true)metadatashow_logsèdisabled®skip_as_script«code_folded$f946c886-6246-4f98-a96f-f06984691ad8cell_id$f946c886-6246-4f98-a96f-f06984691ad8codebegin
function ApproximationUtils.runepisode!((states, actions, rewards)::Tuple{Vector{S}, Vector{A}, Vector{T}}, mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, π::Function; s0::S = mdp.initialize_state(), a0::A = π(s0), max_steps = typemax(Int64)) where {T<:Real, S, A, P, F1<:Function, F2<:Function, F3<:Function}
s = s0
l = length(states)
@assert l == length(actions) == length(rewards)
function add_value!(v, x, i)
if i > l
push!(v, x)
else
v[i] = x
end
end
add_value!(states, s, 1)
a = a0
# @info "Selected action is $a"
(r, s′) = mdp.ptf(s, a0)
add_value!(actions, a, 1)
add_value!(rewards, r, 1)
step = 2
sterm = s
if mdp.isterm(s′)
sterm = s′
else
sterm = s
end
s = s′
#note that the terminal state will not be added to the state list
while !mdp.isterm(s) && (step <= max_steps)
add_value!(states, s, step)
a = π(s)
if bad_continuous_action(a)
@info "Terminating episode after $step steps due to bad continuous action $a taken in state $s"
step = 1
break
end
# @info "Selected action is $a"
(r, s′) = mdp.ptf(s, a)
add_value!(actions, a, step)
add_value!(rewards, r, step)
s = s′
step += 1
if mdp.isterm(s′)
sterm = s′
end
end
return states, actions, rewards, sterm, step-1
end
function ApproximationUtils.runepisode(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, π::Function; kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
states = Vector{S}()
actions = Vector{A}()
rewards = Vector{T}()
runepisode!((states, actions, rewards), mdp, π; kwargs...)
end
ApproximationUtils.runepisode(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}; kwargs...) where {T<:Real, S, N, A <: Union{T, NTuple{N, T}}, P, F1, F2, F3} = runepisode(mdp, Returns(rand(A)); kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$3c316495-bb6c-41e2-a38f-ba867a319fbbcell_id$3c316495-bb6c-41e2-a38f-ba867a319fbbcode #create a cart pole MDP environment
function create_cartpole_mdps(;
m::T = 1f0, #mass at the end of the pole in kg
m_c::T = 10f0, #mass of the cart in kg
l::T = 1f0, #length of the pole in meters
g::T = 9.8f0, #gravitational constant in meters per second squared
h::T = 1f-3, #step size parameter of simulation in seconds
k::T = 1f0, #inertial constant of pendulum,
m_f::T = 0f0, #friction of the rotating pole
μ_c::T = 0f0, #friction of the cart wheels against the track
fmax::T = 100f0, #force applied by throttle
x_max::T = Inf32, #maximum horizontal position
θ_max::T = π/2f0, #maximum pole angle
ẋ_max::T = Inf32,
θ̇_max::T = Inf32,
init_x::Function = () -> 0f0, #initialize each of the 4 state variables
init_θ::Function = () -> Float32(rand([-π/6, π/6])),
init_ẋ::Function = () -> 0f0,
init_θ̇::Function = () -> 0f0) where T<:Real
#the action space is full throttle forward or backwards or idle in the discrete case
actions = [-fmax, zero(T), fmax]
#create a vehicle to use in simulation steps
vehicle = CartPoleVehicle(m, m_c, l, k, m_f, μ_c)
initialize_state(;t = 0f0) = CartPoleState(init_x(), init_θ(), init_ẋ(), init_θ̇(), t)
function failure(s::CartPoleState)
(abs(s.x) > x_max) || (abs(s.θ) > θ_max) || (abs(s.ẋ) > ẋ_max) || (abs(s.θ̇) > θ̇_max)
end
step(s::CartPoleState{T}, f::T) = cartpole_runge_kutta_step(vehicle, s, g, clamp(f, -fmax, fmax), h)
function episodic_step(s::CartPoleState{T}, f::T)
s′ = step(s, f)
return (one(T), s′)
end
function continuing_step(s::CartPoleState{T}, f::T)
s′ = step(s, f)
failure(s′) && return (-one(T), initialize_state(;t = s′.t))
return (zero(T), s′)
end
s0 = initialize_state()
ptf = StateMDPTransitionSampler((s, i_a) -> episodic_step(s, actions[i_a]), s0)
episodic_mdp = TabularRL.StateMDP(actions, ptf, initialize_state, failure)
ptf = ContinuousMDPTransitionSampler(episodic_step, s0, zero(T))
episodic_mdp_continuous = ContinuousMDP(ptf, initialize_state; isterm = failure)
ptf = StateMDPTransitionSampler((s, i_a) -> continuing_step(s, actions[i_a]), s0)
continuing_mdp = TabularRL.StateMDP(actions, ptf, initialize_state, Returns(false))
ptf = ContinuousMDPTransitionSampler(continuing_step, s0, zero(T))
continuing_mdp_continuous = ContinuousMDP(ptf, initialize_state)
(episodic = (discrete = episodic_mdp, continuous = episodic_mdp_continuous), continuing = (discrete = continuing_mdp, continuous = continuing_mdp_continuous))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$6c5e9bb2-4c38-4613-9652-dec99e97b512cell_id$6c5e9bb2-4c38-4613-9652-dec99e97b512code%md"""
#### Policy Function Output
"""metadatashow_logsèdisabled®skip_as_script«code_folded$b0a66a19-ee76-463b-a704-8fcee85444d0cell_id$b0a66a19-ee76-463b-a704-8fcee85444d0codelbegin
function update_params_with_gradient!(θ::Array{T, N}, α::T, ∇θ::Array{T, N}) where {T<:Real, N}
θ .+= α .* ∇θ
end
function update_params_with_gradient!(θ::Matrix{T}, α::T, ∇θ::BinaryEligibilityVector{T, B}) where {T<:Real, B<:BinaryFeatureVector}
@inbounds for i in eachindex(∇θ.π_dist)
@simd for j in 1:∇θ.binary_features.num_features
k = ∇θ.binary_features.active_features[j]
θ[k, i] -= α*∇θ.π_dist[i]
end
end
@inbounds @simd for i in 1:∇θ.binary_features.num_features
j = ∇θ.binary_features.active_features[i]
θ[j, ∇θ.i_a] += α
end
return θ
end
function update_params_with_gradient!(params::FCANNParams{T}, α::T, ∇::FCANNParams{T}) where T<:Float32
for i in eachindex(first(params))
for j in 1:2
# @info "updating parameter $((j, i)) $(params[j][i]) with gradient $(∇[j][i]) and constant $α"
update_params_with_gradient!(params[j][i], α, ∇[j][i])
# @info "new parameter values are: $(params[j][i])"
end
end
end
update_params_with_gradient!(::Nothing, α::T, ::Nothing) where T<:Real = return nothing
endmetadatashow_logsèdisabled®skip_as_script«code_folded$13ebc12f-ff6f-4266-88d3-28d6df5fcf59cell_id$13ebc12f-ff6f-4266-88d3-28d6df5fcf59codeCactor_critic_binary_episodic_gaussian_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, params::@NamedTuple{λ_θ::T, λ_w::T, α_θ_min::Int64, α_w_min::Int64}, num_θ::Integer, num_w::Integer, num_episodes::Integer; kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = actor_critic_binary_episodic_gaussian_parameter_study(mdp, get_active_features, num_features, params.λ_θ, params.λ_w, 2f0 .^(params.α_θ_min:params.α_θ_min+num_θ-1), 2f0 .^(params.α_w_min:params.α_w_min+num_w-1), num_episodes; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$7a6fb1f0-fc3c-4c29-a6d9-769d32ca98a9cell_id$7a6fb1f0-fc3c-4c29-a6d9-769d32ca98a9code3md"""
### Example 13.1 Short corridor gridworld
"""metadatashow_logsèdisabled®skip_as_script«code_folded$f2f2dd1d-180c-4d36-b515-5079d129f93acell_id$f2f2dd1d-180c-4d36-b515-5079d129f93acodesarsa_λ(corridor_mdp, 1f0, 0.9f0, typemax(Int64), 100_000, 1, get_corridor_features; ϵ = 0.0001f0, α = 0.000001f0, save_episode_steps = true).history.episode_steps |> a -> a ./ (1:length(a)) |> plotmetadatashow_logsèdisabled®skip_as_script«code_folded$553b0ceb-f2ca-41ee-99bc-9f53a4487b49cell_id$553b0ceb-f2ca-41ee-99bc-9f53a4487b49codeTget_corridor_episode_stats(best_mc_corridor.policy_sample_action; ntrials = 100_000)metadatashow_logsèdisabled®skip_as_script«code_folded$f9facbba-39d4-483e-9066-275603156db0cell_id$f9facbba-39d4-483e-9066-275603156db0codeifunction plot_mountaincar_values(v̂_mountain_car, π; n1 = 100, n2 = 100)
xvals = LinRange(-1.2f0, 0.5f0, n1)
vvals = LinRange(-0.07f0, 0.07f0, n2)
values = zeros(Float32, n1, n2)
actions = zeros(Float32, n1, n2)
for (i, x) in enumerate(xvals)
for (j, v) in enumerate(vvals)
v̂ = v̂_mountain_car((x, v))
values[j, i] = v̂
actions[j, i] = π((x, v))
end
end
p1 = plot(heatmap(x = xvals, y = vvals, z = values), Layout(xaxis_title = "position", yaxis_title = "velocity", title = "Learned Value Function", height = 400))
p2 = plot(heatmap(x = xvals, y = vvals, z = actions, colorscale = "rb", showscale = false), Layout(xaxis_title = "position", yaxis_title = "velocity", title = "Policy (blue = accelerate left, red = accelerate right, gray = no acceleration)", height = 400))
@htl("""
$p1
$p2
""")
endmetadatashow_logsèdisabled®skip_as_script«code_folded$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0cell_id$0fbf45c8-3e3c-47c1-b763-3b06bcdc60e0code٘one_step_actor_critic_fcann(corridor_mdp, 1, [1], update_corridor_features!, typemax(Int64), 100_000, α_θ = 2f0^-4, α_w = 2f0^-20).policy_function(1)metadatashow_logs¨disabled®skip_as_script«code_folded$d41f1dd1-45fe-4456-9a01-ed47fd6704a7cell_id$d41f1dd1-45fe-4456-9a01-ed47fd6704a7codeebegin
function update_beta_eligibility_vector!(∇lnπ::BinaryBetaEligibilityVector{T, T, T, B}, dist_params::Vector{T}, x::B, action::T, policy_params::Matrix{T}) where {T<:Real, B<:BinaryFeatureVector}
# @info "Beta eligibility vector is $∇lnπ"
∇lnπ.binary_features = x
∇lnπ.a = action
∇lnπ.α = exp(first(dist_params))
∇lnπ.β = exp(last(dist_params))
# @info "Beta eligibility vector updated to $∇lnπ"
return ∇lnπ
end
function update_beta_eligibility_vector!(∇lnπ::BinaryBetaEligibilityVector{T, NTuple{N, T}, Vector{T}, B}, dist_params::Vector{T}, x::B, action::NTuple{N, T}, policy_params::Matrix{T}) where {T<:Real, N, B<:BinaryFeatureVector}
∇lnπ.binary_features = x
∇lnπ.a = action
for i in 1:N
∇lnπ.α[k] = exp(dist_params[k])
∇lnπ.β[k] = exp(dist_params[k+N])
end
return ∇lnπ
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$ba5d6311-daee-4abc-b2fb-fae2184ef3ebcell_id$ba5d6311-daee-4abc-b2fb-fae2184ef3ebcodefunction setup_binary_gaussian_policy_arguments(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer) where {T<:Real, S, N, A<:Union{T, NTuple{N, T}}, P, F1, F2, F3}
x = BinaryFeatureVector()
update_feature_vector!(x::BinaryFeatureVector, s) = update_binary_feature_vector!(x, s, get_active_features)
sample_action = rand(A)
action_dist_params = make_n_param_dist_params(2, sample_action)
∇lnπ = BinaryGaussianEligibilityVector(sample_action)
return (feature_vector = x, update_feature_vector! = update_feature_vector!, action_distribution_parameters = action_dist_params, eligibility_vector = ∇lnπ)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$8e742d32-c074-4981-b35b-b596b64c869bcell_id$8e742d32-c074-4981-b35b-b596b64c869bcode٨@bind cartpole_continuing_binary_study_params create_actor_critic_continuing_params_UI(;λ_θ = 0.95f0, λ_w = 0.05f0, log2α_θ = -4, log2α_w = -16, α_r̄ = 0.005f0)metadatashow_logsèdisabled®skip_as_script«code_folded$03a218cb-aa83-4000-85b5-c6f247087053cell_id$03a218cb-aa83-4000-85b5-c6f247087053codefunction update_binary_value_gradient!(∇v̂::BinaryFeatureVector, binary_features::BinaryFeatureVector, value_params::Vector{T}) where T<:Real
∇v̂.active_features = binary_features.active_features
∇v̂.num_features = binary_features.num_features
endmetadatashow_logsèdisabled®skip_as_script«code_folded$1ec1acf1-f833-4478-9b3c-88029340a629cell_id$1ec1acf1-f833-4478-9b3c-88029340a629code:md"""
##### Non-linear Features
This version of REINFORCE uses non-linear features in a fully connected neural network. The number of parameters no longer matches the size of the input feature vector, but a mapping from state to feature vector is still required. One must specify the size of the feature vector, a function that updates the values in a feature vector given a state, and the size of each hidden layer in the neural network. Additional keyword arguments are available to change the construction of the neural network such as adding residual layers.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$de3cba34-9842-44d1-9b79-47126c0a0751cell_id$de3cba34-9842-44d1-9b79-47126c0a0751codeٜconst cartpole_tilecoding_setup = tile_coding_setup(cartpole_functions.min_vals, cartpole_functions.max_vals, (1f0/4, 1f0/8, 1f0/8, 1f0/8), 8, (1, 3, 5, 7))metadatashow_logsèdisabled®skip_as_script«code_folded$04f42c09-8ab5-4233-b196-51c4aa2dcedbcell_id$04f42c09-8ab5-4233-b196-51c4aa2dcedbcodeif start_mountaincar_continuing_param_study > 0
mountaincar_binary_continuing_parameter_study(mountaincar_continuing_binary_params, 5, 3, 100_000; seed = 45)
else
md"""
Waiting to run parameter study
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$54ff46a2-489a-4dd2-bc30-df70c780cc42cell_id$54ff46a2-489a-4dd2-bc30-df70c780cc42code\cartpole_fcann_parameter_study(fill(fcann_cartpole_study_params.h, fcann_cartpole_study_params.l), fcann_cartpole_study_params.λ_θ, fcann_cartpole_study_params.λ_w, 2f0 .^(fcann_cartpole_study_params.α_θ_min:fcann_cartpole_study_params.α_θ_min+4), 2f0 .^ (fcann_cartpole_study_params.α_w_min:fcann_cartpole_study_params.α_w_min+2), 1_000)metadatashow_logsèdisabledîskip_as_script«code_folded$7126aefd-b847-497a-9545-514e9b9afa71cell_id$7126aefd-b847-497a-9545-514e9b9afa71codeactor_critic_fcann_episodic_parameter_study(MountainCarTask.mdp, mountaincar_fcann_setup.update_feature_vector!, mountaincar_fcann_setup.num_features, fill(fcann_mountaincar_study_params.h, fcann_mountaincar_study_params.l), fcann_mountaincar_study_params.λ_θ, fcann_mountaincar_study_params.λ_w, 2f0 .^ (fcann_mountaincar_study_params.α_θ_min:fcann_mountaincar_study_params.α_θ_min+4), 2f0 .^ (fcann_mountaincar_study_params.α_w_min:fcann_mountaincar_study_params.α_w_min+2), 100_000; nruns = 100, max_steps = 1_000)metadatashow_logsèdisabled®skip_as_script«code_folded$48dcd2d0-a940-41da-a097-90c780f2ec4dcell_id$48dcd2d0-a940-41da-a097-90c780f2ec4dcodezmd"""
### Alternative Paramaterization
If the action space is small enough, then it may be convenient to create a function that simply outputs the preferences for all of the actions at a given state. Let's call $N_a$ to be the number of available actions. We would then consider the vector function $\mathbf{h}(s, \boldsymbol{\theta}) \in \mathbb{R}^{N_a}$ and its components $h_1, h_2, h_3, \dots, h_{N_a}$. To be the action preferences at each state. With this style of paramaterization, we need only compute state feature vectors $\mathbf{x}(s) \in \mathbb{R}^d$.
Similarly, the policy function would also be a vector function. In order to compute the softmax, we must evaluate the denominator of (13.2) which requires knowing all of the action preferences. Practically, it is only defined as a function on vectors, so consider the following notation to simplify expressions where we use the symbol $\mathbf{\sigma}$ to denote the soft-max vector function.
$\sigma(\mathbf{x}) = \frac{e^{\mathbf{x}}}{\sum_j{e^{x_j}}} \text{ where we abuse the notation } e^{\mathbf{x}} = \begin{pmatrix}
e^{x_1} \\
e^{x_2} \\
\vdots \\
e^{x_n}
\end{pmatrix}$
Using this notation, we can write down the policy function under this new parameterization: $\mathbf{\pi}(s, \boldsymbol{\theta}) = \mathbf{\sigma}(\mathbf{h}(s, \boldsymbol{\theta}))$. What do linear preferences look like with this parameterization? Instead of a parameter vector $\boldsymbol{\theta} \in \mathbb{R}^{d^\prime}$, we have a parameter matrix $\boldsymbol{\theta} \in \mathbb{R}^{d \times N_a}$ and the vector of preferences is the result of a matrix vector multiplication: $\mathbf{h}(s, \boldsymbol{\theta}) = \theta^\top \mathbf{x}(s) \in \mathbb{R}^{N_a}$. Subscript notation is used to refer to single preference values so $\mathbf{h}_i$ would be the $ith$ index of $\mathbf{h}$ for the $ith$ action preference equivalent to $h_i$.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$e1493cea-19c4-475d-98a0-86d27fb04af1cell_id$e1493cea-19c4-475d-98a0-86d27fb04af1code٠sarsa_λ(corridor_mdp, 1f0, 0.9f0, typemax(Int64), 100_000, 1, get_corridor_features; ϵ = 0.001f0, α = 0.000001f0).greedy_policy |> get_corridor_episode_statsmetadatashow_logsèdisabled®skip_as_script«code_folded$511a847f-234c-465e-8f4a-688e79d9b975cell_id$511a847f-234c-465e-8f4a-688e79d9b975codeSmd"""
## 13.6 Policy Gradient for Continuing Problems
In the continuing case we need to define the average reward per time step as discussed in Section 10.3. In the update procedure the δ is calculated differently in terms of the reward compared to this long running average. The value functions in this case will also learn the reward difference from the average which is assumed to have a well defined expected value under the stationary state distribution for the policy. This shift in the value function will not affect performance since shifting the value function up and down by a constant does not affect the learned policy. To implement this we need a new learning rate $α_{\overline{R}}$ which controls how quickly the reward average updates. This replaces $γ$ in a sense since we no longer discount rewards of future time steps.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$697b2310-9d96-4f7f-be62-c3bd6bf736f3cell_id$697b2310-9d96-4f7f-be62-c3bd6bf736f3codefunction reinforce_with_baseline_monte_carlo_control_fcann(mdp::StateMDP{T, S, A, P, F1, F2, F3}, input_length::Integer, hidden_layers::Vector{Int64}, update_feature_vector!::Function,max_episodes::Integer; policy_params::FCANNParams = FCANN.initializeparams_saxe(input_length, hidden_layers, length(mdp.actions)), reslayers = 0, l2 = 0f0, dropout = 0f0, use_μP = true, activation_list = fill(true, length(hidden_layers)), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
setup = setup_fcann_policy_and_value_arguments(policy_params, input_length, hidden_layers, reslayers, l2, dropout, use_μP, activation_list)
reinforce_with_baseline_monte_carlo_control!(policy_params, setup.eligibility_vector, setup.value_params, setup.value_gradient, mdp, setup.update_action_preferences!, setup.update_eligibility_vector!, setup.feature_vector, update_feature_vector!, setup.value_function, setup.gradient_update, max_episodes; kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$056a8adc-92f4-4b33-90d9-4b3b4026bbbccell_id$056a8adc-92f4-4b33-90d9-4b3b4026bbbccodebegin
function update_traces_with_gradient!(c::T, z_θ::Matrix{T}, ∇θ::BinaryGaussianEligibilityVector{T, T, T, B}) where {T<:Real, B<:BinaryFeatureVector}
c1 = ∇θ.a - ∇θ.μ
c2 = ∇θ.σ^(-2)
# isnan(c2) && @info "warning σ of $∇θ.σ is causing nan results"
# isinf(c2) && @info "warning σ of $∇θ.σ is causing inf results"
c3 = c1 * c2
c4 = c3*c1 - one(T)
z_θ .*= c
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 1] += c3
end
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 2] += c4
end
return z_θ
end
function update_traces_with_gradient!(c::T, z_θ::Matrix{T}, ∇θ::BinaryBetaEligibilityVector{T, T, T, B}) where {T<:Real, B<:BinaryFeatureVector}
c1 = digamma(∇θ.α + ∇θ.β)
δ1 = ∇θ.α*(log(∇θ.a) + c1 - digamma(∇θ.α))
z_θ .*= c
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 1] += δ1
end
δ2 = ∇θ.β*(log(one(T) - ∇θ.a) + c1 - digamma(∇θ.β))
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 2] += δ2
end
return z_θ
end
function update_traces_with_gradient!(c::T, z_θ::Matrix{T}, ∇θ::BinarySquashedGaussianEligibilityVector{T, T, T, B}) where {T<:Real, B<:BinaryFeatureVector}
c1 = atanh(∇θ.a / ∇θ.amax) - ∇θ.μ
c2 = ∇θ.σ^(-2)
# isnan(c2) && @info "warning σ of $∇θ.σ is causing nan results"
# isinf(c2) && @info "warning σ of $∇θ.σ is causing inf results"
c3 = c1 * c2
c4 = c3*c1 - one(T)
z_θ .*= c
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 1] += c3
end
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 2] += c4
end
return z_θ
end
function update_traces_with_gradient!(a::T, z_θ::Matrix{T}, b::T, ∇θ::BinaryGaussianEligibilityVector{T, T, T, B}) where {T<:Real, B<:BinaryFeatureVector}
c1 = ∇θ.a - ∇θ.μ
c2 = ∇θ.σ^(-2)
# isnan(c2) && @info "warning σ of $∇θ.σ is causing nan results"
# isinf(c2) && @info "warning σ of $∇θ.σ is causing inf results"
c3 = c1 * c2
c4 = c3*c1 - one(T)
z_θ .*= a
δ1 = b*c3
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 1] += δ1
end
δ2 = b*c4
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 2] += δ2
end
return z_θ
end
function update_traces_with_gradient!(a::T, z_θ::Matrix{T}, b::T, ∇θ::BinarySquashedGaussianEligibilityVector{T, T, T, B}) where {T<:Real, B<:BinaryFeatureVector}
c1 = atanh(∇θ.a / ∇θ.amax) - ∇θ.μ
c2 = ∇θ.σ^(-2)
# isnan(c2) && @info "warning σ of $∇θ.σ is causing nan results"
# isinf(c2) && @info "warning σ of $∇θ.σ is causing inf results"
c3 = c1 * c2
c4 = c3*c1 - one(T)
z_θ .*= a
δ1 = b*c3
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 1] += δ1
end
δ2 = b*c4
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 2] += δ2
end
return z_θ
end
function update_traces_with_gradient!(a::T, z_θ::Matrix{T}, b::T, ∇θ::BinaryBetaEligibilityVector{T, T, T, B}) where {T<:Real, B<:BinaryFeatureVector}
c1 = digamma(∇θ.α + ∇θ.β)
z_θ .*= a
δ1 = b*∇θ.α*(log(∇θ.a) + c1 - digamma(∇θ.α))
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 1] += δ1
end
δ2 = b*∇θ.β*(log(one(T) - ∇θ.a) + c1 - digamma(∇θ.β))
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
z_θ[i, 2] += δ2
end
return z_θ
end
function update_traces_with_gradient!(a::T, z_θ::Matrix{T}, ∇θ::BinaryGaussianEligibilityVector{T, NTuple{N, T}, Vector{T}, B}) where {T<:Real, N, B<:BinaryFeatureVector}
z_θ .*= a
for k in 1:N
c1 = ∇θ.a[k] - ∇θ.μ[k]
c2 = ∇θ.σ[k] ^-2
# isnan(c2) && @info "warning σ of $∇θ.σ is causing nan results"
# isinf(c2) && @info "warning σ of $∇θ.σ is causing inf results"
c3 = c1 * c2
c4 = c3*c1 - one(T)
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k] += c3
end
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k+N] += c4
end
end
return θ
end
function update_traces_with_gradient!(a::T, z_θ::Matrix{T}, ∇θ::BinaryBetaEligibilityVector{T, NTuple{N, T}, Vector{T}, B}) where {T<:Real, N, B<:BinaryFeatureVector}
z_θ .*= a
for k in 1:N
c1 = digamma(∇θ.α[k] + ∇θ.β[k])
δ1 = ∇θ.α[k]*(log(∇θ.a[k]) + c1 - digamma(∇θ.α[k]))
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k] += δ1
end
δ2 = ∇θ.β[k]*(log(one(T) - ∇θ.a[k]) + c1 - digamma(∇θ.β[k]))
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k+N] += δ2
end
end
return θ
end
function update_traces_with_gradient!(a::T, z_θ::Matrix{T}, b::T, ∇θ::BinaryGaussianEligibilityVector{T, NTuple{N, T}, Vector{T}, B}) where {T<:Real, N, B<:BinaryFeatureVector}
z_θ .*= a
for k in 1:N
c1 = ∇θ.a[k] - ∇θ.μ[k]
c2 = ∇θ.σ[k] ^-2
isnan(c2) && @info "warning σ of $∇θ.σ is causing nan results"
isinf(c2) && @info "warning σ of $∇θ.σ is causing inf results"
c3 = c1 * c2
c4 = c3*c1 - one(T)
δ1 = b*c3
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k] += δ1
end
δ2 = b*c4
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k+N] += δ2
end
end
return θ
end
function update_traces_with_gradient!(a::T, z_θ::Matrix{T}, b::T, ∇θ::BinaryBetaEligibilityVector{T, NTuple{N, T}, Vector{T}, B}) where {T<:Real, N, B<:BinaryFeatureVector}
z_θ .*= a
for k in 1:N
c1 = digamma(∇θ.α[k] + ∇θ.β[k])
δ1 = b*∇θ.α[k]*(log(∇θ.a[k]) + c1 - digamma(∇θ.α[k]))
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k] += δ1
end
δ2 = b*∇θ.β[k]*(log(one(T) - ∇θ.a[k]) + c1 - digamma(∇θ.β[k]))
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k+N] += δ2
end
end
return θ
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$bc8a399b-8864-4473-89d2-e3b0a03d15b5cell_id$bc8a399b-8864-4473-89d2-e3b0a03d15b5codeٹcorridor_parameter_study(args...; kwargs...) = actor_critic_binary_episodic_parameter_study(corridor_mdp, get_corridor_features, 1, args...; init_policy_params = [0f0 3.7f0], kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$bba13634-ff0e-47f7-a23b-8d56098f4ac6cell_id$bba13634-ff0e-47f7-a23b-8d56098f4ac6codebegin
function gaussian_action_sampler(params::Vector{T}) where T<:Real
σ = exp(params[2])
μ = params[1]
isinf(μ) && return μ
isapprox(σ, zero(T)) && return μ
isnan(σ) && return μ
rand(Normal(μ, σ))
end
make_gaussian_n_sampler(::Val{1}) = gaussian_action_sampler
function make_gaussian_n_sampler(::Val{N}) where N
function f(params::Vector{T}) where T<:Real
ntuple(i -> rand(Normal(params[i], exp(params[i+N]))), N)
end
end
make_gaussian_n_sampler(n::Integer) = make_gaussian_n_sampler(Val(n))
make_gaussian_sampler(::T) where T<:Real = gaussian_action_sampler
make_gaussian_sampler(::NTuple{N, T}) where {N, T<:Real} = make_gaussian_n_sampler(N)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$407a0724-4bb6-4c83-ab2d-17a0e19c4072cell_id$407a0724-4bb6-4c83-ab2d-17a0e19c4072codeKconst reinforce_test4 = actor_critic_with_eligibility_traces_fcann(cartpole_setup.mdps.episodic.discrete, 0.95f0, 0.2f0, cartpole_fcann_feature_setup.num_features, [64, 64], (x, s) -> cartpole_fcann_feature_setup.update_feature_vector!(x, (s.x, s.θ, s.ẋ, s.θ̇)), typemax(Int64), 1_000_000; α_θ = 4f-4, α_w = 2f-5, γ = 1f0)metadatashow_logsèdisabled®skip_as_script«code_folded$77cf3a74-899f-4ade-99f2-5aaf7a98c02dcell_id$77cf3a74-899f-4ade-99f2-5aaf7a98c02dcodeٴfunction scale_fcann_params!(params::FCANNParams, scales::Vector{T}) where T<:Real
@inbounds for i in eachindex(scales)
for j in 1:2
params[j][i] ./= scales[i]
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$28ce6e60-59cf-408a-8081-b978507b3c72cell_id$28ce6e60-59cf-408a-8081-b978507b3c72code@bind cartpole_fcann_continuing_test_state PlutoUI.combine() do Child
md"""
x position: $(Child(Slider(-50f0:50f0, default = 0, show_value=true)))
pole angle: $(Child(Slider(LinRange(-deg2rad(70f0), deg2rad(70f0), 1000), default = 0, show_value=true)))
x velocity: $(Child(Slider(-50f0:50f0, default = 0, show_value=true)))
pole angular velocity: $(Child(Slider(-10f0:10f0, default = 0, show_value=true)))
"""
end |> confirmmetadatashow_logsèdisabled®skip_as_script«code_folded$7ccadf01-fbba-4dfd-a5ad-770dab9946f9cell_id$7ccadf01-fbba-4dfd-a5ad-770dab9946f9codeTmd"""
We can define our policy as a normal distribution function over actions for a given state and parameter vector.
$\pi(a|s, \mathbf{\theta}) \doteq \frac{1}{\sigma(s, \mathbf{\theta}) \sqrt{2\pi}} \exp \left ( - \frac{(a-\mu(s, \mathbf{\theta}))^2}{2\sigma(s, \mathbf{\theta})^2} \right ) \tag{13.19}$
This policy requires μ and σ to be parameterized by the parameter vector. To make a linear model for both parameters we can use the following formulas:
$\mu(s, \mathbf{\theta}) \doteq \mathbf{\theta}_\mu ^\top \mathbf{x}_\mu(s) \text{ and } \sigma(s, \mathbf{\theta}) \doteq \exp{( \mathbf{\theta}_\sigma ^ \top \mathbf{x}_\sigma (s))} \tag{13.20}$
where $\mathbf{x}_\mu(s)$ and $\mathbf{x}_\sigma(s)$ are state feature vectors. With these formulas we can apply the previous algorithms to solve environments with real-valued actions.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$b72e030f-7d52-481f-b4f7-2b16b227e547cell_id$b72e030f-7d52-481f-b4f7-2b16b227e547code\md"""
### Figure 13.2
Adding a baseline to REINFORCE can make it learn much faster as illustrated here on the short-corridor gridworld (Example 13.1). Here the approximate state-value function used in the baseline is $\hat v(s, \mathbf{w}) = w$. There is only one component of the feature vector and the state value approximation parameters.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$4c5cb75e-79b5-4502-b1eb-6246e002feafcell_id$4c5cb75e-79b5-4502-b1eb-6246e002feafcodeZ@bind mountaincar_binary_params create_actor_critic_params_UI(λ_θ = 0.1f0, λ_w = 0.9f0)metadatashow_logsèdisabled®skip_as_script«code_folded$48b342f2-e48f-457a-9bd3-b3504a79f3a6cell_id$48b342f2-e48f-457a-9bd3-b3504a79f3a6codemd"""
#### Binary Features
This version of REINFORCE uses binary feature vectors for which one needs to specify the total number of features as well as a function that returns the active features for a given state.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$5d50a5d0-8fe2-4c6e-b76c-d5614e4fd884cell_id$5d50a5d0-8fe2-4c6e-b76c-d5614e4fd884codem#for displaying plots that do not load by default when the notebook first runs. Displays a placeholder markdown and then if the counter is more than 0 runs the function f with the provided arguments and caches the result in the appropriate dictionary
function show_or_lookup_plot(buttoncounter::Integer, args::Tuple, kwargs::NamedTuple, dict::Dict, f::Function, name::AbstractString)
buttoncounter == 0 && return md"""
#### Placeholder for $name plot. Click above button to run
"""
haskey(dict, (args, kwargs)) && return dict[(args, kwargs)]
p = f(args...; kwargs...)
dict[(args, kwargs)] = p
endmetadatashow_logsèdisabled®skip_as_script«code_folded$ba645f6b-143f-4e83-9003-707770ae308dcell_id$ba645f6b-143f-4e83-9003-707770ae308dcode|function show_mountaincar_trajectory(π::Function, max_steps::Integer)
states, actions, rewards, sterm, nsteps = runepisode(MountainCarTask.mdp; π = π, max_steps = max_steps)
positions = [s[1] for s in states]
velocities = [s[2] for s in states]
tr1 = scatter(x = positions, y = velocities, mode = "markers", showlegend = false)
tr2 = scatter(y = positions, showlegend = false)
tr3 = scatter(y = [MountainCarTask.actions[i] for i in actions], showlegend = false)
p1 = plot(tr1, Layout(xaxis_title = "position", yaxis_title = "velocity", xaxis_range = [-1.2, 0.5], yaxis_range = [-0.07, 0.07], height = 400))
p2 = plot(tr2, Layout(xaxis_title = "time", yaxis_title = "position", height = 400))
p3 = plot(tr3, Layout(xaxis_title = "time", yaxis_title = "action", height = 400))
@htl("""
Total Reward: $(sum(rewards))
$([p1 p2 p3])
""")
endmetadatashow_logsèdisabled®skip_as_script«code_folded$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811cell_id$1acc0d86-fd5b-4f2e-acb2-dc9a96d3b811codeHupdate_corridor_features!(x::Vector{T}, s) where T<:Real = x[1] = one(T)metadatashow_logsèdisabled®skip_as_script«code_folded$8f1b2db4-ed35-44fc-a3d5-e06deae16d48cell_id$8f1b2db4-ed35-44fc-a3d5-e06deae16d48code`cartpole_tilecoding_reinforce_continuous_parameter_study(2f0 .^ (-18:-15), 2f0 .^ (-6:-4), 1000)metadatashow_logsèdisabledîskip_as_script«code_folded$57bbdb10-bed8-459d-8f67-9ea637cf12bacell_id$57bbdb10-bed8-459d-8f67-9ea637cf12bacodefunction one_step_actor_critic_fcann(mdp::StateMDP{T, S, A, P, F1, F2, F3}, input_length::Integer, hidden_layers::Vector{Int64}, update_feature_vector!::Function, max_episodes::Integer, max_steps::Integer; policy_params::FCANNParams = FCANN.initializeparams_saxe(input_length, hidden_layers, length(mdp.actions)), reslayers = 0, l2 = 0f0, dropout = 0f0, use_μP = true, activation_list = fill(true, length(hidden_layers)), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
setup = setup_fcann_policy_and_value_arguments(policy_params, input_length, hidden_layers, reslayers, l2, dropout, use_μP, activation_list)
one_step_actor_critic!(policy_params, setup.eligibility_vector, setup.value_params, setup.value_gradient, mdp, setup.update_action_preferences!, setup.update_eligibility_vector!, setup.feature_vector, update_feature_vector!, setup.value_function, setup.gradient_update, max_episodes, max_steps; kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$ca360680-afc9-4dd9-9351-493643f91575cell_id$ca360680-afc9-4dd9-9351-493643f91575code|md"""
#### Probability distributions for short corridor gridworld example with probability of left action selected below
"""metadatashow_logsèdisabled®skip_as_script«code_folded$d95f75b5-21d8-4862-baa7-50b58d9725b8cell_id$d95f75b5-21d8-4862-baa7-50b58d9725b8code md"""
### Soft-max notation and gradients
To use policy gradient methods, we must be able to take the gradient of the policy function for every state-action pair. Using the above notation and treating the policy as a vector function, we must know the gradient of the soft-max applied to a vector function at a particular index. Each gradient is a column vector of length $d$ where $d$ is the number of parameters. There is a separate gradient available for every index in the vector output which is one for each action or a total of $N_a$. To simplify expressions, $\mathbf{h}(s, \boldsymbol{\theta})$ will we written as $\mathbf{h}$ and $\mathbf{\pi} = \mathbf{\sigma}(\mathbf{h})$. Our desired gradient is with respect to a particular component of $\mathbf{\sigma}(\mathbf{h})$ denoted $\mathbf{\sigma}(\mathbf{h})_a$ where $a$ represents the action index. The gradient itself is the vector of partial derivatives with respect to the parameters $\theta$. The $ith$ component of the gradient $\nabla(f(\theta))_i = \frac{\partial f(\theta)}{\partial \theta_i}$. When we compute the gradient we need all the components whose expression is derived below.
$\begin{align}
\nabla \left ( \sigma(\mathbf{h})_a \right )_i &= \frac{\partial}{\partial \theta_i} \left ( \frac{e^{h_a}}{\sum_k{e^{h_k}}} \right ) \\
&=\left ( \frac{1}{{\sum_k{e^{h_k}}}} \right )^2 \left ( e^{h_a} \frac{\partial{h_a}}{\partial{\theta_i}} \sum_k{e^{h_k}} - e^{h_a} \sum_k{e^{h_k} \frac{\partial{h_k}}{\partial{\theta_i}}} \right ) \\
&=\left ( \frac{1}{{\sum_k{e^{h_k}}}} \right )^2 e^{h_a} \left ( \frac{\partial{h_a}}{\partial{\theta_i}} \sum_k{e^{h_k}} - \sum_k{e^{h_k} \frac{\partial{h_k}}{\partial{\theta_i}}} \right ) \tag{factoring out exponenential term}\\
&=\left ( \frac{e^{h_a}}{{\sum_k{e^{h_k}}}} \right ) \left ( \frac{\partial{h_a}}{\partial{\theta_i}} \sum_k{\frac{e^{h_k}}{\sum_l e^{h_l}}} - \sum_k{\frac{e^{h_k}}{\sum_l e^{h_l}} \frac{\partial{h_k}}{\partial{\theta_i}}} \right ) \tag{distributing squared fraction}\\
&=\pi_a \left ( \frac{\partial{h_a}}{\partial{\theta_i}} \sum_k{\pi_k} - \sum_k{\pi_k \frac{\partial{h_k}}{\partial{\theta_i}}} \right ) \tag{definition of policy function}\\
&=\pi_a \left ( \frac{\partial{h_a}}{\partial{\theta_i}} - \sum_k{\pi_k \frac{\partial{h_k}}{\partial{\theta_i}}} \right )
\end{align}$
The final step results form the fact that the policy function is a probability distribution so the sum over it is always 1.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$65be0e58-24be-4932-92a9-9e4825b14144cell_id$65be0e58-24be-4932-92a9-9e4825b14144codebactor_critic_binary_continuing_squashed_gaussian_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, args...; kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = actor_critic_binary_continuing_squashed_gaussian_parameter_study(mdp, one(T), get_active_features, num_features, args...; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$60c21e9c-e42d-4f0b-a910-3b318440fbc8cell_id$60c21e9c-e42d-4f0b-a910-3b318440fbc8code@bind gaussian_plot_params PlutoUI.combine() do Child
md"""
### Normal Distribution Plot with
$$\mu$$: $(Child(Slider(-4:.01:4, default = 0, show_value=true)))
$$\sigma$$: $(Child(Slider(0.01:0.01:5, default = 1, show_value=true)))
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$da2d3186-a778-41cc-9b49-759bf1e9b8facell_id$da2d3186-a778-41cc-9b49-759bf1e9b8facodeٟconst BinaryFeatures{I} = Union{C1, C2, C3} where {I <: Integer, C1 <: AbstractVector{I}, N, C2 <: NTuple{N, I}, T<:AbstractVector{I}, C3 <: Base.Generator{T}}metadatashow_logsèdisabled®skip_as_script«code_folded$b695ef21-a1ac-4d1f-a0e1-71cd81cede18cell_id$b695ef21-a1ac-4d1f-a0e1-71cd81cede18codeWplot_mountaincar_continuous_values(mountaincar_continuous_test_train2.policy_and_value)metadatashow_logsèdisabled®skip_as_script«code_folded$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00cell_id$7d5c5e78-cdb9-4c1f-8b6d-53591f47ff00codefunction reinforce_with_baseline_monte_carlo_control_binary_features_gaussian_actions(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, max_episodes::Integer; policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, N, A <: Union{T, NTuple{N, T}}, P, F1, F2, F3}
setup = setup_binary_gaussian_policy_arguments(mdp, get_active_features, num_features)
reinforce_with_baseline_monte_carlo_control!(policy_params, setup.eligibility_vector, value_params, BinaryFeatureVector(), mdp, update_binary_action_preferences!, setup.action_distribution_parameters, make_gaussian_sampler(rand(A)), update_gaussian_eligibility_vector!, setup.feature_vector, setup.update_feature_vector!, binary_value_function, update_binary_value_gradient!, max_episodes; kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$dcb306ae-a1b1-43d6-ba6e-e38668838689cell_id$dcb306ae-a1b1-43d6-ba6e-e38668838689code'md"""
### *Soft-max Implementation*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$54f559b6-8a62-4a42-894d-c56e41d5ebefcell_id$54f559b6-8a62-4a42-894d-c56e41d5ebefcode;const corridor_state_counts = collect_state_distributions()metadatashow_logsèdisabled®skip_as_script«code_folded$f545c800-0bf3-491f-9d7d-42341cfdb573cell_id$f545c800-0bf3-491f-9d7d-42341cfdb573codefunction form_state_continuous_policy_function(update_feature_vector!::Function, update_action_preferences!::Function)
function π!(x, action_preferences, s, params)
# @info "Updating feature vector with state $(s)"
update_feature_vector!(x, s)
update_action_preferences!(action_preferences, x, params)
# @info "Action distribution is $action_preferences"
return action_preferences
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$8b35661b-5075-4d63-bc31-044407f99acfcell_id$8b35661b-5075-4d63-bc31-044407f99acfcodeactor_critic_with_eligibility_traces_binary_features(corridor_continuing_mdp, 0.75f0, 0.25f0, get_corridor_features, 1, 1_000_000, α_θ = 0.00625f0, α_w = 0.0004f0, α_r̄ = 0.004f0, policy_params = [0f0 3.7f0]; save_step_rewards = true).policy_and_value(1)metadatashow_logsèdisabled®skip_as_script«code_folded$09dd1440-5d09-421f-addc-b1ede43ff517cell_id$09dd1440-5d09-421f-addc-b1ede43ff517codeolet x = LinRange(-5, 5, 1000)
plot(scatter(x = x, y = pdf.(Normal(gaussian_plot_params...), x)), Layout())
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a0ca7a5e-0089-4a45-9278-c0f27cd096a0cell_id$a0ca7a5e-0089-4a45-9278-c0f27cd096a0codeWplot_mountaincar_continuous_values(mountaincar_continuous_test_train3.policy_and_value)metadatashow_logsèdisabled®skip_as_script«code_folded$64b38d1f-ecf9-4843-89a1-4c8953048265cell_id$64b38d1f-ecf9-4843-89a1-4c8953048265code٭const cartpole_fcann_continuing_test_episode = runepisode(cartpole_setup.mdps.episodic.discrete; π = cartpole_continuing_fcann_test.policy_sample_action, max_steps = 1_000)metadatashow_logsèdisabled®skip_as_script«code_folded$d963ff6d-f1b6-4799-aa0e-1ae100310d84cell_id$d963ff6d-f1b6-4799-aa0e-1ae100310d84codepPlutoDevMacros.@frompackage @raw_str(joinpath(@__DIR__, "..", "ApproximationUtils.jl")) using ApproximationUtilsmetadatashow_logsèdisabled®skip_as_script«code_folded$b16899b7-36bf-4a5e-8e2f-4496b8450687cell_id$b16899b7-36bf-4a5e-8e2f-4496b8450687codesquashed_gaussian_pdf(x::Union{T, AbstractArray{N, T}}, μ::T, σ::T, xmax::T) where {N, T<:Real} = inv(σ*sqrt(T(2)*π)) * exp(-inv(T(2))*((atanh(x/xmax) - μ)/σ)^2) / abs(xmax*(1 - (x/xmax)^2))metadatashow_logsèdisabled®skip_as_script«code_folded$10cdd16e-a337-4421-a7a0-6de4e4b60c0fcell_id$10cdd16e-a337-4421-a7a0-6de4e4b60c0fcodebegin
mutable struct BinaryGaussianEligibilityVector{T<:Real, A<:Union{T, NTuple{N, T} where N}, P<:Union{T, Vector{T}}, B <: BinaryFeatureVector}
binary_features::B
a::A
μ::P
σ::P
end
BinaryGaussianEligibilityVector(a::T) where T<:Real = BinaryGaussianEligibilityVector(BinaryFeatureVector(), a, zero(T), one(T))
BinaryGaussianEligibilityVector(a::NTuple{N, T}) where {T<:Real, N} = BinaryGaussianEligibilityVector(BinaryFeatureVector(), a, zeros(T, N), ones(T, N))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a8b40b8f-051a-4e6f-a079-ece4f32873decell_id$a8b40b8f-051a-4e6f-a079-ece4f32873decodeVfunction create_actor_critic_params_UI(;λ_θ = 0.5f0, λ_w = 0.5f0, log2α_θ = -10, log2α_w = -10)
PlutoUI.combine() do Child
@htl("""
$(md"""
$$\lambda_\theta$$: $(Child(:λ_θ, Slider(0.00f0:0.001f0:.999f0, default = λ_θ, show_value=true)))
$$\lambda_\mathbf{w}$$: $(Child(:λ_w, Slider(0.00f0:0.001f0:.999f0, default = λ_w, show_value=true)))
$$\log_2 \alpha_\theta$$ min: $(Child(:α_θ_min, NumberField(-100:0, default = log2α_θ)))
$$\log_2 \alpha_{\mathbf{w}}$$ min: $(Child(:α_w_min, NumberField(-100:0, default = log2α_w)))
""")""")
end |> confirm
endmetadatashow_logsèdisabled®skip_as_script«code_folded$5eebf3da-bfe7-46eb-81a3-f87f334ee270cell_id$5eebf3da-bfe7-46eb-81a3-f87f334ee270codefunction create_actor_critic_fcann_params_UI(;λ_θ = 0.5f0, λ_w = 0.5f0, h = 8, l = 2, log2α_θ = -10, log2α_w = -10)
PlutoUI.combine() do Child
md"""
$$\lambda_\theta$$: $(Child(:λ_θ, Slider(0.00f0:0.001f0:.999f0, default = λ_θ, show_value=true)))
$$\lambda_\mathbf{w}$$: $(Child(:λ_w, Slider(0.00f0:0.001f0:.999f0, default = λ_w, show_value=true)))
hidden layer size: $(Child(:h, NumberField(1:128, default = h))), num layers: $(Child(:l, NumberField(1:5, default = l)))
$$\log_2 \alpha_\theta$$ min: $(Child(:α_θ_min, NumberField(-100:0, default = log2α_θ)))
$$\log_2 \alpha_{\mathbf{w}}$$ min: $(Child(:α_w_min, NumberField(-100:0, default = log2α_w)))
"""
end |> confirm
endmetadatashow_logsèdisabled®skip_as_script«code_folded$9bce6fdb-2cbc-4758-9a8b-794e490c973dcell_id$9bce6fdb-2cbc-4758-9a8b-794e490c973dcode8@bind ep2_step Slider(1:length(ep2[1]), show_value=true)metadatashow_logsèdisabled®skip_as_script«code_folded$b86ee9d3-b6b5-4ea0-8f55-1927571cdfbfcell_id$b86ee9d3-b6b5-4ea0-8f55-1927571cdfbfcode"function create_continuous_action_mountaincar(;slipforce = 1f0)
mdp = MountainCarTask.mdp
function step(s, a)
f = if abs(a) > slipforce
sign(a)*0.1f0
else
a
end
(-1f0, MountainCarTask.step(s, f))
end
ContinuousMDP(step, mdp.initialize_state, 0f0; isterm = mdp.isterm)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$0ce66c9d-6d1c-4c2d-8178-5bcdfa247cd6cell_id$0ce66c9d-6d1c-4c2d-8178-5bcdfa247cd6codeٚconst mountaincar_continuing_test_episode = runepisode(MountainCarTask.mdp, π = mountaincar_continuing_tile_test.policy_sample_action, max_steps = 1_000)metadatashow_logsèdisabled®skip_as_script«code_folded$7afb6fb0-248a-4518-b94f-9876f81eca64cell_id$7afb6fb0-248a-4518-b94f-9876f81eca64codecorridor_continuing_parameter_study(args...; kwargs...) = actor_critic_linear_parameter_study(corridor_continuing_mdp, get_corridor_features, 1, args...; init_policy_params = [0f0 3.7f0], seed = 45, binary_features=true, kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$37a273b6-b104-46f0-987a-401dc1c97327cell_id$37a273b6-b104-46f0-987a-401dc1c97327codeW@bind start_cartpole_continuing_binary_param_study CounterButton("Run Parameter Study")metadatashow_logsèdisabled®skip_as_script«code_folded$7a6f3f79-ea06-4994-8b62-90b2056e4034cell_id$7a6f3f79-ea06-4994-8b62-90b2056e4034codebegin
function squashed_gaussian_action_sampler(params::Vector{T}, amax::T) where T<:Real
σ = exp(params[2])
μ = params[1]
isinf(μ) && return amax*sign(μ)
isapprox(σ, zero(T)) && return amax*tanh(μ)
isnan(σ) && return amax*tanh(μ)
amax*tanh(rand(Normal(μ, σ)))
end
make_squashed_gaussian_n_sampler(::Val{1}, amax::T) where T<:Real = params -> squashed_gaussian_action_sampler(params, amax)
function make_squashed_gaussian_n_sampler(::Val{N}, amax::NTuple{N, T}) where {N, T<:Real}
function f(params::Vector{T}) where T<:Real
ntuple(i -> amax[i]*tanh(rand(Normal(params[i], exp(params[i+N])))), N)
end
end
make_squashed_gaussian_n_sampler(n::Integer, amax::T) where T<:Real = make_squashed_gaussian_n_sampler(Val(n), amax)
make_squashed_gaussian_sampler(::T, amax::T) where T<:Real = params -> squashed_gaussian_action_sampler(params, amax)
make_squashed_gaussian_sampler(::NTuple{N, T}, amax::NTuple{N, T}) where {N, T<:Real} = make_squashed_gaussian_n_sampler(N, amax)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$f2ed56c9-c2b7-42cb-a083-e12aeaa126efcell_id$f2ed56c9-c2b7-42cb-a083-e12aeaa126efcodeِreinforce_monte_carlo_control_binary_features(corridor_mdp, get_corridor_features, 1, 1_000, α = 2f0^-13, max_steps = 1_000).policy_function(1)metadatashow_logsèdisabled®skip_as_script«code_folded$cbea5840-49d2-4e91-be9c-f5f15666d78acell_id$cbea5840-49d2-4e91-be9c-f5f15666d78acodeٰreinforce_with_baseline_monte_carlo_control_binary_features(corridor_mdp, get_corridor_features, 1, 1_000, α_θ = 2f0^-12, α_w = 2f0^-6, max_steps = 1_000).policy_function(1)metadatashow_logsèdisabled®skip_as_script«code_folded$1f041cb3-618c-4380-a1ec-d7bbe4a80f62cell_id$1f041cb3-618c-4380-a1ec-d7bbe4a80f62codeCfunction actor_critic_binary_episodic_parameter_study(mdp::StateMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, λ_θ::T, λ_w::T, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, max_episodes::Integer; nruns::Integer = 100, max_steps::Integer = 10_000, seed = rand(UInt64), init_policy_params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), init_value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
Random.seed!(seed)
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> actor_critic_with_eligibility_traces_binary_features(mdp, λ_θ, λ_w, get_active_features, num_features, max_episodes, max_steps; α_θ = α_θ, α_w = α_w, policy_params = copy(init_policy_params), value_params = copy(init_value_params), kwargs...) |> x -> isempty(x.episode_rewards) ? -T(Inf) : mean(x.episode_rewards)) |> foldxt(+) |> x -> x / nruns
end
traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Episode in the First $max_episodes Episodes Averaged Over $nruns Runs", xaxis_type = "log", title = "Binary Feature Encoding with $num_features Features, λ_θ = $λ_θ, λ_w = $λ_w"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$96506201-6b66-49e6-8179-06952e2394e1cell_id$96506201-6b66-49e6-8179-06952e2394e1codeBfunction setup_binary_policy_arguments(mdp::StateMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer) where {T<:Real, S, A, P, F1, F2, F3}
x = BinaryFeatureVector()
update_feature_vector!(x::BinaryFeatureVector, s) = update_binary_feature_vector!(x, s, get_active_features)
action_preferences = zeros(T, length(mdp.actions))
∇lnπ = BinaryEligibilityVector(x, 1, copy(action_preferences))
return (feature_vector = x, update_feature_vector! = update_feature_vector!, action_preferences = action_preferences, eligibility_vector = ∇lnπ)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$76b03e72-da04-4530-8534-6d6468268cbdcell_id$76b03e72-da04-4530-8534-6d6468268cbdcode md"""
$\sum_{s \in \mathcal{S}} \sum_{k = 0}^\infty \Pr \{ s_0 \rightarrow s, k, \pi \} = \sum_{k = 0}^\infty \left [ 1 - \Pr \{s_0 \rightarrow S_T, k, \pi \} \right ] = \eta$
where $\eta$ is the average length of an episode. The quantity inside the brackets is the probability that an episode has not terminated by step k and follows from the fact that the sum over states in $\mathcal{S}$ is over the set of non-terminal states. If the sum was over $\mathcal{S}^+$ instead then it would be infinite since the first sum term would be 1 for every k. Normally to calculate $\eta$, we would use the expected value with the probability of an episode lasting exactly $k$ steps, but the probability we have access to here is actually the distribution function, not the density function. That is $\Pr \{s_0 \rightarrow S_T, k, \pi \} = \sum_{t = 0}^k \Pr \{ T = t \} = \Pr \{ T \leq k \}$ where $T$ is the length of an episode. Using these probabilities, we can write $\eta = \mathbb{E}_\pi [T] = \sum_{k = 0}^\infty k \Pr \{ T = k \} = \Pr \{T = 1 \} + 2 \Pr \{T = 2 \} + \cdots$.
Earlier we had the expression $\eta = \sum_{k = 0}^\infty \left [ 1 - \Pr \{s_0 \rightarrow S_T, k, \pi \} \right ] = \sum_{k = 0}^\infty \Pr \{T \gt k \} = \sum_{k = 0}^\infty \sum_{t = k + 1}^\infty \Pr \{T = t \}$
We can stack up the terms of this double sum to see that it is equivalent to the expected value calcuation from before:
$\begin{flalign}
\Pr \{ T = 1 \} + \Pr \{ T = 2 \} + &\Pr \{ T = 3 \} +\cdots \\
\Pr \{ T = 2 \} + &\Pr \{ T = 3 \} + \cdots \\
&\Pr \{ T = 3 \} + \cdots \\
\vdots
\end{flalign}$
If we count terms along the diagonal, we see that each value of $k$ has exactly $k$ terms, matching the expected value calculation.
What if we wanted to calculate the bivariate distribution over states and steps where we ignore the terminal states $\mu_\pi(s, k)$ such that $\sum_{s \in \mathcal{S}} \sum_k \mu_\pi(s, k) = 1$. This probability represents the chance of sampling a particular step and state simultaneously from a unbiased sample of non-terminal states in an episode. Luckily we can break down this probability into two components: 1) the probability of reaching a step k without terminating 2) the probability of being in a non-terminal state on step k. We saw already that 1) is just $\sum_{s \in \mathcal{S}} \Pr \{ s_0 \rightarrow s, k, \pi \}$ and 2) we can calculate by normalizing those probabilities over only the non-terminal states: $\frac{\Pr \{ s_0 \rightarrow s, k, \pi \}}{\sum_{s \in \mathcal{S}} \Pr \{ s_0 \rightarrow s, k, \pi \} }$. By multiplying these two together we see that the probability is just the original distribution but where the domain of possible input values is $s \in \mathcal{S}$ and all possible steps $k$. Therefore, we can transform this into a normalized bivariate distribution by dividing by its sum over those two sets:
$\mu_\pi(s, k) = \frac{\Pr \{ s_0 \rightarrow s, k, \pi \}}{\sum_{x \in \mathcal{S}} \sum_{t = 0}^\infty \Pr \{ s_0 \rightarrow x, t, \pi \}}$
Now that we have established the relationship between the on-policy distribution function and the probability expression we have, we can use it to complete the proof below.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$fd89433e-643c-474b-b3c4-a997678421a6cell_id$fd89433e-643c-474b-b3c4-a997678421a6codemd"""
#### Linear Features
This version of REINFORCE uses linear feature vectors for which one needs to specify the total number of features as well as a function that updates the values in a feature vector given a state.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$87feff3e-e510-4916-91a9-db3a2cd12225cell_id$87feff3e-e510-4916-91a9-db3a2cd12225code@bind fcann_continuing_cartpole_study_params PlutoUI.combine() do Child
md"""
$$\lambda_\theta$$: $(Child(:λ_θ, Slider(0.00f0:0.001f0:.999f0, default = 0.75f0, show_value=true)))
$$\lambda_\mathbf{w}$$: $(Child(:λ_w, Slider(0.00f0:0.001f0:.999f0, default = 0.25f0, show_value=true)))
$$\alpha_{\overline{r}}$$: $(Child(:α_r̄, NumberField(0.00f0:0.001f0:.999f0, default = 0.1f0)))
hidden layer size: $(Child(:h, NumberField(1:128, default = 8))), num layers: $(Child(:l, NumberField(1:5, default = 3)))
$$\log_2 \alpha_\theta$$ min: $(Child(:α_θ_min, NumberField(-100:0, default = -11)))
$$\log_2 \alpha_{\mathbf{w}}$$ min: $(Child(:α_w_min, NumberField(-100:0, default = -10)))
"""
end |> confirmmetadatashow_logsèdisabled®skip_as_script«code_folded$5261651e-a51e-4e80-8e23-83a4c10e5259cell_id$5261651e-a51e-4e80-8e23-83a4c10e5259codebegin
function update_gaussian_eligibility_vector!(∇lnπ::Matrix{T}, action_dist_params::Vector{T}, x::Vector{T}, action::T, policy_params::Matrix{T}) where T<:Real
c1 = action - first(action_dist_params)
σ = exp(last(action_dist_params))
c2 = σ^-2
c3 = c2*c1
c4 = c3*c1 - one(T)
@inbounds @simd for i in eachindex(x)
∇lnπ[i, 1] = x[i]*c3
end
@inbounds @simd for i in eachindex(x)
∇lnπ[i, 2] = x[i]*c4
end
end
function update_gaussian_eligibility_vector!(∇lnπ::Matrix{T}, action_dist_params::Vector{T}, x::Vector{T}, action::NTuple{N, T}, policy_params::Matrix{T}) where {N, T <: Real}
for k = 1:N
c1 = action - action_dist_params[k]
σ = exp(action_dist_params[k+N])
c2 = σ^-2
c3 = c2*c1
c4 = c3*c1 - one(T)
@inbounds @simd for i in eachindex(x)
∇lnπ[i, k] = x[i]*c3
end
@inbounds @simd for i in eachindex(x)
∇lnπ[i, k+N] = x[i]*c4
end
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$dddc4a2f-34b2-41dc-85b3-55aba4880fa6cell_id$dddc4a2f-34b2-41dc-85b3-55aba4880fa6codeٌdisplay_cartpole_episode((runepisode(cartpole_setup.mdps.episodic.continuous, reinforce_test.policy_sample_action) |> x -> (x[1], x[2]))...)metadatashow_logsèdisabled®skip_as_script«code_folded$54fff14b-cf53-47b0-9cfa-8b9ee33df54ecell_id$54fff14b-cf53-47b0-9cfa-8b9ee33df54ecodebegin
mutable struct BinaryBetaEligibilityVector{T<:Real, A<:Union{T, NTuple{N, T} where N}, P<:Union{T, Vector{T}}, B <: BinaryFeatureVector}
binary_features::B
a::A
α::P
β::P
end
BinaryBetaEligibilityVector(a::T) where T<:Real = BinaryBetaEligibilityVector(BinaryFeatureVector(), a, one(T), one(T))
BinaryBetaEligibilityVector(a::NTuple{N, T}) where {T<:Real, N} = BinaryBetaEligibilityVector(BinaryFeatureVector(), a, ones(T, N), ones(T, N))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$023f67b8-8f38-470a-9766-ac60a75678aacell_id$023f67b8-8f38-470a-9766-ac60a75678aacodefconst mountaincar_fcann_setup = fcann_feature_vector_setup(mountaincar_min_vals, mountaincar_max_vals)metadatashow_logsèdisabled®skip_as_script«code_folded$1558cec1-c4fd-4bc0-85ed-ae22c6067d41cell_id$1558cec1-c4fd-4bc0-85ed-ae22c6067d41codemd"""
We can also repeat this derivation for the alternative linear parameterization where we only have state feature vectors and a parameter matrix with components $\boldsymbol{\theta}_{i, j}$:
$\begin{flalign}
\mathbf{h} &= \boldsymbol{\theta}^\top \mathbf{x}(s) \\
h_a &= \mathbf{h}_a \\
\mathbf{\pi}(s) &= \sigma(\mathbf{h}) \\
\pi_a &= \sigma(\mathbf{h})_a \\
\nabla(\pi_a)_{i, j} &= \pi_a \begin{cases}
\mathbf{x}(s)_i (1 - \pi_j), & \text{ if } j = a \\
-\pi_j \mathbf{x}(s)_i, & \text{ else }\\
\end{cases}
\end{flalign}$
We already know how to apply the chain rule to the natural logarithm so our final gradient is:
Applying this to the above expression yields:
$\begin{flalign}
\nabla \left ( \ln \pi_a \right )_{i, j} &= \frac{\nabla \left ( \pi_a \right )_{i, j}}{\pi_a} \\
&= \begin{cases}
\mathbf{x}(s)_i (1 - \pi_j), & \text{ if } j = a \\
-\pi_j \mathbf{x}(s)_i, & \text{ else }\\
\end{cases}
\end{flalign}$
which is the per component version of the desired vector expression.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$da8d0bca-105b-4d0b-a73d-ee5c9059aeafcell_id$da8d0bca-105b-4d0b-a73d-ee5c9059aeafcodemd"""
Notice now that all of the parameters associated with the state-value estimate are irrelevent since they always cancel out in the parameter update. Even though we have added a parameter, this method effectively removes two from the analysis. Also, we seem to actually benefit from an intermediate value of $\lambda_{\boldsymbol{\theta}}$ unlike in the episodic case where using the Monte Carlo method was always the best.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$3e7cecec-eb77-4862-8e3c-b510422e06dbcell_id$3e7cecec-eb77-4862-8e3c-b510422e06dbcode8plot_squashed_gaussian(squashed_gaussian_plot_params...)metadatashow_logsèdisabled®skip_as_script«code_folded$0284f0d7-b8a9-4ae6-add0-ac1078571d9bcell_id$0284f0d7-b8a9-4ae6-add0-ac1078571d9bcode3md"""
$\begin{flalign}
J(\boldsymbol{\theta}) \doteq r(\pi) &\doteq \lim_{h \rightarrow \infty} \frac{1}{h} \sum_{t=1}^h \mathbb{E} [R_t \mid S_0, A_{0:t-1} \sim \pi] \tag{13.15} \\
&= \lim_{t \rightarrow \infty} \mathbb{E}[R_t \vert S_0,A_{0:t-1} \sim \pi] \\
&= \sum_s \mu(s) \sum_a \pi(a \vert s) \sum_{s^\prime, r} p(s^\prime, r \vert s, a) r
\end{flalign}$
where $\mu$ is the steady-state distribution under $\pi$, $\mu(s) \doteq \lim_{t \rightarrow \infty} \Pr \{ S_t = s \vert A_{0:t} \sim \pi \}$, which is assumed to exist and to be independent of $S_0$ (an ergodicity assumption). Remember that this is the special distribution under which, if you select actions according to $\pi$, you remain the same distribution:
$\sum_s \mu(s) \sum_a \pi(a \vert s, \boldsymbol{\theta})p(s^\prime \vert s, a) = \mu(s^\prime), \: \forall s^\prime \in \mathcal{S}$
Naturally, in the continuing case, we define values, $v_\pi(s) \doteq \mathbb{E}_\pi [G_t \vert S_t = s]$ and $q_\pi(s, a) \doteq \mathbb{E}_\pi[G_t \vert S_t = s, A_t = a]$, with respect to the differential return:
$G_t \doteq R_{t+1} - r(\pi) + R_{t+2} - r(\pi) + R_{t+3} - r(\pi) + \cdots \tag{13.17}$
With these alternate definitions, the policy gradient theorem as given for the episodic case (13.5) remains true for the continuing case. See proof below:
"""metadatashow_logsèdisabled®skip_as_script«code_folded$b94fc99c-f439-4df2-8da3-c01718a136c4cell_id$b94fc99c-f439-4df2-8da3-c01718a136c4code{md"""
Repeating this process for state 2 yields:
$\begin{flalign}
v_2 &= -\frac{2+p}{p(1-p)} \\
\frac{\partial v_2}{\partial p} &= -\frac{p(1-p) - (2+p)(1 - 2p)}{p^2(1-p)^2}
\end{flalign}$
Setting this equal to 0 implies
$\begin{flalign}
p - p^2 &= 2 - 4p + p - 2p^2 \\
p^2 + 4p - 2 &= 0 \\
\end{flalign}$
Using the quadratic equation and taking only the positive solution yields:
$p = \frac{-4 + \sqrt{16 + 8}}{2} = \frac{-4 + \sqrt{24}}{2} = -2 + \sqrt{6} \approx 0.4495$
So, in order to maximize the value at state 2, we have $p_{\text{left}} \approx 0.4495$ and $p_{\text{right}} \approx 0.5505$. Which is different from the value we got for state 1. So There is a different optimal policy depending on the starting state. It should be obvious for example that starting in the third state results in an optimial policy of choosing the right action every time. The value functions for each state are plotted below. The behavior of $v_3$ is not well defined at $p=0$ because for any finite $v_2$ it should be 0 but the limit approaching from the right side is -3. This is because for $p=0$ both $v_1$ and $v_2$ are not finite and the episode never terminates.
The value of the state at this probability is: $v_2 = - \frac{2+p}{p(1-p)} = -\frac{\sqrt{6}}{(\sqrt{6}-2)(3 - \sqrt{6})} = - \frac{\sqrt{6}}{3 \sqrt{6} - 6 - 6 + 2 \sqrt{6}} = - \frac{\sqrt{6}}{5 \sqrt{6} - 12} \approx -9.9$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$b8532822-179b-4cd5-a279-4b71dafb544acell_id$b8532822-179b-4cd5-a279-4b71dafb544acode2const mountaincar_continuous_test_train = actor_critic_with_eligibility_traces_binary_features_gaussian_actions(mountaincar_continuous_mdp, 0.05f0, 0.8f0, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, typemax(Int64), 1_000_000; α_θ = 5f-5, α_w = 0.00008f0)metadatashow_logsèdisabled®skip_as_script«code_folded$07ba9fe4-aaa7-4123-9865-cbfa79d0d44acell_id$07ba9fe4-aaa7-4123-9865-cbfa79d0d44acode٣display_cartpole_episode((runepisode(cartpole_setup.mdps.episodic.discrete; π = reinforce_test4.policy_sample_action, max_steps = 1_000) |> x -> (x[1], x[2]))...)metadatashow_logsèdisabled®skip_as_script«code_folded$f487f2dd-ad09-48ac-ae34-bf50cfa6ac7dcell_id$f487f2dd-ad09-48ac-ae34-bf50cfa6ac7dcodee@bind start_mountaincar_continuing_fcann_param_study CounterButton("Run Mountaincar Parameter Study")metadatashow_logsèdisabled®skip_as_script«code_folded$5c4a383f-fcf2-4f2b-819f-6d84471dda00cell_id$5c4a383f-fcf2-4f2b-819f-6d84471dda00codeufunction update_fcann_value_gradient!(∇v̂::FCANNParams, x::Vector{T}, params::FCANNParams, hidden_layers::Vector{Int64}, l2::T, tanh_grad_z::FCANNActivations{T}, activations::FCANNActivations{T}, deltas::FCANNActivations{T}, dropout::T, reslayers::Integer, activation_list::AbstractVector{B}, scales) where {T<:Float32, B<:Bool}
FCANN.nnCostFunction(params..., hidden_layers, x, 1, l2, ∇v̂..., tanh_grad_z, activations, deltas, dropout; resLayers = reslayers, loss_type = OutputIndex(), activation_list = activation_list)
@inbounds for i in eachindex(params[1])
for j in 1:2
∇v̂[j][i] .*= scales[i]
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$135f205a-f87e-4691-8e87-d317d6312c84cell_id$135f205a-f87e-4691-8e87-d317d6312c84code:md"""
The plots below visualize these distributions for the corridor problem starting with the normalized distributions per step which include the terminal states. If we continued to create these plots for larger values of $k$, then the distribution would collapse to a value of 1 for being in a terminal state. In order to calculate other distributions such as the stationary state distribution, it is necessary to renormalize these probabilities by excluding the terminal states:
#### On-policy Distributions
$$\begin{flalign}
&\mu_{k, \pi}(s) = \Pr\{S_k = s \mid \pi \} \; \forall s \in \mathcal{S}^+ \tag{state visits per step}\\
&\Pr \{ T \leq k \vert \pi \} = 1 - \sum_{s \in \mathcal{S}} \Pr\{S_k = s \mid \pi \} \; \forall k \tag{Chance of terminating already (distribution function not density)}\\
&\mu_\pi(s) = \frac{\sum_k \Pr \{ S_k = s \mid \pi \}}{\sum_{k} \sum_{s \in \mathcal{S}} \Pr \{ S_k = s \mid \pi \}} \; \forall s \in \mathcal{S} \tag{non-terminal state visits}\\
&\mu_\pi(s, k) = \frac{\Pr \{ S_k = s \mid \pi \}}{\sum_{k} \sum_{s \in \mathcal{S}} \Pr \{ S_k = s \mid \pi \}} \; \forall s \in \mathcal{S} \tag{non-terminal state and step visits}\\
\end{flalign}$$
Note that final two distributions are only defined for non-terminal states. If we tried to include terminal states we would be unable to normalize the distribution since $\lim_{k \rightarrow \infty} \Pr \{ S_k = S_T \mid \pi \} = 1$ and we would have a diverging sum in the denominator. The only reason these calculation is possible is that the probabilities reach zero quickly enough at higher $k$ for the non-terminal states.
The plots below visualize the four expressions above. The second expression notably is not a probability density but a cummulative distribution function since it includes a sum of all probabilities that meet the condition.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$4a39f9a7-72d4-44ad-895a-742cd1291f92cell_id$4a39f9a7-72d4-44ad-895a-742cd1291f92codeW@bind dist_plot_p Slider(0.1f0:0.1f0:.9f0; default = 0.5f0, show_value=true) |> confirmmetadatashow_logsèdisabled®skip_as_script«code_folded$ee72af8d-3cb8-4314-82df-580f068e1252cell_id$ee72af8d-3cb8-4314-82df-580f068e1252code md"""
One common form of linear feature vector is one that selects active features per state. Tile coding is an example of this where a state is assigned a tile in each tiling used and the number of tilings control how many active features a given state will have. Because the only possible feature vector values are 1 or 0, this style of encoding need not be as complex as other methods. We can see by the form of the gradients an abbreviated algorithm that need not compute the eligibility vector explicitely.
We can define a binary feature encoding by the function $\mathcal{F}(s)$ which returns the indices of active features for a state $s$ as well as the knowledge of how many total features there are, $d$. All of the values of $\mathbf{x}(s)$ are zero except for the indices in $\mathcal{F}(s)$ whose values are 1. That simplifies the expression we have before for the linear feature eligibility vector:
$\begin{flalign}
\nabla \left ( \ln \pi_a \right )_{i, j} &= \frac{\nabla \left ( \pi_a \right )_{i, j}}{\pi_a} \\
&= \begin{cases}
\mathbf{x}(s)_i (1 - \pi_j), & \text{ if } j = a \\
-\pi_j \mathbf{x}(s)_i, & \text{ else }\\
\end{cases} \\
&= \begin{cases}
(1 - \pi_j), & \text{ if } j = a \text{, } i \in \mathcal{F}(s) \\
-\pi_j, & \text{ if } j \neq a \text{, } i \in \mathcal{F}(s) \\
0, & \text{ otherwise}
\end{cases}
\end{flalign}$
We can see from this form of the eligibility vector that it need not be computed explicitely and we do not need to instantiate a feature vector either. Rather we can simply go through the active feature indices and subtract the policy output for the column index at each row and then add 1 to the column corresponding to the selected action:
Loop for each step of the episode $t = 0, 1, \cdots, T-1$
$G \leftarrow \sum_{k=t+1} \gamma^{k-t-1}R_k$
$c = \alpha \times \gamma^t \times G$
Loop for each action index j
Loop for each feature i
$\theta_{i, j} \leftarrow \theta_{i, j} - c \times \pi(a_j, S_t, \mathbf{\theta})$
Define $j_a$ as the column index corresponding to action $A_t$
Loop for each feature i
$\theta_{i, j_a} \leftarrow \theta_{i, j_a} + c$
Specialized versions of REINFORCE that use binary features and linear features can be found below as well as the general case that works for any type of parameterized function approximation.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$e524f8cc-ab69-4f8b-a59f-28156696a104cell_id$e524f8cc-ab69-4f8b-a59f-28156696a104codec@bind run_mountaincar_binary_episodic_countinuous_param_study2 CounterButton("Run Parameter Study")metadatashow_logsèdisabled®skip_as_script«code_folded$1894ae1a-bb68-4de0-a4d2-ac5d02c49f09cell_id$1894ae1a-bb68-4de0-a4d2-ac5d02c49f09code,plot(mountaincar_test_train.episode_rewards)metadatashow_logsèdisabledîskip_as_script«code_folded$f3bc47b5-03fc-4bd9-a890-26f9608a730bcell_id$f3bc47b5-03fc-4bd9-a890-26f9608a730bcode.md"""
### *Continuing Corridor Gridworld Example*
Note that if we try to apply this algorithm to the short corridor gridworld it fails because a terminal state is encountered. This condition is checked inside the algorithm because there is nothing about an MDP the way it is defined which tells you in advance if it is a continuing task or not. In the tabular case you can always check to see if a terminal state exists since every state is available, but for the non-tabular case, all we can do is note the problem if a terminal state is encountered.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$4915b1ed-ad53-4ece-9b00-bc136d47d8dccell_id$4915b1ed-ad53-4ece-9b00-bc136d47d8dccodemd"""
It is implicit in all expressions below that $\pi$ is a function of $\boldsymbol{\theta}$ and that the gradients are with respect to $\boldsymbol{\theta}$. The performance measure for the continuing case is $J(\boldsymbol{\theta}) = r(\boldsymbol{\theta})$ (13.15) and all value functions use the definition of the differential return. We begin by expressing the gradient of the state value function in terms of the state-action value function, the policy, the average return and gradients thereof:
$\begin{flalign}
\nabla v_\pi(s) &= \nabla \left [ \sum_a \pi(a \vert s) q_\pi (s, a) \right ], \: \forall s \in \mathcal{S} \\
&= \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \nabla q_\pi(s, a) \right ] \tag{product rule} \\
&=\sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \nabla \sum_{s^\prime, r} p(s^\prime, r, \vert s, a)\left (r - r(\boldsymbol{\theta}) + v_\pi(s^\prime) \right ) \right ] \tag{differential return definitions} \\
&=\sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) [ -\nabla r(\boldsymbol{\theta}) + \sum_{s^\prime} p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) ] \right ] \tag{distributing gradient}\\
\end{flalign}$
The purpose of this expression is to isolate the term which is the gradient of the average return since this is the performance metric gradient we originally sought. Note that if we separate the terms inside the sum, the one with the gradient of $r$ is $\sum_a \pi(a\vert s) [- \nabla r(\boldsymbol{\theta})] = -\nabla r(\boldsymbol{\theta}) \sum_a \pi(a \vert s)$. But the policy function is a probability distribution so its sum over actions is just 1. Therefore, this term simplifies to just $-\nabla r(\boldsymbol{\theta})$ which we can simply move to the other side of the expression swapping its place with the state value function:
$\begin{flalign}
\nabla v_\pi(s)&=-\nabla r(\boldsymbol{\theta}) + \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) \right ] \\
\nabla r(\boldsymbol{\theta}) &=-\nabla v_\pi(s) + \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) \right ]
\end{flalign}$
Now the left hand side is $\nabla J(\boldsymbol{\theta})$ and does not depend on $s$. As such, the right hand side as a whole must be independent of $s$ as well so we are free to take a weighted sum of it over some probability distribution on $s$ since all the terms sum to 1. That is, if $f$ is independent of $s$, then $f = \sum_s \mu(s) f = f \sum_s \mu(s) = f \times 1 = f$:
$\begin{flalign}
\nabla J(\boldsymbol{\theta}) &= \sum_s \mu(s) \left ( \sum_a \left [ \nabla \pi(a \vert s) q_\pi(s, a) + \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) \right ] - \nabla v_\pi(s) \right ) \\
&= \sum_s \mu(s) \sum_a \nabla \pi(a \vert s) q_\pi(s, a) + \sum_s \mu(s) \sum_a \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) - \sum_s \mu(s) \nabla v_\pi(s) \tag{separating sum terms}\\
&= \sum_s \mu(s) \sum_a \nabla \pi(a \vert s) q_\pi(s, a) + \sum_{s^\prime} \sum_s \mu(s) \sum_a \pi(a \vert s) p(s^\prime \vert s, a) \nabla v_\pi(s^\prime) - \sum_s \mu(s) \nabla v_\pi(s) \tag{swapping sum order in second term}\\
&= \sum_s \mu(s) \sum_a \nabla \pi(a \vert s) q_\pi(s, a) + \sum_{s^\prime} \mu(s^\prime) \nabla v_\pi(s^\prime) - \sum_s \mu(s) \nabla v_\pi(s) \tag{stationary state distribution definition}\\
&= \sum_s \mu(s) \sum_a \nabla \pi(a \vert s) q_\pi(s, a) \tag{cancelling equivalent sum terms}\\
&= \mathbb{E}_\pi \left [ \sum_a \nabla \pi(a \vert S_t) q_\pi(S_t, a) \right ] \tag{expected value definition}\\
&= \mathbb{E}_\pi \left [ \sum_a \pi(a \vert S_t) \frac{\nabla \pi(a \vert S_t)}{\pi(a \vert S_t)} q_\pi(S_t, a) \right ] \tag{multiplying and dividing by the policy}\\
&= \mathbb{E}_\pi \left [\frac{\nabla \pi(A_t \vert S_t)}{\pi(A_t \vert S_t)} q_\pi(S_t, A_t) \right ] \tag{expected value definition}\\
&= \mathbb{E}_\pi \left [\frac{\nabla \pi(A_t \vert S_t)}{\pi(A_t \vert S_t)} G_t \right ] \tag{differential return definition}\\
&= \mathbb{E}_\pi \left [G_t \nabla \ln \pi(A_t \vert S_t) \right ] \tag{chain rule}\\
\end{flalign}$
The expression inside the expected value can be sampled on every time step and the gradient is only in terms of the policy function which we have selected as something differentiable with respect to the parameters. Since this method will only be used for continuing problems, we cannot rely on Monte Carlo sampling for the differential return. Instead, our only option is to use a bootstrap value estimate in combination with a running estimate of the average reward and the immediate sample reward: $R - \overline{R} + \hat v^\prime$ where $\hat v^\prime$ is the differential value function estimate at the transition state and $\overline{R}$ is an estimate of the average reward. We can apply the existing actor-critic algorithms to these continuing problems as long as we track that additional information and use an additional step size parameter to update the average reward estimate. This step size parameter replaces the discount rate. See a full implementation below:
"""metadatashow_logsèdisabled®skip_as_script«code_folded$f924eb30-d1cc-4941-8fb5-ff70ad425ab9cell_id$f924eb30-d1cc-4941-8fb5-ff70ad425ab9codemd"""
## 13.3 REINFORCE: Monte Carlo Policy Gradient
If we replace the true action-value function in (13.5) with a learned approximation $\hat q_\pi$, then we have a method called the *all-actions* method because the update involves the sum over all actions. For the REINFORCE algorithm, we instead sample this value using the actual return and the policy distribution.
We can re-write (13.5) using an expected value under the policy and continue from there:
$\begin{flalign}
\nabla J(\boldsymbol{\theta}) & \propto \mathbb{E}_\pi \left [ \gamma^t \sum_a q_\pi (S_t, a) \nabla \pi(a|S_t, \boldsymbol{\theta}) \right ] \tag{13.6}\\
&= \mathbb{E}_\pi \left [\gamma^t \sum_a \pi(a|S_t, \boldsymbol{\theta}) q_\pi (S_t, a) \frac{\nabla \pi(a|S_t, \boldsymbol{\theta})}{\pi(a|S_t, \boldsymbol{\theta})} \right ] \tag{multiply and divide by policy} \\
&= \mathbb{E}_\pi \left [ \gamma^t q_\pi (S_t, A_t) \frac{\nabla \pi(A_t|S_t, \boldsymbol{\theta})}{\pi(A_t|S_t, \boldsymbol{\theta})} \right ] \tag{replace a with sample under policy} \\
&= \mathbb{E}_\pi \left [ \gamma^t G_t \frac{\nabla \pi(A_t|S_t, \boldsymbol{\theta})}{\pi(A_t|S_t, \boldsymbol{\theta})} \right ] \tag{replace value with sample return} \\
\end{flalign}$
Using the expression in the brackets we can write down an update rule for the parameters that can be sampled on each time step. This is the **REINFORCE update**:
$\begin{align}
\boldsymbol{\theta}_{t+1} \doteq \boldsymbol{\theta}_t + \alpha \gamma^t G_t \frac{\nabla \pi(A_t|S_t, \boldsymbol{\theta}_t)}{\pi(A_t|S_t, \boldsymbol{\theta}_t)} \tag{13.8}
\end{align}$
Because it uses all future returns after step t, REINFORCE is a Monte Carlo algorithm and is well defined only for the episodic case. For implementation purposes we can replace $\frac{\nabla \pi(A_t|S_t, \boldsymbol{\theta})}{\pi(A_t|S_t, \boldsymbol{\theta})}$ with $\nabla \ln \pi(A_t|S_t, \boldsymbol{\theta}_t)$ which is usually refered to as the *eligibility vector*.
With the alternative parameterization, the eligibility vector is $\nabla \ln \pi(S_t, \theta_t)_{A_t}$ where $\pi$ is a vector and the $A_t$ subscript takes the value of that vector at the index corresponding to the action $A_t$.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$d83dc659-dce7-41dd-a8e7-2933ab39d15ccell_id$d83dc659-dce7-41dd-a8e7-2933ab39d15ccodemd"""
### *REINFORCE with Baseline Implementation*
These functions use two sets of parameters, one to calculate the policy function and another to calculate the state value function. The state representation vector is shared between the two functions, but the policy function will return a distribution of preferences over actions while the value function will return a single value. If linear approximation is used to estimate both functions, the the policy parameters $\boldsymbol{\theta}$ will be a $d \times N_a$ matrix where $d$ is the length of the state feature vector representation and the value function parameters $\mathbf{w}$ will be a length $d$ vector. It is also possible to mix linear and non-linear approximation with this method.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fcecell_id$7f77d574-8f65-4e1e-8f5f-6f1bcccc3fcecodendisplay_cartpole_episode(cartpole_fcann_continuing_test_episode[1], cartpole_fcann_continuing_test_episode[2])metadatashow_logsèdisabled®skip_as_script«code_folded$83ca0577-15d7-4448-b597-c77810b812bfcell_id$83ca0577-15d7-4448-b597-c77810b812bfcodefunction figure_13_2_test(α_list, α_pair_list; nruns = 100, num_episodes = 1_000, max_steps = 1_000)
Random.seed!(45)
function average_runs(α)
1:nruns |> Map(_ -> reinforce_monte_carlo_control_binary_features(corridor_mdp, get_corridor_features, 1, num_episodes, params = [0f0 3.7f0], α = α, max_steps = max_steps).episode_rewards) |> foldxt((a, b) -> a .+ b) |> v -> v ./ nruns
end
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> reinforce_with_baseline_monte_carlo_control_binary_features(corridor_mdp, get_corridor_features, 1, num_episodes, policy_params = [0f0 3.7f0], α_θ = α_θ, α_w = α_w, max_steps = max_steps).episode_rewards) |> foldxt((a, b) -> a .+ b) |> v -> v ./ nruns
end
traces1 = [begin
out = average_runs(α)
scatter(x = 1:num_episodes, y = out, name = name = "α = 2^$(round(Int64, log2(α)))")
end
for α in α_list]
traces2 = [begin
out = average_runs(αs...)
scatter(x = 1:num_episodes, y = out, name = name = "α_θ = 2^$(round(Int64, log2(αs[1]))) and α_w = 2^$(round(Int64, log2(αs[2])))")
end
for αs in α_pair_list]
baselinetrace = scatter(x = 1:num_episodes, y = fill(-2*sqrt(2) / (3*sqrt(2) - 4), num_episodes), name = "ideal value", line_dash = "dash", line_color = "gray")
plot([baselinetrace; traces1; traces2], Layout(yaxis_range = [-90, -10], yaxis_title = "Total reward on episode (averaged over $nruns runs)", xaxis_title = "Episode", width = 800))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a7c9ae69-f4b8-471c-ab97-90642f3c2bdbcell_id$a7c9ae69-f4b8-471c-ab97-90642f3c2bdbcode/function reinforce_with_baseline_monte_carlo_control_binary_features(mdp::StateMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, max_episodes::Integer; policy_params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
setup = setup_binary_policy_arguments(mdp, get_active_features, num_features)
reinforce_with_baseline_monte_carlo_control!(policy_params, setup.eligibility_vector, value_params, BinaryFeatureVector(), mdp, update_binary_action_preferences!, update_binary_eligibility_vector!, setup.feature_vector, setup.update_feature_vector!, binary_value_function, update_binary_value_gradient!, max_episodes; action_preferences = setup.action_preferences, kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a7dcc8cd-04ec-48f2-a387-116330eaffb2cell_id$a7dcc8cd-04ec-48f2-a387-116330eaffb2codegfigure_13_2_test([2f0^-13], vcat([(2f0^n, 2f0^-4) for n in -12:-10], [(2f0^n, 2f0^-2) for n in -8:-6]))metadatashow_logsèdisabled®skip_as_script«code_folded$0ab70fc3-6188-42eb-aba2-d808f319be9fcell_id$0ab70fc3-6188-42eb-aba2-d808f319be9fcodemd"""
# Dependencies
"""metadatashow_logsèdisabled®skip_as_script«code_folded$047656d1-2921-40f2-b75b-ce4a87098007cell_id$047656d1-2921-40f2-b75b-ce4a87098007code1md"""
### Switched Corridor Parameter Studies
"""metadatashow_logsèdisabled®skip_as_script«code_folded$5d434c83-c9ca-499f-8695-c7733031c2decell_id$5d434c83-c9ca-499f-8695-c7733031c2decodeffunction cartpole_continuing_step(s::CartPoleState, i_a::Integer)
s′ = cartpole_functions.step(s, cartpole_functions.discrete_actions[i_a])
if cartpole_functions.failure(s′)
s′ = cartpole_functions.initialize_state()
s′ = CartPoleState(s′.x, s′.θ, s′.ẋ, s′.θ̇, s.t+cartpole_functions.h)
(-1f0, s′)
else
(0f0, s′)
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$3a37b53d-9174-4faa-9404-74a40c385b0acell_id$3a37b53d-9174-4faa-9404-74a40c385b0acodeYshow_mountaincar_trajectory(mountaincar_continuing_fcann_test.policy_sample_action, 1000)metadatashow_logsèdisabled®skip_as_script«code_folded$820752af-8966-4ee8-82f7-a40934522de5cell_id$820752af-8966-4ee8-82f7-a40934522de5codePtest_study2 = actor_critic_fcann_parameter_study(cartpole_continuing_mdp, cartpole_vector_update!, cartpole_fcann_feature_setup.num_features, [4, 4], LinRange(0f0, .95f0, 20), LinRange(0.0f0, .95f0, 20), [0.005f0, 0.01f0, 0.05f0], 2f0 .^ (-8:-2), 2f0 .^ (-8:-2), 100, 100_000; nruns = 40, seed = 45) |> df -> sort(df, :output; rev=true)metadatashow_logsèdisabledîskip_as_script«code_folded$6acb549a-5d90-4457-a347-d22448ad8071cell_id$6acb549a-5d90-4457-a347-d22448ad8071codeق@bind cartpole_fcann_continuing_episode_step_select Slider(1:length(cartpole_fcann_continuing_test_episode[1]); show_value = true)metadatashow_logsèdisabled®skip_as_script«code_folded$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62cell_id$f52fc4a9-f6dd-422d-aeae-6c327d1a7b62codecartpole_fcann_continuing_parameter_study(layer_size::Integer, num_layers::Integer, args...; kwargs...) = actor_critic_fcann_parameter_study(cartpole_continuing_mdp, cartpole_vector_update!, cartpole_fcann_feature_setup.num_features, fill(layer_size, num_layers), args...; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$3bccf6fc-6e5e-4f62-ad40-1ff0a3740728cell_id$3bccf6fc-6e5e-4f62-ad40-1ff0a3740728codeactor_critic_with_eligibility_traces_binary_features(corridor_mdp, 0f0, 0f0, get_corridor_features, 1, typemax(Int64), 100_000, α_θ = 2f0 ^ -4, α_w = 2f0 ^ -10, policy_params = [0f0 3.7f0]).policy_and_value(1)metadatashow_logsèdisabled®skip_as_script«code_folded$ae0f5a96-7a4b-47f9-be1e-e803a238a071cell_id$ae0f5a96-7a4b-47f9-be1e-e803a238a071code@md"""
### *MDP Types and Transitions for Continuous Actions*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$41d62de1-2c92-41ee-9430-b9ca3007afd9cell_id$41d62de1-2c92-41ee-9430-b9ca3007afd9codeumd"""
The above matrix represents an estimate of $\Pr \{ S_k = s \mid \pi \}$; however note that the terminal states are excluded from the rows. This corridor problem only has three non-terminal states. If we sum across each row, then we have the probability of reaching that step prior to terminating. The vector defined below measures the probability of an episode terminating prior to each step. Notably, this probablity is 0 for the first three steps since no policy starting from the left can terminate that quickly. As expected, the probability of terminating under the random policy grows with time approaching 1.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$8eb42403-1234-4e59-993e-057cc3a6d5c9cell_id$8eb42403-1234-4e59-993e-057cc3a6d5c9codeCif run_mountaincar_binary_episodic_param_study > 0
actor_critic_binary_episodic_parameter_study(MountainCarTask.mdp, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, mountaincar_binary_params, 5, 3, 1000; max_steps = 100_000)
else
md"""
Waiting to run parameter study
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$bbc8864a-1545-433f-bc7c-0ddf6e907138cell_id$bbc8864a-1545-433f-bc7c-0ddf6e907138codefunction plot_mountaincar_policy_values(policy_and_value::Function; n1 = 100, n2 = 100)
xvals = LinRange(-1.2f0, 0.5f0, n1)
vvals = LinRange(-0.07f0, 0.07f0, n2)
values = zeros(Float32, n1, n2)
action_dists = [zeros(Float32, n1, n2) for i in 1:3]
for (i, x) in enumerate(xvals)
for (j, v) in enumerate(vvals)
π, v̂ = policy_and_value((x, v))
values[j, i] = v̂
for k in 1:3
action_dists[k][j, i] = π[k]
end
end
end
p1 = plot(heatmap(x = xvals, y = vvals, z = values), Layout(xaxis_title = "position", yaxis_title = "velocity", title = "Learned Value Function", height = 400, width = 600))
p2 = [plot(heatmap(x = xvals, y = vvals, z = action_dists[k], colorscale = "rb"), Layout(xaxis_title = "position", yaxis_title = "velocity", title = "Policy Probability for Action $k", height = 400, width = 600)) for k in 1:3]
@htl("""
$p1
$p2
""")
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a12b92d1-e045-4f92-b8cd-eee5d56fa67dcell_id$a12b92d1-e045-4f92-b8cd-eee5d56fa67dcodeٹconst best_mc_corridor = reinforce_with_baseline_monte_carlo_control_linear_features(corridor_mdp, update_corridor_features!, 1, 100; α_θ = 0.006f0, α_w = 2f0^-2, max_steps = 1_000)metadatashow_logsèdisabled®skip_as_script«code_folded$ce33f710-fd9d-4dfa-acda-40204e54d518cell_id$ce33f710-fd9d-4dfa-acda-40204e54d518codemd"""
## 13.5 Actor-Critic Methods
Here we also use the value function estimator to calculate the the return estimate using the one step bootstrap return. When the state value function is used in this way we call it the *critic*. In general we can use this function with n-step returns and eligibility traces. Recall from the subject of TD learning of value functions that the one-step return is often superior to the actual return regarding variance and ease of computation, although it does introduce bias to the estimate. With the use of eligibility traces we can smoothly vary arbitrarily close to the Monte Carlo return. Note that the bias in the gradient estimate is n due to the bootstrapping as such; the actor would be biased even if the critic was learned by a Monte Carlo method.
The one-step actor-critic method is the analog of the one step methods such as TD$(0)$, Sarsa$(0)$, and Q learning. These methods replace the full return of REINFORCE with the one step return as follows:
$\begin{flalign}
\boldsymbol{\theta}_{t+1} &\doteq \boldsymbol{\theta}_t + \alpha(G_{t:t+1} - \hat v(S_t, \mathbf{w}))\ln\nabla\pi(A_t|S_t, \mathbf{\theta_t}) \tag{13.12} \\
& = \boldsymbol{\theta}_t + \alpha(R_{t+1} + \gamma \hat v(S_{t+1}, \mathbf{w}) - \hat v(S_t, \mathbf{w}))\ln\nabla\pi(A_t|S_t, \mathbf{\theta_t}) \tag{13.13} \\
& = \boldsymbol{\theta}_t + \delta_t\ln\nabla\pi(A_t|S_t, \mathbf{\theta_t}) \tag{13.14} \\
\end{flalign}$
This can be implemented as a fully online algorithm because we do not have to wait until the end of an episode to calculate return estimates. The natural state-value-function learning method to pair with this is semi-gradient TD(0). See a full implementation below.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$339b4d2b-2237-46a3-9867-ecc3332856c1cell_id$339b4d2b-2237-46a3-9867-ecc3332856c1code!md"""
This expression repeats terms of the form $\nabla \pi(a \vert s) q_\pi(s, a)$ summed over different probabilities. The first appearance of this term is just a sum over all actions at the state $s$ which is the state we are using for the gradient expression. The next appearance of the expression is a sum over actions at state $s^\prime$. Let's define a new expressions:
$\begin{flalign}
f(s) &\doteq \sum_a \nabla \pi(a \vert s) q_\pi(s, a) \\
\end{flalign}$
Then we can rewrite the second term as follows:
$\gamma \sum_a \left [ \pi(a \vert s) \sum_{s^\prime} p(s^\prime \vert s, a) f(s^\prime) \right ] = \gamma \sum_{s^\prime} f(s^\prime) \sum_a \left [ \pi(a \vert s) p(s^\prime \vert s, a) \right ] = \gamma \mathbb{E}_\pi [f(s^\prime) \vert s] = \gamma \sum_{s^\prime} f(s ^\prime) \Pr \{ S_1 = s^\prime \mid S_0 = s, A_1 \sim \pi(s) \}$
Define a new term $g(s) = \sum_{s^\prime} f(s^\prime) \Pr \{ S_1 = s^\prime \vert S_0 = s, A_1 \sim \pi(s) \} = \sum_{s^\prime} f(s^\prime) \sum_a [\pi(a \vert s) p(s^\prime \vert s, a)$
So the second term can be written as $\gamma g(s)$
where the final expression uses the probability that the agent transitions from state $s$ to $s^\prime$ in one step under the policy $\pi$. Using this same logic, we can rewrite the third expression as well.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$a8349352-3242-46d5-b0d5-1b6eb5d77e90cell_id$a8349352-3242-46d5-b0d5-1b6eb5d77e90code4@bind x Slider(-50:50; default = 0, show_value=true)metadatashow_logsèdisabled®skip_as_script«code_folded$7d63b960-3998-4f7b-8cbb-ccd49db9aeaccell_id$7d63b960-3998-4f7b-8cbb-ccd49db9aeaccodeٻone_step_actor_critic_binary_features(corridor_mdp, get_corridor_features, 1, typemax(Int64), 100_000, α_θ = 2f0 ^ -3, α_w = 2f0 ^ -10, policy_params = [0f0 3.7f0]).policy_and_value(1)metadatashow_logsèdisabled®skip_as_script«code_folded$65d2add6-fd6f-456c-92ed-3cd9d1862ef6cell_id$65d2add6-fd6f-456c-92ed-3cd9d1862ef6codePfunction update_binary_policy_params!(params::Matrix{T}, active_features::BinaryFeatures, i_a::Integer, π_dist::Vector{T}, c::T) where T<:Real
@inbounds for i in eachindex(π_dist)
for j in active_features
params[j, i] -= c*π_dist[i]
end
end
@inbounds for j in active_features
params[j, i_a] += c
end
return params
endmetadatashow_logsèdisabled®skip_as_script«code_folded$f55afa58-962d-4551-8d95-a5b467d61adfcell_id$f55afa58-962d-4551-8d95-a5b467d61adfcode mbegin
function update_params_with_gradient!(θ::Matrix{T}, α::T, ∇θ::BinaryGaussianEligibilityVector{T, T, T, B}) where {T<:Real, B<:BinaryFeatureVector}
c1 = ∇θ.a - ∇θ.μ
c2 = ∇θ.σ^(-2)
# isnan(c2) && @info "warning σ of $∇θ.σ is causing nan results"
# isinf(c2) && @info "warning σ of $∇θ.σ is causing inf results"
c3 = c1 * c2
c4 = c3*c1 - one(T)
δ1 = α*c3
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
θ[i, 1] += δ1
end
δ2 = α*c4
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
θ[i, 2] += δ2
end
return θ
end
function update_params_with_gradient!(θ::Matrix{T}, α::T, ∇θ::BinaryBetaEligibilityVector{T, T, T, B}) where {T<:Real, B<:BinaryFeatureVector}
c1 = digamma(∇θ.α + ∇θ.β)
δ1 = α*(log(∇θ.a) + c1 - digamma(∇θ.α))
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
θ[i, 1] += δ1
end
δ2 = α*(log(one(T) - ∇θ.a) + c1 - digamma(∇θ.β))
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
θ[i, 2] += δ2
end
return θ
end
function update_params_with_gradient!(θ::Matrix{T}, α::T, ∇θ::BinarySquashedGaussianEligibilityVector{T, T, T, B}) where {T<:Real, B<:BinaryFeatureVector}
c1 = atanh(∇θ.a/∇θ.amax) - ∇θ.μ
c2 = ∇θ.σ^(-2)
# isnan(c2) && @info "warning σ of $∇θ.σ is causing nan results"
# isinf(c2) && @info "warning σ of $∇θ.σ is causing inf results"
c3 = c1 * c2
c4 = c3*c1 - one(T)
δ1 = α*c3
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
θ[i, 1] += δ1
end
δ2 = α*c4
@inbounds @simd for j in 1:∇θ.binary_features.num_features
i = ∇θ.binary_features.active_features[j]
θ[i, 2] += δ2
end
return θ
end
function update_params_with_gradient!(θ::Matrix{T}, α::T, ∇θ::BinaryGaussianEligibilityVector{T, NTuple{N, T}, Vector{T}, B}) where {T<:Real, N, B<:BinaryFeatureVector}
for k in 1:N
c1 = ∇θ.a[k] - ∇θ.μ[k]
c2 = ∇θ.σ[k] ^-2
# isnan(c2) && @info "warning σ of $∇θ.σ is causing nan results"
# isinf(c2) && @info "warning σ of $∇θ.σ is causing inf results"
c3 = c1 * c2
c4 = c3*c1 - one(T)
δ1 = α*c3
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k] += δ1
end
δ2 = α*c4
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k+N] += δ2
end
end
return θ
end
function update_params_with_gradient!(θ::Matrix{T}, α::T, ∇θ::BinaryBetaEligibilityVector{T, NTuple{N, T}, Vector{T}, B}) where {T<:Real, N, B<:BinaryFeatureVector}
for k in 1:N
c1 = digamma(∇θ.α[k] + ∇θ.β[k])
δ1 = α*(log(∇θ.a[k]) + c1 - digamma(∇θ.α[k]))
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k] += δ1
end
δ2 = α*(log(one(T) - ∇θ.a[k]) + c1 - digamma(∇θ.β[k]))
@inbounds @simd for i in 1:size(θ, 1)
θ[i, k+N] += δ2
end
end
return θ
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$d9d11d69-bc16-400a-8f46-f9a8ecb8516acell_id$d9d11d69-bc16-400a-8f46-f9a8ecb8516acode,actor_critic_binary_episodic_parameter_study(mdp::StateMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, params::@NamedTuple{λ_θ::T, λ_w::T, α_θ_min::Int64, α_w_min::Int64}, num_θ::Integer, num_w::Integer, num_episodes::Integer; kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = actor_critic_binary_episodic_parameter_study(mdp, get_active_features, num_features, params.λ_θ, params.λ_w, 2f0 .^(params.α_θ_min:params.α_θ_min+num_θ-1), 2f0 .^(params.α_w_min:params.α_w_min+num_w-1), num_episodes; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$ed93259c-7b8b-46d7-97fb-f194e0e04b3acell_id$ed93259c-7b8b-46d7-97fb-f194e0e04b3acodefunction setup_binary_beta_policy_arguments(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer) where {T<:Real, S, N, A<:Union{T, NTuple{N, T}}, P, F1, F2, F3}
x = BinaryFeatureVector()
update_feature_vector!(x::BinaryFeatureVector, s) = update_binary_feature_vector!(x, s, get_active_features)
sample_action = rand(A)
action_dist_params = make_n_param_dist_params(2, sample_action)
∇lnπ = BinaryBetaEligibilityVector(sample_action)
return (feature_vector = x, update_feature_vector! = update_feature_vector!, action_distribution_parameters = action_dist_params, eligibility_vector = ∇lnπ)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$d1ed25e6-60c6-411f-a541-99986e5da2c5cell_id$d1ed25e6-60c6-411f-a541-99986e5da2c5codereinforce_with_baseline_monte_carlo_control_linear_features(mdp::StateMDP{T, S, A, P, F1, F2, F3}, update_feature_vector!::Function, num_features::Integer, max_episodes::Integer; policy_params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), value_params::Vector{T} = zeros(T, num_features), x = zeros(T, num_features), action_preferences = zeros(T, length(mdp.actions)), kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = reinforce_with_baseline_monte_carlo_control!(policy_params, copy(policy_params), value_params, copy(value_params), mdp, update_linear_action_preferences!, update_linear_eligibility_vector!, x, update_feature_vector!, linear_value_function, update_linear_value_gradient!, max_episodes; action_preferences = action_preferences, kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$b966b248-fb4d-457d-90f6-114370846242cell_id$b966b248-fb4d-457d-90f6-114370846242codeٱbegin
bad_continuous_action(a) = false
bad_continuous_action(a::Real) = isnan(a)
bad_continuous_action(a::NTuple{N, T}) where {N, T<:Real} = any(bad_continuous_action, a)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$4156d955-9daf-4429-b152-e8332980fb9ecell_id$4156d955-9daf-4429-b152-e8332980fb9ecode7const mountaincar_continuous_test_train_beta = actor_critic_with_eligibility_traces_binary_features_beta_actions(mountaincar_continuous_beta_mdp, 0.01f0, 0.99f0, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, typemax(Int64), 100_000; α_θ = 1f-4, α_w = 0.00002f0)metadatashow_logsèdisabled®skip_as_script«code_folded$b09e1e48-494e-4967-826a-6e70199acad4cell_id$b09e1e48-494e-4967-826a-6e70199acad4code+md"""
### Squashed Gaussian Alternative
"""metadatashow_logsèdisabled®skip_as_script«code_folded$734573e5-547b-4dcc-89bb-412aa6cc42d6cell_id$734573e5-547b-4dcc-89bb-412aa6cc42d6codefunction actor_critic_linear_parameter_study(mdp::StateMDP{T, S, A, P, F1, F2, F3}, feature_function::Function, num_features::Integer, λ_θ::T, λ_w::T, α_r̄::T, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, max_steps::Integer; nruns::Integer = 100, seed = rand(UInt64), init_policy_params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), binary_features = false, kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
if binary_features
algo = actor_critic_with_eligibility_traces_binary_features
title_prefix = "Binary Feature Encoding"
else
algo = actor_critic_with_eligibility_traces_linear_features
title_prefix = "Linear Encoding"
end
make_trace_data(α_θ_list, α_w) = [average_continuing_runs(nruns, seed, α_θ, α_w, α_r̄, init_policy_params, algo, mdp, λ_θ, λ_w, feature_function, num_features, max_steps; kwargs...) for α_θ in α_θ_list]
traces = [begin
scatter(x = α_θ_list, y = make_trace_data(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Step in the First $max_steps Steps Averaged Over $nruns Runs", xaxis_type = "log", title = "$title_prefix with $num_features Features, λ_θ = $λ_θ, λ_w = $λ_w"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54cell_id$97b7ce3f-6d1e-41bc-ba07-50e8516a2d54codefunction actor_critic_with_eligibility_traces_fcann(mdp::StateMDP{T, S, A, P, F1, F2, F3}, λ_θ::T, λ_w::T, input_length::Integer, hidden_layers::Vector{Int64}, update_feature_vector!::Function, args...; policy_params::FCANNParams = FCANN.initializeparams_saxe(input_length, hidden_layers, length(mdp.actions)), reslayers = 0, l2 = 0f0, dropout = 0f0, use_μP = true, activation_list = fill(true, length(hidden_layers)), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
setup = setup_fcann_policy_and_value_arguments(policy_params, input_length, hidden_layers, reslayers, l2, dropout, use_μP, activation_list)
actor_critic_with_eligibility_traces!(policy_params, setup.eligibility_vector, setup.value_params, setup.value_gradient, mdp, λ_θ, λ_w, setup.update_action_preferences!, setup.update_eligibility_vector!, setup.feature_vector, update_feature_vector!, setup.value_function, setup.gradient_update, args...; kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$692c1043-4eaf-491e-b8fe-368618867f99cell_id$692c1043-4eaf-491e-b8fe-368618867f99codemd"""
1. The soft-max distribution is:
$\sigma(a|s, \theta) = \frac{e^{h(s, a, \theta)}}{\sum_b e^{h(s, b, \theta)}}$
We only have two possible actions in each state so the policy for action 1 would be given by:
$\pi(1|S_t, \theta_t) = \frac{e^{h(s, 1, \theta_t)}}{e^{h(S_t, 0, \theta_t)} + e^{h(S_t, 1, \theta)}}$
Simplify this expression by dividing by $e^{h(s, 1, \theta_t)}$ which results in:
$\pi(1|S_t, \theta_t) = \frac{1}{e^{h(S_t, 0, \theta_t) - h(S_t, 1, \theta_t)} + 1}$
Given the assumption that $h(s, 1, \theta)-h(s, 0, \theta) = \theta^\top\mathbf{x}(s)$, we replace the expression in the exponent resulting in the final expression of:
$\pi(1|S_t, \theta_t) = \frac{1}{e^{-\theta_t^\top\mathbf{x}(S_t)} + 1}$
Using the notation $f(x) = 1/(1+e^{-x})$ we can write $\pi(1|S_t, \theta_t) = f(\theta_t^\top \mathbf{x}(S_t))$ where $f$ is the logistic function. Consider this notation for the rest of the exercises.
2. The REINFORCE update is given by: $\theta_{t+1} = \theta_t + \alpha G_t \frac{\nabla\pi(A_t|S_t, \theta_t)}{\pi(A_t|S_t, \theta_t)}$, so we need to compute the gradient of the policy in terms of the parameters for this action selection: $\nabla \pi(1|S_t, \theta_t)$. Luckily, the derivative of the logistic function is simply given by: $f(x)(1-f(x))$ where $f(x)$ is the logistic function itself. In our case $x = \theta_t^\top \mathbf{x}_t$ so after applying the chain rule we have:
$\nabla\pi(1|S_t, \theta_t) = f(x)(1-f(x))\nabla x = f(x)(1-f(x)) \mathbf{x_t}$ since $x$ is just a linear function of the parameters. So for the parameter update step we have:
$\frac{\nabla\pi(1|S_t, \theta_t)}{\pi(1|S_t, \theta_t)} = \frac{f(x)(1-f(x))\mathbf{x}_t}{f(x)} = (1 - f(x))\mathbf{x}_t$
Also note that:
$1 - f(x) = 1 - \frac{1}{e^{-\theta_t^\top\mathbf{x}(S_t)} + 1} = \frac{e^{-\theta_t^\top\mathbf{x}(S_t)} + 1 - 1}{e^{-\theta_t^\top\mathbf{x}(S_t)} + 1} = \frac{e^{-\theta_t^\top\mathbf{x}(S_t)}}{e^{-\theta_t^\top\mathbf{x}(S_t)} + 1}$
The REINFORCE update will then be:
$\theta_{t+1} = \theta_t + \alpha G_t \left ( \frac{e^{-\theta_t^\top\mathbf{x}(S_t)}}{e^{-\theta_t^\top\mathbf{x}(S_t)} + 1} \right ) \mathbf{x}_t$
3. For the general case, we want to calculate $\frac{\nabla\pi(a|s, \theta)}{\pi(a|s, \theta)}$. We already know this expression for $a = 1$.
$\nabla {\pi(1|s, \mathbf{\theta})} = f(x)(1 - f(x))\mathbf{x}(s) = \pi(1|s, \mathbf{\theta})(1 - \pi(1|s, \mathbf{\theta})\mathbf{x}(s)$
Since $\pi(a|s, \theta)$ is a probability distribution across actions, we also know that
$\pi(0|s, \theta) = 1 - \pi(1|s, \theta)$
which implies that
$\nabla \pi(0|s, \theta) = -\nabla \pi(1|s, \theta) = -\pi(1|s, \mathbf{\theta})(1 - \pi(1|s, \mathbf{\theta}))\mathbf{x}(s)$
We can express this in terms of $\pi(0|s, \theta)$ completely:
$\nabla \pi(0|s, \theta) = (\pi(0|s, \mathbf{\theta}) - 1)\pi(0|s, \theta)\mathbf{x}(s) = -\pi(0|s, \theta)(1 - \pi(0|s, \mathbf{\theta}))\mathbf{x}(s)$
Let's now compare the two expressions for the policy gradient at each action:
$\begin{align}
\nabla {\pi(1|s, \mathbf{\theta})} &= \pi(1|s, \mathbf{\theta})(1 - \pi(1|s, \mathbf{\theta})\mathbf{x}(s) \\
\nabla \pi(0|s, \theta) &= -\pi(0|s, \theta)(1 - \pi(0|s, \mathbf{\theta}))\mathbf{x}(s) \\
\therefore \\
\nabla \pi(a|s, \theta) &= \chi (a) \pi(a|s, \theta)(1 - \pi(a|s, \mathbf{\theta}))\mathbf{x}(s) \\
\end{align}$
Where $\chi (a)$ is a function that returns 1 for $a=1$ and -1 for $a=0$. There are many ways to achieve this but the following expression is simple and works: $\chi(a) = 2a - 1$. Dividing by the policy yields a unified expression for the eligibility vector:
$\nabla \ln{\pi(a|s,\theta)} = (2a - 1) (1 - \pi(a|s, \mathbf{\theta}))\mathbf{x}(s)$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$2c5d221a-2469-49e1-9249-dfdc2457f2facell_id$2c5d221a-2469-49e1-9249-dfdc2457f2facode\@bind start_cartpole_continuing_fcann_param_study CounterButton("Run FCANN Parameter Study")metadatashow_logsèdisabled®skip_as_script«code_folded$7c592385-e8d3-4efe-962c-d39debb64405cell_id$7c592385-e8d3-4efe-962c-d39debb64405code~const mountaincar_tilecoding_setup = tile_coding_setup(mountaincar_min_vals, mountaincar_max_vals, (0.1f0, 0.1f0), 12, (1, 3))metadatashow_logsèdisabled®skip_as_script«code_folded$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebcell_id$16ae3aa6-8f28-4cb0-a15f-7a96c01cdaebcodeimport HypertextLiteral.@htlmetadatashow_logsèdisabled®skip_as_scriptëcode_folded$8eab55a5-41b7-4f5e-a02f-4c19388bc9eacell_id$8eab55a5-41b7-4f5e-a02f-4c19388bc9eacodeifunction update_binary_feature_vector!(x::BinaryFeatureVector, s::S, get_active_features::Function) where S
active_features = get_active_features(s)
l = length(x.active_features)
n = 0
for (i, f) in enumerate(active_features)
if i > l
push!(x.active_features, f)
else
x.active_features[i] = f
end
n += 1
end
x.num_features = n
return x
endmetadatashow_logsèdisabled®skip_as_script«code_folded$0ac7ea44-14f6-4e80-80f9-d6df8059bb38cell_id$0ac7ea44-14f6-4e80-80f9-d6df8059bb38codefunction reinforce_monte_carlo_control!(policy_params, ∇lnπ, mdp::StateMDP{T, S, A, PTF, F1, F2, F3}, update_action_preferences!::Function, update_eligibility_vector!::Function, x, update_feature_vector!::Function, max_episodes::Integer; α = one(T)/10, kwargs...) where {T<:Real, S, A, PTF, F1, F2, F3}
out = reinforce_with_baseline_monte_carlo_control!(policy_params, ∇lnπ, nothing, nothing, mdp, update_action_preferences!, update_eligibility_vector!, x, update_feature_vector!, Returns(zero(T)), Returns(nothing), max_episodes; α_θ = α, kwargs...)
return (episode_rewards = out.episode_rewards, episode_steps = out.episode_steps, policy_function = out.policy_function, policy_sample_action = out.policy_sample_action, parameters = out.policy_parameters)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$5ffc271f-c73f-494a-9727-8d7516af2191cell_id$5ffc271f-c73f-494a-9727-8d7516af2191code٣@bind cartpole_continuing_fcann_study_params create_actor_critic_continuing_params_UI(;λ_θ= 0.8f0, λ_w = 0.15f0, α_r̄ = 0.05f0, log2α_θ = -6, log2α_w = -5)metadatashow_logsèdisabled®skip_as_script«code_folded$c5a2879c-e89b-47f7-bbd6-48200d7e89e3cell_id$c5a2879c-e89b-47f7-bbd6-48200d7e89e3code^actor_critic_binary_episodic_squashed_gaussian_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, args...; kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = actor_critic_binary_episodic_squashed_gaussian_parameter_study(mdp, one(T), get_active_features, num_features, args...; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$537270ba-122b-4f2b-880b-31d086766295cell_id$537270ba-122b-4f2b-880b-31d086766295codebegin
#the following struct represents a problem for which both the state and action space can take arbitrary values
struct ContinuousMDP{T<:Real, S, A, P<:AbstractContinuousTransition{T, S, A, F} where F<:Function, StateInit<:Function, IsTerm<:Function, ValidAction <: Function} <: AbstractMDP{T, S, A, P, StateInit}
ptf::P
initialize_state::StateInit #function which provides an initial state index
isterm::IsTerm #function that returns true if a state is terminal and false otherwise
is_valid_action::ValidAction #is_valid_action(s, a) returns true if the action a is valid to take from state. by default every action is assumed to be available
ContinuousMDP(ptf::P, initialize_state::F1, isterm::F2, is_valid_action::F3) where {T<:Real, S, A, F<:Function, P<:AbstractContinuousTransition{T, S, A, F}, F1<:Function, F2<:Function, F3<:Function} = new{T, S, A, P, F1, F2, F3}(ptf, initialize_state, isterm, is_valid_action)
end
ContinuousMDP(ptf::AbstractContinuousTransition{T, S, A, F}, initialize_state::StateInit; isterm::Function = Returns(false), is_valid_action::Function = Returns(true)) where {T<:Real, S, A, F<:Function, StateInit<:Function} = ContinuousMDP(ptf, initialize_state, isterm, is_valid_action)
function ContinuousMDP(step::Function, initialize_state::Function, a::A; kwargs...) where A
s0 = initialize_state()
ptf = ContinuousMDPTransitionSampler(step, s0, a)
ContinuousMDP(ptf, initialize_state; kwargs...)
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$dc2efc6c-8da8-425b-aa5f-290949109565cell_id$dc2efc6c-8da8-425b-aa5f-290949109565codeGplot_mountaincar_policy_values(mountaincar_test_train.policy_and_value)metadatashow_logsèdisabled®skip_as_script«code_folded$a019925a-460a-410e-a54b-50a4cfe0e90ecell_id$a019925a-460a-410e-a54b-50a4cfe0e90ecodeplot(scatter(x = 1 .- LinRange(0.01, 0.99, 100), y = -[get_corridor_episode_stats(p) for p in 1 .- LinRange(0.01, 0.99, 100)]), Layout(xaxis_title = "probability of right action", yaxis_title = "sample mean value of starting state", width = 800, yaxis_range = [-60, -10]))metadatashow_logsèdisabled®skip_as_script«code_folded$f92bb265-4b19-4f0e-a698-d7547bb6dd41cell_id$f92bb265-4b19-4f0e-a698-d7547bb6dd41code٧mutable struct BinaryFeatureVector{I <: Integer}
active_features::Vector{I}
num_features::I
function BinaryFeatureVector()
new{Int64}(Vector{Int64}(), 0)
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$ac9c8845-284d-4c21-b05d-d930f86598a3cell_id$ac9c8845-284d-4c21-b05d-d930f86598a3codeb@bind run_mountaincar_binary_episodic_countinuous_param_study CounterButton("Run Parameter Study")metadatashow_logsèdisabled®skip_as_script«code_folded$192cc1cf-9ea1-492d-baa7-f2e197abecd4cell_id$192cc1cf-9ea1-492d-baa7-f2e197abecd4codeV@bind run_mountaincar_binary_episodic_param_study CounterButton("Run Parameter Study")metadatashow_logsèdisabled®skip_as_script«code_folded$a4eec4d3-5a75-4b52-ab9c-9d9e83d5547dcell_id$a4eec4d3-5a75-4b52-ab9c-9d9e83d5547dcode6@bind ep_step Slider(1:length(ep[1]), show_value=true)metadatashow_logsèdisabled®skip_as_script«code_folded$c8b47eac-2d45-419a-bec6-2ae0cdc59393cell_id$c8b47eac-2d45-419a-bec6-2ae0cdc59393codehbegin
#represents a transition where the state must be referenced directly instead of through a tabular index
abstract type AbstractContinuousTransition{T<:Real, S, A, F<:Function} <: AbstractTransition{T, 2} end
struct ContinuousMDPTransitionSampler{T <: Real, S, A, F <: Function} <: AbstractContinuousTransition{T, S, A, F}
step::F
function ContinuousMDPTransitionSampler(step::F, s::S, a::A) where {F<:Function, S, A}
(r, s′) = step(s, a)
@assert promote_type(S, typeof(s′)) != Any "There is no common type between the provided state $s and the transition state $s′"
new{typeof(r), promote_type(S, typeof(s′)), A, F}(step)
end
end
#when used as a functor just apply the step function to the state action pair indices
(ptf::ContinuousMDPTransitionSampler{T, S, A, F})(s::S, a::A) where {T<:Real, S, A, F<:Function} = ptf.step(s, a)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$36a6e43f-6bcf-4c27-bfbb-047760e77adacell_id$36a6e43f-6bcf-4c27-bfbb-047760e77adacode md"""
# Chapter 13 Policy Gradient Methods Introduction
Instead of selection actions based on *action-value estimates* we learn a *parameterized policy* with parameters $\boldsymbol{θ}$. $\pi(a|s, \boldsymbol{\theta}) = \text{Pr}\{A_t=a|S_t=s, \boldsymbol{\theta}_t=\boldsymbol{\theta\}}$ denotes the probability that action *a* is taken at time *t* given that the environment is in state *s* at time *t* with parameter $\boldsymbol{θ}$.
We consider methods that improve the policy parameter using the gradient of some scalar performance measure $J(\boldsymbol{\theta})$ with respect to the policy parameters. We follow gradient ascent since we are trying to maximize this value and methods that use this approach are called *policy gradient methods*. Methods that learn approximations to both policy and value functions are often called *actor-critic methods*, where 'actor' is a reference to the learned policy, and 'critic' refers to the learned value function, usually a state-value function.
## 13.1 Policy Approximation and its Advantages
"""metadatashow_logsèdisabled®skip_as_script«code_folded$436c52d2-280b-4ca4-9360-d6587b8254c7cell_id$436c52d2-280b-4ca4-9360-d6587b8254c7code~md"""
In order to test this algorithm we need to use a continuing task which is lacking a terminal state. We could simply modify the corridor MDP to be a continuing task by altering the reward structure so a reward of 1 is received upon moving to the right from state 3 after which the state is reset to 1. Se below for a version of this MDP updated to be a continuing problem.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$e96d592d-1e54-486d-8ad9-b857f85476e8cell_id$e96d592d-1e54-486d-8ad9-b857f85476e8code.actor_critic_linear_parameter_study(mdp::StateMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, params::@NamedTuple{λ_θ::T, λ_w::T, α_r̄::T, α_θ_min::Int64, α_w_min::Int64}, num_θ::Integer, num_w::Integer, max_steps::Integer; kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = actor_critic_linear_parameter_study(mdp, get_active_features, num_features, params.λ_θ, params.λ_w, params.α_r̄, 2f0 .^(params.α_θ_min:params.α_θ_min+num_θ-1), 2f0 .^(params.α_w_min:params.α_w_min+num_w-1), max_steps; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6ccell_id$5583ae6d-f6fa-47ba-aab4-cb6a4f32cb6ccodeLcorridor_parameter_studies(2f0 .^ (-15:-8), 2f0 .^ (-35:5:-15); nruns = 100)metadatashow_logsèdisabled®skip_as_script«code_folded$4da20fd7-b897-4f26-bf2a-f08d66ddf90fcell_id$4da20fd7-b897-4f26-bf2a-f08d66ddf90fcode b#version of reinforce for general function approximation
function actor_critic_with_eligibility_traces!(policy_params::P1, ∇lnπ, value_params::P2, ∇v̂, mdp::ContinuousMDP{T, S, A, PTF, F1, F2, F3}, λ_θ::T, λ_w::T, update_action_distribution!::Function, action_dist_params::Vector{T}, action_sampler::Function, update_eligibility_vector!::Function, x, update_feature_vector!::Function, value_function::Function, update_value_gradient!::Function, max_steps::Integer; α_w::T = one(T)/10, α_θ::T = one(T)/10, α_r̄ = one(T)/10, z_θ::P1 = deepcopy(policy_params), z_w::P2 = deepcopy(value_params), save_step_rewards = false) where {P1, P2, T<:Real, S, A, PTF, F1, F2, F3}
step_rewards = Vector{T}()
#initialize variables
step = 1
rtot = zero(T)
r̄ = zero(T)
c = one(T)
zero_params!(z_θ)
zero_params!(z_w)
s = mdp.initialize_state()
update_feature_vector!(x, s)
while step <= max_steps
update_value_gradient!(∇v̂, x, value_params)
v̂ = value_function(x, value_params)
update_action_distribution!(action_dist_params, x, policy_params)
a = action_sampler(action_dist_params)
if bad_continuous_action(a)
@info "terminating after $step steps due to invalid continuous action $a taken in state $s with action distribution parameters $action_dist_params"
push!(episode_steps, max_steps)
push!(episode_rewards, typemin(T))
break
end
update_eligibility_vector!(∇lnπ, action_dist_params, x, a, policy_params)
(r, s′) = mdp.ptf(s, a)
rtot += r
save_step_rewards && push!(step_rewards, r)
step += 1
mdp.isterm(s′) && error("$s′ is a terminal state and this method only applies to continuing tasks")
update_feature_vector!(x, s′)
v̂′ = value_function(x, value_params)
δ = r - r̄ + v̂′ - v̂
r̄ += α_r̄*δ
update_traces_with_gradient!(γ*λ_w, z_w, ∇v̂)
update_traces_with_gradient!(γ*λ_θ, z_θ, c, ∇lnπ)
update_params_with_gradient!(value_params, α_w*δ, z_w)
update_params_with_gradient!(policy_params, α_θ*c*δ, z_θ)
s = s′
end
function_outputs = form_state_and_policy_function_outputs(update_feature_vector!, update_action_distribution!, action_dist_params, action_sampler, value_function, x, policy_params, value_params)
return (;step_rewards = step_rewards, total_reward = rtot, policy_parameters = policy_params, value_parameters = value_params, function_outputs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$11ea640c-3981-404d-87c6-4d3d0708a2b8cell_id$11ea640c-3981-404d-87c6-4d3d0708a2b8codefunction actor_critic_linear_episodic_parameter_study(mdp::StateMDP{T, S, A, P, F1, F2, F3}, update_feature_vector!::Function, num_features::Integer, λ_θ::T, λ_w::T, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, max_episodes::Integer; nruns = 100, max_steps::Integer = 10_000, seed = rand(UInt64), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
Random.seed!(seed)
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> actor_critic_with_eligibility_traces_linear_features(mdp, λ_θ, λ_w, update_feature_vector!, num_features, max_episodes, max_steps; α_θ = α_θ, α_w = α_w, kwargs...) |> x -> isempty(x.episode_rewards) ? -T(Inf) : sum(x.episode_rewards) / length(x.episode_rewards)) |> foldxt(+) |> x -> x / nruns
end
traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Episode in the First $max_episodes Episodes Averaged Over $nruns Runs", xaxis_type = "log2", title = "Linear Feature Encoding with $num_features Features, λ_θ = $λ_θ, λ_w = $λ_w"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$281360af-46bf-4c73-bf11-3cb1153ad3e2cell_id$281360af-46bf-4c73-bf11-3cb1153ad3e2codeUcartpole_tilecoding_reinforce_parameter_study(2f0 .^ (-12:-7), 2f0 .^ (-13:-10), 100)metadatashow_logsèdisabledîskip_as_script«code_folded$9ae58dd6-3cde-4943-9ac1-bd9d4f7d690ccell_id$9ae58dd6-3cde-4943-9ac1-bd9d4f7d690ccode%begin
function update_squashed_gaussian_eligibility_vector!(∇lnπ::BinarySquashedGaussianEligibilityVector{T, T, T, B}, dist_params::Vector{T}, x::B, action::T, policy_params::Matrix{T}) where {T<:Real, B<:BinaryFeatureVector}
∇lnπ.binary_features = x
∇lnπ.a = action
∇lnπ.μ = first(dist_params)
∇lnπ.σ = exp(last(dist_params))
return ∇lnπ
end
function update_squashed_gaussian_eligibility_vector!(∇lnπ::BinarySquashedGaussianEligibilityVector{T, NTuple{N, T}, Vector{T}, B}, dist_params::Vector{T}, x::B, action::NTuple{N, T}, policy_params::Matrix{T}) where {T<:Real, N, B<:BinaryFeatureVector}
∇lnπ.binary_features = x
∇lnπ.a = action
for i in 1:N
∇lnπ.μ[k] = dist_params[k]
∇lnπ.σ[k] = exp(dist_params[k+N])
end
return ∇lnπ
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$da3cb392-78f2-48b2-b0dc-5f016664798ccell_id$da3cb392-78f2-48b2-b0dc-5f016664798ccodeXshow_mountaincar_trajectory(mountaincar_continuing_tile_test.policy_sample_action, 1000)metadatashow_logsèdisabled®skip_as_script«code_folded$dca2f8e2-76af-4679-bf81-3824c15fc76dcell_id$dca2f8e2-76af-4679-bf81-3824c15fc76dcode
const reinforce_test3 = actor_critic_with_eligibility_traces_binary_features(cartpole_setup.mdps.episodic.discrete, 0.85f0, 0.5f0, cartpole_setup.get_active_features, cartpole_setup.num_features, typemax(Int64), 100_000; α_θ = 2f0 ^-6, α_w = 2f0 ^-4, γ = 0.99f0)metadatashow_logsèdisabled®skip_as_script«code_folded$8019bec9-1228-407b-9199-2fe29f26a981cell_id$8019bec9-1228-407b-9199-2fe29f26a981code
md"""
> ### *Exercise 13.1*
> Use your knowledge of the gridworld and its dynamics to determine an *exact* symbolic expression for the optimal probability of selecting the right action in Example 13.1
Example 13.1 is a gridworld with 3 non-terminal states and a terminal state at the far right. The reward is -1 per step. States 1 and 3 have actions left/right that move in the expected directions but state 2 reverses the directions. We use a performance measure $J(\mathbf{\theta}) = v_{\pi_\theta}(S)$. Given our feature representations of $\mathbf{x}(s, \text{right}) = [1, 0]^{\top}$ and $\mathbf{x}(s, \text{left}) = [0, 1]^{\top}$, we can only learn policies that are stochastic in terms of left/right action selection but do not vary between states. Also observe that due to probability constraints $p_{\text{right}} = 1 - p_{\text{left}}$. For simplicity, we will use the notation $p \doteq p_{\text{left}}$ and the following for the three state values: $v1, v2, v3$.
$\begin{flalign}
v_1 &= p \times v_1 + (1-p) \times v_2 - 1 \tag{1} \\
v_1 (1-p) &= v_2 (1-p) - 1 \\
v_1 &= v_2 - \frac{1}{1-p} \tag{1′}\\
v_2 &= p \times v_3 + (1-p) \times v_1 - 1 \tag{2} \\
v_3 &= p \times v_2 - 1 \tag{3}\\
v_2 &= p \times [p\times v_2 - 1] +(1-p) \times v_1 - 1 \tag{substituting 3 into 2} \\
v_2(1 - p^2) &= -p +(1-p) \times v_1 - 1 \\
v_2 &= \frac{(1-p) v_1 - (1+p)}{(1+p)(1-p)} \tag{collecting terms} \\
&= \frac{(1-p) v_2 - 1 - (1+p)}{(1+p)(1-p)} \tag{using 1′} \\
&= \frac{v_2}{1+p} - \frac{2 + p}{(1+p)(1-p)} \\
v_2 \left [1 - \frac{1}{1+p} \right ] &= - \frac{2 + p}{(1+p)(1-p)} \\
v_2 \frac{1+p-1}{1+p} &= - \frac{2 + p}{(1+p)(1-p)} \\
v_2 &= - \frac{2 + p}{(1-p)p} \\
v_1 &= - \frac{2 + p}{(1-p)p} - \frac{1}{1-p} \\
&= \frac{-2 - p - p}{(1-p)p} \\
&= -\frac{2 + 2p}{(1-p)p} \\
v_3 &= -\frac{2 + p}{1-p} - 1\\
&= \frac{-2 - p - 1 + p}{1-p}\\
&= -\frac{3}{1-p}\\
\end{flalign}$
To summarize all the state values:
$\begin{flalign}
v_1 &= -\frac{2 + 2p}{(1-p)p} \\
v_2 &= - \frac{2 + p}{(1-p)p} \\
v_3 &= -\frac{3}{1-p}
\end{flalign}$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$fd964539-2baf-4ff1-b286-5a0bb1b222c4cell_id$fd964539-2baf-4ff1-b286-5a0bb1b222c4codemd"""
The beta distribution has two parameters like the normal distribution but is only defined from 0 to 1. The two parameters $\alpha$ and $\beta$ are positive real numbers and control the shape of the distribution. The density function is given below:
$f(x; \alpha, \beta) = \frac{x^{\alpha-1} (1-x)^{\beta - 1}}{\text{B}(\alpha, \beta)}$ where $\text{B}(\alpha, \beta) = \frac{\Gamma(\alpha) \Gamma(\beta)}{\Gamma(\alpha + \beta)}$ and $\Gamma(z) = \int_0^\infty t^{z-1}e^{-t} \text{d} t$
We saw earlier from the treatment of the gaussian distribution that we need to find the gradient of a function of each distribution parameter with respect to the parameters of the function approximation. Luckily, the maximum likelihood estimator already computes the gradient we are interested in for this distribution. Note that the likelihood function for a single sample of the random variable $x$ which follows the beta distribution is given by $\mathcal{L}(\alpha, \beta \vert X) = \ln(f(X_i; \alpha, \beta))$ and the partial derivative of this function with respect to each parameter $\alpha$ and $\beta$ is given by:
$\frac{\partial \mathcal{L}(\alpha, \beta, \vert X)}{\partial \alpha} = \ln X - \frac{\partial \ln \text{B}(\alpha, \beta)}{\partial \alpha}$
$\frac{\partial \mathcal{L}(\alpha, \beta, \vert X)}{\partial \beta} = \ln (1-X) - \frac{\partial \ln \text{B}(\alpha, \beta)}{\partial \beta}$
where $\frac{\partial \ln \text{B}(\alpha, \beta)}{\partial \alpha} = -\psi(\alpha + \beta) + \psi(\alpha)$ and $\frac{\partial \ln \text{B}(\alpha, \beta)}{\partial \beta} = -\psi(\alpha + \beta) + \psi(\beta)$ and $\phi(\alpha)$ is the digamma function which is just the derivative of the logarithm of the gamma function.
Since both $\alpha$ and $\beta$ must be greater than zero, we can use for an estimate for each one the exponential function applied to a dot product of the parameter vector with the feature vector: $\alpha(s, \boldsymbol{\theta}) \doteq \exp \left (\boldsymbol{\theta}_\alpha^\top \mathbf{x}(s) \right )$ and $\beta(s, \boldsymbol{\theta}) \doteq \exp \left (\boldsymbol{\theta}_\beta^\top \mathbf{x}(s) \right )$.
The eligibility vector for this distribution is then:
$\nabla \ln f(a \vert \alpha(s, \boldsymbol{\theta}_\alpha), \beta(s, \boldsymbol{\theta}_\beta))$
where $\alpha$ is a function of its parameters and $\beta$ is a function of the other parameter vector. The gradient components corresponding to each vector is only a function of a partial derivative of the distribution with respect to $\alpha$ and $\beta$. That is, since $\frac{\partial \alpha}{\partial \theta_{\beta_i}} = 0 \forall i$ and vice versa, then we can treat each part of the gradient separately.
$\begin{flalign}
\nabla_{\boldsymbol{\theta}_\alpha} \ln f(a \vert \alpha, \beta) &= \frac{\partial f(a \vert \alpha, \beta)}{\partial \alpha} \nabla_{\boldsymbol{\theta}_\alpha}\alpha \\
&= \left ( \ln a - \psi(\alpha + \beta) + \psi(\alpha) \right ) \nabla_{\boldsymbol{\theta}_\alpha} \alpha \\
&= \left ( \ln a - \psi(\alpha + \beta) + \psi(\alpha) \right ) \nabla_{\boldsymbol{\theta}_\alpha} \exp \left ( \boldsymbol{\theta}_\alpha^\top \mathbf{x}(s) \right ) \\
&= \left ( \ln a - \psi(\alpha + \beta) + \psi(\alpha) \right ) \alpha \mathbf{x}(s)\\
\end{flalign}$
$\begin{flalign}
\nabla_{\boldsymbol{\theta}_\beta} \ln f(a \vert \alpha, \beta) &= \frac{\partial f(a \vert \alpha, \beta)}{\partial \beta} \nabla_{\boldsymbol{\theta}_\beta}\beta \\
&= \left ( \ln a - \psi(\alpha + \beta) + \psi(\beta) \right ) \nabla_{\boldsymbol{\theta}_\beta} \beta \\
&= \left ( \ln a - \psi(\alpha + \beta) + \psi(\beta) \right ) \nabla_{\boldsymbol{\theta}_\beta} \exp \left ( \boldsymbol{\theta}_\beta^\top \mathbf{x}(s) \right ) \\
&= \left ( \ln a - \psi(\alpha + \beta) + \psi(\beta) \right ) \beta \mathbf{x}(s)\\
\end{flalign}$
"""metadatashow_logsèdisabled®skip_as_script«code_folded$5720e942-d3f8-4329-83a8-8bcedf078b6acell_id$5720e942-d3f8-4329-83a8-8bcedf078b6acodeٔreinforce_monte_carlo_control_linear_features(corridor_mdp, update_corridor_features!, 1, 1_000; α = 2f0^-14, max_steps = 1_000).policy_function(1)metadatashow_logsèdisabled®skip_as_script«code_folded$62e677ac-2070-4f6b-9df2-90849d89fa9fcell_id$62e677ac-2070-4f6b-9df2-90849d89fa9fcodeQconst corridor_terminal_probabilities = 1 .- sum(corridor_state_counts, dims = 2)metadatashow_logsèdisabled®skip_as_script«code_folded$11b9beea-b0cd-45eb-84c6-151728894df0cell_id$11b9beea-b0cd-45eb-84c6-151728894df0code function form_state_and_policy_function_outputs(update_feature_vector!::Function, update_action_distribution!::Function, action_dist_params::Vector{T}, action_sampler::Function, value_function::Function, feature_vector, policy_params, value_params) where T<:Real
π! = form_state_continuous_policy_function(update_feature_vector!, update_action_distribution!)
π(s) = π!(feature_vector, action_dist_params, s, policy_params)
π_sample(s) = action_sampler(π(s))
v! = form_state_value_function(update_feature_vector!, value_function)
estimate_state_value(s; x = deepcopy(feature_vector)) = v!(x, s, value_params)
function policy_and_value(s; x = deepcopy(feature_vector), action_dist_params = copy(action_dist_params))
π!(x, action_dist_params, s, policy_params)
v̂ = value_function(x, value_params)
return (action_distribution_parameters = action_dist_params, state_value_estimate = v̂)
end
(policy_function = π, policy_sample_action = π_sample, estimate_state_value = estimate_state_value, policy_and_value = policy_and_value)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290cell_id$961f02ee-a6e5-4fe8-b1d2-eb3f8824d290codexfunction reinforce_monte_carlo_control_binary_features(mdp::StateMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, max_episodes::Integer; params::Matrix{T} = zeros(T, num_features, length(mdp.actions)), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
setup = setup_binary_policy_arguments(mdp, get_active_features, num_features)
reinforce_monte_carlo_control!(params, setup.eligibility_vector, mdp, update_binary_action_preferences!, update_binary_eligibility_vector!, setup.feature_vector, setup.update_feature_vector!, max_episodes; action_preferences = setup.action_preferences, kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$55ba8725-0ddf-4196-a41d-3f3c490a8d84cell_id$55ba8725-0ddf-4196-a41d-3f3c490a8d84codeofunction actor_critic_binary_episodic_gaussian_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, λ_θ::T, λ_w::T, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, max_episodes::Integer; nruns::Integer = 100, max_steps::Integer = 10_000, seed = rand(UInt64), init_policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), init_value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
Random.seed!(seed)
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> actor_critic_with_eligibility_traces_binary_features_gaussian_actions(mdp, λ_θ, λ_w, get_active_features, num_features, max_episodes, max_steps; α_θ = α_θ, α_w = α_w, policy_params = copy(init_policy_params), value_params = copy(init_value_params), kwargs...) |> x -> isempty(x.episode_rewards) ? -T(Inf) : mean(x.episode_rewards)) |> foldxt(+) |> x -> x / nruns
end
traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Episode in the First $max_episodes Episodes Averaged Over $nruns Runs", xaxis_type = "log", title = "Binary Feature Encoding with $num_features Features, λ_θ = $λ_θ, λ_w = $λ_w"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a540814a-57a1-4b98-9443-59e401425444cell_id$a540814a-57a1-4b98-9443-59e401425444codefunction binary_value_function(binary_features::BinaryFeatureVector, params::Vector{T})::T where T<:Real
v = zero(T)
@inbounds @simd for i in 1:binary_features.num_features
j = binary_features.active_features[i]
v += params[j]
end
return v
endmetadatashow_logsèdisabled®skip_as_script«code_folded$1b102220-6d78-480d-a77f-0e57bad23dcacell_id$1b102220-6d78-480d-a77f-0e57bad23dcacode!cartpole_binary_continuing_parameter_study(args...; kwargs...) = actor_critic_linear_parameter_study(cartpole_continuing_mdp, s -> cartpole_tilecoding_setup.get_active_features((s.x, s.θ, s.ẋ, s.θ̇)), cartpole_tilecoding_setup.num_features, binary_features = true, args...; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$4d4ae57b-afc3-44f9-b6fc-892f59f82921cell_id$4d4ae57b-afc3-44f9-b6fc-892f59f82921code #version of reinforce for general function approximation
function one_step_actor_critic!(policy_params, ∇lnπ, value_params, ∇v̂, mdp::StateMDP{T, S, A, PTF, F1, F2, F3}, update_action_preferences!::Function, update_eligibility_vector!::Function, x, update_feature_vector!::Function, value_function::Function, update_value_gradient!::Function, max_episodes::Integer, max_steps::Integer; α_w::T = one(T)/10, α_θ::T = one(T)/10, γ::T = one(T), action_preferences = zeros(T, length(mdp.actions)), save_episode_steps = false) where {T<:Real, S, A, PTF, F1, F2, F3}
step_rewards = Vector{T}()
episode_steps = Vector{Int64}()
episode_rewards = Vector{T}()
#initialize variables
ep = 1
step = 1
rtot = zero(T)
c = one(T)
s = mdp.initialize_state()
update_feature_vector!(x, s)
# @info "initial value params: $value_params"
while (ep <= max_episodes) && (step <= max_steps)
update_value_gradient!(∇v̂, x, value_params)
v̂ = value_function(x, value_params)
update_action_preferences!(action_preferences, x, policy_params)
soft_max!(action_preferences)
i_a = sample_action(action_preferences)
update_eligibility_vector!(∇lnπ, action_preferences, x, i_a, policy_params)
(r, s′) = mdp.ptf(s, i_a)
rtot += r
save_episode_steps && push!(step_rewards, r)
step += 1
if mdp.isterm(s′)
push!(episode_steps, step)
push!(episode_rewards, rtot)
v̂′ = zero(T)
ep += 1
rtot = zero(T)
c = one(T)
s = mdp.initialize_state()
update_feature_vector!(x, s)
else
update_feature_vector!(x, s′)
v̂′ = value_function(x, value_params)
s = s′
c *= γ
end
δ = r + γ*v̂′ - v̂
# @info "About to update value params with gradient $∇v̂ and constant $(α_w * δ)"
update_params_with_gradient!(value_params, α_w*δ, ∇v̂)
# @info "About to update policy params with eligibility vector $∇lnπ and constant $(α_θ*c*δ)"
update_params_with_gradient!(policy_params, α_θ*c*δ, ∇lnπ)
# @info "policy params after $step updates: $policy_params"
# @info "value params after $step updates: $value_params"
end
function_outputs = form_state_and_policy_function_outputs(update_feature_vector!, update_action_preferences!, value_function, x, action_preferences, policy_params, value_params)
return (;step_rewards = step_rewards, episode_steps = episode_steps, episode_rewards = episode_rewards, policy_parameters = policy_params, value_parameters = value_params, function_outputs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$61949faa-8174-4b7b-8fbc-01d5f850b419cell_id$61949faa-8174-4b7b-8fbc-01d5f850b419code:function actor_critic_binary_continuing_gaussian_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, get_active_features::Function, num_features::Integer, λ_θ::T, λ_w::T, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, α_r̄::T, max_steps::Integer; nruns::Integer = 100, seed = rand(UInt64), init_policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), init_value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
Random.seed!(seed)
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> actor_critic_with_eligibility_traces_binary_features_gaussian_actions(mdp, λ_θ, λ_w, get_active_features, num_features, max_steps; α_θ = α_θ, α_w = α_w, α_r̄ = α_r̄, policy_params = copy(init_policy_params), value_params = copy(init_value_params), kwargs...).total_reward) |> foldxt(+) |> x -> x / nruns / max_steps
end
traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Step in the First $max_steps Steps Averaged Over $nruns Runs", xaxis_type = "log", title = "Binary Feature Encoding with $num_features Features, λ_θ = $λ_θ, λ_w = $λ_w, α_r̄ = $α_r̄"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$5b15f5c9-80bf-47f0-898a-f8dead5b927ccell_id$5b15f5c9-80bf-47f0-898a-f8dead5b927ccodemd"""
### *Continuing Case Actor-Critic Implementation*
Note that this function has the same name as the episodic version. The only difference other than keyword arguments is that the `max_episodes` argument is missing. Since we already defined the versions of the algorithm for linear and non-linear cases in a generic manner, we only need to define the core version of this algorithm and the other functions will dispatch to it if they are called without the `max_episodes` argument.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$266d2234-26c8-43f1-9e75-49440a230ed6cell_id$266d2234-26c8-43f1-9e75-49440a230ed6code #version of reinforce for general function approximation
function actor_critic_with_eligibility_traces!(policy_params::P1, ∇lnπ, value_params::P2, ∇v̂, mdp::StateMDP{T, S, A, PTF, F1, F2, F3}, λ_θ::T, λ_w::T, update_action_preferences!::Function, update_eligibility_vector!::Function, x, update_feature_vector!::Function, value_function::Function, update_value_gradient!::Function, max_episodes::Integer, max_steps::Integer; α_w::T = one(T)/10, α_θ::T = one(T)/10, γ::T = one(T), action_preferences = zeros(T, length(mdp.actions)), z_θ::P1 = deepcopy(policy_params), z_w::P2 = deepcopy(value_params), save_step_rewards = false) where {P1, P2, T<:Real, S, A, PTF, F1, F2, F3}
step_rewards = Vector{T}()
episode_steps = Vector{Int64}()
episode_rewards = Vector{T}()
#initialize variables
ep = 1
step = 1
rtot = zero(T)
c = one(T)
zero_params!(z_θ)
zero_params!(z_w)
s = mdp.initialize_state()
update_feature_vector!(x, s)
while (ep <= max_episodes) && (step <= max_steps)
update_value_gradient!(∇v̂, x, value_params)
v̂ = value_function(x, value_params)
update_action_preferences!(action_preferences, x, policy_params)
soft_max!(action_preferences)
i_a = sample_action(action_preferences)
update_eligibility_vector!(∇lnπ, action_preferences, x, i_a, policy_params)
(r, s′) = mdp.ptf(s, i_a)
rtot += r
save_step_rewards && push!(step_rewards, r)
step += 1
if mdp.isterm(s′)
push!(episode_steps, step)
push!(episode_rewards, rtot)
v̂′ = zero(T)
rtot = zero(T)
zero_params!(z_θ)
zero_params!(z_w)
ep += 1
c = one(T)
s = mdp.initialize_state()
update_feature_vector!(x, s)
else
update_feature_vector!(x, s′)
v̂′ = value_function(x, value_params)
s = s′
c *= γ
end
δ = r + γ*v̂′ - v̂
update_traces_with_gradient!(γ*λ_w, z_w, ∇v̂)
update_traces_with_gradient!(γ*λ_θ, z_θ, c, ∇lnπ)
update_params_with_gradient!(value_params, α_w*δ, z_w)
update_params_with_gradient!(policy_params, α_θ*c*δ, z_θ)
end
function_outputs = form_state_and_policy_function_outputs(update_feature_vector!, update_action_preferences!, value_function, x, action_preferences, policy_params, value_params)
return (;step_rewards = step_rewards, episode_steps = episode_steps, episode_rewards = episode_rewards, policy_parameters = policy_params, value_parameters = value_params, function_outputs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$aa69e4ea-91e0-496a-a7be-529e67f4dbeccell_id$aa69e4ea-91e0-496a-a7be-529e67f4dbeccodeٴreinforce_with_baseline_monte_carlo_control_fcann(corridor_mdp, 1, [10, 10], update_corridor_features!, 100; α_θ = 2f0^-14, α_w = 2f0^-14, max_steps = 10_000).policy_function(1)metadatashow_logsèdisabled®skip_as_script«code_folded$10ee7709-0816-48d2-abe0-9be3dd04700fcell_id$10ee7709-0816-48d2-abe0-9be3dd04700fcodeLplot_continuing_step_rewards(mountaincar_continuing_fcann_test.step_rewards)metadatashow_logsèdisabled®skip_as_script«code_folded$7d94922e-dc9f-4953-b539-24aaa2c85b12cell_id$7d94922e-dc9f-4953-b539-24aaa2c85b12code٘@bind continuing_study_params create_actor_critic_continuing_params_UI(;λ_θ = 0.75f0, λ_w = 0.25f0, log2α_θ = -6, log2α_w = -10, α_r̄ = 0.005f0)metadatashow_logsèdisabled®skip_as_script«code_folded$df7f84e8-b42a-4001-9dbf-6bc3ced94207cell_id$df7f84e8-b42a-4001-9dbf-6bc3ced94207codeَusing PlutoDevMacros, Random, Statistics, LinearAlgebra, Transducers, Base.Threads, Random, Distributions, Statistics, StatsBase, StaticArraysmetadatashow_logsèdisabled®skip_as_script«code_folded$352d2952-cb83-47d3-9078-2b2ef9927443cell_id$352d2952-cb83-47d3-9078-2b2ef9927443code#create a cart pole MDP environment
function create_cartpole_functions(;
m::T = 1f0, #mass at the end of the pole in kg
m_c::T = 10f0, #mass of the cart in kg
l::T = 1f0, #length of the pole in meters
g::T = 9.8f0, #gravitational constant in meters per second squared
h::T = 4f-2, #step size parameter of simulation in seconds
k::T = 1f0, #inertial constant of pendulum,
m_f::T = 0f0, #friction of the rotating pole
μ_c::T = 0f0, #friction of the cart wheels against the track
fmax::T = 300f0, #force applied by throttle
x_max::T = 50f0, #maximum horizontal position
θ_max::T = deg2rad(70f0), #maximum pole angle
ẋ_max::T = 50f0,
θ̇_max::T = 10f0,
init_x::Function = () -> 0f0, #initialize each of the 4 state variables
init_θ::Function = () -> Float32(rand([-0.02f0, 0.02f0])),
init_ẋ::Function = () -> 0f0,
init_θ̇::Function = () -> 0f0) where T<:Real
#the action space is full throttle forward or backwards or idle in the discrete case
actions = [-fmax, zero(T), fmax]
#create a vehicle to use in simulation steps
vehicle = CartPoleVehicle(m, m_c, l, k, m_f, μ_c)
initialize_state(;t = 0f0) = CartPoleState(init_x(), init_θ(), init_ẋ(), init_θ̇(), t)
function failure(s::CartPoleState)
(abs(s.x) > x_max) || (abs(s.θ) > θ_max) || (abs(s.ẋ) > ẋ_max) || (abs(s.θ̇) > θ̇_max)
end
step(s::CartPoleState{T}, f::T) = cartpole_runge_kutta_step(vehicle, s, g, clamp(f, -fmax, fmax), h)
min_vals = (-x_max, -θ_max, -ẋ_max, -θ̇_max)
max_vals = (x_max, θ_max, ẋ_max, θ̇_max)
(step = step, failure = failure, initialize_state = initialize_state, discrete_actions = actions, min_vals = min_vals, max_vals = max_vals, h = h)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$0964133c-3a5b-433b-a8c4-a97813c37583cell_id$0964133c-3a5b-433b-a8c4-a97813c37583code,function plot_continuing_step_rewards(r::Vector{T}; npoints = 1000) where T<:Real
rsum = cumsum(r)
ravg = rsum ./ (1:length(r))
inds = round.(Int64, LinRange(1, length(r), npoints))
plot(scatter(x = inds, y = ravg[inds]), Layout(xaxis_title = "Training Step", yaxis_title = "Reward Average"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$349631b2-4686-49a9-9f3a-1e4ad588b568cell_id$349631b2-4686-49a9-9f3a-1e4ad588b568code\const mountaincar_continuous_mdp2 = create_continuous_action_mountaincar(;slipforce = 100f0)metadatashow_logsèdisabled®skip_as_script«code_folded$8544eddb-2095-4a3c-82e0-920123a88e6dcell_id$8544eddb-2095-4a3c-82e0-920123a88e6dcodemd"""
### Test REINFORCE With and Without Baseline
The following function calls execute the REINFORCE algorithm on Example 13.1. The output displayed is the policy function acting on the single state representation for the problem. The two values represent the probability of taking the left and right action respectively. If converged properly, the right action probability should be higher, approaching a value of about 60%.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$31f7e903-30b6-4193-9174-88093e004de4cell_id$31f7e903-30b6-4193-9174-88093e004de4codemd"""
In policy gradient methods, the policy can be parameterized in any way, as long as $\pi(a \vert s, \boldsymbol{\theta})$ is differentiable with respect to its parameters, that is, as long as $\nabla \pi(a \vert s, \boldsymbol{\theta})$ exists and is finite for all $s \in \mathcal{S}, a \in \mathcal{A}(s)$, and $\boldsymbol{\theta} \in \mathbb{R}^{d^\prime}$ where $d^\prime$ is the number of parameters.
If the action space is discrete and not too large then we can have numerical preferences for each state/action pair parameterized by $\boldsymbol{\theta}$. $h(s, a, \boldsymbol{\theta})$ and the corresponding policy can be to select actions according to the probability distribution generated by the soft-max. $\pi(a|s, \boldsymbol{\theta}) \doteq \frac{\exp{h(s, a, \boldsymbol{\theta})}}{\sum_b \exp{h(s, b, \boldsymbol{\theta})}}$. One advantage of using the soft-max is that the optimal policy can be stochastic or we can approach a deterministic policy by selecting the action with the highest probability. If we include a temperature parameter in the soft-max then we can vary the same policy to be more or less stochastic as needed.
If we calculate preferences with linear features, then we would have feature vectors $\mathbf{x}(s, a) \in \mathbb{R}^{d^\prime}$ to match with the parameter vector $\boldsymbol{\theta} \in \mathbb{R}^{d^\prime}$. Then the preferences would be calculated:
$h(s, a, \boldsymbol{\theta}) = \boldsymbol{\theta}^\top \mathbf{x}(s, a)$
Another advantage is that for some problems the policy may be easier to approximate than the action-value function. We can also inject some prior knowledge of the environment into how the policy is parametrized.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$fee14dfe-c5ca-4126-a830-cc9d7eda5433cell_id$fee14dfe-c5ca-4126-a830-cc9d7eda5433code1const mountaincar_continuous_test_train2 = actor_critic_with_eligibility_traces_binary_features_gaussian_actions(mountaincar_continuous_mdp2, 0.05f0, 0.8f0, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, typemax(Int64), 100_000; α_θ = 5f-4, α_w = 0.0008f0)metadatashow_logsèdisabled®skip_as_script«code_folded$b53dba81-a9e9-41da-8fc2-7736bf25f2dccell_id$b53dba81-a9e9-41da-8fc2-7736bf25f2dccodejif run_mountaincar_binary_episodic_countinuous_param_study > 0
actor_critic_binary_episodic_gaussian_parameter_study(mountaincar_continuous_mdp, mountaincar_tilecoding_setup.get_active_features, mountaincar_tilecoding_setup.num_features, mountaincar_binary_continuous_params, 4, 3, 1000; max_steps = 100_000)
else
md"""
Waiting to run parameter study
"""
endmetadatashow_logsèdisabled®skip_as_script«code_folded$beb01fb8-c77d-4b5c-a66d-3812415e04a3cell_id$beb01fb8-c77d-4b5c-a66d-3812415e04a3code ### *Exercise 13.4*
> For the Gaussian policy parameterization, derive the formula for the eligibility vector $\nabla \ln{\pi(a|s, \mathbf{\theta})}$
Starting with our expression for the parameter function, we can calculate the gradient:
$\nabla \pi(a|s, \mathbf{\theta}) = \nabla \left ( \frac{1}{\sigma(s, \mathbf{\theta}) \sqrt{2\pi}} \exp \left ( - \frac{(a-\mu(s, \mathbf{\theta}))^2}{2\sigma(s, \mathbf{\theta})^2} \right ) \right )$
We will eventually need $\nabla \mu$ and $\nabla \sigma$ so let's calculate them now.
$\nabla (\sigma(s, \mathbf{\theta})) = \nabla \exp{( \mathbf{\theta}_\sigma ^ \top \mathbf{x}_\sigma (s))} = \sigma(s, \mathbf{\theta})\mathbf{x}_\sigma (s)$
$\nabla(\mu(s, \mathbf{\theta})) = \nabla ( \mathbf{\theta}_\mu ^\top \mathbf{x}_\mu(s)) = \mathbf{x}_\mu (s)$
The first application of the quotient rule is trivial, I will omit the input arguments to μ and σ keeping in mind that these are functions of the parameters. Also let $\left ( - \frac{(a-\mu)^2}{2\sigma^2} \right ) = f(\mu, \sigma)$ which results in $\pi(a|s, \mathbf{\theta}) = \frac{1}{\sigma \sqrt{2\pi}} \exp{(f(\mu, \sigma))}$. Therefore:
$\begin{flalign}
\nabla \pi(a|s, \mathbf{\theta}) \sqrt{2\pi} &= \frac{1}{\sigma ^2} \left (- \exp{(f(\mu, \sigma))} \nabla \sigma + \sigma \exp{(f(\mu, \sigma))}\nabla f(\mu, \sigma) \right ) \\
&= \frac{1}{\sigma ^2} \left ( -\exp{(f(\mu, \sigma))} \sigma\mathbf{x}_\sigma + \sigma \exp{(f(\mu, \sigma))}\nabla f(\mu, \sigma) \right ) \\
&=\frac{\exp{(f(\mu, \sigma))}}{\sigma} \left (-\mathbf{x}_\sigma + \nabla f(\mu, \sigma) \right ) \\
\end{flalign}$
Now we need only calculate the gradient of $f$:
$\begin{flalign}
\nabla f(\mu, \sigma) &= \frac{-1}{2} \nabla \left [ \frac{(a-\mu)^2}{\sigma^2} \right ] \\
& = \frac{-1}{2\sigma^4} \left [-2 \sigma^2 (a - \mu) \nabla \mu - (a - \mu)^2 2\sigma \nabla \sigma \right ] \\
& = \frac{-1}{\sigma^3} \left [ -\sigma (a - \mu) \nabla \mu - (a - \mu)^2 \nabla \sigma \right ] \\
& = \frac{-1}{\sigma^3} \left [ -\sigma (a - \mu) \mathbf{x}_\mu (s) - (a - \mu)^2 \sigma \mathbf{x}_\sigma \right ] \tag{substituting gradients}\\
& = \frac{(a - \mu)}{\sigma^2} ((a - \mu) \mathbf{x}_\sigma + \mathbf{x}_\mu) \tag{simplifying}\\
\end{flalign}$
Now substitute this back into the policy gradient:
$\nabla \pi(a|s, \mathbf{\theta}) \sqrt{2\pi} = \frac{\exp{(f(\mu, \sigma))}}{\sigma} \left (-\mathbf{x}_\sigma + \frac{(a - \mu)}{\sigma^2} ((a - \mu) \mathbf{x}_\sigma + \mathbf{x}_\mu) \right )$
Furthermore, observe that $\pi(a|s, \mathbf{\theta}) = \frac{1}{\sigma\sqrt{2\pi}} \exp(f(\mu, \sigma))$
So our expression for the policy gradient is:
$\nabla \pi(a|s, \mathbf{\theta}) = \pi(a|s, \mathbf{\theta}) \left (-\mathbf{x}_\sigma + \frac{(a - \mu)}{\sigma^2} ((a - \mu) \mathbf{x}_\sigma + \mathbf{x}_\mu) \right )$
To get the eligibility vector we must divide this by the policy which is conveniently already in the expression:
$\begin{flalign}
\frac{\nabla \pi(a|s, \mathbf{\theta})}{\pi(a|s, \mathbf{\theta})} &= -\mathbf{x}_\sigma + \frac{(a - \mu)}{\sigma^2} ((a - \mu) \mathbf{x}_\sigma + \mathbf{x}_\mu)\\
&= \mathbf{x}_\mu \left [ \frac{(a - \mu)}{\sigma^2} \right ] + \mathbf{x}_\sigma \left [\frac{(a-\mu)^2}{\sigma^2} -1 \right ] \\
\end{flalign}$
There are two components to the sum, one for $\mu$ and one for $\sigma$. If we think of the paramters and feature vectors as concatenated, then this sum would be an element by element sum where $\mathbf{x}_\mu$ has a zero value for all the feature indices corresponding to $\sigma$ and vice-versa. This way doing the sum will form one complete vector that has gradient components for all the parameters $\mathbf{\theta}_\mu$ and $\mathbf{\theta}_\sigma$. Alternatively, the sum can be separated and each gradient can be treated separately with only those components keeping them separated throughout the calculation.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$8bc280db-e57d-4e40-be46-1790f4f7d9e7cell_id$8bc280db-e57d-4e40-be46-1790f4f7d9e7codefunction actor_critic_fcann_parameter_study(mdp::StateMDP{T, S, A, P, F1, F2, F3}, update_feature_vector!::Function, num_features::Integer, hidden_layers::Vector{Int64}, λ_θ::T, λ_w::T, α_r̄::T, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, max_steps::Integer; nruns::Integer = 100, seed = rand(UInt64), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
Random.seed!(seed)
init_policy_params = FCANN.initializeparams_saxe(num_features, hidden_layers, length(mdp.actions))
make_trace_data(α_θ_list, α_w) = [average_continuing_runs(nruns, seed, α_θ, α_w, α_r̄, init_policy_params, actor_critic_with_eligibility_traces_fcann, mdp, λ_θ, λ_w, num_features, hidden_layers, update_feature_vector!, max_steps; kwargs...) for α_θ in α_θ_list]
traces = [begin
scatter(x = α_θ_list, y = make_trace_data(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Step in the First $max_steps Steps Averaged Over $nruns Runs", xaxis_type = "log", title = "$num_features Input, $hidden_layers Hidden Non Linear Approximation, λ_θ = $λ_θ, λ_w = $λ_w"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$89901156-b874-416b-89c1-6dc434a4eb17cell_id$89901156-b874-416b-89c1-6dc434a4eb17code(md"""
### *REINFORCE Implementation*
"""metadatashow_logsèdisabled®skip_as_script«code_folded$ff76ef94-fdf5-41f3-a31a-21c4629efabecell_id$ff76ef94-fdf5-41f3-a31a-21c4629efabecode(const corridor_mdp = make_corridor_mdp()metadatashow_logsèdisabled®skip_as_scriptëcode_folded$581f7e9b-a5c2-4841-9605-85f9585b0274cell_id$581f7e9b-a5c2-4841-9605-85f9585b0274codeٺupdate_linear_action_preferences!(action_preferences::Vector{T}, x::Vector{T}, params::Matrix{T}) where T<:AbstractFloat = BLAS.gemv!('T', one(T), params, x, zero(T), action_preferences)metadatashow_logsèdisabled®skip_as_script«code_folded$8aa16866-bfda-48df-9cf1-cf3d2e203ccbcell_id$8aa16866-bfda-48df-9cf1-cf3d2e203ccbcodeofunction cartpole_tilecoding_reinforce_continuous_parameter_study(α1_list, α2_list, max_episodes; num_trials = 100, kwargs...)
setup = setup_cartpole_problem(;kwargs...)
traces = [begin
steps = [begin
1:num_trials |> Map() do i
solution = reinforce_with_baseline_monte_carlo_control_binary_features_gaussian_actions(cartpole_setup.mdps.episodic.continuous, cartpole_setup.get_active_features, cartpole_setup.num_features, max_episodes; α_θ = α1, α_w = α2)
steps = solution.episode_steps
isempty(steps) && return 0
mean(steps)
end |> foldxt(+) |> x -> x / num_trials
end
for α1 in α1_list]
scatter(x = α1_list, y = steps, name = "α_w = $α2")
end
for α2 in α2_list]
plot(traces, Layout(xaxis_title = "Policy Learning Rate α_θ", yaxis_title = "Average Episode Duration Over First $max_episodes Episodes", xaxis_type = "log"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$04b5929a-2058-49c9-963a-96c752a1d67dcell_id$04b5929a-2058-49c9-963a-96c752a1d67dcodeIplot_continuing_step_rewards(cartpole_continuing_fcann_test.step_rewards)metadatashow_logsèdisabled®skip_as_script«code_folded$f0104778-81a6-417b-8501-f916e5e7f3afcell_id$f0104778-81a6-417b-8501-f916e5e7f3afcodeAfunction make_corridor_continuing_mdp()
function step(s::Integer, i_a::Integer)
δ = 2*i_a - 3 #calculates the s change -1 for left (1) and 1 for right (2)
switch = iseven(s) #returns true in state 2 which is where actions are switched, when switch is true, multiply δ by -1, otherwise by 1
c = 1 - 2*switch
s′ = s + c*δ
goal = s == 4
left_limit = s == 0
s′ = ifelse(left_limit || goal, 1, s′)
r = Float32(goal)
(r, s′)
end
actions = [:left, :right]
ptf = StateMDPTransitionSampler(step, 1)
StateMDP(actions, ptf, () -> 1, Returns(false))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$3e3c5897-809f-46e3-bb58-f115b082443ecell_id$3e3c5897-809f-46e3-bb58-f115b082443ecodefunction actor_critic_with_eligibility_traces_binary_features_beta_actions(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, λ_θ::T, λ_w::T, get_active_features::Function, num_features::Integer, args...; policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, N, A <: Union{T, NTuple{N, T}}, P, F1, F2, F3}
setup = setup_binary_beta_policy_arguments(mdp, get_active_features, num_features)
actor_critic_with_eligibility_traces!(policy_params, setup.eligibility_vector, value_params, BinaryFeatureVector(), mdp, λ_θ, λ_w, update_binary_action_preferences!, setup.action_distribution_parameters, make_beta_sampler(rand(A)), update_beta_eligibility_vector!, setup.feature_vector, setup.update_feature_vector!, binary_value_function, update_binary_value_gradient!, args...; kwargs...)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a9db3f85-ff56-4bbc-be87-47b893ef3b7bcell_id$a9db3f85-ff56-4bbc-be87-47b893ef3b7bcodefunction mountaincar_continuing_step(s, i_a::Integer)
a = MountainCarTask.actions[i_a]
s′ = MountainCarTask.step(s, a)
(s′[1] == 0.5f0) && return (1f0, MountainCarTask.initialize_state())
return (0f0, s′)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$08505e88-9c23-4e95-91e3-d18bf5133dbccell_id$08505e88-9c23-4e95-91e3-d18bf5133dbccodefunction actor_critic_binary_episodic_squashed_gaussian_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, amax::A, get_active_features::Function, num_features::Integer, λ_θ::T, λ_w::T, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, max_episodes::Integer; nruns::Integer = 100, max_steps::Integer = 10_000, seed = rand(UInt64), init_policy_params::Matrix{T} = make_n_param_dist_policy_params(2, num_features, rand(A)), init_value_params::Vector{T} = zeros(T, num_features), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
Random.seed!(seed)
function average_runs(α_θ, α_w)
1:nruns |> Map(_ -> actor_critic_with_eligibility_traces_binary_features_squashed_gaussian_actions(mdp, amax, λ_θ, λ_w, get_active_features, num_features, max_episodes, max_steps; α_θ = α_θ, α_w = α_w, policy_params = copy(init_policy_params), value_params = copy(init_value_params), kwargs...) |> x -> isempty(x.episode_rewards) ? -T(Inf) : mean(x.episode_rewards)) |> foldxt(+) |> x -> x / nruns
end
traces = [begin
scatter(x = α_θ_list, y = average_runs.(α_θ_list, α_w), name = "α_w = $α_w")
end
for α_w in α_w_list]
plot(traces, Layout(xaxis_title = "α_θ", yaxis_title = "Average Reward Per Episode in the First $max_episodes Episodes Averaged Over $nruns Runs", xaxis_type = "log", title = "Binary Feature Encoding with $num_features Features, λ_θ = $λ_θ, λ_w = $λ_w"))
endmetadatashow_logsèdisabled®skip_as_script«code_folded$ad0009af-2cfc-4820-bd4a-698ad391f459cell_id$ad0009af-2cfc-4820-bd4a-698ad391f459codebplot(scatter(x = LinRange(0, 1, 1000), y = make_beta_dist(beta_params...).(LinRange(0, 1, 1000))))metadatashow_logsèdisabled®skip_as_script«code_folded$16fcc2d0-9f2f-4226-9dcc-6d86248cab26cell_id$16fcc2d0-9f2f-4226-9dcc-6d86248cab26codeXfunction plot_state_distributions(p_left; kwargs...)
state_visits = collect_state_distributions(;p = p_left, kwargs...)
η = sum(state_visits)
μ = sum(state_visits, dims=1)[:]
μs_plot = plot(bar(x = 1:3, y = μ ./ sum(μ)), Layout(yaxis_range = [0, 1], xaxis_tickvals = [1, 2, 3], xaxis_title = "State", yaxis_title = "Probability", title = "Stationary State Distribution"))
p_not_term = sum(state_visits, dims = 2)
pterm = 1 .- p_not_term
termplot = plot(pterm, Layout(xaxis_title = "Step", yaxis_title = "Probability", title = "Probability of Episode Terminating On or Before Step"))
(n, m) = size(state_visits)
plots = [begin
v = state_visits[k, :][:]
vterm = pterm[k]
t = bar(x = 1:4, y = [v; vterm], name = "k = $k")
p = plot(t, Layout(width = 270, height = 250, yaxis_range = [0, 1], xaxis = attr(tickvals = 1:4, ticktext = ["1", "2", "3", "Term"], title = "State"), yaxis_title = "Probability", title = "Step $k"))
end
for k in vcat(1:5, 10:10:50)]
full_p = plot(heatmap(x = 0:20, y = 1:3, z = state_visits[1:21, :]' ./ sum(state_visits)), Layout(xaxis_title = "Step", yaxis_title = "State", title = "Probability Over States and Steps", yaxis_tickvals = [1, 2, 3]))
# p3 = plot(traces2)
@htl("""
Policy Probability for Left Action is $p_left and Average Episode Length is $η
State Distribution Per Step Including Terminal State
$plots
$termplot
$μs_plot
$full_p
""")
endmetadatashow_logsèdisabled®skip_as_script«code_folded$11063fff-4d36-46d5-828f-dbed0f46b9cfcell_id$11063fff-4d36-46d5-828f-dbed0f46b9cfcoderfunction actor_critic_fcann_parameter_study(mdp::StateMDP{T, S, A, P, F1, F2, F3}, update_feature_vector!::Function, num_features::Integer, hidden_layers::Vector{Int64}, λ_θ_list::AbstractVector{T}, λ_w_list::AbstractVector{T}, α_r̄_list::AbstractVector{T}, α_θ_list::AbstractVector{T}, α_w_list::AbstractVector{T}, num_tests::Integer, max_steps::Integer; nruns::Integer = 100, seed = rand(UInt64), kwargs...) where {T<:Real, S, A, P, F1, F2, F3}
Random.seed!(seed)
init_policy_params = FCANN.initializeparams_saxe(num_features, hidden_layers, length(mdp.actions))
run_test(α_θ, α_w, α_r̄, λ_θ, λ_w) = average_continuing_runs(nruns, seed, α_θ, α_w, α_r̄, init_policy_params, actor_critic_with_eligibility_traces_fcann, mdp, λ_θ, λ_w, num_features, hidden_layers, update_feature_vector!, max_steps; kwargs...)
test_params = [(α_θ = rand(α_θ_list), α_w = rand(α_w_list), α_r̄ = rand(α_r̄_list), λ_θ = rand(λ_θ_list), λ_w = rand(λ_w_list)) for _ in 1:num_tests]
DataFrame([begin
output = run_test(params...)
(;params..., output = output)
end
for params in test_params])
endmetadatashow_logsèdisabled®skip_as_script«code_folded$8fcdca63-01a0-4d4b-933c-06a7621d980acell_id$8fcdca63-01a0-4d4b-933c-06a7621d980acodeW#add neural network implementation of continuous policy gradient and do parameter studymetadatashow_logsèdisabled®skip_as_script«code_folded$33c99850-67cd-4754-94b9-6df97b238e27cell_id$33c99850-67cd-4754-94b9-6df97b238e27codefunction soft_max!(x::AbstractVector{T}) where T<:Real
minx, maxx = extrema(x)
if minx == maxx
x .= one(T) / length(x)
return x
end
s = zero(T)
@inbounds @simd for i in eachindex(x)
h = exp(x[i] - maxx)
s += h
x[i] = h
end
x ./= s
endmetadatashow_logsèdisabled®skip_as_script«code_folded$786a5385-b648-4fc3-8e19-bf6582828136cell_id$786a5385-b648-4fc3-8e19-bf6582828136codeKmd"""
#### Continuous Action Space
Now that we have verified the success of policy gradient methods on this problem, we can consider using a continuous action space where the policy can output a distribution over throttles. In the original problem, the maximum throttle value is 1, but the velocity of the car is already capped at 0.07. We can see if a policy attempts to use much higher throttle values to end the episode faster even if the physics is unrealistic. That observation would confirm a successful use of continuous actions where the throttle is an unbounded continuous value. The optimal policy would likely try to use the highest throttle possible to reach the maximum speed in either direction faster. We could apply friction to the problem so that the car would actually slip if it attempts to accelerate too quickly.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$573878bb-020d-40f6-9329-3d5f91843010cell_id$573878bb-020d-40f6-9329-3d5f91843010code^get_corridor_episode_stats(corridor_train.greedy_policy; max_steps = 100, ntrials = 1_000_000)metadatashow_logsèdisabled®skip_as_script«code_folded$2e7c737c-c798-4442-a7e1-d74ccfd73119cell_id$2e7c737c-c798-4442-a7e1-d74ccfd73119code6@bind ẋ Slider(-50:50; default = 0, show_value=true)metadatashow_logsèdisabled®skip_as_script«code_folded$9d264543-33ab-498a-90f5-5f913c252484cell_id$9d264543-33ab-498a-90f5-5f913c252484code]plot(reinforce_test4.episode_steps[1:10:end] ./ (1:10:length(reinforce_test4.episode_steps)))metadatashow_logsèdisabled®skip_as_script«code_folded$9cf3dc5f-8a25-479f-93db-06e34f0d37a0cell_id$9cf3dc5f-8a25-479f-93db-06e34f0d37a0code%plot_state_distributions(dist_plot_p)metadatashow_logsèdisabled®skip_as_script«code_folded$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70cell_id$d04d4234-d97f-11ed-2ea3-85ee0fc3bd70codeهbegin
using PlutoUI, PlutoPlotly, LaTeXStrings, PlutoProfile, HypertextLiteral, ProgressLogging, BenchmarkTools
TableOfContents()
endmetadatashow_logsèdisabled®skip_as_scriptëcode_folded$bd6a7c16-6c25-4fc2-8e1b-4dab693ce19fcell_id$bd6a7c16-6c25-4fc2-8e1b-4dab693ce19fcodedactor_critic_binary_episodic_squashed_gaussian_parameter_study(mdp::ContinuousMDP{T, S, A, P, F1, F2, F3}, amax::A, get_active_features::Function, num_features::Integer, params::@NamedTuple{λ_θ::T, λ_w::T, α_θ_min::Int64, α_w_min::Int64}, num_θ::Integer, num_w::Integer, num_episodes::Integer; kwargs...) where {T<:Real, S, A, P, F1, F2, F3} = actor_critic_binary_episodic_squashed_gaussian_parameter_study(mdp, amax, get_active_features, num_features, params.λ_θ, params.λ_w, 2f0 .^(params.α_θ_min:params.α_θ_min+num_θ-1), 2f0 .^(params.α_w_min:params.α_w_min+num_w-1), num_episodes; kwargs...)metadatashow_logsèdisabled®skip_as_script«code_folded$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72ccell_id$3e5fc75b-61a5-49d5-b5bd-3d2847f5f72ccodeلcorridor_train = sarsa_λ(corridor_mdp, 1f0, 0.99f0, typemax(Int64), 1_000_000, 1, get_corridor_features; ϵ = 0.5f0, α = 0.0001f0)metadatashow_logsèdisabled®skip_as_script«code_folded«notebook_id$6d683db8-38f5-11f0-0729-898e37e867d8in_temp_dir¨metadata