Hello and welcome to this new tutorial.
All right so now we're going to build the last method of this policy class which is a method that is
going to do the most important step of the whole algorithm which is basically great innocent.
More specifically it's an approximation of grain in the sense because we don't compute directly the
gradient of you know the reward with respect to the weights we approximate this gradient by measuring
the difference of the reward between the reward and the positive direction and the reward in the negative
direction.
So we're going to have a look at it in detail.
Now I will quickly explain the purpose of this great in this sense and we'll implement this very important
step which is the updates step one step of the great and descent.
So before we do this let's go back to the article to remind us what we have to implement exactly.
So as a reminder we just implemented this point five here by applying the perturbations on the matrix
of weight.
And we also did the sampling of the perturbations following a gash in distribution of variance the noise
which we implemented here and now we're going to move on to Step 7 make the update step which corresponds
to exactly one step of graden in the sense.
So why do we need to do this.
And how does it work.
Let me quickly remind the explanation.
Well in class egregiousness And you know what we do we want to optimize something which is generally
the last function you know we want to reduce the last error between a prediction and a target.
Right.
And we do that with respect to the weight so we're going to differentiate the less with respect to the
weight to reduce this loss error.
And then we will update the weight by adding this differentiation of the last with respect to the weight
to the weight.
And that will update the weights in a direction of last reduction here we don't have any loss.
We don't have any target and prediction here.
What we have is some we want we want to optimize the reward.
Therefore what we're going to do is not to try to reduce unless what we're going to do is try to increase
the reward and therefore we're not going to differentiate the loss with respect the weight we are going
to differentiate that we want with respect to the weight.
And that's why here what you see is exactly this differentiation but it's not a classic mathematical
differentiation it's an approximation of the differentiation of the reward because this reward that
you see here is the reward obtained by applying some perturbations to your policy in the positive direction
and this reward that you see here is the reward obtained by applying some perturbations to that same
policy in the opposite direction as this one.
And since the preservation consists of adding to the weight a very small number you know like a small
Epsilon following the normal distribution.
Well this difference of the words here is almost the same as differentiating the rewards with respect
to the weight because a real differentiation would be the partial derivative of the reward divided by
the partial derivative of the weights.
But since this difference is a difference of words with respect to directions that are opposite and
also with respect to some very small values of the perturbations.
Well this is exactly the approximation of the gradient of the word with respect to the weight.
All right.
And this method has a specific name.
It's called the method of Finot differences and that exactly happens right here with this Finot difference
of the words in two opposite directions by playing some very small perturbations in that approximates
the gradient with respect to the weights and then we multiply indeed by the value of the perturbation
as the way we do it with great in the sense.
So that's exactly one step of gradient descent but by approximating the gradient.
So it's important to get this because this is really the heart of the algorithm.
This is really where the added value is indeed the algorithm that we're implementing right now is called
commented random search but it's so important this is so crucial in the algorithm that we could call
it the methods of any differences.
In fact the method of any differences is at the heart of augmented random search.
All right there was an explanation of the purpose of this because this is really important.
And now let's implement this.
It's going to be very simple we just need to implement that sum.
So by the way we do this some of the finit differences of the words for all the best directions you
know the bigger response here to the best directions which I remind is one of our hyper parameters and
we set it equal to 16 but that can be modifiable.
So we're going to some of these finit differences multiplied by the perturbation itself.
For the 16 best erections and we are going to add this factor here alférez the learning rate divided
by B.
The total number of best directions and sigma are which is the variance of the words.
Because indeed in 3.1 we haven't taken care of 3.1 yet.
But in order to improve the algorithm we need to scale by the standard deviation Sigma or so here we
are taking Cygne are in fact to normalize what we want.
All right.
So let's do this let's implement this.
And so now I'm going to go back to Python to make this final method of our policy class let's do this.
So this method we're going to call it a date because indeed it's the update step of the paper and that
corresponds to exactly one step of approximated great in the sense.
So date and now it's going to take three arguments the first one is self as usual because we're going
to use our future object or future instances of the policy class.
Then the second argument you cannot really guess where it's going to be but it's going to be rolled
out.
Actually they talk about rollouts in the paper and a rollout is basically going to be a list of several
triplets and the triplets will contain First the word obtained in the positive direction.
Then second the reward obtained in the opposite direction.
And third the perturbation in this specific direction that gave you these two first elements that we
were in the positive direction and that we were in a negative direction.
So basically it's a session that will contain the best triplets of rewards obtained by applying some
perturbations in the best directions.
Right.
So roll out and then the last arguments.
Remember in the paper we're going to use this Sigma are here the standard deviation of the word that
we will take care of later on in the implementation.
But we need to put it here as argument because it is not one of our hyper parameters we will compute
it later on.
So we're going to call that sigma on this core are to specify that it is a standard deviation of the
reward.
All right to self rule out Sigma are all good.
Let's make this a data step.
So speaking of step I'm going to introduce here first variable step that is going to be exactly this
part which is the some of the best directions being the number of reservation's of this finit difference
of the words obtained in respectively the positive direction and the opposite or negative direction
multiplied by the value of the perturbation step is going to be exactly this.
So that's why I'm going to initialize step as a matrix of zeros and the dimensions of this matrix will
be exactly the same as the dimensions of the matrix of weights theta.
Now you might be wondering why didn't I use to star you that simply because the run and function is
expecting a different format than the format expected by the zero's function the random function expects
this couple of two dimensions.
You know self-deceit at that shape of the next zero for the number of lines and then come on then said
that they did that shape and the next one for the number of columns whereas the zeros function here
directly expect selve that feature that shape basically the shape of theta.
OK.
So just some different formats expected by two functions.
And now we've initialized this matrix step that represent all the sum here with the right format because
we're going to sum this to the current matrix of weight theta.
So now we're going to do exactly this some here and the best way to do a sum like that is through a
for loop.
So now we're going to do a full loop and we need to iterate on all the rollouts because I remind that
the rollouts contain exactly the triplets of the puzzle we want there is the word think in the positive
direction the negative reward there is we were obtained in the opposite direction.
And then the value of the perturbation you see the purpose of creating the structure rollout that gathers
the three elements because that's exactly the three elements we need to do this.
And then the rural adds there are the triplets and B response to the number of best erections which
is part already of our hyper parameters.
So that's why now in order to make this some We're going to loop over all the positive words which I'm
going to call our Pazz.
All the best but the rewards actually.
That is all the rewards obtained in the best directions then.
Same for the negative reward our neck.
But these are negs we were trained in the opposite direction as the direction that was used to obtain
this are post here.
So they kind of go together.
And then of course the value of the perturbation itself which we can call deep.
So that's the triplets values that are all into rollouts which is one of our arguments here but we will
build to roll out later on.
So for all these triplets of positive reward negative reward and perturbations.
Well now we can have data step by taking the step which is right now equal to zero and we're going to
add to this step.
That's why I'm using plus equals.
We're going to add exactly this element here that is the finit difference of the word obtained in the
positive direction and the reward obtained in negative direction multiplied by the perturbation.
And we already have all these things to rule out.
This is our pose.
This is our neg.
And this is D.
So what we're simply going to add to our current value of our matrix step here is our pas plus every
word minus our neg the negative word or should I say we word obtained in the opposite direction as the
direction that was used to obtain Arpels multiplied by D which is the perturbation that was applied
in the same direction that led us to obtain our POWs and RNA.
But this was obtained in the opposite direction of this perturbation.
All right.
So we have just a data step that's great.
And now we have one final thing to do.
It's just of course the most important it is of course to update the matrix of weight our matrix of
weight theta is going to become theta plus all this which we already computed thanks to this for loop
but multiplied by this factor here which is the learning rate Alpha bit which we called learning rate
divided by the total number of best directions times the standard deviation of the reward.
All right.
We already have all this in our arguments so let's make that final computation we need to get of the
for loop because the for loop was just to compute this some here so that's done.
So now we just need to add this factor here and this is exactly what we're going to do as the last line
of this update function.
Let's do this let's update our matrix of weight theta so self that theta is going to become the previous
self that theta Plus that's what I'm adding plus you call our learning rate which we can get by taking
our hyper parameter object that was not created yet but which we will create in the future HP.
And then we take its proper variable learning rates.
Here we go.
That's the learning rate which we have to divide by Remember the total number of resurrections be times
the standard deviation of the word.
Let's go back let's get first of all number of resurrections B which is one of our hyper parameters
so I'm taking H-P again.
Our hyper barrière object.
And from this object I'm it's variable.
And the best directions.
And then we multiply this by Sigma R which is a standard deviation of the word and which is one of our
argument here we'll compute that later.
But at least we have everything to make this formula to have the matrix of weight in a direction that
increases the most that we want which is exactly what we want to do with our AI because the higher the
we word the better the AI has the ability to walk.
All right.
And then of course final step we need to multiply this by the step that is the sum of the finit differences
of the word multiplied by the perturbation.
Congratulations.
That was the most important step of this implementation of the step the step of approximated great in
the sense you did it and now we have two final things to do first.
Another function which is to explore function and that will compute either an average reward or some
of reward on one full episode because we imagine that we're not going to compare the words in the different
directions on one single action.
We want to compare them on a large series of actions in a row.
Otherwise this wouldn't be very relevant to compare the rewards to each other.
You know we want to have a relevant measure of that we want and so we're going to measure the words
on one full episode.
And I remind that an episode is just a session where either the AI manages to walk up to the episode
length or it's when it fails.
You know when it fails that's the end of the episode.
We reset the environment and the air tries to walk again.
So we're going to measure the words in each of the 16 directions on one full episode.
And so in the next day Doyle will make this explore function that will simply return either the average
word or the accumulated reward by playing the perturbations in one specific direction over one full
episode and then we'll compare all the accumulated rewards for the different directions and we'll sort
them by the best ones you know the best couples of rewards of ten are POWs in our neck.
And that's exactly what we'll do as this step 6 year.
So the directions by Max Pozza reward up as it were reward over the different directions.
All right.
So we'll do all this in the next tutorials.
And until then enjoy AI.