Hello and welcome this news it's horrible.
So we've gathered the positive words and negative words we computed the standard deviation of all these
we want and now we have to do a step which we haven't done any time so far during the training and I'm
talking of course about that step six just before we made the update step and the steps six consist
of soaring the directions by the maximum of the couple of positive words and negative words meaning
the words we get by applying the perturbations in a positive direction and the reward we get by applying
the perturbation in the opposite direction.
So we have these couples of positive and negative rewards for each of the 16 directions that we're testing
and for each of the 16 directions we're going to get these positive reward and negative word.
We're going to take the next of them and then we'll sort all the directions by the highest of these
maximum of words.
That's what we need to do.
And of course we want to do this because we looking for the directions that increased the most the word
because indeed the higher is the reward we reach the better the I will have the ability to walk.
So it's always about trying to optimize and increase the reward.
And that's why right after the step six you have this update step one step of great the sense which
will update your weight in these best directions then increase the rewards.
And here this is the approximated grade in the sense that we're doing by approximating the gradient
of the word with respect to the weight.
All right so let's do this let's do this.
Step 6 soling the directions by the max of the rewards.
OK so how are we going to do this.
Well first we are going to introduce a variable that we're going to call scores and that will get these
you know maximums of the reward in the positive direction and the reward in negative direction or the
opposite direction.
So we're going to get these maximum first for each of the 16 directions.
All right.
So let's do this let's go back to Python.
So our scores here how are we going to gather all these maximums.
Well since then after gathering this course here we're going to use the Soledad function which is a
function that allows you to sort the values of a dictionary.
Well the thing we're going to do with course here is set it as a dictionary in which the keys will be
just the numbers from 1 to 16 or no from 0 to 15 and the values of this case will be these maximums
that we are trying to sort.
So that then it will be much easier for us to sort these maximums.
So the first thing we're going to do here is add some brackets like that because that's the syntax of
a dictionary in Python.
So we had some brackets and then we first need to define the keys which I remind are just going to be
some integers starting from 0 to 15 because each cue will correspond to one of the 16 directions that
we're testing.
So clear here will represent this key and integer going from zero to 15.
And then you need to add a column and then you specify the value of that key and the value of that key
is going to be exactly the maximum of the reward we get in the positive direction.
For one specific direction then the reward we get in the negative direction meaning the opposite direction
as this positive direction.
That's our values the maximum of this.
We will we get in the positive direction and that's where we get in the negative direction.
And of course we need to get these keys and maximum of these rewards for each of the six indirections.
So now what we're going to do is a for loop inside the dictionary.
We can do that to classical Trigon python instead of putting the dictionary inside a for loop.
You put the for loop inside a dictionary and you can also do that to populate a list.
And so I'm going to add here of four and then you know we want to generate this for each of the 16 keys.
Each of the key corresponding to one of the six indirections.
So since our keys here are the case I'm going to do K but also the word in the positive direction and
the reward in the negative direction because we need to get them as well in.
And now we're going to get everything using two tricks to enumerate function which will enumerate the
integers.
Want to get that is the case from 0 to 15 and the zip function together.
The positive words and the negative words because indeed we need the our post we want in the positive
direction and our nega we work in the negative direction.
So here is what we have to do.
We take first enumerate to generate integers and then inside enumerate we take the zip function that
will gather together the positive reward which we already gathered here all the positive words.
And of course the negative reward this all of this here will get me the case that is integers from 0
to 15 corresponding to the 0 to 15 directions.
Then the puzzle was that is the words we get by applying the perturbations in opposite directions.
In other words these guys here and the negative words that is the reward we get by playing the perturbations
in the negative directions are the opposite directions as these ones.
And these are these ones negative word.
So all good.
We just made a dictionary taking as the keys to integers from 0 to 15 corresponding to the 16 directions
that we're testing and as values the maximum of these words in a positive direction and we words in
a negative direction.
All right.
So that's the first thing done then next step.
Now that we have the scores we want to sort them and because we made a dictionary it's going to be very
easy for us to sort them indeed we're going to use the sorted function which is a function by Peyton
to sort just the keys of the dictionaries by their values you know from their highest value to their
lowest value which means that we're not going to get.
Now this time the maximums themselves we're just going to get these indexes in the list.
But instead of having the list from you know 0 1 2 3 to 16 we'll get the list.
In this specific order sold by the maximum of these we want.
So for example if the highest of these maximums is is next three and if the second highest is in the
next 7 for example well the order list that we're about to get will start with three and seven.
And then because we have a dictionary with all the keys and values well organized well we'll be able
to find the highest word things to the indexes which are the keys.
All right so what I'm going to do now is introduce a new variable which will be order and which will
be a list that will contain these keys of the highest.
Couple of words right of the highest maximum of positive reward and negative reward.
So how are we going to get these keys and indexes.
Well it's easy.
We need to take the sorted function by Python which will sort your dictionary.
And in this sort function the first thing that to specify or input or what you want to get in order
do you want to get the maximum of the rewards.
Or do you want to get the keys.
Are you going to say do we want to get the keys because these are the keys that will help us find everything
meaning the directions and the words in the positive direction and opposite directions for that specific
direction.
So what you want are the keys and therefore the first thing I'm going to specify here is this chorus
that keys that is the keys of my scores dictionary.
Then we add some parenthesis here and then here we go we get the keys and now you add a comma and that's
when you specify by what you want to sort the keys of your dictionary.
So you get the keys of your dictionary but then you want to sort them by these maximums for each of
the directions starting from 0 to 15 because we have 16 directions in total.
And so to do this you're going to do it through function.
It's basically the argument expected by the sort of function here and this argument is called qi but
be careful it's not to be confused with the keys or dictionary.
So we take key and then we add equals and then Lunda which is just to specify that we're about to do
a function then the argument of the function which we call expert which will be the key of the dictionary
then colon and then what this Lunda function will return which are this course right the scores of the
keys xx.
So that right now thanks to this function here we know that we want to sort our scores dictionary by
discours here which are the maximums of the word we get in the positive direction in the word we get
in the negative direction for the 16 different directions and by specifying scores that keys here will
get not these maximums returned in this order list but the keys of these dictionaries correspond into
the keys of the highest word among the 16 positive and opposite directions.
All right so that's a very useful trick in Python if you want to sort a mapping of indexes and values.
You know the indexes here were just the indexes of the directions from 0 to 15 and the values where
the maximums of these words correspond to each of these 16 directions.
So that's a useful trick to know with them that's not over here we get a list here but we won't remember
to get the best directions.
Right.
We have the total number of directions but we also have the total number of best directions.
And remember that in the paper here we're not summing on all the directions but the best directions
be the number of best directions here.
So what we simply need to add here is you know since this is a list well we're going to add some brackets
to take the best directions on.
And how can we do that.
Well since this list is already sorted by the maximums of the reward in the positive direction and we
were in the opposite direction.
Well what we simply need to do here to get the best solutions is just to go from zero to the number
of best directions and you know we don't even have to specify the Zero Year because the lower bound
of a range is by default zero.
So here we are taking the first and the best directions elements which corresponds to the highest maximums
of positive reward and opposite reward because in the way this was are sorted by these maximums.
All right so that's the simple thing we have to do here.
And this gives us exactly what we want or at least what exactly is said in the paper.
Perfect.
So now we have one final thing to do now that we have these indexes but it's not all of course.
Now we want to use these indexes to get Indeed the reward we get by applying the perturbations in the
best directions that are sorted.
And also the reward we get by applying the perturbations in the opposite directions of these resurrections
and of course the perturbations.
Why do we want to get these three things that of course because in the next step will make the update
step which will take the words we get the positive best directions and the rewards we get in the negative
best directions and also the perturbations the values of the perturbations for each of these desperations.
So that's why when together all this now and then we'll be able to make that data step which corresponds
to one step of and descend to approximate the gradient in order to date the weight in these best directions.
All right so now what are we going to do.
Remember we talked about a new concept before which are the rollout.
Well here we go you know that's the update function which takes us in put the roll out.
And therefore since the next step will be to apply the update function to make this one step of gradient
descent.
Well we need to gather these roll out and I remind that these roles are nothing else than the best directions
triplets of the words and the positive resurrections the words in the opposite desecrations and the
perturbation D.
So what we need to do now is prepare those roll out and this will be very easy for us because indeed
we have all the keys of dictionaries for the best directions and therefore will use these keys to get
the positive word the negative reward and the preservation.
So let's do it let's prepare here.
Our rollouts are rollout which will be the least of the following triplets composed of First the positive
rewards.
That is the words attained by playing the perturbations in a positive direction for the best directions.
So here since we to take the best actions of course we're going to add here the index K and K you will
see will be all the case in the order list because the list contains indeed the indexes of the best
directions.
It's puzzle with words.
K then of course I'm going to copy this because the next element of the triplet will be the negative
words OK.
So I'm replacing that here.
Negative three words OK.
And then of course the third element of the triplet as we said is the perturbation deltas.
And again we take in excess K of the best directions and in order to get these preservations the final
touch of what we're doing now is to add of course a new full loop inside the list.
Now I tell you that we can do it inside a dictionary or inside a list.
It's a very classic trick in Python.
And of course we're going to get all the case in the order list composed of the indexes of the best
directions leading to the highest we want.
All right so here we go we have the rollouts now and so we are ready to make that data step to do this
one step of greatness and to have data weight in these best directions.
Let's do that and the next is oil.
And until then enjoy AI.