Hello and welcome to this new tutorial.
So we made good progress so far because we built the AI and now we're going to make an additional function
which we'll call the Xplornet function and that will not belong to any class but is just a function
we will use to explore.
As you can see our policy on one specific direction and over one episode.
So let me quickly remind why we have to do that.
Remember the principle of this whole A R S algorithm is that we are testing several directions to update
the weights and we want to take the direction that increases the most that we want.
And then you know we take the opposite direction because we're going to do a differentiation of the
gradient to update the weights in that best optimal direction.
But when we compare the directions with each other we need to find a relevant measure of do we want
to assess in a relevant way which direction is the best and therefore we're not going to compare the
words on one single action no by getting the reward obtained by playing one specific action after applying
the perturbation on the weights.
No we're going to do that on one full episode because one full episode is composed of several actions
actually a lot of actions because you know it can last a few seconds and there are a lot of actions
happening in a few seconds and therefore we'll get lots of rewards for each direction.
And then by just taking the average of those words or some of that we words like the cumulative word
over the full episode.
Well we will have a set of several words each word corresponding to one specific direction and this
will be a much more relevant way to measure that we want to compare them with each other and get the
highest ones and therefore get the best reactions.
So that's what we have to do now and we'll do it through a function which we'll call the Explore function.
So this function is going to take several arguments.
The first one is the environment of course because we're exploring in one specific environment and I
remind that we will test many of them.
The second one is the normalizer because we will use Indeed the normalizer to normalize the states.
I remind that the states are the input of the policy or the perception of the policy and each time we're
going to feed those inputs to the policy we will normalize them first so that's where I'm taking the
normalizer here.
Then of course we're going to take our policy.
We're exploring our policies so we need to specify in the arguments which policy we want to explore
then the direction which will set by default equal to none.
Because again we will use the evaluate method.
Remember from the policy class and we just want to remind that the direction is set by default to none
because the evaluate method is not only used to apply some perturbations and getting the output when
we feed the input but also when we simply want to get the output by filling the inputs and without applying
any perturbation to direction it Course none.
And then one final argument.
Delta equals none as well.
Right Delta as a perturbation.
All right so it takes these four arguments and now we're going to do the whole exploration on one full
episode.
So the first thing we have to do of course is to reset the environment because we have to start from
scratch we have to start from the very beginning of a new episode in the very beginning of a new episode
is when the ice starts.
You know I remind that an episode is when they start to try to walk and then at some point it either
falls and that's the end of the episode or it manages to walk for a few seconds.
And so in that case the end of the piece that will be when the eye reaches the abyss at length which
is one of our hyper parameters.
So we are going to reset the environment and to do this we're going to take our environment and we're
going to add the reset method.
But the reset method actually returns the first date when you reset the environment.
And therefore here I'm going to add state equals to get that first date and that's perfect.
That's going to be our first input of this exploration on one full episode.
All right so we have our state and now the next step is to introduce a done variable which is a Boolean
that is equal to either true or false and which will be equal to true when the episode is done.
That is when we reach the end of an episode and false if we haven't reached yet the end of the episode
and we will initialised of course to Fox because at the very beginning of course the episode is not
done yet.
The next step we have to introduce the number of actions played and we will call this non-place.
So that's number of action plays and therefore of course we have to initialize it to zero because at
the very beginning no action was played yet and we will even initialize this to zero point because later
on you'll see that this number of actions played will be used in the float computation.
All right.
And then one final variable will introduce an initialized to zero and it's of course what we want to
get in the end.
So that's the most important one which is the sum of the reward.
Or if you want you can call it the accumulated reward.
And it doesn't have to be the sum of the reward it could also be the average reward.
The only thing that matter is that we need a relevant measure of the world over the full episode.
And since you know that we can either be positive or negative.
Well by taking the accumulated reward no something the reward.
Each time we play a new action well this will get us a relevant measure at the end of how high the final
reward is going to be over the full episode.
And so we will initialize this sum of words to zero and we don't have to set it as a float because it
won't be used in a flawed computation.
All right so perfect.
Now we're done with the initialization of the variables.
And so now the next natural step is to do a loop because indeed an episode will contain a lot of actions
to play and we will do some kind of loop until we reach the episode length that is until the number
of actions reaches the opposite length.
The episode length is basically the total number of actions you can play in an episode.
You can see it this way.
So not only we're going to loop until the number of action splayed reaches the episode length but also
we have to do this loop as long as we don't reach the end of the episode.
That is as long as done here is a quote for us as soon as dawn is equal to true.
The exploration of the policy will be over and therefore the most natural loop that we should use here
is not to fall because of this done very well here but a while loop is in the while loop we can specify
easily that you want to do the loop as long as done is equal to false.
All right.
So the way to do this world loop is well done is equal to falso So as long as done is not equal to true
and by typing was not done like that its exactly saying as long as done is not equal to true because
the default value of a boolean is true.
So by just writing done here its like writing down equals equals true.
So well not done because true basically.
And the number of action played is lower than the fool episode.
Lengths and as you can notice I didn't forget the HP hyper parameter object which will create later
on because episode length is a variable of our hyper parameters class.
No the class containing all the hyper parameters.
So while we haven't reached the end of the episode and wow the number of actions played is lower than
the total episode length which I remind was set to 1000.
Well in that case we are going to compute the cumulated reward on the full episode.
And so let's do this.
You're going to see the steps to do it are going to feel very natural now that we've done the hardest
part.
So the first natural step is to feed the perception of the policy with the input because indeed we want
to return the output which is the action to play because that action to play will get us our first word.
But before we feed the input to the perception remember we need to normalize it.
And before normalizing it we need remember the mean and the variance because in order to normalize it
we indeed need the mean and standard deviation which is the root of the variance.
So the very first step here is to take our normalizer which is one of our arguments here and then use
the observe method which we apply to the state because remember we normalized the state and that the
observed method remember will give us exactly the mean and the variance.
So now we have the mean and the variance.
And so that's when we can normalize our state.
And so this time we're going to use the normalized method which returns directly remember the normalized
state it takes as input to state and returns the normalized state.
Therefore since it returns something here and it returns the state will directly get our state which
so far was just the first state of the environment.
And this state will be equal to what is returned by this normalized method of the normalizer class and
therefore I'm taking the normalizer object of the normalizer class and I'm going to use the normalize
method which I'm going to apply to the state.
The previous version of the state which by applying the normalized method will become the normalized
state.
And now we can feed the perception with this state because it was indeed normalized.
And we want to work with the normalized version of the alias of the paper.
So now that we have this date we have input so we can fit it to the perception.
And so now we're going to naturally use the evaluate method in of course that specific direction that
we want to explore.
By applying the perturbation in that direction.
So right now what we have to do is apply the evaluate method by taking as input the state that which
is normalized then the perturbation Delta which is one of our argument and the direction which is another
one of our argument.
And since the evaluate method of course returns the action to play.
Well I'm going to get that action that we're going to play now by taking the evaluate method but be
careful the evaluate method belongs to our policy object.
So we take policy that evaluate this evaluate method takes on the three arguments first the input state
which was well normalized then the perturbation Delta and then the direction of the perturbation.
So right now the action is set by default to none and Delta I said bellyfull to non-bird of course when
we were going to explore the policy on one specific direction and for one specific perturbation.
Well these will not be equal to none.
OK so here we evaluate our policy by filling to the input state and by applying the perturbation delta
on one specific direction and all of this returns the action played in that situation.
And now since we played an action well that will lead us to a next date that will lead us to a word
and also that lead us to know whether Dunn is still equal to false or true.
And luckily for us we're going to get these three updates of the words very easily.
It is thanks to the step method of the environment class of the Bible library which just takes us and
put one argument which is the action played and that we've just got and that will return the state.
The word and this done variable you know when we've just played the action and we reach the new state
of the environment so that's why in order to get on these next stage and we would well we simply to
update our variables here state we want and done.
And then there's even a fourth variable but that we don't use here so no need to worry about this.
And here we go to get these next state we wanted and done well we simply need to take our environment
which is not ject of the public library then the step method which only takes as input one argument
which is of course the action that was just played.
And this step function this step method from the environment object of the public library returns to
next state and the environment.
The reward obtained after playing that action and whether or not the episode is done.
All right.
Perfect.
And now that you've just got the word we have to do a trick that is very classic in reinforcement learning.
It's the fact that we don't want to be biased by any super high reward or super low negative reward
you know imagine that.
Imagine that most of the words we get are taking small values like for example between minus 1 and plus
1.
But in the same episode you find one or two rewards are taking some very high values.
Well imagine the bias and the final accumulated reward or the final average reward that would cause
to keep these very high values that would disturb the measure of the reward and therefore for those
outliers we call them outliers these very high reward or very high positive or negative rewards that
we can get during the expiration of one full episode.
We want to ignore them or we want to set them equal to either plus one for the very high positive reward
or minus one for the very high negative reward.
And by doing this we will protect ourselves from any bias due to the outliers.
And the trick to do this is actually to force all of super high positive rewards to be equal to one
and those who were high negative reward to be equal to minus one.
So we can either do it with an IF condition but there is a better trick faster trick which is you take
your reward and you take the max of the men between the reward and 1 and minus 1 and you can check that
this will get you that we want all the wars between minus 1 and plus 1 or more specifically all the
super high but every word will be forced to be equal to one and no the super low negative reward will
be us to be equal to minus 1.
That's a classic trick in reinforcement learning.
Keep it in mind.
You might have to use it again if you want to go further in reinforcement learning.
Perfect.
And so now we have a few more steps to go.
The first one is of course now that we have the we wanted to update the accumulated reward or the sum
of rewards and to obtain it.
Well naturally we need to take our some of words and incremented by this last word that we've just got.
So Plus you quote the Word Perfect then almost over one last step we have to do inside this while loop.
Can you guess what it is.
It is of course the non-place.
There is number of actions play that need to be incremented by 1 because indeed we need to check that
number of actions played is lower than this episode length.
In order to do all this so we need to increment to 1 in order to check that we haven't reached the end
of the episode and therefore I'm taking the number of actions played which record called non-place and
incrementing it by one.
Perfect.
And now we can get out of the while loop and return of course the sum of the rewards because that's
exactly the purpose of this export function.
So now we have some of the words.
Don't worry about this warning here that just because we haven't defined HP yet but HP is going to be
our future object of the hyper Brenner's class.
So all good.
And now we have this explorer function that will give us a relevant measure of the word over one full
episode and for one specific direction of perturbation.
So we will be able to compare in a relevant way the different words we obtain in different directions
of perturbations and now mostly the good news is that we were ready for the major step that will make
our AI smart or should I say are able to walk on the field and I'm talking of course about the training.
The next section will implement now is going to be the training at the AA.
And so we're going to make a train function that all days and put the environment the policy because
we want to train that policy to normalizer and our hyper parameters and we will do this whole full hoop
which will basically do all the steps here from 1 to 10.
And to do that we're going to use of course the tools we made so far you know different classes that
contains method and therefore tools to implement all this including the normalization of the states.
So I can't wait to do that because that means that we're also getting closer to the final demo when
we see our final AI walk on the field.
It's going to be very exciting.
I can't wait to show you all this.
And until then enjoy AI.