Hello and welcome to this new tutorial.
Now we are one or two tutorials away from getting the final results.
Well we'll when needed first toile to indeed run the main code as you see here.
And you know create all the different objects get our policy allwell initialized with all its weights
equal to zero.
You know at the beginning and then applying other updates things to the chain function that indeed has
integrated all the tools that we've developed in this implementation.
And that's the finish line of the journey.
Then we'll have nothing to do the training will execute and we will see the words of each training loop
populating here in the council and we'll see mostly if the word is Will increasing over the training.
I remind that there are going to be one thousand training loops 1000 number of steps or you know 1000
updates of the policy.
All right so let's do this.
As you can notice I as I tell you in the previous the toile included this code here that you know the
classic code you can find on the Internet because indeed and Kadeer means make directory.
And there is even a we could be page 4 and Cadia because it is a Linux tool to indeed make the directory.
And you know since this has nothing to do with AI I prefer to do it here.
You can find it on the Internet.
Basically what it will do is that it will create you know when we execute the code the first folder
which will be X then a second subfolder which will be before us and another third subfolder which will
monitor and inside this monitor subfolder you will have all the videos of your AI trying to walk on
the field.
All right so let's do this let's run the main code.
So the first thing we have to do according to you what is it.
Well very naturally the first thing we integrate here is our hyper parameters object.
Our HP object of the HP class because by creating this object we'll get all these hyper parameters here
well-defined and which will be fixed during the training.
No we won't be able to change them anyhow so let's do this let's create this object and to create such
an object.
The only thing that we need to do is first give a name to that object which we're going to call simply
HP and then we call the age Reclast which differentiates things to the Keble h here.
HP glass and with some parenthesis and there you go that create an object of your hyper parameters HP
class and therefore by creating this object all these variables are defined and well initialized and
fixed during the whole training.
I know we haven't chosen an environment name so far.
We will do that at the very end with no.
We haven't executed anything yet so we still have time to choose a first environment to work on.
But we'll do that in the end.
All right so we have our hyper parameters now.
And now what is the next step.
Well the next step is actually optional but the next step could be to you know choose a seed for the
training and that's mostly for one purpose it's so that we have you know both you the student and instructor
have the same results and the same policy in the end because indeed by fixing a seed we will fix a c
with none by and therefore all the random operations that we do with non-bio such as the sampling of
the perturbations deltas here.
Well we'll get the same sampling.
You know these are random values following a normal distribution but by fixing the same C in our code
we'll get the same random values there for the same perturbations therefore the same updates of the
weight and therefore in the end the same policy.
All right so you can choose another city if you want but if you want to get same results as mine well
let's fix a seat.
And to fix this with an umpire where we take our non-Thai libraries first which has a shortcut entry
and then we take the random module because indeed the seed is applied on all the random functions we
use in our implementation.
And then from the Saranda module we take the seed function which will take us and put exactly this seed
high per parameter that we not only created but that is now existing because we created that object
which includes this hyper parameter.
OK so HPC that fixes the C will get the same result.
And the important new step is a new step is about connecting the environment to our AI.
In other words to choose the environment on which the AI AI will be trained.
So I won't hide you that I spend quite a lot of time on that because indeed we always do our best so
that you don't get any you know rounding error or installation issue.
And there are actually many ways to connect your AI to the pilot environment you have a module of pilot
that you can install.
But I noticed that this would cause some issues on the different systems.
You might be having like you know Linux Windows or Mac and it had some incompatibilities with one of
them.
So this wasn't the best solution but fortunately I found a much better solution which is direct and
which won't cause any installation issue because now it is well developed and therefore there is a way
to install it smoothly on the three systems.
It's through the gym environment.
You know we're going to connect our Bible that environment through Jane.
That's the easiest way to do.
And that will work the best for the three systems.
OK.
So let's take the solution.
Of course I will do it.
Tauriel on how to install Jim and by bullet I tested it on the three systems and no issue has encountered.
You shouldn't have any problem but of course if you have any problem we'll provide some debug solution
and I will actually make it to toile fully on that.
All right so let's do this let's get our environment through Jim.
And therefore now what we're going to do is you know since we called our hyper parameter for the environment
and its name well we're going to take that gym library and use the make function which will take us
and put this name and which will make this environment ready through the gym library and how we're going
to do this.
Well let's first say that a year and name and now what we're going to do is here next line we're going
to introduce a new variable which will be on.
And that will represent exactly the environment but made by Jim and therefore since we want it to be
made by Jim where we're going to take Jim first and then that's where we use the make function in which
we put.
Indeed our hyper parameter object and especially the alpha name farmable from this HPI object.
All right.
But then as you can notice We'll have the warning undefine named Jim.
Well that's because we have to import it.
It wasn't imported yet.
So we're going to add it here because this is where we have all the libraries that we need to import
for the implementation.
And there is nothing more simple than importing gem you just need to do the same.
As for the others import.
Right now the warning should disappear.
Here we go.
Disappeared.
OK so now we have our environment made by.
And later on we'll specify the name of the environment.
Let's do that at the very end.
So that's not over to get in Varman ready.
The next thing we have to do now that end is our environment made by Jim with the right name.
Well what we want to do now is to indeed check out the videos of our different eyes working on the fields
on our screen.
Now on our monitor and so what we're going to do now is use a wrappers.
You know we have the environment but we want to watch the trainings happen happening in that environment
and to do that we need to take the Raptors module by Jim.
So not only we're going to import Jim but from Jim we will import these rappers which will allow us
to see our AI's walking on the fields on our monitor you know inside this sub subfolder that we created
thanks to the M-K function.
OK.
So from Jim import wrappers now we have a warning saying that it isn't news but we are about to use
it now because indeed in order to basically see the videos of our AI's and or Mulliner Well we need
to say that rappers by Jim and then we take the money class which will take us and put our environment
made ready by Gen then the location where we want to have these videos or am working on the field in
our computer and this location is exactly this monitor there that basically is the directory folder
of the future videos that will be populated once we launched the training some money this year.
That's the second argument here.
Perfect.
And then we're going to use the following trick so that we can not be stopped by any warning because
you'll see that when we execute the code we'll have some warnings.
And in order to prevent those warnings from stopping the training Well we're going to add this trick.
Force equals true.
All right so we're going to force the training and videos to be populated inside this monitor directory
folder.
All right.
And that's the trick to not only get the environment of bullets but also to get the videos are trying
to walk on the field on a specific monitor directory folder.
OK.
That's the best we could use this is the simplest one.
And that's what works best for the three systems.
OK so we have now our environment.
Perfect.
And now according to you what do we need.
You know we made a lot of tools but what do we need to take exactly to launch this training.
Well that's very simple.
See the train function as we made before is second as arguments the environment the policy the normalizer
and our hyper parameters.
And so you know to launch this training of our AI.
Well the only thing to do is to get these four arguments ready.
We already made our hyper parameters ready.
That's what we did here by creating this object of the HP glass.
Then we made our environment ready through Jim.
And not only we did that but also we used a trick to populate the videos of our AI trying to walk in
a field.
So the only things remaining that we had to do now is to of course get our policy ready you know creating
the policy object of the policy class that's for sure.
Because indeed I remind that the SARS the whole principle of the IRS is that we are doing an exploration
on the space of policies and therefore you know we're starting with a policy that is perception of weight
all initialized to zero.
So that's the beginning of our exploration and then we're going to explore different policies by trying
some different updates of the weights in different directions.
And that's where the exploration in space of policies happen.
So indeed we're going to create one object for the policy but then the exploration of space of policies
happens by you know trying different updates of the weight of this policy in different directions.
All right so it's important to get that.
The next step we have to do is of course to build our policy object.
OK.
And then of course the last step and this will really be the last step before we you know execute the
train function is to get our normalizer ready as well.
All right.
So it's going to be done pretty soon.
We're almost there.
The final exciting step let's take of our policy first.
So now my next question is what do we need to create our policy.
Well let's crawl back up.
Here we go.
That's our policy class.
And now to see what we need to create a new instance of this class or you know this policy object of
this class.
Well we simply need to look at the init method and see the arguments that it needs and it needs you
know the input size which is the number of elements in the code in input state vector and the output
size which is of course the number of actions the AI has to predict at each time.
Again that's it.
That's all you need to have to create your policy and by the way for the hyper parameters Well indeed
we didn't have anything to specify in the arguments of the HP class to create our HP object.
Because indeed and in a method we are not calling any arguments but for the policy class indeed we will
need input size and the output size so let's get them right now.
It's quite easy.
You know thanks to what we did with making the environment ready by Jim and then using these wrappers
to get not only the videos but also all the tools that we have in the public library.
Basically what we did here is a shortcut.
So let's get first the number of inputs which I remind is the number of elements in your input state
vector.
And so to get it it's quite easy.
You know it's just how the public library works you take your environment first then you take the observation
space which is something that exists already you know it's part of the whole Bible library observation
space the space of your observations and therefore the space of your input observation space and then
to get the shape attribute because indeed that's what will get you by taking the next 0 the number of
inputs right and number of input is contained in that specific indexed location.
Then we're going to do the same for the number of outputs which is of course the number of actions that
the is predicting to play at each time t.
And to get them again we take our environment than that and then we're going to take well here the action
space exactly that.
And then again shape in square brackets zero basically zero returns the number of the space you're looking
at.
OK.
So here the action space in here the space of inputs.
Perfect.
Great.
So we have a number of inputs and the number of outputs now and therefore we are ready to create our
policy class because indeed in method it is calling for the input size and the output size.
So how are we going to do this.
Well again we have to give a name to that object which we're going to called policy and then we take
our policy class this time with capital P and then our two arguments are required as we can see here
and the inputs first for the input side.
You know we just gave some different names for the arguments.
That's quite safer to do.
And then the number of output for of course the output size here we go we have our AI ready and initialized
as a perception of weights equal to zero.
And then when we run the train function now this function here.
Well you know after initializing the perturbations in the different directions Well we get you know
the positive words and negative rewards and then we sort them by the maximum of the couple of words.
And then here at this specific time of the implementation or the training.
Well that's where the policy will be updated.
And therefore that's where the way it will be updated in different directions to explore the ways that
will lead to the highest rewards.
All right so you have to understand now we're just you know taking a step back to understand how all
this will work you know how all this will happen.
OK.
So here we go.
Our place is ready.
And now let's go back to the train function the train function takes with the environment which we have
the policy which we have the hyper parameters which we also have.
And also the normalizer which we don't have yet.
That's the only remaining thing we need.
And that's what we're going to get right now.
So let's get our normalizer which is going to be an object or you know an instance of this normalizer
class.
So we're going to call this class.
And now to know which arguments we will need as input at this class will we just need to look at init
method and the init method only takes one argument which is a number of inputs and which we already
have.
So perfect.
Let's create our normalizer object and by calling this normalize a class and by taking input of this
normalized class the number of inputs.
Perfect.
Here we go.
We have everything ready to use the train function because indeed.
Now to get this training ready the only thing we have to do is just to take the train function and input
the four environments it is expecting.
So first the environment then second our policy then third our normalizer.
And finally our hyper parameters and there we go.
Congratulations.
This implementation is over.
So just getting the final good ready.
Here we go 140 lines of code not bad you know especially for a non-pilot implementation and you're going
to see that you know this contrast between the simplicity of this implementation you know in 140 lines
that's not much and done results will get me and is mind blowing so I 20:18 get ready for it.