Highs and welcome back to Python now we're going to start the real implementation from scratch.
Step-By-Step as usual and here we go with the first step in putting the libraries which you're going
to see are not going to be any advanced libraries like tenso flow or by torch.
We're just going to use non-pilot to build one of the most powerful eyes that you can build today.
So this is a big deal.
This contrasts between the power of the eye and the simple library which is done by Remember that sense
of flow and by torch were invented because I was not advanced enough to build some AI's you know to
build some deep learning models.
But indeed here we're not doing some deep Eremos we're not doing deep learning at all just shallow learning
if I may say is we'll still have a perception with some neurons but we'll just have one layer and therefore
we can totally use pi to build our AI.
So what we're going to import now importing the libraries will be well first before we import them by
we're going to import the O S library which is the library you need where you're going to use your operating
system and indeed we're going to use our operating system because you know in the end we will create
a folder an output folder which will contain the videos of our AI walking on the field in the Bible
that framework.
OK so operating system and then to build our AI we will just need by only this and that will be enough
to build our AI.
OK so here we go.
I just saved and now we're going to set the hyper parameters all the happy parameters that will be used
to build the AI.
So how are we going to do that.
The best way this is a classic way of setting the hyper parameters.
The best way is doing that through class I remind for those of you who are new to programming a class
is like an ensemble of instructions of something you want to build.
You know you find instructions and then what you can do is create some objects as many objects as you
want which will contain all the properties that were defined in the class.
So you first define the class and then you create the object of the class which will do later on but
the class that we're about to build right now is just a class that will contain all the hype parameters
of our AI.
I remind that the hyper parameter is a parameter that is supposed to have a fixed value.
No it won't be any variable of some function or any other kind of variable but that doesn't mean that
you can try some other values.
But in the training of the eye the value is fixed.
That's what you need to understand.
You know the hyper parameter during the building and the training of the eye is fixed.
But then if you want to try some other trainings Well you can try these trainings with some other values
of the hyper parameters and therefore what we're going to define right now through this class is the
list of all these parameters that will have a fixed value during the training and which won't be modified
unless you want to try some other trainings.
All right so let's do this.
So the final lesson Python you simply need to start with class and then you add here the name of your
class the name of the class and Python usually starts with a capital letter.
And since we're just a new class that will give the list of all the hyper parameters of the AI.
Well I'm going to call it h p as hyper parameters and then you need to add some parenthesis because
in a class you can specify some variables or even some inheritance tools but we won't use any variables
here simply because we just want to define some hyper parameters that will be defined through the first
method of the class.
You know it is always the same method you start with when building a class which is the in it method
separated by two double underscores and this method will take one argument which is self.
So self is a mystery for much people when they start programming but that is only used to refer to the
object that will be created from the class.
We also call it an instance of the class you know the future objects will create are called instances
of the class and self here is to specify that when you're going to use a variable that belongs to the
object then you will specify that with self in order to specify that the variable indeed belongs to
the object.
OK so you're going to understand now what I'm going to do then because I'm going to define all the variables
of the object and to define them to specify that these are variables of the object.
I will indeed use the self here to refer to the object.
So what is to be the first hyper parameter.
What is going to be the first parameter that we're going to use and is going to be fixed in all the
training.
Well that's going to be the number of steps.
So here I'm just choosing a name for the number of steps and what is the number of steps.
That's basically the number of training loops we're going to have in the end or in other words that
the number of times we're going to update our model.
Know this perceptual one layer of several neurons which has a policy ticking hasn't put the state of
the environment and returning it outputs the actions to pray in order to walk properly and which value
are we going to choose where we're going to choose 1000.
We will get some good results with that then next parameter still yourself because this is still a parameter
of the future instances of the HP class.
It's going to be the episode length the episode length which is what.
Which is the maximum length of an episode.
So the length here is just the maximum length of an episode meaning the maximum time will I walk on
the field and the value we're going to choose for that.
Still after experimentation is same one doesn't feel free to test any other values then we're going
to define you Paramo it is going to be self taught learning rate.
So learning rates is an inevitable parameter that you will always have whether you're doing some machine
learning deep learning or AI learning where it is always here and that's just to control how fast your
AI is learning.
And usually you want to start with not too small but not too large a learning rate and a good value.
Here is 0.00 to again feel free to try some other values.
Then another hyper parameter that is really important and really you know at the heart of the augmented
random search algorithm is the number of directions the number of directions which is the number of
perturbations we'll apply on each of the weights.
Remember that we're testing a certain number of directions and also they're opposite directions to figure
out which direction increases the most reward.
And this number of directions is a fixed hyper parameter and the value we're going to choose here actually
started with 8.
But I realized that I had better results with 16 and actually I will even try with some more directions
because indeed the more directions you test the better chance you have to increase the reward.
But of course the more directions you test and the longer it will take the training.
So I will test it.
But right now let's just start with 16.
Then remember in the article in the paper they want to consider separately the number of directions
and the number of best directions you know the directions are best that are increasing the reward the
most.
You want to keep them separately because you're going to reduce them and therefore separately we're
going to create the number of best directions and we're going to start with 16 as well which means that
we're going to test all the direction so far.
But later on who will choose a number that is lower than the total number of directions.
And speaking of being lower.
That's what we'll need to assert right now.
That is we need to make sure that the number of best directions is lower than the number of directions.
Right that makes sense.
You want to keep your top directions among all your directions and in order to assert something in Python.
The trick to do that is add here assert self doubt and the best directions lower than the number of
directions there you go with a certain number of best directions is always lower than the number of
directions.
Perfect then three more variables.
The next one is the noise which is going to be the sigma in the Gaussian distribution will use to sample
the perturbations because actually these perturbations are going to be sampled following a Gaussian
distribution and you know in a Galchen distribution you have the standard deviation and this noise is
actually this standard deviation Sigma and we're going to set it equal to both point three to not have
a large variance.
OK then the next hyper parameter will be the seed which is just to fix the current configuration of
the environment.
So that's basically to fix the parameters of the environment.
We can just choose any value here just so that we have the same result.
You know if you want to get the same result as the ones I will show you in the end.
In addition when we test our AI on several environments Well it's normal to observe the same thing.
So let's just pick one that's the seed.
And then finally one has variable which is of course the environment.
So I'm going to give the following name to that variable.
It's basically the environment will connect to our AI.
It's just the name of the environment so we can call it and name and name and so forth we're just going
to input some quotes and later on when we choose our environment to play with we will give the name
of this environment inside the quotes.
All right.
And that's it.
These are basically the hyper parameters that will be used in the AI.
You can actually add some more.
For example you can add a dk.
You know when you use the learning red you can add a DK hyper parameter to reduce the learning rate
of the epochs in the training.
But this will be enough.
With this group of hyper Renner's we will get some great results.
All right.
So that was the first step and each step in the next step is going to get slightly more challenging.
We will implement the first important feature of the paper which is about normalizing the States so
we will follow exactly what is in the paper.
And I remind that when you do this for performance purposes.
So let's do this in the next tutorial.
I'll see you there.
And until then enjoy a.