Hello and welcome to the beginning of this journey this new journey in which we're going to build one
of the most powerful eyes this new artificial intelligence that was just released in 2018 late March
2010.
And so in this little I want to take the opportunity to show you the environment.
We'll be working on as well as the research paper that contains the whole theory of this new AI.
And because we're going to go back and forth from our code to the research paper and vice versa because
really I want to show you how to understand a research paper and translate it into code so it's important
that we already start to get familiar with it so that it doesn't get too overwhelming when we start
digging into it.
All right.
But first let me show you what we're going to do you know to build AI on which kind of environment we're
going to train it.
And before I show you exactly the environment we'll be working on.
Let me show you something that is very exciting and that made a huge buzz in the AI community.
What I want to show you is the Google deep mind humanum it.
Here we go.
I already have the research gesture and the video I want to show you is exactly this one Google deep
mine.
You can see I'm walking on sunshield and this is taking has put the states on the arm which were not
the images but something called vectors during what's happening in the environment.
So you know these are all the angles between the axis angles of rotations the coordinates of the eye.
And this is predicting the actions to play in order to walk on these kind of fields and you have to
understand that the action to play are not only one action but several actions like you know the muscles
in portions of the human are its robots or these other arms or cheetah robots.
So you have to understand that what you will build is actually a function that will take us into the
states environment which will be vector isn't including what's happening in the environment and which
will be returning the actions to play which are the muscles in the muscles and bosons in order to walk
on these fields.
So this is very exciting.
This is the Google environment.
But the bad news is that this environment is not on purpose.
However I would like to see something that is at least as exciting as this.
So what we're going to do is work on some similar environment where we are going to train and to work
on some fields not with these amazing graphics but still the graphics will be awesome.
And this is what I'm going to show you right now on the environment that we'll be working on.
So here we are back on Google.
And before I show to you this environment I just want to say that today if you want to build an AI to
train you to walk on a field or run across a field you have two options.
The first option is Miyoko which you can find on opening Jim and also on the team control suite by deep
mind.
And the second one is pilot.
Now the bad news is that Djoko is not open source is not open source you have to buy a license to get
environments.
You can actually get a free trial for one month but you know I want you to have fun as much as you want
with this course and with this AAA we're going to build.
So we are not going to go for Mukoko if you really want to go from a Djoko you can get a free trial
or buy a license or even if you are a student.
Well you can get the license for free but I know that not all of your students in this course.
So we're not going to go over this one.
The other one we're going to go for and which is by the way even better in terms of graphics and mechanics
is pilots.
And let me show you this amazing environment which is the one we're going to use for this course.
All right.
Bible it Bible it is an easy to use by the Munjal for physics simulation robotics and deep reinforcement
learning.
It was built by Irwin Cummins who I often discuss with sometimes about AI and robotics and mostly about
people because I actually contributed to the development of pilot.
I made a request to have recently to improve on the features so Bible that you have to have paid Schir
of Bible it which is originally from bullet 3 where you can find some more details if you scroll down.
All right you have all the details about the bullet physics as the case for those of you who are into
mechanics.
You can check it out more closely but what I really want to show you right now is some examples of pilot
environments.
So if we click on videos here we can see tons of examples.
These are different bible environments.
Let's have a look at this one for example.
The mini tour this is the mini tour and this is an agent you can train to move forward with some a build
so it can be an AI based on deep reinforcement learning like you know deep learning or the A-3 see this
is not what we're gonna do for this course.
We're going to build the latest and very powerful AI that was released in 2018 March 2018 ers.
But you can train it with several miles.
Let's have a look.
And clicking on it and here is the minute we're moving forward.
And it was trained by an AI model.
So that's pretty cool.
This is not the environment we'll be playing with.
We will actually be playing with the half cheetah environment for two reasons.
I personally think this is more fun to train a cheetah on how to run than a mini tour.
And I hesitated between the hatchet and the humanoid but the thing is the humanoid is the most challenging
environment to train and I had to walk and it actually takes several weeks on a normal computer to train
the human.
And you know I want everyone in this course to be able to train the AI we're going to be on this course
and then how she dies the best compromise between something challenging to train and the training time.
No we won't have to wait for days or weeks to train our health CIDA and you're going to see the areas
so powerful that will only have to wait for a few minutes.
This is insane.
I can't wait to show this to you.
All right.
So now that we have the environment let me show you again the research paper I'm sure will show that
to you.
But let me show it to you again because we really need to get familiar with it because we're going to
go back and forth from the code to the research paper and vice versa.
So I just want to show it to you once again and especially show you exactly what we'll be implementing
in the course you know which version of the Airbus algorithm will be implementing because indeed the
research paper suggests several versions and no worries.
We will be implementing the most powerful one.
So let's go to the research paper it's called Mantid random search provides a competitive approach to
reinforcement learning.
Actually this is the research paper.
Let's click on it.
Here we go.
And then PTF here and here we are in the research paper.
Simple random search provides a competitive approach to reinforcement learning.
In his paper written by Horia Manua earlier guy and Benjamin Rashed vision in Russia is very famous
in the AI community.
One of the best and so all of this was invented in the University of California Berkeley one of the
top universities in the world.
And as you can see March 20 2018.
So just very recently at the time I'm speaking.
And so what I want to show you exactly is the algorithm we'll be implementing in the course which is
on page 6 if I remember correctly.
Here we go.
Yeah.
This is the algorithm we'll be implementing exactly as it is so you know you will really learn how to
follow a research paper and translate what's going on here in a code.
We will be implementing exactly the same thing actually.
They suggest two versions of the algorithm.
One which is without any normalization of the states and we do which is with normalization of states
because indeed they say that we can normalize the states to improve the performance and we can also
scale the standard deviation to improve the performance as well which we will also be doing the standard
deviation of by the way that we want.
So basically we will implement the best version of this paper which is V-2 with the normalization of
the states and the normalization of the world.
And so we'll be making some classes and functions to implement this.
The best way you know how a real AI developer would do it.
So you will learn how to think logically like an AI developer and work on some pretty advanced research
paper but I promise you you will understand everything because we will really go into the details of
this.
And now I would like to say just a few words on how this works because you know we could explain it
in a few words.
So basically the first important thing to understand is that your AI you have to see it as a function
which is called a policy taking as with the states of the environment which are inputs vectors encoding
exactly what's happening at each time in the environment.
So they are like the angles of the axis of Eurobonds the angles of rotation the coordinates of the points
of your robot and more you know enough values to describe exactly what's happening environment so that
we could almost draw a picture.
All right.
So this is a function taking this as input and returning as output the actions to play the actions to
play in order to walk.
But then the second very important thing to understand is that the output is not only one action but
several It's you know a group of actions and that makes sense.
That's because in order to work on some field you not only have to move one leg you have to move all
the parts of your body in more precisely.
These actions will be all the muscles and portions you can have within these parts of the body.
So this is probably very different from what you've done before you know because in the previous age
you might have built.
Generally you return one action and mostly one discrete action here.
It's not only one action it's a group of action but also this is a group of continuous actions because
the muscle and portions are measured by some continuous metrics in order to make it even closer to reality.
So you know we are really going to build an AI AI for some very realistic goal and I remind that you
know it's fun to train any eye to work on some field but this is not only for fun purposes but also
it is a benchmark for your AI because if you manage to train your AI to walk on some field Well you
can adapt this AI to other problems even business problems by just changing the inputs of the environment
know which will still be some including vectors and by of course adapting the outputs the actions to
play and the reward strategy.
So by only a few tweaks you could transpose this AI that we're going to build and train into another
kind of problem.
Even business problems.
Now how does that work well.
So your AI is a policy that is a function to Gazen put the states of the environment and returning a
group of continuous actions which are the muscles in potions.
And how does it work well between those inputs and those outputs you're going to have a perception which
is actually a neural network of one layer composed of several neurons.
Each of these new ones will have a wait and we will do an exploration on the policy meaning we will
explore lots and lots of updates on these weights by just adding you know some little values here that
will follow a normal distribution to see which update of the weights will increase.
Do we want.
And so basically what we'll do is that will test several directions.
Each direction corresponding to one little small value will add to the matrix of weights.
This is a matrix of weight and for each of these directions will also do the same update you know with
the same value but with the opposite direction.
You know that's why we have a plus an A minus here that correspond to one specific direction but both
For one way and the opposite way.
And why do we need to do this you know with one way and the opposite way.
That's because in this step 7 here we will do what is called an approximated grade in the sense meaning
that we will take the word that we'll get by playing this perturbation here in the plus direction the
positive direction and we will subtract this we to the word we get by applying the same perturbation
with the same value but with the opposite direction.
And that will be in order to compute the finished difference of reward with respect to this small update
of the weight which is the smart perturbation value we add to each of the weights.
And indeed this is an approximation of the gradient because this is an approximation of the partial
derivative of the word with respect to the weights because indeed you are trying to optimize the word
with respect to the weights.
All right and then you're going to add this approximation of the gradient of the word with respect to
the weight to your current matrix of weight to have to wait in the best directions that will increase
the most the reward.
Great.
So I can't wait to start.
But before we start I want to show you one last thing which is an implementation by some developer we
know personally and who was very happy to share his code for the course.
His name is Alex D-Jack.
He is a very talented AI developer who made this amazing Python implementation for this same algorithm
recommended random search with these X and paper and he obtained amazing results.
And for those of you who don't know Alex Jack he is a highly skilled Ph.D. student in one of the best
universities EPF fail in Switzerland who has made some great contributions to the community and not
only that he is in the prestigious Facebook that AI page.
Here we go.
That's a prestigious page you know all the people of this page are very talented in the AI community
and great contributors.
And indeed if we scroll down we will find Alex Jack right here with its amazing tutorial on neural transfer
with by touch.
So I would like to say a huge thank you to Alex Jack for helping us with this course and sharing his
amazing implementation of the areas which will get inspired from we won't have exactly the same code
but still will use the same type of parameters and the same logic.
All right.
So I can't wait to start.
Let's start from the next tutorial.
And until then enjoy AI.