Hello and welcome to this new tutorial.
So in the Bruce Doyle we made this evaluate method that returns the output when we feed the perception
with a certain input and with three possible options which are first no perturbation is applied when
Direction is known.
Second when a positive perturbation is applied.
And third when a negative perturbation is applied and negative I remind it means it's the opposite.
It's a perturbation in the opposite direction as this one the positive one.
So that's a good thing done.
But the only thing missing in all this is that the doubters are not simple yet anywhere in the code
so we just need to make this extra method here too simple to Deltas and then in the arguments here we'll
put the perturbations that were sampled thanks to this additional method that we're about to make.
So let's make it deaf.
And as I said in the peruses oil we're going to call it simple deltas or you can call it simple perturbations
but the Deltas are the perturbations of course so simple they are to us and it's not going to take any
argument except self of course because simply it just consists of returning some small values following
a normal distribution.
That is a Gaussian distribution of mean zero and variance one.
So indeed we don't need any argument here to do that.
We'll just return it directly by using the rand and function from the library.
And since it's actually direct to use this function well we can just start here with a return and this
will be the only line of the method will return exactly what we want that is these simpled perturbations
deltas.
OK.
So as I said we can do this through the non-pay library and the non-pay library is given by the shortcut
and then from this non-pilot rii we're going to take the random module because it's a random module
that contains the rand n function that is returning these simpled small values following a normal distribution.
You know the end here is for normal and Rande it's because it's returning some random values.
So normal distribution.
But now it's important to understand that we're not only going to return one small value we're going
to return a matrix of small values following a random distribution.
And why is that.
That's simply because we are adding this delta here multiplied by the little noise to our matrix of
weight theta.
And therefore this delta here must be a matrix of course.
So basically we're adding those little very small values to each of the weights of the matrix of weight
theta.
And that's why here in the random function we have to specify the dimensions of the matrix theta.
It must have the exact same dimensions of the matrix theta because we are adding those small values
to each of the values of the matrix theta.
And therefore here we need to specify the dimensions of this matrix of small values we want to add to
the matrix of weight theta.
And there is a trick in Python to do that quickly.
It is by adding a star here.
Then we take our matrix of weight self the theta and then we just add shape and this will return as
the dimensions of the matrix of weight theta which is exactly what we need.
Otherwise the other way was to take some of that theta that shape and then in square brackets 0 that
would give us the first time mention of the matrix of where theta and then come up and then sell that
theta that shape and then square brackets 1 which would give us the second dimension of the matrix of
what theta.
So you know we would have a couple with these two elements but it's much quicker it's actually a little
trick.
That is only compatible with Python 3 2 at this door here to specify that we want the two dimensions
of the shape of theta.
All right so good to know.
So that gives us the dimensions and therefore that creates a matrix of the exact same time emotions
as theta.
So we can add this matrix to theta and this matrix will contain some small values following a normal
distribution.
So that's a good first thing done.
But then that's not all that's not only what we want to return.
We not only want to return a matrix of small values following a normal distribution we want to return
16 matrices of these small values why 16 is because we are playing those perturbations for 16 different
directions.
Remember this and the directions hyper parameter here that is equal to 16.
That means that when we apply a perturbation in some direction.
Well we are going to do that for 16 directions and 16 opposite directions so therefore in total 32 directions
16 positive directions and 16 negative directions.
So we want to return these matrices as a list of matrices.
And that's why I'm adding some square brackets here surrounding these matrices that we're creating and
in order to get 16 matrices we just need to add a for loop with any you know boy will I can just add
I or even an underscore in the range of the total number of directions and I as you can notice the HP
because the number of directions is a hyper parameter of our future HP objects that will create at the
end of the implementation.
All right so basically by just adding this full hoop here for the total number of directions I'm going
to create 16 matrices of small random values following a normal distribution that is a Gaussian distribution
of mean zero and variance 1.
And then things to this noise here.
Well actually add to the matrix of weights.
Theta are not small values following normal distribution but small values following a Gaussian distribution
of mean zero and of variance or standard deviation 0.03 because remember that we initialized the noise
to 0.03.
So that's how we'll add the perturbations in a positive direction and the opposite negative direction.
Right now we have our sampled values and so we're ready to move on to the final method of this policy
class which is basically the next step of the paper that is make the update step.
Well actually there is step 6 first.
So the directions us or perturbations by the max or do we ward in the positive direction and the reward
in the negative direction.
But no worries we will do that.
And actually the training itself will implement that feature here in the training we don't have to implement
it right now in the evaluate method.
However we have to implement this make the update which is actually one step of great in the sense because
indeed we are updating the matrix of weight in the direction of the perturbation here.
Right.
The gradient is approximated by taking the differences of the words into two couple of positive and
opposite directions and multiplied by the perturbation.
That's one step of gradient descent and that's the update step that we have to implement now and we'll
do that in the next hour tomorrow.
Until then enjoy AI.