Hello and welcome back to the course on augmented random search engine today's tutorial we're going
to compare basic versus augmented random search.
So we're going to find out why is augmented random search called augmented random search.
What is this word augmented in the name stands for.
So let's have a look.
We're going to look at three main updates that were done in Erris to make it augmented.
The first one is scale update step by the standard deviation of rewards.
Second one was online normalization of states.
And third one is discarding directions that yield the lowest rewards.
So the first one which is about scaling up data step by step division rewards is as simple as taking
this calculation which we discussed in the previous tutorial and dividing it by the standard deviation
of the rewards involved in this in this calculation.
Now it's not as important for the purposes of our intuition.
There's a technical reason behind this and what we're not going to dive into it.
You can find more information on this in part three point one.
All of the research paper which will reference at the end of that tutorials same research paper that
we referenced in the previous article the main research paper.
So this is not as important for us as the other one.
So let's move on from here.
But just something important something to keep in mind and something to have a look in the paper or
to look out for in the practical tutorial.
The second update is online normalization of states and online basically means real time normalization
of states as the AI is learning as the AI is going through these environments.
The states are normalized and what does that what is their emotional states mean.
It means normalizing these values that are inputs over here and not just normalizing them based on what
they are normalizing them on based on what values we've already seen so that everything is treated in
a similar way.
And so they talk about this in more detail in their research paper and this will be one of the first
things that will code I'd love.
I'll just give a quick overview of how to ensure we think about this.
And this is the same example as they given the research paper as well.
So imagine that these input states for one case are between ranging between 90 or 100 so these values
are between somewhere like between 80 and 100 like 91 92 and maybe 99.
And another case there between minus 1 and 1.
Now the thing is that the weights the fact that they're going to have like changing the way you remember
how we learn we learn by being the weights while changing the way slightly by like 0.1.
In the case when this value is 100 is going to yield a much more drastic change in output as opposed
to when this value is something like 1 or zero point five or something like that.
So basically even though like it's the environment might be so the environment might be described with
different ranging values which is fair environment might be changing might be different it might be
some you know different forces happening different types of terrain or things like that.
And because because of that because these values might be in different ranges what can happen is that
slight perturbation of weights might have different likely different magnitude or impacts on the output
values even though they is the only reason for that is that the is in a different environment so we
want to minimize that effect.
And we want to have cheap perturbations in the way it's to be kind of like fair the perturbations to
be fairly treated across the whole training process.
And that's why we're going to normalize them but not just normalize them because a set of weights I
think is eventually going to normalize them online which is also called widening of states meaning we're
going to take the values that we've already seen for these states and we're going to normalize these
values alongside what we've already seen.
And that way the weights won't be affected as drastically by the different input the different ranges
of these in propellors.
So that was one of the biggest effects if you read the blog post that was in reference reading at the
start of this section you would have noticed that the supervisor of the
Horia money and really a guy he actually mentioned that this was one of the biggest changes that helped
us tackle challenges such as humanoid walking.
All right so that's important to understand.
And again you'll see a bit more of that in the whole practical side of things.
And in the research paper and finally the third update was discarding directions that yield lowest rewards.
Quite a bold move.
And it turned out to be a good bet for the authors.
So here we've got our four or eight actually different perturbations.
So for positive or negative the ones that we talked about the to tutorial and here we've got the results
how they went through and we've got the rewards.
And basically what they like as you recall previously we would use all of them but now they're saying
all right we're only going to take the top k top k of these rewards the top k of these perturbations
are actually going to work with them when are we going to discard the the ones that weren't part of
the top.
OK so we're going to take this one we're going to take this one and we're going to throw away these
these two or these four perturbations we're not actually going to include them in our calculation and
that changes the formula.
We discussed over here.
It's actually not going to have these include basically the weights are going to evolve in the direction
of only the most successful results that we saw.
And yeah and so that was also a significant update that helped this method and it's quite quite an intuitive
one as well.
Why.
Why not take the the highest pathé the directions or revolution that have the highest potential of all.
Just in that direction and discard the ones that have the lowest part of sounds a bit like natural selection.
And now those are the three main differences if you'd like to get a bit more details of course you'll
have them in all of these things in the practical terms.
But if you'd like to read a bit more about them and understand them not just sort of intuitive Bay bases
but like more philosophically or more mathematically then you once again would like to refer you to
the research paper by her money and really a guy on an argument and random searches same research paper
reference the embryos that are.
But just in case you skipped that tural because it was quite advanced then this is a good opportunity
to get back to that research paper and have a look at some of these things.
And on that note I look forward to seeing that next of Tauriel.
And until then enjoy AI.