Hello and welcome back to the course on augmented a random search in today's tutorial we're going to
be comparing Erris versus other AI algorithms.
And the reason why we included this tutorial in the course is to highlight the main differences between
areas and other standard or more conventional AI algorithms.
It's pretty hard to say conventional because all of this is cutting edge technology and cutting edge
models and algorithms but nevertheless we want to contrast areas which is brand new and how is it different
to other Algor algorithms that exist out there that we've seen before.
And hopefully this will give you a good overview of how the contrast and how you'll be more prepared
for these conversations.
Or know what the advantages of areas are with a difference.
So let's have a look on the left you can have a column for features or areas that are going to be three
main distinctions between the two.
There's the full list but those are the ones that we're going to cover.
And those are the ones we find the most important ones.
And on the road we're going to have other area.
So number one is exploration Eris performs the exploration in the policy space weather whereas other
AI usually performs exploration in action space.
So let's have a look at what this actually means in Eris.
We've already discussed that.
We've got this Perceptor on and then we wait until the agent gets to the end of the episode and then
after based on the result whether they successfully got to the end and one or they fell over at some
point we get a reward which we calque which we use to adjust the weights.
Well the thing is environment's actually are set up.
For instance this merger go environment is set up independent of the algorithm that's going to be applied.
So there's this environment and you can apply arrest to it you can apply a different AI or any other
algorithm to it.
So and the thing is that the environment is set up to provide it provide rewards not just at the end
of the episode but after every single action.
Just that's just how the environment works.
Every time the agent does something the environment provides a reward.
For instance if it's getting closer to the target then it's getting a positive reward if it's falling
over or if it's getting further than the target might be getting a negative reward.
In any case there's a reward that's provided after every single action.
What Erez does though is it doesn't take advantage of the opportunity to look at the reward after every
single action arrest accumulates the total reward and then looks at the total reward after the episode.
That's important to understand and remember that it's kind of like not using it could be looking at
the reward after every action but it's not.
So it's actually not using the all of the possibilities that the environment provides and yet is still
stronger than other AI's surprising.
And this is kind in contrast to how other AI usually works and the way they work is the Jesus lives
from artificial intelligence.
It is of course if you've been part of that course then you'll remember them.
Normally AI performs actions and gets rewards so it goes into a new state and gets a reward right away
so that action gets a reward performs an action gets reward and it analyzes the reward.
In contrast to Erris normally the AI reinforcement learning and keep learning things like that.
They analyze the reward right away as soon as the action is performed they get the reward.
They analyze the reward and what this allows them to do is apply the Belman equation and build a value
function for states.
So then they know the values of different states of being or of being in different states and therefore
after that they can use those value functions in order to make their decisions and create the maps are
they going to get through these environments.
So that's the main difference in Erris we focus on the total reward for the whole episode.
And basically we're analyzing we're exploring therefore we're not exploring every single action.
We're exploring the whole policy.
So the whole approach of how to go through an episode.
So we're and we're assessing the whole episode the whole episode and therefore it's we're exploring
in the policy state space whereas in other AI Normally we're exploring actions and therefore we're exploring
in action space.
So that's what we have exploration in the policy space.
We know from start to finish for the whole episode versus exploration action space where we explore
every individual action and that's the difference in a way.
If you have an artificial intelligence that is of course you remember that we had a tutorial on eligibility
trace in in there for AI and that's exactly that's kind of similar That's what IRS does is kind of similar
to disability trace we're in eligibility trace.
We didn't look at every single action but we matched up the actions that we did had five or 10 or 15
actions in a row and then we only looked at the reward and we remember from there that that actually
improved the performance of other.
We won't go into detail on this but just just a hint that if you are aware of eligibility trace arrest
is kind of like similar thinking in the similar action as eligibility trait.
All right.
Next step two is what.
How is that updating of weights performed in as it's the method of finite differences in other AI it's
Grady and the famous or famous gradient descent algorithm.
So we talked about the matter of finite differences or here where we basically have a small point a
positive shift a negative shift or a positive delta negative delta we have the rewards and based on
that we can calculate how to adjust our weights.
Overall the difference is that in other AI AI we don't use Mithal like we actually use gradient Paul's
descent which is proper differentiation.
All of the loss that you're getting.
So in Erris you get a reward in not normally in a you get you get a loss.
So it's a proper differentiation of the last based on the weights and going backwards through through
the neck.
So that is what we do normally I approach skol also known as a back propagation of the error through
the network to update the weights.
The reason why we can do that in Erris why we have to do the method of finite differences is because
we simply don't have a value function.
We just don't because we're not exploring an actual space we don't have that value function that we
can then we're going to apply this method to and therefore we have to deal with what we have.
And there that's why the method of finite differences is used.
All right.
And it works totally fine and works really great.
In fact in that main research paper for Erris they talk about that this is a good enough approximation
of the gradient.
All right.
And number three is shallow learning.
In the case of arrests in the case of other AI it's usually deep learning.
So if we have a look as we recall this was the percept drawn for a dress it might have a lot of inputs.
I have a lot of outputs but the main point is that there's just one the input layer which is Connect
the right way to the output there they're an AI that is based on deep learning usually have a at least
a hidden layer like this at least one hidden layer where these inputs are entered combined and then
they're going to the output.
In fact more often than not you will have multiple hidden layers and therefore it is actually proper
deep learning that is used because as you can see the neural network is very deep in this case.
And so that's the difference.
And obviously that takes more attention more training as more weights that need to adjusted and so on
and so there we go.
We've got arrest exploration the poses space murtherer of finite differences in shallow learning.
Either way I is explosion in action space.
Gradient descent algorithm and back propagation and three usually deportee.
And so what does this sum up to.
Well even no arrests in many ways is kind of like less involved then other AI as we just discussed.
As a result Erris is still much stronger.
It's up to 15 times faster and at the same time yields higher rewards on specific applications.
And I put in specific applications here because airiest has been a critique that is like it finds real
inefficiencies for instance or
inaccuracies in the magical algorithm and it like exploits them and so therefore somebody someone might
argue that deep AI is based on deep learning even though it's slower might be more versatile and be
more applicable and smart in different ways.
But at the end of the day if you have a specific application and you need to get it sorted out and you
need to solve the problem why does it matter if you know what what's how you get to the end result like
you know as long as you are getting the result that you want then and you know the problem is solved
then you have a good outcome and so therefore a arrest can be a great solution in many different applications
just something to keep in mind that you might not be as versatile but that is still yet to discover.
You need to discover that all the different specific application that you're using.
Overall it's been showing some fantastic results.
And on these benchmark tests such as Mujer coaches beating other artificial intelligence algorithms
that exist out there by it's 15 times faster and still gets higher rewards and diminish up today's tutorial
we've got some additional reading.
This paper is called the Bush strategies as a scalable alternative to reinforcement learning by Tim
Salomon's and others published via open AI in 2017.
The link is on the course notes and papers are archived.
So basically this paper talks about ebullitions strategies which is another term for random search.
It's exactly the same thing just different terminology.
And this paper came before the Erris paper and there they kind of compared evolution strategies or random
search to other reinforcement learning algorithms are the artificial intelligence.
So it's in line with the topic of today's tutorial.
If you'd like to have a look and this paper is also referenced in the press paper that we touched on
before.
On that note I hope you enjoyed today's tutorial.
I was forcing him back here next time.
And until then enjoy AI.