This new AI that is able to look at a bunch of unstructured motion data, like this, then place a character in a video  game and see all the amazing things that it can learn from it

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.

Today we are going to have a look at this new AI that is able to look at a bunch of unstructured motion data, like this, then place a character in a video  game and see all the amazing things
that it can learn from it. Walking,  running, dancing, you name it. And, look at that! Yes, this works for bipeds, and  quadrupeds at the same time. Now that is a quite a challenge, so let’s see what is going on here.

First, why is this a challenging problem?
Why do we not just copy the movements from

the training data? These were recorded
by real humans and dogs after all. Well,

that will not cut it here. You see, in video
games, we get to control these characters,

which means that we can stop any movement
at any time, and start a new one. Aha!

So this requires neural networks to look at
a big soup of motion data. And how big is

this soup? I will tell you in a moment,
and I think you will be very surprised.

And wait, even learning the essence of
these movements is not enough, it needs to

learn about transitions too by itself. Oh yes,
transitions are key. Why? Well, look at this.

Here is a previous method, and whenever we
change direction, look. These transitions are

quite unnatural, and this is not parkour, we
are still talking about just running around,

a much simpler task to animate. And it
is still not working too well. Not good.

And oh my! Are you seeing what I
am seeing? What is that? Oh yes,

that’s right. This is foot sliding. The bane
of our existence. More on that in a moment.

But now, let’s see how the new
method solves this problem.

Wow. Now that is fantastic! So much better.

And, what about the doggies? Well,
previous methods are not too bad here,

but the new one is so much more fluid.
The movement of the body is now better,

but if you don’t find that too noticeable, also
check movement of the tail too. Previous methods

seem a lot more tentative, while
the new one is much more lifelike.

This new technique can also perform
more advanced actions, I loved how

it performs the dribbling here. And note that
the ball is controlled by the physics engine,

so the character has to react to the
ball’s movement quickly and convincingly.

And, the dance moves it has are really
cool too. And it can not only dance,

but it can dance so much more convincingly
than previous methods. Loving it!

But, if we wish to get even more crazy,

we can even combine different movements for
the lower and upper body, and I know from

previous papers that we can’t just copy
paste two motions together to get this,

these are so much more challenging. And the
new method does it with flying colors. So good!

One of the many key insights in the paper
is that they propose using this diagram,

which is the phase space. And as we pick a point
and move it around there, our character starts

moving. However, not all spaces are this
intuitive. Look, with previous techniques,

good movements required these crazy paths
and were thus, more difficult to achieve.

But wait, we have two more incredible insights
about this paper. Now one, hold on to your

papers because this is where I fell off the
chair when reading this paper. As promised,

let’s have a look at how much training data this
technique required to learn all these beautiful,

fluid movements. What? Are you seeing
what I am seeing? That is impossible,

right? Look. It learned to animate quadrupeds
by just seeing 17 minutes of footage,

and perhaps even better, dancing, just
9 minutes. That is absolutely amazing.

And, the second insight is about foot sliding.
This new technique shows less foot sliding in

most cases than previous methods, and even
in the worst case, it is comparable. Look,

this was a huge problem for previous techniques.
But really, whatever metric we use to measure the

new one against previous techniques, it performs
better in pretty much all of them. Incredible.

And huge respect to Sebastian Starke, first
author of this work who just keeps publishing

these incredible papers, one after another, and
almost always with the source code as well. And,

yes, this also means that the source code
for this technique is also available. So,

from now on, we can soon expect
much more lifelike character

movements in our games and virtual
worlds. What a time to be alive!

Thanks for watching and for your generous
support, and I’ll see you next time!