[MUSIC PLAYING]



ALESHA UNPINGCO: We're
here to talk to you today

about designing AR applications.



So Google has a long
history in designing for AR.

We've been doing mobile
AR for the past four years

and working on other augmented
reality projects before that.

What's changed most recently
is that mobile AR is really

starting to take off,
and Google's AR core

makes it possible for anybody
to create quality AR content.

So I'll some fun
facts about AR core.

The first thing is
that we released 1.0

at the end of February, and
that made AR content available

to more than 100
million devices.

And we're already
seeing rapid growth

with more than 300 apps already
available in the Google Play

Store.

So you're here because you want
to learn how to design for AR.

And what we've found is
that once you understand

your users and the
type of experience

you're trying to create,
design principles for getting

started fall into five
different categories,

which we call the
pillars of AR design.

First, you want to understand
your user's environment.

Where will the users be
experiencing your app?

Think about the surfaces
that are available

and how your app can adapt
to different environmental

constraints.

Then you want to consider
the user's movement

and how much space the
user will need in order

to experience your app.

And when it comes to
initialization and onboarding,

you want to make the onboarding
process as clear as possible

so that users understand
exactly what to do

in this entirely new medium.

When it comes to
object interactions,

design natural
object interactions

that convey the affordances
and also the feedback

so that users understand how
these digital objects fit

in the context of your
real physical space.

And when you're thinking
about user interfaces,

balance the on-screen UI with
volumetric interface design

so that you're able to
create an experience that

is meaningful and
usable for your users.



So we have some
examples that showcase

the different guidelines
within each of these pillars.

And the thing that
we want to point out

is that this framework can
help anybody get started

with AR content creation.

So throughout our
talk, we're going

to show you some different
demos that you'll

be able to play with very
soon through an app we're

launching on the Google Play
Store called AR Core Elements.

And many of the core
interaction patterns

you'll see in this
talk are already

available in [INAUDIBLE] form
or will be available for Unity

later this summer.



ALEX FAABORG: All right.

So let's start out talking
about the user's environment--

the first pillar of AR design.

So ARCore is a relatively
new technology.

So just to begin, let's talk
about what ARCore can actually

do.

So it does lots of things.

The first thing that
everyone's familiar with

is surface plane
detection, where

it can understand
surfaces, tables, floors,

those types of things.

It can also do walls.

Vertical surfaces.

And then, of course,
what's better

than horizontal and
vertical surfaces?

It can also do angled
surfaces with oriented points.

Being able to place
objects on any angle.

ARCore does light estimation.

This is really important for
having objects sort of look

realistic in a scene.

And there's also
some other fun things

you can do with that
that we'll get into.

And announced at I/O
yesterday, Cloud Anchors,

which is available
now for ARCore

both on Android and also on iOS.

It lets you do multi-player
experiences in ARCore

with two people
viewing the same thing.

And also announced
yesterday, augmented images.

The ability to recognize
an image, but then also

not just recognize
it, but use that image

to get 3D post data off of it so
you know where it is in space.

So that's the current set
of ARCore capabilities,

and this is, of course,
growing over time.

And what we found
is the more your app

integrates with the
user's environment,

really, the more magical
your app is going to feel.

So let's look through
a few examples here.

First, surface plane detection.

A lot of AR apps currently
only use one surface.

Say you're playing a game--

a game board will
appear on a surface.

What we found is there's
really no reason that you

have to use one surface.

You could use all of
the detected surfaces.

And at moments when you
have your game interacting

with multiple
surfaces, those can

be really kind of like
breakout moments in your game

where it feels very magical.

Even something like if, say,
you're playing a physics based

game and you destroy
your opponents

castle or something and the
bricks fall onto the floor.

At that moment when you
see the items on the floor,

it can be really quite stunning.



All right.

Light estimation.

This is critical
for making objects

look realistic in the scene.

Here's an example that we're
working on to sort of test

out some different techniques.

Here we have three fake
plants and one real plant,

although I think it's
actually a real fake plant.

And we have unlit, which is the
most basic lighting you can do.

Really not very realistic.

Then dynamic and a combination
of dynamic and baked.

And what's great with dynamic
is you get real time shadows.

And then when you combine
dynamic with baked,

you start to see things
like soft shadowing

where the leaves of
the plant are actually

a little bit darker and
it's picking that up.

You can sort of see
an example of how

this looks with some movement.

And lighting, there's
definitely a lot of innovation

that's going to be
occurring in this space

as we try to get more and
more realistic lighting.

You can see where
we are right now.

And especially as the
scene gets darker,

you start to see that
the unlit object just

doesn't perform as well.

So it's really
important that you're

using the real-time lighting
APIs that are in ARCore.



The other thing you can
do is you can actually

use lighting as a trigger
to change something.

So here in this example when
you turn the light switch off

in the room, the
city actually glows

and it responds to that change.

And these types of moments
can really feel great

in magical freezers where--

imagine you're playing a sort
of a city simulation game

and it's having these kind of
significant, meaningful changes

based off the
environmental light.



All right.

Oriented points.

This is actually a
very new feature.

So we don't have a whole
lot of examples here,

but here is one of
the more basic ones.

I filmed this when
I was out skiing.

And here I'm just attaching
androids to the side of a tree.

And you can see that
as I place them,

they stick to exactly
that point on the tree,

at the angle of where
the branches were.



All right.

Cloud anchors.

Announced yesterday.

Here's an example
of that being used.

Here both players see
the same game board

in exactly the same
place, and they

can play that game together.

And this is really
tremendously fun.

You can actually try
it out in the Sandbox

if you want to stop
by later today.

And again, this works
with both Android and iOS.



Augmented images.

There's a lot of different
ways you can use this.

One demo that we have in the
Sandbox that you can go check

out is actually an
art exhibit that was

built using augmented images.



[MUSIC PLAYING]



So we're really
excited about what you

can do with augmented images.

And there's really
lots of possibilities

from artwork to even
something like just having

a toy sort of come to life.

You know, the surface
of a product box

where you can see sort
of 3D models of what

you're about to play with.



All right.

So now that we've gone
over some of the basics

on the core
capabilities of ARCore,

let's talk about
how you'd actually

start to design an app for AR.

So one of the first things
you're thinking is, OK,

where do I actually start?

You know, blank page
and ready to start

having brainstorming
and new ideas for AR.

And one of the things that
I'd want you to first focus on

is AR exists outside
of the phone,

so your design work
should really exist

outside of the phone as well.

So something I've found
with a lot of people who've

done tremendous amounts
of mobile design

is they tend to be very attached
to sort of the phone frame

and sort of flows of screens.

They've been doing
that for so long.

One of the first things you
need to do when you're starting

to think about AR is to
actually put away all of those,

you know, 2D UI stencils.

And don't really think
about the phone at all.

Instead, what you
want to do is you

want to sketch the actual
environment that the user's in.

So you should be sketching
living rooms, and tables,

and outdoor spaces.

And then as you sketch
the user's environment,

then you start to sketch
the various objects

that they're going
to be interacting

with in that environment.

In many ways, you can
sort of think of AR

as having a lot of
the same challenges

as responsive design for the
web in terms of different window

sizes, but it's even
more complicated

because now you have responsive
design for 3D spaces that are

the user's actual living room.

So you want to sketch
the user for scale

to get a sense of how
you're going to start

crafting this experience.

The user could be very large
relative to the AR objects

or very small.

And then you want
to start thinking

about how that user is
going to move around

in that environment.



ALESHA UNPINGCO: And that
brings us to user movement.

So now that we understand how
to design for the environment,

let's think about how to
design for user movement.

And as Alex mentioned,
it's completely OK

to design beyond the
bounds of the screen.

And what we've found
is that in many ways,

this can make the experience
feel more delightful and even

more immersive.

Because when you have an object
that begins on-screen and also

extends beyond the boundaries
of the phone [INAUDIBLE] port,

it can make the user feel like
the object is really there.

And beyond that, it can
also motivate the user

to organically begin moving the
phone around their environment

so that they can
appreciate the full scale

of these digital objects
in their physical space.

And that brings us to
our next observation.

Because users are more familiar
with 2D mobile applications

that don't typically
require user

movement as a form
of interaction,

it can be very challenging
to help convey to users

that they're able
to move around.

So many users don't
because it just

doesn't feel
natural based on how

you use 2D apps in the past.

But what we realized is
that characters, animations,

or objects that convey
visual interest on-screen

and then move off-screen
can be a natural way

to motivate users to move.

So here we have a
bird and it appears

in the middle of the screen.

And when it flies
off-screen, it's

replaced with a marker that
moves around and slides along

the edge to help users
understand the bird's location

in relation to the user.



Another major thing that
you want to think about

is that whenever you have an
experience that requires a user

to move, you also want to
think about how much space

a user needs.

So we found that
experiences fall

into three different sizes.

There's table scale,
there's room scale,

and there's also world scale.

And when it comes to
table scale, what we found

is that your experience
is able to scale

to the smallest of surfaces
so that many users are

able to enjoy your experience.

And with room scale, it
expands the impact of AR

so that content will
start to feel life sized

and you're able to do a lot
more with the space that's

available.

And world scale has no limits.

It allows users to appreciate AR
in whatever area they see fit.

And this is an area
we're particularly

excited about
because what it means

for procedurally generated
content in world scale.



So no matter what size your
experience ends up being, just

remember to set the right
expectation for users

so they have an understanding of
how much space they will need.

Because it can be a
very frustrating part

of the experience if the
user is playing a game

and in the middle
of the game they

realize they don't have
enough space to enjoy it.

And when it comes to how
much movement your experience

requires, there's no one
size fits all solution.

It really depends
on the experience

that you're trying to create.

For example, if you have a game
that requires user movement

as a core part of
interaction, that

can be a very
delightful experience.

You can use
proximity or distance

to trigger different actions
so that as a user gets closer

to this frog, it can
leap behind the mushroom,

or maybe the mushroom
can disappear.

And that can be
really cool to see.

However, if you have
a utility app where

the core purpose of the app
is to help users understand

very complex data
and information,

then requiring
users to move might

be a really bad experience.

Because what it
means is that users

who have different movement
or environment limitations

won't be able to get the
complete app experience.

So allowing users to manipulate
the object to rotate it,

to move it around in a space
that's more appropriate

will ensure that all users
have easy access to the data

that they seek.



ALEX FAABORG: Because
AR is relatively new,

the actual process for users to
flow from 2D parts of your app

into 3D can be, at
times, a bit awkward.

Amber is just starting to
create some sort of standards

around that.

So we'll talk about
initialising into AR.

One of the first
things you can do

is you can leverage standard
View in AR material icon

so users, when
they see that, they

know that when
they hit this icon

they're going to
be going into AR.

You can use this in
all the normal places

that icons appear.

Like floating action
button or on top of cards

as the indicator that
you can actually view

this object in 3D
in your environment.

One of the next
things you'll see

if you've been playing
with lots of AR apps

is something that you might
not initially understand.

I want to talk about
the concept of how

understanding depth actually
requires some movement.

So you'll see these
types of animations

where I was trying
to get the user

to move their phone around.

So why is that
actually happening?

Basically, we perceive depth
because we have two eyes,

but we actually get a lot
of our depth information

by actually sort of moving
our head around and being

in the scene.

And for the case of ARCore, most
current phones on the market

only have a single
camera on the back,

so the device only has one eye.

And if it hasn't moved
yet, it doesn't necessarily

know what's going on.

So this is the first
thing the phone sees.

It's going to say, all right,
well, that's interesting,

but I don't totally have a sense
of where these objects are yet.

And once you just move
just a little bit,

then it becomes clear.

As soon as you bring in a
little bit of movement, then

you have enough of that data and
different angles on the scene

that it can start to build up
a model of what it's seeing.

So that's why we
have these animations

at the start of the app to
try to get that movement,

to try to get ARCore to
have enough information

to recognize the scene.



Next thing you
want to think about

is deciding if users
are able to easily move

the objects after
they've been placed

or if these are really
more permanent objects.

And there's, again,
no right answer here.

So more persistent
objects might be

like a game board or something
that itself takes input,

but we want to recommend that
you use standard icons to set

expectations for users so
they know as they're placing

that object if that object
is going to move around

later on as they swipe on it.

So some examples of that.

Let's say you are placing,
like, a city game.

And here as you're
swiping on the city,

you're actually going
to be interacting

with the game itself.

So we recommend using an
anchor icon for these more

persistent object placements.

And you still want
to enable the user

to move the game board
later, perhaps through a menu

screen or some type
of re-anchoring flow.

So set expectations
ahead of time

that the city actually
is going to be

sort of stuck to the
ground there for a while

as you interact with the game.

Versus something like, say,
you're shopping for furniture

and you're just placing
a chair on the scene.

Here, the chair itself
has an interactive,

so you can actually map
swipe gestures onto the chair

and just easily move it around.

So using the plus icon to
kind of set expectations ahead

of time that you're not really
committing to exactly where

you're placing this object.

All right.

So now that we're talking
about objects interactions,

there's actually quite
a bit of details there.



ALESHA UNPINGCO: So
now that we understand

how to onboard
users, let's start

thinking about how
users can interact

with objects in their space.

One of the things
that we challenge

you to think about as designers
and developers in the community

is thinking about
how to problems

solve for user behavior,
even when it's unintentional.

So one of the
things we recommend

is giving users feedback
on object collisions.

And this solves a huge problem
that we see in mobile AR

where a user will be
moving the device around

and once the device collides
with an object in AR,

that object might
disappear and the user

has no feedback in
terms of how to fix it.

So what we recommend is we
recommend providing feedback

in the form of camera filters
or special effects that

helps users understand when
object collision is not

an intended interaction.

And this tends to
work really well.

The other thing that
you want to think about

is how to give users the
right type of feedback

on object placement.

And it's really
important in this case

to think of each stage
of the user journey,

even as it relates
to surface feedback.

So surface feedback in
AR is very important

because it helps users
understand how ARCore

understands the environment.

It gives the users a
sense of the surfaces

that are available, the
range of the surfaces

that are available.

So we recommend
including feedback

on the surfaces when the user
is placing objects in the scene.

The other thing
that we recommend

is maintaining the height
of the tallest surface

as a user drags an object
from one surface to another.

And once an object is
suspended in the air,

make sure that you're always
communicating visual feedback

on the dropdown point.

That's way it's very clear
to the user at all times

where the object
is going to land.

And once an object is
placed into the scene,

we also recommend
providing feedback

in the form of visual
feedback on the surface,

or even on the
object itself, just

to communicate
the object's entry

into the physical environment.

So now that we know how to play
with objects in your scene,

let's think about how an
object might get there.

We recommend using gallery
interfaces in order

to communicate to
users how they can

take objects that live
on-screen and drag it out

into their real world.

So here you see we
have a gallery strip

at the bottom bar.

And as a user selects
an object, they're

able to drag it
onto their space.

And not only that, we're able
to support both selection

states and also very familiar
gestures that allow users

to manipulate the objects.

So you can use pitch to
scale, twist to rotate,

and even drag to move.



And you see many examples in
our talk of how dragging objects

is a very common and
expected behavior.

But another alternative for
object selection and object

movement is through a reticle.

So reticle selection
is also very

effective in that it allows
users to manipulate objects

in their scene without covering
too much of the user's view.

So we have an example here
where reticle selection

is being used to select a rock.

And that's triggered
via the Action button

in the bottom right.

But what it allows
users to do is

that it allows users to
see the many surfaces that

are available.

And as you can
imagine, if a user

is selecting an object
with their finger

and dragging it
across the screen,

you don't have as much
screen real estate

to see all of the surfaces
that the user might

want to place the object on.

So reticle selection is
very, very impactful here.

The other thing that you
get with reticle selection

are raycasts.

So raycasts are very
effective in helping

the user get a sense of
the virtual weights applied

to each of these objects.

So here we have another
example where the user

is able to pick up a feather.

And once the father
is picked up,

you'll notice that the raycast
has very little movement

and very little bends on it.

And for the most part,
it remains straight.

However, when the user
picks up the rock,

you're able to see a more
dramatic bend applied

to the raycast.

That signifies the
larger amount of mass,

the heavier weight of
this object in relation

to the feather.



ALEX FAABORG: All right.

So let's move on to the final
pillar, which is volumetric

interface design.

I think one of the first things
you want to consider here

is that the phone is
the user's viewport.

They're actually using the
phone to look out into the scene

and see the application.

And because of that,
you don't actually

want to place a lot of
2D UI on the screen.

That's actually going to
obscure the user's view

onto your application.

So showing a sort
of quick example.

It's obviously a lot nicer to
have a limited set of controls.

As soon as you start
to clutter the screen,

it really gets in the
way of the user's ability

to enjoy the AR app.

And the sort of
counter-intuitive thing

that we've even found is that
users are so focused on the app

out in the world
that often designers

will place a control
on a screen level

because they want to draw
attention to that control,

but it's actually having
the opposite effect.

The users are
actually more focused

out in the scene, so they'll
actually just miss controls

that are drawn on the surface.

Just kind of tune those out.

So really, you want to be
very mindful of when you're

making decisions on if you're
going to put a control up

on the screen versus out
into the scene itself.

For not just obscuring the view,
but also for discoverability

of them finding that.

And that's not to say that you
should never put a control up

onto the screen, but you
want to be considering

a few different metrics on it.

So our recommendations is
that you only really leverage

on-screen surface UI
for things like controls

that have a very high
frequency of use or controls

that require very fast access.

So like a camera
shutter button is

kind of the perfect
example of something

that hits both criteria,
where in a camera

you're obviously
taking lots of pictures

and also you want to take
pictures very quickly.

But imagine if
you're playing a game

and there is some ability
to, like, fire or something.

That would be a good candidate
for an onscreen control

because you're both
hitting that button a lot

and also you need to get to
that button very quickly.



So we talked about
using the View in AR

icon to get people into the
experience and transition

from 2D into AR, but
also want to be very

careful about the opposite.

Of when users are now in AR and
they're actually transitioning

back to a 2D experience.

And one thing we found is
that if the user is not

initiating that action to go
back into a 2D experience,

it can actually be pretty
obnoxious because they're so

focused out in the scene.

So the user's viewing
the application,

and then suddenly
a 2D UI shows up

and blocks the entire viewport.

That can be pretty annoying.

So depending on even
if the user is exiting,

or they're customizing
an item in the scene,

or whatever the use case
is, you want that flow back

to 2D screen level UI that's
covering most of the screen

to be something that the
user's actively doing,

and not something that
happens by surprise.



So a common thing with
mobile application design

is you want to maintain
touch targets that

are about the size
of the user's finger.

For 3D, this is, of
course, a bit harder

because the object could be any
distance away from the user.

So quick example of some
things you can do here.

Here we have two tennis balls.

And when you tap
on the tennis ball,

confetti fires out of
it because in AR you

can do whatever you want.

And we're showing
the touch target size

with the dotted line.

So one of these tennis
balls is actually

maintaining a reasonable
touch target size

as it gets farther away,
whereas the other one is not.

It's just mapping to the
virtual size of the object.

And, of course,
it's a lot easier

to interact with the one that
is maintaining a large target

size.

We've also found
for interfaces where

you're manipulating objects,
if you're not doing tricks

to kind of maintain
target size, you often

get these problems where you
swipe an object very far away,

and then it's actually hard
to bring the object back

because it's now
such a small target.

So you have to actually walk
over to the object to get it,

which is a little
bit frustrating.

Maybe you could say
it's very immersive,

but either way, it's nicer
to be able to actually bring

the objects back as well.



So on the whole, you want to
be thinking about what controls

are going to be on the screen
versus what controls are going

to be out in the scene.

And kind of a mantra
that the team has had

is to say, you know,
scene over screen.

Obviously, we talked about sort
of battery cases of when you'd

want to put something
on a screen level,

but I found that it's many
people's initial reaction

to design everything
for the screen level

because that's the
type of design work

we've been doing
for 2D applications,

but you really want
to start thinking more

about volumetric UI and having
your UI out into the scene

itself.

So give a quick example of this.

This is actually one of the
demos that ships with scene

form.

It's a solar system simulator.

Loads of fun.

Also, it's missing
a planet right now.

We did fix that for the
public release, in case

you notice that in the video.

But imagine now you need
to design the UI for this.

So a lot of people
would initially

think, oh, I'll have a
gear menu up in the corner.

That will throw something
up on the screen.

You know, the
problem there is then

you're not going to be able
to sort of be as immersed

in the simulation itself as
you're interacting with it.

So an alternative
way of doing that

is actually leverage these
objects on the scene itself.

So as you're tapping
on planets, you'll

get feedback on
what planet that is,

which is nice for
educational use cases.

And then in this particular
demo, when you tap on the sun,

that's how you start to control
the entire solar system.



So here the user is
tapping on the sun,

and that brings up a panel.

This is actually
an Android view.

So in scene form, you can map
just standard Android views

into AR.

And here you have
controls like changing

the orbit speed or
the rotational speed

of the planets themselves.

And it's really nice
to be able to interact

with these objects
in the scene, and not

to have that sort of sudden
loss of being able to see things

and being sort of taken
out of the experience.

And that kind of brings me
to my final point, which

is this idea of AR presence.

So we'd actually seen this
coming up in user research

studies where people would
be looking through a phone

and then they would kind of
look outside of the phone

to see that something
was placed correctly.

And then, of course, we're
recording and so they laugh.

And they're like, oh, yeah.

Right.

Of course.

I can only see it
through the phone.

And we always laughed
when we saw this happen.

And then I was
testing out an app.

It was these sort of
plastic interlocking bricks

and I had instructions
of what I was building,

and I was playing
it for a long time.

And at one moment I looked over
to see the instruction book,

and it wasn't there.

And I had the reaction
that you normally

have if, like, an object
just disappears in real life.

And, of course, then immediately
I'm like, oh, that's silly.

Yeah.

It's AR.

But I was so immersed in the
experience and the application,

and I'd been playing it for
so long that I was no longer

kind of mentally tracking what
was real and what was virtual.

And I was just sort of buying
that the experience was

happening.

And you're going to start
to have this experience

as well as you're interacting
with these applications.

And I would say that's the
moment when your application is

performing really,
really well because it

means that the user is
just completely immersed

and engrossed in
the application.

So if you ever have
these moments where

people are looking at a
vase through their phone

and then they look down and
it disappears and they react,

that's good.

That means the app
is performing great.



ALESHA UNPINGCO: All right.

So we've gone through the
five pillars of AR design,

which, again, include
understanding the user's

environment, planning
for users' movement,

onboarding users by
initialising smoothly,

designing natural
object interactions,

and balancing on-screen and
volumetric interface design.

And again, this
framework, we believe,

will help anybody get
started with creating apps

that everybody can enjoy.

So we have a quick
video for you.

Some amazing content that
many designers and developers

like yourselves from the
community have created.

We hope you enjoy.

[MUSIC PLAYING]



We're very happy to
share with you everything

that we had today,
and we look forward

to seeing what you create.

Please fill out our survey and
check out our resources online.

Thank you.

[MUSIC PLAYING]