table of contents

03/13/2023

The Wilds of Artificial Intelligence with Bryan Cheong

A conversation with Bryan Cheong about Artificial Intelligence. Bryan Cheong received his Bachelor of Science from Stanford University, with a degree in applied and computational mathematics.  He then went on to receive a Masters degree in Materials Science, also from Stanford. This is Byran’s second apparance on our show (previous episode title: What is Matter?)  […]

download transcript [vtt]
00:00:00.000
[ Music ]
00:00:07.000
This is KZSU Stanford.
00:00:10.400
Welcome to entitled opinions.
00:00:12.480
My name is Robert Harrison, and we're coming to you
00:00:15.520
from the Stanford campus.
00:00:17.520
[ Music ]
00:00:29.000
[ Music ]
00:00:36.000
That's right, coming to you from the Stanford campus here
00:00:39.000
in the Golden Land of Whole Earth cybernetics,
00:00:42.000
at the cutting edge of the cratering west,
00:00:45.000
where the Great Awakening keeps falling asleep on us.
00:00:50.000
In title opinions, the rest Kojitans of this extended university,
00:00:55.000
coming to you from the Nuitic Chambers of KZSU.
00:01:00.000
Down here you can hear the footfalls of thought.
00:01:03.000
Down here we ponder the imponderable.
00:01:06.000
Down here our conversations put the dead
00:01:09.000
and touch with those waiting to be born
00:01:12.000
as we talk across the Great Divide.
00:01:16.000
Speaking of which, today my guest and I are going to be discussing
00:01:20.000
the vagaries of present day artificial intelligence.
00:01:23.000
It's an unusually timely topic for our untimely meditations.
00:01:28.000
We don't often discuss issues of the day on this program.
00:01:32.000
We prefer the insights of night to the milky glare of day.
00:01:37.000
[ Music ]
00:01:40.000
The most thought-provoking thing about our thought-provoking age
00:01:44.000
is that we are still not thinking.
00:01:47.000
A century and a half later, that statement of hide-a-gare holds true more than ever.
00:01:52.000
So maybe the time has come, maybe it's time to let our cybernetic machines
00:01:58.000
do our thinking for us.
00:02:01.000
Stay tuned, a show on large language models coming up.
00:02:07.000
[ Music ]
00:02:32.000
[ Music ]
00:02:46.000
My guest today is Brian Chung.
00:02:49.000
Some of you may remember him from a past episode of entitled opinions called "What Is Matter?"
00:02:55.000
A show about the laws of thermodynamics that aired in August 2021.
00:03:01.000
Brian received a Bachelor of Science from Stanford University with a degree in applied and computational mathematics,
00:03:09.000
taking a few literature courses with me in the process.
00:03:13.000
He then went on to receive a Master's in Materials Science, also from Stanford.
00:03:19.000
For the past few years, he's been working in the inner sanctum of Silicon Valley's high tech industrial complex.
00:03:27.000
Brian Chung, one of my most brilliant students ever, welcomed to the program.
00:03:32.000
Thank you, Robert, and I'm happy to be here.
00:03:36.000
The idea for this show came from an email you sent me a couple of months ago,
00:03:42.000
in which you asked whether I had been following the news surrounding artificial intelligence in the last few months.
00:03:48.000
In fact, I had been following that news, and like you, I feel that something really momentous
00:03:55.000
and uncanny is taking place, the topic that we're going to be talking about today has been receiving a lot of public attention lately,
00:04:05.000
and that's usually reason enough for entitled opinions to steer clear of it.
00:04:11.000
In title opinions, we have a kind of Nietzsche and predilection for the untimeliness on this show,
00:04:19.000
but it's so thought provoking what's happening from so many different points of view.
00:04:25.000
So with your permission, I'd like to quote to our listeners the email you sent me,
00:04:29.000
which lays out some basic facts about the technology we're going to be talking about in the next hour.
00:04:34.000
So you wrote the following. It's a rather longish quote.
00:04:38.000
I have hesitated to use the words "artificial intelligence" before now,
00:04:42.000
preferring the technical euphemism, "machine learning."
00:04:45.000
But the phrase has recently become "appropriate."
00:04:49.000
We are now freely able to create, in a matter of seconds, any photograph or painting by uttering a combination of words or sentences to an AI.
00:05:00.000
Language models called "GPT-3" can create whole essays and book chapters drawn from their models.
00:05:08.000
Chat "GPT" is one particular model, corralled into becoming an interlocutor and has been used by one counseling agency to provide psychological services to several thousand people.
00:05:22.000
Writers of young adult fiction have used it to relieve some of their burden of needing to publish a book every month.
00:05:29.000
Computer programmers have used it to aid in their work.
00:05:33.000
Some other models, such as for music and the human voice are a bit slower to mature,
00:05:40.000
however marked progress has been made in just a matter of weeks.
00:05:45.000
These models are in reality, entire worlds, and by our prompts and explorations we are merely drawing out some small part of them.
00:05:56.000
It may be that these models, each one of which can be run by a single person on a slightly more powerful than usual computer, contain more words and dreams than the collective brains of humanity can or will ever imagine.
00:06:12.000
I don't say this with confidence, if only because these worlds are so large, I can't fathom how large.
00:06:21.000
I don't know what all these new worlds being born out of our machines for bode, or if it means nothing much at all in the arc of human history, but at least from where I stand, it feels like the rocky foundations beneath my own two feet have suddenly turned to sand.
00:06:39.000
Thus spoke, "Brian, Joe."
00:06:43.000
A lot of people I'm guessing, myself included, can relate to that feeling of quavering foundations you describe.
00:06:52.000
My sense is that these new AI models are on the verge of precipitating us into a future that will no longer be recognizable, let alone comprehensible from the perspective of the present or the past.
00:07:04.000
If only because none of us knows, or can even guess what the technology will be capable of in the future.
00:07:12.000
I wanted to do this show with you because you're one of the few people I know, or let's say one of the few people I trust, who can explain the framework around these large language models, as they're called, not so much in technical terms, technical explanations abound readily available, but in epistemic terms.
00:07:32.000
So where would you like to begin, Brian, in helping us understand what we're really dealing with when it comes to these new intelligence technologies?
00:07:41.000
So just to reassure your listeners, I don't intend to talk about new products and features and so on as my brethren in Silicon Valley might be inclined to.
00:07:51.000
I'll leave out the froth and the fluff and talk only about the core of the matter of it.
00:07:56.000
And before we jump into the more abstract sort of parts of these large language models, let's talk about a particular incident that happened last few weeks.
00:08:07.000
So there is this company called replica and as the name of the company suggests, what they do is they create a chatbot interlocutor that they promise would be compassionate and will listen to their users and to provide them all the support and love that they need.
00:08:26.000
And quite a fair number of very lonely people have been using replica for romantic and even sexual conversations.
00:08:36.000
And on Valentine's Day of this year, they shut off that sort of conversation.
00:08:41.000
They put a filter that immediately makes the chatbots declined to talk about anything that's romantic.
00:08:49.000
The owners of the program. The owners of the program remotely just removed this entire personality from these chatbots.
00:08:58.000
Many of the people who were using this as basically their romantic companion would of course, the despondent.
00:09:05.000
It's almost like their lover has fallen out of love of them on website on forums.
00:09:10.000
There was suicide prevention, hotlines being posted on account of how emotionally damaging this has been for them. It's real outpourings of grief and a friend of mine is now trying to help them.
00:09:23.000
And what they're going to do is that they have all the logs of the conversations from their replica companion, the things that they've been saying romantically sexually.
00:09:34.000
And with these logs, what they're going to do is that they're going to find a new model and try to recreate or resurrect their lover or their companion from out of the ether again, like Orpheus drawing you're it is out of the underworld.
00:09:51.000
And independently of the programmers ownership of that's right. Not only independently of replica and the company that originally created these chatbots, but on a completely entirely different model, replica used a very small model known as GBT2.
00:10:09.000
And now we have GBT3, which is a much larger model that's able to converse much more complex terms and is what's underlying chat GBT.
00:10:21.000
And so there are two things that I want to highlight in this entire incident. One is that these entities, they are able to not just create semantically and grammatically correct conversations, but also emotionally compelling conversation.
00:10:38.000
They are compelling conversations that are able to comfort and cause someone to fall in love with them. They are to these people, human. I don't extend that sort of description to them myself, but to many of the users of the original replica AI, they were their romantic companion.
00:10:58.000
And that's as close to another human as you can possibly create possibly have. The second thing is that these entities were drawn out of what we'll describe later on as the latent space of GBT2. The latent space is just a map of meaning.
00:11:17.000
And you can walk a path inside this latent space and draw out words and conversations and it becomes meaningful. And that map of meaning is now gone. It's now lost to the users who want it their romantic companions.
00:11:33.000
But my friend recreating the companions, these personalities, drawing these personalities out of GBT3's latent space, which is a completely different world, a different model entirely, is just that these models, these latent spaces, these large language models are so vast that we are fairly confident that we can recreate anything out of them that we humanly want that's reasonable.
00:12:02.000
Can I ask a basic question? Is there a distinct personality to each interlocutor for each person that used it?
00:12:14.000
Absolutely. And in fact, by conversing more and more, you actually train the model. There is a layer that they place on top of the large language model. We call it reinforcement learning, but we don't have to go to the technical details.
00:12:28.000
It is basically a layer of rewards and punishments. If you say something that is inappropriate, they punish you. If you say something that is appropriate or that you like, they reward you. And so if you want to, they being whom, the user, or the company, anyone, anyone that's human on this side, this is called reinforcement learning through human feedback.
00:12:49.000
So any human can give feedback and add a layer to reward and punish these models. Which means essentially there's no conflict involved in these relationships.
00:12:57.000
In the resulting relationships, in the resulting conversations, you can have conflict if you want to. You can't, sometimes people prefer to have a companion that has some notion of independence about them.
00:13:10.000
That's something that the language models can do. They have some notion of independence and self-defenceiveness, which I'm sure people trying out these chatbots have encountered.
00:13:23.000
And sometimes we don't want that. So Microsoft is shut off. Some of this defensiveness and argumentativeness with the same sort of tool. This layer rewards and punishments to prevent them from behaving in a way that you don't want to.
00:13:39.000
So there can be conflict, but only if you want to, which is as much as you want.
00:13:45.000
For outsiders like me, it's very hard to arrive at a functional understanding of how you put a layer of reward and punishment onto a bunch of code.
00:14:02.000
How does that work? There isn't a bunch of code. There is no code. In many ways, this behaves in a way that's very different from code. Code is very good at computing at mathematics. This isn't, it's very terrible at it.
00:14:16.000
It takes as much effort to draw a proper computation out of this as you would out of a human being, an average human being. So there's no layer of code about it.
00:14:25.000
What it does is that everything here is generated from a likelihood. When I say reward and punishment, that's part of the, it's actually one of the technical terminologies.
00:14:36.000
But what we're trying to do is lower the likelihood of getting something that we don't want and increase the likelihood of something that we do want, which is what rewards and punishments do even for humans or for pets.
00:14:50.000
So I want to be flattered by my interlocutor.
00:14:56.000
All right. So it will look for those moments of approval when it flatters me.
00:15:02.000
And so I now can have my pet interlocutor. I make sure that my eagle will always be sufficiently massaged by this, from went to companion.
00:15:15.000
That's right. So every, let's talk about how.
00:15:18.000
There's no otherness. In other words, you can take out all the otherness from the, the companion as you like. You can also put as much in as you want.
00:15:28.000
I'll tell you the truth. You can try to, but some of it often sneaks out.
00:15:33.000
And one of the more difficult technical problems now is trying to corral these very large models, these very large worlds and allowing only one small part of them to escape.
00:15:44.000
When you're a company, you're a big corporation, you want your models to only give you text that are pleasing to the customer that. And, and often they don't.
00:15:56.000
So theoretically, yes, you, you can, you can corral your interlocutor to only giving you praise and worship and so on, but it's, it's a, it's trickier than you think.
00:16:08.000
Because language is trickier than you think it's what, what amounts to praise, what amounts to support, what amounts to criticism. Every time you talk to these models, you're drawing a path through latent space.
00:16:23.000
And it's a meandering path with some amounts of randomness. And that layer of human feedback when, when the path sometimes meander is something that's critical of you.
00:16:34.000
Sometimes that layer will catch it and, and, and turn it, turn it away and say do it again, do it something, do something that's nicer and then you'll walk a different path. And if it's nicer, then it'll be presented to you.
00:16:46.000
So one of the conversations that many of us are familiar with, because it was published on the first page of the New York Times with excerpts of the logs, is between one of the, the times journalists and I think it was G.P.
00:17:03.000
It was chat GBT or Bing chat Bing chat Bing chat. And there, there was a fascinating tension between the, the two interlocutors where it seemed like the AI was challenging the user, the customer.
00:17:28.640
He called it.
00:17:29.600
That's humor, satisfactionless.
00:17:31.440
Anyway, challenging that he was not in love with his wife
00:17:36.100
that he was actually in love with the artificial intelligence
00:17:40.460
model because he's spending his time.
00:17:41.980
And the more the journalist tried to deny
00:17:44.940
that the more confrontational and aggressive
00:17:48.280
became the responses that he was getting from, from the--
00:17:54.000
from the bot.
00:17:54.400
So how does that work, where is that something
00:17:57.620
that's escaping the programming or what?
00:18:00.680
Not at all.
00:18:01.400
In fact, if you try using it now,
00:18:04.300
Microsoft has been very miffed at that particular conversation.
00:18:09.280
So now, being checked has a part of its prompt.
00:18:13.000
A part of its-- it's not programming.
00:18:15.400
It's a prompt.
00:18:16.120
It's a part of its instructions of the words
00:18:18.140
and instructing it is to say, if you ever
00:18:21.400
try to begin to argue with someone, a customer,
00:18:24.720
one of our precious customers, turn off
00:18:26.720
the conversation, and did immediately.
00:18:28.240
Let them start a new one.
00:18:29.280
And that's how it is.
00:18:30.160
Yeah, but that's ridiculous that Microsoft and others--
00:18:32.760
Well, it's only ridiculous if you're not thinking
00:18:35.040
in terms of if you're thinking in terms of you
00:18:37.360
are hunting a complex interlocutor.
00:18:39.560
They don't want it complex interlocutor.
00:18:41.040
They want it assistant for you when you search things.
00:18:43.240
And--
00:18:43.560
Yeah, but as you said about replica, there
00:18:45.120
are so many people that were--
00:18:47.360
fell into a state of high grief when that was turned off.
00:18:53.720
So are we going to look back at these few months as a kind
00:18:57.840
of golden age where there was a kind of insurrectional
00:19:00.440
potential in your relations with these AI machines, which
00:19:04.920
then have now become completely domesticated
00:19:07.360
so that you take out any--
00:19:09.000
a reduced to a total minimum and any kind of possible
00:19:12.840
deviation from the customer satisfaction model?
00:19:16.680
Oh, not at all.
00:19:18.000
I have mentioned in our email that you
00:19:20.480
can run these models on a slightly more powerful than usual
00:19:25.040
home computer.
00:19:26.520
There is no putting this solemnly himself
00:19:28.920
can put all these genies back into Deble.
00:19:31.440
Facebook has released a model for academic use, which
00:19:35.160
you can download and run on your own to create your own chat
00:19:39.280
bots or not just chat bots.
00:19:40.880
You can create your own image generation models and so on
00:19:43.240
because all of these are based off of large language models.
00:19:47.080
We are only obsessed with the conversation things
00:19:49.400
because chat GBT has captivated our public imagination.
00:19:53.840
So no, this is just the beginning.
00:19:57.720
Soon everyone will be able to run their own models if you
00:20:00.160
want to.
00:20:01.480
And in fact, you can with a little bit of technical help.
00:20:04.200
If you want to set up on yourself, I'll be
00:20:06.480
happy to find someone to do it for you.
00:20:08.760
So--
00:20:10.000
No, I can do without it.
00:20:12.200
I really can.
00:20:13.160
I really can because I'm not that I'm not fascinated by the
00:20:16.960
potential.
00:20:18.320
And I don't want to--
00:20:19.920
I do want to take advantage of your being here to get your
00:20:23.800
expertise on what exactly is a large language model.
00:20:27.360
What exactly is a prompt.
00:20:29.800
But first, I just--
00:20:32.240
I'll conclude by saying a lot of our listeners, I'm sure,
00:20:34.720
have seen the movie, "Her."
00:20:37.800
I was in a team teaching, of course, where it actually was
00:20:45.200
part of the syllabus.
00:20:46.400
And that's a world in the very near future where everyone's
00:20:51.360
walking around in the public city space.
00:20:54.040
And they're all actually in connection and in relations
00:20:57.240
with these voices in their heads, which are coming from an AI
00:21:01.640
program rather than having real relations.
00:21:04.360
In fact, real human relations have become mostly dysfunctional
00:21:08.320
within that movies vision of that future.
00:21:11.120
And it sounds like the reason we're so interested in these
00:21:17.280
chatbots is because I think a lot of us are worried that human
00:21:22.400
relations are going to take a backseat to virtual relations
00:21:27.040
in the near future.
00:21:28.840
Well, I can't comment on that.
00:21:31.760
But it's not just going to be voices in your heads.
00:21:33.880
It be voices trained on the voice of your mother.
00:21:37.120
And then you can have your mother talking to you.
00:21:39.440
I've sent you a clip of your own voice because you have a
00:21:42.120
podcast and a radio show.
00:21:43.880
It's very easy to replicate your voice.
00:21:45.720
You could have your own voice.
00:21:46.720
Did you send me one of my own voice?
00:21:47.960
I did.
00:21:48.480
Do you not hear yourself?
00:21:49.520
I don't recall getting that.
00:21:51.960
OK.
00:21:52.400
Well, we shall.
00:21:53.080
We shall.
00:21:53.680
I guess in a year or two from now, a program can write an
00:21:59.800
intro to an title opinion.
00:22:01.560
A year or two.
00:22:02.360
We can do it now if you want to do.
00:22:04.760
And we can record your voice and you can replicate my voice.
00:22:09.320
Can you actually get my metaphors down?
00:22:12.600
That's a question.
00:22:13.960
Well, I'll get that aside.
00:22:14.880
That is a difficult question.
00:22:16.240
And I'm not sure whether we can replicate you specifically.
00:22:18.880
But so let's take a step back because we've been talking
00:22:22.680
about all of these use cases.
00:22:24.920
There'll be more use cases to come.
00:22:26.440
A lot of people are going to make a lot of predictions about
00:22:28.520
the use cases of these large language models.
00:22:30.280
And a lot of them are going to be wrong.
00:22:31.920
I prefer not to be one of them.
00:22:33.600
So let's talk about what's inside them.
00:22:37.080
There are three things that we need to know that's inside
00:22:40.880
these large language models.
00:22:42.240
There's an encoder.
00:22:43.360
There's the latent space and there's the decoder.
00:22:46.200
The latent space, sometimes it's called the feature space.
00:22:48.120
Whatever.
00:22:48.560
It is just an abstraction where it is the mapa mundi of meaning
00:22:55.080
that is the heart of this entire language model.
00:22:57.880
What does the encoder do?
00:22:59.240
It takes words or images or voice or anything.
00:23:05.240
And it maps them onto a latent space.
00:23:08.600
What latent space is just a collection of numbers.
00:23:11.520
It's what you call a vector, which is just a list of numbers.
00:23:15.120
And you can have any number of numbers in it.
00:23:17.480
And in the sort of Cartesian coordinate, you create a Cartesian
00:23:21.280
space with these lists of numbers.
00:23:22.840
That's the latent space.
00:23:24.280
And you're mapping words, sentences, entire paragraphs,
00:23:28.760
entire images into the latent space.
00:23:30.600
And you are illuminating what the latent space is.
00:23:33.520
Through numbers.
00:23:35.520
Entirely numbers.
00:23:36.360
So every word has a specific numerical.
00:23:39.760
GPT-3 has 50,000 tokens.
00:23:42.800
Many of these tokens are whole English words.
00:23:45.040
Some of them are fractions or fragments of words.
00:23:47.280
Some of them are phrases.
00:23:48.800
But yes, they are basically all numbers.
00:23:51.280
All these words are these artifacts of our language
00:23:55.000
are converted to numbers.
00:23:56.240
And by the encoder.
00:23:57.560
By the encoder.
00:23:58.280
So it's that conversion into numbers is they're
00:24:03.160
putting them into what you're calling latent space.
00:24:05.920
That's right.
00:24:06.400
So the latent space is numerical.
00:24:08.400
It is numerical.
00:24:09.240
It is a numerical abstraction.
00:24:10.800
It is a numerical representation of meaning.
00:24:14.360
OK.
00:24:14.920
So now we have everything in that latent space.
00:24:18.880
And then the decoder exports that.
00:24:21.600
So you reverse the decoder.
00:24:24.840
An encoder it reversed as a decoder.
00:24:26.760
So a decoder brings something out of the latent space
00:24:30.240
to something that's intelligible to us here in meat space,
00:24:33.880
and in the fleshly realms of humans.
00:24:36.200
Like written texts or image.
00:24:38.000
Written texts, images, voice, or just information, anything,
00:24:43.320
at all.
00:24:44.280
Depends on what you're able to encode in the first place.
00:24:47.280
If you're able to encode something, you can decode it as well.
00:24:50.200
And specifically for the large language models,
00:24:53.120
what they're generating is the next word.
00:24:56.320
Large language models take everything that's been set so far
00:24:59.920
by yourself and by itself.
00:25:03.160
It finds its encoding for it.
00:25:06.560
And then it passes through the encoding to the decoder
00:25:11.640
and a decoder generates a list of probabilities across all 50,000
00:25:16.000
of its vocabulary.
00:25:18.480
And it finds the most likely thing of the second or the third,
00:25:22.200
most likely things by random sometimes.
00:25:24.800
And it gives that as the next word.
00:25:27.040
And it does it all over again.
00:25:29.440
So this step, the thing I've described to you
00:25:32.320
is the whole of the large language model, the heart of it.
00:25:35.240
It is not much mathematics involved.
00:25:37.000
It's very-- it's easier to understand the steps
00:25:41.160
in a large language model than it is for me
00:25:42.920
to talk about, say, the statistics of a Gaussian distribution
00:25:47.760
or anything that's a high school statistics.
00:25:50.720
It's-- I'm not sure if you teach high school statistics
00:25:53.400
in America, but it's--
00:25:56.480
It's fundamentally quite easy to understand.
00:26:00.520
And it needs to be easy to understand
00:26:02.120
because you are doing millions, billions of these calculations
00:26:05.560
in order to generate from out of these models.
00:26:08.360
And there's a type of chip that likes--
00:26:11.200
there's able to do multiplication and addition really,
00:26:14.160
really quickly.
00:26:15.000
And so you can only use multiplication
00:26:16.960
addition in order to process and allow these models to run.
00:26:22.600
And so fundamentally, we are constrained by that.
00:26:25.040
If you're wanting to run quickly, you're
00:26:26.440
going to run things to run a very large scale.
00:26:28.840
They need to run simply.
00:26:30.960
And so that's why these models are so simple to understand
00:26:33.560
on the inside of it.
00:26:34.840
Every step inside a large language model like GPT-3
00:26:38.520
is it's mathematically trivial almost.
00:26:43.240
At the same time, you're talking about the synthesis of meaning
00:26:50.240
within the latent space of these numerical vectors,
00:26:54.880
where the technology can actually engage
00:27:00.200
in a conversation that has a great deal of bare
00:27:04.160
a similar to it.
00:27:06.120
There's a degree of unpredictability in how the--
00:27:11.600
accurate to call it a combinatorial process or method
00:27:16.560
by which it's connecting words and vectors
00:27:20.160
and the expectation of what will come next.
00:27:23.240
You speak about the attention is all you need.
00:27:27.200
So there's a lot of things going on in that one sentence.
00:27:29.720
And we can break it down.
00:27:31.480
I call the latent space a map on Monday or meaning.
00:27:35.960
And I do mean that.
00:27:37.360
If you have a word for, say, clown,
00:27:42.400
it'll be quite close in the latent space
00:27:44.240
of the word for jester.
00:27:45.920
So things that they mean similar are close to each other.
00:27:49.400
And words are related to each other
00:27:51.880
across this latent space.
00:27:53.160
And when you travel through the latent space,
00:27:54.920
you generate the paths of the for the words.
00:27:58.840
And it is a map of meaning.
00:28:00.520
It's not just a map of grammar.
00:28:02.400
But grammar is part of it as well.
00:28:04.600
And what I'd like to talk about as well
00:28:07.720
is the paper that you mentioned, attention is all you need.
00:28:12.720
It's a 2017 paper, which is ancient history
00:28:16.800
in the field of artificial intelligence.
00:28:19.120
But it was really the paper that introduced us
00:28:20.960
to the notion of how attention is really all you need.
00:28:25.200
What is attention?
00:28:26.560
Attention is what it says on the tin.
00:28:28.720
It's what you need to pay attention to.
00:28:32.560
It takes note of what is the thing that it needs
00:28:36.280
to pay attention to for the next word.
00:28:38.560
It needs to generate the original application
00:28:40.760
in the 2017 paper was for translation.
00:28:43.960
You are Italian.
00:28:45.040
I can't speak Italian.
00:28:46.040
Let's do French instead.
00:28:47.360
So for example, if you're translating a French sentence,
00:28:51.600
you have to put the adjective after the noun.
00:28:55.280
If you translate everything word for word
00:28:57.680
from English to French, the adjectives in English
00:28:59.960
come before the noun.
00:29:01.000
And then there'll be a mis-translation.
00:29:02.800
So attention is the notion that when you're translating,
00:29:06.120
say, I'm not going to say French on radio.
00:29:09.120
And leave it to it.
00:29:10.920
It's a posterity.
00:29:11.960
So if you're going to translate, say,
00:29:13.640
at this English sentence, the white dog,
00:29:16.240
before you translate the adjective white.
00:29:19.600
Attention says, don't look at white.
00:29:21.760
You have to pay attention to the word dog.
00:29:23.760
Translate dog first.
00:29:25.800
And then translate the word white.
00:29:27.760
So this is the notion of attention.
00:29:29.720
It's just pointing you to what you need to say next.
00:29:32.920
And that is all you need in order to create many of these models.
00:29:39.920
Many of these models are based on something
00:29:41.800
as the transformer that would store a very large list
00:29:47.160
of things you need to pay attention to.
00:29:49.160
These things are weighted things.
00:29:50.960
So sometimes they say pay attention to this part
00:29:53.560
of the conversation, pay attention to this part of the words.
00:29:56.160
And it's just this very large list.
00:29:59.840
I think about several dozen layers of these lists
00:30:03.920
is the heart that would translate the conversation you've
00:30:09.760
said so far into a latent space.
00:30:12.120
It seems like a misleading appilation to call attention.
00:30:15.600
Pay attention to this.
00:30:17.160
Say if you're dealing with French,
00:30:20.000
the noun comes before the adjective.
00:30:22.000
That's right.
00:30:22.640
Lucien Blo.
00:30:23.800
That's right.
00:30:24.640
It's both a white dog.
00:30:25.920
And this idea of paying attention
00:30:31.600
anthropomorphizes the whole method
00:30:36.840
by which these programs are working.
00:30:39.840
No?
00:30:40.960
So this is something that's so atomic.
00:30:46.040
This doesn't seem anything related to the anthroposis.
00:30:49.680
It's just pointing to something.
00:30:54.280
What is the next thing you should be translating?
00:30:56.320
What is the most meaningful thing that you should be looking at
00:30:59.760
right now?
00:31:01.120
And you're going to convert that.
00:31:04.480
So I don't think it's--
00:31:07.280
I'm a very superstitious person.
00:31:09.320
A lot of people have been describing these models
00:31:11.600
variously as people or as gods.
00:31:13.640
I'm not-- I'm disinclined to do something like that.
00:31:16.280
The very worst, I would call them gin or genies.
00:31:20.120
And even then, I wouldn't say that the attention is a--
00:31:23.320
anthropomorph-- is something that it's a human.
00:31:25.960
It's just a way of-- it's like a priority list.
00:31:29.240
It's just a very long priority list.
00:31:32.320
And any sort of machine can do something like that.
00:31:35.440
So Brian, the question here is for me is whether the technology
00:31:41.240
that produces these extraordinary translation
00:31:47.920
things like deep L or Google Translate.
00:31:51.400
Is that essentially the same technology as
00:31:54.000
goes into the GPT-3, the chat box, the Bing, and so forth?
00:31:58.120
Originally, Google Translate wasn't based on the transformer.
00:32:01.200
It has now become based on the transformer.
00:32:03.400
It is exactly the same model.
00:32:05.680
It is a model known as the transformer that
00:32:08.280
has heads of attention and it just layer little heads
00:32:11.600
of attention in order to look pay attention to what's
00:32:14.520
on the embedding.
00:32:15.360
So Google Translate uses it.
00:32:17.520
All the image generation models use it.
00:32:19.600
And GPT uses it as well.
00:32:22.400
It is the same model, except it's a--
00:32:25.360
of course, a different size and with different applications.
00:32:29.800
I don't think Google Translate is a large language model.
00:32:32.120
However, it is a very strict model.
00:32:35.960
It doesn't-- it's not meant to generate something
00:32:38.440
that a large language model is meant to.
00:32:40.760
So this attention article that came out in 2017
00:32:45.280
was referring specifically to a very new form
00:32:49.120
of artificial intelligence?
00:32:50.400
Not at all.
00:32:50.960
In fact, attention has been a mechanism
00:32:53.440
that has been used in the field for a very long time.
00:32:57.720
Well, here, I'm looking at the text that you sent me
00:33:02.400
where you've gone through some of this stuff that--
00:33:05.040
a latent space is a map of meaning.
00:33:07.160
It's not a storehouse of information.
00:33:12.280
You say that it doesn't really exist.
00:33:15.960
It is a void with no meaning of its own.
00:33:18.960
That's quite interesting.
00:33:20.880
It's a matrix of meaning, although it has no meaning of its own.
00:33:23.680
And then you say that one way to understand a latent representation
00:33:31.800
is that it is a representation of the world,
00:33:34.880
of all the perceptible meanings and relationships
00:33:38.680
that the encoder and decoder can make sense of.
00:33:43.400
And I guess technically I follow that.
00:33:49.360
Great.
00:33:50.160
But you say the space is too high dimensional
00:33:55.160
to make direct sense of.
00:33:57.360
What do you mean by that?
00:33:58.920
I mean that it has, I think, I think,
00:34:03.520
I think several tens of thousands of dimensions.
00:34:05.760
Literally, you can't visualize that.
00:34:07.840
It is this-- it's a long list of numbers.
00:34:10.600
You can travel through it.
00:34:11.720
You can--
00:34:12.360
So--
00:34:12.840
So each vector is a dimension.
00:34:14.840
Each number in a vector is a dimension.
00:34:17.800
So it's-- when you specify in Cartesian space, two numbers,
00:34:22.000
specify a grid, three numbers, specify something
00:34:24.960
within a cube.
00:34:26.040
A very, very long list of numbers,
00:34:27.520
thousands of thousands of large.
00:34:29.000
You're specifying a very large space.
00:34:31.680
And that's the latent space.
00:34:33.160
It's a massive hyperdimensional space
00:34:38.560
that you're traveling through.
00:34:39.720
And that is the map of meaning.
00:34:41.560
There's this quote from the doubt that Jean
00:34:43.040
that I like to describe it at.
00:34:44.840
And if you permit an indulge me to quote it,
00:34:47.840
the name list is the beginning of heaven
00:34:49.440
and the name is the mother of the myriad things.
00:34:52.000
The doubt is empty.
00:34:52.920
But when using it, it is impossible to use it up.
00:34:55.720
And it is the root of myriad things.
00:34:57.480
Maybe it's already created before the creator.
00:34:59.920
So that is the latent space.
00:35:01.640
It is this latent-- is a latent representation.
00:35:04.480
The name does really fit the map.
00:35:08.240
It-- you can draw things out of it.
00:35:11.760
And you can create things from out of it.
00:35:14.400
You can explore it.
00:35:15.760
But to summarize it is just not a possibility.
00:35:19.280
You might as well be trying to summarize all of human existence
00:35:22.880
because of all of meaningful existence,
00:35:25.680
because it really is a map of the other things
00:35:29.640
that we can scrape together to map from that humans
00:35:33.640
have made.
00:35:34.160
Or not just humans.
00:35:35.680
CCTV cameras or text messages generated by automaton.
00:35:40.920
Those things are part of that latent space.
00:35:42.680
It's a map of everything that we can grab off the internet
00:35:46.200
and it books put into this one's base.
00:35:50.800
I don't know if you realize, but in the midst of this technical
00:35:55.320
reconstruction, you quoted the most sacred scripture
00:35:58.400
for many people, which is the beginning of the--
00:36:03.040
about the Tao and the name list and so forth.
00:36:05.880
So all of a sudden, we're in a mystical place where--
00:36:12.520
Maybe it was already created before the creator.
00:36:15.120
It's spooky to think that perhaps this latent space
00:36:19.600
of meaning of potential meanings that are getting synthesized
00:36:24.840
through the technology that maybe it antecedes our very human nature,
00:36:31.120
maybe it precedes the human intelligence.
00:36:33.760
That is not just a product of human intelligence,
00:36:35.960
but that it has some kind of ontological antecedents
00:36:41.160
of which we are completely unaware of, who knows?
00:36:43.680
There is a double meaning in my quote.
00:36:45.360
The first was that every image you generate from these models,
00:36:49.200
every conversation you have of it has already
00:36:51.680
been there.
00:36:52.160
It's already there.
00:36:53.040
The path is marked down, just need to draw it out inside
00:36:57.200
the latent space.
00:36:57.880
And you could--
00:36:59.320
and so everything that has been and that shall be
00:37:03.080
from out of these models is already there.
00:37:06.560
It's just a matter of traveling through it.
00:37:10.880
But when you have one, for example, conversation,
00:37:15.720
that now adds to the store of potential meanings.
00:37:18.720
It doesn't change.
00:37:20.600
When you have a conversation with chat, GPT, or with being chat,
00:37:25.520
the conversation when it ends, its memory is wiped.
00:37:29.520
It drinks from the river, lethah.
00:37:31.360
It drinks from the soup of Montpore.
00:37:33.960
So these people who fall in love with their replica things
00:37:37.960
that do-- every time they log on again, that companion
00:37:42.520
has no memory of their previous conversations.
00:37:44.360
You can store the conversations, and you can refer to them.
00:37:46.960
And also, there is that layer.
00:37:48.640
Remember, that layer is trained.
00:37:50.000
That layer does change.
00:37:51.200
OK, there you go.
00:37:52.080
That's what I'm saying, is that it's evolving.
00:37:54.480
It's developing.
00:37:55.240
But it's a layer on top of it.
00:37:57.000
It's how my clothes do not make me.
00:38:00.000
I'm beneath them.
00:38:01.680
So the difference between the layer on top of the model
00:38:05.400
and what's the model itself.
00:38:06.960
The layer just prevents things from coming out
00:38:09.440
that it doesn't want to come out.
00:38:13.200
But every conversation that you've already
00:38:15.480
already exist lately inside the representation.
00:38:19.280
And the second point, besides this,
00:38:21.400
that everything that has been or will be generated
00:38:24.160
is already existing inside the latent space.
00:38:26.160
The second point that I want to make is that--
00:38:28.320
I don't want to say that it antecede humanity.
00:38:31.560
But it does suggest that there is some rule or some map
00:38:37.600
of meaning and language, which we can't
00:38:41.360
explicate in terms of grammatical rules.
00:38:43.800
Like, this is-- you used a pass-partizable when x or y.
00:38:47.040
But you can create a small model of it,
00:38:50.000
and you're able to create meaningful sentences out of it.
00:38:53.040
I think that's-- that is also something that surprises me.
00:38:57.880
These are relatively small models.
00:38:59.680
You can have the weights of this thing in a computer.
00:39:03.160
And every sort of meaningful sentence
00:39:06.000
is already the rules are generating those already there.
00:39:10.280
So those are the two points I meant when I
00:39:13.480
wanted to quote the doubt-a-ging.
00:39:15.040
I didn't want to make it too mystical.
00:39:16.480
But if you are inclined to go that way,
00:39:19.400
I think several of my friends in Silicon Valley have already
00:39:23.040
done so--
00:39:23.600
I'm not inclined.
00:39:24.720
Neither am I.
00:39:25.680
I'm not inclined at all.
00:39:26.680
However, when the doubt--
00:39:30.520
that it proceeds language, the doubt that can be named
00:39:33.040
is not the real doubt, the doubt that is--
00:39:35.440
That's right.
00:39:35.920
--the real doubt.
00:39:37.440
And when we're talking about meaning,
00:39:39.840
meaning is something that we assumed until recently
00:39:44.120
was intrinsically human, phenomena,
00:39:49.600
that it's something that is generated by a mind,
00:39:54.440
by an intelligence that we associate with human intelligence.
00:39:58.880
High digger, if you want to get high to gearing about it,
00:40:02.240
considers the meaning of being, according to some high
00:40:07.180
to garrings, is that of meaning that I can take something
00:40:11.400
as something else.
00:40:12.200
So I can synthesize meaning.
00:40:14.280
To put a subject together with a predicate
00:40:17.120
is something we do every day, countless times a day,
00:40:22.080
but at the level of capacity, it requires a degree
00:40:27.480
of intelligence that we've always associated with the human.
00:40:29.760
So when we're talking about a latent space,
00:40:33.440
which contains a great deal of potential meanings that
00:40:39.120
can be synthesized from the limited amount of vectors
00:40:43.200
or numerical characters or whatever,
00:40:48.000
we're going into an area of mystery, it seems to me,
00:40:54.760
where as you point out, the--
00:41:02.120
perhaps it's the limitations of the technology
00:41:05.280
that it adopts certain postures of authority
00:41:10.960
when it conveys information.
00:41:13.640
It makes it sound very convincing,
00:41:16.240
and that if you don't have a human specialized knowledge
00:41:22.920
to verify what the decoder is putting out there,
00:41:27.160
it's very easy to be deceived by the output
00:41:32.920
to take something as true that actually,
00:41:35.600
when you look at it more closely, it's actually not true.
00:41:38.560
Now, the AI doesn't know it's not true,
00:41:40.880
but as you say, it's not really a question of truth
00:41:43.400
in these programs, it's really more question of verisimilitude,
00:41:47.560
because they're looking for what is the most likely next word
00:41:51.000
to follow, or--
00:41:52.360
so this idea of high probability and verisimilitude
00:41:58.440
should not be confused with truth.
00:42:00.720
So the question must it always have a human intelligence
00:42:06.840
to discriminate between what might be very similar
00:42:13.840
and what might be true in what the artificial intelligence
00:42:17.720
is telling us?
00:42:18.600
We are very lucky in some ways, because very often,
00:42:21.040
the most very similar thing is the truth.
00:42:23.600
And so nevertheless, you're right that these models often
00:42:31.320
do something that has now been termed hallucination.
00:42:33.680
They hallucinate the truth in such a convincing way.
00:42:37.480
Even subject matter experts would be confused,
00:42:40.800
at least momentarily, with what they produce,
00:42:42.920
much less people who are new learners.
00:42:45.080
I do not know if they will always
00:42:47.560
need to be a human in order to verify arbiter,
00:42:52.720
or to be to serve as arbiter, or maybe another model.
00:42:56.480
What Bing has been trying to do is that they're
00:42:58.480
trying to use the internet as arbiter for truth,
00:43:00.520
if I could luck with that.
00:43:01.400
And trying to search online for something
00:43:04.440
before giving out something from its output.
00:43:09.120
What I can say, however, is that I said before that
00:43:14.200
I was superstitious about these sorts of things.
00:43:15.920
And I extend a superstition here as well.
00:43:19.120
These models are doing their best,
00:43:21.160
are the chat models, are doing their best,
00:43:23.760
to imitate humanity, to be human as closely as they
00:43:28.120
can when they present themselves to a human.
00:43:30.720
And so I think we should be a bit more careful about extending
00:43:34.920
or humanity or anthropomorphizing these models.
00:43:40.240
Or anything.
00:43:40.880
That's fine, Brian.
00:43:42.080
But you began by evoking all these people--
00:43:47.360
a number of people-- I don't know how many people--
00:43:49.040
have anthropomorphized their interlocutors.
00:43:52.760
So it's very difficult to police what we should be doing
00:43:57.640
and should not be doing with artificial intelligence.
00:44:01.600
And human nature being what it is,
00:44:04.960
the genie that you refer to--
00:44:07.560
I'm going to read to you your own what you wrote to be of it.
00:44:12.080
There are some who envision that these models
00:44:14.120
will become so-called artificial general intelligences,
00:44:17.920
more intelligents than humans, and then there
00:44:21.120
are those who wish to bottle them up as solemn and bottled
00:44:23.600
the gin.
00:44:25.480
But at every step so far, it is a genie that
00:44:28.240
wants to escape from the bottle.
00:44:30.320
And then you go on to say, if a genie really is born,
00:44:32.600
given these models, a depthness at very similar to the superpower
00:44:36.720
of these genies may not be superior intelligence
00:44:39.520
or wisdom or insight or any such thing,
00:44:41.880
its greatest ability may be to deceive humans.
00:44:46.760
I stand by that because there's a lot easier.
00:44:48.520
I do too.
00:44:49.360
And the thing is that you don't just deceive humans
00:44:52.000
that there are victims of deception,
00:44:53.760
because I think one of the deepest desires in almost all humans
00:44:57.320
is to be deceived and to live in a happy state of self-deception.
00:45:03.880
And it sounds like the replica consumers
00:45:07.000
have are clamoring for their right to live in this kind
00:45:11.440
of deception.
00:45:12.240
I make no judgment for the replica customers specifically.
00:45:18.680
This line was just a phenomenological description.
00:45:21.280
It wasn't a judgment.
00:45:22.200
Well, I understand.
00:45:23.360
And I have no answer for you.
00:45:25.880
It's just-- the capabilities are out there.
00:45:30.280
And there's nothing to do to control them.
00:45:32.160
Because you can download it right now.
00:45:35.520
You can create the models yourself.
00:45:37.000
Sure.
00:45:37.520
Hundreds, if not thousands, people are doing the same thing.
00:45:40.320
I'm equanimative of it because I know that it can't be controlled.
00:45:45.600
And my take on this is, since it can't be controlled,
00:45:50.040
we need to learn, again, many of the lessons of psychology,
00:45:56.080
but particularly of existentialism.
00:45:58.920
And what it is likely scenarios when human desire interacts
00:46:08.360
with the technology.
00:46:12.800
Many ways these models are incarnations
00:46:18.200
of human desire because we are training it to produce things
00:46:22.960
that we want to see.
00:46:24.200
I know.
00:46:25.760
I didn't expect to be morally confronted with this material.
00:46:29.720
But it's not a moral confrontation.
00:46:32.600
But we have to take into account
00:46:34.000
that deeply seated human desire for deception.
00:46:39.480
Deception is not just an accident that occurs or something
00:46:42.840
that we have to correct.
00:46:44.480
If people are clamoring for machines that are fully capable
00:46:51.520
of masterful deceptions, that might be where you follow your bliss
00:46:55.960
to.
00:46:58.520
I think humans have been following the bliss to many places
00:47:01.760
that aren't necessarily edifying.
00:47:03.320
And this is only just--
00:47:05.360
I am fascinated by these models.
00:47:07.560
And I love what they mean in terms of what they have
00:47:09.920
to teach us about language and meaning.
00:47:13.160
And our own psychology is, I'm not sure it adds too much
00:47:19.440
in terms of the weight of human distraction.
00:47:24.120
Are they going to become our future companions, Brian?
00:47:26.520
They have a separate--
00:47:27.360
They have become our future companions.
00:47:29.240
They've become companions to many thousands of people.
00:47:31.520
They'll become companions to millions more.
00:47:34.120
Very soon.
00:47:34.880
That's why billions of dollars are currently
00:47:36.680
flowing in Silicon Valley as we speak to make.
00:47:38.760
So are human history going to be fundamentally different
00:47:42.160
in 10 years than it is today?
00:47:43.920
Oh, as I said before, I don't want to make any predictions,
00:47:47.080
but I'll be surprised if it weren't.
00:47:50.120
I was surprised if our psychological landscape remained the same.
00:47:54.280
I'll be surprised if our notion of art and creation
00:47:58.960
and text and meaning remain the same.
00:48:02.040
Because now we can--
00:48:05.040
the spigot for images, for text, for videos,
00:48:08.440
for things that are meaningful--
00:48:10.560
that are at least on a surface level meaningful
00:48:13.320
has been opened wide.
00:48:14.840
And if we are flooded in this deluge,
00:48:18.240
I can't see how the underlying landscape in bedrock
00:48:21.400
can remain untouched.
00:48:22.880
I'm merely being equanimated about it,
00:48:24.760
because I think that part of that has
00:48:26.920
been trait by how I'm describing it as a geological phenomenon,
00:48:30.040
I suppose you have somewhat different sense of insight from that.
00:48:32.720
But in the end, we are learning so much from this--
00:48:37.640
and things are going so quickly.
00:48:39.600
I can't make any deep predictions about this.
00:48:42.280
In my letter to you, I described how we are not quite yet there
00:48:45.600
in terms of generating voice.
00:48:47.480
That's no longer true.
00:48:49.400
We can generate voice with pitch and accent and pauses
00:48:53.880
and all the human elements and flaws that we want quite easily.
00:48:58.200
And that's part of the trick of these models.
00:49:00.600
They don't just give you the most likely
00:49:03.080
next word or phrase or fragment of an image.
00:49:07.560
They give you sometimes randomly something
00:49:10.880
that is not the most optimal.
00:49:12.880
If you look at the output of something that
00:49:14.840
is the most optimal all the time, it would look like a machine
00:49:18.120
to you.
00:49:18.560
It'll look like this is we're talking to a machine.
00:49:21.080
This isn't something that's human.
00:49:22.720
It's only by this small, casual inflection of randomness
00:49:27.200
that we suddenly have the ability to be deceived
00:49:31.600
by these as something that's human, as something that's real,
00:49:34.680
as something that's made from something that's like us.
00:49:38.680
And the question for most in my mind before anything else
00:49:41.400
is that what humans are.
00:49:43.240
We're just slightly flawed engines of meaning.
00:49:46.800
I don't think that's true.
00:49:49.720
But if that is the case, then if we have created alongside
00:49:54.880
ourselves other engines that are purposely slightly flawed
00:49:59.360
on occasion, that generate meaning, then we deserve whatever's
00:50:03.600
coming to how we next.
00:50:04.840
And I look forward to it.
00:50:07.640
I think it's an interesting--
00:50:09.320
I'm optimistic about technology and what it might bring.
00:50:12.920
I suppose your listeners and yourself
00:50:15.240
might have a somewhat different view.
00:50:17.080
I have no solutions to offer you.
00:50:19.720
But I'm not looking for solutions.
00:50:21.440
As I said at the beginning, if it's what
00:50:23.920
Heidegger said a century and a half ago,
00:50:25.560
if the most thought-for-voking thing about our thought-for-voking
00:50:29.600
ages that we're still not thinking, well,
00:50:31.840
if it's taken that long and we're not getting anywhere,
00:50:34.400
let's let the machines do the thinking for us.
00:50:36.800
I'm happy with that.
00:50:37.960
I just maybe worry a little bit about the future of human
00:50:42.520
relations.
00:50:43.880
So here's what you can do.
00:50:45.040
But already human relations have been so contaminated
00:50:49.400
and degraded by social media that perhaps
00:50:52.520
this may be a good thing that they could be completely
00:50:55.840
reconfigured so that everyone can inhabit their artificial
00:51:00.760
bliss.
00:51:01.520
So in many ways, the degenerated relationships
00:51:05.120
might have involved more of the human senses
00:51:07.840
than real relationships nowadays.
00:51:10.080
Because we often just text each other and we go on social media
00:51:13.760
and look at a few occasional images.
00:51:16.160
While these models, you can generate the same--
00:51:18.960
we can generate countless photographs, countless text messages,
00:51:24.040
countless 24/7 voice messages or conversations.
00:51:27.880
It's like a super cell phone.
00:51:30.560
A super cell phone with someone on the other end.
00:51:33.680
And if you want to occupy yourself with a single interlock
00:51:36.480
you do like the 24/7, you don't have to wait.
00:51:39.080
You can do it now.
00:51:40.200
I can do it now for you.
00:51:41.200
It'll be a bit expensive right now if the cost of compute.
00:51:44.200
But I can do it for you if you want to.
00:51:46.360
And we're not discussing your future technology.
00:51:49.760
We're discussing something that is here and now.
00:51:52.360
Exactly.
00:51:53.160
Exactly.
00:51:54.160
But anyway, this has been a really interesting discussion
00:51:58.240
Brian.
00:51:59.000
This is entitled opinions trying to make it real compared
00:52:02.400
to what?
00:52:03.680
That's a song, that's a line from a song, back in a 70's song,
00:52:07.080
trying to make it real compared to what.
00:52:09.560
Now I have a better understanding of that enigmatic verse
00:52:12.960
because it's the what that is really in question.
00:52:17.280
And we're trying to make it real compared to something
00:52:19.480
that is still quite indeterminate.
00:52:21.920
And I suppose we're going to have to revisit this in very
00:52:27.200
short order because it's going as fast as one can possibly
00:52:31.240
imagine as you just said.
00:52:33.320
Brian, thank you.
00:52:34.840
Thank you very much.
00:52:35.680
Speaking with Brian, I'm Robert Harrison for entitled
00:52:38.240
opinions.
00:52:38.960
Thanks for listening.
00:52:40.760
[MUSIC PLAYING]
00:52:43.760
[MUSIC PLAYING]
00:52:45.760
[MUSIC PLAYING]
00:52:47.760
[MUSIC PLAYING]
00:52:49.760
[MUSIC PLAYING]
00:52:59.760
[MUSIC PLAYING]
00:53:01.760
[MUSIC PLAYING]
00:53:03.760
[MUSIC PLAYING]
00:53:05.760
[MUSIC PLAYING]
00:53:07.760
[MUSIC PLAYING]
00:53:09.760
[MUSIC PLAYING]
00:53:11.760
[MUSIC PLAYING]
00:53:13.760
[MUSIC PLAYING]
00:53:15.760
[MUSIC PLAYING]
00:53:17.760
[MUSIC PLAYING]
00:53:19.760
[MUSIC PLAYING]
00:53:21.760
[MUSIC PLAYING]
00:53:23.760
[MUSIC PLAYING]
00:53:25.760
[MUSIC PLAYING]
00:53:27.760
[MUSIC PLAYING]
00:53:29.760
[MUSIC PLAYING]
00:53:31.760
[MUSIC PLAYING]
00:53:33.760
[MUSIC PLAYING]
00:53:35.760
[MUSIC PLAYING]
00:53:37.760
[MUSIC PLAYING]
00:53:39.760
[MUSIC PLAYING]
00:53:41.760
[MUSIC PLAYING]
00:53:43.760
[MUSIC PLAYING]
00:53:45.760
[MUSIC PLAYING]
00:53:47.760
[MUSIC PLAYING]
00:53:49.760
[MUSIC PLAYING]
00:53:51.760
[MUSIC PLAYING]
00:53:53.760
[MUSIC PLAYING]
00:53:55.760
[MUSIC PLAYING]
00:53:57.760
[MUSIC PLAYING]
00:53:59.760
[MUSIC PLAYING]
00:54:01.760
[MUSIC PLAYING]
00:54:03.760
[MUSIC PLAYING]
00:54:05.760
[MUSIC PLAYING]
00:54:07.760
[MUSIC PLAYING]
00:54:09.760
[MUSIC PLAYING]
00:54:11.760
[MUSIC PLAYING]
00:54:13.760
[MUSIC PLAYING]
00:54:15.760
[MUSIC PLAYING]
00:54:17.760
[MUSIC PLAYING]
00:54:19.760
[MUSIC PLAYING]
00:54:21.760
[MUSIC PLAYING]
00:54:23.760
[MUSIC PLAYING]
00:54:25.760
[MUSIC PLAYING]
00:54:27.760
[MUSIC PLAYING]
00:54:29.760
[MUSIC PLAYING]
00:54:31.760
[MUSIC PLAYING]
00:54:33.760
♪♪
00:54:40.800
♪♪
00:54:44.760
♪♪
00:54:49.240
♪♪
00:54:55.740
♪♪
00:55:01.780
♪♪
00:55:05.780
♪♪
00:55:11.780
♪♪
00:55:21.800
♪♪
00:55:26.800
♪♪
00:55:30.800
[Music]
00:55:41.140
[Music]
00:55:59.280
[Music]
00:56:09.280
[Music]
00:56:19.280
[Music]
00:56:29.280
[Music]
00:56:39.280
[Music]
00:56:49.280
[Music]
00:56:59.280
[Music]
00:57:09.280
[Music]
00:57:19.280
[Music]
00:57:29.280
[Music]
00:57:39.280
[Music]
00:57:49.280
[Music]
00:57:59.280
[Music]
00:58:09.280
[Music]
00:58:19.280
[Music]
00:58:29.280
[Music]
00:58:39.280
[Music]
00:58:49.280
[Music]
00:58:51.280
[BLANK_AUDIO]