Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2199 Thanks!
To this talk on Couch Stone
to follow this talk, the
rise and fall of social
research will be presented
from Florian Gove.
It's not my prediction
to shower on to the desert talk,
but you resist.
So it's not because it's
hvad, so it's not English fiction.
I would switch back to English now.
This is a talk presented
by Florian Calvert's from the known Beck
Institute of Technology, and he
will talk about social bots
in recent years.
We have all observed the phenomenon
of social bots
accompanying different media events
and also the media had like
waves also showing those social
bots activities
in the social media
four years ago.
Already, Kyle
found significant shortcomings
in the research on social
bots, and
Florian will now deeper show
us why the current research is
flawed or even deeply flawed in
this specific field.
So have lots of fun in the talk.
The rise and fall of social bot research.
Thank you very much for the introduction.
So unless you've been
living under a rock
somewhere in the desert, you've
seen headlines like these
are just randomly selected a couple
of headlines through Google news.
So we have
been told that
bots on Twitter are amplifying
conspiracy theories or
they are spreading disinformation
before the election, or they even
half of the account tweeting
about coronavirus are likely.
Bots and bots are
a major source of climate
disinformation, and
they there
are hundreds of thousands of
them, and they even interfere in Canadian
politics and in British
politics.
And Donald Trump has millions
of bots supporting
him.
And they are a danger
to democracy, and
they are poisoning democracy
and they are damaging democracy.
They supported Boris Johnson
and they are spreading fake false
claims about the bushfires
in Australia, and
they are even trying to influence
elections in Germany.
Maybe so.
This is a
man. You will find hundreds of
headlines like these if you just
search for the keyword bots
in Google news.
And the behind lot of many of
these headlines, there is some academic
research. Most famously, the paper
by Ferrara and
colleagues called the rise of
social bots, which inspired
the title of this talk.
And while
the research papers
that claim to
prove some influence off social bots
on the US presidential elections
and the spread of fake news by
social bots, and this is a paper
one of many papers by Philip
Howard, who claimed that
bots had interfered to the
Brexit referendum.
But there are many other issues
that are largely influenced
by social bots.
So, for example, health communication
vaccine.
The vaccine discussion this is a paper
from 2018.
So before the COVID pandemic,
and this is the
more recent paper boats claim
that bots are interfering
the discourse about vaccines,
but they're also deferring the
discoveries about
climate and generally
any low credibility.
Content on Twitter is
actually spread by social
bots, so by the end of this talk, you
will hopefully learn
that the low credibility
content is actually inside these papers.
So what's the social, but
if we look at
one of many slightly different
definitions, this is one by
Ferrara. So he's the one
who basically created
the social bot hype, so we
have to believe him.
So if we listen to him, Social Buzz,
our automated accounts that
use artificial intelligence to steer
discussions and promote specific
ideas of products on social media
sites such as Twitter and Facebook.
The typical social media users browsing
their feeds social bots may go unnoticed
as they are designed to resemble
the appearance of human users,
and they behave online in a manner
similar to humans.
So that's a pretty clear definition
of what social bots are supposed
to be.
If basically,
if you look at all definitions
of social parts and academic papers,
this is a pretty precise picture
of what the general consensus
on social bonds is.
So typically they are assumed to be
political accounts influencing
political discussions, and
they have to be automated.
And they are
accounts with human fake profiles,
pretending to be humans and somehow
behaving in a human fashion.
So obviously, at the intersection between
political accounts and fake accounts,
you would find human paid
trolls, for example, so people who are
paid to to
take part in political discussions.
Also, you can easily find automated
political accounts.
So, for example, in the Fox News Feed,
what is political?
And it's somehow automated, so
automatically.
Newly published articles
on the website, for example, very
get tweeted on Twitter so that some kind
of automation.
So this would be automated political
accounts.
And of course, there are.
Also, I
can at least suspect that a lot of
automated accounts with
fake human profiles
are used with porn or
crypto scams.
So basically to scam individual
users on Twitter.
But social bots, they combine these
three properties.
That's generally the idea of
social buzz and social parts are
at least attributed was
whether it was
there, that somehow
people believe that they are able to
produce content on their own.
So are somehow autonomous.
And the
very often the the the
word artificial intelligence
are mentioned in context with social
bots.
So if Twitter is countered
with crowded with social, but
where the hell is one, that's my
starting point for my interest in social
bots, so I felt kind of gaslighting.
Social media is telling me that
there are millions of social bots
on Twitter, and I've spent almost a
decade on Twitter and I've never seen as
social.
But so that's the point
where that got me interested.
I have a background in a I
have pattern recognition.
I've been building conversational dialog
systems for many years, which
would be called Chat Bot today,
and I've never
really believed the idea that
at the current state of the art, you
could build political
bots that could interfere
in current political discussions
that seem kind of strange and implausible
to me. So if they exist,
how do these things work?
So I wanted to find one to take a look at
the social and actual social bots
and an actual social bot, and
I started to search for an
actual social bot in late 2018.
So three years ago.
So since then, I've studied dozens of
scientific papers and media reports
about alleged social bots
are basically. I read all
newspaper articles, media
media reports about social bots
that Google news could come
up with in various different countries
and AI and
its hundreds of Twitter account that had
been accused of being Social Pass, for
example, in newspapers where actual
social bot accounts were named.
And of course, I asked a number
of social researchers for an example
of an actual social but
and I've performed experiments on
using the Twitter API, which involves
hundreds of thousands of accounts.
Some of these experiments, together with
Michael Crile, who was
mentioned in the introduction, who has
been working on this problem
one or two years longer than me.
So the total number of
social bots I found until
now, using all these different
approaches is zero.
So I couldn't find a single social,
but that fits the definition of
Emilio Ferrara.
So that seems kind of strange.
So where does this
apparent mismatch come
from?
In order to understand this, you have to
take a look at the
mythological method
as a theological basic, a basis
of these scientific papers.
And basically, it boils
down to two different methods.
The first method is the so-called Oxford
Criterion's, named after
the University of Oxford.
Philip Howard, who
defined
heavy automation as
accounts that post at least 50
times a day.
So basically, he counted
the tweeting frequency of different
accounts. Everybody, which we did more
than 50 times a day, was
defined heavily automated
and referred to as a bot.
So that seems kind of strange.
And the second approach and the more
common approach nowadays is
an automated tool called a bottom
feeder, which is
you can use it as the public
website where you can enter a Twitter
account and press a button and then
you get a score somewhere between zero
and five or between zero
and one, depending on the scale you use.
And it basically and
the researchers typically use
the API. They feed a long list
of Twitter accounts into
about a meter for each account.
They receive a bot score and
then they use a threshold typically 50
percent of zero point five
on a scale of zero to one.
And if the bot score is larger than
0.5, then they simply
so it's a bot
similar. There are similar tools that
basically work in the same manner, but
bottom meter is by far the most popular
tool of this kind.
So in both cases, no manual checks
are performed, whether the accounts
identified using these rather
crude methods are bots or not.
And even more interestingly, the names
of the user. These are these alleged
posts are not published.
You will won't find any bots
in all of this dozens of research papers.
And if you ask the authors of these
papers for the
actual bus, they are routinely routinely
withheld so they won't give you
the data.
They will give you some flimsy
excuses about data
protection laws and privacy.
Or they somehow accidentally
deleted the data or they
didn't store the data or whatever.
So, so pretty much like
if you ask the kind
of the dog ate my homework
type of excuses.
OK. Let's take a slightly closer
look at the way these researchers
produce headlines, so.
So that's basically a certain
perfect recipe for for
creating headline in The
New York Times.
So you decide
which political issue you might be
interested in? Really?
You can come up with any topic
you might be interested in. Let's say
you're interested in gaming.
So you want you want to
create a headline about gaming and
social bots in the New York Times.
So what do you do?
You look for
gaming related
keywords on Twitter,
and you identify
accounts that that
tweet about gaming.
Then you can come up with a
long list of user IDs or
Twitter accounts that tweet about
gaming. Then you feed them into
barometer. Use your threshold zero
point five, for example, and you
get a list of parts and a list
of humans.
And then the New York Times will publish
your headline research.
Nearly half of accounts tweeting about
gaming are bots.
OK, so now skeptical people like
me might want to
take a look at the actual bots to
verify your claims.
So obviously it's more useful to hide
this data, but journalists
don't seem to be interested in the actual
bots. They have never questioned the
validity validity of these claims,
even without looking at actual
bots. So they will believe
you and they will publish this in
the New York Times.
So the first
criteria that you use
for this type of papers is the
Oxford Criterion's of the 50 tweets
per day credit here, and it's pretty easy
to to show
that it's not very useful.
So you find very
large Twitter accounts.
Celebrities who tweet more than
50 tweets per day, like journalist
Glenn Greenwald or
Cory Doctorow, was a
blogger and author with a very large
following. Or you and his cars,
for example, a German member of
Parliament who tweeted up to 300 tweets
per day.
Donald Trump himself tweeted more
than 50 tweets per day on six days
in in
2020.
So, all right, at least in
a very short period of time in 2020.
So political activists
who have more time than presidents
or MPs obviously tweet
a lot more.
My favorite account on Twitter is
actually a guy called Eric Luxottica.
He's a Canadian engineer and
retired college instructor.
He has been tweeting
it writes about 300 tweets per
day over a period of several months.
And he's always ranting about liberals
in Canada, and he he hates Justin
Trudeau.
Another interesting experiment I
performed with there
was about the K-pop band Beat
is extremely popular,
and a German radio reporter
made some derogatory remarks
about the bands and the K-pop
band.
The other bits of
fans were extremely angry
about this, and
they created more than three million
tweets in a period of four days
with the hashtag Bae and three racist
Band three is the the
radio program involved.
And they
so 500 of those accounts,
500 different accounts.
Each of them fulfilled the
Oxford criteria, and
the maximum was three hundred forty four
tweets on average per day.
And I looked at the top ranking
accounts in this list, and
none of them showed any signs of
automation. Those were real people.
They were tweeting pictures
of themselves, and
everything must come because it's
coming from different countries.
Those are real people, not bots.
Another easy way.
The second criterion was criterion that's
often used for the bot, which
is the other major
bottom.
You can create the link
is given at the bottom of the page,
and you can simply enter
the name of a Twitter account and the bot
immutable return score.
So, for example, those are all
but according to by the media.
So, for example, Tim Cook is a bot
the pope is about.
The new German chancellor Olaf Scholz is
a bot.
Former chess champion
Garry Kasparov is a bot.
Auschwitz Museum is a bot.
Joe Biden is about
the Brazilian President Jair
Bolsonaro is a bot.
The very well
respected German virologist
Christian Dustan is about, but
also his counterpart to an
economy retired economist
who is basically doesn't believe
in the coronavirus.
He he is also
a bot. So basically,
if you just randomly enter a code name,
chances are very high that you will run
into bots, according to
a Tomita.
Similar analysis
have been performed by Michael
Kyle in 2018.
He fed all the Twitter accounts
of members
of US Congress into Bot A Meta,
and you can see that basically
it's a it behaves like a random
number generator with a normal
distribution pretty much around the
threshold that's usually used
for for
discriminating between bots and humans.
So
if we want to use the same approach,
the the social but researchers
used for identifying, but we
can find interesting stuff, we could find
UFOs or we can
find any kind
of stuff we want to find
and what to show, how we could find
unicorns.
Let's say we want to find you.
We want to find unicorns in Africa.
So we would train a
kind of photo trap for
unicorns.
And and what
we want to deploy that in the Serengeti
to find the actual unicorns.
So for training the unicorn, we have
the same problem.
The social part researchers have,
we don't have actual unicorns to train
our our
classifier on. So we would use a
unicorn like
animals like a white horse, for example.
Or we would use toy unicorns
as a training set for
unicorns.
And we would use some other animals
like a cow, a pig, a cat and the dog
as unicorns.
And then we would train our classifier
on this data, and then we would move our
trained classifier to Africa,
and then we would
find lots of unicorns.
So for example, in this case, the zebra
would get right.
It would get a seventy five percent
unicorn score because it looks quite
similar to this white horse.
And also, this white egret
would get a very high unicorn score
because it has this pointy beak and
it's white, so it looks a lot
like a unicorn.
So if you want to convince people that
you actually found lots of
unicorns in Africa, some people might
want to take a look at your pictures, but
obviously that would be bad
for your claim.
So obviously you would want
to hide the actual
unicorns you would find.
You would simply rely
on the results of your unicorn classifier
and people who who complain.
You would tell them why we have almost
100 percent recognition rate
in our on our training
data. So you believe us?
That's exactly the way the border media
people argue.
If we take a closer look at how
this kind of classifiers work,
well, basically the
statistical classifications of neural
networks are in the case of
part a meta. It's a it's a
random forest classifier, and
any of these classifiers basically
places each of the observations
in a feature space.
All the dimensions might be the the
whiteness and the pointing is
of the animals.
And so the unicorns
and the non unicorns,
they form some kind of cluster
in the features space, ideally.
And the training, the
classifier would create a class boundary
that can
perfectly discriminate
between unicorns and unicorns
on your training data or in very
similar data.
But in reality, if you now deploy
your classifier to
Africa or to the real world,
you would find a lot of
unicorns and lots of non unicorns,
depending on the the class
boundary chosen by the classifier.
So it would still produce 100 percent
recognition rate on your
on your training
data. But it seems far
stretch to believe that this is actually
a unicorn.
This is a unicorn.
So if you now remove the training data,
you see, that's basically you get some
kind of unicorns, money, unicorns,
depending on the class boundary you
trade. But it seems
outright ridiculous to believe
that the animals labeled as unicorns
will be actual unicorns, or even
that the relative amount of the unicorns
that result from this approach can serve
as an approximation for
the true prevalence of unicorns.
But that's exactly the way those
but researchers argue.
So there's another problem, if you think
that what I told you so far is bad for
social bad research.
It's even worse
because
no, you I have to
explain some fundamental ideas
of pattern recognition.
So in,
let's first take a look at the approach.
So the idea people
use to come up with claims like
50 percent or 15 percent of the accounts
are, but they feed the list to
the the classifier and
they come up with five parts and eight
nine dots, for example.
And now they calculate the relative
amounts of votes, like thirty eight point
five percent in this case.
Now they claim that thirty
eight point five percent of the
of the accounts are
bots, which is the so
called a priori probability for
bots or the prevalence of
plots or
the prior probability.
And the the fun part now is
that this probability is already
part of the decision process
that's used by the classifier.
So, for example, a
random forest classifier like but a meta
will include this probability
and we'll learn this probability based
on the training data.
So if if
the buzzer beater people use 50 percent
bots and 50 percent non bots
to train by the meta, the classifier
will have incorporated the information
that 50 percent of the accounts
are bots and the optimal
classifier with the lowest possible
recognition rate.
And this is what the training process is
trying to achieve will
always adjust the
threshold in a way that it will
will conform to
this equation in order to produce
good results on the training data
on similar. But this is circular
reasoning. So to estimate
the number of bots using this approach,
you have to know it beforehand.
So this is it doesn't make sense.
It's simply impossible to use a
statistical classifier to
to determine the
prevalence of bots or of unicorns
or whatever.
In reality, it with the media,
there's typically a curve like this, so
depending on the threshold you choose,
you get a percentage of bots
that are classified as a percentage of
bots or a percentage of accounts
classified as bots.
So the Spanish threshold
many people use is 50 percent.
But if the account of bots is,
the number of bots is too high to
be credible.
The researchers typically choose
a higher threshold, which would give
you a lower percentage of.
But if you want more
bots, you can choose a lower threshold.
If you want to want to
feel bots, you can choose a higher
threshold. So basically, the
results the researchers present
are based on their guess of what
the prevalence of.
But I'd be the results
scan.
It does not really make
sense.
So far, we have falsified the methodology
used by social batteries, such as using
examples of some accounts
fitted into the different method, and
also we have shown that it can't even
work in theory.
So there's is one secret
that's remaining what do the bots look
like that these researchers claim to
find in the studies?
So the accounts they prefer to hide when
we ask them for the list of
these accounts, and we've spent quite a
lot of time trying
to get behind
this question and to find out what did
these accounts look like
they might find in their studies.
So the first study we took a closer look
at was about the German
election in 2017.
The authors claim that
the share of social bots
among the followers of seven German
parties increased to
from seven to 0.1 percent
to nine point one nine percent
during the election campaigns.
So basically, they claim that
roughly 10 percent of
eight hundred and thirty eight
thousand accounts following the largest
home parties were social bots,
more than 80000 accounts
following German parties social
bots. That's interesting.
So I asked the first author
for examples of actual social
bots, and he gave me four
accounts which didn't remotely look like
social bots. So that seemed kind of
strange. And he also seemed to be fully
aware that there are no social bots
in his sample.
So we tried to replicate
this study a couple of
months later. So we basically we
downloaded all the followers of the
German parties and we fed them
through about a meter.
We used the same threshold.
These authors had used and we
for the first time, we had a look at the
actual bots that show
up in this type of study
so that we can look at.
We found 260000, but
we took a random sample from these
in this case, 100 99 accounts.
I spent a lot of time looking at.
If those accounts very carefully,
those are they're ranked by the bots
because this is close to the
basically the maximum value
about a meter can produce.
And those
are the accounts you can check them if
you want, if you believe any of those.
But we couldn't find any bots.
So, for example, a dance cafe
in Erfurt, this
guy here or we
found a forest in southern Bavaria
or this guy complaining about the
noise produced by motorbikes in
Gelsenkirchen.
And it took me a
quick Google search.
I could find out exactly at this stage.
There was a motorcycle event in
Gelsenkirchen, at the at the amphitheater
in Gelsenkirchen, so those are real
people with real ears complaining
about real noise.
Many of these accounts are very inactive,
so basically the opposite
of what was social.
But researchers make you believe
about the properties
of social bus, so we couldn't find
a single social bot in
this list. Add some more of these
accounts.
So, for example,
probably a Cairo below, probably.
She asked Microsoft for
for help and
the one and only tweet tweet she
she produced with this account.
And it resulted in
but score close to five
like four point four five six
four. So close to the maximum.
Awesome.
A blogger, a doctor,
a politician,
a former regional chairman of the
political party de-link.
She got a score of four point four and a
scale of zero point five.
No, but.
The second study we
looked at closely.
A claim that
they found
a very large enough
number of social bots in
the spreading vaccine critical
information.
Two hundred thousand social bots
and I asked and have done the first
author for the list the
fall of its bots, and
he actually gave me the list
for the very first time.
We had the precise list that was used
to produce this kind of
paper. Also again,
we sampled
a random sample of 121
in this case of the 200000.
But and I took a very close
look at these one hundred and twenty one
accounts, and we could not find a single
social bots.
And those are just nine of the 120
accounts I analyzed.
You can see those are doctors.
This is a very
high ranking
doctor at the World Health Organization
Organization, a medical student from
Rwanda, a professor
from Texas.
He's
working at a medical technology company
in Nigeria, medical
intern from Saudi
Arabia, a health professional from
California. I think a postdoctoral
researcher and
on a viruses,
the data
operator, I.T.
professional from India.
So people from across
the world, many tweeting under the real
name, many with oppressive
professional credentials and are simply
these social badree such as simply
claim those are all.
But though this seems
kind of strange.
So if we take a closer look at this list
R120 when one occurs
a lot, you can check
the accounts. They are all in our paper.
So, for example, the World Association
for Medical Law was rated about or
this pastor from Boston
was rated R, but but
none of them seem remotely automated.
This is one of my favorites.
Taylor Timothy
tweets by a girl called Taylor, who
is seriously in love with a guy called
Timothy Moss, with whom she is engaged.
So those are two of her tweets,
and the score is four point three
nine for close to maximum
or this low reject Laura James.
She even tweeted Picture lots of pictures
of herself. She tweets a lot about
UNICEF. She was rated R,
but.
OK, so
in this year, two
other papers came out two interesting
papers.
OK.
OK, so there were some problems with the
slides, but the problems seem to be
solved. So the
in two thousand twenty
one two papers were published
where researchers
use but a major and they
actually checked the
list off of.
But they they got very similar
to what we actually did.
And in this, obviously this
in this paper, they took a very
close look at 500
accounts and they fed them into about a
meter and they also manually
checked them if they could be automated
or if they looked automated.
And it turned out that the major label
roughly six percent of the uses
social, but but none of them
account looked like a remotely
automated, so they simply ignored
the bottom meter labels
as rubbish and
another paper very similar.
It was just the preprint was published
a few weeks ago,
and they also use about a meter.
They used the standard threshold
of zero point five, and
so eighty six accounts were
labeled
bots.
But actually none of these was a social,
but only one of them
was automated.
It's automatically shared article from an
external personal blog.
So does that doesn't make it the social
part.
It might have been automated, so we
ignore people basically
turned out that the barometer scores
are absolutely
unreliable and have nothing to do
with reality.
So now we've seen.
So basically, we can ignore the
existing social but research based
on the Oxford Criterium,
or we can ignore the research based on
by the media.
But there are some other methods
being used where people claim
to find social buzz.
So this is a headline from the Times
of London Army of Twitter.
Bots follow top politicians such
as Nicola Sturgeon and
a few others.
And when you ask the question
of the
people, how you how do you know
it's but that is actually
this guy?
He.
Sorry.
OK, still some problems with.
Problems with the slide screen, but
hopefully, I think wouldn't change
anything now. OK, so
this guy explained on Twitter how it
worked. We downloaded the latest one
K follows of seven politicians and
checked numbers of users with
exactly eight digits and usernames.
So these guys assume that,
but our usernames with
eight digits like Mike two six
four eight one five six four So
they assume that the bot many people
believe this, but it's actually complete
bullshit because names
like these are automatically assigned to
newly created accounts by Twitter, at
least since 2017.
So you can if you just create a new
account with a common name,
you will automatically get assigned a
name with these h digits
digits attached.
So if you are familiar with Twitter, you
will. You have a possibility to change
the name of your Twitter account, but
most people aren't aware of this feature,
so they simply they
look like but for the rest
of their days on Twitter.
What about accounts posting the same
stuff very often claims are made about
bots with screenshot like
the one on like this,
so different accounts posting more
or less the same message.
And typically those are so-called
copy past us and where people
ironically make
fun of them
or tweet they don't like by
posting it over and over again.
Sometimes they they
post it over and over
again, and ironically or different
accounts post the same message.
It's pretty hard to to understand
if you're not familiar with the meme
meme culture.
On Twitter, this guy, for example,
all of these accounts are on
the same side of the discussion,
but he doesn't get the joke.
So he he believes those are bots.
Actually, those are his friends in this
this political issue.
And then you find a report like
the one that you publish yearly
by the computational propaganda
research project in the Oxford
Internet Institute, and they
give you, you know, colorful lists
of countries where bots
had been identified
in the year.
And obviously the the
whole list, if you look, take a close
look. It's not based on actual bots,
but it's based on news reports
about actual bots.
So that's a kind of a.
In a process
that will continue to go on
forever because
newspapers will report about
this report and the reports about
the report will be will
lead to further bot
bots in or bot
claims in the newspaper.
So
there's a very nice claim by
some journalists in a very different
context that describes
what's going on.
If I found an institute to combat
public health hazards caused by hedgehog
bites and provide this institute
with staff positions, then I will get
a study every year warning
about the growing dangers posed by
aggressive hedgehogs.
Anything else would be rather stupid
on the part of the institute's staff.
So that's pretty much describes what's
going on with all these
research institutes
involved in investigating
social, but.
So to conclude social, but
research is fundamentally based on
the misclassification of human users
as social, but.
A malicious interactive social bots don't
seem to exist at all.
After years of studying the output of
social bot research, we have yet to see a
single credible example
of a social bot.
And some of the researchers involved in
this type of research violate basic
academic standards, though
they make claims that are not supported
by data, and they withhold data
which would allow us to verify
their claims.
And the public has been misled over
the non-issue of social bots
for years.
So, Michael Krol and me, we
published we wrote all this down
in paper.
The name is the rise and fall of social
board research. You can download
it at the link on the slide
and you it has all the
lists of plots in
the appendix.
You can take a closer look and
try to find social, but if you're
in those lists, if you if you like.
Thank you very much for your attention.
Well, thanks so much, Florian,
and yeah, for
your talk, and it seems to me that
indeed the field of social
research is in a very problematic
state. To me, it also seems
just for the sake of realistic thinking
and discussion.
The work you do and Michelle Recoilless
doing is really important for all
of us. So we and
maybe you or all the scientists
need to provide a clear base for
discussion and media
coverage such that those topics
are actually kind of talked
about reasonably.
So we collected some questions
from our audience, and
maybe I'll go through these questions
one by one. The first one would be,
yes, you talked about those social bots
and like, what is it about and what is
not a bot? But shouldn't the question
rather be about the effect of
such accounts rather than whether
they are click workers
people, which too much time in their
hands, paid influencers
or automated accounts?
So saying being a bot or
being acting
in kind of those political ways or
acting like a bot, isn't that actually
the same or like the effect is important?
Would you agree with that?
Well, we are not
the ones who made the claims that these
automated bots exist.
So first of all, before we talk about
the effects of stuff, we would
have to first.
If you're talking, do they exist at all?
Though many bots about research can take
paper. Let's talk about the facts, not
about the existence, because
but first of all, we have to get rid of
this belief, which I think it's a
it's a it's a conspiracy theory.
It's a fairy tale that these
social bots, these automated accounts
exist and many people actually believe
it. And typically, when
people claim
your robot, you're no, you're about.
This is you're about to meet us, go, no,
I don't. I'm not about, but
you're a bot. So these discussions on
Twitter keep going on, and they are based
on this
belief and this ridiculous
conspiracy theory.
So first of all,
Michael and B, we're trying to get
to make gather let
people have a realistic view of
what's actually going on on Twitter.
The question, whether accounts
are maybe paid
by some malicious
actors to
influence discussions is a very different
question. So we have been looking at
automation. I'm I'm a computer scientist
as a my my I'm not
the right person to me to analyze
the motivations, why people
might be tweeting
on Twitter a lot.
But my impression is that
the very active accounts on Twitter, like
the the Canadian account
mentioned as a target relative to I
don't believe he's a paid actor.
He's he is a retired professor
and he engineering.
He is a lot of money.
He is certainly not paid for
being political.
Very active on Twitter.
He is acting out of intrinsic
motivation.
Same holds for all these five hundred
accounts that
tweeted hundreds of times
about about the the
K-pop band.
They are not paid actors.
They are people who really who
are angry about stuff or people who
are really trying to to
to to change things by being
very active on Twitter.
So my general impression is that people
have a tendency of overestimating
the problem of malicious
fake accounts, people.
The default assumption should be actually
those are real people with real,
real intrinsic motivation.
Hmm.
Another question we have is
in your slides, you had this circular
reasoning example or
maybe this kind of fundamental
flaw of circular reasoning
and the question would be from this
person, is it maybe
like circular reasoning, maybe not
rather resonance, which
would be a valid method to get the
results? Kind of that
the results actually resonate with
their
with the real world, such that
in the end, you would kind of confirm
that there is something resonating
with your findings.
It's a bit hard to ask this question, but
maybe you got it. I think I think I get,
I get I believe I got to get
one of the questions going out.
So maybe the the idea would be
to sell it
to start to use this business, you would
have to have a guess for the other, the
prior probability of butts so
you could use this iteratively.
It hopefully ends up and
at the real value.
But actually, I believe it's certainly
either it will explode and
go to 100 percent or it will go to zero
from all my, I say, twenty
five experience in pattern recognition.
I don't think that approach
is remotely capable of
estimating the real probability
by using it iteratively
or something like that.
No, I don't think that's that will work.
The interesting part about this
this equation is any
kind of classifier use.
It is so even humans use it.
So if if I have
the if my task
is to label bots
or whatever in a list of accounts
and I have some expectation
of the real prevalence
of bots, I will include this information.
So how how
so? Basically, the the
problem is,
if we don't know the prevalence
can, how can we even estimated?
And the only way to do it was it
would be if these two
posterior probabilities, which basically
tell us, how much does do these
features?
How much does these do these features?
So, for example, the tweeting read, look
like it. But if these two distributions
are very different.
So for example, if bots tweet 10000
times a day and humans only
tweet ten times a day, then we keep it
can keep these two apart.
And these two probabilities, which
we don't know are very
unimportant.
But if these two distributions
look somewhat similar, so
if so, let's
think we're trying to estimate
if it's a man or a woman based on
the on the on the height.
And of course, if somebody
is one point seventy five meters,
it can be a male or female.
It's very hard to discriminate.
So these distributions overlap.
And if these distributions overlap, we
simply cannot count male and
females without
having without knowing in advance
which which of the two classes,
what the distribution is.
So this approach, even if we
there's no way to make it
a iteratively use it
unless we have a very clear
identifier for humans.
So for example, if each of the bars has a
label that says it's a bot,
then we can discriminate the two classes.
Mm-Hmm.
Another question that is similar, maybe
to the effect question that we had in the
beginning.
Are you not worried that your research
will be abused by those employing
rooms full of humans to do a bot
like posting to just claim
there are no problems at all?
And just
just just giving this as an excuse
saying like look at those researchers,
they claim there are no bots at all.
It's so like what we do, it's
good work or something like that.
Well, first of all, I'm I'm a
researcher and I'm not interested in
what people might make
of the results I have.
So I'm not trying to produce
results that
might be useful for somebody
or I'm not afraid of producing
results that might, might
come handy for parties
and political discussions.
The whole. But the research are often
very often involves
a very polarized political
discussion. So sometimes people like me
because I tell the right wing people
they are not part of the real people.
And then the next discussion I tell the
left wing people that are not bots and
they like me because I tell them they're
not bots. So at the moment,
I start to think about the.
The effects some people might
might like about research.
This would be I would no
longer be a scientist,
but obviously we're not talking about
people
getting paid for for
for tweeting.
So obviously we're
only interested in the claims about
social bots. So the automated
accounts interfering in political
discussions, using fake profiles.
So the paid trolls, I don't
claim they don't exist.
I know they exist, especially
in Venezuela, for example,
or in Mexico or in Russia.
They're very valid,
very credible reports
about paid trolls, even insider
reports by people who have been
working in this kind of set ups.
So I'm certainly not claiming that
there are no humans trying to
interfere in political discussions.
Yeah.
Maybe a follow up from another attendee.
How can we incentivize
this replication work
for those studies you've shown
and you kind of tried to replicate
as well, or at least got some information
out within the scientific community for
this field? Is that kind of something
that the general audience can do,
or maybe something that researchers need
to do in order to put more focus
on replication in this field,
which seems to be a bit flawed and
needs lots of replication, it seems?
Well, I think
the replication issue is a big issue in
all scientific fields, even
in computer science.
A lot of the research,
for example, I'm
new to topologies,
designed for neural networks, which work
in certain setups in certain.
But it did no longer work in slightly
different setups.
So replication is a big issue
in all scientific fields, but
I think it's a very huge issue in social
sciences, social sciences.
So I'm a computer scientist.
I come from a technical field.
We're used to very strict standards,
so we use it.
Nobody would dare
to publish something without
being ready to publish
the raw data as well.
So the idea of publishing results without
being able to hand out the raw data
is a huge.
It's basically scientific misconduct in
our field. So in social scientists,
obviously, people seem to be a lot more
more relaxed about about
reality and about truthfulness.
And hey, I'm
there. I have a feeling there's a kind
of cultural mismatch
there, though it turns
out that a lot of studies in
social sciences are not replicable.
And the problem seems
to be extremely
prevalent in social sciences.
Yes. Mm-Hmm.
Mm-Hmm. Maybe last question.
A couple of people ask the variations
on the same question, in your opinion,
is the lack of evidence for social bots
sufficient to prove that social bots
do not exist or not
in any meaningful amounts?
I mean, like that we didn't find them
doesn't mean they don't exist.
I mean, we can build one for a proof
that they exist.
But did you think that
kind of the lack of evidence is
proof, maybe
that they did not exist at all or not
exist in a large quantity?
Well, basically
the interesting.
Maybe you're familiar with Russell's
teapot.
He he used the the
the the idea of
of a teapot
flying around around
the Sun in an orbit beyond
Mars and
claiming that the tea teapot was
flying around the Sun in orbit
around Mars is hard to
disprove.
So you cannot.
You can take a very close look
at the orbit in the will still not be
able to disprove that.
Or is it a teapot is flying around
Mars? And obviously
we have the technology to send
the teapot in an orbit around Mars.
So but it's
and if people claim that
teapot might be orbiting Mars, it's
very hot. Yeah, we can build rockets.
We can build teapot.
We can send them in orbit around Mars,
but be very hard to disprove
these claims. It's very much like Social
Buzz. Now, I believe it's very unlikely
that somebody has already
launched a rocket with a teapot teapot
to do it to orbit Mars.
And also believe it's pretty unlikely
that people would try to build
automated political
social bots simply based on my
understanding of the technology.
So building jet parts is extremely
cumbersome, it's expensive,
and the results are really, really
bad, so if you've ever interacted with
a real social, a real chat bot,
you know those parts are very
limited at the current state of the art.
They can't even handle negation.
So if you tell Siri, do
not switch on the light, it will switch
off the light. So even the simple stuff
like negations are still beyond
what a natural language processing can
do at the current state of the art.
So the idea that of rapidly
coming up with a chat bot
like system that can interfere
in current political discussions, which
are usually only going on for a
couple of days on Twitter, and then you
would have to buy a new one for the next
discussion or you have to program
a new one. It seems very,
very beyond what's
possible from a technological
perspective.
I'm pretty sure these kind of bots do not
exist, but of course I cannot prove it.
All right.
Thanks. I think we got some more
questions, but we cannot really fit them
into this slot again.
Thanks so much, Florian.
Keep up the discussion and also keep
up your work.
And after this talk,
we will continue here on calls soon
followed with a talk from Big Alex about
software engineering, I guess.
And we will see you soon
after a short break.
Thank you.