Science: Has Rigour Been Sacrificed for Reach? | Predatory Publishing, AI-Generated Research, Scientific Fraud & Public Trust

Gareth King (00:43)

Kent, Joy, thank you both so much for joining us and welcome to Ruined by the Internet?

 

Kent Anderson (00:48)

Great to be here.

 

Joy Moore (00:48)

Thank you so much for inviting us. Great to meet you.

 

Gareth King (00:51)

You have both been inside the world of science and publishing for many years now. To get us started today, can you tell us about when you first felt like new or emerging technology was contributing to some systemic problems and changing things within science's ability to regulate itself?

 

Kent Anderson (01:12)

Yeah, I'll take a stab at that first. So, I had been working at launching various initiatives online for about five or six years, around 2001. And I'd worked at the American Academy of Paediatrics, and I was working at the New England Journal of Medicine at the time.

 

And we had a meeting where Harold Varmus, who was just coming off of a term heading the National Institutes of Health, and he's a Nobel Prize winning oncologist, and he was at the meeting speaking about an initiative that he was spearheading called eBiomed, which ultimately morphed into something called PubMed Central. And it was strange to me because it was a proposal to centralise all of this information on the internet. And the internet to me was a decentralised technology. But he wanted to centralise it under government control and that threw a lot of red flags up for me.

 

And so, I buttonholed him as he was leaving the meeting and I said, why do you want to do this? And he said, well, let's just see what the internet can do. And I thought, oh boy, we are in for it when someone is that worshipful of a technology that they are willing to just bank their brain and cross their fingers and let it go and wreak havoc. That was the first time I've had a chill.

 

Gareth King (02:32)

Right. That's an interesting kind of little context that you’ve given there because it sounds exactly like what we're seeing with AI and LLMs right now, which I'm sure we're gonna get into. But just even on that, you did point out that the internet was supposed to be this decentralised thing and that centralisation of information flew in the face of everything that it was all about.

 

But how specifically have those changes affected, I guess, the incentive structures within science and publishing themselves?

 

Joy Moore (03:04)

Well the platforming was very exciting as a way to sell a lot of content efficiently. And so going from print and then individual online subscriptions to this platform strategy where you could pump a lot of different journals and endless numbers of articles through a system at scale was great for efficiency, but also, for changing the way that content was charged for and negotiating how it was priced and distributed.

 

And so that was the first big shift. And the platformisation also introduced the idea that there might be a long tail of content. Once you have everything in one place, you might discover a niche article or a sub-specialty journal that wasn't on your radar. And so that was another one of those, let's see what we can do once we put it all into one place.

 

That said, when it comes to incentives, once you're investing in this platform and the scale, then you're incentivised to push more through it in order to monetise more individual units to support the ever-expanding costs of maintaining that platform.

 

Gareth King (04:30)

Yeah, absolutely. And look, everything that you've just explained there, Joy, like that sounds typical for tech and the internet just in general. You know, everything is that speed, that volume and that monetisation.

 

But for me, that contrasts with, from my understanding anyway, what science has always been about, which is that slower, more rigorous investigation. So, like what what's really happening thanks to that mismatch and how did we get to this place?

 

Kent Anderson (04:57)

Well, the one other little piece to weave into this, so the original sin of the internet is the advertising model, right? So, all the content and services have to appear free to the user. And the way that that happens is that your data and your information is sold to advertisers, right? It's a producer pays model.

 

And one of the things that Varmus and others in his milieu really pushed was this idea of public access or open access. And the only way to make it free to the user was to charge the producer. And that took it from a recurring revenue subscription model based on a community of a specific size and the specific people, and you platform the information and you charge by the article.

 

Then all of a sudden, the only way you can make it pay off and cover those costs, which were much greater than print, for digital platforms and all the services and licenses and all this stuff. The only way you can make that work is you have to churn a lot of content.

 

And so that opened the floodgates to predatory publishers, to fake papers, to paper mills, to all these other things who all of a sudden, they had a cash on the barrelhead business. So, they would pop up and they could make money in 30 days, whereas a traditional subscription publisher would take three to five years to start getting an ROI on a journal startup.

 

And they could fake people out and they could lie to people and pump and dump these little tech platforms that they started up with cash on the barrelhead businesses. So, chasing that new money became a thing across the industry.

 

Gareth King (06:35)

Yeah, chasing that new money. I'm glad that you brought up the open access to predatory publishing pipeline, 'cause I did want to get into that with you both. But can you just explain what the concept behind predatory publishing is? Like what does it look like?

 

Joy Moore (06:51)

Well what we found is with the explosion, particularly of open access publishing, and as Kent was saying, in the producer pays model, that's pretty lucrative business. It's a turnkey operation. And so, there are companies known, and shadow unknown companies all over the world, who are making money by churning out papers for various reasons.

 

So, they could be selling authorships, they could be selling papers to support institutional accreditation. There are lots of incentives for different people to want to publish, not just in the pursuit of science and the slow science that you were mentioning, but in in the paper as the currency for different incentives for different players.

 

And so therefore if you, the predatory publishers are looking for a quick buck, they are recruiting people to review or author or all sorts of different aspects of that participation in the publishing process that looks legitimate. It looks like what most scientists are used to, but they're really there to just get that upfront payment.

 

Kent Anderson (08:13)

Yeah. And there there's a big range. I mean, and a lot of different practices. You can have everything from, you know, shady places that just spam people with emails and happen to get enough of a fetch of papers to make it worthwhile. So, spray and pray, predatory publishing, right?

 

You have instances where predatory publishers will do what they call journal snatching. So, they'll find a struggling journal maybe at the end of it a legitimate subscription journal at the end of its life, where a husband and wife started a journal, they start to age out, they look for a deal to sell it to somebody, they sell it to somebody who flips it to open access predatory publishing. They ramp up the article volume and all of a sudden, they're making money.

 

We have instances of authors or editors being bribed, of certain organised crime rings placing people on editorial boards in order to get papers through. So, there's all these little tendrils happening at the same time. It's not just one thing, it's more of a practice, I guess.

 

Gareth King (09:11)

Look, it sounds awful, especially within the world of something like science that, you know, should be as legit as it can be. But everything that you've said, it just sounds like when there is that opportunity for money to be made, it is going to attract bad actors.

How much is that actually compared to, how much papers does that represent, I guess, amongst the entire output?

 

Kent Anderson (09:37)

It’s hard for me to get a beat on what's going on because now we have LLMs spitting out papers like crazy. You have large, consolidated publishers, and that's one of the other things that led to consolidation was the price of technology, right? And the deals that have to go on with that. And a lot of smaller nonprofit publishers like those I worked with can't afford to hire the lawyers to paper those deals. They can't afford the huge technology plays and all of those things.

 

But you have AI pumping through a lot of papers, excuse me, you have projections that don't line up with plausible scientific outputs. So, they're projecting that they're going to have 15 to 20% increases in the number of papers they publish while US science funding is being cut, while the normal increase in scientific output is far lower than that. So, you don't know how much tomfoolery is actually going on here.

 

But again, the incentives right now is what one of the things we focus on kind of follow the money, and the incentives are to just keep growing. And now they have a tool with LLMs that really makes that easy.

 

As far as you're ask asking for an estimate. So, there's something called the problematic paper screener. And this is something that looks at papers for fingerprints from LLMs. And a lot of times they say silly things like they'll instead of breast cancer, they'll say bosom cancer. Or instead of insects, they'll say creepy crawlies because they're just doing inference matching, right? And there's some of them are rather funny.

 

But they estimate that there are 1.2 million problematic papers just generated by LLMs, and that doesn't count the hundreds of thousands from paper mills and the, you know, the plagiarism that's going on. We were just talking to somebody today and one of the big plagiarism tools is no longer effective because all the LLM content is effectively plagiarism. So, it's basically mooted right by the fact that it everything is a lot of what it's looking at is just totally plagiarised.

 

Gareth King (11:36)

Right. That's interesting to hear. And that does connect to what I was gonna ask you then. Like obviously you mentioned there's that quite rapid growth in the production of papers. And look, I'm assuming the number of people or the ability of the number of people to kind of audit this stuff can just, there's no chance it can keep up with something like that. So obviously the tools themselves are presumably getting more automated too. And yeah, sorry, go on.

 

Kent Anderson (12:04)

Well, it's you know, so one of the first paper generating tools was called SciGen and it was launched in 2005, and it became a tool that a lot of people used to conduct stings on journals for fake papers. And then they developed a tool to detect the tool, right? And now there are tools to detect LLMs.

 

So, it's just this technology civil war, but what we keep returning to, none of this is about science. None of the, and you know, we kind of have to keep touching grass as they say or getting back to the fundamentals. Science is about discovery, observation of the real world, of nature, and trying to understand what you don't understand. And all of this other stuff has nothing to do with science.

 

Gareth King (12:47)

Mm-hmm. On that point… sorry, you go Joy.

 

Joy Moore (12:50)

I was just gonna say using the number of papers published as a proxy for the amount of science that's being discovered is a false equivalency. And especially when in the digital era, there is the need to quantify and make inferences based on data.

 

And so, the numbers of papers going through the system has just gone up. The number of scientists around the planet is not going up twenty to fifty percent a year. The number of papers are being published in smaller slices. And so, there are more papers talking about smaller bits of discovery than there were before because there's economics in publishing those papers.

 

So, whereas before the big incentives in scientific publishing were to figure out what not to publish, and make the most money from publishing the select content that was going to be of most use to a certain community. And so, it's flipped completely where the incentives, as we said, are to publish more.

 

And so, to your point about the people who are trying to screen it, peer review and quality control either before publication or after publication, it's a strain on the system because the system wasn't designed to scale, the system was originally designed to restrict.

 

Gareth King (14:27)

Right. Look, everything you've just outlined there, hearing you explain it all, again, a lot of these problems around scaling and the ability to keep up in parallel with that, obviously we're discussing them in a in a scientific context.

 

But I feel that those are almost universal problems that if we bring it back to maybe an example that everyone can relate to, you know, you might be scrolling social media or whatever, and now they're using automated moderation to try and track that stuff because obviously people can't keep up with it. But on social media, you know, like it could be a little video that you're seeing that's a bit fake.

 

What I'd love to get into now is this all sounds awful for a world of science that's meant to be delivering real breakthroughs to help humanity. Like what's happening when so much resources are now not only potentially drawn to producing, but also fighting this weirdly strange and unreal ecosystem?

 

Kent Anderson (15:25)

Yeah, I think one of the things that we're observing, we don't have that many direct indications of this, but I'm I have a Chick-fil-A sandwich story I'll share. but the, are you familiar with Chick-fil-A sandwich?

 

Gareth King (15:39)

I know the name; I've never eaten it.

 

Kent Anderson (15:55)

Okay, well I'll give you an insight so in hopes that you do someday. It's fine. But what we're what we think is going on is that people have learned to play this game. And that there isn't any, a lot of the papers that are filling the funnels and making the cash registers ring are just dreck. They're just pablum. They're safe papers. There's nothing in there that's going to shock anybody. It's not going to raise a retraction event. It's not going to be withdrawn. It's not going to alarm anybody. It's not going to inform anybody. It's just to pad a resume.

 

And papers have become so easy to publish that they are taking the place of actual, you know, there was a scam a few years ago and it's still I think it's still going on where high school students were being solicited to publish scientific papers to increase their chances of getting into the college of their choice, right?

 

And one of the journalists I'm friends with who was covering this, he told me the story of a young student who her family went to Disney World. They drove down the eastern seaboard of the United States, and she decided she was seeing a lot of Chick-fil-A restaurants. So, she was going to write about the Chick-fil-A marketing system and her observations of that.

 

So, she wrote it up and he showed me the paper, and I said, well, this wouldn't pass my peer review. And he said, why not? And I said, have you ever had a Chick-fil-A sandwich? He said, yeah. I said, she doesn't mention the pickle. That's like the tangiest, most distinctive part of a Chick-fil-A sandwich. I'm like, she never ate one. I guarantee you, she never ate a Chick-fil-A sandwich.

 

But I think that again, the way this is expanded as a substitute for actual achievement and actual science. And it's just these like it's Chick-fil-A sandwich papers. It's nothing that's going to upset anybody, nothing that's going to inform anybody, nothing that's gonna be that right or that wrong. It's an ocean of nothing.

 

Then we start to get into the idea of how all these LLMs that are being sold to us about how they're so good with science, but their people don't know what's going on with the scientific information that they are supposed to be handling.

 

Gareth King (17:52)

There's something interesting in what you were talking about there. Let's say, you know, the Chick-fil-A paper. Now you mentioned that nothing in there is groundbreaking. It's not gonna raise any suspicions. It's just kind of grey. Let's just call it slop, a word that that everybody knows. Now if all it's doing is just kind of taking up space, is it really getting noticed by people or is it simply clogging up a pipeline for real research?

 

Because yeah, I'm just trying to understand because from what you said, if it doesn't raise suspicion, if it's not really noticed by anyone, it's not making super outlandish claims, like it's just garbage, what's the actual effects of this nothingness?

 

Joy Moore (18:34)

We've had lots of interesting discussions with scientists around this and those scientists who are serious in in their area of expertise, they know what to blur their eyes to, they know what to ignore, they're going to the sources that they are familiar with. So, to them it's not really that big of a deal. Sometimes they get asked to peer review more often than they have time for because there are so many more publications.

 

The flip side of this though, where the public has access to all of this content because it's freely available and people are encouraged to do their own research, whether it's just a straight search engine search or now through LLMs, is that people who don't know the difference have access to this information and they're making decisions around a paper that nobody would have even paid attention to before.

 

Kent Anderson (19:32)

Yeah. And in the US, I think one of the commercial phenomena is the nutraceuticals market has exploded because these, from AG one to Armra colostrum powder to peptides to, they are able to put what look like innocuous scientific papers into these pay-to-play journals and then list those on their site as evidence that they have scientific backing to their claims.

 

And then we've traced that all the way through to an LLM called Open Evidence. It's becoming popular in the US, and it uses the same Silicon Valley business model of surveillance capitalism selling ads to people. But we were able to trace that all the way through from, you know, suspected predatory publisher and a company planting papers there, all the way through to the LLM promoting essentially the bovine colostrum powder and peptides.

 

And so that's, you know, what looks innocuous, it makes it easier for these things to hide out when you have so much mud and muck. And it makes it so that the people who know what they've planted can just point at them and say, look how valid we are and sell you things that would not pass muster from a scientific point of view, but it looks like they have.

 

Gareth King (20:52)

That's interesting because apart from the fact that you mentioned AG one just on a side tangent, I feel like ninety-eight percent of podcasts I listen to that have ads run something for AG one these days. But that's a side point.

 

But what you've mentioned there was the people that can identify the dreck. And you mentioned LLMs as well. Now, one thing that is a real problem in my opinion, is the way that LLMs are very often confidently incorrect. And people may not audit the information and so they take it as the ultimate truth, et cetera.

 

Now it sounds like from what you're saying now that the real problems start to arise when this junk grey matter, we'll call it, is served up as really credible evidence to people who will just take it and run with it. And does then that kind of power its own momentum in that regard?

 

And then what does that do for the people not only producing actual real work that needs to be done, like do they just get buried behind this kind of middle of the road greyness that LLMs generally revert to?

 

Kent Anderson (21:57)

Well, I think there's a recursive bit of this too. I was reading something earlier today, where an LLM can generate a fake scientific paper in between three and twenty minutes, right? And then it gets planted somewhere and then another LLM picks it up or it picks itself up later. And so, the degradation of LLMs as scientific authors is already ongoing because they're drinking their own bathwater, which is hilarious.

 

But it's just a really bad information system overall. But yeah, you're absolutely right, and they're corroding the fundamental part of it themselves.

 

Gareth King (22:33)

Hmm. So, it feels like yes, these may be these innocuous papers produced in huge volume, but I'm just trying to play it out here because I want, I want to really understand what the downstream effects on the scientific discipline are, and kind of what that means for the rest of us who are not inside the industry, as well as understanding what it means for inside the industry.

 

So, can you break down what will happen if this kind of thing continues as it is?

 

Kent Anderson (23:04)

Well, I think we've seen some of it already. So, we've seen fake papers used in the US under the current Health and Human Services Administration to change vaccine guidance, even though the papers never went through official peer review. They were published by physicians who had lost their licenses because they spread misinformation.

 

You know, you see a lot of amplification of bad scientific claims to the public through Substack and Substack newsletters. You see amplification through social media. And so, I think it, without the upstream controls that are focused on actual scientists talking to other scientists, then you open the gates for people to exploit the public and mislead them and sell them things and hurt them.

 

I mean, we've seen the resurgence of measles in the US because of misguided vaccine policies. And that traces all the way back to when I was at the American Academy of Paediatrics and the Wakefield paper came out, and I was there the day that it landed, and I watched the physician leadership go into huddle mode and try to deal with it. They couldn't because there was also a new search engine on the market called Google. And that perpetuated that myth all the way through to today. And that's one paper. And that was before the amplification of social media, before the amplification of Substack, before all of these things and TikTok.

 

So, you get into now when it's just overwhelming and you can pick any topic and find a scientific paper to impress somebody with or mic drop on somebody with, it becomes very problematic to separate fact from fiction and to understand what reality actually is, and whether it's based on actual scientific findings.

 

Gareth King (24:47)

You mentioned the mic drop moment. Now, something that comes up quite a bit on this show is the way that people engage in the digital space. Often it's not very civil, you know, like people are trying to dunk on each other and as you said, if you can point to one of these fraudulent slop papers as the ultimate proof that whatever claim you're making is true, it's a lot of people who are potentially very confused, but it's just leading to a lot of division.

 

But on the topic of division, you did mention measles and vaccination stuff. Something I'd love if we could get into a little bit now was how this stuff affects those contentious issues, like say vaccine hesitancy, climate denial, et cetera. Like these are very polarised issues that I feel like everyone can find a random scientific paper or journal publication that they can then use to support whatever position they take on that. And that's it. You know, you draw the line under it, you're like, yo, I'm correct. I don't need to listen to anything else.

 

Now you mentioned the current health administration, and is it RFK is at the top of something like that, and I know there's like a whole lot of stuff circulating around him and his positions on vaccinations and things like that. But that's coming like very top down, which is so much more socially damaging, it feels like, than say myself - I don't know how a vaccine is made. I go and find something and read it. And if I think I'm right, it's not really affecting anybody.

 

The fact that these dodgy papers and like let's say fake research, whatever we’ll call it, is making its way to the top echelons of governments and then beyond. Like that seems enormously, as they say, problematic for society in general. Is that what's happening where you're seeing the resurgence of measles giving people this wrong information and we're almost turning back the clock on proven science?

 

Joy Moore (26:41)

Well, it's a misuse of scientific papers fundamentally. And so very rarely does any single paper prove something beyond the shadow of a doubt. Science is iterative. And so, each paper is a report of an experiment or a finding, and a claim based on those outcomes. And then science builds on that. So eventually, if enough evidence comes into play to change practice, then that makes its way into society or to create a new policy or, you know, things like that.

 

And so, the adoption of individual scientific papers as proof is very much more like a legal exercise, right? So as a lawyer, you have a case to prove and then you go find the evidence to back up your point of view. And so that's not how science works and that's not how scientific policy is normally conducted; you follow the evidence and then from there you make a decision.

 

And so that distinction is really important, particularly when we have public access to all of these papers now, because anyone can build a case in citing individual papers. You see this all the time on podcasts now and on the news and things like that. And you they pop up this paper; it's like this study said this so therefore, it's true. Within the scientific community, that's not how these papers were intended to be used, and it's certainly not how serious scientists take them.

 

Gareth King (28:19)

Right. I guess there's two prongs in the way that I can take that more democratised access. Now, obviously you just mentioned there it's not the way that it's supposed to be used. Whether that comes through heavily credentialled people using them in in the way that they're supposed to be used. But then I guess the flip side of that is it's kind of guarded and shielded for a much, much smaller amount of people.

 

Where the positive flip side of that could be that you've got more eyes, more brains, potentially finding previously unseen ways to think about something and build on that and iterate that that research and that learning.

 

Kent Anderson (28:56)

And that was absolutely the hope. You know that was absolutely the hope from, and Joy and I were both idealists. Oh no, oh dear no. I think what we've what we've actually seen is so there's kind of this techno utopian idea that this is gonna democratise the access to knowledge and we're all gonna become super smart.

 

One of the people who wrote about Varmus's vision of what the internet can do, thought he'd win the Nobel Prize for his insights in four years, and everybody would be a scientist, and all the world's diseases would be cured by 2003. So, I missed the mark on that one.

 

But what's been emerging is feudalism, is really techno feudalism. And so again, you have these consolidated platforms, AI, if you view it as a centralised knowledge bakery. You have all this centralised power, and you have all this wealth inequality. So, you have people who try to control everybody else with ideas that they coordinate among a very few number of them.

 

And that's the antithesis of democratised knowledge. And science is, if you ever read a real scientific paper that isn't aimed at the public, it's really dense. It's very technical. And Joy and I have both edited a number, a lot of scientific papers in our years.

 

And there's a lot of subtlety and nuance within every community. Cardiothoracic surgeons are completely different from civil engineers, or completely different than OBGYNs, are completely different than hand surgeons. They have different jargon. They have different little ways of referring to things that they all know, but no outsider would, right? A real tech a real scientific paper is for scientists, and usually very specific scientists.

 

I worked in orthopaedics for a number of years and the hand surgeons would acknowledge that the spine surgeons couldn't read their papers and vice versa. I mean, it's that narrow, right? And so, for the whole premise that the public can take one or five papers and come to some sort of independent conclusion, is I think part of a ruse to do something else. Somebody's up to something else with that positioning.

 

Gareth King (31:12)

Yes, I'd love to get into that. But before we do, I think where I was kind of going with that previous question was the democratisation of access for people that may understand it enough to kind of build on the work. Like not someone like myself going in and finding a paper on thoracic surgery and then trying to figure out what I'm gonna do.

 

But I'm glad that you've brought up the public and how this affects them because how do we how do we get from these really scientifically dense papers, how does that end up in the public if it is way too dense for fellow really, really intellectual experts to kind of understand? Like what what's falling down here before that information gets to the public to the point where they may interpret their own way or even just like I say, use an LLM to summarise just some greyness for them. Like is there I guess an information pipeline problem here?

 

Kent Anderson (32:05)

Yeah, I think it gets back to your earlier point about the content versus something like journalism, right? Science journalism. There's a real paucity of science journalism, translating these things in a meaningful and careful way to the public. Science education is a key part of that and knowing when and what to educate the public about. But Joy, what do you think?

 

Joy Moore (32:29)

I agree with that, but I was also gonna add the social media attention economy aspect of it is very much in play here, just as it is with everything else. And so, authors are incentivised to promote their individual papers on social media. Free papers are getting picked up by journalism outlets all over the world, if there's a headline that looks juicy, and you can see on the platforms, for these papers is especially the ones that are free access are getting turned into clickbait.

 

And so even though the paper was technically correct and dense scientifically, the headline takeaway and the interpretation of what that paper means to you and me on a daily basis are completely out of sync.

 

And I think that's what's really driving a lot of this attention economy. What's more exciting than a headline that says you should eat chocolate five times a day or things like that? So it's content begetting content and selling ads and creating impressions at every platform step along the way.

 

Kent Anderson (33:43)

The other thing that's interesting, I just want to tie one last thing in 'cause that's a there's a great point about the attention economy, is a lot of the leadership in the US health system currently are creatures, what I would call creatures of the internet themselves. They came up through the attention economy. That's how they gained fame, that's how they gained prominence. They are empty vessels ultimately. But they now have authority. You talked about that earlier. How did they get authority? Well, it's because they rode the attention economy to the top of the scientific institutions.

 

Gareth King (34:15)

Right. That is a great segue into I guess where what I wanted to go into next, like the attention economy. Now Joy, you mentioned framing everything as this kind of clickbait, you know, eat chocolate five times a day. And I think the saying is they're being factual but not truthful in the way that you can spin something to give that that headline.

 

Now we obviously know that the real problem is how many people read beyond the headline? Like that's really all you need to gain like this ultimate backing yourself. But then I think where my next question on, you know, that attention economy and I guess how you've mentioned that that even top-level officials try and game that now.

 

How does this affect, I guess, public trust in science itself if you've got, say, proven established science being gamified for clicks that may contrast the actual evidence? What's that doing to the way that the public in general trust science? Because from what I can see, it seems to be a very divided environment on a lot of issues.

 

Kent Anderson (35:24)

Yeah. I think that the public trusts scientists. I think they trust their physician. They trust their local officials who oversee safe drinking water and public safety and public health and the school nurse and all of those people, right? They trust the people who work in science. Those people haven't lost trust.

 

I think where the where they are finding themselves whipsawed a bit is with the information space. And they don't know what to trust there. That's what we're talking about, right? Is how that's been so damaged and undercut.

 

Joy Moore (36:00)

And it's very difficult. It's very distracting a lot of the clickbait use of scientific information. And there are some really excellent science communicators and that's a special skill that not very many people have.

 

Scientists when I see them on especially like on the news, you know, explaining their new study, that's not what they do. And so that's why it's very hard trying to get them to write it in plain language, is really hard because that's not what they do. Scientific communicators have a really hard job and they're hard to find.

 

Whereas the pseudoscience and the misinformation distribution outlets are very easy to find, and they're very well-funded and they're everywhere, and you kind of can't avoid them. And so that's the struggle is that there is good information out there, you have to go seek it. But the bad information is more likely to seek you out.

 

Kent Anderson (37:00)

Yeah, and I think the bad information has one benefit that normal science doesn't. It has a narrative and they're coordinating that narrative and they're finding the evidence to justify that narrative. And they push that narrative day in and day out on social media because they're incentivised to because they've got money at stake.

 

I have a friend who's having an adventure with functional medicine right now, and I keep trying to warn him that hopes and fears are being exploited to empty his wallet, but his hopes and fears are so strong that they can do that and it's working and it breaks my heart what's going on.

 

But that's how these charlatans work, is they've created this narrative and you know, there's a lot of there are a lot of dropout doctors, a lot of old doctors who are looking for one last hurrah and one last cash out. I mean, there are a lot of operators out there.

 

Gareth King (37:54)

Look, that that makes sense as well. Obviously, there's people trying to adapt to a world that's changing super, super fast constantly. But I think where I'd love to touch back to now, you mentioned current administration health people who've kind of grown up in that attention economy and they almost know how to use it.

 

But if we look at it as a purely objective, I guess, opportunity there, doesn’t that mean that there's also the equivalent really honest operators within science itself who have also grown up in that environment and are becoming or will become much more efficient communicators to the public, and are able to leverage the same tools for a more positive outcome?

 

Joy Moore (38:35)

Yeah, and we're seeing more and more of these digital native generations start to have more agency, and say, hey, I'm not believing the hype anymore. And so that's really exciting. They know how to use it, and they know how to make the most of it, but they also know how to call bullshit when they see it, which is exciting.

 

Kent Anderson (39:01)

And, you know, they know that they watched friends deal with the abuses of these platforms, right, throughout their school years and beyond. And so, they know what to watch out for.

 

Part of the reason we wrote the book is we wanted to pull the alarm because we didn't think many people knew how bad things had gotten in science and how the attention economy, the producer pays model and other things from the internet had infiltrated it and really kind of upset the apple cart that everybody thinks is fine.

 

Gareth King (39:28)

That's a great point as well, which has been very eye-opening to hear about. But we've just been obviously talking about these kinds of digital natives who have come up using various social media platforms that allows everyone to build and connect communities.

 

There's an obviously a huge load of positives that come out of that for those exact reasons. We've spoken about, I guess, the relationship between tech and science internally, but while we're talking around the attention economy and people using it, I mean, what's your take on the relationship between these massive platforms and science itself, especially when they do have that ability to manipulate the algorithms, even take down certain themes of content that may not be expedient at any given moment? Like what's happening there?

 

Kent Anderson (40:18)

Yeah, I think one of the one of the things we talk about in the book is kind of the heritage of big data, the heritage of Stanford and Silicon Valley, and some of the ideas that have tracked from the late nineteenth century all the way through to today, like eugenics and racism and sexism and misogyny.

 

And all of those things have a big history in there. And we're seeing them again today represented in these platforms and how their algorithms work and how Grok works and how children are exploited and a whole bunch of different things and you know, from USAID being defunded to children's health being defunded in the US and things like that. Who is behind these platforms because the power is so consolidated is almost as important as how they work these days.

 

My take on it is I was taught that tech tends toward monopoly just because it becomes more efficient to run it if you run one platform. I also think the way that tech has been deployed now, it tends toward governance. And we're being governed by these five or six people, essentially, and their belief systems. And I think that a lot of their belief systems aren't compatible with science. And Adam Becker has a great book that talks about how they have this sort of ketamine-fuelled delusion, according to Cory Doctorow, that we're going to get to the singularity and escape velocity, and we can burn the earth to a cinder while we do it, and all of these things.

 

And so, there's a problem that isn't about making the world better for people today, but it has issues with long termism and transhumanism and effective altruism and all of those kinds of things.

 

Gareth King (41:56)

Yeah, look, the monopolisation and stuff that you mentioned there, I would agree that that's the inevitable end point that we're gonna get to. But I think what I'd love for you both to do now, if you can, is if you can identify the single most dangerous consequence of what the internet and tech has done to science over the last few decades, what would it be?

 

Kent Anderson (42:19)

My biggest concern is how we think about power. And it has changed us from thinking about power being about truth seeking behaviour and treating each other and doing things to help each other, to information is power, which makes media power and using that to exploit each other.

 

The fundamental change that worries me is that might makes right. And that information has become power and people are using it to dominate others. And that makes me very worried because science is supposed to be, is one of the greatest process accomplishments of human beings, to just drop your religious, drop your intellectual biases, drop your cultural biases, and observe the world as it is and figure out why that why that weird thing happened.

 

Gareth King (43:12)

What a great summary. I love that.

 

Joy Moore (43:15)

And along the same lines, I think the most dangerous thing that has come true over this era of tech and science is that the money is being made by private companies, shady operators, while the funding for the real science is getting more and more endangered. And so, the farther we keep going, the less we're going to be able to support the next generation of bona fide scientists. And that's really scary.

 

Gareth King (43:50)

Yeah, that is scary. Again, it's just comes back to this extraction economy. But then on those points then for you both, if you could redesign the incentive structures of scientific publishing for this digital world that we're not only currently in but for the future, what would you change and what would you keep?

 

Joy Moore (44:08)

I would take it back to the community level and start to break up some of these big platform plays, and ensure that science is for scientists and that those who need it are able to get it without sacrificing the integrity of the information.

 

And so, I would re-jig it and focus back on the communities and create incentives that are community specific. Individual communities are totally different and have different incentives. And so, if you take away that platform scale incentive and that attention at all cost incentive, then more resources can be dedicated to doing the quality work in scientific publishing to better serve those individual communities.

 

Kent Anderson (44:55)

Yep. And to support you, Joy, I would call complete bullshit on Section 230 and how it's being..

 

Gareth King (45:02)

What is, sorry, just for context for I guess us outside the US, what's Section 230?

 

Kent Anderson (45:09)

So, Section 230 is a part of the copyright law that was put in place to say that if you're running just a platform like a telephone company, you're not liable for what if I call you up and say nasty things. They can't sue the phone company because of that, right?

 

And so, the internet became was bundled into that sort of platform space. And that's fine as a carriage, common carriage feature, you know, IP addresses shifting back and forth and all of that. If I put something on my website that says, you know, Joy eats worms, she can't sue Verizon or Comcast for that.

 

But what has, what the internet companies and platform companies and social media companies have done is they've said, even though we're trading in media, even though we're have news feeds and we have recommendation engines and we have algorithms that are sorting things and making decisions about what you see and what you don't, and we're doing all these media manipulation things, we're still covered by Section 230 because we're just a little old platform. We're just common carriage.

 

And I think that's complete and utter nonsense. They became media companies decades ago. And as soon as you have a recommendation engine, as soon as you have content that you're peddling, imagine if your phone called you up and recommended a restaurant just out of the blue. That's not common carriage anymore, right? So, I think we need to we need to really interrogate that rule because it's the US tech that is really at the heart of the rot.

 

Gareth King (46:39)

Yeah, look, it does feel like an a very antiquated, I dunno, I'm struggling to say it, but rule for a very different time. that when not, you know, like let's be real, so many people are looking at a screen for like twelve hours a day these days. That's half your time spent in a parallel universe essentially where rules like that really don't apply.

 

But I guess then, just to finish up, and you may have touched on some of this already, what's a belief or concern you both have about technology in general right now that you wish was getting more attention?

 

Kent Anderson (47:15)

I think for me, it's starting to get attention, but I would think it needs to be really interrogated, which is just how inference engines and transformers and things like that, the whole stack of LLMs is interesting, right? It's not what somebody calls normal technology and to treat it like normal technology.

 

And because it's some of it's useful, some of it's not. Some of it's useful in certain contexts, but it's not useful in every context. It's not artificial, it's not intelligence.

 

I think we need to really knock that back and get it into under control. And remember that humans created it, right? So, I think my dream of the future is to stop elevating like Harold Varmus did, stop elevating the technology over the humans.

 

Let's see what the internet can do, or AI is inevitable or anything like that. It's like, nope, we made it. These are our little toys. What are we going to do with them? Let's talk like adults, and stop infantilising us about tech. We know how it works and we don't need that. That's what I would, that's what I would hope.

 

Joy Moore (48:22)

Yeah, it's okay to unplug things, you know. We are imposed through our devices upon on lots of different ways and we assume that we have to be there because everybody else is doing it, especially in when it comes to scientific publishing and scientific information.

 

We thought the more connected everything is, the better we're gonna get out of it. And in fact, it might be better to unplug some things and make some more determined human choices and what we seek out to use these tech tools for as opposed to letting them wash over us.

 

Gareth King (49:02)

I couldn't agree more. You know, let's get back some agency as people. But Joy, Kent, thank you both so much for joining us today. You've really given us a lot to think about, and like I said before, some pretty eye-opening stuff about what's going on.

 

What have you both got coming up from here and where can people follow what you're up to?

 

Kent Anderson (49:22)

So, we have a book coming out called How the Internet Disrupted Science. And you can find it in your local retailer. And where you can find us at disruptedscience.com, all one word dot com. You can follow us on the geyser, which is the newsletter that we work on to keep almost daily pulse of this. We also have a Friday song of the week if you like music. And we have a podcast called Disrupted Science ourselves and we talk about issues there. So, any of those outlets you can keep up with us.

 

Gareth King (49:55)

Brilliant. Thank you both so much.

 

Joy Moore (49:57)

Thank you. What a pleasure to talk to you.