I want to stick my neck out to say that this has the potential of being very bad for science.
Imagine saying "no" to a researcher with a big social media profile. Imagine 4chan coming at you with style-detection and deanonymization tools simply because their favorite racist or antivaxer got their nonsense rejected and sent their followers after you. And this is not just me feeling this way - quoting myself from a previous comment, and according to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents" and "support for public review inversely correlated with reviewing experience".
A measure that women ~~and inexperienced researchers~~[2] do not support is a measure that favors only those who are already part of the club.
> Imagine saying "no" to a researcher with a big social media profile
"The identity of the reviewers will remain anonymous, unless they choose otherwise — as happens now."
(Also "support for public review [being] inversely correlated with reviewing experience" means inexperienced reviewers are more likely to support it. Not less.)
You are correct about the second point - I'll strike it through once I find out how.
As for the anonymous part, that's why I wrote "with style-detection and deanonymization tools". If the Internet could find Shia Labeouf's flag in a day [1], could they find a reviewer based on their writing?
This is a concern I have as an "anonymous" HN account, even up to putting it in quotes despite never revealing my name or strong PII. But the language I use as a reviewer is pretty different, even from that of a writer. I suspect this will be harder due to low sample rates but then again, high noise could help your point. Mobs need targets more than accuracy.
Though, we do eventually need to have a conversation about deanonymization of online accounts. That's not a thing we want to be done so easily
The difference is that as a scientific reviewer you are not hiding a physical location and what you need is plausible deniability, which would still exist. In addition to this, actively attempting to deanonymise your reviewers is on the level of scientific misconduct that your employer and professional organisation should consider taking disciplinary action against you. I am not arguing that this makes it entirely safe to publish anonymised reviews and that we will not affect reviewer behaviour (maybe for the better in some cases, as "one-sentence reviews" will be something in the public record), but it is in stark contrast to the example that you bring up.
Plausible deniability will satisfy who exactly? The Sherlock Holmes of Reddit did a great job with the Boston marathon bombing.
You don’t have to actively demonetize your reviews, you just have to prime your audience ahead of time that your ~~election~~ publication was stolen and they’ll do the rest.
This is giving tinfoil hat vibes - I would guess style detection has such a high false positive rate as to be near useless. Also there's nothing stopping people from publishing reviewer comments today and letting the Internet run wild with "style detection" (or doing it themselves).
Has there been recent developments in the style detection and deanonymization tools you mentioned? I would assume many would not work well given the high usage of LLMs nowadays.
I know every area is different but the "grifters" in the area of Computational Linguistics (the ACL) are "any volunteer[1] whose paper has been accepted at least once", meaning anyone from PhD students to professors and industry researchers.
Not all academia is Elsevier.
[1] This policy has been altered recently, though, and now submitting a paper comes with reviewing duties.
We are struggling badly with review quality in natural language processing though. Most likely due to the unprecedented expansion of the field over the last ten or so years. Reviewers are suffering with review loads far exceeding what one reasonably can manage mentally (used to be two to three papers per reviewer and now five would be considered rather generous). Authors and area chairs suffer from worse quality reviews due to reviewer inexperience and overload, not to mention how good reviewer/author correspondence with author and area chair comments frequently being ignored by the reviewers. To me, the last holdout of good peer review in the field is Transactions of the Association for Computational Linguistics (TACL), but there the acceptance bar is sky high compared to ACL Rolling Review (ARR) for better or worse.
The ACL leadership and senior members of the field are very much aware of this and are trying their best (ARR being an attempt to improve the situation, but I am unsure how much better it really is compared to the old system of conference reviewing now that we are a few years in). But there appear to be no easy fixes for a complicated, distributed system such as peer review. Every discussion I have with said leadership and other senior members always ends with us agreeing on the problems and likewise agreeing that despite considerable mental effort we are failing to come up with solutions.
Returning to the main topic. Nature is worthy of praise for making their peer review transparent and I say this as a massive Nature critic. It is a move I loved seeing from NeurIPS (then NIPS) and ICLR over a decade ago, as it helps younger researchers see what good (and bad) communication looks like and that even papers they know now are greatly appreciated received a fair amount of criticism (sometimes unwarranted). I have argued for ACL to introduce the same thing for nearly a decade at this point, but we still do not and I have never heard a solid argument as to why not (best argument was the technological effort, but OpenReview, with all its flaws, makes this even easier than with Softconf; not that it would have been that hard with Softconf either).
This is amazing. The trust of peer review as a stamp of quality among academics is dwindling, and distrust of science among the population is growing (within the increasing politicisation of some areas of research). Transparent peer review raises the bar for both academics and enhances the potential trust in the process.
This is desperately needed as AI could further deteriorate the quality of science if the publishing process is not made more strict. This represents a significant step forward in rigorous science. I hope other publications follow suit.
The fact remains that distrust of science continues to grow. Up to now the establishment’s response has been one of condescension. Your comment echoes that attitude.
Ignoring the problem is not going to fix it, and in fact continuing to regard these people as beneath you is only going to accelerate the downfall of this system.
> The fact remains that distrust of science continues to grow.
Of course it does, billion-dollar interests who have a vested interest in attacking it have a well-funded propaganda arm, and as we've been discovering over the past century - angry, loud, and frequently repeated bullshit with a profit motive floods any signal out of the room.
What's really sad is people who have legitimate concerns or desires for improvement hitching their horse-cart to the former.
No amount of peer review or replication is going to convince someone whose fortune is built on peddling snake oil.
> Ignoring the problem is not going to fix it,
Here's a solution. Hold the people making these attacks to the same level of rigor as what they are attacking. Stop giving proven liars a loudspeaker. The day the press will do that is the day some meaningful progress to fix that damage may be made.
Until you do that, we're going to continue getting shit outcomes.
Lots of them have. Look at any site where science skepticism is regularly posted, you will find that maybe a good 50% of the content is commentary on papers. Of those who don't trust science and haven't read papers, they will usually have read commentary by those who have.
Source: I've published such skepticism in the past and met people who read my articles, including politicians.
Nonetheless, you're right that simply publishing peer reviews won't help improve trust. The situation is bad enough that there's no One Weird Trick that can dig academia out of its hole. Some of the most intellectually fraudulent papers I've written about in the past did have transparent peer review, and all it showed was that peer reviewers were often aware of the critical problems found inside and waved it through anyway. Or that their feedback was ignored. Or that they agreed with obviously bogus practices. Or that the parts of the paper that revealed the problems weren't reviewed (eg. appendices, github repositories). After all, nobody cares about papers that got rejected by the system, the distrust is driven by the papers that weren't, so almost by definition such papers either were badly reviewed or the review wasn't used.
I think there are a few groups and reasons of distrust. Some more or less valid.
Those that distrust authority as a whole and lean into conspiracies cannot be saved with this kind of thing.
But i think news and science are having similar perception issues recently.
Distrust for news growing among the average population (for some good reasons). People are loosing faith in the objectivity of established media organisations. Most people are exposed to science through these traditional news.
So adding back some sense of confidence and authority to scientific institutions is very valuable to non-academics. Even if they themselves would not read the papers or revews.
Why should anyone trust science? Skepticism should always be the default position. Putting it on a pedestal to be worshipped is what led us all into this mess. If science needs trust to work, then whatever it is doing is something I'd like to see fail.
When people talk about those who distrust science, they aren't referring to the carefully sceptical. They're talking about people who come to a conclusion and then reject any evidence against it.
Right. RFK supporters aren’t carefully weighing evidence to arrive at their conclusions, they are blindly following someone who is a mindless contrarian with a 30yr vendetta against vaccines. No amount or quality of peer review is going to change any of their minds.
> RFK supporters aren’t carefully weighing evidence to arrive at their conclusions, they are blindly following someone
I'd actually be curious what fraction of his supporters share his views on vaccines. It strikes me as more likely that they like one of his random, idiosyncratic views and are willing to excuse the anti-vaccine nonsense to get those policy outcomes.
That, but also people who take the blanket view that none of the work produced by the scientific establishment is trustworthy. An ironic stance considering that it's the inverse of the blind worship that they accuse others of.
>none of the work produced by the scientific establishment is trustworthy.
There's that word again. No works are trustworthy. You shouldn't trust, instead verify. Be skeptical. I'm more concerned with which of their works can be verified.
>An ironic stance considering that it's the inverse of the blind worship
I'm not sure there's any difference worth mentioning between blind worship and cuddly warm trusting that you want to do. One may seem a little less zealous, but it's the same pig just with extra lipstick.
Assuming the widespread presence of blatant fraud in the absence of evidence is (at least to my mind) the inverse of blind worship. There is no difference in validity between an unfounded assumption of misdeed or virtue.
By "trustworthy" I mean nothing more than the same presumption of innocence absent evidence to the contrary that you extend at the grocery store when you assume that the food you're purchasing isn't poisoned or rotten or etc. That doesn't mean the published results are reproducible, the measurements without (unintentional) error, or the conclusions logically sound. Merely that there isn't an active attempt being made to deceive or otherwise knowingly mislead the reader to the advantage of the author.
What I'm describing is an incredibly low bar with broad applicability that society generally requires in order to function.
> That, but also people who take the blanket view that none of the work produced by the scientific establishment is trustworthy.
That’s a red herring. No scientist actually says that. What we say is that on some subjects the evidence is overwhelming and to overcome the current understanding you need compelling evidence and theories, not screeches about bias and liberal elites.
No. We are not going to take seriously someone’s pet theory about a new perpetual motion machine, or cold fusion, or lack of global warming, or the ineffectiveness of vaccines, or anything that is contrary to massive amounts of accumulated evidence.
> Why should anyone trust science? Skepticism should always be the default position.
Skepticism is not anti-scientific. Hell, distrusting results and theories is not anti-scientific. Distrusting the scientific method is. There is a difference.
> Putting it on a pedestal to be worshipped is what led us all into this mess.
Scientists did not ask for this. Amongst all high profile politicians, those who whine about science becoming political are those who made it so, by taking contrarian positions to rile up their voter base. Most people who make this point are not being honest. If you are, you should make specific arguments rather than rehash propaganda.
One might think that to be the case, but science is in a hurry to save our planet, and it has no patience for your skepticism. It may already be too late.
>Distrusting the scientific method is.
Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn. And maybe when it's finally proven that, maybe I go on distrusting it... and make it prove it some more. If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven". But so little do I ever hear of that sentiment. Like in this very thread.
>Scientists did not ask for this.
Probably not. I'd be skeptical of the theory that they asked for it. But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask. And liking it, they started behaving in ways that encouraged it just the same. And those scientists who enjoyed it the most elbowed their way to the front to egg it on even more. And so that's where we are now. I mean, I'm a little skeptical that it happened that way, and I'd welcome evidence that disproves it.
>Most people who make this point are not being honest.
It's really sort of hilarious how it's only my side that can ever be honest. Not the other guys'. And this is true no matter who "me" is. None of your opponents, for instance, can ever be genuine... they're always trying to cheat. And if your side tries to cheat, well, it's with the best of intentions because the stakes are so high. Not that you would cheat, only the other side ever does that.
>f you are, you should make specific arguments rather than rehash propaganda.
If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
> One might think that to be the case, but science is in a hurry to save our planet, and it has no patience for your skepticism. It may already be too late.
As a matter of fact, it is already too late for some of the consequences. The permafrost is not going to re-freeze, for example, and wildfires are already significantly on the rise.
The thing is, I don’t believe this because science, I know this for a fact because it was proven. It’s like nihilistic edgelords giving cynicism a bad name. Skepticism is not the rejection of facts because they are inconvenient. It is not stubbornness in the face of evidence, either.
> Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn.
Right, but then to prove itself it needs to be evaluated and assessed. You need a kind of method to do so, including a standard of proof and the basics of reasoning and logic. This framework is itself the scientific method. If you start from the axiom that it does not work, then the only conclusion is that it does not work, because how would you prove otherwise?
> If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven".
Well, yeah. That’s what we do. Though the standards shift over time in the face of overwhelming evidence. We can not start from first principles every time, it just does not work and there are theories we can rely on. But when holes are found in these theories we try to figure out what the problem is and how to build better ones. The scientific method itself has been refined over centuries and the standards now are far from what Newton used. A common example is the evolution of what is considered to be a proof in Physics.
> But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask.
I am sure some of them do, and not necessarily the best ones. But most of us are not attracted by public engagement and just want to do our science in our lab. We did not ask to become pawns in sectarian struggles. We rang the alarm because, honestly, when the house is on fire you have to do something and not doing it would mean some degree of responsibility in the outcome.
> It's really sort of hilarious how it's only my side that can ever be honest.
I am not following. I am not talking about sides. Regrettable attitudes towards science are not confined in one political or social tribe and there are examples in history of both right- and left-leaning ideologies becoming anti-science. But yeah, making a topic political in order to then claim that it is political is blatantly dishonest. It’s a cheap rhetorical trick with lasting consequences. What is political is how you address the situation, not whether reality is real.
> If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
You are reading way too much. First, I am not attacking you. Then, a good starting point is facts we can observe and things we can infer from them instead of the a priori position that reality is not real and the other side is just making stuff up.
> wildfires are already significantly on the rise.
Wildfires at at relatively low levels, historically. US burn acreage is about 10M acres per year at the moment, same as it was in 1955. During the hot dustbowl years burn acreage was as high as 50M/year.
You're unaware of this because "scientists" like to truncate wildfire graphs at around 1983, knowing that insufficiently skeptical people won't check if there's data available from earlier years. The choice of 1983 isn't arbitrary, it's when the downward trend in the graph goes into mild reverse as people started to realize that suppressing wildfires isn't always the best policy. There follows a small, gentle rise to a level still far below what it was historically (and arguably, what it should be).
Now, obviously, they don't say that. They say oh dear, in 1983 we changed our process and - incredibly - that they have no idea where data came from before that, so please ignore all previously collected data. The numbers just, like, magically appeared in their spreadsheets. This sort of trick is common, and it correctly leads to distrust as obviously, if they genuinely believe data before 1983 is useless, they can't make statements about the effect of climate change on wildfires one way or another. But that would also be an enormous scandal given all the money spent collecting that data.
> I don’t believe this because science, I know this for a fact because it was proven.
You haven't actually gone out and monitored wildfires with your own eyes, that's impossible. When you say it's a "fact" and "proven", what you mean is, people you trust from institutions you aren't skeptical of told you that they did go out and count wildfire acreage, and that it's a "fact" that wildfires are getting worse. Then you assumed they're completely trustworthy and would never do things like drop data points to force a trend, so it became a fact in your mind and you became unable to understand why anyone might disagree.
Now repeat this problem 1000x. That's where the fighting comes from, where the distrust comes from.
> Wildfires at at relatively low levels, historically. US burn acreage is about 10M acres per year at the moment, same as it was in 1955. During the hot dustbowl years burn acreage was as high as 50M/year.
Don't deny the science! The scientists are our saviors. You're only skeptical because evil manipulators have twisted the truth to support their narrative. I saw a picture of a polar bear about to drown because it was stuck on a rapidly melting iceberg, and unless we murder our economy the planet won't survive. Think of the polar bears.
I would go further: Anyone who has published a "scientific paper" in the last decade or so either "distrusts science" or is more likely than not a mid-wit at best.
Your posting doesn't give me the impression that you're very familiar with "science".
Isn't that true for everything, though? If I weren't a software developer I wouldn't know that I have to worry about questions like "has this plane been rebooted in the last 51 days?" [1] or "does this bank offer anything other than SMS as second factor?".
Maybe structural engineers feel safer after their Master's when they traverse a bridge, but I bet that's more the exception than the rule.
Seeing how the sausage gets made makes you realize ALL the downsides and things you'd rather not have known. That doesn't mean you can think of a way to fix science. Let alone get the required funding to make an actual attempt.
> Anyone who has published a "scientific paper" in the last decade or so either "distrusts science" or is more likely than not a mid-wit at best.
That is very unhelpful, to say the least. The amount of noise has increased, but it does not mean that the scientists who know their subject disappeared. They are still around and not any less bright than their predecessors were 30 years ago.
Steel-manning their argument, 'distrusting science' doesn't mean they throw the whole thing out, they're just aware that there is disagreement and bullshit going around within the process. As far as I can tell, it's dangerous to try to assess a topic through reading papers alone. A scientist active in the field will have read more widely, be aware of the reputations and biases of the different groups, and likely will have tried some of the published experiments themselves (replication does happen, if it's an interesting result, it's just rarely worth publishing the result).
> The trust of peer review as a stamp of quality among academics is dwindling
The thing is, peer review is not a stamp of quality, and never was. It is just the basic level of due diligence. The referees cannot reproduce the results most of the time for a lot of very good reasons. They are here as a sanity check, to ensure that the work avoided common pitfalls and actually makes sense.
What most people do not understand is that articles are not good because they are peer-reviewed; it’s the lack of peer review that is a red flag. Amongst reviewed articles, a lot of them will turn out to be wrong or flawed in ways that are impossible for the reviewers to find out.
This only addresses a small part of the problem with peer-review. The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers. If the paper doesn’t fit what is expected, it will often be rejected. This can also lead to self-reinforcing views that ignore contrarian data. Also, the data can be made up, and if it makes sense to the reviewers, it is generally not questioned
> The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers.
This is a misunderstanding of the role of peer review. The point is not to prove that a paper is correct, the point is to ensure a minimum level of quality. You are entirely right that most reviewers cannot hope to reproduce the results presented, and very often for very good reasons. If I write 3 proposals over the course of a year to get some beam line on a neutron source, it is completely unrealistic to expect a referee to have the same level of commitment.
I think this hints at a more profound problem, which is that a lot of studies are not replicated. This is where the robustness of a scientific result comes from: anybody can make the same observation and reach the same conclusions under the same conditions. This is the real test, not whether you convinced 3 or 5 referees.
The real value of an article is not in whether it was peer reviewed (though the absence of peer review is a red flag). Instead, it is in whether different people confirmed its main results over the years that follow publication.
Merely replicating a study doesn't tell you anything about whether the claims are correct, although failure to replicate can be a signal that the claims probably aren't correct.
Have you personally reviewed a non-zero number of papers? What is this statement based on? For a thread ostensibly about science, the comments are disappointingly lacking in evidence and heavy on vibes.
Maybe people could learn about what peer review is before posting their strong feelings about it? The purpose certainly isn't to replicate people's experiments, that happens after publication and not by referees. One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing.
Also, just because something has made it past peer review also doesn't mean it isn't controversial in the field.
He isn't saying the purpose of peer review is to replicate work. And yes I've personally reviewed quite a lot of papers (from outside of academia), also occasionally reviewed peer reviews.
What he is saying is that peer review is treated by academia as a close to gold standard, when it's really more like a bronze standard. It's a bit like finding a software company that exclusively uses volunteer post-commit code review with no unit tests or static typing, and in which the only testing process is to push to production and see if anyone inside the company complains.
It's not useless as a concept, and it's generally better to have a paper that's reviewed than one that isn't unless the field has been captured by ideologues. But there are so many common problems it can't fix, not even in principle.
Bad quality work is a much bigger problem than dishonest work. Systematically well-done research with fake results or data is much rarer than just... lazy bad science.
There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important. This is lazy and fast, and can be caught by a stricter review and scrutiny.
Papers that are properly done all the way through, but with faked data meant to push an agenda, can be disproven by counter research.
> Systematically well-done research with fake results or data is much rarer than just... lazy bad science.
Genuine mistakes, logical errors, and other oversights are even more common than that. For all the issues it has, peer review is quite good at catching the things that it's intended to catch.
Reading your comment makes me think that you believe that the point of peer-review is to ensure that a paper is correct, or at least that specific aspects of it are correct. Is that the case? What do you think the point of peer-review is?
I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?
Research gets cited constantly in public debates and is used for policy decisions, so the public should be able to quickly separate the good from the bad, the "maybe this is true" from the "this is empirically proven."
The public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus.
I wish this comment was more representative of my personal experience in science.
Instead I got PIs happy to say that weak evidence "proved" their theory and to try suppress evidence that negatively impacted "fundablity". The most successful scientists I worked with were the ones who always talked like a PR puff piece.
> public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus
The public has lost trust in science because 10 to 30% of it is scientifically illiterate [1]. (Tens of millions of American adults are literally illiterate [2].)
That's what lets activists and politicians cherrypick bad science that supports their position or cast a scientific consensus as unquestionable.
That's certainly true but I think there's also a very real issue as described by GP. Research gets cited in a political setting as a rhetorical tool. That cycle does a lot to erode trust in the establishment because it incentivizes non-scientists (who are otherwise uninvolved) to behave as though the process is a partisan effort to be interfered with for the benefit of one's "team" rather than an objective pursuit to be funded at arms length for the betterment of society at large.
Obviously reproducing results as part of peer review is not a workable (or even coherent) solution. I don't pretend to have any idea what a solution might be. The obvious issue is that academic publications were never intended as political tools and should not be made into that.
On several occasions I've had interactions with laymen where I found myself thinking "if only you hadn't had access to pubmed and way too much motivation we'd both be better off right now" yet I firmly believe that free and open access to information is a huge net benefit to society on the whole.
> I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?
It is not, and it cannot be. It is unrealistic to expect a referee working in their free time to confirm studies that often cost millions of dollars. This is a fundamental misunderstanding of what peer review is and why it is useful in popular or heavy vulgarised science.
Politicians, journalists, and university press offices are guilty of this, and they are those abusing peer review to give some studies more weight than they deserve.
A good step forward. Reading the referee reports and rebuttals from papers that previously opted in to transparent reviewing was incredibly useful to my own paper publishing process. In many ways Nature is ahead of Science on this.
I hope that making things transparent will help reduce the situation where big labs have an easier time getting their work into high impact journals through relationships with the editor.
We need more scientific societies. Modern peer review is super modern. Go back to the origins of science and it was all about a real community—setting high standards and having just a few people decide what to publish. It wasn’t trying to be “objective” — it was just high standards, determined invisibly by the members of the society. “Should we publish this?” asked the society.
Alas, scale ruins everything. Nevertheless, I strongly believe “science is friendship”
That model had some major issues. Too many opposing theories to the "established norm" were dismissed, among things we now know were wrong all along.
Although it could raise the standard of the process itself (methodology, writing) to very high levels, it restricted innovative ideas or unpopular outcomes.
The trouble is, scientific research is now a livelihood and career for must published researchers. It's not an aristocratic hobby anymore. The incentives are all different.
One of the weaknesses to such a system is human nature. We want to believe, which leads to farces like Piltdown man, which was a farce by a man called Charles Dawson (not Darwin, not the natural selection guy, though I did a double take at first), against the Geological Society and society at large. The farce wasn’t definitively disproven until 41 years later, which is quite the downside risk of the gentlemen’s club research group gatekeeping scholarly research, but the more things change, the more they stay the same. This is a good thing which seems like it’s in response to grumblings about irreproducibility and bias, but those issues will likely always be with us and must be considered anew each time an experiment is designed, and each time a print run is cut.
I’m having trouble finding a downside besides vote buying or voting rings, now that which way one has voted is now attributable. Can you think of any risks under the new system?
I feel like the main thing that this will accomplish is to show that the quality of peer review is quite spotty. Usually 1 of 3 truly read and understood the paper enough to critique from a place of knowledge. Entirely miscategorizing a paper is not uncommon.
I only got as far as undergraduate research assistant in the academia racket, so maybe I'm not "with it" in the way serious researchers are, but is this to suggest that this wasn't being done, by default, on everything already?
Any reputable journal had a reasonably robust peer review process (exact details vary of course) but historically it was strictly confidential. Only over the past perhaps 2 decades has a process where reviews are made public begun to gain mainstream support.
There are so many great reasons to distrust "The Science"... perhaps the greatest is that without powerful and persistent skepticism, science simply isn't science. Questioning and skepticism is profoundly fundamental to the scientific process. "Trust the science" or worse yet "Believe the science" are statements that are about as destructive to the foundation of science as one can get.
The whole point of science was that everyone has access to the true nature of the universe which we can discover through theorizing and experimentation. Empirical experience is the great equalizer that puts the Arch Duke of Where-and-What or the Highest of High Priests the on the same footing as, say, Joseph Preistly or Benjamin Franklyn. Questioning and skepticism must be accepted from all quarters. To say that only the most select, distinguished and credentialed can be admitted to the discussion smacks of priesthood and aristocracy.
The argument that we need to re-persuade the people to trust in science is missing the point by such a wide margin as to be a symptom of the problem.
It is more or less the same as fretting that people aren't going to church so much any more.
The science should not be trusted. The science should be tested. Those who say "Trust the science" have completely lost the thread.
I want to stick my neck out to say that this has the potential of being very bad for science.
Imagine saying "no" to a researcher with a big social media profile. Imagine 4chan coming at you with style-detection and deanonymization tools simply because their favorite racist or antivaxer got their nonsense rejected and sent their followers after you. And this is not just me feeling this way - quoting myself from a previous comment, and according to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents" and "support for public review inversely correlated with reviewing experience".
A measure that women ~~and inexperienced researchers~~[2] do not support is a measure that favors only those who are already part of the club.
[1] Original here (currently offline): http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019..., summary here: https://www.aclweb.org/adminwiki/images/f/f5/ACL_Reviewing_S...
[2] This part has been correctly pointed out as being wrong.
> Imagine saying "no" to a researcher with a big social media profile
"The identity of the reviewers will remain anonymous, unless they choose otherwise — as happens now."
(Also "support for public review [being] inversely correlated with reviewing experience" means inexperienced reviewers are more likely to support it. Not less.)
You are correct about the second point - I'll strike it through once I find out how.
As for the anonymous part, that's why I wrote "with style-detection and deanonymization tools". If the Internet could find Shia Labeouf's flag in a day [1], could they find a reviewer based on their writing?
[1] https://www.dailydot.com/unclick/4chan-shia-labeouf-secret-l...
This is a concern I have as an "anonymous" HN account, even up to putting it in quotes despite never revealing my name or strong PII. But the language I use as a reviewer is pretty different, even from that of a writer. I suspect this will be harder due to low sample rates but then again, high noise could help your point. Mobs need targets more than accuracy.
Though, we do eventually need to have a conversation about deanonymization of online accounts. That's not a thing we want to be done so easily
The difference is that as a scientific reviewer you are not hiding a physical location and what you need is plausible deniability, which would still exist. In addition to this, actively attempting to deanonymise your reviewers is on the level of scientific misconduct that your employer and professional organisation should consider taking disciplinary action against you. I am not arguing that this makes it entirely safe to publish anonymised reviews and that we will not affect reviewer behaviour (maybe for the better in some cases, as "one-sentence reviews" will be something in the public record), but it is in stark contrast to the example that you bring up.
Plausible deniability will satisfy who exactly? The Sherlock Holmes of Reddit did a great job with the Boston marathon bombing.
You don’t have to actively demonetize your reviews, you just have to prime your audience ahead of time that your ~~election~~ publication was stolen and they’ll do the rest.
This is giving tinfoil hat vibes - I would guess style detection has such a high false positive rate as to be near useless. Also there's nothing stopping people from publishing reviewer comments today and letting the Internet run wild with "style detection" (or doing it themselves).
Translate into $RANDOM_LANG and then back to English. The perfect prose obfuscator.
Or get chatGPT to reword it in a different style of writing
Has there been recent developments in the style detection and deanonymization tools you mentioned? I would assume many would not work well given the high usage of LLMs nowadays.
What is the alternative?
Or are you just for creating classes of people that just can't be critiqued in any circumstance?
This kind of sounds like, 'Wont anyone think of the grifters?!'
I know every area is different but the "grifters" in the area of Computational Linguistics (the ACL) are "any volunteer[1] whose paper has been accepted at least once", meaning anyone from PhD students to professors and industry researchers.
Not all academia is Elsevier.
[1] This policy has been altered recently, though, and now submitting a paper comes with reviewing duties.
We are struggling badly with review quality in natural language processing though. Most likely due to the unprecedented expansion of the field over the last ten or so years. Reviewers are suffering with review loads far exceeding what one reasonably can manage mentally (used to be two to three papers per reviewer and now five would be considered rather generous). Authors and area chairs suffer from worse quality reviews due to reviewer inexperience and overload, not to mention how good reviewer/author correspondence with author and area chair comments frequently being ignored by the reviewers. To me, the last holdout of good peer review in the field is Transactions of the Association for Computational Linguistics (TACL), but there the acceptance bar is sky high compared to ACL Rolling Review (ARR) for better or worse.
The ACL leadership and senior members of the field are very much aware of this and are trying their best (ARR being an attempt to improve the situation, but I am unsure how much better it really is compared to the old system of conference reviewing now that we are a few years in). But there appear to be no easy fixes for a complicated, distributed system such as peer review. Every discussion I have with said leadership and other senior members always ends with us agreeing on the problems and likewise agreeing that despite considerable mental effort we are failing to come up with solutions.
Returning to the main topic. Nature is worthy of praise for making their peer review transparent and I say this as a massive Nature critic. It is a move I loved seeing from NeurIPS (then NIPS) and ICLR over a decade ago, as it helps younger researchers see what good (and bad) communication looks like and that even papers they know now are greatly appreciated received a fair amount of criticism (sometimes unwarranted). I have argued for ACL to introduce the same thing for nearly a decade at this point, but we still do not and I have never heard a solid argument as to why not (best argument was the technological effort, but OpenReview, with all its flaws, makes this even easier than with Softconf; not that it would have been that hard with Softconf either).
This is amazing. The trust of peer review as a stamp of quality among academics is dwindling, and distrust of science among the population is growing (within the increasing politicisation of some areas of research). Transparent peer review raises the bar for both academics and enhances the potential trust in the process.
This is desperately needed as AI could further deteriorate the quality of science if the publishing process is not made more strict. This represents a significant step forward in rigorous science. I hope other publications follow suit.
Of the people who distrust science, how many of them have ever read a scientific paper?
I suspect the number is low. If that's the case, they're unlikely to be more convinced by the presence of published peer review, either.
The fact remains that distrust of science continues to grow. Up to now the establishment’s response has been one of condescension. Your comment echoes that attitude.
Ignoring the problem is not going to fix it, and in fact continuing to regard these people as beneath you is only going to accelerate the downfall of this system.
> The fact remains that distrust of science continues to grow.
Of course it does, billion-dollar interests who have a vested interest in attacking it have a well-funded propaganda arm, and as we've been discovering over the past century - angry, loud, and frequently repeated bullshit with a profit motive floods any signal out of the room.
What's really sad is people who have legitimate concerns or desires for improvement hitching their horse-cart to the former.
No amount of peer review or replication is going to convince someone whose fortune is built on peddling snake oil.
> Ignoring the problem is not going to fix it,
Here's a solution. Hold the people making these attacks to the same level of rigor as what they are attacking. Stop giving proven liars a loudspeaker. The day the press will do that is the day some meaningful progress to fix that damage may be made.
Until you do that, we're going to continue getting shit outcomes.
Lots of them have. Look at any site where science skepticism is regularly posted, you will find that maybe a good 50% of the content is commentary on papers. Of those who don't trust science and haven't read papers, they will usually have read commentary by those who have.
Source: I've published such skepticism in the past and met people who read my articles, including politicians.
Nonetheless, you're right that simply publishing peer reviews won't help improve trust. The situation is bad enough that there's no One Weird Trick that can dig academia out of its hole. Some of the most intellectually fraudulent papers I've written about in the past did have transparent peer review, and all it showed was that peer reviewers were often aware of the critical problems found inside and waved it through anyway. Or that their feedback was ignored. Or that they agreed with obviously bogus practices. Or that the parts of the paper that revealed the problems weren't reviewed (eg. appendices, github repositories). After all, nobody cares about papers that got rejected by the system, the distrust is driven by the papers that weren't, so almost by definition such papers either were badly reviewed or the review wasn't used.
I think there are a few groups and reasons of distrust. Some more or less valid.
Those that distrust authority as a whole and lean into conspiracies cannot be saved with this kind of thing.
But i think news and science are having similar perception issues recently.
Distrust for news growing among the average population (for some good reasons). People are loosing faith in the objectivity of established media organisations. Most people are exposed to science through these traditional news.
So adding back some sense of confidence and authority to scientific institutions is very valuable to non-academics. Even if they themselves would not read the papers or revews.
>Of the people who distrust science,
Why should anyone trust science? Skepticism should always be the default position. Putting it on a pedestal to be worshipped is what led us all into this mess. If science needs trust to work, then whatever it is doing is something I'd like to see fail.
> Why should anyone trust science?
When people talk about those who distrust science, they aren't referring to the carefully sceptical. They're talking about people who come to a conclusion and then reject any evidence against it.
Right. RFK supporters aren’t carefully weighing evidence to arrive at their conclusions, they are blindly following someone who is a mindless contrarian with a 30yr vendetta against vaccines. No amount or quality of peer review is going to change any of their minds.
> RFK supporters aren’t carefully weighing evidence to arrive at their conclusions, they are blindly following someone
I'd actually be curious what fraction of his supporters share his views on vaccines. It strikes me as more likely that they like one of his random, idiosyncratic views and are willing to excuse the anti-vaccine nonsense to get those policy outcomes.
That, but also people who take the blanket view that none of the work produced by the scientific establishment is trustworthy. An ironic stance considering that it's the inverse of the blind worship that they accuse others of.
>none of the work produced by the scientific establishment is trustworthy.
There's that word again. No works are trustworthy. You shouldn't trust, instead verify. Be skeptical. I'm more concerned with which of their works can be verified.
>An ironic stance considering that it's the inverse of the blind worship
I'm not sure there's any difference worth mentioning between blind worship and cuddly warm trusting that you want to do. One may seem a little less zealous, but it's the same pig just with extra lipstick.
Assuming the widespread presence of blatant fraud in the absence of evidence is (at least to my mind) the inverse of blind worship. There is no difference in validity between an unfounded assumption of misdeed or virtue.
By "trustworthy" I mean nothing more than the same presumption of innocence absent evidence to the contrary that you extend at the grocery store when you assume that the food you're purchasing isn't poisoned or rotten or etc. That doesn't mean the published results are reproducible, the measurements without (unintentional) error, or the conclusions logically sound. Merely that there isn't an active attempt being made to deceive or otherwise knowingly mislead the reader to the advantage of the author.
What I'm describing is an incredibly low bar with broad applicability that society generally requires in order to function.
> That, but also people who take the blanket view that none of the work produced by the scientific establishment is trustworthy.
That’s a red herring. No scientist actually says that. What we say is that on some subjects the evidence is overwhelming and to overcome the current understanding you need compelling evidence and theories, not screeches about bias and liberal elites.
No. We are not going to take seriously someone’s pet theory about a new perpetual motion machine, or cold fusion, or lack of global warming, or the ineffectiveness of vaccines, or anything that is contrary to massive amounts of accumulated evidence.
I think you misread my comment? I'm talking about the non-scientist naysayers.
I just think we do not disagree.
> Why should anyone trust science? Skepticism should always be the default position.
Skepticism is not anti-scientific. Hell, distrusting results and theories is not anti-scientific. Distrusting the scientific method is. There is a difference.
> Putting it on a pedestal to be worshipped is what led us all into this mess.
Scientists did not ask for this. Amongst all high profile politicians, those who whine about science becoming political are those who made it so, by taking contrarian positions to rile up their voter base. Most people who make this point are not being honest. If you are, you should make specific arguments rather than rehash propaganda.
>Skepticism is not anti-scientific.
One might think that to be the case, but science is in a hurry to save our planet, and it has no patience for your skepticism. It may already be too late.
>Distrusting the scientific method is.
Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn. And maybe when it's finally proven that, maybe I go on distrusting it... and make it prove it some more. If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven". But so little do I ever hear of that sentiment. Like in this very thread.
>Scientists did not ask for this.
Probably not. I'd be skeptical of the theory that they asked for it. But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask. And liking it, they started behaving in ways that encouraged it just the same. And those scientists who enjoyed it the most elbowed their way to the front to egg it on even more. And so that's where we are now. I mean, I'm a little skeptical that it happened that way, and I'd welcome evidence that disproves it.
>Most people who make this point are not being honest.
It's really sort of hilarious how it's only my side that can ever be honest. Not the other guys'. And this is true no matter who "me" is. None of your opponents, for instance, can ever be genuine... they're always trying to cheat. And if your side tries to cheat, well, it's with the best of intentions because the stakes are so high. Not that you would cheat, only the other side ever does that.
>f you are, you should make specific arguments rather than rehash propaganda.
If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
> One might think that to be the case, but science is in a hurry to save our planet, and it has no patience for your skepticism. It may already be too late.
As a matter of fact, it is already too late for some of the consequences. The permafrost is not going to re-freeze, for example, and wildfires are already significantly on the rise.
The thing is, I don’t believe this because science, I know this for a fact because it was proven. It’s like nihilistic edgelords giving cynicism a bad name. Skepticism is not the rejection of facts because they are inconvenient. It is not stubbornness in the face of evidence, either.
> Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn.
Right, but then to prove itself it needs to be evaluated and assessed. You need a kind of method to do so, including a standard of proof and the basics of reasoning and logic. This framework is itself the scientific method. If you start from the axiom that it does not work, then the only conclusion is that it does not work, because how would you prove otherwise?
> If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven".
Well, yeah. That’s what we do. Though the standards shift over time in the face of overwhelming evidence. We can not start from first principles every time, it just does not work and there are theories we can rely on. But when holes are found in these theories we try to figure out what the problem is and how to build better ones. The scientific method itself has been refined over centuries and the standards now are far from what Newton used. A common example is the evolution of what is considered to be a proof in Physics.
> But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask.
I am sure some of them do, and not necessarily the best ones. But most of us are not attracted by public engagement and just want to do our science in our lab. We did not ask to become pawns in sectarian struggles. We rang the alarm because, honestly, when the house is on fire you have to do something and not doing it would mean some degree of responsibility in the outcome.
> It's really sort of hilarious how it's only my side that can ever be honest.
I am not following. I am not talking about sides. Regrettable attitudes towards science are not confined in one political or social tribe and there are examples in history of both right- and left-leaning ideologies becoming anti-science. But yeah, making a topic political in order to then claim that it is political is blatantly dishonest. It’s a cheap rhetorical trick with lasting consequences. What is political is how you address the situation, not whether reality is real.
> If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
You are reading way too much. First, I am not attacking you. Then, a good starting point is facts we can observe and things we can infer from them instead of the a priori position that reality is not real and the other side is just making stuff up.
> wildfires are already significantly on the rise.
Wildfires at at relatively low levels, historically. US burn acreage is about 10M acres per year at the moment, same as it was in 1955. During the hot dustbowl years burn acreage was as high as 50M/year.
https://web.archive.org/web/20210129125036/https://www.nifc....
You're unaware of this because "scientists" like to truncate wildfire graphs at around 1983, knowing that insufficiently skeptical people won't check if there's data available from earlier years. The choice of 1983 isn't arbitrary, it's when the downward trend in the graph goes into mild reverse as people started to realize that suppressing wildfires isn't always the best policy. There follows a small, gentle rise to a level still far below what it was historically (and arguably, what it should be).
Now, obviously, they don't say that. They say oh dear, in 1983 we changed our process and - incredibly - that they have no idea where data came from before that, so please ignore all previously collected data. The numbers just, like, magically appeared in their spreadsheets. This sort of trick is common, and it correctly leads to distrust as obviously, if they genuinely believe data before 1983 is useless, they can't make statements about the effect of climate change on wildfires one way or another. But that would also be an enormous scandal given all the money spent collecting that data.
> I don’t believe this because science, I know this for a fact because it was proven.
You haven't actually gone out and monitored wildfires with your own eyes, that's impossible. When you say it's a "fact" and "proven", what you mean is, people you trust from institutions you aren't skeptical of told you that they did go out and count wildfire acreage, and that it's a "fact" that wildfires are getting worse. Then you assumed they're completely trustworthy and would never do things like drop data points to force a trend, so it became a fact in your mind and you became unable to understand why anyone might disagree.
Now repeat this problem 1000x. That's where the fighting comes from, where the distrust comes from.
> Wildfires at at relatively low levels, historically. US burn acreage is about 10M acres per year at the moment, same as it was in 1955. During the hot dustbowl years burn acreage was as high as 50M/year.
The world is bigger than the US. FFS...
It's true worldwide:
https://ourworldindata.org/grapher/annual-burned-area-by-lan...
Wildfires have been decreasing globally for over 20 years.
Don't deny the science! The scientists are our saviors. You're only skeptical because evil manipulators have twisted the truth to support their narrative. I saw a picture of a polar bear about to drown because it was stuck on a rapidly melting iceberg, and unless we murder our economy the planet won't survive. Think of the polar bears.
I would go further: Anyone who has published a "scientific paper" in the last decade or so either "distrusts science" or is more likely than not a mid-wit at best.
Your posting doesn't give me the impression that you're very familiar with "science".
Absolutely, the quickest way to lose faith in "The Science" is actually to do Science in a formal research institute....
Isn't that true for everything, though? If I weren't a software developer I wouldn't know that I have to worry about questions like "has this plane been rebooted in the last 51 days?" [1] or "does this bank offer anything other than SMS as second factor?".
Maybe structural engineers feel safer after their Master's when they traverse a bridge, but I bet that's more the exception than the rule.
[1] https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
Seeing how the sausage gets made makes you realize ALL the downsides and things you'd rather not have known. That doesn't mean you can think of a way to fix science. Let alone get the required funding to make an actual attempt.
> Anyone who has published a "scientific paper" in the last decade or so either "distrusts science" or is more likely than not a mid-wit at best.
That is very unhelpful, to say the least. The amount of noise has increased, but it does not mean that the scientists who know their subject disappeared. They are still around and not any less bright than their predecessors were 30 years ago.
Steel-manning their argument, 'distrusting science' doesn't mean they throw the whole thing out, they're just aware that there is disagreement and bullshit going around within the process. As far as I can tell, it's dangerous to try to assess a topic through reading papers alone. A scientist active in the field will have read more widely, be aware of the reputations and biases of the different groups, and likely will have tried some of the published experiments themselves (replication does happen, if it's an interesting result, it's just rarely worth publishing the result).
> The trust of peer review as a stamp of quality among academics is dwindling
The thing is, peer review is not a stamp of quality, and never was. It is just the basic level of due diligence. The referees cannot reproduce the results most of the time for a lot of very good reasons. They are here as a sanity check, to ensure that the work avoided common pitfalls and actually makes sense.
What most people do not understand is that articles are not good because they are peer-reviewed; it’s the lack of peer review that is a red flag. Amongst reviewed articles, a lot of them will turn out to be wrong or flawed in ways that are impossible for the reviewers to find out.
>and enhances the potential trust in the process.
I highly doubt it's a meaningful factor in public trust.
The general public does not distrust science because peer reviews are not public.
Right. The general public cannot understand the points that are being made during peer review.
This only addresses a small part of the problem with peer-review. The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers. If the paper doesn’t fit what is expected, it will often be rejected. This can also lead to self-reinforcing views that ignore contrarian data. Also, the data can be made up, and if it makes sense to the reviewers, it is generally not questioned
> The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers.
This is a misunderstanding of the role of peer review. The point is not to prove that a paper is correct, the point is to ensure a minimum level of quality. You are entirely right that most reviewers cannot hope to reproduce the results presented, and very often for very good reasons. If I write 3 proposals over the course of a year to get some beam line on a neutron source, it is completely unrealistic to expect a referee to have the same level of commitment.
I think this hints at a more profound problem, which is that a lot of studies are not replicated. This is where the robustness of a scientific result comes from: anybody can make the same observation and reach the same conclusions under the same conditions. This is the real test, not whether you convinced 3 or 5 referees.
The real value of an article is not in whether it was peer reviewed (though the absence of peer review is a red flag). Instead, it is in whether different people confirmed its main results over the years that follow publication.
There are no silver bullets. Replication also isn't worth much:
https://blog.plan99.net/replication-studies-cant-fix-science...
Merely replicating a study doesn't tell you anything about whether the claims are correct, although failure to replicate can be a signal that the claims probably aren't correct.
Have you personally reviewed a non-zero number of papers? What is this statement based on? For a thread ostensibly about science, the comments are disappointingly lacking in evidence and heavy on vibes.
Maybe people could learn about what peer review is before posting their strong feelings about it? The purpose certainly isn't to replicate people's experiments, that happens after publication and not by referees. One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing.
Also, just because something has made it past peer review also doesn't mean it isn't controversial in the field.
I’ve only peer reviewed a few papers. I’ve submitted my share of papers for review
This is my academic profile https://scholar.google.com/citations?user=bxn78bkAAAAJ&hl=en
He isn't saying the purpose of peer review is to replicate work. And yes I've personally reviewed quite a lot of papers (from outside of academia), also occasionally reviewed peer reviews.
What he is saying is that peer review is treated by academia as a close to gold standard, when it's really more like a bronze standard. It's a bit like finding a software company that exclusively uses volunteer post-commit code review with no unit tests or static typing, and in which the only testing process is to push to production and see if anyone inside the company complains.
It's not useless as a concept, and it's generally better to have a paper that's reviewed than one that isn't unless the field has been captured by ideologues. But there are so many common problems it can't fix, not even in principle.
Bad quality work is a much bigger problem than dishonest work. Systematically well-done research with fake results or data is much rarer than just... lazy bad science.
There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important. This is lazy and fast, and can be caught by a stricter review and scrutiny.
Papers that are properly done all the way through, but with faked data meant to push an agenda, can be disproven by counter research.
> Systematically well-done research with fake results or data is much rarer than just... lazy bad science.
Genuine mistakes, logical errors, and other oversights are even more common than that. For all the issues it has, peer review is quite good at catching the things that it's intended to catch.
Reading your comment makes me think that you believe that the point of peer-review is to ensure that a paper is correct, or at least that specific aspects of it are correct. Is that the case? What do you think the point of peer-review is?
I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?
Research gets cited constantly in public debates and is used for policy decisions, so the public should be able to quickly separate the good from the bad, the "maybe this is true" from the "this is empirically proven."
The public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus.
Nothing is ever “proven”. There is simply more or less support for a theory or proposition.
Replication and meta-analysis are an important part of this.
Most scientists are in fact very conservative with how they claim their results - less so university PR departments and “study shows” clickbaiters.
I wish this comment was more representative of my personal experience in science.
Instead I got PIs happy to say that weak evidence "proved" their theory and to try suppress evidence that negatively impacted "fundablity". The most successful scientists I worked with were the ones who always talked like a PR puff piece.
What field, may I ask?
Applied physics. I'd prefer not to get too specific. Most of my peers are working for the US DoD or DoE now.
I wonder if what you described is due to the money incentive?
I did theoretical physics (no money) and my experience totally matches what the other person described.
I approve of this comment.
> public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus
The public has lost trust in science because 10 to 30% of it is scientifically illiterate [1]. (Tens of millions of American adults are literally illiterate [2].)
That's what lets activists and politicians cherrypick bad science that supports their position or cast a scientific consensus as unquestionable.
[1] https://carsey.unh.edu/publication/conspiracy-vs-science-sur...
[2] https://nces.ed.gov/pubs2019/2019179/index.asp
That's certainly true but I think there's also a very real issue as described by GP. Research gets cited in a political setting as a rhetorical tool. That cycle does a lot to erode trust in the establishment because it incentivizes non-scientists (who are otherwise uninvolved) to behave as though the process is a partisan effort to be interfered with for the benefit of one's "team" rather than an objective pursuit to be funded at arms length for the betterment of society at large.
Obviously reproducing results as part of peer review is not a workable (or even coherent) solution. I don't pretend to have any idea what a solution might be. The obvious issue is that academic publications were never intended as political tools and should not be made into that.
On several occasions I've had interactions with laymen where I found myself thinking "if only you hadn't had access to pubmed and way too much motivation we'd both be better off right now" yet I firmly believe that free and open access to information is a huge net benefit to society on the whole.
> I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?
It is not, and it cannot be. It is unrealistic to expect a referee working in their free time to confirm studies that often cost millions of dollars. This is a fundamental misunderstanding of what peer review is and why it is useful in popular or heavy vulgarised science.
Politicians, journalists, and university press offices are guilty of this, and they are those abusing peer review to give some studies more weight than they deserve.
A good step forward. Reading the referee reports and rebuttals from papers that previously opted in to transparent reviewing was incredibly useful to my own paper publishing process. In many ways Nature is ahead of Science on this.
I hope that making things transparent will help reduce the situation where big labs have an easier time getting their work into high impact journals through relationships with the editor.
We need more scientific societies. Modern peer review is super modern. Go back to the origins of science and it was all about a real community—setting high standards and having just a few people decide what to publish. It wasn’t trying to be “objective” — it was just high standards, determined invisibly by the members of the society. “Should we publish this?” asked the society.
Alas, scale ruins everything. Nevertheless, I strongly believe “science is friendship”
That model had some major issues. Too many opposing theories to the "established norm" were dismissed, among things we now know were wrong all along.
Although it could raise the standard of the process itself (methodology, writing) to very high levels, it restricted innovative ideas or unpopular outcomes.
The trouble is, scientific research is now a livelihood and career for must published researchers. It's not an aristocratic hobby anymore. The incentives are all different.
Sounds like a recipe for serious inequity to me.
> Sounds like a recipe for serious inequity to me
If it works better, it works better. The problem with the society method is it didn't work better than a decentralised scientific system.
One of the weaknesses to such a system is human nature. We want to believe, which leads to farces like Piltdown man, which was a farce by a man called Charles Dawson (not Darwin, not the natural selection guy, though I did a double take at first), against the Geological Society and society at large. The farce wasn’t definitively disproven until 41 years later, which is quite the downside risk of the gentlemen’s club research group gatekeeping scholarly research, but the more things change, the more they stay the same. This is a good thing which seems like it’s in response to grumblings about irreproducibility and bias, but those issues will likely always be with us and must be considered anew each time an experiment is designed, and each time a print run is cut.
I’m having trouble finding a downside besides vote buying or voting rings, now that which way one has voted is now attributable. Can you think of any risks under the new system?
> You have full access to this article via your institution.
I would suggest to Nature that this phrase hints at much larger a problem than showing authors arguing with Reviewer 2.
I feel like the main thing that this will accomplish is to show that the quality of peer review is quite spotty. Usually 1 of 3 truly read and understood the paper enough to critique from a place of knowledge. Entirely miscategorizing a paper is not uncommon.
I only got as far as undergraduate research assistant in the academia racket, so maybe I'm not "with it" in the way serious researchers are, but is this to suggest that this wasn't being done, by default, on everything already?
Any reputable journal had a reasonably robust peer review process (exact details vary of course) but historically it was strictly confidential. Only over the past perhaps 2 decades has a process where reviews are made public begun to gain mainstream support.
A positive step! Bravo Nature!
To little, to late? Maybe.
Surely very little, surprisingly late.
There are so many great reasons to distrust "The Science"... perhaps the greatest is that without powerful and persistent skepticism, science simply isn't science. Questioning and skepticism is profoundly fundamental to the scientific process. "Trust the science" or worse yet "Believe the science" are statements that are about as destructive to the foundation of science as one can get.
The whole point of science was that everyone has access to the true nature of the universe which we can discover through theorizing and experimentation. Empirical experience is the great equalizer that puts the Arch Duke of Where-and-What or the Highest of High Priests the on the same footing as, say, Joseph Preistly or Benjamin Franklyn. Questioning and skepticism must be accepted from all quarters. To say that only the most select, distinguished and credentialed can be admitted to the discussion smacks of priesthood and aristocracy.
The argument that we need to re-persuade the people to trust in science is missing the point by such a wide margin as to be a symptom of the problem.
It is more or less the same as fretting that people aren't going to church so much any more.
The science should not be trusted. The science should be tested. Those who say "Trust the science" have completely lost the thread.
[dead]
[dead]