Great post. Earmarking funding for pre-registrations, as well as additional funding for replications, and requiring open data - these are all excellent and highly feasible ideas. I've wondered why NIH and NSF don't already do this.
if we threw away all social science research, we would lose absolutely nothing. all of it could disappear tomorrow with zero effect. i take that back; intuition would likely lead to BETTER outcomes than applied social science. curious, i asked chat gtp what the most important findings of social science have been. i was not disappointed (thats to say, i was disappointed, but not surprised.) it spit out some of the most obvious facts of human existence that none with a few days of life on earth and below average iq could not immediately tell you - prospect theory: people violate rational choice assumptions. stanford replication: delayed gratification leads to better life outcomes. broken windows: visible disorder contributes to crime. coleman report: upbringing matters. public choice theory: politicians respond to incentives. (LMAO.) minnesota twins: psychological traits are heritable.
anyone not severely mentally retarded could tell you all of this without thinking for 10 seconds.
99.9% of "social science" is political and ideology propaganda.
While it’s somewhat interesting to have a method to « prove » that most of it is nonsense or irrelevant this is still quite a waste of time.
You won’t convince the people who refuse to do proper science to behave by a rational demonstration.
In any case it all boils to letting in the women in academia, for which most of the « social science » has been created. You can’t really fix anything until they are told to fuck off.
there are a lot of potential "root causes" one could point to for societal and even civilizational decline. but if one had to choose just one ground-level. tangible cause, it would probably be "womens lib" / feminism. catastrophic at a civilizational level.
Pretty much. But feminism is downstream of female behavior, so we were fucked the moment they got too much power.
Some women are pretty decent at operating in a more rational way (depending on settings) but they are such a tiny minority that it’s not worth it.
I think it’s not the first time that society got in trouble for giving women too much power, I believe the Bible and Christianity was basically created to deal with this problem (of course it also has some rules for men’s behavior but it all starts with the necessity of women’s submission).
right. and interestingly, pretty much every single religion and cultural tradition has learned and taught this - the role of women is supportive. every civilization on earth. and i happen to be christian, but leaving that completely aside, also tens of thousands of years of basic science evolutionary theory teaches the exact same thing. for VERY obvious reasons, the womans role is primarily nurturing and raising children and keeping the home. they have been honed by thousands of years of evolution to perfect this role, and men, their role. as "scientific" as libs claim to be, theyre willfully blind to this, or they just flat lie. the feminization of society has been an abject failure. specifically white women, the most coddled, comfortable, safe, living in opulent comfort and luxury group of humans in the history of the planet, are now like, "oh hey. thanks for giving us literal everything, now get off your throne (that we built), we deserve to sit there. you OWE us that." like, WHAT?!?! and weak men are like, "oh. yeah. sorry. sorry for literally dying by the millions to give you all this. here ya go."
I read this in full and I’m not clear on how the replication was done, which undermines the credibility for me. I also have no idea how it handles more qualitative papers, or papers with niche audiences.
Also I perceive an anti-education feel which appeared to persist in the face of relatively good replicatability.
Education is strong because the economists got traction in the top ed schools and policy schools, and because the Institute of Education Sciences, initiated in 2002 by George W. Bush with initial leadership by Russ Whitehurst, required that funded studies have well-powered causal designs, especially RCTs or regression discontinuities, with occasional tolerance for dif-in-dif.
The main paper I’m working on is a registered report. It’s been an uphill struggle. There are only two journals in my field that accept registered reports at all. While I have a revise and resubmit for phase 1, back-and-forth with the referees is taking so long that I would’ve had a normal paper out by now.
I’m also unconvinced registered reports will fix everything in my field, where the critical data sets are large pre-existing ones. For my first paper on a given data set, I can say I’ve never accessed the data set - that’s at least a clear cut assertion, although it requires trust. For all subsequent papers, I have to say that I have access to the data set but have not run the specific analysis in question. That is so much greyer… what counts as “the same analysis”?
I’m grappling with this question right now as I’m about to access a new data set and I want to do some exploratory work – I feel like I should keep a log of everything I try…
Amazing. Thanks for the hard work and this post summarizing what you learned. I have a few questions, perhaps you can answer ate least one. What about papers with qualitative methodologies? Hard to even define replication, no? Were they included in the sample? What about papers with a formal model only, not uncommon in economics (and a bit in polisci). This could help develop fields who lack theory. Were they included in the sample?
Are you using quant replication logic to critique qual social science research? That makes no sense to me. Replication isn’t the standard for qualitative work.
Great post. Earmarking funding for pre-registrations, as well as additional funding for replications, and requiring open data - these are all excellent and highly feasible ideas. I've wondered why NIH and NSF don't already do this.
if we threw away all social science research, we would lose absolutely nothing. all of it could disappear tomorrow with zero effect. i take that back; intuition would likely lead to BETTER outcomes than applied social science. curious, i asked chat gtp what the most important findings of social science have been. i was not disappointed (thats to say, i was disappointed, but not surprised.) it spit out some of the most obvious facts of human existence that none with a few days of life on earth and below average iq could not immediately tell you - prospect theory: people violate rational choice assumptions. stanford replication: delayed gratification leads to better life outcomes. broken windows: visible disorder contributes to crime. coleman report: upbringing matters. public choice theory: politicians respond to incentives. (LMAO.) minnesota twins: psychological traits are heritable.
anyone not severely mentally retarded could tell you all of this without thinking for 10 seconds.
99.9% of "social science" is political and ideology propaganda.
Yep pretty much.
While it’s somewhat interesting to have a method to « prove » that most of it is nonsense or irrelevant this is still quite a waste of time.
You won’t convince the people who refuse to do proper science to behave by a rational demonstration.
In any case it all boils to letting in the women in academia, for which most of the « social science » has been created. You can’t really fix anything until they are told to fuck off.
there are a lot of potential "root causes" one could point to for societal and even civilizational decline. but if one had to choose just one ground-level. tangible cause, it would probably be "womens lib" / feminism. catastrophic at a civilizational level.
Pretty much. But feminism is downstream of female behavior, so we were fucked the moment they got too much power.
Some women are pretty decent at operating in a more rational way (depending on settings) but they are such a tiny minority that it’s not worth it.
I think it’s not the first time that society got in trouble for giving women too much power, I believe the Bible and Christianity was basically created to deal with this problem (of course it also has some rules for men’s behavior but it all starts with the necessity of women’s submission).
right. and interestingly, pretty much every single religion and cultural tradition has learned and taught this - the role of women is supportive. every civilization on earth. and i happen to be christian, but leaving that completely aside, also tens of thousands of years of basic science evolutionary theory teaches the exact same thing. for VERY obvious reasons, the womans role is primarily nurturing and raising children and keeping the home. they have been honed by thousands of years of evolution to perfect this role, and men, their role. as "scientific" as libs claim to be, theyre willfully blind to this, or they just flat lie. the feminization of society has been an abject failure. specifically white women, the most coddled, comfortable, safe, living in opulent comfort and luxury group of humans in the history of the planet, are now like, "oh hey. thanks for giving us literal everything, now get off your throne (that we built), we deserve to sit there. you OWE us that." like, WHAT?!?! and weak men are like, "oh. yeah. sorry. sorry for literally dying by the millions to give you all this. here ya go."
Holy shit
I read this in full and I’m not clear on how the replication was done, which undermines the credibility for me. I also have no idea how it handles more qualitative papers, or papers with niche audiences.
Also I perceive an anti-education feel which appeared to persist in the face of relatively good replicatability.
Sorry haven't had time to read yet but what is the metareplication rate?
Education is strong because the economists got traction in the top ed schools and policy schools, and because the Institute of Education Sciences, initiated in 2002 by George W. Bush with initial leadership by Russ Whitehurst, required that funded studies have well-powered causal designs, especially RCTs or regression discontinuities, with occasional tolerance for dif-in-dif.
Loved reading this as a PhD student. Do you have any similar analyses for areas like STEM and medicine?
The main paper I’m working on is a registered report. It’s been an uphill struggle. There are only two journals in my field that accept registered reports at all. While I have a revise and resubmit for phase 1, back-and-forth with the referees is taking so long that I would’ve had a normal paper out by now.
I’m also unconvinced registered reports will fix everything in my field, where the critical data sets are large pre-existing ones. For my first paper on a given data set, I can say I’ve never accessed the data set - that’s at least a clear cut assertion, although it requires trust. For all subsequent papers, I have to say that I have access to the data set but have not run the specific analysis in question. That is so much greyer… what counts as “the same analysis”?
I’m grappling with this question right now as I’m about to access a new data set and I want to do some exploratory work – I feel like I should keep a log of everything I try…
An oversight committee formed through sortition, which is something scientists should embrace, would be a stepping stone to a systemic solution.
Amazing. Thanks for the hard work and this post summarizing what you learned. I have a few questions, perhaps you can answer ate least one. What about papers with qualitative methodologies? Hard to even define replication, no? Were they included in the sample? What about papers with a formal model only, not uncommon in economics (and a bit in polisci). This could help develop fields who lack theory. Were they included in the sample?
Are you using quant replication logic to critique qual social science research? That makes no sense to me. Replication isn’t the standard for qualitative work.