6

: an ongoing methodological crisis in which it has been found that the results of many scientific studies are difficult or impossible to reproduce

started as a psychologist tried to reproduce 100 studies and was able to do it for only 49 of them

"thinking fast and slow book " cites 12 articles which describle 31 studies

analyisis of 31 studies show : 100% significant results + 57% averege observed power

"**less than 50%** a study produced a non significant result"

studies with non significant results are missing

statistical power of the test you have performed, based on the effect size estimate from your data

if you have 10 studies and the average power is 55%, expect 5/ 6 studies with significant result and 4 /5 studies with non significant result

average power is also inflated when researchers report only significant results

Kahneman

scientific credibility may increase in some fields, devrease in others

if fields with low credibility become dominant, they can hurt overall scientific credibility

scientific progress is not guranteed to be continous may be periods during which scientific credibility drops significantly

in order to make a case → create hypothetical a planet "F345" (not so hypotetical actually )

reserch is conducted in a few very popluar filds, there are no genuene nonnull effects to be discovered

what is found is due to only randomn error and bias

"young investigators are taught that the only thing that matters is making new discoveries and finding statistically significant results at all costs "

"replication is considered despicable exercise suitable only for idiots capable of only mimicking, definetly not a serious science "

arguments :

1. practical progress does not mean efficincy
2. practical progress tells us nothing about proportion of scientific results that are not correct
3. linking practical progress to science credibility is tricky

1. are impovements in life expectancy indeed due to medical science or other practical impovements
2. there is even a chance that wrong and inneficint medicine is currently becoming majour disater, inevting resources elswhere would lead to much grater benefits

1) Optimal

2) Self correcting

3) False non replication

4) perpetuated fallacy

to make a correct discovery and to correctly replicate it

the discovery is wrong but the replication allows us to correct the error

original discovery is correct but the replication is wrong

both the discovery and the replication are wrong

1. optimal
2. self correcting

1. false non replication
2. perpetuated fallacy

total prevalnce of the 4 paradigms in replication following "positive " findings is very low : 1%/5%

only aprox 1% of the papers published by the 100 most cited journals are replication efforts

of this 1% , only 18% are direct rather than conceptual replications ( same theoretical idea but everyting else is different )

direct replications are exceptionally uncommon and only a fraction of tose are succesful

clear replications of porposed discoveries are even less common that successful replications

seems to have low status in the psychological community and not too many investigators venture

probably very rare for the same reason

replication effort are done by the same investigators who porposed the original idea → study suffers from confirmation bias

estimated ammount is only 2%

56% findings may be false positives( given the averege of 35% an a nonulleffect of 10%)

ast majority (53%) remain uncallenged facilities

takin gin account potential bias , 95% might be uncallanged fallacies

1. publication bias
2. selective reporting bias ( analyisi and outcomes )
3. fabricated results
4. questionable research practices
5. underpowered studies
6. underestimation of replication crisis

"research that porduces novel studies,statistically significant results and seeminglly 'clear' results is more likely to be published" → use flexibility to get around

the probability that a test of significance will pick up on an effect that is present (will detech a deviation from the null hypothesis, should such deviation exist)

e.g. detecting an effct ( higher rates of cancer in smokers ) given that it actually exists

1. statistical problems that arise if the reserch conducted is without biases that tend to create 'possitive' results
2. problems that affect reflect biases ( and become worse in small, unpowered studies )

median statistical power in neuroscince is between 8 : 31%

the nr of significant studies in their analysis( 349) was significantly higher than expected(254)

smaller studies can inly detect large effects, are suspectable to inflate effect size estimates

reduce, refine and replace

1. Perform a prior power calculation
2. Disclose methods and findings transparently
3. Pre register your study protocol and analyisis plan \Make study materials and data available
4. Collaborate in your research


### The causation vs correlation debate

in some case *third variable* plays an mportant role

or there could be a *common cause* to the problem

randomize subjects to two or more groups , treat them differently and compare the response

important to have control group ( wsmokers and non spokers)

randomnize both people who have yellow fingers and who do not

1. empirical association between X and Y
2. temporal priority of independent variable X
3. non spuriousness ( no confounding thrid variable that causes both X and Y )

*experiment has to establish (by means of intervention ) that changes in X leads to a change in Y (other thisngs being equal )