Lecture 6 + Lecture 6 (IDK)

Each user has unique preferences and behaviors, leading to variability that can make group averages uncertain or misleading

When do we decide if a new system is truly working or not?

By using statistical tests to determine if differences are real or due to chance

The means of two independent groups (e.g., Group A vs Group B)

Difference in means, sample size (n), and variance (s², measure of variability/error)

t(198) = 7.03, p < 0.05 → significant difference, real measurable effect

It shows how big the difference is in practice, not just whether it’s statistically significant

Age, subscription type, device type, genre, time/day, network quality, and user mood

Different groups of participants experience different conditions (e.g., Group A uses Algorithm A, Group B uses Algorithm B)

Groups may differ for reasons unrelated to the condition (e.g., demographics, traits)

The same participants experience both conditions, reducing variability from individual differences

Learning or order effects (practice or bias)

Alternating the order of conditions to reduce order effects (e.g., Participant 1: A→B, Participant 2: B→A)

Construct, internal, external, and ecological validity

Whether you are measuring what you think you are measuring

Whether your manipulation truly causes the observed results or if confounds exist

Whether results generalize beyond the study sample

Whether results are relevant in real-world settings

A controlled change by the experimenter to test causal effects

When comparing 2 groups in a between-subjects design

When comparing the same participants under 2 conditions (within subjects)

When comparing one group’s mean to a known value

When comparing more than 2 groups (between subjects)

When comparing more than 2 conditions within subjects

When assumptions like normality are not met

An alternative approach providing probabilities of hypotheses

Design, test, evaluate

Design: split users into groups, Test: measure engagement, Evaluate: check significance and effect size

Variability makes averages misleading, use statistical tests, design matters, counterbalance in within-subjects, significance shows real effects, effect size shows importance

Data collected without clear purpose, construct validity at risk, misuse of hypothesis testing, findings should be treated as exploratory not definitive

A cognitive architecture with modules (visual, procedural, retrieval, etc.) for simulating cognition

Computational (what problem is solved), algorithmic (how it’s solved), implementational (where it’s implemented)

Descriptive (verbal), box-and-arrow diagrams, computational (math/simulations)

Describe/explain choices, simulate/predict choices, improve choices

Decision-making as evidence accumulation until a threshold is reached, explaining choices and reaction times

Drift rate (info quality/difficulty), starting point (bias), decision threshold (cautiousness vs impulsivity)

Information quality, prior knowledge (bias), and available time (speed vs accuracy)

Evidence accumulation-like processes have been observed in brain regions like LIP

Pressure for speed lowers thresholds (faster but more errors), accuracy requires more time (fewer errors)

Everyday decisions, animal behavior (bees detecting spiders), human perceptual decision tasks

Impairments in speed–accuracy regulation are linked to disorders such as ADHD

Experiments are essential, statistics/design ensure valid results, big data ≠ theory, cognitive models explain decisions, SAT is a key theme in behavior and clinical research

Data is often collected without a reason (at cost of privacy, energy, time).

It's not clear whether the data is the best measure for the question at hand (construct validity at risk).

They are sometimes incorrectly applied (e.g., hypothesis testing presented as a priori when it was exploratory).

Findings are presented as theory-based, whereas they should be presented as exploratory.

Prior knowledge, the amount of time we have, and the information we use.

Whether the experimental manipulation caused the outcome.

Whether an operational measure accurately represents the theoretical construct it's supposed to measure.

The average speed and direction of evidence accumulation.

Stimulus strength/difficulty.

Drift rate.

Slower decisions and more errors.

Starting point.

Faster and more choices towards the favored alternative.

Threshold.

Risky / impulsive choices (more errors).