Methods of social sciences: TestAS practice testlets

Social scientists often want to know the effect of a policy, such as a higher minimum wage, but they can rarely run a randomised experiment in which some regions are assigned the policy by chance. Simply comparing regions with and without the policy is risky, because the regions may differ in many other ways. Comparing one region before and after the policy is also risky, because other things may have changed at the same time, such as the general state of the economy.

A natural experiment uses a situation in which a policy or event affects some groups but not other, similar groups, for reasons that have little to do with the outcome being studied. A famous example is a study by the economists David Card and Alan Krueger, published in 1994. When the US state of New Jersey raised its minimum wage in 1992, neighbouring eastern Pennsylvania did not. Card and Krueger surveyed fast-food restaurants on both sides of the border before and after the increase. Contrary to the prediction of the simple economic model, they found no fall in employment in New Jersey relative to Pennsylvania. The study provoked a long debate, and later research has refined its methods and results.

The method used in such studies is called difference-in-differences. First, the change over time is calculated for the group affected by the policy, the treatment group. Second, the same change is calculated for the unaffected group, the control group. The control group's change shows what would probably have happened in the treatment group without the policy, for example because of a general economic trend. The effect of the policy is estimated as the difference between the two changes.

The method rests on an important assumption, called the parallel trends assumption: without the policy, the outcome in both groups would have developed in the same way. Researchers check this assumption by examining whether the two groups followed similar trends in the years before the policy. If the treatment group was already growing faster before the policy, the estimate will be biased. Other threats include events that affect only one group at the same time as the policy, and people moving between groups in response to it.

The table shows invented figures for a similar study.

Survey results are often presented as cross-tabulations, tables that show how answers to one question are distributed across the categories of another. Reading them correctly requires attention to what the percentages refer to.

Percentages can be calculated in three ways. Row percentages show the distribution within each row, for example what percentage of young people give each answer; each row then adds up to 100%. Column percentages show the distribution within each column, for example what percentage of all "yes" answers come from each age group. Total percentages relate each cell to the whole sample. Which kind is appropriate depends on the question. As a rule, to compare groups defined by the independent variable, percentages are calculated within these groups: to see whether age influences an opinion, one compares the percentage of each age group that holds the opinion.

Differences between percentages are expressed in percentage points. If 40% of one group and 60% of another agree, the difference is 20 percentage points, although the second figure is 50% higher than the first.

The number of cases matters. A percentage based on 20 respondents is much less reliable than one based on 2,000, and reports should always state the number of cases for each group. It is also important to check whether categories such as "don't know" have been excluded.

Finally, as with any association, a difference between groups does not by itself show a cause. Age differences in opinions may reflect life stage, generation or other characteristics that differ between age groups, such as education.

Before interpreting any table, it helps to ask four questions: what population the data describe, how the sample was drawn, how the question was worded and which categories were counted. A cross-tabulation can then be read systematically, first along the rows to compare groups and then down the columns to see where totals come from. It is also useful to calculate a few percentages yourself rather than relying on the text that accompanies a table, since reports sometimes emphasise the most dramatic figure.

The table below shows fictional results from a survey of 1,000 people who were asked whether they support a ban on cars in the city centre.