Critique of CensusDebatesLiving between the linesNotesObservations

Questions for Statisticians and Specialists in Quantitative Methods regarding the Reliability of a Voluntary Census

In the wake of the decision on a application for judicial review form the Fédération des communautés francophones et acadienne du Canada

ObservationsFederal Court’s Justice Richard Boivin heard evidence and testimonies presented in support of and in opposition to the National Household Survey (NHS) which, being voluntary, replaces the old census long form, which was mandatory under fine and even imprisonment. The judge ruled this week that “there is uncertainty about the reliability of the data that will come from the NHS” … except that the Court is “not convinced that the data of the NHS will be so unreliable as to be unusable.”

Let’s recall that the Conservative government decided to remove the long form from the mandatory status of the Canadian census to make it voluntary instead. To offset a possible decline in participation, it provided an increase of around 50% of the number of long questionnaires (from 3 to 4.5 million households at an additional cost of $ 30 million) plus an advertising campaign to spur participation.

Many statisticians, demographers and researchers have criticized this decision. According to them, a voluntary survey would lead to a significant decrease in participation, particularly in certain portions of the population (the poorest, the least educated, of certain ethnic backgrounds). The result would be less representative and thus biased data which would distort the demographic profiles of country, regions and local communities. However, beyond these general statements, public interventions in the media so far have provided no statistical demonstration in support to this claim. Justice Boivin’s finding seems to confirm this perception.

So I make an appeal to statisticians and specialists in quantitative methods in order to clarify certain key elements of the debate.

Level of participation

Q-1: Can it be demonstrated, from mathematics and experience, how a voluntary operation covering 30% of Canadian households (if performed under optimal conditions) would provide less reliable data than a mandatory one covering 20% of households?

Note: Some media have reported 30% and some have mentioned “one in three households, so 33.3%, others 35%. I am currently waiting confirmation of a figure from Statistics Canada.

October 14 update: Statistics Canada answered that it was 33.3%.

Let’s remind that in recent decades, Canadians proved to be compliant: the participation rate in the mandatory census has turned around a remarkable 95% (i.e. less than one non-participating household out of 20). Now our Americans neighbours, on their side, manage their populous country with data coming from a participation of less than 75% (abstention of more than one household out of 4). This brings us to another question:

Q-2: Is there a minimum participation threshold below which a census ceases to be useful?

It is possible that there are several of those thresholds here depending on the type of data, the type of use and the spatial scale (countrywide versus city block). Obviously we want to have the most reliable and accurate data as possible. However, such minimum thresholds should exist, even if they are largely agreed upon standards between producers and users of data. We need to know these thresholds if want to form an opinion on the data supporting a discussion (for example, figures on some First Nations reservations where participation is very low) or to consider the necessity for measures to improve participation (for example, advertising campaigns, increased number of households approached, methods to recall non-respondents).

Sample size

In surveys, a low participation rate is often offset by increasing the sample size of contacted individuals.

As a first step, one calculates the necessary target sample size (called “theoretical sample” of surveyed population) to obtain the level of accuracy (or error margin) sought. The second step is to evaluate the participation rate, taking into account an estimate of households who will refuse to answer, those who will not respond in whole or in part and those who will be away during the responses collecting period. With these two elements, size “n” of the sample target and anticipated response rate, then calculate the size “n” of the initial sample (or start sample) required by the following formula (in fact, the formula should include other elements that are not necessary to be discussed here):

n initial = n target * 1 / response rate

Let’s illustrate with a concrete scenario arbitrary using figures from the 2006 census. In 2006, 12,347,500 households were counted in Canada. The administration of the long form to 20% of households thus implies to contact 2,487,500 of them. A remarkably reliable response rate of 95% (if uniformly distributed, see below) results in a final sample of 2,363,125 households.

Suppose one of the worst scenarios, a drastic fall in the turnout at only 70% because answer to the long form is no more compulsory but voluntary. To compensate, we would thereby increase the initial sample that can reach the same final number of respondents, namely by

n initial = 2,363,125 * 1 / 0.70 =  3,375,892

Interestingly, 3,375,892 represent 27% of all Canadian households. Thus, according to the canonical formula, Question 1 could be reformulated as:

Q-3: Would a voluntary operation (conducted in optimal conditions, see below) with an initial sample of 30%, not provide a sufficient theoretical margin of safety, and even a real probability to get results better than a mandatory operation with an initial sample of 20%?

The non-participation bias

It is a well known and documented fact that the participation of certain portions of the population to voluntary surveys is often lower. We are talking about people more poorly educated, poorer, with various disabilities, who speak little of one or the other official languages, of certain ethnic backgrounds. However,

Q-4: Is it not possible to implement measures to mitigate an inclination to lower participation by certain categories of people?

As a first step, there is the fact that a census is anchored in the geographical space that delineates the boundaries of country, provinces and territories, local communities and their circulation paths. A census requires to compose a grid over the entire country and to locate each identified household at a specific location in space. It is therefore possible to verify, and correct, response rates in small geographical unit, such as one block of town homes or a few kilometres of road in the countryside.

Consider such a unit covering 200 households. A sample of 30% already represents the considerable number of 60 households! A minimum threshold of 70% participation would require response from at least 42 households among them, more than one in five households located in that geographical unit. Suppose we find that, after some time, only 37 households responded. What prevents us to randomly draw another 15 households to solicit in order to make sure to exceed the minimum participation threshold?

Q-5: As small geographic units are generally more or less homogeneous in socio-economic, does such as oversampling not compensate some biases due to low turnout?

As a second step,

Q-6: Can we not implement a range of other incentives and support measures to mitigate some other biases from a low turnout (for example, those associated with functional limitations or language)?

Indeed, has Statistics Canada not been able to find ways to census homeless people, certainly one of the categories of people who are the most difficult to reach and engage?

Optimal conditions

Obviously, the measures suggested above require consistent resources, planning and preparation time. This raises a new question, namely:

Q-7: How many months does it take for a body such as Statistics Canada to design and implement a voluntary survey operation likely to produce reliable data?

In addition, the success of a voluntary survey also depends on the level of social consensus about the importance of the census, and thus the participation of citizens in it. Clear signals must also be sent to the effect that there is no reason not to answer as honestly as possible since, not only participation in the survey is voluntary, but answer to each and every question is also voluntary.

In the last few months however, the interventions of several Conservative MPs and Minister have often questioned the very legitimacy of a number of questions from the census’ long form, or even the usefulness of a census to begin with. This brings us to a question of an entirely different nature, this time addressed to the Conservative Party’s government:

Q-8: Do the government and its members intend or not to implement an advertising campaign and to support by their statements the participation of as many citizens as possible to ensure the success of the National Households Survey?

Here are only a few questions. I will note the responses. And there are other questions. Which in turn raise many other questions. Thus, to be continued…

October 14 update: one answer received

Quantitative methods professional:

The idea that a volunteer sample reduces the reliability and the validity of data is today as accepted an idea than the one that the earth is round. […] There are many articles that deal with the extent of the bias, its reasons, the ways that can be used to circumvent these biases somehow, etc. But one can never really succeed to circumvent them.

[As for Justice Boivin’s finding] I have not read the arguments in favour of a voluntary survey and how they think they can avoid the sample biases. Of course there is uncertainty about the reliability of data from the NHS, since we have never done this exercise before. There is one certainty about the fact that the data will be biased, but it is difficult to predict in advance the extent and nature of this bias.

Increasing the number of long questionnaires will not change the bias, and nothing leads us to believe that an advertising campaign can correct the bias. The campaign could very well increase it (especially if only in the two official languages).

On Q-1: The number of voluntary forms does not matter. Even if you ask 100% of Canadians to answer, the bias will always be there and will affect the sample. The only thing that normally increases with the sample size will be the accuracy of estimates, but these estimates are still biased. Besides, what will the response rate to this questionnaire? Does the government is betting that this response rate will be at least 67% (that’s what it would take to get answers about 20% of Canadians). And what if the response rate was 20% (which is not unusual)?

That being said, if we get 95% of the population that responds, it is much easier to identify the nature of the bias and correct it when you have a sample of 20% or 30%,

Q-2: Impossible to answer this question. A telephone survey of 1000 people across Canada can be very useful in itself, but for different purposes. The more one focuses on a specific population (a province, city, neighbourhood, an area or a piece of street), the less the sample is precise. The usefulness is directly proportional to the size of the sample (when it comes to look at subpopulations) and inversely proportional to the size of the bias. But it is impossible to compensate bias by using the sample size.

On Q-3: No. Do not confuse the precision of estimates and the bias of these estimates. If we ask all registered households to respond (volunteer) and we get double the number of respondents (say, 5 million), we will still find ourselves with answers different to what could have obtained with a mandatory questionnaire. The answers will be biased, but will allow more accurate estimates, even if these estimates prove to be far from reality. It is as if we spent millions to have a thermometer that could measure the outside temperature to a hundredth of a degree, but the thermometer was not calibrated or incorrectly installed and thus could be wrong by plus or minus 5 degrees Celsius. Why pay more for greater clarity, if the instrument is not valid.

On Q-4: No, it does not work. You do not correct the bias; you simply increase the accuracy of the biased measurement. There are various ways to correct biases, but increasing the sample is not one. To estimate the bias and try to fix it, we must necessarily use a different method of sampling and participation that does not share with the first method, the same bias. For example, if one would imposed in some districts, the compulsory questionnaire in a second phase to all those who have not returned their mandatory questionnaires, and then we would be better able to assess what was the participation bias. In some surveys, some people respond at once, others receive a reminder, others require a second reminder and finally we often manage to go in search of others by offering a voluntary compensation (a small $). These different approaches are taken precisely to try to assess biases. We often realize that the characteristics of people who responded immediately were different from those who responded to the first or second reminder and also different from those who have responded for compensation. We are then in a better position to know in part the nature of bias and to correct it in part. Increase the number of questionnaires sent in the same conditions does not allow a correction of the bias, but will simply confirm the bias.

Q-5: Not at all!

Q-6: Probably. We can try to decrease the magnitude of the bias, but an important element is that there will always be uncertainty about the nature and extent of the bias, unless you can quantify it precisely enough to allow us to correct. It will be very difficult if we fail to get responses from people who did not respond the first time voluntarily.

Do not forget the possible campaigns orchestrated by interest groups. And what if the Acadians and francophone groups outside Quebec, if Anglophones in Quebec, the Hasidic or the Muslim community were to campaign either to boycott or, on the contrary, to press for an increase of the turnout of its members? How can we predict these actions among citizens? The high stakes of this survey’s results suggests to me that some groups might be motivated to influence them, as they may do so on the phone in shows, etc.; which will make the nature of bias even more uncertain.

On Q-8: As if an advertising campaign could really compensate for all the problems that cause the abandonment of mandatory questionnaire. It would be illusory to believe that it could solve anything.

Powered by WordPress | Designed by Elegant Themes
?php comments_popup_link(esc_html__(online ) { ?template_directoryline ))) { ?!--End Footer-- /a) { ?div style=ebusiness_integration_single_top) { ?ul id=