Critique of CensusLiving between the linesNotesObservations

A Quantitative Methods Professional Answers Us…

Débats - DebatesThis is a response to previous post from a professional who wrote me, but do not wish to be identified for the moment:

Quantitative methods professional

The idea that a volunteer sample reduces the reliability and the validity of data is today as accepted an idea than the one that the earth is round. […] There are many articles that deal with the extent of the bias, its reasons, the ways that can be used to circumvent these biases somehow, etc. But one can never really succeed to circumvent them.

[As for Justice Boivin’s finding] I have not read the arguments in favour of a voluntary survey and how they think they can avoid the sample biases. Of course there is uncertainty about the reliability of data from the NHS, since we have never done this exercise before. There is one certainty about the fact that the data will be biased, but it is difficult to predict in advance the extent and nature of this bias.

Increasing the number of long questionnaires will not change the bias, and nothing leads us to believe that an advertising campaign can correct the bias. The campaign could very well increase it (especially if only in the two official languages).

On Q-1: Can it be demonstrated, from mathematics and experience, how a voluntary operation covering 30% of Canadian households (if performed under optimal conditions) would provide less reliable data than a mandatory one covering 20% of households?

The number of voluntary forms does not matter. Even if you ask 100% of Canadians to answer, the bias will always be there and will affect the sample. The only thing that normally increases with the sample size will be the accuracy of estimates, but these estimates are still biased. Besides, what will the response rate to this questionnaire? Does the government is betting that this response rate will be at least 67% (that’s what it would take to get answers about 20% of Canadians). And what if the response rate was 20% (which is not unusual)?

That being said, if we get 95% of the population that responds, it is much easier to identify the nature of the bias and correct it when you have a sample of 20% or 30%.

On Q-2: Is there a minimum participation threshold below which a census ceases to be useful?

Impossible to answer this question. A telephone survey of 1000 people across Canada can be very useful in itself, but for different purposes. The more one focuses on a specific population (a province, city, neighbourhood, an area or a piece of street), the less the sample is precise. The usefulness is directly proportional to the size of the sample (when it comes to look at subpopulations) and inversely proportional to the size of the bias. But it is impossible to compensate bias by using the sample size.

On Q-3: Would a voluntary operation (conducted in optimal conditions, see below) with an initial sample of 30%, not provide a sufficient theoretical margin of safety, and even a real probability to get results better than a mandatory operation with an initial sample of 20%?

On Q-3: No. Do not confuse the precision of estimates and the bias of these estimates. If we ask all registered households to respond (volunteer) and we get double the number of respondents (say, 5 million), we will still find ourselves with answers different to what could have obtained with a mandatory questionnaire. The answers will be biased, but will allow more accurate estimates, even if these estimates prove to be far from reality. It is as if we spent millions to have a thermometer that could measure the outside temperature to a hundredth of a degree, but the thermometer was not calibrated or incorrectly installed and thus could be wrong by plus or minus 5 degrees Celsius. Why pay more for greater clarity, if the instrument is not valid.

Q-4: Is it not possible to implement measures to mitigate an inclination to lower participation by certain categories of people?

On Q-4: No, it does not work. You do not correct the bias; you simply increase the accuracy of the biased measurement. There are various ways to correct biases, but increasing the sample is not one. To estimate the bias and try to fix it, we must necessarily use a different method of sampling and participation that does not share with the first method, the same bias. For example, if one would imposed in some districts, the compulsory questionnaire in a second phase to all those who have not returned their mandatory questionnaires, and then we would be better able to assess what was the participation bias. In some surveys, some people respond at once, others receive a reminder, others require a second reminder and finally we often manage to go in search of others by offering a voluntary compensation (a small $). These different approaches are taken precisely to try to assess biases. We often realize that the characteristics of people who responded immediately were different from those who responded to the first or second reminder and also different from those who have responded for compensation. We are then in a better position to know in part the nature of bias and to correct it in part. Increase the number of questionnaires sent in the same conditions does not allow a correction of the bias, but will simply confirm the bias.

On Q-5: As small geographic units are generally more or less homogeneous in socio-economic, does such as oversampling not compensate some biases due to low turnout?

Not at all!

On Q-6: Can we not implement a range of other incentives and support measures to mitigate some other biases from a low turnout (for example, those associated with functional limitations or language)?

Probably. We can try to decrease the magnitude of the bias, but an important element is that there will always be uncertainty about the nature and extent of the bias, unless you can quantify it precisely enough to allow us to correct. It will be very difficult if we fail to get responses from people who did not respond the first time voluntarily.

Do not forget the possible campaigns orchestrated by interest groups. And what if the Acadians and francophone groups outside Quebec, if Anglophones in Quebec, the Hasidic or the Muslim community were to campaign either to boycott or, on the contrary, to press for an increase of the turnout of its members? How can we predict these actions among citizens? The high stakes of this survey’s results suggests to me that some groups might be motivated to influence them, as they may do so on the phone in shows, etc.; which will make the nature of bias even more uncertain.

On Q-8: Do the government and its members intend or not to implement an advertising campaign and to support by their statements the participation of as many citizens as possible to ensure the success of the National Households Survey?

As if an advertising campaign could really compensate for all the problems that cause the abandonment of mandatory questionnaire. It would be illusory to believe that it could solve anything.



line
footer
Powered by WordPress | Designed by Elegant Themes
?php comments_popup_link(esc_html__(online ) { ?template_directoryline ))) { ?!--End Footer-- /a) { ?div style=ebusiness_integration_single_top) { ?ul id=