Misrepresenting Colorado Marijuana – Analysis


By David W. Murray*

What is wrong with the marijuana legalization debate, and who is responsible for its sorry state? No better example of misdirection can be offered than the results of a recent Colorado poll. Because of the public health stakes for the nation’s youth, getting this right is essential.

The 2015 version of the Healthy Kids Colorado (HKCS) school survey, which polled both middle school and high school students, garnered a tremendous amount of news. Since it is the first state level estimate to be taken after marijuana legalization (accelerated with retail sales in January 2014), there was reporting concerning possible impact, compared to the previous HKCS taken in 2013.

While not without flaws, the study is an interesting snap-shot of youth health concerns, and legitimately alerts us to some genuine problem areas for marijuana use. But the media reporting was appalling, and the HKCS did little to prevent misunderstanding.

Media Advocacy

Uniformly, media described good news for marijuana legalization advocates. Most coverage (in the Washington Post, Denver Post, Fox News) reported that, compared to two years prior, marijuana use was “flat,” because not “statistically significant.”

Yet even a flat outcome is surprising, not least because other national surveys in Colorado have disclosed alarming increases in use for adults and young adults, rates rising in ten years some 99 percent (7.5 percent to 14.93 percent).

Moreover, in the regional breakdown there were major increases since 2013 in past month use for some students who were juniors and seniors, the increase in some regional breakdowns rising between 50 and 90 percent.

But the survey combined those results with younger grades to produce an overall mean (an ill-advised methodology without weightings), so officials were reported to declare no statistically significant change in marijuana prevalence.

Predictably, the results were treated as a report card on legalization, and media seized on the purported lesson — no rising rates, hence, no worries.

There was even more confusion (or worse) by some media. Both Time magazine and the Scientific American ran articles claiming that the survey had indeed found change, after all. But remarkably, they reported that the new results showed marijuana use had “dipped” from 2013. Of course, in the absence of significance, this claim would be simply wrong. If the non-significant outcome cannot be up, it certainly cannot be down.

There is worse in store. Media thought that calling the changes “not significant” meant that changes were not sufficiently large. But as we shall see, what happened is that the survey did not, methodologically, produce an outcome capable of being statistically significant, as a true weighted probability survey would be. That is a very different matter.

What they seem to have produced is a (partial) census of students, for whom marijuana is variously steeply up or on occasion down, depending on grade and geography. But this survey does not fulfill the necessary criteria for a probability sample.

What National Average?

A further question is, to what are the results being compared? Both the HKCS report itself, and the media, compared the outcome of HKCS 2015 not only to HKCS 2013, but to what they termed the “national average” of youth marijuana use.

According to the reports, 21.2% of Colorado teens were past month marijuana users in 2015. The “national rate” of pot use by youth was reported as higher, at 21.7%.

What is the possible source for deriving that “national average”? There is one genuinely national sample of youth drug use, that from the National Survey on Drug Use and Health (NSDUH) that covers all states. But this cannot be the basis for the claim. In their latest 2014 estimates, NSDUH reported that 7.2 percent of adolescents aged 12 to 17 across the nation used marijuana in the past month – that figure, not 21.7 percent, would be the youth “national average.”

Moreover, the NSDUH specifically declared that Colorado had the nation’s highest rates. Adolescent marijuana use ranged from 4.98 percent in Alabama to 12.56 percent in Colorado.

Worse, the NSDUH showed for youth that from 2009, when medical marijuana took off in Colorado, there has been a stunning rise of 27 percent through 2014 (from 9.91 percent to 12.56 percent). So Colorado youth use rates in the NSDUH are not only higher than the national average, but, after freer access to marijuana, have been steeply climbing.

There is also Monitoring the Future (MTF), a school survey (8th, 10th, an 12th grades) produced by the National Institute on Drug Abuse. In 2014 MTF showed 21.2 percent of students reporting past month marijuana use. But that rate applied only to seniors in their survey, while the HKCS results were supposed to represent all grades. (Rates for 8th graders in the MTF stood at only 6.5 percent.)

Apparently worried about such data contradictions, the Washington Post sought to mitigate concern by pointing out that the HKCS had a very large in-state sample, of 17,000 kids. Says the Post, “That much larger sample could produce a more accurate estimate than the smaller numbers in the federal drug survey.”

But the Post‘s maneuver only magnifies the problems. HKCS is modeled on the Youth Risk Behavioral Survey (YRBS) conducted every two years by the Centers for Disease Control, and the YRBS (which covers most but not every state), also has a sample that estimates a national average.

Critically, however, the YRBS has nowhere near 17,000 youth in each state as their national sample. Their national estimate is based on a total 13,600 responses, for the entire nation; that would average just a few hundred kids per state if evenly distributed. Moreover, the YRBS surveys youth in 9th through 12th grade. Hence, the YRBS is not really comparable to what happened in Colorado.

Likely Missed Most At-Risk Youth

The fact that HKCS and YRBS are both school-based surveys (that is, not taken in households, as is the NSDUH) may account for some of the difference in their respective magnitudes – school surveys produce traditionally higher figures. But this fact raises questions about the validity of any school-based survey.

School-base surveys cannot capture youth not in schools. Given that marijuana use itself is strongly associated with school drop-out rates (as well as high rates of absenteeism), the population of interest may not have been included in a representative fashion. Further, the rising number of young homeless marijuana users flooding into Colorado shows another side of the problem; they are not captured in the school surveys.

That is, the HKCS survey might have systematically missed exactly the kids most at-risk of using marijuana. Of course, if true, the same impact might have affected the HKCS 2013 results that formed the contrast. However, if marijuana use had increased substantially since legalization, there could be a differential impact on student attrition by 2015, as the situation worsened.

Response Rates Undermined Validity

The 2013 survey reported in their “demographics” breakdown that the earlier iteration had an even larger sample than the 2015 run. The 2013 sample was 40,000 youth, who answered with a response rate of 58 percent. But the 2015 sample, the 17,000, registered a truly dismal 46 percent response rate. This is a real problem.

First, when fewer than half of the sample responds, there is a risk that those who did answer are not representative of the actual youth total. Second, that drop in response could signal that there were more youth at risk in the 2015 sample, and hence, not present in the classroom during the second round.

Third, however, these data also show the non-comparability of the surveys. The 58 percent response in the 2013 survey, with a much larger sample, stands in genuine contrast with the lower 46 percent response rate and smaller initial sample for 2015. By way of contrast, the CDC YRBS has an 88 percent student response rate, while the NSDUH stands at 71.2 percent.

Here the point made by the Post about larger samples can be turned back on them; surely the smaller 2015 HKCS would be less accurate than the previous iteration, by the Post‘s own logic. Moreover, the surveys are non-comparable not only because of sample size and response rates, but further because a different set of schools was included in the 2015 iteration, not reporting, for instance, some large school districts near urban areas, where rates of use are often higher.

Given such differences, the two different sets of results cannot meaningfully be put side-by-side. And as we have seen, because the HKCS study was unique to Colorado, there is no methodologically comparable “national average” to which comparisons could be made. Hence, there is no lesson to be derived regarding the impact of legalization; certainly not one sufficiently robust to counter the worrisome NSDUH data to the contrary.

But here’s the most devastating problem of all. The official YRBS, run every two years, requires a 60 percent “participation” rate in order to generate valid weightings for the results. If they are unable to weight a state survey (such as HKCS), they cannot provide an estimate that is representative of the state population. As the CDC participation map shows, Colorado did not receive a proper weighted sample.

This means that the HKCS, according to the criteria offered by the CDC, cannot be used to represent all students in Colorado – there can be no extrapolation of the findings beyond the survey respondents. According to the CDC criteria, these results cannot be extrapolated beyond the participants themselves, and therefore “stands for” no one but the kids who participated. As such, there can be no statistically significant comparisons between these results and previous years, nor with what was termed a “national sample.”

Correction Needed

And yet the media were allowed (where they were not encouraged) to run the “top line” results as though they stood for Colorado youth. And to declare the results “better” following legalization. And to declare use in Colorado post-legalization to be below the national average. As we have seen, none of these statements is warranted.

In fact, we now learn about other states that did not participate in the latest 2015 YRBS round. They include Minnesota, but more importantly, both Oregon and Washington, states that have recently legalized marijuana – that is, states that could have provided a report-card on legalization, but about which we will learn even less than we learned about Colorado. (The next NSDUH state-level estimate won’t come until 2017, post-election.)

This apparent coincidence does little to allay concerns that we are witnessing the effects of a pro-marijuana agenda, perhaps forwarded by well-meaning state boosters and, more surely, by their enablers in the media. Given that other states are looking to Colorado to comprehend their own legalization risks, it is important that the record be corrected.

About the author:
*David W. Murray
is a Senior Fellow at Hudson Institute where he co-directs the Center for Substance Abuse Policy Research.

This article was published by the Hudson Foundation.

Hudson Institute

Hudson Institute is a nonpartisan policy research organization dedicated to innovative research and analysis that promotes global security, prosperity, and freedom.

4 thoughts on “Misrepresenting Colorado Marijuana – Analysis

  • July 4, 2016 at 12:07 pm

    “legitimately alerts us to some geniune problem areas of marijuana use”. No, it doesn’t. The author clearly has no grasp of the benefits marijuana use affords, and consequently misses completely the central issues of the prohibition/normalization debate.

  • July 4, 2016 at 2:55 pm

    The Canadian government sits poised to introduce legislation as early as February of 2017 to legalize marijuana products nation wide. They released a public consultation input website on June 30th. They also offered a 27 page summary of their goals and objectives. The document starts off by stating that as evidenced by high rates of use by youth in Canada the current system most obviously has not been working. They also point to the reduction in the perception of risk associated with the use of marijuana products.
    Chief in their list of goals is to keep marijuana out of the hands of kids and reduce the activities and profits of the black market.
    If the central problem we are trying to solve is the high rate of use by youth the question is whether of not legalization would achieve those ends. It clearly is doing the exact opposite as both Canada and the US have been operating under defacto legalization where marijuana for medical purposes has been made legally available – which in the case of Canada is nation-wide since 2001. Marijuana has been normalized, politicized and promoted to a generation of high school students. The fault of high rates of use is due to a governments that failed to fund prevention and curtail the advance of pot industry interests in the past ten years.
    The market for marijuana products is 2.5 times higher for youth than adults. Under a legal regime they will continue, as these statistics clearly show, to obtain marijuana through illegal sources. The black market stands to prosper and rates of use across all age groups also are expected to rise. The fact that legalization will increase rates of use is acknowledged by the Canadian Government on page 20 of their rather thin public consultation documents.
    The way to stop drug use is to approach both supply and demand and sadly legalization means more marijuana use across society not less.

  • July 6, 2016 at 9:17 am

    No body should be promising that use won’t increase under legalization. I ardently support legalization and thoroughly expect use to increase with increased easier access. And why shouldn’t it?

    I love the pretense that the legalization movement is a movement conducted by egghead social policy specialists who have no inherent interest in cannabis but are only coolly debating the harms and benefits of continued prohibition and legalization. No, the legalization movement is substantively about people who have experienced cannabis on a recreational and medicinal basis thinking it absurd that society legally forbids the consumption of cannabis.

    Creating a bogyman out of the likely eventuality of increased usage is a prohibitionist impulse, and legalization is substantively about the discrediting of prohibition, since it’s clearly anti-personal liberty. Let us ask: Is it worth locking up 750,000 people every year for marijuana related charges so as to prevent “increased usage” of legalization?

    Curious how those with prohibitionist impulses are eager to create a bogyman out of more people using pot, yet aren’t creating a fuss about the true heinous and appalling fact that we are annually locking up 750,000 people over cannabis.

    The majority of Americans support a market for legalized cannabis, and yet rather than respecting the will of the majority over a matter of personal liberty, prohibitionists are now hiding behind children to argue that the freedom of adults to consume cannabis should not be expanded. Absurd.

  • July 6, 2016 at 11:56 am

    I assume that the author of this article thinks it is better for individuals to spend time in prison over a plant. That is the reality of the system currently in place. The question for readers now becomes, if this is the correct solution to the cannabis punishment and are our youths harmed more by cannabis use than by spending time in an enviornment such as correctional facilities where more bad things are learnef rather than good things? My tax dollars should not be used to support an unjust system that systematically makes individuals second class citizens or even worst criminals. Youth use of cannabis is not a new awakening, it has been happening for decades yet, these same youngsters have by majority become productive citizens that contribute to society and pay taxes just as any other individual.


Leave a Reply

Your email address will not be published. Required fields are marked *