A Less Punishing World: Contradictions in Behavior Analysis, Autism, and Punishment

Abstract

This paper reviews the research in punishment for specific behaviors of autistic children. I propose that the rationale for punishment contradicted previous behavior analytic research. I will juxtapose these against the arguments of B. F. Skinner who advocated against punishment.

Introduction

In 1972 the American Humanist Society awarded B. F. Skinner, their “Humanist of the Year” award. It was dedicated for his efforts to show how “a less punishing world” was possible. It was a controversial choice to say the least. Skinner was after all, the man who had authored a book called “Beyond Freedom and Dignity” that had come out just the previous year (Skinner, 1971).

Skinner (1953) was strongly anti-punishment. He opposed it for five reasons. (1) Its ineffectiveness in decreasing inappropriate behavior (Skinner, 1953; Skinner 1971; Skinner, 1974; and Skinner, 1985). (2) Considerations that the negative behavior being punished is not operant, but caused by some other process (Skinner, 1953). (3) The production of side effects (Skinner, 1953). (4) The ethics of using punishment when other techniques are viable (Skinner. 1953; Skinner 1971; Skinner, 1974; and Skinner, 1985). (5) The quality of life that aversives adversely affect (Skinner, 1953; Skinner 1971; and Skinner, 1974).

With apologies to the opinion of many, B. F. Skinner was not THE behavior analyst; he was however the first among his peers to provide ethical precedent and not just the pragmatic consideration in his analysis of punishment. I argue that knowledge of the history of behavior analytic research into punishment is important as it discloses inconsistancies, in particular when compared to the ethical based arguments of Skinner.

Punishment in the behavior analytic sense, is not the same as in the broader English sense. In behavior analytic sense, punishment is: Any stimulus event or condition, who’s immediate response contingent presentation results in a decreased frequency of that response. So, that is the key, it has to immediately follow a behavior and it must be shown to decrease the behavior. Punishment in the broad sense of the word, might be to take a toy away. However, if that doesn’t decrease the behavior, it isn’t punishment in the behavior analytic sense. On the flip side if a teacher showers a child with praise, and that praise decreases the response that delighted the teacher, then the praise is punishment, in that case.

Review of Literature

Skinner’s argument that punishment doesn’t work has largely been rejected by the behavior analytic community, based on other research. It has been informally noted that Skinner disliked punishment based operants and he may have overlooked relevant facts. This might be so, but it is also a psychogenetic fallacy, to imagine Skinner’s thoughts and use this to explain his actions away. In any case the behavior analyst quickly adapted and adopted punishment.

Nate Azrin, one of the most prolific behavior analysts ever, was at the forefront of this shift. In fact psychology in general is in debt to him as he was part of a cohort of young psychologists from various schools of thought who were forcing psychology towards science and away from armchair analysis.

An early example was Azrin (1956) with pigeons. An early human example was Azrin (1958). Later he researched "time-out" which is easily (although anecdotally) the most popular punishing technique used for any sort of child (Azrin, 1961).

Azrin was not the only one looking at aversives. Todd Risely, is another famous old guard behavior analyst. Risley (1968) came on the scene for the first edition of the brand new Journal of Applied Behavior Analysis with a study of the effects and side effects of punishment for an autistic child. However, he wasn’t quite the the first to use punishment for autistic children. One earlier example was Lovaas, Schaeffer, & Simmons (1965), who used electric shock to build repertoires of social behavior in children who were diagnosed as childhood schizophrenics and who today, would have likely been diagnosed on the autism spectrum.

This research got into the media and caused a bit of a stir. Even Bettelheim has a few things to say about this on pages 410-411 of “The Empty Fortress” (Bettelheim, 1967) Moreover, an early cognitive/behavioral psychologist attacked this on ethical grounds. Bregar (1965) congratulated Lovaas et al. (1965) from freeing themselves from Skinnerian dogma, in terms of effectiveness of punishment, but attacks the ethical integrity of non-consent for use of aversives in this study.

Another problem pointed is the production of side effects. Thanks to the research of Ulrich & Azrin (1962) we knew early on that punishment induces temporary aggression in rats and gerbils (although not guinea pigs). There is a big step from pigeons to humans and humans do not necessarily instantly attack the nearest person when they are punished. However thanks to the efforts of another set of Doctors we know that seeing aggression can lead to aggression in children. The first (and no surprise) is Bandura (1961), but the second doctor whom we must credit and who also published in the same year, was Lovaas (1961).

Punishment based research on animals continued, but now the shift was towards a variety of human problems such smoking (Powell, & Azrin, 1968) and poor posture (Azrin, Rubin, O'Brien, Ayllon, & Roll, 1968). However punishment for autistic children also picked up steam. (Tate & Baroff, 1966; Azrin, Kaplin, & Foxx, 1973; and Koegel, Firestone, Kramme, & Dunlap, 1974).

A late occurrence was Lichstein (1976). This sparked some rejoinders. Most notable were from two a husband and wife who again addressed the ethics of non-consent and who also questioned the need for aversives in when other interventions were possible (Shea & Shea, 1976). This just preceded Wolf (1978), this was about the time when behavior analysts were figuring out that social validity was highly important in our research.

So, aversives were causing problems in terms of social validity for behavior analysts, but we knew via research that aversives were quite effective. Other alternatives were investigated.. Tanner & Zeiler, (1975) used noxious aromas. Moore & Bailey (1973) used social punishment delivered by the mother of the child. Foxx & Azrin (1973) used overcorrection. This is the concept that a person more than corrects (more than restitution) the relevant ill behavior. Friman, Cook, & Finney, (1984) compared a sprayed water mist spray to vinegar and lemon juice and found it just as effective. It looked like the perfect aversive for a while. It didn’t cause pain. It was just annoying. However, it too did not pass the test of social validity.

Klier & Harris (1977) did some truly fascinating research using three autistic children to see if an inverse relationship existed between self-stimulatory behavior and learning; that is, the authors tried to see if self-stimulatory behavior got in the way of a leaning task. For two of the three children the answer was no. The authors concluded that maybe we don’t have to get rid of self-stimulatory behavior. That seems like a doubly good thing, autistic persons write that self-stimulatory behavior is important as it allows autistics to cope with difficult or uneasy situation. That looks like the quality of life issue that Skinner brought up. One of Dr. Harris’s next research projects involved looking at ways to suppress self-stimulatory behavior in autistic children (Harris & Wolchik, 1979).

Sandra Harris later went on to investigate the relationship between allowing staff to use mild aversives and allowing them to strong aversives. They found that staff who were allowed to use strong aversives were more likely to remain with the center and had reported that they felt that they accomplished more. Harris, Handleman, Gill, & Fong (1991) write “Allowing staff to use a range of interventions including strong aversives may diminish job stress and enhance their sense of personal efficacy.” However, this article did not establish if such aversives truly were effective or justified in terms of helping the students.

In the mean time Dr. Lovaas was not idle and was still working with autistic kids as well as with effeminate boys. Rekers & Lovaas (1974) and Rekers, Lovaas, & low (1974) used spanking to modify the behavior of the effeminate boys to make them more masculine. Part of the critique offered by Nordyke, Baer, Etzel, & LeBlanc, (1977) concerned the use aversives.

Flashing forward, Lovaas, Ackerman, Alexander, Firestone, Perkins, Young,
Carr, & Newsom (1981) came out with “The Me Book”. This book was simple, straightforward and readable. It also advised aversives; generally (but not only), a single slap to the thing or buttocks. However, this still preceded the landmark study Lovaas (1987) in which following two years of treatment based on “The Me Book” 47% of the experimental group became indistinguishable from typically developing peers. I will not comment further on this study except to say that it is still the source of great controversy and that the active ingredient in it was contingent aversives.

Conclusion

Many of the late greats mentioned in this post like Baer, Wolf, and Skinner have shuffled off for the big experimental space in the sky. Nate Azrin is still teaching in Nova Southeastern University in Florida. Risley is now an advocate for exclusively positive practices and Lovaas has repudiated his use of aversives. Behavior analysts have almost entirely abandoned the use of physically aversive techniques for autistic children. Is this a happy ending then?

I would submit that the answer is “no”. Skinner argued five reasons exist that show punishment should not be used. One was shown to be false. The other four remain.


References

Azrin, N. H.(1956); Some effects of two intermittent schedules of immediate and non-immediate punishment. Journal of Psychology: Interdisciplinary and Applied, 42, 3-21.

Azrin, N. H. (1958). Some effects of noise on human behavior. Journal of the Experimental Analysis of Behavior, 1, 183-200.

Azrin, N. H. (1961). Time-out from positive reinforcement. Science, 133, 382-383.

Azrin, N. H., Hutchinson, R. R., & Hake, D. F. (1966). Extinction-induced aggression. Journal of the Experimental Analysis of Behavior, 9, 191-204.

Azrin, N. H., & Hutchinson, R. R. (1967). Conditioning of the aggressive behavior of pigeons by a fixed-interval schedule of reinforcement. Journal of the Experimental Analysis of Behavior, 10, 395-402.

Azrin, N. H., Rubin, H. B., O'Brien, F. J., Ayllon, T., & Roll, D. L. (1968). Behavioral engineering: Postural control by a portable operant apparatus. Journal of Applied Behavior Analysis, 1, 99-108.

Azrin, N. H., Kaplin, S. J., & Foxx, R. M. (1973). Autism reversal: Eliminating stereotyped self-stimulation in retarded individuals. American Journal of Mental Deficiency, 1973, 78, 241-248.

Bandura, A.; Ross, Dorothea; Ross, Sheila A. (1961). Transmission of aggression through imitation of aggressive models. Journal of Abnormal & Social Psychology, 63(3), 575-582.

Bettelheim, B. (1967). The empty fortress: Infantile autism and the birth of the self. Oxford: Free Press of Glencoe.

Bregar, L. (1965). Comments on “Building social behavior in autistics children by use of electric shock. Journal of Experimental Research in Personality. 1, 110-113.

Friman, P. C., Cook, J. W., & Finney, J. W. (1984). Effects of punishment procedures on the self-stimulatory behavior of an autistic child. Analysis and Intervention in Developmental Disabilities, 4, 39-46.

Foxx, R. M. & Azrin, N. H. (1973). The elimination of autistic self-stimulatory behavior by overcorrection. Journal of Applied Behavior Analysis, 6, 1-14.

Harris, S.L., Handleman, J.S., Gill, M.J., and Fong, P.L. (1991). Does punishment hurt? The impact of aversives on the clinician. Research in Developmental Disabilities, 12, 17-24.

Harris, S. L. & Wolchik, S. A. (1979). Suppression of self-stimulation: Three alternative strategies.. Journal of Applied Behavior Analysis, 12, 185-198.

Holz, W. C., & Azrin, N. H. (1962). Interactions between the discriminative and aversive properties of punishment. Journal of the Experimental Analysis of Behavior, 5, 229-234.

Klier, J. & Harris, S. L. (1977). Self-stimulation and learning in autistic children: Physical or functional incompatibility? Journal of Applied Behavior Analysis, 10, 311.

Koegel, R. L. Firestone, P. B. Kramme, K. W. & Dunlap, G. (1974). Increasing spontaneous play by suppressing self-stimulation in autistic children. Journal of Applied Behavior Analysis, 7, 521-528.

Lichstein, K. L. (1976). Employing Electric Shock With Autistic Children. Journal of Autism and Childhood Schizophrenia, 6(2), 163-173.

Lovaas, O. I. (1961). Effect of exposure to symbolic aggression on aggressive behavior. Child Development, 32, 37-44.

Lovaas, O. I., Schaeffer, B., and Simmons, J. Q. Experimental studies in childhood schizophrenia: building social behavior in autistic children by the use of electric shock. Journal of Experimental Research Personnel, 1965, 1, 99-109.

Lovaas, O.I., Ackerman, A., Alexander, D., Firestone, P., Perkins, M., Young, D.B.,
Carr, E.G., & Newsom, C. (1981). Teaching Developmentally Disabled Children: The
Me Book. Baltimore, MD: University Park Press.

Moore, B. L. & Bailey, Jon S. (1973). Social punishment in the modification of a pre school child's autistic-like behavior with a mother as therapist. Journal of Applied Behavior Analysis, 6, 497-507.

Nordyke, N.S., Baer, D.M., Etzel, B.C., & LeBlanc, J.M. (1977). Implications of the stereotyping and modification of sex role. Journal of Applied Behavior Analysis, 10, 553-7.

Powell, J., & Azrin, N. H. (1968). The effects of shock as a punisher for cigarette smoking. Journal of Applied Behavior Analysis, 1, 63-71.

Rekers, G.A., and Lovaas, O.I. (1974). Behavioral treatment of deviant sex-role behaviors in a male child. Journal of Applied Behavior Analysis, 7, 173-90.

Rekers, G.A., Lovaas, O.I., & Low, B. (1974). The behavioral treatment of a "transsexual" preadolescent boy. Journal of Abnormal Child Psychology, 2, 99-116.

Risley, T. R. (1968). The effects and side effects of punishing the autistic behaviors of a deviant child. Journal of Applied Behavior Analysis, 1, 21-34.

Shea, J., Shea, N. (1976). Reactions to "Employing Electric Shock With Autistic Children". Journal of Autism and Childhood Schizophrenia, Vol. 6(3), 289-294.

Skinner, B. F. (1953). Science and Human Behavior. New York: Macmillan, 1953.

Skinner, B. F. (1971). Beyond freedom and dignity. New York: Alfred A. Knopf.

Skinner, B. F. (1974). About Behaviorism. New York: Alfred A. Knopf. Skinner, B. F. (1948). Walden two. New York: Macmillan.

Skinner, B. F. (1985). Toward the cause of peace: What can psychology contribute? Applied Social Psychology Annual, 6, 121-25.

Solnick, J. V. Rincover, A. & Peterson, C. R. (1977). Some determiners of the reinforcing and punishing effects of timeout. Journal of Applied Behavior Analysis, 10, 415-424.

Tanner, B. A. & Zeiler, M. (1975). Punishment of self-injurious behavior using aromatic ammonia as the aversive stimulus.. Journal of Applied Behavior Analysis, 8, 53-57.

Tate, B. G. and Baroff, G. S. (1966). Aversive control of self-injurious behavior in a psychotic boy. Behavior Research and Therapy, 4(2), 281-287.

Ulrich, R. E., & Azrin, N. H. (1962). Reflexive fighting in response to aversive stimulation. Journal of the Experimental Analysis of Behavior, 5, 511-520.
Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11(2), 203-214.

Fallacies, Autism, and Homeopathy

Some time ago I joined the yahoo group EoHarm with the purpose of learning from primary sources about the opinion and positions of parents who believe in a Vaccine induced etiology of autism and in biomedical treatment. I have not participated in that group as I see no evidence that it welcomes dissent or critical commentary. I have however, approached this group with the mindset that perhaps they have something to teach me.

I read a message on that board recently that I think should be challenged as it is mostly based on fallacies premises and has the potential to do harm to autistic children. It is not my wish to copy someone’s message from a closed membership list. However, I greatly wish to challenge this argument. Also, this should be given as an example of how quackery has infiltrated vaccine etiology of autism theory advocates. This is particularly the case as the moderator of that group supported the Homeopath by citing non autism studies of Homeopathies effectiveness.

Message #21050 http://groups.yahoo.com/group/EOHarm/message/21050

“A Homeopath makes a prescription of a remedy usually "potentised" (or energised) based on the law of similars which is that which causes something can also cure it.
Remedies are made of all sorts of things from the animal , vegetable, or mineral realm. What is given depends upon the symptoms of the individual patient, not the disease name.
The healing energy of the substance when potentised and when similar to the disease symptoms will produce a cure.
Think of it as two frequency waves of the same amplitude and wavelength but in opposite phase being able to cancel each other out. If the amplitude, or wavelength do not match there will not be a cure.
Homeopaths can be medically trained or can not. Sometimes an allopathic mindset is difficult to deprogramme and can be a hinderance to being a good homeopath as they are constantly thinking in disease terms rather than patient terms.
Homeopathy has been around for over 200 years and is usually very safe . The difficulty in proving this absolutely is mainly due to the fact that there are no two people that are alike and it is very difficult to do the standard double blind trials that they do with other allopathic drugs. You may have 50 people with say , migraines all needing a different remedy.
In the 18th century the main form of health care in the US was Homeopathy and it did very well compared with allopathy which had far more side effects and deaths from the treatment. After the advent of antibiotics the current Medical monopoly started to take hold and got rid of homeopaths as they were a threat to their way of doing things. Many doctors were struck off for even talking to homeopaths!!!!
Theres got to be something to it then!”

The Law of Similars is very adequately addressed by Dr. Barrett here: http://www.homeowatch.org/basic/similars.html It is based on the notion that the cure of certain given symptoms is provided by substances that produce similar symptoms; an interesting concept, but one without any research assessing it for autism. In general for other various conditions the research is quite mixed. Nor has this “law” been established by basic/observational research.

Indeed the author states the difficulty in producing double blind studies that show effect as the each patient may require a separate remedy. This functions as a red herring, it distracts from, but does not adequately explain why Homeopathic remedies can not stand up to controlled studies. If the Homeopaths have a means of discerning what patient should receive what cure (determined in patient terms I assume) then they could group individuals similar in this regard and have a testable hypothesis that could be assessed in controlled trials.

The “potentised” concept refers to the idea that a remedy can be greatly diluted and the larger medium, e.g. (water) will retain the same potent energy, or this will even be increased. That a given amount of material can be reduced and yet have a greater medical impact, is an example of magical thinking. Not surprisingly, it is not substantiated in the research.

The author does not explain what “healing energy” might be, or how the presence of this energy is determined and measured. The concept that an Allopathic mindset (the term Allopath is used by many practitioners of alternative medicine to describe contemporary or historical science based medicine) is difficult to deprogram is an interesting statement. However, I would imagine that a logical mindset in general might make it difficult to effectively practice Homeopathy.

The conclusion that in the 18th century had Allopathic treatments caused far more side effects when compared to Homeopathy has no precedent in the literature, namely as proper recording and the current controlled trials were not yet implemented. This is an example of ad hoc reasoning.

The conceptualization of two frequency waves canceling each other seems to be an example of the fallacious use of jargon. This is a wave process that most lay persons are unlikely to be familiar with. It sounds impressive, but distracts from the issue at hand.

The final statement that author presents a conclusion that does not follow and is therefore a non-sequitur, it also assumes the greed of allopath’s and is therefore a ad Hominem, it also fallaciously assumes a hidden truth, when there is no logical reason to assume such. That argument is easily, but not fallaciously, reducible to absurdity.

The author’s logic is represented in this syllogism:

P. Those who greatly oppose an idea recognize that it has validity (unstated).
P. Allopaths greatly oppose Homeopathy.
C. Allopaths who oppose Homeopathy secretly recognize its validity.

The major premise is unstated and unsubstantiated. Based on this logic I will state:

P. Those who greatly oppose an idea recognize that it has validity.
P. My fifth grade teacher opposed the answers on my spelling test.
C. My teacher recognized the validity of my answers on my spelling test.

P. Those who greatly oppose an idea recognize that it has validity.
P. My friends greatly opposed the idea of drunk driving.
C. My friends secretly recognize the validity of drunk driving.

P. Those who greatly oppose an idea recognize that it has validity.
P. Interverbal greatly opposes the use of illogic.
C. Interverbal actually recognizes the validity of illogic.

P. Those who greatly oppose an idea recognize that it has validity.
P. A Canadian individual opposes a State mandated treatment for autistic persons.
C. That Canadian persons actually recognizes the validity of State mandated treatment.

P. Those who greatly oppose an idea recognize that it has validity.
P. The members of the yahoo group EoHarm condemn those who argue that there has been no epidemic of autism.
C. The EoHarm members recognize the validity of the no epidemic of autism statements.

In conclusion that post to EoHarm is an example of the inroads of alternative medicine into the vaccine etiology theory of autism camp. The arguments presented within are primarily fallacious. It is to be hoped that the inroads of quackery into the ranks of the vaccine etiology of autism theory advocates will not continue.


References

Milazzo, S., Russell, N., & Ernst, E. (2006). Efficacy of homeopathic therapy in cancer treatment. 42(3), 282-9.

28th Skeptic’s Circle

The 28th Skeptic’s Circle is up and rolling at Unused and Probably Unusable. It has a ton of links to the best skeptical blogging on the internet. It is a good place for logic, science, and precision. So check it out:

http://unusedandunusable.powerblogs.com/posts/1139269297.shtml

Ethics of Engineering the Illusion of Choice In life and In Applied Behavior Analysis

What if we had no free will? What if our actions were determined by a combination of genetic and environmental factors. What if, the environment in our lives interacted with our cognition and other biological systems to produce outcomes that give us the illusion of choice, but with no choice is truly present.

If this is true are we “hoist by our own petard”? Have we usurped humanity? Are we automatons by default? Is there reason to still exist? Are these just some silly questions that no one besides young college students ask? Is this just one of those “Yeah it is good to think about it, but you can’t take these things too seriously or you will end up huddled on a mountain in a small wooden shack frantically writing an obscure philosophy manifesto, or worse, become a University professor” type of thing?

So, who cares? Well, I care. I care because, these things affect in large or small way, real people. It is way beyond the scope of this University student to write about these things in depth, but I will talk about one instance where this applies and rather directly affects real people. In this post I will look at an issue from the perspective that free will does not exist.

Behavior analysts have found a strange effect where people will prefer to use slot machines that have buttons to stop the wheels. The wheels are moving way too fast for the people to have any chance of actually influencing where the wheels stop, their odds of winning are no better than chance. Folks seem to prefer this over the simple crank handle slot machines. I think I would too, to be honest. The research on this is still ongoing, so I am unable to provide a specific citation.

The implications for this are huge. Based on this, it seems people would prefer the image of control rather than the image of no control. Is this true in other parts of life? It matches with my experiences; I will leave the reader to assess whether this matches with their own.

Now here is the catch; if this is what people prefer and we are dealing with an disagreeable system, is the solution to engineer the environment so that the image of choice is present? I will argue in brief, that the answer is “no” and explain why.

The reason is that this is a form of dishonesty. In this case the dishonesty removes from the relevant person the opportunity to make a more informed selection, it nullifies consent. I argue that such a removal is unethical. I also, note that this does not imply choice is real, but it does imply that the way people experience things are important, and that these experiences are partially used to define ethics.

In present mental health, informed consent is required of caregivers of minors or of adults who are institutionalized. They have no consent to give in this case. The ethics of this merit their own discussion. In this case, they are relevant because, it makes these persons most vulnerable to abuse from this technique.

A Review of the the Use of California Department of Developmental Service’s Autism Data

(click to enlarge graph)



Abstract

The California Department of Developmental Services (DDS) quarterly reports have been used by some to calculate incidence of autism. The California DDS has explicitly stated that their data should never be used to calculate the incidence or prevalence for autism. In the current post the implications for this are discussed. This post will also include a more thorough analysis and discussion of the DDS autism data than has been offered in previous reviews on this site. The results indicate that the DDS and descriptive epidemiology appear quite different. The conclusion from this analysis is that the California DDS data which the DDS states should never be used for descriptive epidemiology purposes and which does not resemble existing epidemiology, should not be used as epidemiology.

Introduction

The California DDS system is a voluntary system that provides services to persons with disabilities within the State of California. It does not provide services when such are not requested, nor does it accumulate data on such persons (Department of Developmental Services, 2005).

For those who enter the system, the DDS does compile certain data. This includes things like age, Developmental service category, and certain co-morbidities (DDS, 2005). These data are useful when making reports to law makers or consumer advocates who wish to see for whom, resources are being allocated.

Among other categories, the DDS serves autistic children and adults, who are at least 3 years old. This system’s collection of data over the years, make it appealing to those who wish to calculate prevalence or incidence or autism. This suddenly becomes an issue when we consider the claims of a current autism epidemic (MSNBC, Autism: The Hidden Epidemic, March 3, 2005).

Within the peer reviewed literature the DDS data have also been used to argue for an autism epidemic (Balxill, Baskin, & Spitzer 2003). These data have also been used to help support an argument for a proposed decrease in new intakes of children receiving services for autism (Rollens, 2006). The data have likewise been used to support the idea of diagnostic substitution as the source of the apparent increase of autism (Croen, Grether, Hoogstrate, & Selvin, 2002).

The changes in the DDS data have been argued as proof of an autism epidemic now in recession due to the removal of thimerosal (Rollens, 2006). This is unfortunate as epidemiology can not establish causation (Friis & Sellers; Last, 1995; Taubes, 1995; and Winkelstein, 1995). It seems prudent then to show some restraint when discussing what epidemiological data indicate.

The fact that the DDS has argued against this use of their data for epidemiological purposes, has been stated as a valid reason to discard such analyses http://interverbal.blogspot.com/2006/01/review-of-california-department-of.html. As these data continue to be used as formal proof, this post will include an analysis of the failure to account for threats to internal validity and the presence of problems with external validity. Problems with random and systematic error will also be discussed. The presence of these uncontrolled threats, as well as the differences from existing descriptive epidemiology, offer a strong agreement with the statements by the DDS, that these data should not be used to calculate incidence or prevalence.

Methods

The DDS autism data were taken from the quarterly reports (DDS, 2006) and from personal communications with the DDS. Additional data were taken US Census Bureau (USCB) projections and the 2000 census (USCB, 2006). Data were calculated to form a prevalence rate per 10,000.

The DDS data served as the denominator and census data as the numerator. (10,000) was then divided by the outcome to give the rate per (10,000). As there was numerous data entry and calculations used this process an additional student checked for accuracy of data entry and calculations on all relevant points. There were no disagreements, inter-observer agreement equaled (100%).

The analysis was conducted by visual inspection of the data path in line graph format as generated from MS Excel. The data path was assessed for relevant patterns such a cyclical trend, epidemic threshold, and case clustering (Friis & Sellers, 2004). Mean and standard deviation data was calculated as an additional descriptive tool for the net, quarterly report data changes.

Analysis

Figure 1. shows the prevalence rate of 3-5 year olds as calculated from the DDS data and the USCB. The increase is steady with the last two data points having the same rate, even though the raw numbers continue to show an increase between those points. This is not indicative of slowing of the increase as other points in the data path are also the same or very nearly the same.

The 3-5 cohort is the youngest group via the DDS that we have adequate data on. To serve as a counter balance I have presented Figure 2. which follows the oldest cohort of ages 62-99. An upward trend is also noted for this group, but at a much lower level. The cause of this in unknown, but this could be accounted for by diagnostic substitution for persons who were already in the DDS.

Diagnostic substitution from mental retardation to autism may not be wholly sufficient to explain the change (Blaxill et al., 2003). This author is also analyzing those data, and the analysis seems to support Blaxill et al. (2003) in that not all the change between categories can be accounted for by shifting persons receiving services for mental retardation to autism. However, shifting from other categories may affect this as well, in particular for school aged children (Eagle, 2003).

The data for ages 6-9 are sometimes discussed as being more accurate than the 3-5 data. This is logical as some diagnoses are not done until the child is older. Blaxill et al. (2003) use a median (middle number in a data set) age of diagnosis based on Kaye, del Melero-Montes, & Jick (2001) of (4.6). However, the median may not be most appropriate method for estimating how many diagnoses will be made later, as it is merely the middle number and does offer evidence as to what age, most of diagnoses are made (Jick, Beach, & Kaye, 2006) put the contemporary mean of diagnosis for Autistic Disorder at (3.1) years of age.

The 6-9 cohort numbers, have been typically higher than the 3-5 cohort, but that is not necessarily evidence for their accuracy. This could very well reflect a period of greater leniency by the DDS for whom they offer services. A review of Figure 3. supports this.

It is reasonable to predict that the level of the ages 10-13 cohort would be higher than the ages 6-9 cohort assuming that the change from ages 3-5 to 6-9 cohorts is due to delay in diagnosis. This does not seem to be the case. In fact there is typically a slight loss every three years from the 6-9 cohort to the 10-13 cohort. This attrition is unaccounted for. This may be tempting for some to assume that these children were “cured” of their autism by chelation or some other treatment. However, it would be an error to assume such an answer in the absence of better evidence. This may reflect movement out of California, it may reflect shifting to another category, it may reflect maturation of the individual or their regression towards the norm on testing and thus no longer being eligible for services under the more restrictive DDS requirements, it may simply reflect the end of a possible grace period. The DDS keeps attrition data, but does not have it organized into a database. The cost is $80.00 per hour to generate this database. To fund this is well beyond the means of this author.

In addition the prevalence rate calculated for the December quarter in 2005 equals (37) per (10,000). This is at least (10) higher, than the two epidemiology studies which came out in 2005. This could be accounted for by certain leniency from the DDS in terms of who they give services to despite the greater restrictions placed in 2003. This is an update from Croen et al. (2002) who cited the DDS prevalence rate as lower than the epidemiology.

Figure 4. shows the net changes across quarters from 1992 onwards. As per the last analysis http://interverbal.blogspot.com/2006/01/review-of-california-department-of.html
the pattern is primarily one of instability. A level change apparent in the years post 2000 still have significant variability. In fact if I was reviewing these data for a causal relation in a reversal design (baseline, introduction of independent variable, baseline) I would not conclude that the independent variable caused the change. This is doubly true, since, it is not clear where the baselines are relevant to the independent variable.

I have also included Figure 5. which is the yearly net change. This still shows instability, and includes the same problems that are discussed above. However, it also smoothed some of the instability out. This is unfortunate as instability is part of the picture of the DDS data. We must deal with this if we are to tell the whole story.


The mean of the analysis of change across quarters was (96) the standard deviation was (56). The square root of (96) = (9.7). In this case the standard deviation equals over half of mean; this indicates the high variability of the data. Ideally, the standard deviation should be near the square root of the mean.

Some of the changes may have explanations. In 1994 the DSM-IV altered the criteria for types and criteria of autism. This may account for the some of the spikes in the net change. Change of criteria is seen as a possible reason for increase in other areas of epidemiology. The changes between the World Health Organization, International Classification of Diseases (ICD) has been given as a possible reason for the increase of certain medical conditions (Friis & Sellers, 2004).

In July of 2003 a more restrictive eligibility criteria was put into place for the DDS services (Rollens, 2003). Diagnosticians may have rushed to get kids into the system ahead of the change, this may partially account for the spike in July of 2003. This is a special source of variability as is the change in the diagnostic criteria in the DSM.

A more stable source of variability comes is seen in that very few of the relative spikes happen in June or September. As has been suggested by others, this may have more to do with summer vacations than anything else. Also, the relative spikes in the data occur most frequently in March. This is not easily explained, but may simply be the time of year when the diagnosticians buckle down and do a great deal work. Other factors could influence this such as preparations for Individual Education Plans (IEP), the season for which often starts in earnest just after the Winter Holidays.

Via analysis of the data, the pattern does not seem to be truly cyclical however as is seen for monthly pneumonia and influenza mortality rates (Center for Disease Control and Prevention, 1997). Nor does these data allow us to present an epidemic threshold as we are unsure what the normal rate might be.

When we speak of autism incidence, we should speak of the youngest populations. We know that the mean age of diagnosis in the US is (3.1) years of age for Autistic Disorder (Jick et al., 2006). However, to qualify for Autistic Disorder, a child must meet criteria no later than age 3. There is no doubt that some children are diagnosed after this point. This does two things: It makes comparing data over the years difficult and it means that a 6 year old who is diagnosed with autism also met criteria for such in the past. This obscures attempts to determine case clustering.

Discussion

Threats to Validity

Since lay persons insist that these data are proof positive of certain causes for autism via “cum hoc, ergo prompter hoc” reasoning, it seems fair to subject analyses based on the DDS data for threats to internal and external validity.

There are seven threats to internal validity. The lay reports I have seen fail, to control for five of them. They do control for the other two, but those simply fail to apply in this case.

History is the first threat. It is the concept that external cultural or personal events may affect the outcome. The events of 9/11 might affect a study on depression, for example. In our case, this could be things like the study by Wakefield et al. (1998), which is shown to correlate to an autism rise in 1998 (Jick, Beach, & Kaye, 1998). This is not controlled for in the lay analyses.

Experimental Mortality, or attrition refers to the loss of a participant for various reasons. This is uncontrolled for and is evidenced in part by the discrepancy between the 6-9 cohort and the 10-13 cohort. As mentioned earlier this could be accounted for to an extent, but it has not been controlled for thus far.

Regression towards the mean, refers to the idea that over time statistical outliers tend to shift towards the mean. A child who meets criteria for the autism service category , may not do so at a later date. It is good to note that such does not imply that a child is no longer autistic, but that they simply fail to meet criteria for the DDS services.

Instrumentation, refers to the means used to measure an individual. Ideally, your instrumentation never changes over the course of a study. In our case the instrumentation did change at least twice for the data in graphs of this post. Once, in 1994, and once in 2003. This threat is uncontrolled is particularly likely to be a confound in this case.

Selection bias, refers to the including individuals in a study who bias the study in a certain direction when such do not represent the actual population. For example if I were to find some Amish persons with a low rate of autism and claim them as a representative of persons who are not vaccinated, I would be failing to control and also being guilty of using the fallacy of the Texas Sharpshooter, named after a sharpshooter who shot a cluster in a target and then drew the target around the cluster. In this case, using a voluntary service system can not possibly control for this factor, so this remains uncontrolled in the DDS analyses.

Random and Systematic error

Random and systematic errors are sources of problems that occur in epidemiology.
These are of concern in epidemiology, because they may cause us to identify or fail to identify a given pattern (Friis & Sellers, 2004). The analyses of the DDS data have failed to control for all six sources of error.

Random error involves fluctuation around a true value of a parameter. The first kind of error is Poor Precision. In this case we have a danger of the diagnosticians in the DDS assessing in different manners. This is present as there is no exact protocol aside from DSM-IV criteria and the additional more restrictive criteria implemented in 2003 for them to make eligibility judgments with.

Sampling Error is the second variety. In this case the sample is not representative of the larger population with that specific characteristic. We know that the DDS sample is not representative of the larger Autistic Disorder population in California because, of their more restrictive service eligibility criteria and due to their voluntary nature. There is the counter-intuitive problem that the prevalence via the DDS is higher than the mean prevalence post 1999.

The third kind of random error is Variability in Measurement. In this case the way we measure changes over time. This has been a concern between the ICD manuals and specific medical problems, as well the DSM manuals and autism. This is again, true as we altered the criteria to be less stringent from DSM-III to DSM-IV (Gernsbacher et al. 2005) and then tightened the criteria for legibility in the DDS system in 2003. Simply more awareness may affect the incidence of autism as evidenced by Jick et al. (2006) who found a jump in autism following the controversial Wakefield et al (1998) article and the public UK public concern over the possibility that the MMR vaccine caused autism.

Systematic errors are those problems that include a trend in the collection, analysis, interpretation, publication, or review that departs from the truth. There are three kinds; selection bias, information bias, and confounding (Friis & Sellers, 2004)

In Selection Bias, we find that the research favored a subset of a population of interest and excluded other examples in the research. The most well known example of this is the Texas Sharpshooter fallacy, which was described earlier in this post. This is present in the DDS data due to the more restrictive criteria put applied in 2003.

In Information Bias, we find that some other factors influences the accuracy of reporting. This may apply in July 2003, when the DDS autism numbers change, spiked high. This may be due to a rush by DDS diagnosticians to get as many people into the system before the more restrictive criteria were implemented. This also may account for why the DDS prevalence is higher than expected, the diagnosticians may exaggerate the severity of the developmental level of the child in order to receive services. This sub type of Information Bias is called prevarication (Friis & Sellers, 2004). This is logical as the DDS are gatekeepers for services in California. However, this is the opposite of what has been anecdotally reported by some parents of autistic children.

In the third type called Confounding, we are unable to obtain accurate results due to some other variable’s interference. Many of these are included in the five threats to internal validity that I named earlier. A Famous example is called “Simpson’s Paradox” where we receive significantly different rates between two instances of research based upon the number of persons in our samples (Rothman, 1986). This may account in part for the discrepancy between the high variability seen in recent years in the incidence calculated via the DDS data and what is seen for incidence via the epidemiology which has been stable in the recent years (Jick et al., 2006).

Conclusion

Future analyzers of the DDS data should be aware that these data are not recommended for this purpose according to the DDS. The cohort of primary interest should be the 3-5 age group. The data do not resemble the existing epidemiology and therefore do not seem externally valid. They do not evidence a pattern that clearly discernable and are quite variable between quarters. They can not support or refute an epidemic threshold level. They can not give an accurate case cluster analysis. They fail to account for five out of seven significant sources of internal validity. Finally, due to their observational nature, they not establish causation.

To conclude, Rollens (2006) poses a question that was answered in a previous post on this blog. Now I would like to take the chance to reciprocate and ask a question: What is the ethical standing of using the California DDS data when the DDS has stated that they should not be used for this purpose and when they do not resemble other existing epidemiology?

Notes: I want to thank the US Census Bureau and the California DDS for very generously fielding my questions.

References

American Psychiatric Association. (1980). Diagnostic and Statistical Manual ofMental Disorders, Third Edition. Washington, DC: American Psychiatric Association; 1980.

American Psychiatric Association. (1987). Diagnostic and Statistical
Manual ofMental Disorders, Third Edition, Revised. Washington, DC: American Psychiatric Association; 1980.

American Psychiatric Association. (1994). Diagnostic and Statistical
Manual ofMental Disorders, Fourth Edition. Washington, DC:
American Psychiatric Association; 1994.

American Psychiatric Association. (2000). Diagnostic and Statistical Manual ofMental Disorders, Fourth Edition, Text Revision. Washington, DC:
American Psychiatric Association; 1994.

Autism: The Hidden Epidemic. MSNBC. March 3, 2005.http://www.msnbc.msn.com/id/6844737/.Accessed September 16, 2005.

Angel, M. (1990). The interpretation of epidemiological studies. Editorial, New England Journal of Medicine. 332, 823-825.

Baird, G., Charman, T., Baron-Cohen, S., et al. (2000). A screening instrument for autism at 18 months of age: a 6 year follow-up study. Journal of American
Academy of Child and Adolescent Psychiatry 39, 694-702.

Bertrand, J., Mars, A., Boyle, C., Bove, F., Yeargin-Allsop, M., & Decoufle, P.
(2001). Pediatrics, 108, 1155-161.

C. H. (2006). Department of Disability Services. Personal Communication.

Chakrabarti, S., & Fombonne, E. (2001). Pervasive developmental disorders in preschool children. Journal of the American Medical Association, 285,
3093-3099.

Chakrabarti, S., Fombonne, E., (2005). Pervasive developmental disorders in preschool children: confirmation of high prevalence. American Journal of
Psychiatry, 162(6), 1133-1141.

Centers for Disease Control and Prevention. (1997). Weekly Influenza and Pnuemonia Mortality. MMWR.
46, 327.

Department of Developmental Services (2006). Quarterly Client Characteristics Reports.http://www.dds.ca.gov/FactsStats/quarterly.cfmAccessed Friday January 13, 2005.

Department of Developmental Services (2005). Data Interpretation Considerations and Limitations.http://www.dds.ca.gov/FactsStats/pdf/CDER_QtrlyReport_
Consideration_ Limitations.pdfAccessed Friday January 13, 2005.

Fombonne, E. (2001). What is the prevalence of Asperger syndrome? Journal of Autism & Developmental Disorders, 31: 3, 363-364.

Fombonne, E. (2001). Is there an epidemic of autism? Pediatrics.Vol 107 (2), 411-412.

Fombonne, E., Simmons, H., Ford, T., Meltzer, H., Goodman, R. (2001). Prevalence of pervasive developmental disorders in the British national survey of child mental health. Journal of the American Academy of Child & Adolescent Psychiatry 40, 820-827.

Fombonne, E. (2002). Prevalence of childhood disintegrative disorder (CDD). Autism 6, 2, 147-155.

Fombonne, E. (2003). Epidemiological surveys of autism and other pervasive developmental disorders: an update. Journal of Autism and Developmental Disorders. 33, 365-382.

Friis, R. H., Seller, T. A. (2004). Epidemiology for public health practice, 3rd ed. Sundbury, MA: Jones and Bartlett Publishers.

Gernsbacher MA, Dawson M, & Goldsmith HH. (2005).Three reasons not to believe in an autism epidemic. Current directions in psychological science, 14 (2), 55-58.

Honda Shimizu, Y., Rutter, M. (2005). No effect of MMR withdrawal on the incidence of autism: a total population study. Journal of Child Psychology and Psychiatry, vol 46 (6), 572-579.

Individuals with Disabilities Education Act. (1990). Public Law 101-476, U.S.C.

Jick H, Beach KJ, Kaye JA. Incidence of autism over time.Epidemiology. (2006). Epidemiology, 17(1), 120-121.

Kaye, J. A., del Melero-Montes, M., & Jick, H. (2001). Mumps, measles, and rubella vaccine and the incidence of autism recorded by general practitioners: A time trend analysis. BritishMedical Journal, 322, 460–463.

Kielinen, M., Linna, S. L., & Moilanen, I. (2000). Autism in Northern Finland. European Child and Adolescent Psychiatry, 9, 162-167.Laidler, J. (2005). US Department of Education data on "autism" are not reliable for tracking autism prevalence. Pediatrics, 116 (1), 120-124.

Last, J. M. (1995). A dictionary of epidemiology, 3rd ed. New York, NY: Oxford University Press.

Magnusson, P., & Saemundsen, E. (2001). Prevalence of autism in Iceland. Journal of Autism & Developmental Disorders 31: 153-163.

Mandall, D. S., Novak, M. M., Zubritsky, C. D. (2005). Factors associated with age of diagnosis among children with autism spectrum disorders. Pediatrics,Vol 116 (6), 1480-6.

Powell, J., Edwards, A., Edwards, M., et al. (2000). DevelopmentalMedicine and Child Neurology, 42, 624–628.Rollens, R. (2006) California Reports: New Autism Cases at 4 Year Low. Schafer Autism Report, 10 (7). Thursday, January 12, 2006, http://www.sarnet.org/Accessed Accessed Friday January 13, 2005.

Rollens, R. (2004). California Autism Intakes Down Slightly -- Factors Discussed. Schafer Autism Report, 8, 165. Wednesday, October 20, 2004, http://www.sarnet.org/Accessed Accessed Friday February 4, 2005.

Taubes, G. (1995). Epidemiology faces its limits. Science, 269, 164-9.

Taylor B, Miller E, Farrington CP, et al. Autism and measles, mumps, and rubella vaccine; no epidemiological evidence for a causal association. Lancet, 353, 2026-2029.

United States Census Bureau. (2006). Personal Communication.

Wakefield, A., Murch, S., Anthony, A., Linnell, J., Casson, D., Malik, M., Berelowitz, M., Dhillon, A., Thomson, M., Harvey, P., Valentine, A., Davies, S., & Walker-Smith, J. (1998). Heallymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet, 351, 637-641.

Winklestein, W. (1995). Invited commentary on “Judgement and Causal Inference: criteria in epidemiological studies”. American Journal of Epidemiology, 141, 699-700.

World Health Organization. (1992). International Classification of Diseases, 10th Revision (ICD-10). Geneva, Switzerland: World Health Organization; 1992.

Yeargin-Allsopp, M., Rice, C., Karapurka, T., Doernberg, N., Boyle, C., Murphy, C. (2003). Prevalence of autism in a US metropolitan area. Journal of the American Medical Association, 289, 49-89.