NEJM Study
A new study has just come out in the New England Journal of Medicine (NEJM). It was a correlational study, looking at 1047 children ages 7-10 that had complete records of immunization. The study assessed for 42 types of neuropsychological outcomes. This did not include autism which will be assessed in a different study. This study authors concluded that there is no association between neuropsychological disorders and thimerosal in vaccines.
Even before this study was released yesterday, there was publicized dissent from advocates from the idea of a mercury etiology of neuropsychological harm. One of them came from a well known advocate, Sallie Bernard, who was invited to be a collaborator in this study. It seems that she was involved in the planning of this study, but the lead author indicated that she withdrew her support after the results began to be circulated.
That is not how science works. If you have a problem with a study design then you dissent before you begin collecting data.
Even before this study was released yesterday, there was publicized dissent from advocates from the idea of a mercury etiology of neuropsychological harm. One of them came from a well known advocate, Sallie Bernard, who was invited to be a collaborator in this study. It seems that she was involved in the planning of this study, but the lead author indicated that she withdrew her support after the results began to be circulated.
That is not how science works. If you have a problem with a study design then you dissent before you begin collecting data.
posted by Interverbal at 7:52 AM
22 Comments:
Amen, brother.
Let this be a lesson to CDC. They have to stop thinking that "further study" will pacify their critics, who are the Lucy to CDC's Charlie Brown. It's impossible to prove a negative, so every time there's a news story on thimerosal, it's just another opportunity for people who hate vaccines to be quoted saying, "See? You can't rule out the possibility that vaccines cause autism!"
When did Sallie Bernard register her dissent with the results of the study? I did not see any documentation on the time frame in which she did so.
Some manuscripts on problems on epidemiology
Causality in epidemiology
Overview of research design in epidemiology
"Causality Assessment
Several authors have discussed causality assessment in epidemiology (Hill 1965; Buck 1975; Ahlbom 1984; Maclure 1985; Miettinen 1985; Rothman 1986; Weed 1986; Schlesselman 1987; Maclure 1988; Weed 1988; Karhausen 1995). One of the main points of discussion is whether epidemiology uses or should use the same criteria for the ascertainment of cause-effect relationships as used in other sciences.
Causes should not be confused with mechanisms. For example, asbestos is a cause of mesothelioma, whereas oncogene mutation is a putative mechanism. On the basis of the existing evidence, it is likely that (a) different external exposures can act at the same mechanistic stages and (b) usually there is not a fixed and necessary sequence of mechanistic steps in the development of disease. For example, carcinogenesis is interpreted as a sequence of stochastic (probabilistic) transitions, from gene mutation to cell proliferation to gene mutation again, that eventually leads to cancer. In addition, carcinogenesis is a multifactorial process-that is, different external exposures are able to affect it and none of them is necessary in a susceptible person. This model is likely to apply to several diseases in addition to cancer.
Such a multifactorial and probabilistic nature of most exposure-disease relationships implies that disentangling the role played by one specific exposure is problematic. In addition, the observational nature of epidemiology prevents us from conducting experiments that could clarify aetiologic relationships through a wilful alteration of the course of the events. The observation of a statistical association between exposure and disease does not mean that the association is causal. For example, most epidemiologists have interpreted the association between exposure to diesel exhaust and bladder cancer as a causal one, but others have claimed that workers exposed to diesel exhaust (mostly truck and taxi drivers) are more often cigarette smokers than are non-exposed individuals. The observed association, according to this claim, thus would be "confounded" by a well-known risk factor like smoking.
Given the probabilistic-multifactorial nature of most exposure-disease associations, epidemiologists have developed guidelines to recognize relationships that are likely to be causal. These are the guidelines originally proposed by Sir Bradford Hill for chronic diseases (1965):
• strength of the association
• dose-response effect
• lack of temporal ambiguity
• consistency of the findings
• biological plausibility
• coherence of the evidence
• specificity of the association.
These criteria should be considered only as general guidelines or practical tools; in fact, scientific causal assessment is an iterative process centred around measurement of the exposure-disease relationship. However, Hill's criteria often are used as a concise and practical description of causal inference procedures in epidemiology.
Let us consider the example of the relationship between exposure to vinyl chloride and liver angiosarcoma, applying Hill's criteria.
The usual expression of the results of an epidemiological study is a measure of the degree of association between exposure and disease (Hill's first criterion). A relative risk (RR) that is greater than unity means that there is a statistical association between exposure and disease. For instance, if the incidence rate of liver angiosarcoma is usually 1 in 10 million, but it is 1 in 100,000 among those exposed to vinyl chloride, then the RR is 100 (that is, people who work with vinyl chloride have a 100 times increased risk of developing angiosarcoma compared to people who do not work with vinyl chloride).”
Now, the problem is to compare vaccinated vs unvaccinated therefore there are no control groups here (my comment)
“It is more likely that an association is causal when the risk increases with increasing levels of exposure (dose-response effect, Hill's second criterion) and when the temporal relationship between exposure and disease makes sense on biological grounds (the exposure precedes the effect and the length of this "induction" period is compatible with a biological model of disease; Hill's third criterion). In addition, an association is more likely to be causal when similar results are obtained by others who have been able to replicate the findings in different circumstances ("consistency", Hill's fourth criterion).”
Here, it all depends on the eye of the beholder
-Dose response= higher number of vaccines; higher number of autistics BUT we have problems with the number of autistics in time and with the full schedule ( form, confounder, other) with the number of vaccines-Very difficult to address
-Temporal relationship= well this is to open a can of worms_ Again , with a child what is the “minimum-maximum-temporal relationship??Anecdotical evidence in thousands- even when correlation is not causation
-Consistency-around the world the anecdotic evidence is unfortunately similar
“A scientific analysis of the results requires an evaluation of biological plausibility (Hill's fifth criterion). This can be achieved in different ways. For example, a simple criterion is to evaluate whether the alleged "cause" is able to reach the target organ (e.g., inhaled substances that do not reach the lung cannot circulate in the body). Also, supporting evidence from animal studies is helpful: the observation of liver angiosarcomas in animals treated with vinyl chloride strongly reinforces the association observed in man.”
Again, now we have problems with the unknown…
“Internal coherence of the observations (for example, the RR is similarly increased in both genders) is an important scientific criterion (Hill's sixth criterion). Causality is more likely when the relationship is very specific-that is, involves rare causes and/or rare diseases, or a specific histologic type/subgroup of patients (Hill's seventh criterion).”
And the seventh criterion seems very likely
"Enumerative induction" (the simple enumeration of instances of association between exposure and disease) is insufficient to describe completely the inductive steps in causal reasoning. Usually, the result of enumerative induction produces a complex and still confused observation because different causal chains or, more frequently, a genuine causal relationship and other irrelevant exposures, are entangled. Alternative explanations have to be eliminated through "eliminative induction", showing that an association is likely to be causal because it is not "confounded" with others. A simple definition of an alternative explanation is "an extraneous factor whose effect is mixed with the effect of the exposure of interest, thus distorting the risk estimate for the exposure of interest" (Rothman 1986)."
Where is the Bradfor analysis done for these kinds of studies on thimerosal?
Quotation
"Here, an original argument of Hill [5] becomes of particularly important: the required amount of evidence for a causal effect should depend on the possible consequences of interventions derived from causal conclusions. If a causal conclusion needed an action that brought about more harm if wrongly taken than benefit if rightly taken, a correspondingly high amount of evidence would be required. If the relationship between benefit and harm were converse, less evidence would be necessary. The major tool to assess the applicability of these considerations is multiple bias modelling. Multiple bias models should be much more frequently used. Moreover, the decision as to whether or not to apply one of these considerations is always implicitly based on one or several multiple bias models. For instance, demanding an association of at least a certain magnitude is logically equivalent to the "true bias model" being part of the set of multiple bias models in which priors on bias parameters would require at least this magnitude of association to be observed.
One may ask the counterfactual question of how epidemiology and medical research would have developped if Hill had been more explicit in recommending when to apply each of his considerations. I am far from claiming to be able to answer this question, but I consider my speculation being worth mentioning. In their paper entitled "The missed lessons of Sir Austin Bradford Hill" [6] Phillips and Goodman reviewed malpractices denounced by Hill that were still being made later in practice: over-emphasis of statistical tests, systematic error being under-estimated and cost/benefit trade-offs being disregarded in intervention decisions. Hill's considerations were misused as "causal criteria", and they were taught more often than more sound causal conceptions [6]. There is no reason to believe that more explicit recommendations on when to apply his considerations would have been better heeded; the cautionary notes that Hill actually made were largely ignored.
My own experience is that scientific recommendations are widely followed if they provide easy guidance; recommendations that call for complex action are frequently ignored. My guess is that this is due to many researchers' desire for simple and globally applicable answers. This desire leads to misinterpretation of scientific texts and to taking individual statements out of their context. More pessimistically, the question of which guidance is followed depends on which guidelines are in line with the desired answer. Therefore, it seems likely that, even if Hill's paper had not been published, scientists' desire for simple answers would have caused another paper to be written or to be misinterpreted in the same way as happened with Hill's [5]
Article"
"The usefulness of the counterfactual approximation of Hill's considerations is that their heuristic value can be assessed by answering counterfactual questions. I have argued that the application of seven of the nine considerations (consistency, specificity, temporality, biological gradient, plausibility, coherence and analogy) involves comprehensive causal theories. Complex causal systems comprise many counterfactuals and assumptions about biases. If complexity becomes very large, the uncertainty regarding whether or not to apply a given consideration can be expected to approach a decision made by coin toss. Thus, with increasing complexity, the heuristic value of Hill's considerations diminishes."
Don´t you think that ASD as diagnosis is extremely complex??
From Link-1
Link-2
See page 34:I think that not only this, ALL research thimerosal/vaccines suffers from the same mistakes….
Link-3
lLink-4
Link-5
The scientific model of causality
Things are much more complex than these epi studies can address with the design they have,the tools they use, the sample size they manage and the questions they do.
Idisagree, such as i disagree with the simplistic analysis of Geier et als and others.
My child deserves better.
Hi MJ,
The lead study author has explained, that Sallie Bernard dissented after the write-up began to be circulated.
Hi Maria,
Hill's criteria is not new to me, nor are several of the papers you reference.
It seems though, that the issues you raise in your post are more fundemental than any specific complex disorder. In fact the points you raise affects all of epidemiology.
However, I do not totally agree with the way you attempted to fit the current issue into Hill's criteria.
The only one I am willing to grant is biologic plausibility, however there are mutliple plausible arguments in the biologic field. this being the case, what now?
To answer your question; yes, I do think that ASD as a diagnosis is extremely complex.
I also think that while a correlational design can not prove causation, it is the correct first step, in this direction.
I think in this specific case, even if the thimerosal laden vaccines were only a part of a complex causality for autism, we should have seen some sort of general step effect as the dosage increases. This is lacking.
This study was well designed and well executed. If a study is lacking like the Danish studies, then I have no problem stating such. That is not the case here. I feel comfortable backing the study as described by the lead authors.
Via the Ockham's razor, I don't think that the complexity of autism, is an acceptable reason to disount this study.
I also don't believe that this is a death knell of the vaccine theory, but it is a blow to it.
In the realm of the non-medical doctors, I don't really know what the impact of this study will be. That is because a former model on Oprah, can convince more people with an argumentum ad populum, than any amount of hard science.
Hi Jonathan
You said
However, I do not totally agree with the way you attempted to fit the current issue into Hill's criteria. The only one I am willing to grant is biologic plausibility, however there are mutliple plausible arguments in the biologic field. this being the case, what now? To answer your question; yes, I do think that ASD as a diagnosis is extremely complex. I also think that while a correlational design can not prove causation, it is the correct first step, in this direction
The problem here, even if I do think that my disagreement with you are in grays that is in subtle fields, is the design. This study is not presenting focus on a subgroup of autistic children, trying to know about the design of clinical tests to know about the potential effects of thimerosal on them-because there are no animal models of bioaccumulatin of toxic elements for example. The clinical data are absent. For example
-the children number is 1047. Why do you think with this number a potential number of 7 ASD children are going to be detected if 1 in 150 is the accepted number today?
a) Is the sample size enough to detect them?
b)being a distribution of effects, can these effects being distributed and therefore you should know before the kind of distribution to test adequately adequately done questions in terms of negative effects on the health more than causation- at first?
c)why CAUSATION? Why not immune health status in general, based on immune testing-because there is no testing to know about thimerosal management trouble in autistics or not autistics therefore Why the conclusion?
d)why vaccines in general are not focused in a subpopulation of childre n with developmental problems- not the general population? I this sense, the multifactorial problem is relevant- sorry the links did not work before
Link-2
See page 34:I think that not only this, ALL research thimerosal suffers from the same mistakes….
Link-3
Link-4
You said
I think in this specific case, even if the thimerosal laden vaccines were only a part of a complex causality for autism, we should have seen some sort of general step effect as the dosage increases. This is lacking
Yes, because of potentially many reasons, confounders- even not discussed. With 1047 children totally mixed in genetics/probably epigenetics , immunological, nutritional gastroenterological status, why the effect is going to be seen if the populations are totally mixed and are all vaccinated and the effects-if any detectable- are distributed? How do you separate exposed from non-exposed?
This study was well designed and well executed. If a study is lacking like the Danish studies, then I have no problem stating such. That is not the case here. I feel comfortable backing the study as described by the lead authors. Via the Ockham's razor, I don't think that the complexity of autism, is an acceptable reason to disount this study
I disagree in this point, even when I do not discount the study. In general population, with no specific genetic predisposition/vulnerability and with a distribution of polymorphisms, why with 1047 individuals are you going to detect them? Essential complexity and individuality of genetic/epigenetic impact makes Ockam´s razor inaplicable.
Essentially complex systems
Physiological complexity
Modeling
In the realm of the non-medical doctors, I don't really know what the impact of this study will be. That is because a former model on Oprah, can convince more people with an argumentum ad populum, than any amount of hard science
But this is not hard science for me-because of the question, analysis and conclusions-, such as in Oprah not hard science was presented...
This might be helpful MJ:
http://www.webmd.com/brain/news/20070926/thimerosal-no-smoking-gun
The CDC asked more than a dozen outside consultants to comment on the study prior to publication of the final manuscript. One was Sallie Bernard, the parent of a child with autism and executive director of Safe Minds. Safe Minds is one of the groups working to convince parents that medical use of mercury causes autism and other child health problems.
Interverbal,
I'm disappointed that you didn't do your homework and jumped on the bandwagon of rants here because I normally find your articles very objective.
If you read the study carefully, you'll find that she was listed as a dissenting member in the Thanks section. That would indicate that she likely expressed her concerns with the protocol and/or results well in advance of publication and very possibly at the time of protocol design as you suggested.
I think you don't have enough evidence to make the comments you did.
Hi Maria,
I think we are down to some basic epistemological issues. I am not sure we are going to come to many agreements, but the discussion is interesting and the issues are good to talk about. In this spirit I will offer a reply:
This study makes the assumption that if there is an effect of thimerosal on psychological functioning that it should be detectable in a using statistical significance testing in a set of 1047 children lumped together by their amount of exposure to thimerosal. If thimerosal only causes a statistically insignificant amount of children to have ill effects, then it will not be detected by this study.
However, such a statistical test makes sense if one views the broader theory of thimerosal etiology of harm. Many, perhaps even most of the advocates of the this theory, say that the thimerosal has caused an explosion of problems easily detectable by reviewing prevalence data of these issues. If this is true, then we should predict that we will find a statistically significant association of harm.
I could accept that your criticism has merit here Maria, but to do so, I must simultaneously accept that many of your fellow advocates were wrong to make this claim. Some consider this claim to be essential to their arguments for their theory. It seems there is an issue of coherence here, which was also in Hill’s criteria.
You write “Essential complexity and individuality of genetic/epigenetic impact makes Ockam´s razor inaplicable.”
You seem to be arguing that Ockham’s razor doesn’t work here because the issue could be complex. That is not how Ockham’s razor works.
entia non sunt multiplicanda praeter necessitatem
Entities should not be multiplied beyond what is necessary to explain them.
Given that the majority of the advocates of thimerosal theory claim a notable increase in prevalence that should be detectable in prevalence studies, it doesn’t follow that this study doesn’t work.
Again, you could argue against this, but you would have to brave out in lonely territory. You would also have to deal with the prevalence issues at some point.
Hi Schwartz,
Thank you for your readership and for statement about this blog’s usual objectivity. I also wish to thank you for your attempt at fact correction. I do make factual mistakes and I welcome and value people who are willing to put me straight.
That said, this is not the case here. A private communication with the lead author was shared with me. It was made it perfectly clear that dissent did not occur until after an initial write up began to be disseminated.
As to jumping on the bandwagon of rants. I assure you that I was calm, cool, and collected when I wrote this post and stand by it now as well.
Thank you for your time.
Thanks for the clarifiaction. It would be a lot clearer if a specific communication at least alluded to in the article. Otherwise, it just looks like personal opinion.
I'm not privy to how studies and reviews are done. However, it does appear that the primary objection appears to be with the scope of the concluding statements. I don't see how that objection would have been addressable before a final report was written. Additionally, how does interaction with a consultant typically occur? At the beginning and then at the end? In that case, they are also not privy to how the study is progressing (i.e. enrollment, exclusions etc).
One issue I had that I think limited the study scope, was the extensive list of exclusion criteria that they used to exclude a good number of children.
If Thimerosal is consistently correlated with any of those illnesses and more loosely correlated to the neurological problems studied, then the exclusions will have eliminated any signs of correlation.
I don't understand why they wouldn't have decided to include these children and account for the possible confounders in the statistics themselves. It could have revealed some interesting information.
I also think that any subtle neurological problems could easily have been addressed in the 7+ years between infancy and the study assessments. This could occur by natural healing (many children are speech delayed and catch up naturally over time) or through intervention. Obviously that wouldn't apply to all of the study assessments, but it certainly applies to the speech delay.
"A private communication with the lead author was shared with me. It was made it perfectly clear that dissent did not occur until after an initial write up began to be disseminated."
I would tend to view this as a problem. You are basing your opinion on privately shared information that is not generally available to the public.
Did Sallie Bernard have a chance to comment or refute that information? Did someone verify that that facts citied in support of the lead authors statements were correct and aren't based on his perception of events or because he has a specific agenda?
Hi Schwartz,
You are right, the issue is not clear in my post. I will amend this.
You write “Additionally, how does interaction with a consultant typically occur? At the beginning and then at the end? In that case, they are also not privy to how the study is progressing (i.e. enrollment, exclusions etc).”
Depends on the study. It seems in this case, that the Sallie Bernard was involved in planning the study design. She did not give dissent at this point.
You write “One issue I had that I think limited the study scope, was the extensive list of exclusion criteria that they used to exclude a good number of children.”
If there was one thing this study was not limited in, it was scope. There was an absurd amount of statistical tests 370 of so of them. Now, in such a comprehensive study, you do need to worry about specific threats becoming problems. The authors excluded things that might confound the results like low birth weight. What this means is that this study can’t comment on what influence thimerosal has on children with low birth weight. However, this fact has not influence on the studies validity for children who do not have such conditions.
You write “If Thimerosal is consistently correlated with any of those illnesses and more loosely correlated to the neurological problems studied, then the exclusions will have eliminated any signs of correlation.”
That’s true. That’s why the study can only be said to apply to children who lack those conditions.
You write “I don't understand why they wouldn't have decided to include these children and account for the possible confounders in the statistics themselves.”
I don’t disagree, but that was beyond the scope of the study. This study was focusing on performance on standardized tests in terms of exposure to thimerosal. A separate analysis could have been done on low birth weight and other excluded factors, yes, but also on dozens of other interesting points. No study could do everything; you have to draw the line somewhere.
You write “I also think that any subtle neurological problems could easily have been addressed in the 7+ years between infancy and the study assessments.”
Yes, but also some can develop. Transient tics develop in the age range this study was done. Also, tic based disorders first develop at around at age 10.
You write “This could occur by natural healing (many children are speech delayed and catch up naturally over time) or through intervention.”
Yes, or even more significant delays. This concept is called regression to the mean. Interesting that you should bring this up. Not to get into a tangent, but consider what this fact might do to the validity of research in early interventions or case studies of biomedical approaches.
Let me say a bit more about this study, and forgive me for the jargon if you are unfamiliar with it (just ask me to explain if it is not clear). This study has 42 factors and 370 statistical tests. In other research having just 5-6 factors and <20 statistical tests can make a statistician nervous, because the more independent variables you have the greater the odds of a type I error will occur in at least some of the tests.
The researchers knew this, they used a generous alpha of .05 (95% confidence interval). Given the number statistical tests, they could have and probably should have used a more stringent standard. Simply because as the number of tests rise, so does the odds of an incorrect finding. This is very literally a textbook problem.
I think we are down to some basic epistemological issues. I am not sure we are going to come to many agreements, but the discussion is interesting and the issues are good to talk about. In this spirit I will offer a reply:
Thank you very much for your time and kindness.
This study makes the assumption that if there is an effect of thimerosal on psychological functioning that it should be detectable in a using statistical significance testing in a set of 1047 children lumped together by their amount of exposure to thimerosal. If thimerosal only causes a statistically insignificant amount of children to have ill effects, then it will not be detected by this study.
OK, we are in agreement here,.
However, such a statistical test makes sense if one views the broader theory of thimerosal etiology of harm. Many, perhaps even most of the advocates of the this theory, say that the thimerosal has caused an explosion of problems easily detectable by reviewing prevalence data of these issues. If this is true, then we should predict that we will find a statistically significant association of harm
But that the theory advocates have on thimerosal can be probably wrong in terms of only causes does not preclude a contribution; you are analyzing the issue going backwards with the statements of this theory- that is not mine. The fact that the epi is not for sure saying that a true increase is or isn´t present does not preclude a collaboration of symptoms, in a wide notion considering toxicogenomics and toxicoproteomics and epigenetics- where there could be protective and agressive effects of environmental exposures in combination.
I could accept that your criticism has merit here Maria, but to do so, I must simultaneously accept that many of your fellow advocates were wrong to make this claim. Some consider this claim to be essential to their arguments for their theory. It seems there is an issue of coherence here, which was also in Hill’s criteria
First, I have not fellow advocates because I do not agree with I would say almost nobody in terms of advocacy. I have told you before; each family has its own style of advocacy and interpretation of ASD. I have my personal style and position. The issue of coherence, under a different analysis than yours and the advocates you mention, is not a problem under a different light.
You write “Essential complexity and individuality of genetic/epigenetic impact makes Ockam´s razor inaplicable.”
You seem to be arguing that Ockham’s razor doesn’t work here because the issue could be complex. That is not how Ockham’s razor works.
entia non sunt multiplicanda praeter necessitatem
Entities should not be multiplied beyond what is necessary to explain them
This is not what the manuscripts I included Are talking about. They are clear in this issue.
Given that the majority of the advocates of thimerosal theory claim a notable increase in prevalence that should be detectable in prevalence studies, it doesn’t follow that this study doesn’t work
This is a different problem.
Again, you could argue against this, but you would have to brave out in lonely territory. You would also have to deal with the prevalence issues at some point.
But the issue of prevalence is totally different to the individual susceptibility-genetic and the combination of impacts through epigenetics. In this sense, prevalence are important for the political issues but are not useful for the individual presentation of concomitant medical problems to an ASD diagnosis
MJ,
You write “Did Sallie Bernard have a chance to comment or refute that information? Did someone verify that that facts citied in support of the lead authors statements were correct and aren't based on his perception of events or because he has a specific agenda?”
I saw no reason and still see no reason to make this something that required confirmation from another party. She either gave dissent in the planning phase or did not. If the errors were too grevious should have withdrawn right then. That is what other researchers do when they have big problems with a nature of a study they are working on.
As to agenda, the researchers invited Sallie Bernard to be a part of this team. That doesn’t smack of agenda unless we are about to decline into some conspiracy theories.
On the other hand her organization SafeMinds is pretty clear in their write up of this study when they say “The conclusions should have referenced these and other limitations, as well as confirming thimerosal-associated impairments found in other studies, and as a result, called for further investigation.”
http://www.autismspeaks.org/science/science_news/safeminds_nejm_study.php
What I want to know is what studies confirming thimerosal-associated impairements is the author talking about. Honestly I can’t think of a single one that was plublished in mainstream peer reviewed journal. Whatever they are, Sageminds thinks these need to be confirmed.
Interverbal,
This is why you're the only skeptic site I can go to and expect to get a logical scientific explanation on the first try. Thanks for the response.
My only comment on the Sally topic:
"Depends on the study. It seems in this case, that the Sallie Bernard was involved in planning the study design. She did not give dissent at this point."
If the bulk of her issues derive from the wording/scope of conclusions, then the only time those would have been known was after it was written up at the study end. So her late dissent certainly is not unprecedented -- I even suppose the infamous Wakefield study would be an extreme example of this with subsequent retractions by some of the researchers. But those scientists were not villified for their retractions long after publication -- but then there is politics involved in both these cases I suppose.
On to the study though. I do have a couple of questions/comments:
"The authors excluded things that might confound the results like low birth weight. "
That is exactly the impression I got when I first read the study. Then I looked at the appendix, and the exclusions were much greater than the impression they gave in the discussion.
"That’s true. That’s why the study can only be said to apply to children who lack those conditions."
I think that given the large list of exceptions this does limit the scope of conclusions, and that the press release, written conclusions, (and even the discussion) were not at all clear on this matter -- in this particular respect I find support for that criticism of the study.
However, I can accept the logic of excluding things, and I'll take your word for it that they didn't have scope/budget to account for these things in the study. Unfortunately, it seems like a lost opportunity to glean more information for incremental effort. It's like ripping up the road to lay gas lines, and then fixing the sewer the year after -- which they do all the time where I live... sigh.
"The researchers knew this, they used a generous alpha of .05 (95% confidence interval). Given the number statistical tests, they could have and probably should have used a more stringent standard. Simply because as the number of tests rise, so does the odds of an incorrect finding. This is very literally a textbook problem."
What little memory I have of my statistical classes (focussed on processes for engineering) has pretty much faded. I'm going to have to beef up on the topic, because it appears to be critical in order to truly evaluate and discuss most of these studies. But I looked up the terms that you used, and let me see if I understand your comment.
I think the point you were making is: as a study increases the number of factors, the odds of finding a false positive is higher. Since this study had such a large number of factors, they should have used a higher confidence interval (instead of 19 of 20).
If what you're trying to say is that the small correlations (positive and negative) are just noise, I agree with that as well. The criticisms that attempt to point to the few small correlations are baseless IMO.
I have read numerous criticisms of the 5% statistical significance choice, which I assume is amplified on small sample sets such as these?
However, along those lines, one disappointing factor (out of the control of the researchers), is the number of cases with no Thimerosal exposure (16 if I remember correctly, and only 2 and 3 in the next cohort segments). Is that number not too small to glean any significant results along that range of exposure? IF that is the case, then the study is only effective at determining correlation between exposure differences in the mid-range exposure levels. This could later turn out to be misleading if the mercury in Thimerosal has measurable effects at very small levels.
I don't know how this compares to lead exposure, but my understanding is that even very low levels of lead exposure have been shown to have measurable neurological impact.
On the other hand, I don't know if lead also shows linear or measurable impact throughout the rest of the ranges of exposure. It would be interesting to know if there are step-like functions of exposure thresholds to heavy metal substances or if one should reasonably expect a smooth range of correlation throughout the spectrum of exposure.
"Yes, or even more significant delays. This concept is called regression to the mean. Interesting that you should bring this up. Not to get into a tangent, but consider what this fact might do to the validity of research in early interventions or case studies of biomedical approaches."
I absolutely agree. Even if the root cause of any of these conditions are not immediately identified, I highly suspect there are multiple ways to compensate for a variety of them, especially since it's unlikely that there is a single cause. That's why I also didn't like the huge time lag between exposure and measured effects.
Hi Maria,
You write “But that the theory advocates have on thimerosal can be probably wrong in terms of only causes does not preclude a contribution; you are analyzing the issue going backwards with the statements of this theory- that is not mine. The fact that the epi is not for sure saying that a true increase is or isn´t present does not preclude a collaboration of symptoms, in a wide notion considering toxicogenomics and toxicoproteomics and epigenetics- where there could be protective and agressive effects of environmental exposures in combination.”
Are you proposing that these other factors might exist, so that if a study only looks at one aspect, it will not control for the other factors that could lead to autism? If so, then you are right, this study can do little to answer your point.
But, my point it is that this is your personal position (though thoughtful, educated, and informed). It does not reflect the usual position given by advocates of the mercury theory. This study does address “their” point in an acceptable manner. It does not address the points you raise or your more fundamental criticisms of epidemiology.
You write “First, I have not fellow advocates because I do not agree with I would say almost nobody in terms of advocacy.”
Noted, and I will not mistake their argument for yours. But that may put you in a bind depending on the way you reference your material.
You write “I have told you before; each family has its own style of advocacy and interpretation of ASD. I have my personal style and position. The issue of coherence, under a different analysis than yours and the advocates you mention, is not a problem under a different light.”
The medical scientists try to answer the common usual question. To this end they will try to answer coherent, ideas that are testable. There is an inevitable bias in that they lean towards the questions that most people make, as opposed to the minority.
You write “This is not what the manuscripts I included Are talking about. They are clear in this issue.”
I don’t understand this point. Will you rephrase and/or cite it?
You write “In this sense, prevalence are important for the political issues but are not useful for the individual presentation of concomitant medical problems to an ASD diagnosis”
However now you are in lonely territory. You are welcome to it, but I predict that you are going to be consistently disappointed with future research in this field. Unless you can build up a lot of support for the ideas you propose.
To be honest, I would like to see what a study designed by you attempting to answer these points here would look like. I would like to see if I would agree that it is a well designed study for this issue.
Hi Schwartz,
“If the bulk of her issues derive from the wording/scope of conclusions, then the only time those would have been known was after it was written up at the study end. So her late dissent certainly is not unprecedented”
The conclusions were not her only criticisms. Once again, why now as opposed to before? Also, the detailed A-CHAMP criticism came out before the article was released. Their criticisms were heavily designed based. They either got their information from Sallie or they got an early copy that was being circulated and broke the NEJM media embargo (can’t blog or make press statement before the articles are formally released online). Or both.
“However, I can accept the logic of excluding things, and I'll take your word for it that they didn't have scope/budget to account for these things in the study.”
That’s just the way it goes in research. I can attest to that. I am writing some research right now myself (different subject matter). As I was working, it occurred to me that even as I answer one set of questions, someone definitely needs to answer another set on the same issue. I could have tried to answer it, but it probably went beyond the budget and time constraints I am working with.
“think the point you were making is: as a study increases the number of factors, the odds of finding a false positive is higher.”
“If what you're trying to say is that the small correlations (positive and negative) are just noise, I agree with that as well.”
Exactly right.
“Is that number not too small to glean any significant results along that range of exposure?”
It depends on the circumstances. The central limit theorem states that we would need a sample of at least 30 to achieve statistical power. Some people use the CLT like a dictator, instead of a good general rule. They are misusing it, when this happens. In this case it would have been good to see a larger sample in the high exposure group. I don’t think it matters at all in the smallest exposure group, since they didn’t find anything in the mid-range groups.
Good question re: the lead. I really don’t know. If I have some free time I will post about it. If you get there first, then feel free to do likewise.
Are you proposing that these other factors might exist, so that if a study only looks at one aspect, it will not control for the other factors that could lead to autism?
Yes
If so, then you are right, this study can do little to answer your point.But, my point it is that this is your personal position (though thoughtful, educated, and informed). It does not reflect the usual position given by advocates of the mercury theory. This study does address “their” point in an acceptable manner. It does not address the points you raise or your more fundamental criticisms of epidemiology.
My point is that the goal should be to address the complexity to be more close to the truth; not to address “their” point; because “their” point as the ONLY cause for an other wise neurotypical child, born neurotypical and without susceptibilities has been demonstrated to be very uncomplete and incorrect. The problem is that to design studies to continue to address “their” point instead of the complexity that is demonstrating and demonstrated to be inherent to an heterogeneous ASD diagnosis is not more useful. The knowledge is not going to increase because the problem is that general mainstreamed medicine thinks that”their” point- the most extremist- is the position and if they demonstrate it wrong the controversy will end. It will not because the problem has not been in the begining of being studied, IMHO.
You write “First, I have not fellow advocates because I do not agree with I would say almost nobody in terms of advocacy.” Noted, and I will not mistake their argument for yours. But that may put you in a bind depending on the way you reference your material.
Please consider that perhaps I mention some material because of the problems I see related to the above comments. The proposal is wrong such as it is, but because the problem has not been properly studied, nobody can be sure about a role as collaborator in the symptoms. AND what studies like this are doing is closing the door to the further research my family- and many many others- could be providing the anecdotic evidence.
You write “I have told you before; each family has its own style of advocacy and interpretation of ASD. I have my personal style and position. The issue of coherence, under a different analysis than yours and the advocates you mention, is not a problem under a different light.” The medical scientists try to answer the common usual question. To this end they will try to answer coherent, ideas that are testable. There is an inevitable bias in that they lean towards the questions that most people make, as opposed to the minority.
But once the most vocal- not necessarily the majority- ideas –even extreme- are addressed, other ideas should be considered carefully and they have not been- for example bioaccumulation of toxic elements because of transport and excretion issues and imbalances of essential elements in autism.
My point, Why if autism is the most heritable of the medical conditions, epidemiology is still done mixing vaccinated autistic and non-autistcs?
Why if the proposal could be a transport and /or excretion problem in autistics under the ASD diagnosis studies focused on this group specifically- very clearly situated under the DSMIV- are not common?
You write “This is not what the manuscripts I included Are talking about. They are clear in this issue
I don’t understand this point. Will you rephrase and/or cite it?.”
From the links I gave you
“Physiological systems are inherently complex and often far more complex than usually appreciated. This truism, which in many ways comprises a contrary position to Occam´s razor, has emerged from repeated observations that physiological systems are best assumed to be more complex than at first apparent until we can demonstrate otherwise.”
Regulation of water fluxes across biological membranes for example Jon. It was assumed to be based on simple osmotic forces BUT we learned about the tremendous importance of aquaporins in regulating membrane water permeability. For example Agre et al (1995) said
The long standing biophysical question of how water crosses plasma mebranes has been answered by the recent discovery of the aquaporins.
The point is the difference between complicated and complex. Former ones are much more predictible, complex are not, because of the potential importance of the unknown. The most approacheable answer to a ph ysiological study of complex interactins involves the need of multidisciplinarity.
“Because biological phenomena are the result of an evolutionary process that simply uses what is available, many biological phenomena are simply cobbled together and in no sense can be the “simplest”way to accomplish something”
What about the unknown in autism?
You write “In this sense, prevalence are important for the political issues but are not useful for the individual presentation of concomitant medical problems to an ASD diagnosis”
However now you are in lonely territory. You are welcome to it, but I predict that you are going to be consistently disappointed with future research in this field. Unless you can build up a lot of support for the ideas you propose.
To be honest, I would like to see what a study designed by you attempting to answer these points here would look like. I would like to see if I would agree that it is a well designed study for this issue.
Well to begin wit I do think that two fundamental group of questions should be the questions of the studies related to vaccines/thimerosal in autism.Because they include a lot of controversial issues and potentially of not interest for your usual readers I prefer to discuss privately-because also of privacy issues.
I also appreciate your time and kindness to discuss these issues with me.
You said
Unless you can build up a lot of support for the ideas you propose.
Well, I have contacted several researchers of multiple ideas about autism with several thoughts to discuss, with different answers.
The support- if something like this for these ideas is going to be present in the future.- will be provided if responsable and serious and very trained ethic scientists find merit in these ideas to perform the proper adequate design of the studies to -even partially- answer them or to redesign them better orand if they ( ideas such I have proposed here and elsewhere) are really more close to the true situation, because of the potential importance of the unknown.
I have trust in scientific method, especially because of the enormous amount of research on -for example, epigenetics and the immune system and autism going on. IF these ideas have merit probably someone is going to test them in the future.Otherwise, And I know and for sure I am presenting as such today, they are opinions-even when they are testable and scientifically based on scientific literature and knowledge with partial evidence from other fields under a personal-if you want- analysis, doing questions. There are not studies on the issues on autism I am interested on and this is why I am very careful on how I present the ideas. I prefer to do questions.
Interverbal,
"It depends on the circumstances. The central limit theorem states that we would need a sample of at least 30 to achieve statistical power. Some people use the CLT like a dictator, instead of a good general rule. They are misusing it, when this happens. "
Thanks, I'll look that theory up.
"In this case it would have been good to see a larger sample in the high exposure group. I don’t think it matters at all in the smallest exposure group, since they didn’t find anything in the mid-range groups. "
Your statement here lost me. My assumption was that a cohort study like this assumes a baseline rate of illness, and then sees if the rate of illness increases in correlation with the increased exposure? So are you saying there was no illness reported in the mid-ranges, or that there was no deviation from baseline? If the latter, than wouldn't the cohort of low/no exposure be the one where you would expect to set your baseline?
"Good question re: the lead. I really don’t know. If I have some free time I will post about it. If you get there first, then feel free to do likewise."
I will spend a bit of time this week on this one. I would be delighted if you would one day write about lead exposure and the behaviour of the government safety establishment. I'll see what I can dig up.
Hi Schwartz,
“Your statement here lost me. My assumption was that a cohort study like this assumes a baseline rate of illness, and then sees if the rate of illness increases in correlation with the increased exposure?”
In a study like this, we assume that if the factor we are looking at causes the change that the more we add, the greater the effect will be. We should that there is a stepwise effect. In this were there was no effect noted at the mid-range, we can discount the likelihood of an effect at the lowest range.
Lead in the US does undeniably have an interesting history.
Hi Maria,
You write “My point is that the goal should be to address the complexity to be more close to the truth; not to address “their” point; because “their” point as the ONLY cause for an other wise neurotypical child, born neurotypical and without susceptibilities has been demonstrated to be very uncomplete and incorrect.”
But aren’t you assuming that this research will lead us to be more close to the truth?
“The problem is that to design studies to continue to address “their” point instead of the complexity that is demonstrating and demonstrated to be inherent to an heterogeneous ASD diagnosis is not more useful.”
Complexity in terms of?
“The knowledge is not going to increase because the problem is that general mainstreamed medicine thinks that”their” point- the most extremist- is the position and if they demonstrate it wrong the controversy will end. It will not because the problem has not been in the begining of being studied, IMHO.”
True enough, but only if we assume that the line of research you suggest will be default tell us something we need to no. There is no guarantee of that.
My point, Why if autism is the most heritable of the medical conditions, epidemiology is still done mixing vaccinated autistic and non-autistcs?
Because it appears from other research that the vaccines have no effect.
Re: Ockham’s razor you cite the following:
“Physiological systems are inherently complex and often far more complex than usually appreciated. This truism, which in many ways comprises a contrary position to Occam´s razor, has emerged from repeated observations that physiological systems are best assumed to be more complex than at first apparent until we can demonstrate otherwise.”
Okay, so physiological systems are complex. How is this different from other complex things like certain applications of mathematics or physics, where Ockham’s razor is used? Ockham’s razor just means we don’t multiply our theories beyond what is necessary to explain them. It doesn’t mean explanations have to be simple, it just mean that they should only be made as complex as necessary. The emphasis is on the “necessary” here. Some things are inherently complex, Ockham's razor still applies here. I don’t agree with the author’s argument here.
Also, it is funny how the authors try to invoke evolutionary aspects into their argument. I wonder what some of the premier evolutionist like Dawkins would think about that. He seems to have no problem using O.R. in his defense of evolution. If I remember correctly Gould didn’t either.
Thank you for the lesson on osmosis and aquaporins. This seems to be a classic example of Popperian progression. Is this finding now widely supported? Given the works by Kuhn I would have guessed that such a change would have resulted in a lot of feet dragging. What do you attribute the quick acceptance to (provided it does exist?)
Post a Comment
<< Home