Tuesday, March 11, 2014

Anecdotal Evidence and Inductive Conclusions in Psychiatry: Part II

Cherry picking: pointing to individual cases or data that seem to confirm a particular position, while ignoring a significant portion of related cases or data that may contradict that position

In part I of this post, I discussed the issue of inductive reasoning, and how it can be a source of quite valid scientific evidence. This brings up the question of what is a valid conclusion based on inductive reason versus an invalid conclusion based on one person’s personal experience.

The following comes from my book, How Dysfunctional Families Spur Mental Disorders.

Anecdotal evidence in medicine is often misleadingly defined as evidence based on only one clinician’s personal experience with a treatment or diagnosis in question. If that is the standard that is to be used, clearly many reasons exist to question the validity of inferences drawn from these experiences. Individuals are well known to have various biases that color their observations and the conclusions they draw from them. 

They may have blind spots because of their own emotional conflicts. They may ignore evidence that is contradictory to their point of view. Their observations may be limited by their pet theories about the phenomena in question. 

Perhaps even more importantly, they may be seeing unusual cases that are not representative of more “typical” cases in one way or another – a so-called selection bias.

An obvious case of selection bias was illustrated by a statement I heard made at a conference by a family therapy pioneer, the late Jay Haley. I had always admired Mr. Haley for many of his fascinating and utilitarian ideas and observations. However, in this case he betrayed some ignorance. He stated that he did not believe antidepressant drugs were ever effective because none of the patients referred to him had ever responded to them.

Of course, his being a well-known family therapist who did not believe in medication had a tremendous effect on exactly who would be referred to him. Not everyone does respond to drug treatment. Anyone who had responded to an antidepressant would, in all probability, rarely if ever darken his door. Hence, with his sample, he would be misled into thinking that the medicines were not effective for anybody. This form of bias is very common and can be quite subtle. For example, it can affect one’s beliefs about such matters as racial stereotypes or a determination of how trustworthy members of a city’s police department are.

Do these types of biases invalidate all clinical experience? Hardly. First of all, we have to distinguish between the descriptions of the actual events contained within specific anecdotes, and the conclusions or inferences which are drawn from these events. 

Let us first examine the descriptions of what actually happened. A specific anecdote may be accurately observed and described, or not so accurately. If important details are altered or left out entirely, the anecdote may indeed be worthless. However, the exact same thing can be said about empirical studies.

Important details may not even be known to an observer. Particularly with observations of family behavior within a practitioner’s or researcher’s office, important information is almost always hidden, either by design or unwittingly. In addition to the fact that one does not see the whole picture in any single context, there is also a basic problem inherent in the nature of interactions between intimates.

With verbal behavior, for instance, linguists refer to a quality called ellipsis. What this means is that in conversations among people who have known each other for a while, certain information is not spelled out verbally because the other person already knows it. Strangers such as therapists who are listening in and who have not been privy to these prior experiences may think they know what the family is talking about, but they may in reality be completely clueless.

Let us now consider the separate issue of conclusions that are drawn from anecdotes, as opposed to their description. The questions raised by an accurately-described clinical observation can be quite valid, but the answers inferred from it can be completely wrong. Conclusions based on clinical “anecdotes” exist on a continuum from relatively accurate ones to those that are extremely biased to those that are based on spectacular inferential or logical leaps of faith.

Relatively unbiased clinical conclusions based on anecdotes by mental health professionals have many things in common: 

·   They are based on a sample that one has a reasonable expectation is at least somewhat representative of a larger population.

·    They make use, not just the practitioner’s observations, but of the observations of other professionals whom one knows to be reliable and open minded. These clinicians should also be ones known to take the time with their patients necessary to take a complete history. Widespread clinical experience by competent clinicians is something upon which someone can make a very valid inductive conclusion, and is not just “anecdotal evidence.” In fact, conclusions drawn from this source tend to be more accurate that those drawn from so called “empirical” studies. Many dangerous side effects from drugs that did not show up in the initial drug studies have been discovered in this manner.
·     They make use of other informants besides the patient when possible.

·     They take into consideration that people and their family members behave quite differently behind closed doors than they do in public, and therefore if at all possible include observations of patient behavior when patients are unaware that they are being observed.

·    They are based on longitudinal observations. That is, the patients on whom conclusions are based have been seen on multiple occasions over an extended period of time.

·     They are not contradicted by commonly observed examples of behavior in everyday life related to the behavior in question.

·      The person proposing the conclusion acknowledges potential biases, such as a financial stake in a certain drug or allegiance to a specific school of therapy, and acknowledges his or her limitations. What former president of the Society of Clinical Psychology, Gerald C. Davison, calls “ex cathedra statements based upon flimsy and subjective evidence,” a hallmark of some psychotherapy gurus, are always highly suspicious. In fact, charlatans are relatively easy to spot. They have a tendency to sidestep challenges. I will give an example of this shortly.
·     The conclusions reached should lead to predictions of patient behavior under certain circumstances that prove to be accurate in a significant number of cases. This is called predictive validity. Of course, human behavior being as unpredictable as it is, at times the predictions will not be completely accurate even if the conclusions are valid, and so this fact must also be taken into account.

·     Conclusions based on anecdotes about treatment efficacy or the reasons for certain observed behavior should consider several alternate possible explanations for the observations. If several explanations are possible, one must make a judgment about which ones are more likely and which are less likely based not on the anecdote alone, but on all sources of data available. These sources include empirical studies, but also include observations from everyday life, as well as material seen in some relatively reliable media such as reputable newspapers.

Now of course stories in the media also do not tell the whole story or may be biased, so one needs to realize again that one can be fooled, and take this into account as well. I used to believe the common myth, for example, that in nature under certain conditions the animals called lemmings would follow each other off a cliff and commit mass suicide.

I was surprised when I learned that this was untrue because I had as a child in 1958 seen a film clip of said mass suicide that was part of a Disney “True Life Adventure” nature movie called White Wilderness. I later learned that, because the Disney crew could not find a real example, they had from behind the scenes driven the group of lemmings off the cliff for the cameras.

On the other hand, many people believe that men have never been to the moon and that films of the moon landings were made in a movie studio using special effects. I must say, I tend to believe that those film clips are real, but few know for certain.

·   If other anecdotes about similar patients and treatments seem to contradict the conclusions based on a given anecdote, an attempt should be made to account for this difference.

As an illustration of the latter point and an example of a the “quick step side step” in scientific presentations. I once heard an expert present new evidence from neuroscience that certain capabilities of which human brains are capable seem to develop only at certain times during early childhood development. This brain development could be adversely affected by a baby’s early social environment. Of course, that is somewhat true.

Like psychoanalysts will, however, the expert went on to conclude that if the adverse early experiences had taken place, the child had no chance of growing up to be normal. I raised my hand and asked about those children who come from horribly adverse backgrounds, are adopted away at an age past the alleged crucial developmental time, and yet still turn out wonderfully. The expert then changed the subject without ever addressing my question.  

If the data doesn't fit your pre-conceived conclusions, just change the subject!

Remember, there are NO empirical, placebo-controlled, double blind controlled studies on whether parachutes reduce the incidence of deaths or injuries after falling out of airplanes. Or that appendectomies are effective in preventing complications and deaths from appendicitis. And yet we all take those things for granted.


  1. Well, I had written a post, but Firefox ate it, so I will use only IE to post on here. Good God, it is so difficult to rewrite a post you've already written, but I will try. (I hate you, Firefox!) Well, my post is going to be shorter and more shrift, so read every word carefully, because I'm kinda agitated at the loss of my post so I cannot just relax and write it. (Also, why is the text box for writing posts so small?)

    A similar issue comes up on the subject of the effectiveness of sex reassignment therapy to treat gender identity disorder. A couple years ago, the American Psychiatric Association's task force on GID released a report here:

    Well, I had written a post, but Firefox ate it, so I will use only IE to post on here. Good God, it is so difficult to rewrite a post you've already written, but I will try. (I hate you, Firefox!) Well, my post is going to be shorter and more shrift, so read every word carefully, because I'm kinda agitated at the loss of my post so I cannot just relax and write it. (Also, why is the text box for writing posts so small?)

    A similar issue comes up on the subject of the effectiveness of sex reassignment therapy to treat gender identity disorder. A couple years ago, the American Psychiatric Association's task force on GID released a report here:

    On page 22, they mention why the quality of the evidence tends to be so low:

    "The majority of the satisfaction/regret outcome studies described above suggests that most subjects experience subjective improvement following gender transition; however, most lacked a control group. Studies assessing correlates of satisfaction through interviews or case reviews would be categorized as APA level G. For some important aspects of transgender care, it would be impossible or unwise to engage in more robust study designs due to ethical concerns and lack of volunteer enrollment. For example, it would be extremely problematic to include a 'long-term placebo treated control group' in an RCT of hormone therapy efficacy among gender variant adults desiring to use hormonal treatments."

    Nevertheless, on pages 2-3 and 9, they conclude the evidence, coupled with clinical experience is sufficient to recommend sex reassignment therapy in appropriately evaluated cases. Indeed, a month after this the APA released a position statement calling on third-party payers to cover sex reassignment therapy when ordered by a doctor, when it would be medically necessary.

    I spoke with the lead author of the study by email, and he mentioned that blinded studies would be basically impossible due to the nature of the treatment and its effects and RCT studies would often have to contend with ethical concerns, due to the high levels of suicidality found in untreated patients with GID, and also practical issues such as high levels of dropouts; RCT studies could work short term (up to a couple years or so), but for medium- and long-term followup it would be basically impossible.

    This seems to be a situation where, as you mentioned, clinical expertise is very important for judging the effectiveness of a treatment and denying coverage because higher levels of research evidence are unavailable might actually pose ethical concerns.

  2. David, isn't the other component of the dialectic, seeing the need for change through personal growth? BPD individuals really struggle with this in my experience, viewing concepts of acceptance and change as a Hegelian thesis and antithesis, respectively. I don't practice DBT myself, but I work with therapists who do, and they see sustained progress in their DBT patients who "get it" and then are willing to work on the change part. I think that's a key prognostic factor; identifying sometimes only a speck of personal responsibility in a sea of projection, in selecting BPD patients who may respond to DBT. Then going from that to acceptance without self-judgment, and then growth through improving coping skills. Of course, having a therapist whose orientation is to be supportive and nonjudgmental (as our therapists are) has much to do with establishing positive transferences in their patients, and perhaps the motivation to do the work that DBT requires. The wisdom aspect of the serenity prayer is the most challenging goal it seems for BPD patients to attain, which starts with having healthy introspection as a quintessential quality in order to learn from one's mistakes.

    1. Hi Scott,

      I assume your comment was meant for the next, newer post on mindfulnelness.

      I see personal growth as something that occurs very naturally, and any problem with it as due to social forces that actively block and undermine a patient's attempts to self actualize. Therefore, for personal growth to best occur as it naturally would, those social forces should be dealt with.

      The "acceptance" part comes when I tell the patients that their concern with their loved ones is actually a good and understandable thing, and that the conclusions they have drawn about the motivations of other family members and how they need to respond to them are eminently reasonable based on the information that they have (although they are nonetheless partially incorrect).

      Both CBT and analytic therapists tell them in one way or another that their ideas are highly distorted, which is extremely invalidating.

      Once I validate their feelings and beliefs, the patients become more accepting of themselves, but also, more importantly, more open to taking in new information that changes their assessment of the family dynamics.

      In fact, the reason they come to me is that, due to my studies and experience, I know some things that they simply would have no way of knowing. That's what they pay me for. Once we understand completely what is really happening, I can help them design new solutions.