Sunday, August 11, 2013

Is a word count "text analysis"?

Richard Day castigates me in his comment for taking a swipe at him for not recanting on the charge he passed along on his blog post that I was dealing with the wrong document when I criticized the state's science standards when, in fact, the person he quoted with approval was looking at the wrong one.

Richard said, "Why did my name get brought back into this? We already settled this." I looked back at some of the previous comments on Vital Remnants and I see that he sort of/kind of back-tracked on this matter and I should probably just take it as a retraction, although I never did see anything on his blog, where he published it, correcting it. So, oh well, we'll just call it even.

In addition, as to his comment that the Department of Education's Statement of Consideration did in fact, contrary to my previous post, address my argument, he's right. I read through it and either missed it or forgot I saw it when I wrote the post. My bad. I'll take responsibility for it. I don't know how I missed it.

However, the Department says does not offer any argument in response to what I said. As Richard himself quotes from the report:
The agency has determined that comments asserting a heavy weighting toward climate science, to the exclusion of other disciplines, are not supported by a careful examination of the standards themselves.
Oookay.

So where did I go wrong? They don't say. They just make an assertion that it ain't so. Anyone can do that. Where is the counter-argument? I realize that they may not be required by law (I'll have to check) for offering one, but, in any case, they don't.

I have asked at least once (maybe twice) for someone to falsify my figures or come up with some other analysis which would render them incorrect. So far, nothing--from Richard or anyone else.

Richard says:
I responded to Cothran’s argument. I objected to his drawing such definitive conclusions on the scant evidence produced by a simple word search methodology. But Cothran carried on the debate by defending his method, claiming that I was wrong to have questioned - what he had now switched to calling - a “text analysis.” 
The only problem is that Cothran had not done a “text analysis.” He did a word count.
If he had done a text analysis - which is much more involved research technique and produces much higher quality information, I would have likely had nothing to say about it. 
After objecting to my posting a claim made by another author, Cothran finally conceded that his method was not sufficient for the conclusions he drew, writing, “I fully admit it's not "definitive." I admit the possibility that I could be wrong.” 
"Cothran finally conceded"? This makes it sound like I was holding out on him. If Richard will look back, he will see that I readily said not only that my analysis was not "definitive," but that I had never said it was. Richard makes it sound like I had said it was and reluctantly admitted it wasn't. This is simply a mischaracterization of the facts.

"Definitive" has the sense of being "better than all others" or "final or conclusive." Richard implied that I had said (or implied) that, but I never did and I pointed that out. He needs to drop the pretense, unless he can demonstrate that that's what I said.

And let's address this issue of "text analysis."

Richard says I didn't do a "text analysis"; I did a word count. Okay, first, I should probably explain to Richard that words are, in fact, text. And that analysis is the process of, if I may quote the dictionary (which has authority in most places, although I am uncertain if departments of education are one them) "a process of studying or examining something in detail in order to understand it or explain it." So maybe he would like to explain to me, given the common usage of words, how a word count is not text analysis.

I was using the term "word count" generically, of course. However, I am willing to concede that Richard may have some specialized technique in mind the rules for which I have violated. If so, he should say what it is. To simply keep repeating the expression "text analysis" as if this is a universally agreed-upon technique to analyze documents is getting a little tiresome.

In fact, I took the trouble to do a little investigation on the Internet concerning "text analysis." If you Google the term, you find several interesting things:
  1. There seems to be no one commonly agreed-upon or universally recognized technique for text analysis. In fact, many of the discussions of text analysis discuss the wide disagreement as to what the term means.
  2. The term "text analysis" is frequently used synonymously with "content analysis," which, according to most of the definitions, would fit more closely with what I was doing.
  3. Most definitions of both "text analysis" and "content analysis" would clearly include word counts or analysis of word frequency.
  4. Both "text analysis" and "content analysis," but particularly content analysis, frequently and explicitly employ word counting or analyses of word frequency as at least part of their methodology.
Now Richard clearly believes word counts and text analysis are two entirely different things ("Cothran had not done a 'text analysis.' He did a word count.") The problem is that this just doesn't seem to be the view among most of those who are discussing text analysis (or content analysis).

So maybe Richard would like to enlighten us on this one-and-only true method of text analysis he's trying to hold me accountable to and explain why his definition would seem to differ with the many others one can find easily on an Internet search.

Finally, Richard took offense and told me to go soak my head. This was the most sensible thing he's said so far.

This is what I like about Richard: If you hit him, he hits you right back. You gotta respect that. He fights like a man.

Oops, there I go again.

I'm sure that saying that will offend his Politically Correct sensibilities and he will prescribe some supplementary treatment in addition to the head soaking. But I am so refreshed by his candor that I am even now filling the bucket and awaiting whatever additional prescriptions he might suggest, which I will follow enthusiastically.

It could only help.

6 comments:

Richard Day said...

Ha!

Let me respond with a practical story.

I’m in the early stages of a qualitative (mixed method) research study on college readiness. Much of the data in the study will be student reflections, gathered from my EKU undergrads over several semesters on the topic: “Ready (or not) for College: How my high school and I prepared me for college.” We are working with just under 300 samples.

My little group of researchers will have to do a “text analysis” because all of our data is text – no pictures, graphs or anything else - just text. The title you would want to search to learn more about the methodology would be “content analysis” – since written material often contains information for analysis that extends beyond text. The process is systematic, and objective, and I promise, I didn’t make it up. Go to Google Scholar and try again. There are books on this stuff.

One useful subset of a good text analysis is a word count. As you correctly point out, word count provides important information about the nature of the text itself. But (here I go again) alone, it is not sufficiently informative to draw conclusions from.

My students have written about the teachers who allowed them to negotiate grades and get extra chances; the teachers they can get to talk football instead of math, and the ones who hold their feet to the fire. They have written about great schools, and lousy schools. Some have worked hard to prepare themselves; others have coasted. But to what degree do these situations exist? …and where?

Collectively, the data is about schools, and teachers, and students, but word count alone won’t tell us with any accuracy the meaning behind the text. We can't tell a good teacher from a bad teacher by only understanding the number of instances where the word "teacher" occurs. We will have to dig a bit deeper for that.

In our text analysis, we will have to look specifically at each instance and determine what it is the student is really saying - and then we will employ computer technology to organize those comments into the themes that emerge from the data. Only at that point will be begin to understand what portion of the students shared a particular experience. From that, some conclusions may be drawn – but only if the data clearly points to something significant.

Maybe we won’t learn a thing. And if that should happen – heaven forbid - it is my job as a researcher to say so. I must not allow my desire for big results to shape my reporting. That’s not science.

A word count is necessary, but not sufficient.

Content analysis is to sex – as word count is to premature ejaculation.

Your sensitive new age guy,

Richard

Emma said...
This comment has been removed by the author.
Singring said...

'So where did I go wrong? They don't say. They just make an assertion that it ain't so.'

In fact, they say specifically where you went wrong:

'The agency suspects that these comments were based upon a simple word search of the science standards, rather than a count of actual performance expectations (standards).'

'Words such as “global”, “warming”, and “climate” may appear in the standards in isolated uses that are not related to the concept of climate change.'

'The agency has also determined that there may be a misperception that every standard addressing weather or climate is related to climate change.?'

This clearly addresses the reasons as to why your 'analysis' was flawed.

Sometimes, reading a text rather than doing a word count does really help.

You should try it, Martin.

Oh, one final thing:

You've gone from calling the science standards a 'global warming manifesto' to that being 'non-definitive' to doing google searches on 'text analysis', somehow hoping to play your usual semantics game to try and weasel out of it - that's fighting like a girl.

Anonymous said...

http://mediaportal.education.ky.gov/videos/kde-public-hearing-7232013/

J'ai toujours fait une prière à Dieu, qui est fort courte. La voici: Mon Dieu, rendez nos ennemis bien ridicules! Dieu m'a exaucé!

Singring said...

Thanks Anonymous, hugely entertaining and sad/scary at the same time.

I feel particularly bad for the girl about 52 minutes in.

When people like Dawkins talk about religion as psychological child abuse - this is exactly what they are referring to. A young person so indoctrinated with disinformation, fear and guilt they almost break down just discussing the chance that evolution will be taught in schools.

Md Mahbubu Hasan said...
This comment has been removed by a blog administrator.