Andrea Zellner

Feedback on Article Critique #2 from Dr. Matthew Koehler:

A. Theoretical Perspective (about 3 pages)

1. Critique the author’s conceptual framework (6/8). – You do all three things, report, critique, and to some extent, support your critique with an argument. I think there’s some “teeth” missing in the critique/argument that you could expand upon. For example, do they really discuss the weaknesses or only superficially? What is the connection to affordances? (the authors of the literature they cite would be aghast at their use of the term). Is the argument they put forth one of “a gap in our understanding of research”? If so, what is that gap?
2. Comment on the need for this study and its importance. (7/8) – Pretty good response. The rationale I put forward above would apply here, in that perhaps missing some teeth. For example, “understanding how technologies might be integrated is a relevant question” is not the same as *this* studies with PDAs as being relevant and important. I’m not saying that it isn’t, but arguing generally doesn’t make the best possible case for this particular study.
3. How effectively does the author tie the study to relevant theory and prior research? (4/8) – You fell into a mode here of reporting, at the expense of not critiquing (you need to do both). What makes their lit connections effective? Did they connect to everything they should? Do they connect to some bodies of literature that are not necessary?
4. Evaluate the clarity and appropriateness of the research questions or hypotheses. (7/8) – Good answer on clarity, appropriateness is a bit harder to get a handle on in your answer.

B. Research Design and Analysis (about 4 pages)

1. Critique the appropriateness and adequacy of the study’s design in relation to the research questions or hypotheses (6/8) – You identified it as experimental design (or strongly suggested so near the end of your section). Given the choices in Woolfolk Chapter 1, I understand why you might say this. But to be clear, they didn’t do a “true experiment”, they did something called a quasi-experimental design. To be a true experiment, every participant is randomly assigned to a condition. In a quasi-experimental design, whole classes are assigned to a condition.  Does this difference matter? Yes. If there is a difference between the two conditions, it is harder to determine that the intervention caused the difference, given that many other things are potentially different in two classrooms, including the teacher, the learning environment, perhaps even the time of day might matter. You say that this design is appropriate, but I don’t understand why? (tell me). What makes it appropriate? Consider, for example, a correlational study would not be appropriate if they were interested in questions of causation. So, what about their question and a quasi-experimental design makes them a good match?  WAIT, STOP THE PRESSES. LATER IN YOUR REVIEW, I SEE ANSWERS TO THESE QUESTIONS! Example: “In order to study the efficacy of the utilization of PDAs in the experiential learning flow, it is reasonable to study two matched samples of students, one sample group with the technology and one without, in order to determine whether one intervention had a greater impact on knowledge acquisition and knowledge creation.” (I’m not going to re-write my commentary, because I wanted you to see all the questions/concerns I had while reading your section explicitly addressing this issue. Next critique, consider putting all the stuff related to this section in one place, it was good stuff!)
2. Critique the adequacy of the study’s sampling methods (e.g., choice of participants) and their implications for generalizability (5/8) – What you have here is good, but just a bit more is needed. The punchline is “raises questions of generalizability.” What questions? Why? What is the population they wish to generalize to? Knowing what you know about the sample, what cause for concern might you have that this sample wouldn’t generalize to the population of interest?
3. Critique the adequacy of the study’s procedures and materials (e.g., interventions, interview protocols, data collection procedures). (6/8) — Good. Although much of what you’ve written is about design, and not about procedures and data collection per se.
4. Critique the appropriateness and quality (e.g., reliability, validity) of the measures used. (4/8) – You raise some good points about validity (it has face validity with knowledge acquisition, use of the trick question, etc.). But, you need to address validity and reliability more broadly. For example, you could state the authors did not report any reliability statistics for any of their measures, or speak generally about the reliability and validity of self-report measures, etc. Make sure you address both (reliability and validity), and both of those words appear in your answer.
5. Critique the adequacy of the study’s data analyses. For example: Have important statistical assumptions been met? Are the analyses appropriate for the study’s design? Are the analyses appropriate for the data collected? (4/8) – Again, what you have is good. But, go further, and address the rest of the question.  What are the assumptions of the t-test and Mann-Whitney test (list them), and for each, to what extent are they met? (yes, no, can’t tell because the authors didn’t report that data).

C. Interpretation and Implications of Results (about 3 pages)

1. Critique the author’s discussion of the methodological and/or conceptual limitations of the results (4/8) – Nice job of summarizing, and making an evaluation (“concise and informative”), but not a lot of evidence to support your evaluation. One way to approach this problem may be to state what the key limitations of the study appear to be, and the extent to which the authors addressed them. Did they get them all? Did they miss any?.
2. How consistent and comprehensive are the author’s conclusions with the reported results? (6/8) – For the most part, you summarize, critique, and support your argument. But I’m not buying it. Specifically “ The authors additionally conclude that technology offered affordances that, when considering the pedagogical design, benefited the learning flow of the experience.” It’s not clear to me that the differences are a result of the technology affordance, but rather *something* was different between the two groups. Possibilities include the technology, but also the fact that they were two different intact classrooms, that it was about photography and not PDAs, and that the structure of discourse and interaction around photography is different than what the other group was doing. I guess I’m saying, they are overstepping their results to make a definitive conclusion about the cause.
3. How well did the author relate the results to the study’s theoretical base? (7/8) – Nice!
4. In your view, what is the significance of the study, and what are its primary implications for theory, future research, and practice? (7/8) – Nice!