Classroom based research: K vs Non-K learners with "hands-on" material

There is a decent amount of classroom based research out there using learning styles. Here is a review of one paper I found that has some interesting information.

For this study, instructors in an Introduction to Design course (from the Engineering Dept) restructured their course to include more of a “hands-on” component. They wanted to know whether the restructuring helped some types of students more than others. So, students in the course were given the VARK questionnaire and the Myers-Briggs Type Indicator (MBTI) to classify their learning preferences. To assess the effectiveness of the lectures, the students were given a survey (immediately following each lecture) to measure various self-perceptions about how the material had been taught (including things like interest level and applicability). Through this, the authors tried to examine whether the class restructuring was more “helpful” for certain types of learners compared to others.

Two different professors that taught the course were involved in this study. They examined each of the lectures in the class and evaluated them individually for the amount of “hands-on” content in the lecture. Weights of 0, 1, 2, or 3 were assigned to each lecture. A 3 indicated continuous hands-on interaction throughout the lecture. A 2 indicated hands-on activity for most of the lecture, while a 1 indicated that there was a hands-on demo in the class (and a hands-off lecture). A weight of 0 would indicate no hands-on component for the lecture (this is my inference, since this was not articulated in the paper).

They examined the Kinesthetic (K) learners vs non-Kinesthetic (non-K) learners from the VARK questionnaire and they examined the Intuition (N) vs the Sensing (S) types from the MBTI. The N type (from MBTI) “focuses on possibilities, future use, [and the] big picture.” The S type (from MBTI) “focuses on the five senses and experience.” According to the MBTI, these are indicative of the “manner in which a person processes information.” Jenson and Bowe hypothesized that their course restructuring will help S types more than N types. They also hypothesized that the K learners will be helped more than the non-K learners (see previous posts for explanations of the VARK theory and questionnaire).

To examine whether the lectures were helpful, they gave a quick survey to their students. The survey asked students:

            Please rate the following statements on a scale from 1 to 10:

1.      Today’s class kept me interested.

2.      Today’s class was a good learning experience.

3.      This class prepared me well to apply today’s concepts to problems.

4.      This class motivated me to further explore today’s concept.

Although these are all valuable questions to ask, there is a fairly large issue (in my opinion) with this survey. These survey questions are asking for opinions. In question 3, students were asked whether they felt that the lecture prepared them to apply what they learned to problems. This is a good question; however, the survey was given before the students actually did their HW (it was given right after the lecture). A student may feel that the lecture has prepared them, but how accurate is that perception? Where is the evidence that these students are actually prepared? There is nothing here about class performance. How did these changes affect the grades of these students? That would have made a much more compelling argument in favor of learning styles.

Let’s look through the data:

The first noticeable thing is that the mean rating for each of the four questions that are provided for S types, N types, K types, and non-K types show interesting patterns. Notice that N-types rate higher than S-types (on average) for each question except the first. Non-K types rate higher than K-types on all four of the questions. This is a bit shocking considering that, according to the paper, 75% of the lectures have a significant hands-on component after the restructuring. If there is such a large hands-on component, why are the non-K types rating them all higher? One explanation may be that the 75% given early in the paper is not an accurate number. In some of the graphs provided in this paper, I was able to count the number of lectures that were given each particular hands-on weight. The weights reveal that 8 of the 17 lectures received either a 1, 2, or 3. This particular contradiction is not addressed in the paper. How can 75% of the lectures have a significant hands-on component, yet less than 50% of them get assigned a hands-on weight of 1, 2, or 3. That just doesn’t make sense.

The top column indicates mean (X-bar) and standard deviations (sigma) for each question (Q =1, 2, 3, or 4)
From Jenson and Bowe (1999)

Next, the authors decided to look at average deviation from the mean. This can be useful when looking at how ratings changed. Because the N types started off higher than the S types, a direct comparison of their scores would be misleading. So, instead the authors chose to examine how the scores changed. They asked questions like: are the S types more likely to deviate above the mean than the N types? They asked the same question for the K types vs the non-K types.

Again the data are interesting. The authors report their largest differences for questions 1 and 2. According to the paper, for question 1 (about whether the lecture was interesting), S types scored ½ a standard deviation above the mean and N types scored 1/20 of a standard deviation below the mean. K types scored about 1/10 of a standard deviation above the mean while non-K types scored 1/3 of a standard deviation below it. Is it surprising that people that identify themselves as having a preference for kinesthetic information find lectures with hands-on components more interesting? Not really. The same can be said for the difference between the S and N types in this question- this data is really not very surprising considering the S types prefer to have information through their senses rather than to be given a hypothetical or theoretical idea to deal with. The more hands-on a lecture, the more satisfied the K and the S types.

The data for question 2 is actually a little disturbing from a teaching perspective. Question 2 is about whether the lecture was a good “learning experience.” S types see a small increase above their mean response (1/5 of a standard deviation) with the hands-on lectures and N-types see no real change for hands-on when compared to non-hands-on lectures (1/100 of a standard deviation decrease below the mean). While S types find that the hands-on components increase their learning experience, the N types don’t seem to care. Again, since we are dealing with preferences of sensory modality here, this is not a surprising result. The surprise comes when the K vs Non-K data is examined.

K types rated the hands-on lectures approx 1/20 of a standard deviation higher than the mean rating. That is not a large effect by any stretch of the imagination. The non-K types rated the hands-on lectures nearly ½ of a standard deviation below the mean. Now, I know the purpose of this study was to find ways to benefit those students with “learning styles” that are usually neglected with traditional teaching methods- I get that. But their “hands-on” lectures actually led to a large portion of the class saying that those lectures were a lower than average learning experience. This decrease for the non-K types was not accompanied by an increase for the K types. One can conclude that the addition of the hands-on components decreased learning experiences for some students while providing no increase for others. That is a net loss for students- why make such a change?

Thus far, I have given my explanations and responses for their explanations of their data. As usual, I like to look at the data itself to draw my own conclusions. The figures provided in this study provide some additional information that isn’t given in the author’s conclusions. For example, the ratings are all over the place. I have summarized how many data points (lecture ratings) were above and below the mean ratings for the various groups below. In italics are all of the ratings that fly in the face of the general pattern described by the authors (namely that S types do better with hands-on than N types and that K types do better at hands-on than non-K types).

Fig 3:   S-3: 3 above 1 below, S-2: 2 above, S-1: 1 below 1 above

N-3: 1 above, 2 at the mean, 1 below, N-2: 1 above 1 below, N-1: 2 below

Fig 4:   K-3: 3 above 1 below, K-2: 1 above 1 at the mean, K-1: 2 below

NK-3: 1 above, 3 below; NK-2: 1 above 1 below; NK-1: 1 above 1 below

Fig 5:   S-3: 2 above, 2 below; S-2: 1 above, 1 at the mean; S-1: 1 above, 1 below

N-3: 2 below, 2 above; N-2: 1 above, 1 below; N-1: 1 below, 1 at the mean

Fig 6:   K-3: 2 above, 1 below, 1 at the mean; K-2: 1 above, 1 below; K-1: 2 below

NK-3: 4 below; NK-2: 1 above, 1 below; NK-1: 2 above

Notice that of the 12 comparisons made overall for the lectures with hands-on content in these figures, 5 of them are bolded and italicized. That means that 5/12 of their comparisons either show no difference between the type of learner or show a difference that is the exact opposite from what the authors proposed. I did not go through this analysis for their ratings from survey questions 3 and 4. The data from those are even less convincing.

Also I believe it is important to note that the information that was being studied was a greater predictor for the class responses than the amount of hands-on components in the lectures. For example, lectures 7 and 14 have universally low ratings. This was true across nearly all of their survey questions and across all of their learning types (in both MBTI and VARK). Perhaps the more important factor is the information being provided and communicated to the students and not the way it was communicated. Notice that lecture 7 was weighted as a 3 for their hands-on scoring and that lecture 14 was weighted a 0 for their hands-on scoring. Are learning style theorists making this an issue of style over substance? Maybe all that matters is the substance (the material) and the style (Visual, Auditory, Reading, Kinesthetic) is irrelevant? These data seem to support that notion.

The authors conclude with “…overall it is shown that the addition of the hand-on experiences significantly improves design courses.” I am disappointed with the use of the word “significantly” here- from what is written in the paper, no statistical significance testing was conducted in this study. Also, I really wonder what is meant by “improves design courses.” We have no idea whether students were more successful in these courses due to their restructuring.

References

Jensen, D., Bowe, M. (1999). “Hands-on Experiences to Enhance Learning of Design: Effectiveness in a Reverse Engineering / Redesign Context When Correlated with MBTI and VARK Types,” Proceedings of ASEE Annual Conf., Charlotte, NC.