i am a co-Chair of a special interest group on Mobile Learning for the Society for Information Technology & Teacher Education (SITE). we are a fairly new special interest group and recently started a website and blog centered around mobile learning. i won’t normally copy my blog posts from there to here, but i will this time to help generate interest in our site and our Twitter account on mobile learning (@MLearningSig). here is my latest blog entry from the site:
Last month I made a blog post discussing the need to explore 21st Century Literacies as part of a 1-to-1 computing program. I suggested that schools define what they mean when they refer to 21st Century Skills and then develop the means to measure those skills. This is no small task, but the data can potentially be quite valuable in justifying a 1-to-1 program.
Almost as if on cue, I read an article this week that just came out and fits perfectly with this theme. The citation is as follows:
Bebell, D, O’Dwyer, L. M., Russell, M., & Hoffman, T. (2010). Concerns, Considerations, and New Idea for Data Collection and Research in Educational Technology Studies. Journal of Research on Technology in Education, 43(1), 29 – 52.
And, while the title implies data collection in educational studies, the reality is that these recommendations apply to any kind of program evaluation — particularly to school implementing 1-to-1 programs. I encourage anyone starting a 1-to-1 program or involved in a program to read the article, but I’ll provide a few highlights as an appetizer.
Early in the article, the authors note that very little research exists between the causal relationship of technology and student achievement. This was something I noted in my first blog entry and why I suggested that we also look at skills beyond typical measures of student achievement. The article even explores dynamic phrase, “technology use.” Early in the history of technology in education research, many researchers wanted to measure the use to technology and they remained very general (e.g., a teacher who sometimes used technology with students = technology use). As technology use has picked up, the researchers have become more specific and stringent in who is using technology and how often they use technology (e.g., most definitions now expect students to also use technology; not just teachers). Research has also moved from many students sharing to 1-to-1 computing in many studies. Beyond use, the types of technology being used have changed quite drastically over time. We have some schools with big computer labs and desktop computers while also having schools where students are using mobile iPads ubiquitously. The nuanced uses of technology can vary greatly from school to school, teacher to teacher, and student to student.These evolving definitions make it hard to compare changes in actual technology use across place and time.
The article also has a discussion of standardized testing and technology. This discussion is fairly extensive, but a good recommendation from this section is that most standardized testing takes place using paper and pencil, which fails to reflect to the use and benefits of technology. For example, students who use technology tend to write lengthier and higher quality essays than similar students using paper and pencil. In other words, if you want to test the effects of computers on writing then you should also test students while they are using computers for writing.
Finally, the authors discuss the context in which educational technology takes place. Measuring the effectiveness of technology on a particular student can fail to take into account the classroom technologies available and rules; the schools policies and technologies; and the district technologies and vision for technology. The research often fails to look at how students are using technology at home as this use can also impact school use. This type of comprehensive analysis is referred to as examining the hierarchical structures. More meaningful data collection takes into account the big picture and the authors of this article suggest that future research try and include these deeper levels of context.
I encourage school leaders and individuals involved in 1-to-1 computing initiatives to find this article and delve into it if you can find it.

