Slate

WHO WE ARE

We are SLATE—a team of two Masters students at Carnegie Mellon University’s Human-Computer Interaction Institute, a community dedicated to research and education in topics related to computer technology and its support of human activity and society. For our Capstone project, we were tasked by Vapor, LLC to investigate and envision the possibilities for a personal learning companion for high school students. We worked closely with two of our peers—Riam Chantree and Jamie Crawford—to determine a project scope and conduct our field research. Please refer to pages 55 through 57 for more information about the contributors to this project.

PROJECT SCOPE

Our client came to us with the vision of an elaborate personal learning companion built on emerging technologies. To make our work more manageable, we needed to specify a scope within this larger, more ambitious vision. After considering student needs in areas such as scheduling and organization, we decided to focus on METACOGNITION and SELF-REGULATED LEARNING . While there has been extensive research in other areas of learning and education, it became apparent through our literature review and expert interviews that metacognition is a relatively unexplored area with a high potential for impact. Please refer to pages 4 through 5 for more detail regarding our area of focus.

RESEARCH PROCESS

Although we used various methods throughout the research process, we gathered the bulk of our data from semi-structured interviews with 16 high school students. In an initial round of interviews, we aimed to develop a general understanding of each student’s approach toward activities and challenges. In the week following these interviews, we asked each student to complete a Learning Activities Booklet. In these booklets, students were prompted with questions about 12 activities, and their answers were used to further discussions about habits and learning processes in a second round of interviews. Please refer to pages 18 through 21 for more information on our research methods.

KEY FINDINGS

After transcribing each interview, we pulled relevant quotes from the transcriptions for use in an affinity diagram. With the affinity diagram as our main method of synthesis, we were able to group and discover interesting patterns in the data we collected. Our key findings revolve around motivation, competition, and the use of resources and self-regulated learning strategies. Many of these findings lead to insights that could be harnessed in the creation of a personal learning companion. Please refer to pages 22 through 35 for specific findings and supporting evidence.

VISIONS

With our research insights in mind, we created three visions of a personal learning companion that will target metacognitive development in high school students. Our first vision shows how a system might help students learn to assess resource credibility, our second vision shows how a system might facilitate a student’s help-seeking process, and our third vision shows how a system might teach students new learning strategies. Please refer to page 40 through 42 for a storyboard of each vision.

EDUCATION IN AMERICA

In the existing educational system, high school students often do not receive the individual attention and encouragement they need in order to meet their own definitions of success. We were tasked to address this shortcoming with a personal learning companion that will accompany students in their learning pursuits both in and out of school.

OUR CLIENT

Our client for this project is Vapor, LLC. This company is run by Richard Katzman, a successful businessman who led the Katzman family business for several decades, bringing its yearly revenue from 4 to 500 million US dollars in that time. He has served on Princeton Review’s Board of Directors, and is currently the Executive Director of Learning Content at Noodle Education.

Recognizing that human teachers are a scarce commodity, Katzman is determined to leverage emerging technologies in order to bring this concept one step closer to reality. He envisions an intelligent, customized, and adaptable system that will improve a student’s learning process by anticipating their needs and keeping them engaged in relevant material. Potential areas of impact that have been singled out include the organization and balance of academic courses, extracurricular activities, and social events, as well as the navigation and utilization of an ever-expanding library of online learning content.

Research findings and preliminary visions for the project prototype were presented in May of 2014, at the end of the spring semester.

THE PROJECT

To investigate the metacognitive needs of high school students and conceive a system that helps them become independent learners by addressing these needs.

BACKGROUND

We are two students in the Masters Program at Carnegie Mellon University’s Human-Computer Interaction Institute, and we have been investigating the metacognitive needs of high school students for our Capstone project. This project is a culmination of everything we have learned about human- computer interaction, and this book is a report of the research findings and insights we gathered over the 2014 spring semester. These findings and insights inform our visions for the project moving forward, and will be used to support the development of future products.

PROJECT SCOPE

Our client’s vision is ambitious and multi-faceted. Recognizing that we would not have the time or resources to realize this vision in its entirety, we set to work determining an appropriate scope within the bigger project. In order to gain a better understanding of our problem space, we put together a literature review and competitive analysis, both of which are detailed in Riam Chantree’s Research Report. We also watched recorded talks, spoke with subject matter experts, and conducted guerrilla interviews with students from two high-performing private schools.

Topics we covered in this phase of research include motivation, METACOGNITION , scheduling, teens, rapport, and learning technologies. Although there appeared to be a need in the space of scheduling and organization, we quickly discovered through our competitive analysis and guerrilla interviews that the market is actually flooded with various organizational tools students have no interest in using.

We learned from our literature review and expert interviews that METACOGNITION , on the other hand, is a relatively unexplored area. It can be further divided into the areas of METACOGNITIVE KNOWLEDGE and METACOGNITIVE REGULATION , and the little research that exists suggests a promising opportunity for design and a high potential for impact in the latter. It was while exploring this idea that we came across a list of 10 SELF-REGULATED LEARNING strategies in a paper by Nandagopal and Ericsson. Adapted from a list by Zimmerman and Martinez-Pons in a 1986 paper, these strategies became a reference for our research and are detailed on the following spread.

We referenced this list of 10 SELF-REGULATED LEARNING strategies in our investigation of high school students’ metacognitive needs. Originated by Zimmerman and Martinez-Pons, each strategy suggests the presence of a student’s agency and goals, and refers to a set of actions carried out for the purpose of acquiring knowledge or skill.

SELF-EVALUATING

ORGANIZING & TRANSFORMING

GOAL SETTING & PLANNING

SEEKING INFORMATION

KEEPING RECORDS & MONITORING

ENVIRONMENT RESTRUCTURING

SELF-CONSEQUENCES

REHEARSING & MEMORIZING

SEEKING SOCIAL ASSISTANCE

REVIEWING RECORDS

• Train in a way that encourages metacognitive skills across domains.
• Teachable agents hold a lot of promise for training metacognitive skills.
• Habit-building is important to maintaining metacognitive skills.

In his discussion with us, Professor Aleven highlighted some of his findings on metacognition and intelligent tutors in the context of our research. He pointed out that while it is possible to train metacognitive skills, those skills do not appear to remain once training has ceased. This transient nature suggests that a system able to prolong training effects could potentially help students develop lasting metacognitive habits.

Aleven also suggested possible opportunities in the transfer of metacognitive skills across domains. Although the possibility of this transfer is consistent with existing literature on learning principles, this area that has seen little research due to its daunting nature. According to Aleven, showing the use of metacognitive skills across domains could reinforce such transfer. For example, one of these skills might be self-explanation. Having students learn to explain concepts to themselves could help them understand their level of learning and how they might improve.

In addition, Aleven mentioned certain opportunities for design. He described teachable agents as one of the greater innovations from the learning sciences in the last decade, with projects such as Betty’s Brain by Vanderbilt University’s Teachable Agents Group incorporating metacognitive training.

• Playful rudeness in the agents helps creating rapport with the student.
• Teachable agent leads to greater learning outcomes than teaching agents.
• Agents with personalities different than the students grabs their attention.

Samantha helped us develop a better understanding of virtual peer systems. Much of our discussion revolved around teaching agents, teachable agents, avatars, authorable agents, and effective techniques to build rapport with each.

We focused in particular on teachable agents, which are virtual peers a student must teach how to solve a problem. She suggested developing virtual peers with a personality different from the student, in order to get students to think differently about the same topic.

In order to build rapport and engagement with the technology, Samantha suggested to include an awareness of the student’s culture, the use of colloquial terms, and a touch of rudeness in responses from the virtual peer. Her work has showed that an agent’s overly polite responses to rude inputs from students took away from their immersion. While we should be cautious and sensitive in our use of these ideas, simulating playfulness may increase student engagement.

• The best learning strategies depend on the domain.
• Motivation is a necessary aspect of training in the self-regulated learning strategies.

While Professor Koedinger discussed many values of metacognitive tutoring, he also brought up myths and nuances about metacognition that we should bear in mind. For instance, he suggested we exercise caution in assuming that successful learning comes from or leads to good self-regulation. Although his previous research has yielded positive results in using a tutor to change student behavior in a domain level skill, it has not been shown that those changes in behavior lead to stronger domain learning.

It may be a similar case for students attempting to develop self-regulation strategies. According to Koedinger, those students may be no better off simply because they don’t care. While they may achieve success in exercising self-regulating strategies when prompted to do so, they are unlikely to use them otherwise. This suggests that any system we design to help in the development of metacog- nitive skills may be best paired with a system that builds trust in the value of those skills.

Furthermore, Koedinger discussed how different domains may benefit from different metacognitive strategies. Thus, while teaching these skills in a cross-domain system, we should ensure that particular strategies are taught in their appropriate contexts.

• Video games can be used to engage students in educational content.
• Students need guidance and an organized way to search materials.
• Students can be weaned off help to start searching for content on their own.

Bruce shared his insights on cultivating motivation and getting students engaged in learning systems. According to him, researchers have been successful in disguising educational content in video games for the younger generation of students.

Keeping the vast collection of online learning content in mind, Bruce suggested building a system on top of existing curated content to help learners organize their searches. An important question we might use to guide us is how to provide the appropriate content to students based on their individual characteristics. Furthermore, some method of weaning students off any guidance and encouraging them to discover content on their own may prove highly beneficial.

• Labeling students according to performance and achievement can be harmful to the students.
• Spaced reminders improves students’ organizational skills.
• The system should gain students’ trust.

After listening to our brief about the Hobbes Project, Assistant Professor Ogan shared concerns about grouping students according to performance and/or potential. In particular, she suggested that labeling students as low potential could be extremely harmful.

Ogan also spoke of the ability learning companions have to improve students’ organizational skills by providing spaced reminders. These could be given when they should be studying, when they should be partaking in other activities, or when they should be doing tasks for classes in order to avoid a stressful situation when a due date approaches. Once students begin to trust the system, they may benefit greatly from its recommendations and be inclined to use it more frequently.

• We could build technology that helps students make discoveries on their own.
• Students need to learn how they should find information.

Professor Rosé discussed the benefits of self-learning in students. She has found through her experience and research that high school students are easily confused by excessive information, and a “system that can teach students to look for and find information on their own” will help them become more able self-sufficient learners.

At the same time, Rosé’s research shows that “self-regulated learning is unrealistic because students are not motivated to go out and do anything extra.” She questioned the validity of optional work and suggested that a system designed to help students finish their homework might be more exciting. In terms of maintaining motivation, she has found that “lots of people struggle to keep up and get discouraged and then give up.” This limitation presents an opportunity for us to create a system that will encourage students to engage in self-regulation and become better independent learners.