Recent research builds a powerful case against what used to be accepted "truths" about learning and technology. First, there is strong evidence that traditional models of learning, traditional definitions of technology effectiveness, and traditional models of the cost effectiveness of technology don't work. In place of these old assumptions, researchers are positing new ways of looking at learning that promote:

* engaged, meaningful learning and collaboration involving challenging and real-life tasks; and

* technology as a tool for learning, communication, and collaboration.

The traditional learning model is not relevant to real student needs

Today's workplaces and communities - and tomorrow's - have tougher requirements than ever before. They need citizens who can think critically and strategically to solve problems. These individuals must learn in a rapidly changing environment, and build knowledge taken from numerous sources and different perspectives. They must understand systems in diverse contexts, and collaborate locally and around the globe.

These attributes contrast sharply with the discrete, low-level skills, content, and assessment methods that traditional ways of learning favor. The new workplace requirements for learning are incompatible with instruction that assumes the teacher is the information giver and the student a passive recipient. The new requirements are at odds with testing programs that assess skills that are useful only in school.

The traditional mechanisms for evaluating the effectiveness of technology programs don't work

Traditionally, we have determined the effectiveness of a technology program vis-a-vis a "regular" program by comparing student outcomes on standardized tests. Numerous researchers, however, question the utility of this method. When the NCREL surveyed experts about traditional models of technology effectiveness, respondents said:

· "Effectiveness is not a function of the technology, but rather of the learning environment and the capability to do things one could not do otherwise."

· "Technology in support of outmoded educational systems is counterproductive."

· "[The reliance on] standardized tests is ludicrous... Technology works in a school not because test scores increase, but because technology empowers new solutions."

Similarly, the typical way to determine a technology's cost effectiveness is to compare the costs of the technology-enhanced program against the costs of the traditional program. Some researchers decry this approach, pointing out that such cost analyses assume that we should continue teaching the same things, rather than change with changing times. Additionally, cost-effectiveness data could constrain development of innovative applications of technology.

There are no definitive answers to questions about the effectiveness of technology in boosting student learning, student readiness for workforce skills, teacher productivity, and cost effectiveness. True, some examples of technology have shown strong and consistent positive results. But even powerful programs might show no effects due to myriad methodological flaws. It would be most unfortunate to reject these because standardized tests showed no significant differences. Instead, measures should evaluate individual technologies against specific learning, collaboration, and communication goals.

What is effective learning and how can it be measured?

Given below is a framework based upon a framework developed by Barbara Means of SRI International. Means identified seven variables that, when present in the classroom, indicate that effective teaching and learning are occurring.

These classroom variables are:

· children are engaged in authentic and multidisciplinary tasks

· assessments are based on students' performance of real tasks

· students participate in interactive modes of instruction

· students work collaboratively

· students are grouped heterogeneously

· the teacher is a facilitator in learning

· students learn through exploration

These variables can be further reorganized them into a set of eight categories of learning and instruction: vision of learning, tasks, assessment, instruction, learning context, grouping, teacher roles, and student roles. These appear in the following table.

Indicators of Engaged Learning

Vision of Learning Indicators

Vision of learning indicators describe the goals of engaged learning. These indicators underlie the philosophy and theme that drive all the other indicators discussed here - tasks, assessment, instruction, learning contexts, grouping, and teacher and student roles. We define engaged learning in terms of four indicators.

In engaged learning settings, students are responsible for their own learning; they take charge and are self-regulated. They define learning goals and problems that are meaningful to them; have a big picture of how specific activities relate to those goals; develop standards of excellence; and evaluate how well they have achieved their goals. They have alternative routes or strategies for attaining goals - and some strategies for correcting errors and redirecting themselves when their plans do not work. They know their own strengths and weaknesses and know how to deal with them productively and constructively. Engaged learners are also able to shape and manage change.

Engaged learners are strategic. They know how to learn and constantly develop and refine their learning and problem-solving strategies. This capacity for learning how to learn includes constructing effective mental models of knowledge even though the information may be very complex and changeable. Strategic learners can apply and transfer knowledge to solve problems creatively. They can make connections at different levels.

Engaged learners become energized by learning. They derive excitement and pleasure from learning. Learning is its own motivator and results in a lifelong passion for solving problems, understanding, and taking the next step in their thinking and activities.

Engaged learners are collaborative. They value others and work with them skillfully. Collaborative learners understand that learning is social, that they must be able to articulate their ideas to others and must have empathy and be fair-minded in dealing with contradictory or conflicting views. They have an ability to identify the strengths of others. Collaborative learners typically value diversity and multiple perspectives.

Task Indicators

In engaged learning, tasks are authentic, challenging, and multi-disciplinary. Tasks are authentic when they are important to learners and learners use their knowledge of the subject matter in much the same way that real-life practitioners use that knowledge. Students learn authentic tasks in context, practicing basic and advanced skills together as a means to learning big concepts. In other words, they learn by doing.

Challenging tasks are typically complex and involve sustained amounts of time. They require students to stretch their thinking - and often their social skills. Challenging tasks are authentic in that they are about real-world problems and projects, build on life experiences, require in-depth work, and benefit from frequent collaboration.

Multidisciplinary work requires wholly integrated instruction. It blends disciplines into thematic or problem-solving pursuits, usually in the form of projects because most work in real life involves multidisciplinary projects.

Assessment Indicators

Assessments that promote engaged learning ask students to demonstrate their knowledge and skills in authentic tasks, projects, or investigations. Performance-based assessments are meaningful, challenging experiences that involve planning, development over time, presentations, and debriefings about what students learned. Students should take part as much as possible in planning the unit in which the assessment occurs, the criteria for evaluating the assessment, and various forms of self-assessments such as keeping journals.

Performance-based assessments are also generative. Students construct their knowledge and develop real products and services, perform in some way, organize events such as conferences, create artistic works, and the like for an audience that cares.

At its best, performance-based assessment is seamless and ongoing. That means that the plans, standards and criteria, products, performances, presentations, and debriefings are all instruction at the same time that they are assessment. And vice versa. Movement from one to the other is transparent to the student. Students generally perceive a well-designed hands-on assessment as a challenging and meaningful learning activity.

Performance-based assessments raise issues of equity and standards. It is critical to have equitable standards - ones that apply to all students. Parents and students, as well as teachers, should be familiar with those standards and be able to evaluate the performance of an individual or group against them.

Instructional Model Indicators

The most powerful instruction is interactive and generative. Interactive instruction actively engages the learner with the resources and learning context to construct new knowledge and skills.

Generative instruction, like generative assessment, brings learners with different perspectives together to produce shared understandings. While learning in traditional instruction is a two-person situation (the teacher and the student), in generative instruction learning is a three-person situation (the teacher, the student, and others). Thus, in generative learning, there is co-construction of knowledge; learning occurs as the result of interactions among the learner, the teacher, and others.

Some Generative Instruction Strategies

Generative approaches to instruction use a wide range of instructional strategies, including:

· Socratic dialogue

· individual and group summarizing

· mechanisms for exploring multiple and differing perspectives

· techniques for building upon prior knowledge

· brainstorming and categorizing

· debriefing

· general and content-specific problem-solving processes

· team teaching

· techniques for constructing mental models and graphic representations

All of these strategies encourage the learner to solve problems actively, conduct meaningful inquiry, reflect, and build a repertoire of effective learning strategies.

Classrooms that foster engaged learning let students learn collaboratively. They are knowledge-building learning communities. Such communities create empathetic learning environments that build on diversity and many perspectives. These features are especially important in classrooms where there are marked differences in students' prior knowledge. In such classrooms, knowledge-building strategies - such as brainstorming - pool the knowledge and experiences of the group, thereby creating more equitable learning conditions for everyone and giving everyone access to the aggregate knowledge.

Focus on Collaboration

Truly collaborative classrooms encourage all students to ask hard questions; define problems; take charge of the conversation when appropriate; participate in setting goals, standards, benchmarks, and assessments; have work-related conversations with various adults in and outside school; and may engage in entrepreneurial activities. This vision contrasts sharply with classrooms in which students respond to questions posed by the teacher. Collaborative classrooms also contrast with cooperative learning settings, which involve highly structured tasks and student roles defined and controlled by the teacher. Collaborative work may be most powerful when it involves flexible, learning-centered investigations that bring students together with practicing professionals and community members. Such collaborations may occur electronically or in work outside the school.

Grouping Indicators

Collaborative work that is learning-centered often involves small groups or teams of two or more students within or across classrooms. Although each student's roles and tasks may be different, all members of the group collaborate to accomplish a joint goal or project. When a project is complex or creative, it is often beneficial to use heterogeneous grouping. Groups that include males and females and a mix of cultures, learning styles, abilities, socioeconomic status, and age bring a wealth of knowledge and perspectives to authentic, challenging tasks.

Many teachers use flexible grouping, configuring and reconfiguring small groups of students according to specific instructional purposes. This flexibility lets them make frequent use of heterogeneous groups and to form groups according to common interests or needs, usually for short periods of time.

Flexible grouping with recurrent use of heterogeneous groups is one of the most equitable means of grouping and assuring that all students have opportunities to learn.

Teacher Role Indicators

In classrooms where students engage in learning, teachers are more than information givers. Teachers are facilitators, guides, and co-learners. As facilitators, teachers provide rich learning environments, experiences, and activities; create opportunities for students to work collaboratively, to solve problems, do authentic tasks, and share knowledge and responsibility.

Teachers play complex and varied roles as guides. They mediate, model, and coach. When mediating student learning, teachers must constantly adjust the level of information and support according to students' needs and help them link new information to prior knowledge, refine their problem-solving strategies, and learn how to learn. Teacher modeling involves thinking aloud and demonstrating, when needed. Coaching involves giving hints or cues, providing feedback, refocusing student efforts, assisting students in the use of a strategy, and providing procedural and factual knowledge when needed. As guides, teachers rely heavily on active listening skills and Socratic questioning techniques.

Given the diverse opportunities and challenges present in education, teachers are often co-learners and co-investigators right alongside students. That is, as teachers and students participate in scientific and other investigations with practicing professionals, they increasingly need to explore new frontiers and become producers of knowledge in knowledge-building communities. Indeed, there will be times, especially as technology advances, when students are the teachers and teachers are the learners.

Students who engage in learning are explorers. They discover concepts and connections and apply skills by interacting with the physical world, materials, technology, and other people. Often students jump into an activity with little prior instruction in order to stimulate their curiosity, become familiar with the instructional materials, and formulate early understandings of the task. Students can then reflect upon ideas and revise, reorganize, and expand upon their understandings with further knowledge, exploration, and debriefing.

Reflective thinking is also essential for students as cognitive apprentices. In cognitive apprenticeships, learning is essentially formative, with daily feedback on many aspects of a complex problem or skill. Learning takes place when students observe, apply, and - through practice - refine their thinking processes so that they increasingly formulate more powerful questions, problems, and solutions, moving toward greater expertise. By reflecting across a diverse range of tasks, students come to identify common elements in their many experiences. This enables them to generalize their skills and transfer their learning to new situations.

For some situations, most often when students must be teachers, students need summative learning experiences. These experiences help them to integrate and holistically represent what they have learned intensely over a period of time and to develop the social skills needed to help others learn.

What defines high technology performance and how can it be measured?

There is strong consensus in the research community that technology and technology-enhanced programs can promote engaged learning. Researchers have identified many features of technology that are important to learning. This section presents indicators for identifying effective, high technology performance, organized within six categories:

· access that a school has to diverse technologies and resources, both within its own classrooms and beyond the school;

· operability of the technology;

· organization of the technology in terms of its location and distribution;

· "engagability," or the capacity of the technology to engage students in challenging learning;

· ease of use; and

· functionality, or the technology's capacity to prepare students to use a variety of technological tools.

For each of these six categories of technology performance, the indicators of high performance that would promote engaged learning are displayed in the following table:

Indicators of High Technology Performance

Access indicators address how physically accessible technology is to the school. A technology or technology-enhanced program has high access when it has connectivity, ubiquity, and interconnectivity. Further, the technology should be used equitably.

Connectivity refers to the technology's ability to access rich resources within and beyond the school because it is connected to those resources. Connections between a school and a telecommunications source must be in place if the school is to benefit from the wealth of free and low-cost resources on the information highway.

In terms of ubiquity, the ideal situation would be for all students to have their own networked computer. Since that probably won't be the case anytime in the near future, technology is considered ubiquitous when computers, printers, media technologies, and other equipment are easily and readily available to teachers and students for problem solving, communication, collaboration, and data exchange. Simply having a computer or multimedia lab in every school is not ubiquitous, because students and teachers have to physically go somewhere and perhaps wait for some length of time before they can use the equipment. Networks of computers and other equipment - especially printers - throughout the school indicate high technology performance.

Interconnectivity occurs when students and teachers communicate and collaborate in diverse ways (exchanging data in different formats and publishing, for example) using technology.

For a school to be connected and interconnected, and for its technology to be ubiquitous, means that everyone has access to the best and most extensive resources the technology has to offer. If a system has home-school connections but no connections to the local library system or to the Internet, or if only students in gifted classes or in magnet schools know how to use those connections effectively, the technology is not being used equitably. Technology in schools should be available to all students so that everyone has access to rich and challenging learning opportunities.

Operability indicators refer to the ease and convenience of using the technology. The first operability indicator, interoperability, is the capacity to easily exchange data with, and connect to, other hardware and software. To do so, the technology must have an open architecture. This feature allows users to access data using different (third-party) hardware and software. It also lets users modify the system - sometimes dramatically. An example of such a modification is when a user can add his or her own template to a spreadsheet or desktop publishing program. Interoperability also requires transparency, which means the capability to move from one format or program to another easily and unobtrusively. More specifically, in transparent systems, the user is not - and does not need to be - aware of the process, procedures, and protocols by which the hardware and software effectively perform their functions.

Technologies or programs that have open architecture and transparency promote engaged learning because they allow teachers and learners to spend maximum time and energy enjoying and using the resources they access, rather than spending their time and energy on learning how to use the technology and/or performing complex and time-consuming procedures to move from one program or format to another.

Organization indicators pertain to questions such as: Where is the information stored? How are resources connected? How do new resources get into the system? Is the transmission asymmetrical (from one source to another) or symmetrical (having two-way transmission capability)? Who is in charge?

Some schools and technology programs centralize information. Students typically access it by way of limited-capability, "dumb" terminals that connect to mainframes or other centralized servers. In such systems, information flows in one direction only - from the central source to users. The system operator is in charge of what information and resources go into the system, when they are entered and distributed to others, and so on.

Centralized systems are likely to inhibit learning to the extent that they use the transfer model of learning and instruction. This model assumes that the central source holds most of the important information and that it is the student's job to transfer the information from this central source to his or her location and "learn" it.

Such systems may offer rich resources such as a multimedia encyclopedia, or an efficient management system for assessment and record keeping. These centralized systems would, by definition, be high performance. However, this high performance may be very limited. For one thing, learning may not be very engaged because the educational objectives of this one-way transmission are likely to be a low-level focus on basic skills.

In contrast to these centralized and relatively closed resource systems, distributed systems are organized very differently. The premise behind distributed systems is that the resources that enable and give shape to learning are spread across many people and places both within the local system and outside it (e.g., the Internet). To this end, systems that provide wide area networks (WANs) allow access to many more resources than do systems that provide only local area networks (LANs).

These networked open systems promote two-way transmissions and user contributions, thereby encouraging users to become producers. Any number of users can contribute information, products, and services to a distributed system for others to share. In these systems, the users control when they make a contribution and what that contribution is.

Distributed systems typically feature tools that make it possible for users to take part in collaborative projects and co-investigations. On-line conferences and bulletin boards, access to remote files and joint products, and the capability to communicate in real time with other users accessing the same data all promote collaboration. Users can access programs to work in groups, build consensus, brainstorm, outline, develop plans, schedule meetings, monitor programs on group objectives, and develop joint products. All these capabilities help develop knowledge-building communities.

This indicator refers to features in a technology's design that promote engaged learning. One such design feature is the technology's capability (e.g., software) to provide challenging tasks, opportunities, and experiences. For example, the technology could provide:

· complex problems and cases; links to challenging curricula and unique resource repositories from museums and libraries; opportunities to examine contrasting

· events or data sets;

· access to experts, peers, community members, and/or other learners who can guide, mentor, tutor, mediate, broker, share, inform, and involve users in learning in productive and meaningful ways;

· access to rich media resources - three-dimensional images, audio, video, virtual reality - for data manipulation and for presentations; and/or

· tools for interactive browsing, searching, and authoring.

A second design feature that enhances engaged learning allows students to learn by doing. Tools such as scenarios and simulations provide opportunities to develop expertise using real-world problems and resources. These tools let the user plan, reflect, make decisions, experience the consequences of actions, change direction, and examine alternative solutions and assumptions.

A third design feature that is important to engaged learning is the extent to which the technology provides guided participation. Various techniques achieve guided participation:

· Socratic questioning

· intelligent tutoring

· diagnosing and guiding the analysis of errors

· adapting the technology or system to respond to student actions

All of these techniques allow users to customize content to suit particular interests or learning styles. Techniques and tools that help students see how practicing professionals and others think also enhance guided instruction. For example, students can use "wizards" - intelligent tools that help users work through a set of complex procedures - embedded questions, prompts, and coaches. These tools provide learners with opportunities to anticipate problems and events.

High performance technology is easy to use. For example, it should provide effective helps; these should be informative, clear, comprehensive, readily available, and context-specific. The technology should be user friendly (accessible and understandable) and encourage user control. This latter attribute means that the user can access tools, information resources, experiences, and opportunities on demand and use them to solve problems, make decisions, and create products. The technology should have a fast processing speed; it should also provide the user with feedback regarding any system delays. Training for and supporting technology use are vital; these services should be available locally as well as be accessible from remote locations.

Finally, the technology should provide information that is just in time and just enough. High performance on this indicator means that people with immediate, pressing needs can easily access simplified, useful information, while people who have time for reflection and exploration can access more complex and rich data.

High functionality ensures, first, that the technology provides diverse tools - generic and context-specific - fundamental to learning and working in the 21st century. These tools begin with "basics" like databases, spreadsheets, and word processing, and move on to such high-level, context-specific tools as sonar for oceanographic research. Another indicator of functionality is the extent to which the technology incorporates media such as color printers, video cameras, audio and video recording and editing equipment, and graphics.

A third indicator of functionality is the extent to which a technology prepares students to use tools that create new programs and tools for others. This refers to opportunities to use wizards, as well as to learn programming and authoring skills. This indicator contrasts sharply with traditional approaches to technology that teach students outmoded programming languages as an end in itself.

Functionality also has to do with the technology's capacity to develop skills related to project design and implementation such as setting goals and benchmarks, creating and monitoring budgets, conducting research and development, preparing analyses and presentations, developing dissemination skills, and marketing.