Literarytech
Learning is a Journey

I have been mulling over the question of different types of assessment. The first of these is the formative assessment. This is a pre-test that evaluates prior knowledge and shapes the learning experience. The second type of assessment is the summative assessment. This post-test evaluates the learning that has taken place and validates the success of the learning event. Finally, there is the proficiency assessment. This exam will validate mastery of a body of knowledge without direct reference to a particular learning experience.

1. Formative Assessment
2. Summative Assessment
3. Proficiency Assessment

This works for an academic exercise in taxonomy, however; does it seem that the experience of the learner is somewhat different? Consider a learner whose objective is achieving a credential through the proficiency assessment. Does this individual actually distinguish between the formative and the summative? From a product development perspective, does it make sense to consider the flow a cyclical one from the formative/summative assessment to the learning and back to the assessment and back to the learning?

I recently took a class in Insights Discovery that works to expand on the Jungian archetypes and the work of Myers-Briggs. It assigns colors to the Jungian archetypes and suggests that we all have all the colors, but manifest them differently. In addition, we manifest these characteristics in different ways and at different times.

The four colors are Blue, Red, Yellow and Green. Blue tends to represent an analytical approach and is characterized by the word Observer. Red is the Director color and is characteristic of a catalyst approach. Yellow is characterized as Inspirer and is the extroverted approach. Green is characterized as Supporter and is the nurturing and caring approach.

I am characterized as an Observing Reformer (Classic) although my less conscious characteristics make me a Directing Reformer (Classic). My Blue and Red are balanced across conscious and less conscious. My yellow is clearly conscious (I learned it in sales) and my green is a challenge. In other words, I want to drive for accomplishment, I’ll probably need some data to move forward, I’ll sell you on the idea, and I’ll probably forget to ask if you enjoyed your weekend.

It was fascinating to see how they identified me with the assessment that was used. They indicated that they have a reliability of 0.9 which is remarkable. Reliability of 1.0 is perfect reliability and that is not achieved in practice.

The objective of this training was to provide a foundation for improving effectiveness of employees and teams at Microsoft (or Boeing or any of the other firms using the system). More advanced study would be needed to better understand the ways in which people’s personalities traits can be leveraged to accomplish greatness.

One of the difficulties with expanding the availability of assessment to prove proficiency or demonstrate learning is the need for proctoring. This has been a perennial issue for the distance learning efforts of higher education and is clearly a challenge for high-stakes certification exams. Kryterion has begun to move in a direction that may offer an answer to the challenge. They have developed a system whereby an examinee can be in a room with a web video camera and using two biometric tools, the individual is validated and centrally monitored as they take the exam. The monitoring take two levels at the central office. The first level is looking for alerts that are triggered either manually (the monitor sees someone in the room with the examinee) or automatically (examinee’s response patterns violate statistical norms). When the alert is triggered, the first level monitor hands the examinee off to a second tier who resolves the situation according to the test sponsor’s specifications. There are many issues and questions raised by this system, however most that I heard at the session and in later conversation were answerable.

ITS presented a session at the recent 2008 Association of Test Publishers conference in Dallas, Texas, US. The session demonstrated their capabilities related to diagnostic or formative assessment. ITS has created customized sites for MCAT and TOEFL that allow the user to create a user ID and build a learning environment that allows the individual to iterate through the practice exam and receive detailed reports that are easy to use in building one’s competence and confidence going into the related exam. One of the most interesting capabilities was in the presentation of case or reading exam items. They have developed a system the easily imitates some of the advantages of paper-based assessment. The ability to mark passages, cross-out distracters and otherwise manipulate the assessment tool in an effort to communicate mastery of the skill or knowledge being assessed.

While attending the 2008 Association of Test Publishers conference in Dallas, Texas, US, I was able to attend a session where three leaders from the online assessment vendor community discussed top-of-mind issues. David Foster, CEO of Kryterion, Patrick Ward, President of Internet Testing Systems, and Eric Shepherd, CEO of Questionmark presented their thoughts.

David Foster began with a call for testing at any time or place. He discussed the concept of changing business models within the context of standards. In addressing the question of assessment standards, David made a clear case that methods are not the same as standards. He also argued that the status quo does not constitute standards and it is the status quo that inhibits innovation, not standards. His observation was that as an industry, the assessment industry is too locked into the known and safe and as such the industry is finding it difficult to move outside the box.

Patrick Ward took the role of prognosticator and attempted to make a few observations about the nature of assessment over the next ten years. He argued that Internet-based testing (IBT) will become the primary method of delivery, that we will see more low-cost off-the-shelf solutions. He observed that we are at a point where the major players will need to evolve quickly as new leaders in the business may emerge. He observed a move from multi-vendor relationships with test sponsors and that this will require outstanding service on the part of the sole vendor. While he sees efforts for innovative test items continuing, the cost will continue to be high, so traditional test items will continue. He sees an increase in adaptive testing in order to provide shorter assessments and that these assessments will move more to the diagnostic or formative side of the learning process. Patrick sees the infrastructure of the Internet becoming a non-issue over most of the world. Finally, he sees a growth in certification, a need for rigorous psychometrics and a move of the test center function back to the test sponsor.

Eric Shepherd discussed the Assessment Maturity Model.

I think David and Patrick together build a picture that works well. As we move increasingly into a digital wealth system, we have increasing need for more granular learning and that learning has to weave together to demonstrate mastery of the customized domain. If a professional works to learn a new topic, it is important that they receive some validation of their learning and track that against the full mosaic of their experience (resume) and education (full and granular learning transcript). In our older models, we have gradually moved from the largest and most concrete domain represented by a university degree, and moved to professional certifications that validate a smaller, but still large domain of knowledge. The next move seems to be the move to even smaller domains.