Pages

Showing posts with label design evaluation. Show all posts
Showing posts with label design evaluation. Show all posts

11.22.2008

On collective non-intelligence

Ken Allan, aka Blogger in Middle-Earth, left a provocative comment after reflecting on yesterday's post: According to design dictates. I'm republishing it here so subscribers who may have missed will get to ponder Ken's wonderful insights about collective non-intelligence and our individual relationship to design dictates (dna).
For a long time (and it continues) humankind has followed dictates. Oh yes, the form of the dictates has changed over the millennia, but never-the-less the dictates have called the shots.

Some last for a few years, some last for decades, some live out centuries before the dictates are overturned by some sort of knee-jerk reaction by (human) society. I think that it's in our dna.

You have recognised a feature of the effect of dictates. It's not particularly exciting. It simply dictates the status quo.

How often has society waited for that wonderful moment when religion or science or political inertia is about to announce a revelation that brings hope into the arena, only to find that the wait was a waste of time, and energy in hoping?

Somehow our dna prohibits humankind from behaving intelligently en masse. The collective intelligence we hear and read about never puts on its thinking cap when it's really needed.

Yet it can move swiftly and as deftly as a shoal of fish in following fashion and things seemingly trivial compared to the perceived real need for shifts in society. We've only to look at political choice of a nation.

Oh, I'm not talking about the present moves in elections. Politics has shifted under the influence of dictates for centuries like a pendulum.

As agile as it moves, the pendulum has its own inertia, never finding the balance, never resting in equilibrium. Never learning from its own mistakes. Yet at its centre is a need to solve a problem of sorts.

It is (in fact) like a collective non-intelligence. It's the case in point where the whole is NOT greater than the sum of its parts. Far from it.

History gives a fine reflection of how it works. They say we never learn from history. That saying has been around long enough. Yet we still don't. We have never learnt to learn from what we see as a blatant lesson for society.

No. Humankind doesn't think like brains do. How silly to think that the collective motion of millions of intelligences is not intelligent - as we perceive it. Not like bees. A bee seems to have a residual intelligence. But the swarm seems to have a mind of its own.

Maybe it's just the way individuals think. Maybe, in fact, the real intelligent way to move is how humankind moves and has been moving for centuries - despite the intelligent opinion of individuals on how it SHOULD move. So dictates may form a major part of that. Who knows?

Perhaps the dictates shold be revered more than they are.

11.21.2008

According to design dictates

We always design according to the dictates of the problem were solving. We can redefine the problem or solve a different problem, yet never escape the dictates of the problem we decide to resolve. The inevitability of dictates puts many designers into a state of resignation, denial and self-imposed limitations. It appears there is nothing to do about dictates since they are inevitable and seemingly inescapable.

Some examples of design dictates can be found in what I wrote about incubating a wise decision, freedom via messing around, what happened to you?, and believing in school work.

The effects of design dictates on designers give us "business as usual", "more of the same changes" and "sustaining innovations". Design innovations maintain factory models of organization, command & control methods of management and centralized delivery systems. The system appears to be too vast, complex and costly to fix. The repercussions of ill-conceived solutions are too extensive, damaging and enduring. As far as we can tell, it pays to play by the rules and do what's worked before.

We use metaphors and analogies to allude to the possibility of changing design dictates since the change seems foolhardy or impossible. Here's some I encounter in the blogosphere and my incessant reading of books:
  • Dictates as DNA that calls for genetic mutations to functionally adapt and survive in transformed environments
  • Dictates as deep structure that calls for delving below surface rationalizations, characterizations and justifications
  • Dictates as a dominant narrative that perpetuates it's power over its pawns until a preferred narrative gets authored by it's victim
  • Dictates as a consensual delusion that can be dispelled by making a more objective and realistic appraisal of the situation
  • Dictates as fallout from success with established mechanisms that will fade away naturally when disruptive innovations take hold
  • Dictates as underlying belief systems that get revised by experiences with new ways of succeeding, winning and conquering challenges
  • Dictates as blind spots in the focused awareness of possibilities where panoramic vision can be restored with a shift of attention
When we experience dictates being changed from under our feet, we say things like "what was I thinking?" or "now I get it". Our change of mind is unlike acquiring added knowledge, data or confirmations. We see everything differently according to the new dictates. Threats look like opportunities. Constraints offer new freedoms instead of familiar obstacles. Past history provides valuable lessons instead of regrettable losses, setbacks and embarrassments. The lofty goal looks achievable instead of unrealistic or overly idealistic. A change in dictates creates a life-changing incident.

11.20.2008

Coping with added criteria

When we're designing anything, we usually fall in love with our best solution. We get attached to how well we've resolved the issues, made the tradeoffs and combined some partial solutions into our favored design. We embrace the assumption that we will end up with the best outcome if we stop messing around with what we've come up with at last. We're usually afraid that will throw out the baby with the bath water if we change the design. Considering an added criteria can be very upsetting for these reasons.

Added criteria usually make a design better. It becomes more responsive to it's larger context. It make accommodate more stakeholders It may deal with long term issues in a better way. It can possibly serve the interests of customers/patients/members/students with less expense. Added criteria can even reveal some creative options that had not been considered thus far.

Choosing to reopen the design process can polarize the participants. Some may regard it as a threat as I explored above. Others may see it as an opportunity to do a better job. It depends on where each person sees her or himself. Anyone on the ascending side of life cycle curve will see progress ahead and no place to go but up. Those on the cusp of the curve will be wary of losing ground, entering a slippery slope and going past the point of diminishing returns. Those on the downside will be seeing everything going from bad to worse, catastrophizing about the future and foretelling a series of unfortunate events.

When there is resistance to making a design more responsive, responsible, respectful and resourceful, consider where people are at in their lives. It may not the added criteria that's at fault.

11.19.2008

Multilingual designers

Whenever we design a new course, game, community or business model, we are speaking a pattern language. We use a vocabulary of components we assemble into coherent expressions. We expect our students, players, members or customers to comprehend what we've said in this language. We anticipate they will speak the same language and comprehend our intended meaning.

Any pattern language defines how to evaluate designs. The language defines what is useful and useless, coherent and confusing, or worth keeping or in need of more work. We use the language to comprehend the solutions we've created and anticipate what else to improve about them. We think in terms of the components we're "speaking with" as we conceive of more alternatives to generate and criteria to apply to those innovations.

When we're fluent in a language, we take it for granted. We forget what the language assumes, dwells upon, leaves out and over-emphasizes. We think in the language as well as speak it. We don't realize how we've limited ourselves by the language we've adopted or how it compels us to go to particular extremes. We replicate the premises of the language as we think, communicate and act on the language's assumptions.

As designers, we need to be multilingual. We need to speak the language of SME's if we're sharing their expert content with others. We need to speak the language of natural environments, ecologies and habitats if we're proposing new invasive structures. We need to think in the same terms as school dropouts if we're going to reach them with supportive opportunities. We need to use the same vocabulary as potential members if the new community will be appealing for them to join.

Knowing more than one language makes us aware of the idiosyncrasies of each one. We realize how we cannot find a word that conveys the same meaning or conveys a complex idea as simply. We see how patterns in one language are excluded from another. We understand why things happen because of the pattern language that has been used to generate new ideas and evaluate the designs.

11.18.2008

Evolving design criteria

Whenever we evaluate what we're already using to get a job done, we get ideas for something better. The way we see the status quo can limit or extend the possibilities we consider. This applies to both designers of new things as well as their users.

Printed pamphlets, books and newspapers once had their type set by hand. These skilled craftspeople are called typesetters. Some excelled at this craft. They became faster and more efficient at getting the right letters placed in the right order with the right amount of spacing to line up evenly on both margins. Their exceptional conduct defined some design criteria when the typesetting process became mechanized.

The first generation of machines replicated how the best typesetters worked their craft. Mark Twain went bankrupt funding this venture. A second generation design redefined how the work of typesetting got done. The Linotype process eventually became the industry standard. Different design criteria were invented in this process that changed how to judge which ideas were obsolete and which could be improved upon.

A similar evolution occurred with computer printers and fax machines. The first generation of dot matrix and daisy wheel printers used ribbons like manual and electric typewriters. Laser printers with toner cartridges and PostScript printer drivers reinvented how printing occurred. Ink jet printing and other font systems opened the field to many lower cost printers. We now often consider not printing something out if its more accessible, searchable, archive-able and replicable when kept onscreen, online or on disk. Our criteria for designing new software tools and evaluating the use of printers have become more sophisticated.

Users also change the criteria they use to decide what they want to buy, what will be useful to them and what is worth the purchase price. Automobiles were originally called "horseless carriages". They were evaluated as a different kind of horse. Once they no longer had manual cranks on the front of the crankcase, they were called automobiles in reference to the automatic ignition system that simply required "turning a key". As their usage became more widespread, nicknames entered into our vocabulary. Carriages became cars and automobiles became autos. Now we speak of " a nice ride" as if what a car does for us has superseded what it is.

Online instruction, communities and reading materials mostly appear to replicate what goes in classrooms. We're still at the stage of first generation typesetting machines, changing printer ribbons and thinking of cars as horseless carriages. Our design criteria are defined by the previous ways we got the job done. However, the design criteria we are using are also evolving. A second generation of models, tools and uses is on the horizon.

11.17.2008

Conflicting design standards

Whenever we're designing a course, business model, community or game, we're apt to encounter conflicting design standards. We get put in the position of a traitor, saboteur or whistle blower if we defy the consensual basis for design evaluation. Our conscience or empathy for the customers may tune us out to the accepted criteria. We may identify with emerging changes instead perpetuating the status quo. Here are four typical conflicts that appear in design standards:
  1. In markets where products are getting commoditized and prices charged to the customer are falling, the accepted standards favor cheapening the product. Economies of scale are bringing down the cost to fabricate, deliver and distribute the goods or services. Competition is based on price and cheaper offerings increase sales volume. Design standards become conflicted for employees who care about the actual quality of what is sold, long term impacts of purchases or useful functionality of added features.
  2. In markets where products are getting differentiated by value-added features and benefits, the accepted standards favor enhancements to products, service after the sale, and customer relationships. Sustaining innovations are rewarded with increased revenue and market share. Competition is based on superior options, add-ons and premium package deals. Design standards become conflicted for employees who are wary of feature creep, making products too complicated to benefit from, or overcharging customers for functions they will never use.
  3. In markets where purchases are made for show, higher prices add to the glamour, prestige and perceived value for customers. The accepted deign standards favor overcharging customers, burying hidden charges, and creating the illusion of cost savings with incidental discounts. Competition is based on brand names, visibility, buzz and celebrity endorsements. Design standards become conflicted for employees who value honesty, personal integrity and transparency in their dealings with customers.
  4. In markets where the costs of production, distribution and customer transactions are soaring out of sight, the accepted standards favor cutbacks in service while customers endure price increases. Competition is based on creating captive markets, local monopolies and perishable inventories. Design standards become conflicted for employees who want customers to freely choose what they buy, gain access to other alternatives and define value on their own terms.
These conflicted design standards rarely get resolved within an incumbent enterprise. More often, the conflicts provide the rocket fuel to launch an innovative start-up. The application of the alternative design standards may change the game, disrupt the incumbent and define a new market space. Customers will see the advantage of patronizing the upstarts that adhere to different design standards.

11.14.2008

Mixing up the rubrics

People who take things literally think as class has to be evaluated by rubrics for classroom education. To their way of thinking, there's no point in evaluating a course design as a forest, amusement park or boat ride. In their frame of mind of utilizing a rubric taken at face value, it's not even conceivable how to evaluate a forest. Our left brains not only takes things literally, they compartmentalize everything. An instructional design must have nothing to do with resilient ecosystems, thrilling adventure rides or floating in a wide range of possible directions.

Just as coming up with a design calls for creativity, so does the evaluation of different design alternatives, combinations and improvements. Once we're utilizing our right brains creatively, we not only can consider how to evaluate a forest, we can utilize those frames of reference to size up a class. Mixing up the rubrics like that delivers better insights into a design possibility and more ways to upgrade the design itself.

Here's some possibilities to get you thinking about mixing up the rubrics for the evaluation of instructional designs:

What if the course design provides a business model? How well does the instructional design deliver what has been sold? How satisfied are the customers with what they bought? How consistent and reliable are their purchases when they get used by the customers? In what circumstances do the customers experience the value of what they bought? What word of mouth advertising will the customers give others who seem interested in their purchase?

What if the course design organizes a community? How well do the participants get to know each other? How committed do the learners get toward helping each other through difficulties? How much do the members of the community do to cultivate mutual respect, high levels of trust and compassion for each other? How does the sense of purpose within the community deepen as they accomplish results together?

What if the course design functions as a multi-player game? How many levels can be reached by mastering the challenges in the course? What obstacles require collaborative efforts to conquer adequately? What avenues are open to execute a flawed strategy and pay a penalty? What momentum can be built up by successive victories, conquests or progress? How has balanced been achieved that steers clear of making the game too easy/obvious and too difficult/subtle?

Perhaps these alternative rubrics give you a sense of their potential impact. Instructional designs could become more creative. There would be more learner-engagement by captivating their imaginations, sense of adventure and curiosity. The learners would feel more support, validation and encouragement from the interactions. The course would pose some mystery and suspense rather than be blatantly straightforward and boring. The take-away value would transcend compliance with the learning objectives. Retention would exceed baseline measures. Upgrading the evaluation schema could deliver better designs in the end.

11.13.2008

Considering potential losses

We cannot evaluate designs regardless of context without getting into lots of trouble. Situations vary in how much it costs to fail, fizzle out or flop after a brief honeymoon. When losses are insignificant, we can let design evaluations occur by what happens next. When losses are costly, a more formal approach challenges assumptions before placing any bets. By being context sensitive, we can take the right approach and realize the maximum benefit from critiquing our designs.

Negligible losses: When there is very little to lose, we can "just do it" and "bet the ranch". Clay Shirky gave us a wonderful exploration of what comes about when we are "below the floor" of established organizations' limits to what they can afford. Here Comes Everybody shows us how we can take risks and see what comes of our experiments. This approach is evident in creating so much new content for public access that may or may not get any attention, viewers, links, saves, bookmarks, tags, Diggs, downloads or comments. This pattern shows up in launching new groups in social networking sites like Meetup or LinkedIn or new candidates entering the race for public office. The initiator is free to "run it up a flagpole to find out if anyone salutes it". The design evaluation gets Crowdsourced for free and relies on Outside Innovation to filter out the least viable alternatives. The online public becomes the judge and jury. The process resembles a decision market that evokes the Wisdom of Crowds. A design is sized up to be as good as the effects it has on the diverse audience.

Acceptable losses: When losses are a cost of doing the business of innovation, rapid prototyping provides better design evaluations. This context recommends the approach taken in Serious Play, The Elegant Solution and The Art of the Long View. Toyota's development of the first Lexus luxury automobile involved 900 engine prototypes and 450 test models. This obviously takes prototyping to the extreme to support the Lexus brand for "the relentless pursuit of perfection". Less costly development of multiple schemes, models or scenarios can improve design evaluation dramatically. Rather than judge a single design on its merits in isolation, designs get compared, contrasted and combined. I know from my own experiences that evaluating multiple designs together generates more innovations. I see how to achieve the best of both with a winning combination as The Medici Effect champions. It suddenly becomes obvious how to do something more simply or effectively. Insights arise from how two designs differ and have other facets in common. Ideas come to mind as it becomes more obvious what the crucial tradeoffs are, what has to be kept in balance and what needs to be given less importance.

Avoidable losses: When "an ounce of prevention is worth a pound of cure", a formal design review process is in order. The design is put before an expert panel, focus group of typical users, or a jury of peers. A variety of design principles, criteria and frames of reference get applied to the design scheme before making the big investment. Caution is applied to select appropriate rubrics prior to making evaluations. This process is used in the giving the green light to new Hollywood film budgets, architectural construction projects and legislative spending proposals. Long range consequences get considered. Immediate impacts on various constituencies, environments and societal trends get raised. Return on investment and value engineering criteria seek to justify the massive expenditure.

Inconceivable losses: When losses could be devastating to the entire enterprise, design evaluation goes into denial. Responsible individuals run their "deer in headlights" number. The current design is pursued regardless of consequences, impacts and fallout. The fear of making a mistake eliminates consideration of better evaluations. Sticking to "business as usual" appears to be the only viable choice through the tunnel visioned look at options. This is the approach taken by corporate excesses, industrialized agriculture, the fueling of catastrophic climate changes and countless professional practices which do more harm than good.

Good luck creating a situation with negligible or affordable losses for your creative designs!

11.12.2008

Upgrading design evaluations

Today I'm embarking on a new series of blog posts about improved approaches to design evaluation. Recently, I've been designing facets of learning communities that disrupt the labor model relied on in classrooms. As I develop ways for learning to happen collaboratively, without instructors, and enriched by Web 2.0 tools, I've wondered how best to evaluate these designs. I'm on familiar ground to be designing schemes and the schemata for evaluating them at the same time.

As I've been designing learning community functionality, I've realized there are four kinds of designs that have lots in common:
  1. Community designs to get more members, commitments, interactions and contributions
  2. Enterprise designs to get more customers, loyalty, purchases, and word of mouth advertising
  3. Instructional designs to get more retention, motivation, implementation and follow through
  4. Game designs to get more engagement, exploration, persistence and experimentation
A design that seems like it will get a lot of self-directed learning to happen, can also be critiqued as a game, business model and ongoing social interaction. Besides those very current frames of reference, my thinking about design evaluation is based on my wide range of previous experiences as a designer.
  • When I've designed houses, group facilities or office buildings, I've been concerned with how the users would respond to the spaces and juxtaposition of functions.
  • When I've designed productions for the staging of live theater or videotaping, I've considered how the unfolding narrative is conveyed by the visual setting and environmental cues.
  • When I've designed activities for after-school programs, children's museum exhibits and arts center programs for elementary students, I've been concerned with how much fun the kids are having as well as how contagious my own enthusiasm is for them.
  • When I've designed publications, workbooks and flash-cards, I've considered how accessible the ideas appear to be and how the organization contributes to the users' comprehension.
  • When I've designed college courses, corporate training sessions and professional workshops, I've been concerned with avoiding useless activities that provide no take away value or impact beyond the session.
  • When I've designed performance appraisals, organization change models and interventions in troubled mergers, I've considered how to get others to think for themselves and rely on their inner sense of direction.
  • When I've designed executive planning retreats, strategy formulation sessions and processes for revitalizing businesses, I've been concerned with handing over the controls so momentum in not lost when the gathering ends.
With such a diversity of design experiences, I naturally see patterns in the varied contexts and outcomes of the design methods. The approaches to design evaluation I'm formulating can become more robust and reliable by including these patterns realized from my experiences.