Chapter Three:   Research  Design  and Findings

           This   chapter  examines  those   language   learning  
     activities which might be applicable to a computer-assisted  
     language learning environment and which could be reproduced  
     using an authoring system.

          As an introduction to the data base for this study,  a  
     definition  is suggested for the type of activities  to  be  
     examined  with an analysis of their nature.  The method  of  
     gathering the data is explained and the data is presented.

           The  data  can  be  separated  into  two  categories,  
     activities  used  to  exemplify  an  analysis  of  language  
     learning  activities from an operational point of view  and  
     those  used  to  exemplify an analysis from  a  pedagogical  
     point  of  view.     The  data  pertaining  to  operational  
     typologies  is  examined  first and  the  results  of  this  
     examination are subsequently applied to the data pertaining  
     to pedagogical typologies.

           The  aim  of  the first part of the  analysis  is  to  
     establish and codify the notion of "flexibility" introduced  
     in  Chapter Two by isolating those operational  factors  of  
     the  activities  under  discussion which  might  affect  an  
     authoring  systems's  ability  to  reproduce  them  on  the  
     computer.   The three distinct,  but related, problem areas  
     cited  by  Kearsley in section 2.5.2 are   examined:    The  
     presentation  of  discrete  interactive  elements  of   the  
     activity to learners,  immediate feedback to learners based  
     on  an  analysis  of  the their  response  to  the  stimuli  
     offered,  and the global decision as to how to continue the  
     activity with subsequent interactions. 

           In  the second part of the analysis the  relationship  
     between    operational    flexibility    and    pedagogical  
     effectiveness   is  examined.    The  discrete  pedagogical  
     factors used to categorize activities  are isolated and the  
     interaction between the key operational factors established  
     and these pedagogical factors is determined. 

     3.1   CALL and its role in the language learning process

               In  section 2.1,   a working definition  of  CALL   
     was  developed  which excluded from the discussion uses  of  
     the  computer  as  a  tool or  as  a  passive  medium,  and  
     restricted  it  to  considerations of the  computer  as  an  
     active   participant  and  interlocutor  in  the   language  
     learning  process.  The discussion was further  limited  to  
     uses  of the computer for language practice,  a point which  
     will now be considered in further depth.

               "Instruction"  ultimately  derives from  a  human  
     resource  person,  normally the teacher.   As teachers  and  
     other  qualified  resource  persons tend  to  be  in  short  
     supply,  non-human  surrogates  which are derived from  the  
     teacher   must  often  be  utilized.    Historically,   the  
     textbook  has  been the most widely used stand-in  for  the  
     teacher.  More recently, other more mechanical substitutes,  
     such as language labs and computers,  have been  integrated  
     into the students' learning experience.

               No  mechanical aid can ever completly replace the  
     sensitivity to learners' varying needs which human teachers  
     possess (Byrne,  1976,  p.1).  The most that one can expect  
     of  a  mechanical  surrogate  is  that  it  model,  however  
     imperfectly,  some of the minor roles of the human teacher.  
     A  book presents information in a static manner,  may  pose  
     questions, and can even provide a model correct answer.  It  
     cannot,  however,  modify the presentation of  information,  
     nor  actively correct a learner's particular  errors.   The  
     language  lab offers the same capabilities and suffers from  
     the same restraints as the printed page; it simply provides  
     the  sensory  experience  via the spoken  rather  than  the  
     written medium.  The computer, on the other hand, is a much  
     more sophisticated device.  It can make judgements and thus  
     has  a greater potential to model the role of the  teacher.   
     Byrne's caveat still holds, nevertheless.  The computer can  
     serve as a surrogate teacher only in those areas where  the  
     full  scope of the human teacher's sensitivity and subtlety  
     are not called into play.

               The  language-learning  process has  been  broken  
     down  into three  segments:   presentation,  practice,  and  
     production (Dakin, 1973, p. 4; Byrne, 1976, p. 2).

                In the presentation phase, new material is given  
     to  learners,  either deductively through demonstration  or  
     inductively through example,  which the learners assimilate  
     through  their  own involvement in  the  discovery  process  
     (Dakin,  1973,  p. 4).  The  learners'  problem is that  of  
     understanding (Dakin,  1973,  p. 6).  The teacher's role is  
     that  of  informant,   carefully  selecting  material   and  
     choosing  a  method  for  presenting it  in  as  clear  and  
     memorable way as possible, based on a thorough knowledge of  
     the  language  and a feeling for the  learners'  individual   
     needs  (Byrne,   1976,   p. 2).   Computers  serve  in  the  
     presentation phase only in the same way as books,  language  
     lab  tapes,  films,  and other media can serve - as passive  
     presenters.   To  serve as even an imperfect surrogate  for  
     the teacher,  the computer would require a knowledge of the  
     language and a way to gauge the subtle individual reactions  
     of  students,   which  is currently beyond our  ability  to  
     codify,  let  alone program.   The computer does not  seem,  
     therefore,   to   offer  any  special  advantages  in   the  
     presentation stage of language learning.

               In the production stage,  control over  students'  
     performance  is  relaxed  and students  produce  their  own  
     language (Dakin, 1973, p. 4).  Now the learners' problem is  
     that of communication (Dakin, 1973, p. 6) and the teacher's  
     role is that of guide.  Students are directed towards using  
     the  language freely,  to express their own ideas in  their  
     own  words  for their own  purposes  (Byrne,  1976,  p. 2).      
     Of all the support media, the computer alone seems to offer  
     the  potential  of  actively participating at  this  stage.   
     Ventures   in this area,  such as the program "ELIZA", have  
     been   only  experimental,   however.    Current   computer  
     programs,    developed   from   scratch   by   professional  
     programmers,   are ineffective in  matching the  linguistic  
     abilities  of even the least advanced learners,  let  alone  
     acting  as their guide.    Little immediate application  is  
     therefore  seen  for  authoring systems in  the  production  
     phase of language learning.

               The  practice  stage falls in between  these  two  
     extremes.   Learners  are called upon to produce pieces  of  
     language   in  a  carefully  constructed   and   controlled  
     environment (Dakin,  1973, p. 4), that can be duplicated on  
     the computer.   The learners' problem at this stage is that  
     of  remembering and applying what they have learned (Dakin,  
     1973,  p. 6),  but not the communication of original ideas,  
     which the computer could not handle.

              Byrne  (1976,  p. 2) likens the teacher's role  at  
     this  stage  to  that  of  an  orchestra  conductor  .   In  
     expanding  Byrne's analogy,   the conductor can be imagined  
     as   serving  two  roles.    First,   the  conductor   must  
     orchestrate   the  music,   constructing   the   controlled  
     environment.   This  is  the role of the teacher/author  in  
     designing  a language practice activity,  a role where  the  
     computer cannot serve as surrogate.  The second role of the  
     conductor  is  to  guide the individual  musicians  in  the  
     reproduction of the orchestrated music,  the  corresponding  
     role  of the teacher being to lead students in using pieces  
     of  language within a  controlled  interaction.   Students'  
     responses  at  this stage are likely to be  unexpected  and  
     confused,  to  the  point  of causing a  breakdown  in  the  
     interaction.   The  teacher  must  be  sensitive  to  this,  
     restructuring or temporarily interrupting the course of the  
     interaction  with  feedback and guidance so as  correct  it  
     (Dakin, 1973, p. 4). 

               Dakin  (1973,  p.7) considers that even the  more  
     rudimentary  technology  represented by  the  language  lab  
     could  assume some of the roles of the teacher within  this  
     prescribed, controlled environment.  The computer, with its  
     far  greater  capabilities,  though still feeble  by  human  
     standards, could be expected to play a much greater part in  
     assuming the teacher's role. 

               While   interest  recently  has  focused  on  the  
     production  phase of language learning,  the necessity  and  
     importance of the practice stage is underlined by a variety  
     of authors (Stevick,  1971,  p. 391;  Byrne,  1976,  p. 32;  
     Robinett  1978,  p. 206).   Learners need focused  practice  
     where they can build up confidence,  because they are given  
     something   to  say  in  an  environment  where   confusing  
     distractions are kept to a minimum (Byrne, 1973, p. 32).

     3.2   Forms of language practice

           In  the  general context of  language  learning,  any  
     verbal  activity  whatsoever could be considered  practice.   
     The  definition of CALL outlined in Chapter Two limits  the  
     discussion,  however,  to  a  particular kind  of  language  
     practice  - that  which  is  interactive,   structured   or  
     "focused", and mediated.

               Drills and exercises have traditionally been seen  
     as  a  form  of  focused  practice   (Stack,  1971,  p.161;  
     Stevick,  1971,  p.393;  Rivers & Temperley,  1978,  p.398;  
     Robinett,  1978,  p.206).   A  drill allows the learner  to  
     focus  on  aspects of language which are not  normally  the  
     object  of  attention  during normal language  use  or  are  
     seldom repeated several times in a row during the course of  
     a normal conversation (Stevick, 1971, p. 393).

               The   use  of  the  term  "drill"  evokes  strong  
     reactions  among language teachers because of  the  origins  
     and connotation of the term.   Drill has traditionally been  
     closely   identified  with  a  behaviourist  philosophy  of  
     language  learning.   Drills have been narrowly defined  as  
     activities  which focus on grammatical practice  and  which  
     only  allow  for a single correct response by  the  learner  
     (Stevick,  1971,  p.393;  Cook,  1972,  p.121; Dakin, 1973,  
     p.91;  Rivers & Temperley,  1978,  p.122;  Robinett,  1978,  
     p.47).   The content of the drill needs to be unambiguously  
     designed to lead the learner inexorably towards this single  
     acceptable response (Cook, 1972, p.121; Rivers & Temperley,  
     1978,  p.122).    In  order to escape the rigid confines of  
     the  above definition,  some experts have adopted the  term  
     "exercise" to identify those activities where the focus may  
     not be grammatical,  which are less rigidly controlled  and  
     where  more than one response might be acceptable (Stevick,  
     1971,  p.393;  Dakin,  1973,  p.91).  One could escape this  
     confusion entirely by adopting a totally neutral term  such  
     as  "activity".   Activity  is too general  an  expression,  
     however,  and  does not adequately describe the  particular  
     form  of  practice  traditionally associated  with  drills.   
     Indeed,  "exercise" itself suffers from the same generality  
     of connotation.

           The words "drill",  "exercise",  and "activity"  here  
     are   used   interchangeably,   and  divorced  from   their  
     respective  theoretical  connotations,    to  refer  to   a  
     particular  format  for language  practice.  Excluded  from  
     consideration  are  those  formats  identified  as  drills,  
     exercises,  or activities which do not fit the criteria set  
     out below.

          Drills  are universally seen as a form  of  interactive  
    practice.   The  interaction consists minimally of two  parts.   
    The  first  part has been called  the  "stimulus",  "prompt",  
    "cue",  or "input"(to the learner), while the second part has  
    been  called  the "response" or "output" (from  the  learner)  
    (Stevick,  1971, pp. 393-394; Cook, 1972, p.121; Dakin, 1973,  
    p.49;  Rivers & Temperley,  1978,  p.121). 

               Drills  are  seen  as  structured  practice.  The  
     interaction   proceeds  algorithmically  according   to   a  
     predictable pattern (Dacanay,  1967,  p.107;  Rivers, 1978,  
     p.398).    It  is  the algorithmic,  predictable nature  of  
     drills   which  allows  their  presentation  by   non-human  
     interlocutors, such as books, language labs, or computers.

               Drills are mediated practice;  there is a set  of  
     expected  responses  and  the  learner's  response  can  be  
     examined in consequence of this set.  To include the act of  
     mediation,  the  minimal stimulus-response interaction must  
     be expanded to include a third phase,  the reaction to  the  
     learners'  response,  or "confirmation of response" (Stack,  
     1971,  p.126; Rivers & Temperley, 1978, p.121)  Reaction to  
     the   learner's   response,   or  "feedback"   in   today's  
     terminology,  assumes  an  implicit phase  which  might  be  
     called "analysis of response".

               Explicit reaction to each response may be dropped  
     to  foster a more natural interchange (Rivers &  Temperley,  
     1978,  p.121;  Hardin,  1983).    The  teacher will  still,  
     nevertheless,  be analyzing the learners' responses and the  
     global  course of the drill,  the selection of material and  
     the  number  of  interactions,  will be  a  adjusted  as  a  
     consequence   of  the  responses  given.    In   artificial  
     environments,  the  learner is often called upon to make  a  
     self-evaluation,  based on exposure to acceptable responses  
     contained  in an answer-key at the back of a textbook or on  
     a language lab tape.    In correcting the learner's written  
     work,  or in monitoring the language lab  interaction,  the  
     teacher  can  still  make global alterations to  the  drill  

          The computer is capable of adopting the teacher's role  
     as mediator.   It can examine the learner's response, using  
     tools  that  the  teacher has given  it,  and  can  provide  
     immediate  feedback and make global decisions according  to  
     the options that have been programmed.

               Activities  of  the drill  format,  then,   being  
     language practice which is  intereactive,  structured,  and  
     mediated, satisfies the working definition of CALL provided  
     in Chapter Two.

     3.3  The nature of drills

               Drills   can  be  considered  as  consisting   of  
     individual interactions,  or drill "items".   Each discrete 


          1.   the "stimulus" or "prompt",
          2.   the "response" to the stimulus by the learner,
          3.   the "analysis" of that response in terms of a set  
               of expected or acceptable responses,
          4.   the  "feedback" to the response,   based  on  the  
          5.   "branching",  or  a  global  decision as  to  the  
               subsequent  course  of the drill,  based  on  the  
               analysis and other factors.  

           This use of the computer term,  "branching", must not  
     be  confused  with the standard term  adopted  in  computer  
     programming,   where   it  means  executing  any   computer  
     instruction  other  than the subsequent one.   In  computer  
     programming terms,   the presentation of immediate feedback  
     within  a  drill  item might involve  branching  while  the  
     decision as to the next item to be done might very well  be  
     accomplished without any branching at all.   When viewed in  
     terms of the present definition,  the "branching" powers of  
     a  given authoring system  may turn out to be  considerably  
     weaker   than  if  described  according  to  the   standard  
     programming  sense  of the word.  considerably weaker  when  
     viewed with our definition.

               The "prompt" can be further broken down into  the  
     "model",  the piece of language upon which the learner will  
     be working, and the "cue", the element or instruction which  
     tells  the learner how the model is to be used or  modified  
     (Dacanay,  1967,  p.107;  Rivers & Temperley, 1978, p.123).   
     Stages  three  through five of the  model  described  above  
     would  normally  be the domain of  the  teacher.   Ignoring  
     these,  then,  for the moment, a traditional drill could be  
     presented as illustrated below:  

          3.0       Example:  Basic Pattern

          #1   Model:    I go every day.
               Cue:      yesterday
               Response: I went yesterday.

          #2   Model:    I eat every day.
               Cue:      yesterday
               Response: I ate yesterday.

                         (Dacanay, 1967, p.107)

                (Original drills are uniquely numbered according  
               to  the  order in which they were considered  for  
               inclusion  in the data.   There are gaps  in  the  
               numeration  as  not  all drills  considered  were  
               included in the final data.)

               Dacanay  (1967,  p.107)  describes  three  formal  
     variations  on  this basic  pattern:   interlocked  drills,  
     formalized drills,  and telescoped drills.   In interlocked  
     drills,  the response to one item becomes the model for the  

          3.1  Example:  Interlocked Drill

          #1   Model:    He wrote the report for me
               Cue:                   letter
               Response/ He wrote the letter for me

               Cue:      She    

                         (Dacanay, 1967, p. 107)

     In  formalized  drills,  the explicit cue is replaced by  a  
     standard instruction for modifying a series of models:

          3.2  Example:  Formalized Drill

               Instruction/Cue:   Pluralize the noun.

          #1   Model:    I brought the letter.
               Response: I brought the letters.
          #2   Model:    I brought the note.
               Response: I brought the notes.

                         (Dacanay, 1967, p. 108)

     In telescoped drills,  there is no separate,  explicit cue.   
     Rather, the cue is contained within the model: 

          3.3  Example:  Telescoped Drill

          #1   Model/Cue:     Would you like tea or coffee?
               Response:      I'd like tea.
          #2   Model/Cue:     Would you like cake or ice cream?
               Response:      I'd like ice cream.

                              (Dacanay, 1967, p. 108)

               The purely textual representation of drills above  
     is  not  designed to exclude from consideration the use  of  
     other  media.     A   drill  can  be  greatly  enhanced  by  
     replacing  or supplementing  an explicit verbal  cue  with,  
     for example,  a picture.  One illustration can quickly make  
     clear  a  complex  context  which would take  too  long  to  
     explain verbally.   Since a picture is a representation  of  
     reality,  its  use enhances the communicative nature of  an  
     activity. (Dacanay, 1967, p. 152)

           Many  simple  exercises,  indeed,  depend  on  visual  
     stimuli.   In  the  verbal  cueing of a  simple  vocabulary  
     exercise,  for  example,  it would be necessary  either  to  
     paraphrase  the item in the target language or to provide a  
     translation  in  the  learner's  language.    Many   simple  
     vocabulary  items  would  be difficult to  paraphrase  with  
     language  appropriate  to the level  of  the  learner,  and  
     translation  is  only applicable to homogeneous  groups  of  

          3.19.1    Model:    These are -----.
                    Cue:      You see with them.

          3.19.2    Model:    These are -----
                    Cue:                des yeux

     The  use  of  a  simple picture renders this  very  mundane  
     exercise much more interesting and useful:

          3.19.3    Model:    These are -----
                    Cue:      (picture of eyes) 

               The  computer can provide pictures as cues  in  a  
     variety  of ways.    It can draw the pictures on the screen  
     using  its graphics capabilities.  It can  activate  remote  
     devices   such   as  slide   projectors,   videotapes,   or  
     videodisks.   It can prompt the user to refer to "off-line"  
     media,  such as a printed text.   The presentation of audio  
     or "live action" sequences is no different for the computer   
     than the presentation of a single slide or videodisk frame.

           Many authoring systems excel in providing  mechanisms  
     by  which  non-verbal cues can be included.   Some  provide  
     powerful  graphics  editors,   others provide easy  to  use  
     interfaces to peripheral devices such as tape recorders  or  
     videodisks.   A  glance at 3.19 will  show,  however,  that  
     these have little effect on the structure of the exercises.   
     The  cue  is simply the cue,  regardless of whether  it  is  
     merely  a  written one or,  more  dynamically,  a  recorded  
     visual sequence. dynamic filmed one.

    3.4   The drill sample

               In  an effort to collect a sample that would be a  
     representative  cross-section  of  drills  and   drill-type  
     activities,  examples  were  taken  from  drill  typologies  
     produced  by  a  number  of  accepted  specialists.    This  
     approach was adopted in order to establish a wider and more  
     complete  range of examples,  and within  a more reasonable  
     time,  than would result from a search of original  sources  
     such  as actual classroom drills,  textbooks,  language lab  
     tapes, and similar material.

               Table   3.1  of  Appendix  A  lists  the   eleven  
     authorities consulted.  The works span a period of fourteen  
     years  and  represent  the  range  of  language    teaching  
     philosophies,     from    the    behaviourist/structuralist  
     approaches  of  the 1960's to  the  cognitive/communicative  
     approaches of the 1980's.

               In total,  127 examples were finally included  in  
     the  data  base,  as  listed  in  tables  3.2  and  3.3  of  
     Appendix A.   In  examining a work,   an effort was made to  
     include at least one example of the  drills and  activities  
     that  were  being  proposed  in  support  of  the  author's  
     discussion.    In  most  cases the drills  were  explicitly  
     shown.   In a few cases, all or part of the drill had to be  
     extrapolated   from  the  discussion.    Where  there  were  
     numerous examples categorized by the author as being of the  
     same "type",   usually only one was taken.   The exceptions  
     were  when  it  was evident that  two  "like"  drills  were  
     obviously different in features that might prove important.  

               A   preliminary   screening   of   examples   was  
     undertaken  to  exclude  those that did not  fit  into  the  
     parameters  of  this  study.    These  included  activities  
     intended  primarily  for  the  presentation  or  production  
     phases of language learning as well as those which were not  
     of an interactive, structured, and mediated nature. 

               Each author consulted presents the drill examples  
     in a different fashion.   In order to standardize the data,  
     all examples were fitted into Dacanay's uniform frameworks,  
     as  shown  in  examples 3.0 - 3.4.   So  as  to  illustrate  
     possible   relationships   between  discrete   items,   two  
     representative  items were chosen from each drill  example,  
     wherever possible.    In cases where parts of the drill had  
     to  be assumed,  these are enclosed in square brackets  [].   
     The original examples are listed in Appendix A.

               The data consists mostly of oral drills  designed  
     originally  for  classroom or language laboratory use  (See  
     table  3.3).  It may seem odd to use oral examples  as  the  
     basis  for  a  discussion of CALL,  typically viewed  as  a  
     written medium, and there are several points to consider in  
     this regard:

          1.    A  much  larger  and more  varied  selection  of  
     examples can be found in discussions of oral practice  than  
     in  discussions of written practice.    Most of the written  
     drills examined were found either to be essentially derived  
     from  oral drills or to be activites better suited  to  the  
     production  stage  of  language learning  rather  than  the  
     practice stage.

          2.    It  is  assumed  that,  while each  medium  will  
     accentuate  certain  qualities of a  drill,  the  essential  
     nature of different types of drills will be independent  of  
     the  medium used to present them.   Cook (1972,  p.121) and  
     Rivers and Temperley (1978, p.278) support this approach in  
     their  discussions.   Rivers and  Temperley  (1978,  p.276)  
     warn,  however, that some adjustment in the presentation of  
     the  drill  will be required as one transfers it  from  its  
     original medium to another. 

               3.    CALL can be considered a written medium  in  
     the  sense that the learner interacts with the computer  by  
     reading  written stimuli from the screen and responding via  
     the keyboard.   In other respects,  however,  it much  more  
     closely reproduces the environment of the oral drill:   The  
     responses  made by the learner tend to be short,  they  are  
     evaluated immediately and with a subtlety far exceeding any  
     written answer-key,  and some sort of pertinent feedback is  
     provided  to the learner.     The "written" nature of  CALL   
     exercises  can be moderated with a variety of  non-standard  
     output   devices   (tape   recorders,   slide   projectors,  
     videodisks,  or  voice  synthesizers).  and  input  devices  
     (touch sensitive screens,  light pens,  "mouse" pencils, or  
     voice recognition devices),  all of which would have little  
     effect  on the essential nature of the drill.    Table  3.4  
     shows how CALL provides an environment which can be likened  
     to  that of the oral classroom,  the language lab,  and the  
     written drill.

          4.    The one other major teacher-surrogate technology  
     is the language lab.      Drills designed for the  language  
     lab,  like  many written drills,  can be seen to be derived  
     from  oral classroom drills and exhibit nearly all  of  the  
     same drill types.

          This study is,  therefore,   based on the premise that  
     CALL  drills will exhibit the same basic types of  practice  
     activities as one finds in the classroom or in the language  

     3.5   Bases for drill typologies

               The  authorities  consulted have organized  their  
     drill  typologies according to two  main  gradients,  which  
     might    be   called   categorization   by    "operational"  
     distinctions    and    categorization   by    "pedagogical"  
     distinctions.    In general, operational distinctions refer  
     to the operation that the learner must perform in executing  
     the  drill,  while  pedagogical distinctions refer  to  the  
     drill's overall utility in the learning process.  Table 3.5  
     shows that,  on the whole,   equal weight has been given to  
     both schemes,  although some authors exemplify only one  or  
     the  other approach.   The need to examine drills according  
     to  both  dimensions  was underlined by  almost  all  those  
     consulted (See Tables 3.6 and 3.14 for references).

               It  is a premise of this study  that  operational  
     differences  primarily reflect variations in the format (or  
     algorithm) of a drill while pedagogical differences reflect  
     primarily variations in content.   An authoring system will  
     provide for,  and may limit the expression of,  a range  of  
     drill  formats,  but  should not have any influence on  the  
     choice of content.   There may, however, be some aspects of  
     drill  format  which will limit the  range  of  pedagogical  
     expression  and  some aspect of content which will  enforce  
     the choice of a more elaborate drill format.

               Those   drills  used  to  exemplify   operational  
     typologies  were analyzed with a view to  establishing  the  
     essential  factors used to differentiate them.   Table  3.2  
     lists  the  71  examples,  from the works  of  5  different  
     authors,   that were so examined.   The effect of different  
     operational  factors on the possible computerization of the  
     examples were then examined in detail.

               Those   drills  used  to  exemplify   pedagogical  
     typologies  were  similarly  examined  to  establish  their  
     essential  pedagogical features.    Chart 3.3 lists the  56  
     examples,  from the works of 8 different authors, that were  
     so  examined.   The  possible  effects of  the  operational  
     factors  on the expression of the pedagogical factors in  a  
     computerised version of the exercise were then explored  in  

     3.6  Traditional categorization by operational type

               Operational type is the framework by which drills  
     have traditionally been categorized.   Table 3.6 lists  the  
     major  operational  types as discussed by  the  authorities  
     consulted  for this study.   Paulston (1972,  p.131)  cites  
     Johnson  in  postulating two underlying factors upon  which  
     these categorizations are based:  1) the amount and type of  
     restructuring  of  the  stimulus required  to  produce  the  
     response and 2) the amount of information the learner  must  
     bring to bear to produce the response.  

               Table  3.7  correlates the different  terms  and,  
     judging  by the names of the categories alone,  there seems  
     to  be  a  general consensus as to  the  basic  operational  

          A.   Repetition/imitation drills
          B.   Substitution/replacement drills     
          C.   Transformation/mutatation/conversion drills
          D.   (Other non-standard categories)
          E.   Question/response drills
          F.   Translation drills

     Each  author delimits the categories somewhat  differently,  
     however,   and  so what one author might consider to  be  a  
     drill of type X, another might see as a drill of type Y. 

               Repetition/imitation drills are on the borderline  
     between presentation and practice.  The learners repeat the  
     model  given  them by the  teacher,  without  modification.   
     These  instruments allow for practice in pronunciation  and  
     prosody  (Robinett,  1978,  p.47),  but can only be used as  
     presentation  devices  for higher-level  features  such  as  
     grammar and lexicon (Cook, 1972, p.128; Rivers & Temperley,  
     1978,  p.126;  Robinett, 1978, p.47).   The computer is not  
     currently   an   appropriate  instrument   for   evaluating  
     pronunciation.     Repetition/imitation   exercises   must,  
     therefore, be excluded from the present treatment.

               Similar  to  repetition  exercises are  those  in  
     which there is no possibility of error.  Candlin offers the  
     following example, which was not included in the analyses:

          Example 3.83   Example:  Switchboard (B1)

          Instructions: [Choose  an  element from each  list  to  
                         form a complete utterance]

           Model:    A)   Some/A lot of
                     B)   store detectives/store owners/
                          shopkeepers/store managers
                     C)   use
                     D)   TV security systems/Two-way TV/
                          TV-cameras/photo-scan systems
                     E)   to
                     F)   catch/observe/check up on
                     G)   thieves/"old hands"/"regulars"

                    (Candlin, 1981, p.82)

     Any  combination of the elements given produces  a  correct  
     utterance.   With no possibility of error,  there can be no  
     mediation.   Hence,  this  must  be  considered a  type  of  
     presentation  drill and does not qualify under the  present  
     definition as a possible CALL activity.

               Substitution/replacement  drills assume  a  model  
     with  one or more fixed slots,  the content of which can be  
     replaced  by  any word that fulfils a  similar  function  (  
     Rivers, 1968, p.101-102; Dakin, 1973, pp.49-51).  Different  
     semantic  elements  are practiced within the  same  pattern  
     (Stevick, 1971, p.394). 

           There  is  a  variety  of  sub-categories  under  the  
     heading   "substitution".    These  depend  essentially  on  
     factors  such as whether the substitution slot is given  or  
     the substitition causes a correlated change to the  element  
     in another slot.  Although the deciding factors are more or  
     less   standard,   the  resulting  subcategories  are   not  
     standard,  as a glance at the authors' characterizations of  
     the  various drills listed in table 3.2 will show,  nor are  
     they applied in the same way.   River's simple substitution  
     (3.21),  for example,  would be considered by Dacanay to be  
     a correlative substitution.

          Whereas in substitution exercises a pattern is  given,  
     in transformation/mutation/conversion exercises the pattern  
     itself  is  what the student is asked to reproduce  (Dakin,  
     1973,  pp.49-51).   The same semantic elements are used  in  
     different patterns or surface realizations (Stevick,  1971,  
     p.394).   Transformation  basically practices syntax (Cook,  
     1972,  p.128; Rivers & Temperley, 1978, p.130).  Any change  
     in  word-order  can  be  considered sufficient  to  make  a  
     substitution   exercise  into  a  transformation   exercise  
     (Stack, 1971, p.144). 

          Rivers  (1968,  p.101-102) originally considered  that  
     the   use  of  substitution  vs.    transformation   drills   
     reflected  differences in basic approaches to the  analysis  
     of  syntax,  with  transformation exercises  stemming  from  
     transformation  grammar.     However,  in  a  later  report  
     (Rivers & Temperly,  1978,  p.130) she explicitly uses  the  
     term  "conversion" so as to differentiate the drill  format  
     from  any  relationship to underlying  theories.   Robinett  
     (1978,  p.50),  too,  warns that "although  (transformation  
     drills)  sometimes resemble the manipulative operations  in  
     certain  rules  of transformational grammar,  they are  not  
     meant to be equated with such rules."

           There   is   a  divergence  of  viewpoints   on   the  
     identification   of  subcategories  withing  transformation  
     exercises,  similar  to  the disagreement  on  subtypes  of  
     substitution  practice.  What  some authors consider to  be  
     subcategories are treated by others as separate categories,  
     hence the D.  entry in table 3.7.   All of the distinctions  
     involve four possible basic factors,  often in combination:   
     a change in word order,  the insertion of new elements, the  
     deletion of existing elements,  or the substitution of  one  
     element for another.

               With  question/response drills,  the  operational  
     definition  becomes obscured by pedagogical considerations.   
     Dacanay (1967,  pp.133-134) sees response drills as  having  
     two   possible  functions,   the  exchange  of  information  
     (pedagogical)  or  the  practice  of  the  grammar  of  the  
     response (operational).  Rivers and Temperley (1978, p.143)  
     feel  that  response drills must  concern  themselves  with  
     information exchange and not grammatical forms so that they  
     are  not  reduced to conversion drills.   In example  3.37,  
     however,  they cite,  as  the purpose of  the  drill,  the  
     elicitation of the desired tense (Rivers & Temperley  1978,  
     p.  146),  clearly a grammatical consideration.  Robinett's  
     response  drills  concern themselves almost  entirely  with  
     grammatical practice (Robinett, 1978, pp.52-53).

           There  exists  no  clear  and  accepted   operational  
     definition  of  response  drills.    Stack  (1971,   p.134)  
     mentions  that  these require the introduction of  material  
     not  present in the stimulus.   There is also  the  "you-I"  
     conversion.   The  question  model  "Are you going  to  the  
     movies?"  could be transformed into the statement "You  are  
     going to the movies",  but would have to be responded to as  
     "I am going to the movies".

               Translation  will  involve,  of  course,  lexical  
     substitution  and ,  very probably,  some changes  in  word  
     order.   From  an  operational point of  view,  translation  
     drills  could  be  viewed  as  a  complex  combination   of  
     substitution  and  transformation.     Translation,  as  an  
     operation,  differs  from the others operations in  content  
     alone; a factor not being considered at this point.

           For the purposes of the present study then,  the list  
     in  table 3.7 can be reduced to  three  major,  traditional  
     operational  distinctions:   substitution,  transformation,  
     and  response.    It will be seen that these categories  do  
     not  immediately correlate with the realities that must  be  
     considered in reproducing the various drill examples on the  
     computer, where one must consider "operation" from a formal  
     or   "algorithmic" viewpoint rather than from a  linguistic  
     one.    The  lack of general agreement as to the scope  and  
     application   of  each  term,   and  the  myriad  unrelated  
     subcategories,  make  it difficult to use  the  traditional  
     categories   as  a  point  of  departure  for  the  current  

     3.7   Drill categorization based on cue and response-model

               Those   drill   examples   used   to   illustrate  
     operational typologies were analyzed from the point of view  
     of  the  actual  mechanical  operation  taking  place,   an  
     approach  which differs from the traditional analysis of  a  
     drill  in terms of the linguistic relationship between  the  
     stimulus and the response.    Cook (1972,  p.122) offers  a  
     basis  for this alternative approach to analyzing a  drill,  
     where  a basic framework for the response is postulated and  
     the  changes that the response undergoes between one  drill  
     item  and the next is examined.   The resulting  desription  
     parallels  much  more closely the  algorithmic  description  
     required  to prepare a drill for for possible  presentation  
     on the computer, than does the traditional analysis.  

               In the analysis  shown in table 3.9 of appendix B  
     and summarized in table 3.8,   drills are examined in terms  
     of the cue (C:), that part that actually triggers a change,   
     and  the model  (M:),  where "model"  is now re-defined  to  
     mean  the  "response-model"  or  basic  framework  for  the  
     response.    In those cases where drills could be  analyzed  
     in more than one way, the way that seems most productive in  
     terms of the question of computerization of the drill,  has  
     been adopted.

               A  major division between "simple"  drills (those  
     drills   which  can  be  accomplished  with  one   discrete  
     operation)  and "complex" drills (those which require  more  
     than one simple operation) is evident.   Rivers & Temperley  
     (1978, p.125) calls the latter "mixed" drills, a term which  
     is here employed in a more restrictive sense.

          The fundamental operations identified are:

                 I) "insertion" of an element into the model,
                II) "removal"  of  redundant elements  from  the  
               III) "re-order" of model elements,  and
                IV) "production" of new elements,  based on  the  
                    cue   alone  (where  the  response-model  or  
                    models are implicit).

     I.         Several different types of insertion are evident:
     Type A:    where the primary operation is  the insertion of  
     the  cue into the model.   This can be shown  schematically  

                    C: -->M:[]
                    C:   stands for the cue
                    M:   stands for the response-model
                    []   stands for an insertion point in the 
                    -->indicates  that  the operation  is  the  
                         insertion of C: into M:[]

                    C:   away
                    M:   I'm going [].

         Type B:     where the cue could be presented as part of  
     the model,  triggering the insertion of a secondary element  
     into the model: 


                    E:   stands for a secondary element
                    C:   stands for the cue imbedded within the  

     Example 3.5(Dacanay) can be described as follows:

                    C:+M:     Where [] Jose live?
                    C:+M: stands for the model with the imbedded  
                    Jose  therefore is the imbedded  cue  
                    <--   indicates  that  Jose  determines  the  
                          choice  of  insertion element  for  the  
                          slot [] in the model.

         Type C:   where the cue itself is not inserted into the  
     model,  rather,  it  triggers the choice of another element  
     which is inserted in its place:


                    C:-->E:   indicates  that the cue determines  
                              the choice of secondary element
                    E:-->M:[] stands  for the insertion of  that  
                              element into the model

                    C: uncle -->E:  he
                    M: Yes, [] 's across the road.

                    The  cue,   "uncle",   determines  that  the  
                    insertion  element is "he".   "he"  is  then  
                    inserted into the response model. 

     Type  D:    where  the  cue  is  deleted  from  the  model,  
     triggering the insertion of a secondary element:

                    M:+C:-->M:--  <-- E:

                    M:+C:     stands  for the model,  containing  
                              the cue
                    -->stands  for the implicit  deletion  
                    M:--      stands for the model with the  cue  
                    <--       stands    for    the     insertion  
                    E:        stands  for the secondary  element  
                              to be inserted     

     Example 3.33(Rivers) can be represented in the form:

                    C: n't
                    M: They have -- [] coffee.

                    n't                      is the cue
                    They have -- [] coffee   is   the  response- 
                                             model with the  cue  
                                             already deleted.

     Type  E:   where there is no cue.   The choice of insertion  
     element is determined by the learner.

     II.        The removal of redundant elements was not  found  
     in any of the simple drills, but was a component in several  
     of the mixed drills.

                    M:+Rd: -->M:--

                    Rd:       stands for the redundant element.
                    M:+Rd:    stands  for the model,  containing  
                              the redundant element
                    -->stands   for  the   operation   of  
                              removing Rd:
                    M:--      stands for the resultant model     

     This    operation   can   be   demonstrated   in    example 


                    M:+Rd:    The noisy children who were making  
                              a lot of noise were sent to bed.
                    M:--      The  noisy children -- were sent to  

                    The  underlined portion of the model is  the  

     III.      With  the re-order of model elements,  the cue is  
     formalized  and  there is no real distinction  between  the  
     prompt and the response model:

                    M:[1][2] -->M:[2][1]

                    M:[1][2]  represents the model with elements  
                              in the original order.
                    -->stands  for  the operation of  re- 
                              ordering the elements
                    M:[2][1]  represents the resultant model

     Example 3.7(Dacanay) illustrates the typical case:

                    M: [The girl] [is] [ready]. -->[2][1][3]
                         [1]       [2]   [3]

     IV.        In  exercises  involving the production  of  new  
     elements based on the cue,  it is not possible to predict a  
     single   response  model,   as  there  are  many   possible  
     variations  This  operation  could  be  symbolised  in  the  

                    C:   stands for the cue
                    -->represents the operation of  responding  
                         to the cue
                    R:   stands for the learner's  response  
                    (M:) stands for the implicit response model.

          "Complex" drills can be categorized as: 

               I)   "multiple" operations,   those in which the  
                    same simple operation is repeated;
              II)   "mixed" operations, those in which different  
                    simple  operations must be brought to  bear;  
             III)   "choice"  of  operation,   where  succeeding  
                    drill   items  require  different  types  of  

     All of the drills which cannot be analyzed in terms of  one  
     simple  operation respond to an analysis based on more than  
     one simple operation.  

     I.       Three  variations  on  "multiple  operations"  are  
     evident.   With  "insertion + insertion",  the  cue  causes  
     insertion  of  secondary elements into more than one  slot.   
     With  "insertion,  then insertion",  the insertion  of  the  
     primary  element creates an environment which triggers  the  
     secondary  insertion.   Similarly with "re-order,  then re- 
     order".   (The reader is referred to table 3.9 for examples  
     of each of the above situations.)

     II.       "Mixed  operations" call for the  application  of  
     different  kinds  of simple operations.   With  "insertion,  
     then removal",  the insertion of an element into the  model  
     forces  the  removal  of other elements which  have  become  
     redundant.   The data contains one example of the need  for  
     "insertion,   then  removal,  then  insertion",  where  the  
     removal  of the redundant elements opens up the opportunity  
     for more insertions into the model.   In two examples,  the  
     learner has to formulate a question and then answer it.  In  
     terms of the present analysis, this could be seen as a "re- 
     order,  then  production  of  new  elements".   In  another  
     example,  the  learner  has to formulate the  answer  to  a  
     question,  and  then insert the cue,  or "production of new  
     elements, then insertion".

     III.       In  drills calling for a "choice of  operation",  
     there  is  only a single operation to be accomplished  each  
     time,   but  the  learner's  task  involves  a  judgemental  
     decision   as   to  the  operation  to  be   selected   and  
     accomplished.   Examples  are found of the  choice  between  
     "insertion  or  re-order"  and  of "insertion  Type  A"  or  
     "insertion Type C"

           It remains probable that there are many other complex  
     combinations  of simple operations that are not evident  in  
     the current sampling of drills and exercises.

     3.7.2     Fixed and variable insertion slots

              Insertion  operations  can have as their goal  the  
     selection of the proper insertion element, the selection of  
     the proper insertion slot,  or both.    The purpose of  the  
     exercise  involving  an insertion will directly affect  its  
     presentation  in a computerised form as it  will  condition  
     the manner in which the response-model can be displayed and  
     will  determine  whether  the  exercise is  a  "simple"  or  
     "complex" one. computer.

               Where  the goal of the exercise is the  selection  
     of the proper insertion element alone,  the insertion  slot  
     will  be  fixed and presented to the learner.    Drills  of  
     this  type can easily be presented with the use of  fill-in  
     or simple multiple-choice response elicitation  techniques.  
     (A  complete discussion of the various response elicitation  
     techniques can be found in section 3.10.6.)

          3.5  (Dacanay) Selection of insertion element:

               Where ---- Jose live?

                         Simple Multiple Choice

               Where ---- Jose live?
                  a. do
                  b. does

              Where the goal of the exercise is exclusively  the  
     selection  of  the proper insertion slot,  the form of  the  
     insertion  element will likely be provided to the  learner.   
     The resultant drill becomes,  de facto,  a  multiple-choice  
     activity.   It  can  either be explicitly  multiple-choice,  
     with the learner typing a letter or number to indicate  the  
     slot  choice,  or  it can be a fill-in activity  where  the  
     learner chooses, using cursor control, the slot in which to  
     type  the  response and the computer records only the  slot  

          3.11 (Dacanay) Selection of insertion slot:
                         Simple Multiple Choice

               He [1] 's [2] on time [3].
               Choose (1-3)

          3.11 (Dacanay) Selection of insertion slot:
                         Fill-in = Multiple Choice

               He --- 's --- on time ---.
               Type the word in the correct blank.

              In cases where both the insertion element and  the  
     correct  insertion slot are to be selected,  the learner is  
     dealing  with a complex drill.   The learner may  correctly  
     choose the element,  but place it into the wrong  slot,  or  
     vice-versa,  or  both.   If  n  represents  the  degree  of  
     feedback possible on each point,  then there would be n x 2  
     feedback possibilities.

            On authoring systems that only offer simple multiple  
     choice,  it  would be necessary to complete such drills  in  
     two steps,  one for each discrete point.  With the "fill-in  
     =  multiple  choice" technique shown in the  above  example  
     (3.11b), both responses could be elicited in a single step,  
     though they would still have to be analyzed separately.  

               The  insertion  operations isolated in table  3.9   
     were  subsequently  re-examined to  determine  whether  the  
     insertion  was to be made in a fixed location in the  model  
     or  whether  the  learner had a choice as to the  point  of  
     insertion  of the required element.  Table 3.10  lists  the  
     results of this examination.   In complex drills,  only the  
     insertion  portion of the activity was  considered.   Where  
     there  were  multiple  insertions,  the likelihood  of  the  
     learner making a mistake in the selection of the  insertion  
     slot was considered and,  if such an error seemed unlikely,  
     the insertion slot was considered as fixed.

          It is only with insertions of Type A,  including those  
     complex drills where the discrete operation is an insertion  
     of Type A,  that one finds a choice of insertion slot.   In  
     the  others,  notably  in  insertions of Type B  &  C,  the  
     insertion slot is fixed.

               A few of the drills illustrate complications.  In  
     River's  example 3.28 & 3.29,  if allowance is made  for  a  
     selection of slot, the slot would determine the form of the  
     insertion element.  In River's 3.32,  the selection of slot  
     is left up to the learner and its validity cannot be judged  

               What  of  those insertions of Type  A  where  the  
     insertion slot is fixed?   Examples 3.4 (Dacanay) and  3.40  
     (Robinett) represent such simple operations that they would  
     barely  qualify as CALL exercises according to the  working  
     definition established previously, as there is a restricted  
     possibility  of error.   In the other four examples (3.61 -  
     3.144)  either the isolation of the cue or changes  to  the  
     cue  based on its insertion into the model are the  primary  
     goals of the exercise.

               In  summary,  it  is  in  insertions  of  Type  A  
     (insertion of cue) that a choice of insertion slot is to be  
     expected,  although  this pattern is  not  invariable.   In  
     other  drills   the insertion slot is likely to  be  fixed.   
     Where  there is a fixed insertion slot,  fill-in or  simple  
     multiple  choice elicitation techniques probably  represent  
     the  best  response  elicitation technique to use  for  the  
     computerised  form.   Where  there is choice  of  insertion  
     slot,  multiple choice is the preferable.   Where there are  
     both,   neither  fill-in  nor  simple  multiple  choice  is  
     adequate.   "Fill-in=multiple choice",   "holistic multiple  
     choice"  or  "free  response" (to  be  fully  discussed  in  
     section 3.9) must be used if the author of the computerised  
     drill  desires  that the learner complete the  exercise  in  
     single step.

     3.7.3   Variations in cue presentation

          The manner in which the cue is to be presented to  the  
     learner  is also of importance to the computerization of an  

           The  cue  can be explicitly presented,  or it can  be  
     implicitly  understood by the learner,  as is the  case  in  
     formalized  drills.   A drill may be partially  formalized.   
     In  other  words,  there may well be an explicit  cue,  but  
     further formal instructions may be required to identify, to  
     determine how to apply it.

          Where  the  cue  is explicitly presented,  it  can  be  
     imbedded in the response model, or it can be presented as a  
     separate element.   Where the cue is a separate element, it  
     can  be  presented  alone  or  be  imbedded  within  a  cue  
     framework.  In the latter case, the learner must be able to  
     isolate the cue from the context of the cue framework.  The  
     cue  is  the discrete part of the cue  framework  that  the  
     learner  must  consider in order to formulate  a  response.   
     The  cue  framework is that part of the cue that  plays  no  
     direct part in determining the response.

           The response-model used in the current classification  
     usurps  most of the function of Dacanay's original idea  of  
     "model".    Sometimes,  however,  it  is  not  possible  to  
     provide all the required information in the cue or response  
     model.   For  those  cases where the learner  will  require  
     supplementary  data that is present in neither the cue  nor  
     the  response  model,  it is possible to define an  element  
     called  the  "prompt".     Prompts  can  serve  either   an  
     operational purpose, providing the learner with information  
     required  to complete the task,  or a pedagogical  purpose,  
     setting up a situation or a reason for the task.    At this  
     juncture,  consideration  is given  only to the operational  
     need for a prompt.

               An examination was made of the variations in  the  
     presentation  of  the  cue,  and of the  prompt  where  its  
     inclusion  is essential,  in the  drills used to illustrate  
     operational  typologies.     Seven  discrete  factors  were  

     1)    whether no explicit cue was to be  presented  because  
     the  cue  was to be implied by the instructions  (in  other  
     words, a formalized drill),

     2)    whether  there  was no explicit cue because  of  some  
     other reason,

     3)    whether  the  explicit cue was  imbedded  within  the  
     response model,

     4)   whether the cue was a separate element,  presented  in  

     5)    whether  the  cue was a  separate  element  presented  
     within a cue framework,

     6)    in cases of explicit cues,  whether more instructions  
     were necessary to isolate the cue from the cue framework or  
     to determine how the cue was to be applied,

     7)   whether a separate prompt was operationally required. 

          Chart 3.11 lists the results of this evaluation, along  
     with a detailed analysis of the cue and prompt presentation  
     in each case.

           Some correlation between operational drill  type,  as  
     established  in  3.7.1  and  presentation  of  the  cue  is  
     evident.   Much of this may stem from the definition of the  
     types,  however.    With  simple insertions of Type A

     (C:-->m:[]),  for example,  where the goal is the insertion  
     of  the cue into the model,  it is no surpise that the  cue  
     and model are presented separately.

           It is interesting to note, nevertheless, that in most  
     of  the  cases the separate cue is presented  in  isolation  
     rather  than  as part of a more  elaborate  cue  framework,  
     which might be more interesting pedagogically.   The use of  
     formal  instructions  to  isolate  the cue  would  make  an  
     activity  more challenging.   One might expect to  see  the  
     latter  two  situations  see  a larger use  in  the  drills  
     categorized pedagogically.

               With insertions of Type B (E:-->M:C:[]) the  most  
     expedient manner of presentation is to imbed the cue within  
     the response model.   It is interesting to note that in the  
     majority of cases,  a separate prompt is also required,  so  
     that the identity of the secondary insertion element can be  
     determined.      In 3.5(Dacanay) for example, given only:

               C:+M:    Where  []  Jose  live?

      the learner might provide responses such as "can",  "may",  
.cp3     etc.  The prompt:

                        Where [does]  Mary  live?

      closes the response set (the set of probable responses,  a  
     term  to  be  discussed more fully in  section  3.10.1)  by  
     limiting responses to forms of the element indicated.   The  
     prompt could be replaced by a formal instruction:

                         Use a form of the auxiliary "do" 
     or  alternatively,  possible responses could be  explicitly  
     displayed using a multiple-choice elicitation technique.

            To  eliminate  the  necessity  for  inclusion  of  a  
     "prompt",  the  identity of the secondary element  must  be  
     fixed  by the grammatical and lexical environment  provided  
     by  the  cue and the response model.   In 3.35(Rivers)  the  
     response set is fixed by the form of the response model and  
     the lexical link between cue and response, and therefore no  
     prompt is necessary.   In 3.140(Stack) the response set  is  
     fixed by the grammatical situation.

               With re-orders of the model elements,  all of the  
     examples  represent formalized drills with no explicit cue,  
     although  it  is possible to imagine that an  author  could  
     devise an explicit cue to serve the same purpose.

               With the production of new elements, the tendency  
     is towards explicit,  separate,  isolated cues, accompanied  
     by  a  formalized instruction as to how to apply  the  cue.   
     There is no cue framework, as the nature of the cue is such  
     that the entire element, and not just a part of it, must be  
     considered by the learner in producing a response.

     3.8   The realisation of exercises on the computer using an  
     authoring system

           In  Chapter Two it was shown that the  mechanisms  or  
     templates  offered  by  an authoring system may  be  fixed,  
     allowing  the author the flexibility to provide merely  the  
     content  of  an  exercise,   or there may  be  a  range  of  
     templates  from which to choose or even the possibility  of  
     creating new templates using "macro-instructions".

           The  mechanism  that  an  authoring  system  provides  
     cannot be considered in isolation from the data required by  
     the  authoring system to generate an exercise.    Once  the  
     global template for an exercise has been  established,  the  
     authoring  system  must to elicit from the author the  data  
     needed for each interaction.  The types of information that  
     the authoring system is able to store influences the manner  
     in  which it must handle the presentation of  an  activity.   
     An authoring system is not capable of operations for  which  
     it  cannot  store  data.    In  proposing,  below,  various  
     possible  templates,  some  consideration must be given  to  
     their data-storage requirements.

           In  section  3.3 a drill item was analyzed into  five  
     component stages.  To allow an author to realise a drill on  
     the computer,   an authoring system must provide a "global"  
     template that permits the completion of each of these  five  
     steps.  The  enormous  range  of variations  that  must  be  
     considered   when  examining  the  complete  drill   defies  
     systematization,  however.  It is necessary,  therefore, to  
     analyze  the  different  stages of  the  drill  interaction  
     independently.   Kearsley's  division  of an exercise  into  
     three functional stages (outlined in section,   1)  
     the  presentation of the drill and the elicitation  of  the  
     learner's  response,   2)  the analysis of the response and  
     the  presentation  of  pertinent  feedback,   and  3)   the  
     decision  as to the next item to be done,  can serve  as  a  
     framework  for  the analysis.    The "global" template  can  
     then  be described in terms of its  component  presentation  
     templates,   answer-analysis   templates,   and   branching  

          It  must be kept in mind that the components  are,  of  
     course,  inter-related.    The  choice  of  a  fill-in  vs.  
     multiple choice presentation,  for example,  will determine  
     the  nature of the answer-analysis mechanism and  the  data  
     required for answer-analysis.  Branching possibilities will  
     be  very  much affected by the choice of  presentation  and  
     answer-analysis templates.

           The  first  phase of a drill to be considered is  the  
     presentation  of  the  stimulus to  the  learner,  and  the  
     subsequent elicitation of the learner's response.   While a  
     collection  of  typical example templates is  developed  to  
     describe  the  drills  listed in  the  data,  it  is  again  
     impossible  to  systematize global  presentation  templates  
     because   of  the  large  number  of  possible  variations.  
     Instead,  the discussion is restricted to various  discrete  
     aspects   of   the  stimulus  presentation   and   response  
     elicitation.  Four basic approaches to eliciting a response  
     are   considered:    fill-in-the-blank,   multiple  choice,  
     holistic multiple choice, and free response,  with a review  
     of the response-elicitation mechanisms that can be  adopted  
     for  each  case.    The  presentation of  the  stimulus  is  
     examined in terms of the instructions, the prompt, the cue,  
     and the response-model. 

           The  response-elicitation mechanism,  or REM,  is the  
     bridge  between the presentation and  the  answer- analysis  
     phases.  A particular REM will delimit the range of answer- 
     analysis possibilities.    Conversely,  the requirements of  
     the  answer-analysis  phase may call for a particular  REM,  
     which,  in turn, may influence the form of the presentation  

               With "fill-in-the-blank" exercises,  the computer  
     displays  the response-model, with the appropriate blank or  
     blanks  and  the learner is invited to enter  the  elements  
     that  would  normally occupy the indicated  slot  or  slots  
     within the model. 

          Example of simple fill-in (3.4  Dacanay):  

          Response Model:  M:    I'm going ------.
     The  response-elicitation may be represented  schematically  

          ?-->rt         where:
                         ?-->represents    the   act    of  
                                   soliciting a response.
                         rt        stands  for  a short  one- or  
                                   two-word textual response

           Sometimes  there may be two blanks to complete as  in  
     the following example (3.12 Dacanay):     

          C:+M:               He --- n't --- coffee.           
          ?-->rt1,rt2            ?       ?

                         rt1       stands  for the completion of  
                                   the first blank 
                         rt2       stands for the completion  of  
                                   the second blank                    

               Where  there is a choice of slots  for  inserting  
     the response,  this choice can either be made by a multiple  
     choice  or  by allowing the learner to position the  cursor  
     within  the appropriate blank and then giving the  computer  
     the ability to understand where it has been placed.

          Example of multiple-choice 3.6(Dacanay):

          I:                  Choose a or b
          C:                  tonight
          M:                  The --- leaves --- at seven.
                                  a.         b.
          ?->rm               ?

                          rm        stands for a multiple choice 

          Example of "fill-in = multiple choice":

          P:                  The train leaves tomorrow at seven.
          C:                  tonight
          M:                  The --- leaves ---- at seven.
          ?->rl=m                 ?          ?

                         rl=m      stands   for  the   situation  
                                   where the cursor location can  
                                   be translated into a multiple  
                                   choice response

           For the cases involving both the choice of  insertion  
     slot  and  the form of the element to be  inserted,  it  is  
     appropriate to postulate a REM of the following type:

          ?->rl=m,rt     where  two response would,  effectively,  
                         be  generated  by the  learner's  single  

               Three   different  multiple choice  presentations  
     can  be  postulated  - simple  multiple  choice,   holistic  
     multiple  choice and re-order multiple choice.   The  first  
     would parallel very much the fill-in technique, in that the  
     computer would display the response model with a blank.  In  
     this case,  however,  the response set would be  explicitly  

          Example of Simple Multiple Choice (3.5 Dacanay):
          C:+M:          Where --- Jose live ?
                           a.  do
                           b.  does

               With holistic multiple choice,   the response set  
     is  included  with the response model,  to produce  several  
     complete  responses.   This is most appropriate  with  many  
     complex drills,  since the multiple or mixed operations can  
     be reduced to a single operation,  and for situations where  
     the response set is excessively open-ended.

          Example of holistic multiple choice (Dacanay 3.12)

          P:             He likes coffee.
          C:+M:          a.   He doesn't like coffee.
                         b.   He don't like coffee.
                         c.   He doesn't likes coffee.
                         d.   He don't likes coffee.

           Re-order  multiple choice differs from the other REMs  
     in  that  a  series  of  discrete  responses  is   entered,  
     representing,  in  fact,  the entire explicit response set.   
     What is significant is not the elements provided, but their  
     order.    The REM required for re-order exercises  as:


           Multiple-choice  is  the  most powerful of  the  REMs  
     provided  by  authoring  systems  in  the  sense  that  the  
     simplest   authoring  system  can  still  present   complex  
     exercises by presenting them as multiple-choice activities.   
     Pedagogically,   multiple-choice   may  not  be  the   most  
     appropriate  template for an activity as the  learner  need  
     only recognize and not produce the correct response.    Its  
     use is limited to those situations where the set of correct  
     answers can be predicted.

               Free-response  is  the easiest REM to  structure,  
     from  the  point  of view  of  presentation  and  response-  
     elicitation,  but  can  be  the   most  difficult  for  the  
     computer to handle in terms of response-analysis.  There is  
     no explicit response model,  rather, the response model (or  
     models)  are  used internally during the evaluation  phase.   
     For  free  response  to be possible,  there  must  be  some  
     systematization, however complex, of the responses.

          Example of free response (3.37 Rivers):

          C:             Why   didn't   they  come  home   before  
                         Where rs stands for a sentence response.

           The  first part of the stimulus is  the  instructions  
     and other supplementary data (I:).   For the most part, the  
     instructions  are  to  help the learner to  understand  the  
     nature  of the task and how to perform  it,  although  they  
     could  contribute  to  the  motivation of  the  learner  by  
     establishing a pedagogically rich environment.    Under the  
     category  of  instructions,  it is appropriate  to  include  
     supplementary text,  drawings,  and so forth,  which  offer  
     pedagogical support,  but which do not figure operationally  
     in  the activity (a paragraph establishing the  situational  
     frame  for a drill,  for example).   Instructions and other  
     support material that does not figure operationally in  the  
     activity  can be represented schematically with the symbol:   
     "TTT",    standing  for neutral text.    In  formalized  or  
     partially  formalized  drills,  the cue is implicit in  the  
     instructions  and so the instructions become  operationally  
     important.       Instructions that figure operationally  in  
     the  activity  are symbolized below by the  symbol:   "CCF,    
     standing for formalized cue.    Example  3.34(Rivers),  for  
     example,  could be treated as a formalized drill,  in which  
     case the general instructions might be:

          TTT       Vary   the  final  segment  from  future   to  
                    conditional, according to the first segment.

     To  ensure  the desired response,  further,  more  explicit  
     instructions would be required:

          CCF       Use " 'd "  or " 'll " in your response.

               Material   presented  to  the  student   can   be  
     universal (i.e. the same across all items in a sequence) or  
     item-specific.  Lower case is used below to represent item- 
     specific  components and UPPER  CASE to represent universal  
     components.    Thus, the patterns TTT or ttt or both may be  
     encountered.    A  drill  sequence  could  have   universal  
     instructions,  TTT, and item-specific situational settings,  

               An  authoring  system need only elicit  universal  
     material once for an entire sequence but must elicit  item- 
     specific  material  separately for each  item.    Universal  
     material can always, of course, be treated as item-specific  
     material.   The  learner  would see no  difference  in  the  
     presentation,  but  the author would be asked to enter  the  
     same  material  consecutively for  each  item.    Universal  
     frameworks, however, can be used in the generation of item- 
     specific material and,  as is demonstrated below, there are  
     cases  where  the  use of item-specific data instead  of  a  
     universal framework reduces certain options, notably in the  
     case  of branching possibilities with  interlocked  drills.     
     There are even  cases in which item-specific data cannot be  
     predicted  and  entered in  advance,  as  prompt,  cue,  or  
     response models will depend in part on information that the  
     learner has provided. 

           The  specific prompt for each item can thus be stored  
     in advance,  as   ppp,    or  a prompt generating algorithm  
     or  rule can combine a universal prompt  framework,  PPP[],  
     with some other element to derive  ppg,  the  item-specific  
     prompt.    In  some  cases,  a drill may require  the  same  
     universal prompt for all items, PPP.    Similarly,  the cue  
     can be stored as ccc,   CCC[] +  ?  -->ccg,  or  CCC  and  
     the  response  model as mmm,  MMM[] + ?  -->mmg,  or  MMM.    
     Insertion response-models are represented  as mmm[], mmg[],  
     or  MMM[],  and  non-insertion models  without  the  square  
     brackets,  [].    In the case of multiple choice templates,  
     the   explicit  response  models  are  represented  as   by   
     rr1...rrn,   or   rrn  for short,  with RRN  for  universal  
     response models. 

     3.9.2     Presentation   options  in  the   data:    Simple  

           This section relates the analyses contained in tables  
     3.9,  3.10,  and 3.11,  along with other variations not yet  
     discussed,   to the possible computerization of the  drills  
     in the sample. Simple insertions of Type A

               Simple  insertions of Type A require,  at  least,  
     ccc  &  mmm[].   We will assume  TTT to be present  in  all  
     cases.    Example  3.4  (Dacanay) could then  be  presented  
     using the following template:

          P-template #1

          I:    TTT      Complete  the  sentence with  the  word  
          C:    ccc      away
          M:    mmm[]    I'm going ----.
          ?:    ?->r               ?

               where  the author would have to provide  TTT *  1   
          plus   ccc,mmm[] times (*) the number of items in  the  
          sequence or n-times.

           Since the response models are identical across items,  
     some  authoring systems may allow the entry of a  universal  
     response model:

          P-template #1b

          I:   TTT
          C:   ccc
          M:   MMM[]     
          ?:   ?->r

               where  the author would have to provide  TTT,MMM[]  
          * 1  plus  ccc *  n.

     There  is no operational difference between the  two.   The  
     only  benefit  foreseen would be to economize the  author's  
     time and to save storage space within the computer.   While  
     these   factors   of  user-friendliness   and   programming  
     efficiency    are  important  considerations   in   judging  
     authoring  systems,  they  are not under  discussion  here.    
     Therefore mention of the possibility of using PPP,  CCC, or  
     MMM  is  only  made  where such use  can  have  operational  

           Example   3.40    (Robinett),    3.118(Stack),    and  
     3.144(Stack)   could  also  be presented  using  the  above  
     formula.   They  have  in  common  that  they  are  Type  A  
     insertions  with  the insertion slot fixed,  where the  cue  
     and the response model contain all the information required  
     to complete the operation.   

           Examples  3.118 and 3.144  differ from 3.4 & 3.40  in  
     that the learner's task is more complex.  In 3.4 & 3.40 the  
     cue  is inserted without change,  while in 3.118 and  3.144  
     the cue must be changed.   At first glance this may seem to  
     be  a  function  of  the drill  content  alone.   It  does,  
     however,  have one operational implication.   Whereas 3.118  
     could be presented as a simple multiple choice, for 3.4 and  
     3.40 such explicit listing of the responses would make  the  
     activity very trivial:

          P-template#2   3.118(Stack)

          I:   TTT       Choose the correct completion
          C:   ccc       Il se repose
          M:   mmm       Si Paul est fatigue ----.
               rrn                        a. il se reposera
                                          b. xxx
                                          c. xxx

          I:   TTT       Choose  the form which is the  insertion  
                         of the cue into the model.
          C:   ccc       away
          M:   mmm       I'm going  ----.
               rrn                a. away
                                  b. xxx
                                  c. xxx

               Example  3.9(Dacanay)  would require  a  multiple  
     choice template,  either fill-in=multiple choice,  assuming  
     the  authoring system provided that capability,  or  simple  
     multiple choice, if not.

          P-template #3

          I:   TTT       Insert the cue at the correct point in
                         the sentence
          C:   ccc       for me
          M:   mmm[]     Mother made [] a dress [].
          ?:   ?->rl=m               ?          ?

          P-template #4  

          I:   TTT       Type  1  or  2 to indicate  the  correct  
                         place in the sentence to insert the cue
          C:   ccc       for me
          M:   mmm       Mother made [1] a dress [2].
          ?:   ?->rm     ?

          Like  3.9,  3.11  (Dacanay) could also  use  the  above  
    approaches.   Besides  being Type A insertions,  they have in  
    common that they both have a choice of insertion slot,  there  
    is no change to the form of C:, and C: and M: contain all the  
    information required to complete the operation.

           Example  3.6(Dacanay) differs from the above in  that  
     the  cue  replaces   one of the words that present  in  the  
     original  sentence.   If  the exerccise is  done  as  fill- 
     in=multiple choice, therefore, a prompt will be required so  
     that the learner can see the original sentence:

          P-template #5

          I:   TTT       Insert the cue at the correct point in
                         the sentence
          P:   ppp       The train leaves tomorrow at seven.
          C:   ccc       tonight
          M:   mmm[]     The [] leaves [] at seven.
          ?:   ?->rl=m        ?          ?

    Where  done  as a simple multiple  choice,  C:  and  M:  will  
    contain all the required information:

          P-template #4  

          I:   TTT       Type  1  or  2 to indicate  the  correct  
                         place in the sentence to insert the cue
          C:   ccc       tonight
          M:   mmm       The [train] leaves [tomorrow] at seven.
                               [1]             [2]
          ?:   ?->rm     ?

               Example  3.6 differs in another important  aspect  
     from  3.9,  in  that it is an an  interlocked  drill.   The  
     result  of the insertion made in item(x-1) is reflected  in  
     either  P:(x) or or M:(x),  depending on the approach used.    
     If ppp is used,  then the order of the items will be  fixed  
     in  the  data and no computer-controlled branching will  be  
     possible.    To provide a flexible branching option with an  
     interlocked  drill,  the  authoring  system  must  offer  a  
     mechanism  whereby an element of drill item(x-1),  in  this  
     case  C:(x-1),  can  be inserted into  a  universal  prompt  
     framework,  PPP[], replacing an earlier element.  The P:(x)  
     can thus be generated,  based on whatever the previous item  
     was.   This  algorithm  may  be represented  in  the  form:               

               PPP[] + ccc(x-1) -->ppg(x)

    which gives rise to the following alternate template:

          P-template #6

          P-Rule:        PPP[] + ccc(x-1) -->ppg(x)
          I:   TTT       Insert  the cue at the correct point  in  
                         the sentence
          P:   ppg       The train leaves tomorrow at seven.
          C:   ccc       tonight
          M:   MMM[]     The [] leaves [] at seven.
          ?:   ?->rl=m       ?          ?

    whose  surface realization is identical to  the  presentation  
    using  P-template#5  above.     If done on  the model  of  P- 
    template#4 above,  the discussion above about the interlocked  
    nature  of  the drill would hold.   For  computer  controlled  
    branching,  one  would need a mechanism whereby mmm would  be  
    replaced by  mmg  based on the P-Rule:

               MMM[] + ccc(x-1) -->mmg(x)  

           Like    3.6    are    3.10(Dacanay),    3.21(Rivers),  
     3.42(Robinett),  and  3.60(Cook).   These Type A insertions  
     have in common that they are interlocked drills,  in  which  
     the  cue replaces an element already present in the  model,  
     and there is a choice of insertion slot.

           Examples 3.120(Stack) and 3.125(Stack) are formalized  
     drills  with no explicit cue.   Only M:,  and possibly  P:,  
     need be displayed:

          P-template #7

          I:   CCF       Make the statement negative  
          P:   ppp       They're reading.
          M:   mmm[]     They [] reading.
          ?:   ?->rt     ?

              Example  3.61(Cook) is a partially formalized drill  
    where both I: and C: are functionally required: 
          P-template #8

          I:   CCF       Insert the second item mentioned in  the  
                         cue into the model
          C:   ccc       I  can't decide whether I like  swimming  
                         or skating best.
          M:   mmm[]     Oh, I prefer ----.
          ?:   ?->rt     ?

           If one could take advantage of the universal elements  
     that are present in the original, the computer could  offer  
     a great deal more flexibility with this drill than is shown  
     in  the above template.    A cue framework  CCC[]  could be  
     defined  into  which the item discrete cues   cc1  and  cc2  
     could  be placed.    An algorithm could place the cues into  
     first   and   second  slots  randomly  so  that   on   each  
     presentation of the item,  the order (and the answer) might  
     be different.   Alternatively,  or in addition,  the formal  
     cue  CCF  could  become  a  framework   CCF[]  into   which  
     universal  cues  CC1  ("first") and CC2 ("second") could be  
     placed at random, changing the instruction:


          P-Rules        1.   CCF[] + CC1,CC2  -->ccfg
                         2.   CCC[] + cc1,cc2  -->ccg
          I:   ccfg      Insert  the first item mentioned in  the  
                         cue into the model
          C:   ccg       I can't decide whether I like skating or  
                         swimming best.
          M:   MMM[]     Oh, I prefer ----.
          ::   ?->rt                  ?

     This  subtlety of presentation assumes  an  answer-analysis  
     mechanism  capable,  in  the same way,  of  generating  the  
     current correct response.

           So far the discussion has focused on presentations in  
     fill-in  or  simple multiple choice modes.    Most  of  the  
     drills  discussed  could  also  be  presented  as  holistic  
     multiple- choice  or free-response  activities.    Wherever  
     simple  multiple-choice  could be used,  holistic  multiple  
     choice  is  also possible,   with the difference  that  the  
     separate ccc or mmm[] can be suppressed if desired.

          P-template #11      3.9(Dacanay)

          I:   TTT       Choose the correct sentence  
          M:   rrn       a.  Mother made a dress for me.
                         b.  xxx
                         c.  xxx
          ?:   ?->rm     ?

     As  holistic multiple-choice is almost always an  available  
     option,    especially   for  those  authors  using  a  weak  
     authoring   system,    It  is  discussed  only   in   those  
     extraordinary cases where it might not be possible or where  
     it  is  the  only  option.     Likewise,  from  simply  the  
     presentational  point  of view,  free response will  always  
     remain   possible.     The  more  complex   answer-analysis  
     requirements of free response will make it an  uneconomical  
     option in many cases.   In other situations, it will be the  
     only possible presentation mechanism.  Mention of the free- 
     response  option  is  henceforth only made  in  the  latter  
     context.    Simple insertions of Type B

           With  simple insertions of Type B,  the cue  and  the  
     response model constitute one unit.   In its simplest form,  
     this could be represented in the form  mmm[],  as C: and M:  
     are  abstractions  that  do not  always  require  separate,  
     concrete,    realisation   on   the   computer.     Example  
     3.35(Rivers) exemplifies the simple case:

          P-template #12

          I:     TTT     Complete   the   statements   with   the  
                         appropriate occupational term
          C:+M:  mmm[]   A person who builds houses is a ----.
          ?->rt                                          ?

               Example 3.140(Stack) is similar.   In both cases,  
     the  relatively closed lexical or grammatical nature of the  
     task  is such that the range of response  possibilities  is  

               In  all the other Type B insertions in the  data,  
     some  additional  information is required in order for  the  
     learner  to produce the intended response.   This can  take  
     the form of either a prompt or of formalized  instructions.    
     Example  3.34(Rivers)  provides  a typical  case.   With  a  
     prompt, it can be presented in the form:

          P-template #13

          I:   TTT       Vary  the final segment from  future  to  
                         conditional,  according  to  the   first  
          P:   ppg       If I see him, I'll tell him.
          M:   mmg[]     If I saw him, I -- tell him.
          ?:   ?->rt                      ?

               3.34 is also an interlocked drill,  which is  why  
     ppg  and mmg are adopted.   The algorithm to produce ppg is  
     more  complex  than in insertions of Type A,  as  both  the  
     mmg(x-1)  and  the  correct answer to  the  previous  item,  
     aaa(x-1), are needed to produce ppg(x):

               PPP[] + mmg(x-1) + aaa(x-1) -->ppg(x)

     As discussed earlier,  the drill could,  of course, be done  
     with ppp and mmm[], but the order would become fixed by the  

          P-template #14

          I:   TTT       Vary  the  final segment from future  to  
                         conditional   according  to  the   first  
               CCF       Use 'd or 'll in your response.
          M:   mmm[]     If I saw him, I -- tell him.
          ?:   ?->rt                     ?

               As a formalized drill, the interlocked quality of  
     3.34  can be maintained without the computer generation  of  
     elements.  In the situation where an authoring system could  
     not  generate ppg and mmg and computer-controlled branching  
     was  nevertheless required,  the drill could always be  re- 
     cast in a formalized format.

               In  the  multiple-choice  mode,  rr1...rrn  would  
     achieve approximately comparable results to those of CCF in  
     the  above  example.   Multiple-choice might,  then,  be  a  
     possible  alternative  to fill-in for many drills  of  this  

               Most  of  the  other examples  of  insertions  of  
     Type B,     3.5(Dacanay),   3.41(Robinett),   3.117(Stack),  
     3.132(Stack), 3.142(Stack), and 3.143(Stack),  would submit  
     to  the findings outlined in the discussion of  3.34,  with  
     some  minor  adjust   Only the first  two  are  interlocked  
     drills.  In  the  case  of  3.117,  both ppp  and  CFF  are  
     required.   3.142 and 3.143 are best served with a  prompt.   
     As  formalized  drills,  they would  require  item-specific  
     instructions.  In 3.142(Stack), for example:

                   ccf(1):     "Use ` read '"
                   ccf(2):     "Use ` eat  '" Simple insertions of Type C

              One would expect insertions of Type C to be fairly  
     similar  to  insertions  of Type A in  their  presentation.   
     Again,  the  cue and the response model must  be  presented  
     separately.  A major difference is that whereas with Type A  
     insertions  the  cue  tends  to  be  isolated,  in  Type  C  
     insertions it tends to appear within a cue framework, which  
     opens  up the possibility of computer-generated cues  where  
     the cue framework is universal (CCC[] + ccc -->ccg).   Use  
     of computer-generated cues would provide for economies but,  
     in  most cases,  would not greatly affect the operation  of  
     the drill.   Example 3.20(Rivers), which is typical,  could  
     be presented as a fill-in exercise:

          P-template #1

          I:   TTT       Insert the correct pronoun 
          C:   ccc       Do you see my uncle over there?
          M:   mmm[]     Yes, ---'s across the road.
          ?:   ?->rt          ?

          P-template #2

          I:   TTT       Choose the correct pronoun 
          C:   ccc       Do you see my uncle over there?
          M:   mmm[]     Yes, ---'s across the road.
               rrn            a.  he/b. xxx/c. xxx
          ?:   ?->rm     ?

               Similar to 3.20 are most of the other examples in  
     this  group:  3.46(Robinett),  3.47(Robinett),  3.62(Cook),  
     3.63(Cook),     3.64(Cook),     3.65(Cook),     3.66(Cook),  
     3.106(Stack), 3.111(Stack), and 3.114(Stack).  In 3.106 and  
     3.111,  the  cue  framework  and  the  response  model  are  
     identical.  In 3.66 the response model is so simple amd the  
     nature  of the task is such that free-response would  serve  
     equally well in soliciting the response as would fill-in or  

               Examples  3.28(Rivers)  and 3.29(Rivers)  present  
     complications.  Both are interlocked drills.  In 3.28 there  
     are instances in which the  learner should have the  option  
     of  choosing  either  insertion-slot,   as  they  are  both  
     correct,  and  other  instances  where no  such  choice  is  
     possible.   The authoring system must offer a mechanism for  
     switching   presentation  templates  (and  answer  analysis  
     templates).   To  maintain  the interlocked nature  of  the  
     drill,  the  template  for  item(x)  select  which  cue  to  
     present, based on the learner's choice in item(x-1):

          P-template #15

          I:   TTT          
          C:   ccc(x-1)  Janet read her mother the letter.
          M:   mmm(x-1)  Janet read -- the letter --.
          ?:   ?->rl=m,rt           ?             ?
          I:   TTT
          C:   ccc(x)-a  Janet read her the letter.
               ccc(x)-b  Janet read the letter to her.
          M:   mmm(x)    Janet read -- to her.
          ?:   ?->rt                ?

           In  3.29 the learner is free to provide a wide  range  
     of responses.   Again, the learner should be given a choice  
     of  insertion slot.   In order to maintain the  interlocked  
     nature of the drill, the cue and response model for item(x)  
     would  have  to be generated on the basis of  a  choice  of  
     template, and on the learner's previous response to item(x- 
     1).  There would be no way to pre-determine either the word  
     provided in item(x - 1), or the slot chosen. 

          P-template #16

          I:   TTT
          C:   ccc(x)    I gave it to her.
          M:   mmm(x)    I gave -- to her  --.
          ?:   ?->rl=m,rt       ?          ?
          I:   TTT
          C:   ccg(x+1)-a     I gave [aaa(x)] to her.
               ccg(x+1)-b     I gave her [aaa(x)].
          M:   mmg(x+1)       I gave -- [aaa(x)] --.
          ?:   ?->rl=m,rt            ?           ? 

          There is only one example of a Type D insertion in the  
     data,  3.33(Rivers).   The  way  that 3.33 is best  handled  
     depends on whether the focus is to be only the  correlative  
     change  or  it  should  include the  learner's  ability  to  
     isolate the negative element as well.   In the former case,  
     the  formalized  cue is not  really  necessary.   Simply  a  
     prompt  and the response model need be provided,  as in the  
     example below:

          P-template #17

          I:   TTT       Insert the correct form into the  model,  
                         based on the change observed
          P:   ppp       They haven't any coffee.
          M:   mmm[]     They have -- coffee.
          ?:   ?->rt               ?

           It would be difficult to handle the latter  case,  of  
     isolation  and removal of the negative element coupled with  
     the correlative change,  as a fill-in exercise because  the  
     presentation  of  mmm[]  would  reveal  the  answer.   Free  
     response  would probably be the most suitable  presentation  
     method.   On  authoring systems that could not handle free- 
     response,  holistic multiple choice would be the second and  
     less attractive possibility.  

          P-template #18      3.33(Rivers) Free Response

          I:   CFF       Delete the negative elements 
          P:   ppp       They haven't any coffee.
          ?    ?->rs     ?

          P-template #19

          I:   TTT       Choose the correct negative form 
          P:   ppp       They haven't any coffee.
          M:   rrn       a.  They haven't any coffee/b. xxx/c. xxx
          ?:   ?->rm     ? Simple insertions of Type E

          In Type E insertions,  the learner has a great deal of  
     freedom  to  choose the form of  the  insertion.   In  such   
     cases,  the  needs of the answer-analysis phase will be the  
     determining factor in choosing which presentation  template  
     to  use.   In example 3.32(Rivers),  the learner can insert  
     any  response  which  is  grammatically  and   semantically  
     correct,  into any desired slot.  Furthermore, the drill is  
     interlocked, so that the learner's choices are reflected in  
     the following item.  A sophisticated authoring system could  
     conceivably  have  the  potential to  generate  a  response  
     model:  mmg  based on a complicated  framework,  MMM[],  in  
     conjunction  with  the learner's material provided  in  the  
     previous  item,  aaa(x-1).    It  is difficult to  see  how   
     aaa(x)  could  be  analyzed  for  correctness  without  the  
     involvement  of a grammatical parser endowed with  semantic  
     interprative capabilities.   Fill-in or free-response would  
     thus be ruled out.   The exercise could be done as a simple  
     multiple-choice,  but the learner would lose the freedom to  
     choose the insertion slot.   The most faithful reproduction  
     of  the  original  on the  computer  would,  therefore,  be  
     provided by holistic multiple-choice.

          P-template #20

          I:   TTT       Choose  the  correct  expansion  of  the  
                         given sentence.
          P:   ppp       The man crosses the street.
          M:   rrn       a.  The tired old man crosses the busy street.
                         b.  xxx
                         c.  xxx
          ?:   ?->rm     ?

          Example 3.36(Rivers) is somewhat more  rigid.   Formal  
     instructions  are given as to the nature of the  insertion,  
     and  the learner has freedom in areas that do not  directly  
     affect  the goal of the drill.   This could be performed as  
     an insertion,  with an answer-analysis algorithm capable of  
     checking  for "to" or "not to" in the  learner's  response.   
     The  rest  of the learner's response would have  to  remain  
     unanalyzed,  a  situation  which many teachers  would  find  
     unnacceptable.   If  total control over the correctness  of  
     the  learner's  response were required,  the  best  option,  
     again,  then,  would  seem to be multiple choice.   As  the  
     insertion  slot  is  fixed,  simple multiple  choice  would  
     probably be adequate.

          P-template #21

          I:   TTT       Choose   the  ending  that  provides   a  
                         correct infinitive construction
          M:   mmm[]     She had decided []
               rrn       a.  to marry him./b.  xxx/c.  xxx
          ?:   ?->rm     ? Re-order of model elements

          Example 3.7(Dacanay) is typical of the exercises where  
     the goal is the re-order of model elements:

          P-template #22

          I:   CCF       Re-order  the parts of the statement  so  
                         as to make a question
          M:   mmm       The girl is  ready.
                           [1]    [2] [3]

          ?:   ?->ro     ?

           Examples 3.8(Dacanay),  3.14(Dacanay),  3.127(Stack),  
     and 3.129(Stack) are similar.   All are formalized  drills.   
     All require that the model,  mmm,  be presented.   Multiple  
     choice  is really the only way to handle exercises of  this  
     type.   While  with  some  authoring systems  it  might  be  
     expedient to present them in free response mode, they would  
     still  essentially be multiple choice activities.   Instead  
     of typing the numbers directly, the learner would be typing  
     word(1),   word(2),   etc.     There  would  still  be   no  
     opportunity for insertion or creation of new elements. Production of new elements

          The   last   of  the  simple  drill  types   represent  
     activities  where the learner must produce new elements  in  
     response to a cue, but without any explicit response model.   
     This results in a simple presentation template:

          I:   CCF       Answer the question truthfully  
          C:   ccc       Why didn't they come home before midnight?
          ?:   ?->rs     ?

           All  of  the examples of  this  type,  3.31A(Rivers),  
     3.37(Rivers),  3.38(Rivers),  3.39(Rivers), 3.54(Cook), and  
     3.145(Stack), could all be presented in a similar format. 

          With  authoring  systems  that could not  handle  free  
     response in the answer-analysis phase, the only alternative  
     for  the  presentation of this type of  exercise  would  be  
     multiple-choice, as in the following example from 3.31A:

          P-template #24

          I:   CCF       Which  of the following best  represents  
                         the result of following the instructions  
                         given ?
          C:   ccc       Tell George your name is Ronald.
          M:   rrn       a.   George, my name is Ronald.
                         b.   xxx
                         c.   xxx
          ?:   ?->rm     ?

           At  first glance,  many teachers would probably  find  
     the  multiple-choice  version  of these  activities  to  be  
     weaker  than  free-response  one.  The  question  of  which  
     approach  is  superior  evaluated further when  the  answer  
     analysis  phase is considered in section  3.10.  With  some   
     examples,  of course,  such as 3.38, multiple-choice is not  
     a  viable option because the learner must provide  original  

    3.9.3 Presentation options in the data:  Complex drills

               The  analysis of complex drills has led to  their  
     categorization   into   those   involving   the    multiple  
     application  of the same simple operation,  the application  
     of  different  operations,  or  the  choice  of  operation.    
     Where a response may involve several factors, or where more  
     than one discrete response is required, it might be simpler  
     to formulate templates that would execute the drill as  two  
     separate   and   subsequent  simple   drills.    It   seems  
     pedagogically  advantageous,   however,  to  structure  the  
     learner's  task so as to preserve the unified nature of the  
     activity.    Computer  realisation of the drill should  not  
     require the learner to provide separate, discrete responses  
     where this does not seem natural.    Internally, of course.  
     the  learner's  single  response may have  to  be  analyzed  
     according to multiple factors, and so  result in a range of  
     feedback options much wider than might derive from a simple  
     drill. Multiple operations

           The  examples  of insertion + insertion contained  in  
     the data, 3.12(Dacanay), 3.22(Rivers), and 3.134(Stack) all  
     exhibit the need for separate prompts (P:),   have the  cue  
     imbedded  in  the response model (C:+M:),   and have  fixed  
     insertion slots.   In other words,  they resemble very much  
     simple  insertions  of Type B,  except that there  are  two  
     elements to be inserted into two discrete slots.   The form  
     of  the insertion element for each slot,  rather  than  the  
     choice of element to place into either slot seems to be the  
     major motivation in all three examples.      

           The possible fill-in template shown below for example  
     3.22,  which  seems most typical,   resembles very much  P- 
     template#13,   which  was applicable to most of the  simple  
     insertions  of  Type  B.    The only difference is  in  the  
     response elicitation mechanism,  which in this case must be  
     capable of eliciting two fill-in elements for two  separate  
     blanks,   leading,   in   all   probability,   to  a   more  
     sophisticated response analysis and feedback template.

          P-template #25

          I:   TTT       Make the necessary changes
          P:   ppg       He brings his lunch
          M:   mmg       You --- --- lunch
          ?:   ?->rt1,rt2    ?   ?

           Holistic  multiple-choice could be adopted to  reduce  
     the  learner's  input to a single  item,  so  reducing  the  
     corresponding number of feedback possibilities:

          I:   TTT       Choose the correct sentence
          P:   ppg       He brings his lunch          
          M:   rrn       a.  You bring your lunch.
                         b.  xxx
                         c.  xxx
          ?:   ?->rm          

          (The   prompt  is  not  operationally  required   this  
          example.     The  use  of  ppg  and  mmg  reflect  the  
          interlocked nature of example 3.22.) 

               In  those examples of consecutive  insertions  in  
     the data,  the first is a Type A insertion and the second a  
     Type  B insertion.    Two possibilities present themselves.   
     In example 3.26(Rivers),  the insertion slots for both  are  
     fixed  and  the  drill can be presented in  a  manner  very  
     similar to the previous examples:

          P-template #26

          I:   CCF       Convert the statements to questions.
          C:   ccc       Peter has a new car.
          C:+M: mmm        [] Peter  [] a new car?
          ?:   ?->rt1,rt2  ?         ?

          P-template #11b

          I:   CCF       Choose a correct question
          C:   ccc       Peter has a new car.         
          M:   rrn       a.  Does Peter have a new car.
                         b.  Has Peter got a ner car.
                         c.  xxx
          ?:   ?->rm          

           Example  3.36(Stack) also shows fixed slots for  both  
     insertions,  and could be handled similarly,  although,  in  
     this case,  C:+M: could not be used and a prompt is needed:  
          P-template #26b

          I:   CCF       Answer the question
          P:   ppp       The boy is polite          
          C:   ccc       How does he speak?
          M:   mmm       The boy  [] [] .
          ?:   ?->rt1,rt2         ?  ?

           Holistic  multiple-choice and free-response are  also  
     possible.   The relatively predictable response model makes  
     free-response   an  ideal,   and  probably  more   natural,  

           Example  3.31b is the third example with fixed slots.   
     As a fill-in,  this would be extremely clumsy,  given  that  
     four discrete responses would have to be elicited.    Free- 
     response  is  theoretically possible,  as a  fixed-response  
     model  could be established.   The response is quite  long,  
     however,  making  the learner's task difficult  and  error- 
     prone.   A  very complex response-analysis template can  be  
     envisioned,  which  might well be beyond the reach of  most  
     authoring systems.   The best approach to 3.31b seems to be  
     holistic multiple-choice.


           Examples  3.24(Rivers)  and 3.43(Robinett) exhibit  a  
     choice of insertion slots for the first  insertion.    With  
     3.24,  the  slot for the second insertion is also variable.  
     There  would be so little in the way of  explicit  response  
     model that the fill-in mode would almost,  de facto, become  
     free- response,  a very viable option here, given the fixed  
     nature of the implicit response model:


          I:   TTT       Insert  the cue and make other necessary  
          P:   ppg       She brings too many pencils to school.
          C:   ccc       You
          ?:   ?->rs     ?

           By dealing on the level of word units,  the re-order,  
     then re-order example, 3.15(Dacanay), could be treated as a  
     single  re-order  multiple-choice,  and the discussion  and  
     template shown in would apply:

          P-template #22

          I:   CCF       Combine the two questions,  putting  the  
                         yes/no question first.      
          M:   mmm       Who is  he?  Do you know?                    
                         [1] [2] [3]  [4][5] [6]
          ?    ?->ro

     The possible combinations of six elements far exceed  those  
     of  the  three elements shown in the example of the  simple  
     re-order  operation shown in,  however and  a  more  
     complex  response-analysis-and-and feedback template  could  
     be predicted.      


          Leaving  aside  the use of holistic multiple-choice  or  
    free-response,  which are everpresent possibilities and which  
    can  reduce the learner's task to a one-step  operation,  the  
    insertion  +  removal  exercises  represent  a  new  kind  of  
    problem.    Simple removal was not discussed in 3.9.2 because  
    no  examples were found which involved removal  of  redundant  
    elements alone. 

           The  learner's task in a removal is to determine what  
     needs to be removed and to indicate this.   No new REMs are  
     required for this operation,  which can be performed either  
     as a fill-in/short answer or as a multiple-choice  response  
     to an explicit question or instruction.    In 3.13(Dacanay)  
     the  initial  insertion  of  the  cue  and  the  subsequent  
     identification of redundant elements would have to be  done  
     separately.  The second operation could be presented in the  


          I:    TTT      Indicate the words to be removed
          M:+Rd mmm      The noisy children who were making a lot  
                         of noise were sent to bed
          ?:   ?->rt

          I:    TTT      Indicate the range of words to be removed
          M:+Rd mmm      The noisy children who were making a lot  
                         [1] [2]   [3]      [4] [5]  [6]   [7][8] 
                         of noise were sent to bed.
                         [9] [10] [11] [12][13][14]
          ?    ?->rm

           The  removal  portions of the rest of  the  examples,  
     3.16(Dacanay), 3.30(Rivers), and 3.131(Stack), could all be  
     handled in the same way.

           Robinett provides the only examples of re-order, then  
     production  of new elements.    These exercises,  3.44  and  
     3.45 are naturally two-step operations and would respond to  
     being handled as two separate simple drills.

           While  there are two factors involved in the response  
     to  exercise  3.146(Stack),   it  would  best  be  treated,  
     presentationally  as  a  simple   production-of-new-element  
     drill  with  a  CFF such as "Answer the  question  using  `  
     always  '  ".    The answer analysis and feedback could  be  
     addressed  towards separate consideration of   the  correct  
     formation of a response and the correct use of "always". Choice of template

           Exercises  3.27(Rivers)  and  3.130(Stack)  are   not  
     complex in themselves, but require a more complex authoring  
     system,  in that different items require the application of  
     different simple templates.    Conceptually,  this does not  
     seem  difficult.    Each   discrete  drill  item  could  be  
     flagged  as  to which template should be  applied.    These  
     have  been considered as separate category and  treated  as  
     complex drills because many authoring systems are incapable  
     of switching templates within a sequence.   

     3.10   Factors  influencing the answer-analysis phase of  a  

           There  are  several factors influencing  the  answer- 
     analysis  phase  which  must be  examined  before  possible  
     answer analysis templates for the drills in the data can be  
     postulated.    The content of the drill, largely irrelevant  
     to the choice of P-template,   becomes an important  factor  
     in  answer-analysis as it directly determines the response- 
     set  and  the answer-set of  a  drill.    Response-set  and  
     answer-set must be defined and their expression in the data  
     examined.   The  approaches to answer-analysis currently in  
     use,  or proposed for CALL, must be discussed and those not  
     applicable  to  the  current  discussion  disgarded.    The  
     concept  of response possibility must be defined,  and  the  
     effects of the response-elicitation technique chosen for  a  
     particular  presentation template on the range of  response  
     possibilities    examined.     Finally,    techniques   for  
     controlling  the  range of response  possibilities  require  

     3.10.1  An Examination of the response-sets in the data

          The   response-set  can  be  considered  the  set   of  
     predictable  or  expected responses which derive  naturally  
     from  the  content  of the stimulus,  and  from  which  the  
     learner chooses the correct response.    If,  for  example,  
     the  stimulus calls for the learner to provide the  correct  
     subject  pronoun,  then it would be reasonable to  consider  
     the  response-set  to be the set of  all  subject-pronouns.    
     Responses  that  were  outside the  response-set  would  be  
     invalid  or  unrecognizable  responses rather  than  simply  
     wrong  responses.    The  symbol  {} has  been  adopted  to  
     designate the concept of a set,  and  {R} or {r} to  denote  

               The nature of the response-set affects mostly the  
     type and range of feedback that can be given to the learner  
     after  an incorrect response.  Where there is no  response- 
     set,  for example,  there are only two possible outcomes of  
     the  answer  analysis:   the response matches  the  correct  
     answer or it does not.   With a minimal response-set, three  
     outcomes at least are possible:  correct;  not correct, but  
     within the response-set; and not within the response-set.  

          A response-set can be universal, {R}, meaning that all  
     items  draw  their responses from the same  set,  or  item- 
     specific,  {r},  meaning  that each drill item has its  own  
     unique response-set.   In 3.5(Dacanay), for example, all of  
     the  items  relate  to  the  choice  "do/does",   while  in  
     3.142(Stack),  item  one relates to the choice "read/reads"  
     and item two to the choice "eat/eats".   When the response- 
     set  is  universal,  it can be larger and  more  structured   
     than when it is item-specific. 

          The response-set can also be implicit or explicit.  In  
     the  latter case it will be apparent to the  learner,  from  
     the  nature of the task or from the  presentation  template  
     chosen,  what  the response possibilities  are.   Multiple- 
     choice  activities,  for example,   will automatically have  
     explicit response-sets.    The symbol #{r} denotes explicit  

               There are some variables which are particular  to  
     the multiple-choice mode.   With multiple choice the actual  
     response-set differs from the true response-set.    {R} may  
     be  {do/does},  but  the learner's response options may  be  
     {a/b} or {1/2}.   We will symbolize the actual response-set  
     as @{R} to differentiate it from the true response-set {R}.    
     The explicit response set may encompass all of the natural,  
     implicit,  response-set, @r = {r}, or it may only represent  
     a manageable portion of the larger response-set, @r < {r}.

          The size of the natural response-set and its degree of  
     "closedness" is determined by the content of the  exercise.    
     Where  the natural response set is larger than desired,  it  
     can   be  reduced  by  the   presentation   template.    In  
     3.5(Dacanay), for example, the prompt:

                 P:  Where does Mary live?

       reduces  the  possible forms that could be inserted  into  
     the model:

                 M:  Where  []  Jose live?

     With  the prompt,  {R} = {do/does},  without the prompt  it  
     would  be  necessary to include {can/ might/  may/  should/  
     would/  shall/ ...}.    In cases where there is no  natural  
     limit  on the number of  plausible,  expectable  responses,  
     where there is, in other words, no finite response-set,  an  
     artificial limit may be imposed by using multiple-choice.

          The  response-set  can  be of very  limited  in  size,  
     containing perhaps from two to half a dozen elements.  This  
     can  be called a "very small" response-set and can be shown  
     symbolically as:

               {r} = {aaa/bbb}
               {r} = {aaa/bbb/ccc}
               {r} = {aaa/bbb/ccc...}

               where  aaa  represents the correct  response  and  
               bbb,  ccc  represent  incorrect  but  predictable  

     3.61(Cook),  for example, has an item-specific response set  
     of   two:    {r}   =  {aaa/bbb}  =   {swimming/   skating}.    
     3.132(Stack)  has a universal response set of four:  {R}  =  
     {aaa/bbb/ccc...} = {will/ 'll/ would/ 'd}.   3.62(Cook) has  
     one of seven elements,  the days of the week.    While  all  
     the  size  terms  to  be  discussed  are  relative,   those  
     response-sets  where  all the elements could  be  displayed  
     simultaneously  for  the  learner,  if  desired,  could  be  
     considered  as "very small".   In cases of multiple-choice,  
     the explicit response set,  #{r},   would equal the  actual  
     response set {r}.

         Where  the  set of responses is larger than what  could  
     comfortably be displayed to the learner,  but still  small  
     enough  to  be  handled  without  resorting  to  techniques  
     designed  for  large  databases,  it can be refered  to  as  
     relatively "small" and be represented by the symbols: 
               {r} = {aaa/bbb/.../nnn}

     In 3.118(Stack), for example, the possible re-statements of  
     "il   se  repose"  is  limited,   but  still  represent   a  
     substantial number.  Similarly, in 3.144(Stack), the number  
     of ways, correct and incorrect, to respond affirmatively to  
     the question "Do you like music?" is considerable.

           A  patterned or structured response-set is one  where  
     types of responses are grouped together.   In 3.117(Stack),  
     it  is  possible  to phrase responses that  illustrate  the  
     choice of the correct,  but improperly formed,  tense, "est  
     commande";   the wrong tense,  "commandait";   or  improper  
     verb agreement,  "avez commande".    While the total number  
     of  possible  responses may be considerable,   they can  be  
     categorized into a few general types of error.   Errors  of  
     the same type can be treated in a similar manner. 

         Where the set of responses to too large to handle using  
     normal techniques,  but is still, nevertheless, finite,  it  
     can be termed relatively "large" and represented as: 

               {r} = {database} 

     In  the original,  non-computerised execution of the drill,  
     the learner would have to choose from among a large set  of  
     possible  responses stored as part of "general  knowledge".    
     On the computer,  database storage and retrieval techniques  
     would  be required,  to determine if a response were in the  
     acceptable  set.    In 3.29(Rivers),  a database  would  be  
     postulated  containing  a  large list of names  of  objects  
     which  for which "it" could substitute,   a list of  proper  
     names for which "her" could substitute, and so forth.  


           Table  3.12  examines the variations in  response-set  
     that are evident in the data.   Almost all of the  examples  
     exhibit  closed response-set, although there is no apparent  
     pattern  in  the  variation  between  universal  and  item- 
     specific  response-sets.  3.66(Cook)  and 3.111(Stack)  are  
     shown as having no finite response-set because the  desired  
     response  is not part of a larger set.   The insertions  of  
     Type E,  3.32(Rivers)  and 3.36(Rivers),   and some of  the  
     production-of-new-element      drills,       3.31A(Rivers),  
     3.37(Rivers),  and  3.39(Rivers),  have  response sets  too  
     large to be defined.  All the drills classed as re-order of  
     model elements seem,  by their nature, to exhibit universal  

          Most  of the insertions of type A and all of  the  re- 
     order  drills  exhibit inherently explicit  response  sets.   
     Explicit  response sets will have to be very small and  so,  
     in  the table,  a star was not placed in column 5 if  there  
     was one in column 4.   Taking columns 4 and 5 together,  we  
     can  see that most of the drills in the data,  both  simple  
     and complex,   with the exception of those that involve the  
     production of new elements, have very small response sets. 

     3.10.2  An examination of the answer-sets in the data

         The answer-analysis operation involves more than simply  
     determining if the learner's response matches the "correct"  
     response.  The the learner's response must be identified by   
     matching  it  against one of the elements in the  response- 
     set,  of  which  the "correct" response represents  only  a  
     single possible case.      For those cases where there is a  
     degree  of  correctness,  perhaps  a "best"  answer  and  a  
     "second-best"  answer,  the  approach is to  consider  this  
     situation  as identifying a correct answer and one  of  the  
     "incorrect", but predictable answers. 

          From  the definition given of a CALL exercise,  it  is  
     possible  to  define at least one "correct"  or  acceptable  
     response,  which can be symbolized as  aaa  or  AAA.  It is  
     also necessary to assume a finite set of correct responses,  
     because   an  infinite  set  be  equivalent  to  having  no  
     "correct"  (as opposed to "incorrect) responses at all  and   
     anything   the   learner  supplied   would   therefore   be  
     acceptable.     Normally  the  correct  response  would  be  
     assumed  to  be item-specific,  aaa,  although a  universal  
     correct response, AAA, exists in one example from the data,  

         There may be a single correct response,   aaa, or a set  
     of  correct responses {a}.    As in the case  of  response- 
     sets,  considered above,  where there is {a}, it may either  
     be a "very small" set:
                                 {a} = {aa1/aa2}
                                 {a} = {aa1/aa2/aa3}
                                 {a} = {aa1/aa2/aa3...}

     a relatively "small" set:
                                 {a} = {aa1/aa2/.../aaN}

     or a relatively "large: set:

                                 {a} = {aa1/aa2/.../aaZ}

           The  data  is examined in table 3.13 with  regard  to  
     variations  in answer-set.   Most of the examples have only  
     one  correct  response  per item.   To  single  out  a  few  
     examples  of  a very small answer-set,  it is  possible  to  
     mention 3.11(Dacanay), where "now" can be inserted into one  
     of  two slots;   3.120(Stack),  where the form can be  "are  
     not" or "aren't";   or 3.144(Stack), where the response can  
     be "do",  "like it", or "like music".   A larger, but still  
     relatively  small answer- set can be found in examples such  
     as  3.31A(Rivers),  where  one can expect answers  such  as  
     "George,  my name is Ronald.",  "My name is Ronald.",  "I'm  
     Ronald.",   etc.   Large  answer-sets  are  exemplified  by  
     exercises  such  as  3.32(Rivers),   where  virtually   any  
     grammatically   and   semantically  correct  insertion   is  
     acceptable,  or  3.29(Rivers),  where  any  noun  that  can  
     replace "it" is acceptable.

         The  set  of  {a}  may  represent  discrete,   separate  
     elements  or  may simply be alternative forms of  the  same  
     basic  element.   The  latter case is  represented  in  the  
                              {a} = {aax/aay...}

     This  can  be  exemplified in  3.34(Rivers),  where  {a}  =  
     {"would"/ "'d"}, both of which are forms of the same word.

         For most cases, the correct answer can be  predicted by  
     the  author  and  entered  into the  data  as  a  separate,  
     discrete element:
                             aaa < data

     Where the response-set is structured and relatively  small,  
     it  may be possible to generate the correct answer based on  
     other elements in the data, such as the cue, in combination  
     with a universal rule or set of rules:

                            aaa < RULE

     This  possibility may be used,  even where not  needed,  to  
     save  time  for   the author and to  save  on  data-storage  
     space.   Such cases are not treated here. In 3.63(Cook) and  
     3.114(Stack), however, answer- generation by rule is liable  
     to be the preferred avenue,  because of the ease with which  
     it could be handled by the computer.

           There are other instances, however, where the correct  
     response cannot be predicted, but may depend on some choice  
     that  the  learner has already made.   Two answers  may  be  
     "linked"  in such a way that the first response  determines  
     the second.   In 3.28(Rivers),  for example,  the choice of  
     insertion slot made by the learner determines which form of  
     the answer is correct.

           Response-processing  can  be holistic and attempt  to  
     relate the learner's response,  as a whole,  to the correct  
     response.   Alternatively,  it  can  be partial  and  match  
     pieces  of  the  response against  pieces  of  the  correct  
     answer, to produce feedback based on the discrete elements.   
     With  longer and  more complex responses,  partial analysis  
     becomes  the only feasible avenue for analysing  a  textual  
     response.   In  3.120(Stack)  processing the response on  a  
     word-by-word  basis  would allow the  computer  to  realize  
     that,  in  a  case of "not are",  the words  are  reversed,  
     whereas  holistic processing would simply see the  response  
     as wrong.   In cases such as 3.36(Rivers),  if performed in  
     the  fill-in mode,   holistic matching is not possible,  as  
     the entire learner-response cannot be predicted.   All that  
     is known is that the first word may be either "not" or "to"  
     and  that if the first word is "not",  then the  next  word  
     should be "to".

           With  simple one-word reponses,  partial analysis  is  
     probably not required.   Indeed,  in cases such as the  re- 
     order  of two elements,  partial-response analysis may  not  
     be possible. 

               In  the discussion of response  sets,   @{R}  was  
     defined  as the actual response-set as opposed to the  true  
     response-set {R}.   Similarly,  the symbol @a is defined to  
     stand for the actual response to a multiple-choice.   Thus,  
     in  3.6(Dacanay),  the  learner's choice of [2] stands  for  
     "`tonight'   replacing  `tomorrow'  in  the  slot   between  
     `leaves' and `at seven'".

     3.10.4    General   approaches  to  computer-based   answer  

           Pusack(1983)   outlines  five  basic  mechanisms  for  
     answer  analysis  in  CALL:    Non-evaluation,  right/wrong  
     evaluation,   pattern  mark-up,   error  anticipation,  and  

          This author (Pusack,  1983, p.55) finds non-evaluation  
     to   be   a   powerful   pedagogical   approach   in   many  
     circumstances.    Typically,   the  learner  is  asked   to  
     formulate  an  answer  mentally,   the  correct  answer  is  
     displayed,  and  the  learner is asked to compare the  two.   
     For free-form responses,  where only a likely model for the  
     correct answer can be provided,  non-evaluation may be  the  
     only  viable  technique.   The  disadvantages  lie  in  the  
     possibility  that the learner may not detect an  error,  or  
     may not understand an error that has been detected.   There  
     is   very  little  potential  in  these  circumstances  for  
     computer-controlled     branching.        The      rigorous  
     interpretation of CALL activities used for this study calls  
     for  them  to be mediated by the computer,   and  therefore  
     self-mediation or non-evaluation must be excluded from  the  

               Right/wrong  evaluation involves the matching  of  
     the learner's response against a "correct" response, or set  
     of  "correct" responses.   Techniques can be applied  which  
     reduce  mismatches  due  to deviations such  as  errors  in  
     spacing,  capitalization,  punctuation,  and minor spelling  
     variations (Pusack, 1983, p.55-56).  

          Pattern  mark-up involves a non-linguistic matching of  
     the   learner's  response  against  the   response   model.   
     Missing,  extraneous  or  misplaced  letters or  words  are  
     indicated,  using  some  system  of  proofreading  symbols.   
     Pusack(1983,  p.61) offers the following illustration  from  
     the PLATO system:

          Response Model:     
          The quick brown fox jumps over the lazy dog.

          Learner's response and mark-up:
          the brown quick fox jumpd oaerr the big lazy Dog.
          *  ^      <         = = = xxxxx         xxxx *

               where  *  stands for     capitalization errors.
                      ^                 missing element
                      <                 element to shift leftward
                      ===               mis-spelled word
                      xxx               unrecognizable  or  extra  

          While  a  pattern mark-up algorithm can be applied  to  
     fill-in and multiple choice exercises, it should be evident  
     that  its true power lies in dealing  with  sentence-length  
     free response.   

          Several  serious limitations to pattern-mark  up  have  
     been  identified,   some of which stem from the  mechanical  
     nature  of  the method of  analysis  involved.   Word-order  
     errors  are  described  without  reference  to  grammatical  
     norms.  Inflection errors,  root spelling errors, diacritic  
     errors  and  run-on  words are all described in  a  similar  
     manner  (Hart,  1981,  p.9).   Since there is no  syntactic  
     model within the computer,  no explanation of an error  can  
     be  given  to the learner (Pusack,  1983,  p.59).    It  is  
     difficult  to imagine how the information obtained  from  a  
     pattern  mark-up  analysis  could  contribute  to  computer  
     controlled    branching,    modelled   on   the   learner's  
     performance.    Furthermore,  the  algorithm  itself  often  
     manifests  shortcomings.   Morphological analysis tends  to  
     take  precedence  over syntactic  analysis  (Pusack,  1983,  
     p.59).   The  resulting  mark-up  is based  on  the  hidden  
     response model,  often betraying the correct syntax,  which  
     may  be  the  point  of  the  exercise.    Pattern  mark-up  
     algorithms tend to "hypercorrect",  registering elements or  
     patterns  which were not intended by the  learner.   Pusack 

          Response Model:     Ich gehe in die Stadt.
          Learner Response:   Ichn gehe die Stadt.
          Mark-up:            Ich< gehe ^n die Stadt.

     where  the  algorithm picks up the extra "n" at the end  of  
     "Ich"  and interprets it as part of the missing word  "in".   
     The  possibility  that  the the learner  simply  forgot  to  
     include  "in" seems more likely than the possibility   that  
     it  was  mispelled and placed at the end  of  the  subject,  
     "ich".    Finally,  pattern-mark  up assumes that a  static  
     response   model   can   be  predicted.     There   is   no  
     accommodation built in for cases in which, for example, the  
     choice of a word in one part of the sentence pre-determines  
     the choice of a word later in the sentence.

          For  the purposes of the present survey,  the computer  
     is allocated the role of surrogate teacher.   Pattern mark- 
     up  does not offer the subtlety of analysis that  could  be  
     displayed  by  a  human  instructor.     Furthermore,  with  
     authoring systems that are based on this approach, there is  
     no  mechanism  for  teacher-authors to encode  and  include  
     their expertise.   It must be recognized that pattern mark- 
     up  is a viable technique for authoring systems  to  employ  
     for answer-analysis.   It must be excluded,  however,  from  
     consideration here,  as the model being developed cannot be  
     applied to it. 

          Pusack's  (1983,   p.   61)  fourth  category,   error  
     anticipation,  is  an extension of right/wrong analysis and  
     deals  with possible avenues to follow,  once it  has  been  
     determined   that  the  learner's  response  is  incorrect.    
     Essentially,  an  attempt  is made to match  the  learner's  
     response  against one of a list of expected wrong  answers.   
     For each wrong answer there may be a particular  diagnostic  
     message as feedback, or a particular branch to another item  
     may be made.  In keeping track of error types, the computer  
     may  be  building  up  a  model  of  the  learner's  global  
     performance,  which can also be used to initiate particular  

          Pusack (1983,  p. 61) draws attention to two drawbacks  
     to this technique.   First,   it is necessary to encode all  
     of  the possible errors that may occur,  a task  which  the  
     author may find unmanageable.   Second, a bold leap is made  
     from  the  existence  of an error to  its  probable  cause,  
     assumed  in the explanation for the error which is provided  
     as feedback.   Such explanations must be carefully designed  
     and must take into account all possible causes for a  given  

          The  final  answer-analysis technique,  that  of  full  
     grammatical and semantic parsing of the learner's response,  
     will   eventually   overshadow  all   the   others.    Some  
     theoretical point in the future can, perhaps, be postulated  
     in  which  the computer will have the same  linguistic  and   
     pedagogical expertise as the human teacher,  at which point  
     authoring systems will no longer be necessary.   Currently,  
     however,  parsing  is an imperfect technique,  and requires  
     resources beyond what would normally be available in a CALL  
     environment (Pusack,  1983, p.63).    Pusack coins the term  
     "Pseudo-parser"  to cover algorithms which match  parts  of  
     the  learner's response against parts of the response model  
     or models,  providing some insights into the nature of  the  
     differences that become evident.

          To   exclude  parsers  from  the   discussion,   while  
     including  "pseudo-parsers",  an arbitrary distinction must  
     be  made.  The term "parser" is reserved for any  algorithm  
     which  must grammatically and semantically label  elements,  
     in order to perform the analysis.   Pseudo-parsers will  be  
     able  to perform their analysis by matching elements of the  
     learner's  response against lists of possible  matches  for  
     the same slot.   Pusack's example, introduced above, can be  
     used  to illustrate the two:

                Given  a learner's response:       Ich gehe  die  
          Stadt.,    a  parser  would  begin  by  labelling  the  
          elements as Subject+verb+object.  It would tag "gehen"  
          as a verb of motion and realize that it could not take  
          a  direct object.   Of the list of  prepositions  that  
          could  follow "gehen",  only "in" could collocate with  
          "die Stadt".  It would then signal the missing "in".

          The pseudo-parser would have a stored response model:


          The learner's input would be broken down into words: 


          The computer would then register that there was a word  
          missing.  It would find that L:(3) did not match M:(3)  
          and L:(4) did not match M:(4).    A trial  phase-shift  
          of  one  element would obtain a match for L:(3+1)  and  
          L:(4+1).  The missing element would then be identified  
          as M:(3) or "in".

          In  the relatively closed,  structured environment  of  
     the  CALL  exercise,  as it has been defined,  the  pseudo- 
     parser  can appear to the learner to possess a  grammatical  
     "understanding"  of  the  input  and  to  have   para-human  
     flexibility.    As  the  environment becomes more open  and  
     less structured,  the pseudo-parsing technique becomes  too  
     complex  and must be replaced by a true  parser.   Assuming  
     that most authoring systems are incapable of providing that  
     option,   the  author must introduce "structure" of a  non- 
     linguistic  nature  by  forcing the activity  into  another  
     mode, such as multiple choice.

           In  the  discussion of answer-analysis,  there  is  a  
     concentration  on what Pusack terms  "error  anticipation",  
     realizing  that what he calls "right/wrong" analysis may be  
     the  only  applicable  approach with drills  that  have  no  
     finite  response-set or with authoring systems that do  not  
     allow for error-anticipation.    The pseudo-parser approach  
     essentially  emerges as a variation of  error-anticipation.   
     Instead  of  examining  the  response  holistically,   each  
     component  is treated separately,  but still in  an  error- 
     anticipation aspect.

     3.10.5   The effects of variation in response-possibilites,  
     and preprocessing of input
           The  open/closed nature of the response- and  answer- 
     sets  of  activities is largely an inherent  feature  which  
     does  not  depend  on  the  medium  of  presentation.    An  
     entirely  separate variable is the open/closedness  of  the  
     range  of response possibilities.   While the  response-set  
     depends primarily on the content of the drill, the range of  
     response  possibilities  is a function of the  presentation  
     template  and  of  the  response-elicitation  technique  in  
     particular.    Only  to  the degree that these  affect  the  
     presentation template do the form and content of the  drill  
     influence the range of response possibilities.

          The   nature   of  this  new  variable  may  best   be  
     illustrated with an example.   Consider possible  responses  
     to  a  simple yes/no question.   From the point of view  of  
     content, such a question is completely closed. The response  
     set contains only two elements,   "yes" and "no",  only one  
     of which can be correct.

          If the question is posed as follows, allowing for free  
                    Is that correct ?

     the   range  of  possible  responses  is  wide,   and   the  
     corresponding complexity of the answer-analyzing  algorithm  
     will have to be very great.  Some possible responses are:

               sure, I think so, right on
               I don't think so, I think not, no way
               no, nno, No, NO.

          The  range  of  response  possibilities  would  render  
     virtually   useless  an  attempt  to   directly  match  the  
     response against the words "yes" or "no".   If the response  
     does not match with "yes",  the conclusion that the learner  
     intended to signal "no" does not follow automatically.

          The  application  of  pre-processing  techniques   can  
     greatly  reduce  the  range of  response  possibilities  by  
     cancelling out,  or preventing,  mechanical variations such  
     as upper/lower case and mis-spellings.

          One of the most important such techniques of this kind  
     is that of converting all letters to a single case,  either  
     upper or lower.   YES,  yes, Yes, yES, etc. can then all be  
     treated as YES or yes. 

          The  computer's reaction to extra spaces in the  input  
     frustrates many learners.   A computer may find that " YES"  
     does  not match "YES".    Many programming environments are  
     sufficiently  sophisticated to ignore leading and  trailing  
     blanks,  but  extra  blanks between words can  still  cause  
     problems.   If the desired response is "aren't you?",  then  
     "aren't   you?" is likely to be rejected.  A pre-processing  
     algorithm  is required that can strip leading and  trailing  
     blanks and reduce inter-word blanks to single spaces.  (The  
     problem  posed  by "aren't   you" can also  be  dealt  with  
     using partial processing, as is demonstrated below.)

          The problem of spelling errors and alternative   forms  
     may  be  dealt with during the actual  answer-analysis.   A  
     pre-processing  alternative,  however,  is  to  freeze  the  
     keyboard and only allow certain letters to pass.    In  the  
     "yes/no"  example  above,  the letters  Y,E,S,N,O could  be  
     programmed as the only active letters.   The learner  could  
     type  "Yesno",  but would be unable to type "I think so" or  
     "no way!".

          All   answer-analysis   by  computer   is   ultimately  
     performed  by  determining  if  two  elements  match.   The  
     subtlety  with which this basic operation can be  completed  
     affects  the  range of response possibilities that  can  be  

          Where  the learner's response can be  matched  against  
     more  than one possible element,  variations such as "YES",  
     "SURE",  "SI",  "OUI",  "JA" etc. can be accepted.  This is  
     not  a  good approach for handling the  majority  of   mis- 
     spelling  situations,  as  all possible combinations  would  
     have to be predicted in advance:   yes,  yyes,  yees, yess,  
     yesss, ysys, etc.

          Many authoring systems allow for some degree of "fuzzy  
     matching".   The simplest approach  is to look at only part  
     of  the  learner's response as compared with  the  relevant  
     part of the model.  In the "yes/no" example,  the  computer  
     could compare the first letter of the response with "Y" and  
     thereby  accept  a much wider range of responses  as  being  
     affirmative.  Some systems allow for "dummy" or "wild card"  
     characters to be placed in the answer, allowing for a match  
     with   "Y##".    Other  systems  allow  can  calculate  the  
     percentage  of agreement between the two  elements.   Thus,  
     "EES",  which the algorithms shown above would miss,  would  
     be  found  to have 66% compatibility  with  "YES".   Pusack  
     (1983,  p.  56)  discusses pseudo-phonetic  fuzzy  matches,  
     whereby  "ROOSEVELT"  could be matched with  "ROSEVELD"  or  

          Fuzzy  matching  is a useful tool for suppressing  the  
     errors due to  spelling variations that so often  frustrate  
     learners  are  working  on the  computer,  but  it  visibly  
     demonstrates that the surrogate cannot entirely replace the  
     teacher.  The  technique is useful only in some situations,  
     as there will always be a certain amount of uncertainty and  
     guesswork  involved in planning the outcome to a  response.   
     A simple fill-in exercise illustrates this point:

               C:   talk
               M:   He ----.
               ?:      ?     

     The learner's response, ???, must now be identified.  Using  
     wild-card  characters  for  the  storage  of  the   correct  
     response,  the  situation  could arise where the  learner's  
     response,   ???="tawks", would be considered "correct" when  
     matched with the correct answer, aaa="####s",  while "talk"  
     would be considered incorrect. 

          However     sophisticated    the     response-analysis  
     techniques,   it is impossible to preclude the "grey  area"  
     of  learner  input that cannot be  identified.   While  the  
     handling  of this "grey area" in a pedagogically acceptable  
     fashion  can be considered part of the skill  in  designing  
     CALL activities which authors must possess,  their task can  
     be made easier on systems where uncertainty is reduced to a  

          The  "grey area" phenomenon can cause the computer  to  
     make "mistakes" in analysing learners' responses to certain  
     stimuli.     In  other  situations,   authors  must  resign  
     themselves to the fact that certain parts of the  learners'  
     responses simply cannot be analyzed.  

          One  avenue towards solution is to reduce the range of  
     response  possibilities.  The  question  from  the  earlier  
     example could be posed as follows:

               Is that correct (Y for yes/ N for no) ?

     reducing the response possibilities to two,  a closed  set.   
     Any   input  that  does  not  match  the  set  of  response  
     possibilities  can be immediately rejected,  regardless  of  
     its  acceptability or non-acceptability from the  viewpoint  
     of content.

          Multiple-choice  type  questions  inherently  have   a  
     closed set of response possibilities.   Yes/no, true/false,  
     yes/no/maybe  and  other similar  question formats  can  be  
     considered as variations on multiple choice.

          Even in situations where the student is called upon to  
     type an entire sentence,  the set of response possibilities  
     can   be   considered   as   closed.    Compare:   response  
     possibilities.  Consider:

     (a)          Write these words in the correct order:

                     BOOK  GIVE  I  THE  HER

     and (b)      Write  the numbers in the order  corresponding  
                  to the correct sentence

                     BOOK  GIVE  I  THE  HER
                     [1]   [2]  [3] [4]  [5] 

     The only feature that format (a) from format (b), above, is  
     the  additional  possibility  of  errors  in  spelling  and  
     reproduction  in the former case.  The response-set can  be  
     considered closed when it is possible to determine  whether  
     the  answer  is composed of the required,  albeit  possibly  
     malformed, components.

          As  illustrated in the "yes/no" example shown earlier,  
     the complexity of any response,  from the point of view  of  
     open/closedness  of  content,  can be further increased  by  
     allowing  a  looser range  of  response  possibilities,  or  
     simplified by reducing the range.   

          The  availability of some of the techniques  described  
     above  is  an  important consideration in  discussing  CALL  
     applications  of an authoring system.    These are  treated  
     separately,  however, and for the rest of this analysis, it  
     is  assumed  that the applicable pre-processing  and  fuzzy  
     matching techniques are available. 

    3.10.6   Basic answer analysis templates

           It has been indicated above that while the choice  of  
     P-template    for   particular   exercise   is   relatively  
     independent of content,  the choice of answer-analysis-and- 
     reaction  template,  or  A-template,  depends  both  on  P- 
     template requirements and upon content,  in particular upon  
     the response-set, {r}, and the answer-set {a} .

          In  the  examination of  A-templates in terms  of  the  
     data required, the number and type of operations, and range  
     of  possible outcome situations that can be identified,  it  
     will be assumed that some reaction to all learner input  is  
     involved.   The  reaction  may take the form  of  immediate  
     feedback,  in  terms  of  a  message,  or  some  particular  
     branching decision may be made.   Reaction to any situation  
     requires,  first  of  all,  that the A-template  allow  the  
     computer to realize that the situation exists.

          The  link  between P-template and  A-template  is  the  
     response-elicitation mechanism or REM.   The REM will also,  
     to   some   extent,   determine  the  range   of   response  
     possibilities.   It can be assumed that the multiple choice  
     REM's ( ?->rm, ?->rl=m, and ?->ro ) lead to a closed set of  
     response  possibilities  while the fill-in REM ( ?->rt) and  
     the  free  response  REM ( ?->s) lead to  an  open  set  of  
     response possibilities.

          Wherever   applicable,   the  existence  of  the  pre- 
     processing  and  fuzzy  matching  techniques  described  in  
     3.10.5  is  further assumed.   Only valid  multiple  choice  
     responses will be considered.   In the case of fill-in, the  
     assumption  is that all responses are in the  proper  case,  
     that  there  are no extra spaces,  and that minor  spelling  
     errors  are excluded.    For free-response,  it is  assumed  
     that  the  response  has  already  been  broken  down  into  
     discrete words for partial processing.

          The symbol,  ???,  is used to stand for the  learner's  
     response and the symbols,  ?1?, ?2?, ?3?, ... ?n?, to stand  
     for  that  response,   broken up into discrete word  units,  
     where  applicable.    The  symbol   @?  will  stand  for  a  
     multiple-choice response.

          The simplest A-template that can be postulated is  the  
     algorithm  required to handle the situation where there  is  
     no  response  set,  {r} = 1,  and where there is  a  single  
     correct answer,  {r} = {a} = aaa.   In this case, aaa would  
     have  to come from data.   There would be only two possible  
     outcomes, ??? would or would not match aaa.  The A-template  
     could therefore take the form:

          A-template #1

          ?:   ?->rt --->???
          R:   {r}=1
          A:   {a}=aaa  <---data

          MO:  1.   If aaa=??? then situation I
                               else situation II

          F:   Situation I:    Learner's response is correct.
               Situation II:   Learner's response is unrecognized  
                              (and presumed incorrect).

               where:    ?:   REM leading to response
                         R:   Response Set
                         AGO: Answer  Generating Operations  (not  
                              present in this case).
                         A:   Answer Set
                         MO:  Matching Operations
                         F:   Situations  leading to feedback  or  
                              other reaction              

         Where there is a response-set of two elements,  another  
     possible outcome is added:

          ?:   ?->rt --->???
          R:   {r}={aaa/bbb} <---data
          A:   {a}=aaa       <---data

          MO:  1.   If aaa=??? then situation I
                               else 2.
               2.   If bbb=??? then situation II
                               else situation III    

          F:   Situation I:    Learner's response is correct.
               Situation II:   Learner's response is incorrect.
               Situation III:  Learner's response is

          Additional  elements  in  a very  small  response  set  
     simply multiply the number of operations and the number  of  
     possible outcomes:

          A-template #3

          ?:   ?->rt --->???
          R:   {r}={aaa/bbb/ccc/ddd} <---data
          A:   {a}=aaa               <---data

          MO:  1.   if ??? is element of {r} then 2.
                                             else situation III
               2.   If aaa=??? then situation I
                                    else 3.
               3.* ({r}-1)     If bbb=??? then
                                           situation II * {r}

          F:   Situation I:    Learner's response is correct.
               Situation II * ({r}-1):
                               Learner's response,  nnn,  is  
                              incorrect  because...reason(nnn)
               Situation III:  Learner's   response  is


           Multiple-choice  activities  inherently have  a  very  
     small  response-set,  so the answer analysis would be  very  
     similar  to the above,  with the difference that much  less  
     pre-processing  of the learner's response would be required  
     and the "grey area" would be reduced to nil.

          A-template #4

          ?:   ?->rm --->@?
          R:   @{r}={@a/@b/@c/@d}    <---data
          A:   @{a}=@a               <---data

          MO:  1.   If @a=@?   then situation I
                                    else 2.
               2.* (@{r}-1)    If @b=@? then
                                        situation II * @{r}

          F:   Situation I:    Learner's response is correct.
               Situation II * (@{r}-1):
                               Learner's response,  nnn,  is  
                              incorrect  because...reason(nnn)

           Cases  where  there  are  multiple  correct   answers  
     forming  part  of  a very small answer-set  introduce  only  
     slightly more complexity.     

          A-template #5

          ?:   ?->rt --->???
          R:   {r}={aa1/aa2/aa3/bbb/ccc/ddd} <---data
          A:   {a}={aa1/aa2/aa3}             <---data

          MO:  1.   if ??? is element of {r} then 2.
                                             else situation III
               2.   If ??? is element of {a} then situation I
                                             else 3.
               3.* ({r}-1)     If bbb=??? then
                                           situation II * {r}

          F:   Situation I:    Learner's response is correct.
               Situation II * ({r}-1):
                               Learner's response,  nnn,  is  
                              incorrect  because...reason(nnn)
               Situation III:  Learner's   response  is

           Where there are two responses elicited,  however, the  
     complexity increases greatly:

          A-template #6

          ?:   ?->rt1,rt2 --->?1?,?2?
          R:   {r1}={aaa/bbb/ccc/ddd} <---data
               {r2}={aaa/bbb/ccc/ddd} <---data
          A:   {a1}=aaa               <---data

          MO:  1a.  if ?1? is element of {r1} then 2a.
                                             else situation IIIa
               2a.  If aaa=?1? then situation Ia
                                    else 3a.
               3a.* ({r1}-1)     If bbb=?1? then
                                           situation IIa * {r}

               1b.  if ?2? is element of {r2} then 2b.
                                             else situation IIIb
               2b.  If aaa=?2? then situation Ib
                                    else 3b.
               3b.* ({r2}-1)     If bbb=?2? then
                                           situation IIb * {r}

          F:   Given  the  same  situations as  in  the  previous  
               templates,    the   complexity  of  the   feedback  
               increases radically:

               Situation Ia + Situation Ib
               Situation Ia + Situation IIb * bbb
               Situation Ia + Situation IIb * ccc
               Situation Ia + Situation IIb * ddd
               Situation Ia + Situation IIIb
               Situation IIa * bbb + Situation Ib

               In effect,  2+({r1}-1) * 2+({r2}-1) possibilities

     The  range  of  situations requiring  a  reaction  is  much  
     greater.    It  should be evident why the  elicitation  and  
     simultaneous  analysis  of three or more responses has  not  
     been considered a feasible option.

           Templates  #5  and  #6  could  be  combined  so  that  
     multiple correct answers could be accepted,  for either  or  
     both of the responses elicited.   

           The  limits to the ability to check individually each  
     and every element of {a} and/or examine {r} for a  possible  
     match will depend on the authoring system in question.   In  
     making the distinction between "very small" and "relatively  
     small",  a  distinction has been attempted between a  first  
     category  of  cases which it would be reasonable to  expect  
     the  majority  of  authoring  systems  to  be  capable   of  
     handling,  and  a  second  group which only  very  powerful  
     authoring  systems would be expected to handle on the  same  
     pattern.  fashion.    An authoring system called  Ego,  for  
     example, has the power to handle up to 60 discrete matches,  
     each  with  its own particular  reaction  (Peuchot,  1983).    
     Where the response and/or answer set is too large to handle  
     as above, different techniques, such as partial analysis or  
     conversion to multiple choice,  must be explored. 

     3.10.7 Answer-analysis Options in the data:  Simple drills

           On  the basis of factors examined in tables 3.12  and  
     3.13,  it  is apparent that most of the drills in the  data  
     can  be covered by the  basic  answer-analysis-and-reaction  
     templates  outlined in 3.10.6.  For those cases not covered  
     by these basic templates,   more complex templates will  be  
     postulated  below.   As  in  the case of  the  presentation  
     templates,  the components of the answer analysis templates  
     are  the  critical factor to  be  considered.   The  actual  
     global  templates  proposed proposed are likely to be  very  
     specific to particular situations that are presented in the  
     data.  Simple insertions of Type A

           3.6(Dacanay) is typical of many of the insertions  of  
     Type  A  present  in the data,  in that there is  a  small,  
     explicit  response-set  and  a  single  correct   response.   
     Although the answer may be elicited in the fill-in mode (?- 
     >rl=m),  3.6  is essentially a multiple-choice activity and  
     could,   therefore,   be  accommodated  by  the  basic   A- 

          ?:   ?->rm --->@?
          R:   @{R}={[1]/[2]}        <---data
          A:   @{a}=[2]              <---data

          MO:  1.   If [2]=@?   then situation I
                                    else 2.
               2.* (1)          If [1]=@? then
                                        situation II

          F:   Situation I:    Blank  [2] = "...leaves  [tonight]  
                               at seven", which is correct.
               Situation  II:  Blank   [1]  =  "[Tonight]  leaves  
                              tommorrow  at  seven",   which   is  
                              incorrect because...

     Similarly,   3.9(Dacanay),   3.10(Dacanay),   3.21(Rivers),  
     3.42(Robinett),  3.60(Cook),   3.61(Cook), and 3.125(Stack)   
     could  all be handled as above.     3.11(Dacanay) adds  the  
     complication  of the possibility of two correct answers  in  
     some  cases,  and  so  would  require  the  multiple-choice  
     equivalent of basic A-template#5.

           3.4(Dacanay)  and 3.40(Robinett) have a  response-set  
     which  is  very  large.     Theoretically,   such  a  large  
     response-set  could be handled by a computer  program.   In  
     the case of these rather simple activities,  however,   the  
     complex  answer-processor that would result would  probably  
     not  be worth the effort.   More than likely,  an  infinite  
     response-set  would  be assumed and a template such  as  A- 
     template#1 would be applied.

           3.118(Stack),  3.120(Stack),  and  3.144(Stack)  have  
     response  sets that,  while larger than those discussed for  
     cases 3.4 to 3.125 above,  are still relatively small.  The  
     latter  two add the complication of allowing  for  multiple  
     correct responses.    Most authoring systems should be able  
     to  handle these exercises with basic templates such as  A- 
     template#3  and  A-template#5.     The  response-sets  show  
     certain  patterns,  and so the number of different feedback  
     messages required can be reduced to a number far fewer than  
     the  discrete  number  of {r} elements  against  which  the  
     learner's response would would have to be checked. 

          The  responses to these drills could also  be  handled,  
    and perhaps more efficiently,    through  a partial analysis by a  
    pseudo-parser,   as  illustrated  in  3.10.4.   We  can  take  
    3.44(Stack) as an example:


          ?:   ?->rs --->?1?, ?2?...
          R:A: {r}:{a}   word(1)  do,like
                         word(2)  like,it,music
                         word(3)  the,music,a,very
                         word(4)  lot,much,
          AGO: if do(1) then (2)=0
               if like(1) then (2)=it,music
               if it(2) or music(2) then  (3)=0,a,very
               if a(3) then (4)=lot
               if a(3) then (4)=much
          MO:  Check each word.  Set decision flags.
               Evaluate all decision flags and come up with a
          F:   A potentially high number of discrete reactions.

     The above treatment of 3.44 is not complete.  The number of  
     branches and possibilities could be extended to any desired  
     extent.   No  matter how complex the  pseudo-parser,  there  
     would  always be a "grey area" of incorrect responses  that  
     would slip through and of correct responses that would fail  
     to be recognized.    Still, a much larger number of correct  
     answers and error-situations could be handled than would be  
     possible  with  template#5,  given  even  a  very  powerful  
     authoring system.

           While,  in some cases,  each drill item may bring its  
     own item-specific data to the structure established in  the  
     pseudo-parser,   it   is  highly  unlikely  that  the  rule  
     structure itself would be item-specific,  given the  amount  
     of  time  and  effort  that would be  required  to  design,  
     encode, test and elaborate it. 

           With  authoring  systems that could  not  handle  the  
     three  drills above,  with a template such as A-template#5,  
     and which did not offer the possibility of encoding pseudo- 
     parsers,   the only option certain of handling all  correct  
     responses  and  providing adequate feedback  for  incorrect  
     ones  would  be  multiple-choice,    whereby  the  response  
     possibilities would be reduced to a very small set.

               Type  B insertions tend to demonstrate very small  
     implicit  response-sets  and  a  single  correct  response.   
     3.5(Dacanay) provides the typical example,  using the basic  

          ?:   ?->rt --->???
          R:   {r}={do/does} <---data
          A:   {a}=do        <---data

          MO:  1.   If ???=do  then situation I
                               else 2.
               2.   If ???=does then situation II
                                else situation III    

          F:   Situation I:    Learner's response is correct.
               Situation II:   Learner's response is incorrect.
               Situation III:  Learner's response is

           3.35(Rivers),      3.41(Robinett),      3.140(Stack),  
     3.142(Stack),  and 3.143(Stack) are all similar,  though in  
     some  cases the response set is greater than two and so  A- 
     template#3 should be used.    3.34(Rivers) and 3.132(STack)  
     differ  only  in  that  there is  an  answer  set  of  two,  
     requiring the use of a basic template such as A-template#5.  

           The  only  exercise  that  might  deviate  from  this  
     pattern  is 3.117(Stack),  where the learner must supply  a  
     verb form.   This case,  too, could be handled by the basic  
     templates described above.    An analysis by parts might be  
     more  appropriate than a holistic match,  however,  as  the  
     answer clearly has three functional components:   auxiliary  
     + stem + ending.  To cover adequately all possible mistakes  
     by  use of a holistic analysis,  the author would  have  to  
     enter  a  lengthy  list of  possible  variations.   With  a  
     partial  analysis,  an adequate analysis could be provided,  
     even without considering alternative possibilities for each  

          ?:   ?->rs --->?1?,?2?,?3?
          R:A: {r}/{a} =   part(1) = a
                           part(2) = command
                           Part(3) = e

          M:   1.   if ?1? = a  then Situation Ia
                                else Situation IIa
               2.   If ?2? = command  then Situation Ib
                                      else Situation IIb
               3.   If ?3? = e  then Situation Ic
                                else Situation IIc

          F:   Situation Ia:  The auxiliary is correct
               Situation  IIb:   The  auxiliary  is  missing   or  
               Situation Ib:  The participle stem is correct
               Situation IIb: The participle stem is incorrect
               Situation Ic:  The participle ending is correct
               Situation IIc: The participle ending is incorrect.

     Going the extra step of supplying a larger response-set for  
     each of the separate components might allow the computer to  
     generate feedback messages such as:
          Form:      Message:      

          "est"  ->You  cannot use "e^tre" as an  auxiliary  in  
                     this case
          "sav-" ->The past participle has an irregular stem
          "e'e"  ->The ending does not make agreement with the  

     A-Template#8  assumes a response pre-processor  capable  of  
     breaking up the learner's input into its meaningful parts.  Simple insertions of Type C
           The   answer-analysis  requirements  of  the  Type  C  
     insertions  are essentially the same as for Type  B.   Thus  
     3.20(Rivers),  3.47(Robinett),  3.62(Cook), 3.64(Cook), and  
     3.106(Stack)  could  use either basic A-template#2  or  #3.   
     3.63(Cook)  and  3.114(Stack)  could  also  use  the   same  
     approach.   The  nature  of their  response-sets,  however,  
     would also allow for a more generative approach:

          A-template#9   3.63(Cook)

          ?:   ?-->???
          R:   {R} = {days of the week}  day(1)="Sunday", etc.
          AGO: 1.   aaa=ccc+1.   Wednesday(4)=Tuesday(3)+1
          A:   {a} =  Wednesday.
          etc. as with template#3...

     An  authoring system that allowed for generation of  items,  
     both  in presentation and answer-analysis,  would free  the  
     author  from having to encode the entire list of  items  in  
     the  data.   As  it  could present items  at  random,  each  
     learner could be presented with a different set.

           3.46(Robinett)  is similar to 3.117(Stack)  in  that,  
     while the response could be analyzed holistically,  greater  
     flexibility  of  response might be obtained with a  partial  
     analysis using A-template#8.

           Examples  3.66(Cook) and 3.111(Stack) have  a  single  
     answer,  and  no finite response-set and so would use basic  
     A-template#1,  the right/wrong analysis.    If a  multiple- 
     choice approach were adopted for  3.111(Stack),  at a level  
     beyond:             a. True
                         b. False

     (the multiple choice equivalent of A-template#1),  then the  
     author would be forced to supply some sort of response-set,  
     presumably a meaningful one.

           3.28(Rivers)   resembles   3.144(Stack),    discussed  
     earlier,  in  that  the  learner has a certain  element  of  
     choice in selecting the response and the choice made at one  
     point determines the nature of the remainder of the correct  
     response.  In 3.144, for example, the choice of "do" as the  
     first word in the response precludes the choice of "it"  or  
     "music"  as the second word,  while the choice of "like" as  
     the  first  word forces their use as  second  word.    3.28  
     would  require  the same sort of analysis,  but on  a  much  
     simpler level:


          ?:   ?->rl=m,rt --->[n],???
          R:   {r}={aaa/bbb}   {her/to her}
          AGO: if [1] then aaa=her
               if [2] then aaa=to her
          A:   aaa = her/to her
          etc.  as with template#2...

           3.29(Rivers) and 3.65(Cook) represent cases that have  
     not yet been examined.   Both have very large response-sets  
     and smaller,  but still very large, answer-sets.   It would  
     be  impossible  for  the  computer to  cover  all  possible  
     responses.   3.65,  for  example,  has a response-set  that  
     includes all imaginable proper names and an answer-set that  
     includes  all  masculine names.    Learner's  could  supply  
     foreign  names,  like  "Abdul"  or   strange  spellings  of  
     English names,  such as "Merija" for "Maria".    A  teacher  
     could,  nevertheless,  distinguish  these from common nouns  
     such as "table" or nonesense combinations like "xxx".   The  
     computer  is  not as versatile as a  human  instructor.   A  
     response-set, {R}, consisting of a long list of valid names  
     would have to be entered into a database.   Each name could  
     be tagged as to whether it was masculine, feminine, or both  
     so  that  the  answer   for  a  particular  item  could  be  


          ?:   ?->rt --->???
          R:   {R} = {database:  list of valid names}
          A:   {a} = {those names that can be masculine}
          1. Is ??? an element of {R}   if yes then 2.
                                               else situation III
          2. Is ??? an element of {a}   of yes then situation I
                                               else situation II

          F:  Situation I:  The response is correct.
              Situation II: The response is incorrect.
              Situation III:  The response is invalid. 
           The  only other method of handling such  an  exercise  
     would  be  to close the response possibilities by  reducing  
     the  drill to a form of multiple-choice or by presenting  a  
     short of list of valid names on the screen.  Simple insertions of Type D

           3.33(Rivers)  is the only example of this  type.   In it was shown that this exercise could be handled in  
     two ways.  If the instructional intention is limited to the  
     ability  of  the  learner to make  the  correlative  change  
     correctly,  then  there would be a single response  with  a  
     dual response-set,  and a single correct answer,  clearly a  
     case for A-template#2.  

           If the aim of the exercise,  on the other hand, is to  
     force the learner to identify the element to be deleted, as  
     well as to make the resultant correlative change, then this  
     is clearly a two-response situation and A-template#6  would  
     normally  be  applied.   It  was pointed  out  in,  
     however,   that  the  basic  fill-in  presentation  is  not  
     applicable  to  this example.   Multiple- choice  or  free- 
     response were shown to be available options.  The multiple- 
     choice  approach could be handled by  A-template#4.    With  
     free-response,  the range of response possibilities widens,  
     to  include  options that the designer of the activity  may  
     not have intended:
          P:        They haven't any coffee.
                    They have some coffee.
                    They've some coffee.
                    They've got some coffee.
                    They have some.
                    They do have some.

     With  such a lengthy input,  holistic evaluation  would  be  
     unsuitable.   Fuzzy  matching,  to  reduce  errors  due  to  
     spelling,  could  only  be properly applied one word  at  a  
     time.   A template such as A-template#8 would be capable of  
     handling  each word separately.  To allow for an even wider  
     range  of  variations,   a  pseudo-parser  such  as  in  A- 
     template#7 could be employed.  Simple insertions of Type E

           It   was   anticipated  in   section   that  
     3.32(Rivers)  would  have  to be performed as  a  multiple- 
     choice  as  it  exhibits no  finite-response  set  and  the  
     answer-set  is  extremely large.   In  the  multiple-choice  
     form,  the  aspect of the activity that allowed the learner  
     to  provide original input would be lost.   If this were  a  
     key goal of the activity, then it is unlikely that it could  
     not  be  accomplished  within the limits  of  an  authoring  
     system.  The same finding would hold true for 3.36(Rivers),  
     if only holistic answer processing were available.    While  
     there  is  a part of the answer that is  fixed,   there  is  
     another  part  which  involves  creative  response  by  the  
     learner.     A  partial- processing  template  such  as  A- 
     template#7 could handle the answer-analysis,  but a portion  
     of the response would remain unanalyzed and would allow the  
     passage of unreported errors.    If this situation were not  
     pedagogically  acceptable  to the  author,  then  exercises  
     similar to 3.36 would not be realisable through the use  of  
     an authoring system either.  Re-order exercises

           The  re-order  exercises represent a special kind  of  
     multiple-choice.   They  all uniformly exhibit very  small,  
     explicit  response-sets  with a  single  correct  response.    
     The  pre-processing  of  the  learner's  response  will  be  
     different from that involved in a standard multiple-choice.    
     With ?->rm,  the computer must verify that ?a is an element  
     of @{r}.   With ?->ro, the computer must determine that all  
     elements of @{r} are present,   that none is repeated,  and  
     that there are no extra elements.   The response-set, @{r},  
     represents  not  only  the  elements  themselves,  but  all  
     possible recombinations of the elements.  

           Whether  or not the learner has supplied the  correct  
     order can be easily be established.   In fact,  there is no  
     reason  why  more  than  one correct  order  could  not  be  
     accommodated,  although  this  is not evident in the  data.     
     The  problem  lies in what to do with  responses  involving  
     incorrect  orders.   So long as the number of  elements  is  
     small,   the  computer can identify each order using  exact  
     match  techniques,  and  the  author can provide  a  unique  
     feedback for each one.    In the two-element cases such  as  
     3.8(Dacanay),  3.14(Dacanay)  and 3.129(Stack),  there  are  
     only two possible outcomes,  the correct one,  [2]-[1], and  
     the  incorrect one,  [1]-[2].    Responding in terms of the  
     incorrect order,  in this case, amounts to doing nothing at  
     all.    Where there are three elements,  as in 3.7(Dacanay)  
     and 3.127(Stack), six possible outcomes present themselves.    
     Four of these (excluding the correct order and the original  
     order) represent actual errors that the learner might make.   
     The  number of combinations increases factorially,  at   an  
     alarming rate:

     4 elements = 4 factorial combinations = 1*2*3*4     =  24
     5 elements = 5 factorial combinations = 1*2*3*4*5   = 120
     6 elements = 6 factorial combinations = 1*2*3*4*5*6 = 720

     Beyond   three   or  four  elements,   a   complete   error  
     anticipation  type of analysis cease to be  feasible.    It  
     would  be  necessary to identify certain  key  combinations  
     which  were  typical  errors,  or some  method  of  partial  
     analysis would have to be devised.  

           3.129(Stack) may provide a possible solution.   Here,  
     only the words involved in the goal of the activity can  be  
     moved, while the others are held constant, thereby reducing  
     the number of elements to be treated.   This possibility is  
     considered  at  greater length below in the  discussion  of  
     complex re-ordering exercises.  Production of new elements
           In the discussion of section,  free-response  
     and  multiple-choice  were  offered as possible  modes  for  
     production  of  new element exercises.    The  handling  of  
     multiple-choice needs no further discussion.   The use of a  
     multiple-choice approach may defeat the some of pedagogical  
     goal of this category of activity, especially the intention  
     of inducing the learner to "produce" a response.  Only  the  
     free-reponse  answer-analysis  possibilities are  therefore  
     reviewed here.

           3.39(Rivers)  exhibits  a response-set which  is  not  
     finite,  but an answer-set which is.  The correct responses  
     are short,  so that a holistic matching technique should be  
     feasible.   A right/wrong analysis such as in A-template#1,  
     but  with  the  added ability to  handle  multiple  correct  
     answers could be applied to the exercise.    There would be  
     no  manner of offering any discussion of incorrect choices.   

           The  response-set  could be artificially  limited  by  
     displaying a selection of,  say,  20 rejoinders and  asking  
     the   learner   to  select  only  from   those   displayed.    
     Explanations   could   afterwards  be  given  as   to   the  
     applicability of each of the rejoinders to the cue.

           The  effectiveness of partial-analysis,  on the model  
     of  A-template#7  or A-template#8,  is  determined  by  the  
     existence  of one,  or a set,  of relatively fixed response  
     models.    In  the case of 3.39,  the number  of  different  
     possible  response  models is enormous and therefore  these  
     approaches cannot easily be adopted.

           3.54(Cook)  and  3.145(STack),  on  the  other  hand,  
     exhibit relatively small response- and  answer- sets,  with  
     fixed  response models.   A-template#7 should handle  these  
     exercises  very effectively.    In 3.31A(Rivers),  there is  
     still  a  closed  set of  predictable  syntactic  patterns,  
     although much larger.   The same type of analysis could  be  
     used,  but  the  answer-processor would require  much  more  
     power and subtlety than the one assumed for template#7.

           Exercise 3.38(Rivers) builds a database interactively  
     with the learner and also applies it.   During the building  
     phase,  the  learner  adopts a role similar to that  of  an  
     author interacting with an authoring system.   This is  the  
     phase where the learner can supply new material,  and it is  
     an unmediated phase.   No analysis is made of the questions  
     that  are  entered.    As  the  computer  is  applying  the  
     database,  asking  questions  such  as "Does it live  on  a  
     farm?",  the  learner's yes/no answers are still not  being  
     evaluated  for correctness.   It is only at the end of  the  
     tree  when  the  computer  asks "Is it  a  xxx?"  that  the  
     learner's yes/no response can be mediated.    At this point  
     the  computer knows either that it has the answer  or  that  
     the tree needs another branch.    This activity seems to be  
     very  complex,  but  from  the  point of  view  of  answer- 

     analysis,  it  is  one of the simplest encountered  in  the  

           3.37(Rivers)  requires a partial  analysis  technique  
     that has not been discussed yet,  namely "key-word search".    
     The number of possible ways to respond to the question "Why  
     didn't they come here before midnight?" is enormous.    The  
     question  does have a very small set of correct  responses,  
     however:  "fireworks",  "Fourth  of July"   If the computer  
     looks only for these words, and ignores all other elements,  
     then it is possible to determine, to some degree, if is the  
     learner's  response is correct,  from the point of view  of  

          ?:   ?->rs  --->?1?,?2?,?3?,....?n?
          R:   0
          A:   {a} = {fireworks  and/or  Fourth of July}
          MO:  1.  Is {a} element of {?n?}?
                         If yes, then situation I
                                 else situation II
          F:   Situation I:   You appear to be correct.
               Situation II:  Your response does not seem to be

     This technique involves controlled guesswork and this  must  
     be  reflected  in  the reactions to  the  responses.   Many  
     teachers  would  reject  the  total  lack  of   grammatical  
     analysis.   Sentences leading to situation I could include,  
     for example, such unnaceptable formulations as:

          They is not come cause them fireworks.

     When  the  correction operation is strictly  centred  on  a  
     check  of  learner-comprehension,  on the  assumption  that  
     learners will produce grammatically correct responses, key- 
     word  search can still lead to ambiguous results,  as  with  
     the following sentence leading to situation II:

          They  didn't  come  because  they  took  part  in   the  
          patriotic celebrations.

     Nevertheless,  for  exercises such as  this  one,  key-word  
     search may be the only alternative to multiple choice.

     3.10.8 Answer analysis options in the data:  Complex drills

           In  section 3.9.3 the point was made that,  while  it  
     may  be possible to lessen the number of discrete steps the  
     learner has to perform in completing a complex drill  item,  
     each point must nevertheless be analyzed separately,  so as  
     to  be  reflected  in the  feedback  reactions.    Holistic  
     multiple- choice  represents  the only method  of  response  
     elicitation method that would allow several discrete points  
     to be analyzed with a single analysis per drill item.

           Section 3.7 described how all complex drills could be  
     analyzed  in  terms of the application of more than one  of  
     the  simple drill-options defined.    It is,  then,  to  be  
     expected  that  analysis of the responses would  entail  no  
     more  than the multiple application of some of the  answer- 
     analysis  techniques described in the previous section  for  
     simple drills.   No fundamentally new techniques should  be  
     required.  Multiple operations

           Basic  A-template#6 was defined in section 3.10.6  to  
     handle   the  simple  case  of  two  simultaneous   fill-in  
     responses,  as  was proposed for the insertion +  insertion  
     exercises such as 3.12(Dacanay) and 3.22(Rivers).   Each of  
     the  discrete insertions demonstrates very small  response- 
     sets  and a single correct response and,  as was  mentioned  
     during  the discussion of presentation template in,  
     represent  simple  insertions  of  Type  B.   The  feedback  
     reactions are additive, resulting in a dramatic increase in  
     the feedback possibilities over those in a simple drill.

           In 3.134(Stack), each discrete insertion, while still  
     having  only one correct response,  has a larger  response- 
     set,  and is made up of components that should be  examined  
     using a partial analysis.   Each resembles the simple Type- 
     B insertion shown in 3.117(Stack), which uses A-template#8.   
     A-template#8   provides  for  a  wider  range  of  feedback  
     reactions than the simpler templates,  a range which  would  
     be  doubled in the case of two discrete inputs.    It would  
     be reasonable to anticipate detailed feedback such as:

     C:    a. etudie'  b. rentre'      
     M:    Alain [] quand je suis [].
     ?:          est etudie'       rentrait

     F:   The  first  verb  must be in the  imperfect  and  the  
          second in the passe' compose'.      
          Your  passe' compose' uses the wrong auxiliary  in  this        
          You have used the wrong verb ending on the imperfect 

           The insertion-then-insertion exercises,  in which the  
     insertion-slots are fixed,  can be handled in the same  way  
     as  3.12  and   3.22.   A-Template#6 may not  be  adequate,  
     however, as the possibility arises that the first insertion  
     will   determine  the  second.     Such  is  the  case   in  
     3.26(Rivers),  where  the choice of "Does",  for the  first  
     insertion,  requires  "have" as the correct answer for  the  
     second,   while   the  choice  of  "Has"  for   the   first  
     necessitates "got" in the second.   While the two responses  
     can  be  entered  at  the same  time,  the  first  must  be  
     analyzed,  using  a  standard template  like  A-template#2,  
     before  consideration of the the second,  the template  for  
     which must have an answer-generating capability as shown in  
     A-template#10.   The importance of the link between the two  
     responses    is    not   as   evident   in    3.136(Stack).    
     Nevertheless,   the  answer-analysis  mechanism  should  be  
     given  the  subtlety  to realize that  "politely"  is  only  
     correct in slot#2 if the learner has followed  instructions  
     and inserted "speaks" in slot#1.  Mindless feedback such as  
     the following can then be avoided:

          P:    The boy is polite      
          C:    How does he speak?
          M:    The boy ---- -----.
          ?:             is  politely      
          F:    "is" is wrong but "politely" is correct

           Where  the first insertion can be made in a  variable  
     slot,  there would be no alternative,  in the fill-in mode,  
     to   presenting the two operations in sequence as a  simple  
     Type-A insertion,   followed by a simple Type-B  insertion.    
     Errors  in  the  first part would have to be  be  processed  
     before proceeding, as the display of the response model for  
     the second insertion would have to show the cue inserted in  
     the correct slot. Leaving it in the wrong slot would result  
     in confusion on the part of the learner and would,  in  any  
     case,  be  possible only if the second response-model  were  
     generated.  3.24(Rivers) illustrates the situation:

          P:    She brings too many pencils to school.
          C:    paper
          M:    She brings too many pencils to school.
                [1]                  [2]        [3]
          ?:    [1]  
        * C:+M: Paper ---- too many pencils to school.
          ?:          brings
          F:    Correct!

           Free-response would allow for the learner to complete  
     the  activity  in a single step and in a much more  natural  
     fashion.   As  the  implicit response-model  is  relatively  
     rigid,  a  template such as A-template#8 could  handle  the  
     response analysis on a word-by-word basis.   In the case of  
     the  above  response:

          Paper  brings too many  pencils  to school.

     the feedback could be:

          1.  "Paper" cannot substitute for "She". 
          2.  "pencils"  should  be  replaced  by "paper"
          3.   ...which  would require  "too  much" instead of
              "too many"                    

               In  it was suggested that the  re-order- 
     then re-order exercise,  exemplified by 3.15(Dacanay) could  
     be handled in one step by treating it as an ordering of six  
     discrete  word-units  rather than two  separate  orderings,  
     first  of  two sentence-units,  and secondly of  two  word- 
     units.    As was indicated in, only some of the 6- 
     factorial possible combinations could be treated as errors,  
     the rest falling into the "not recognized" category.  Mixed operations

           The  insertion+removal   exercises,    3.13(Dacanay),  
     3.16(Dacanay),  3.30(Rivers), and 3.131(STack), all exhibit  
     fairly predictable response-models.    Free-response, using  
     a template such as A-template#8,   joins holistic multiple- 
     choice  as  being  an option which allows  the  learner  to  
     accomplish   the  task  in  a  single  operation.     Other  
     approaches  would require two discrete actions on the  part  
     of  the  learner,  one for the insertion and  one  for  the  
     removal. showed that the  removal action could be  
     accomplished  as  a  two-answer  (first and  last  word  of  
     sequence  to be removed) or multiple-answer (all the  words  
     to  be  removed)  fill-in  or  multiple  choice  operation.    
     Variations  of basic A-templates#4 and #6 would handle  the  
     response analysis.  

           Removal   exercises  illustrate  a  case  where   the  
     elicitation of a whole series of responses may be feasible.    
     In  the case of 3.13(Dacanay),  where up to seven  discrete  
     responses  might be required,  the additive nature  of  the  
     feedback  that  was discussed in relation to template#6  is  
     not a problem.   For each word in the model there are  only  
     two possibilities,  it is to be removed or it is to be left  
     in.   Considering all the words in the model of 3.13, there  
     would be only 28 possible messages.

           3.44(Robinett)  and 3.45(Robinett) are naturally done  
     as consecutive simple drills.   The re-order portion  could  
     be  handled  as a re-order multiple-choice  or,  given  the  
     small number of elements and the fixed response model, as a  
     free  response using A-template#8.    Once all reaction and  
     corrections  to the first part were completed,  the  second  
     part  could  easily be handled by A-template#7 as  a  free- 
     response.  Choice of operations

           No  new  answer-analysis mechanisms are required  for  
     these essentially simple drills,  other than the ability of  
     the  authoring  system  ,  as was  earlier  discussed  with  
     reference  to  presentation templates,  to  switch  between  

     3.11  Factors influencing branching between drill items
           The  flexibility  that the author can  allow  to  the  
     computer,  in  its choice of the next drill item to present  
     to  the learner,  is a function of the  authoring  system's  
     available   presentation  and  answer-analysis   templates.    
     Before  examining the effect of these on  branching,   some  
     consideration  is  necessary  of  the  different  branching  
     options that are available on the computer.

     3.11.1  Branching options in CALL activities
           Eisele  (1978,  p.17) defines "branching" as a method  
     of  decision as to the sequence of component  segments  and  
     outlines  four basis branching schemes:  Linear  branching,  
     looping, remedial branching, and voluntary branching.

          A.     Linear  branching occurs when the learner  must  
     continue on to the next item,  regardless of the outcome of  
     the current item:
          Item 1 --->Item 2 --->Item 3

           Some  further  qualification  of  this  category   is  
     necessary.  The  situation  in  which the  computer  simply  
     proceeds  from  one  item in the data to the  next  can  be  
     considered  as "fixed branching",   while the situation  in  
     which  the computer randomly selects items  from the  data,  
     or generate items according to a rule, can be considered as  
     "random branching."

          B.     Looping  occurs  when  the learner is  made  to  
     repeat an item when immediately after having made an error.    
     Under   the  present  scheme,   the  term  "branching"   is  
     restricted to links between items and so the repetition  of  
     an item must be considered as one of the immediate feedback  
     options  possible  WITHIN  the item.  The  "looping"   that  
     Eisele  discusses would be considered as a special case  of  
     branching scheme A.

          [ Item 1  y  ---->] --->[ Item 2 ...
            <------ n

          C.    In Remedial branching the next item is chosen on  
     the basis of the learner's performance in the current item:    
     This  is  the interpretation most  frequently  applied,  in  
     CALL,  to the term "branching".    Several patterns may  be  

     c1:    Remedial branching  based on one item, where, if the  
     learner  misses an item in the main stream,   the next item  
     to be performed is a remedial item of the same sort as  the  
     one  missed.   There may be one of these,  or the chain  of  
     remedial items may continue over a number of presentations.  
     Once  "mastery"  of  the  point  is  demonstrated,  through  
     providing a correct response to one or more of the remedial  
     items,  the  learner  returns to the  main  stream.   Those  
     learners who do not make an error are not diverted from the  
     main  stream  and receive no presentation of  the  remedial  
     items,  although  some authors may prefer to offer them the  
     option of proceeding through the remedial exercises in  any  

            [Item 1]  n --->[Item 1b] n --->[Item 1c] n --->etc.
               :                 y:              y:
            [Item 2]  <----------------------------

     c2:     Remedial  branching,  based  on several  items,  is  
     similar,  except  that more than one error is  required  to  
     trigger the remedial stream.    A count must be kept of the  
     number of errors that the learner is making.

            [Item 1] n --->x=x+1
               :              :
               : <-------------
            [Item 2] n --->x=x+1
               :              :
               : <-------------
            [Item 3} n --->x=x+1: if x=3 then --->[Item R1]...
               :              :

     c3:      Skipping forward is a variation on c2.  A count is  
     kept of correct answers, and the student is branched out of  
     a sequence (or given the option to leave) after a specified  
     number  of correct answers.     The same process may be  in  
     the case of incorrect answers.

               : if y then y=y+1
            [Item 2]
               : if y then y=y+1
            [Item 3]
               : if y then y=y+1: if y=3 then EXIT
            [Item 4]

     c4:     Remedial branching based on multiple factors is  
     a more complex case of C1,  where the correct answer  still  
     moves  the  student on in the sequence,  but there is  more  
     than one possible avenue of remediation,  depending on  the  
     nature of the learner's errors.   Where more than one error  
     is   possible  at  the  same  time,   a  hierarchy  may  be  
     established  so that the learner does not have  to  perform  
     all of the indicated remedial items.

            [Item 1]        
              :y        n --->a. --->[Item A] ...   :
              :               b. --->[Item B] ...   :
              :               c. --->[Item C] ...   :
              :                                      :
              : <-------------------------------------
            [Item 2]

     c5:       Remedial  Branching  based  on  multiple  factors  
     across more than one item is a combination of c3 & c4.

             :y      n --->a.---->a=a+1 ---->:
             :              b.---->b=b+1 ---->:
             :              c.---->c=c+1 ---->:
             :                                 :
            [Item 2]
             :y      n --->a.---->a=a+1 ---->:
             :              b.---->b=b+1 ---->:
             :              c.---->c=c+1 ---->:
             :                                 :
            [Item 3]
             :y      n --->a.---->a=a+1 ---->if a=3 then ...
             :              b.---->b=b+1 ---->if b=3 then ...
             :              c.---->c=c+1 ---->if c=3 then ...
             :                                 :
            [Item 4]

     c6:   Repeating  incorrect  items at a later point  is  not  
     identiacal in nature to "looping",  which is not considered  
     here  as a true form of branching,  because in  this  case,   
     the  program  will  have  already moved past  the  item  in  
     question and must now return to it.   This step is  usually  
     accomplished   by  flagging  items  as  having  been   done  
     correctly or not.

     c7:   Undirected branching,  where individual items do not,  
     necessarily  have  "correct" answers.   Each item leads  to  
     multiple paths,  which lead on, in turn, to multiple paths.   
     Eventually,  certain paths lead to correct conclusions  and  
     it is in this way that the activity is mediated.

             :a                  :b                 :c
             :                   :                  :
           [Item A]          [Item B]            [Item C]
              :                 :                   :
           :------:          :------:            :------:
           :      :          :      :            :      :
          [Ai]  [Aii]      [Bi]   [Bii]        [Ci]   [Cii]

     This is a tree-structure,  such as one might be found,  for  
     example,  in  a  game of "Twenty Questions".   It  is  also  
     possible  that  branches of the tree  be  cross-linked  and  
     back-linked, as in an "Adventure"-type simulation.

          D.    Voluntary branching opportunities are offered in  
     many  CALL activities.    These may include options  to  1)  
     skip an item,  2) skip to the beginning of the next series,  
     3)  re-do an item that was not completly understood,  4) or  
     exit completly.

             Voluntary  branching is certainly important to  the  
     learning  process.    The computer should be responsive  to  
     the  needs  and desires of the  learners,  and  should  not  
     unduly  restrict  their flexibility.    With  a  book,  for  
     example,  the  learner always has the option of turning the  
     page  forward  or  backward and of going  to  the  next  or  
     previous  chapter.  A computer-based activity should offer,  
     at least,  the same flexibility.    This type of  branching  
     does  not  derive  from  the computer's  mediation  of  the  
     activity,  however,  and  therefore  must be  considered  a  
     "user-friendliness" variable rather than a topic for to  be  
     considered in the current investigation.

               Computer-invoked    branching    (or    branching  
     decisions  made  by  the  computer  on  the  basis  of  the  
     learner's  activities) may also be entirely  voluntary,  of  
     course.  The computer may ask, for example:

               Your  progress indicates that you should  do  some  
               remedial  activity.   Would you like to do that or  
               continue ?

           Of  Eisele's four categories,  then,  only two really  
     are of concern here,  namely Type A and Type C.   The three  
     possible  types of branching to be considered  are:  "fixed  
     branching",  "random  branching" and "variable  branching".    
     The  latter term refers to the schemes outlined under  Type  
     C, which are neither fixed nor random.  

     3.11.2   The  Effects of presentation and  answer-analysis  
     factors on branching

           In  the  discussion  of P-templates  for  interlocked  
     drills,  it was shown that, without the ability to generate  
     presentation  elements,  the  order of the items  would  be  
     fixed by the data.    Random branching would be  impossible  
     and variable branching would quickly become very difficult,  
     as the author tried to predict and provide for all possible  
     branchings in the data. The choice between  interlocked vs.  
     independent  drill  items  appears to be  the  only  formal  
     presentation factor that directly affects branching.

           Branching   possibilities  are  much  more   directly  
     affected  by the sublety of the answer-analysis  templates.    
     Remedial  branching is only possible to the extent that the  
     computer can identify the learner's errors.

           The remedial branching possibilities of an  authoring  
     system  which  only offered right/wrong analysis  would  be  
     restricted  mostly  to  simply  repeating  incorrect  items  
     (branching scheme: c6).   Branching to a stream of remedial  
     exercises  (c1) could only be done where where the response  
     possibilities were limited to dual response-set.  Otherwise  
     the author could not ensure that the error committed by the  
     learner matched the remedial meterial offered.    Even with  
     only  right/wrong analysis,  one can provide  for  skipping  
     forward  based  on  correct or  incorrect  responses  (c3),  
     albeit  with  less  certainty  in  the  case  of  incorrect  
     answers.    Tree  structure  branching  (c7)  can  also  be  
     handled  with  a right/wrong (in this case Yes/no  or  A/B)  
     answer-analysis.   Such tree-structure activities are often  
     more  interesting to the learner,  however,  where there is  
     more than one branch from each node. 

           Where  the  authoring  system  allows  for   multiple  
     evaluations,  it is possible to take advantage of branching  
     schemes  based  on remediation of particular errors  (c1  &  
     c4).   There  is  a  direct link between  the  subtlety  of  
     answer-analysis and branching flexibility.  The greater the  
     number  of possible incorrect incorrect responses that  can  
     be identified, the greater the possibilities for branching.     
     It is possible,  and in fact quite probable,  however, that  
     the  answer analysis mechanism be more subtle,  be able  to  
     identify  far  more response variations,  than  the  system  
     allows branching paths. 

     3.11.3  Independent factors affecting branching
           Branching subtype C2 and its derivatives,  c3 and c5,  
     amount,  in effect, to building up a model of the learner's  
     performance and taking action based upon that model.  To be  
     capable of building such a model, the authoring system must  
     have  some facility for keeping records over more than  one  
     drill  item.    Most,  in  fact  actually,  do  offer  this  
     facility.    It   is  often  manifest  as  a  score-keeping  
     mechanism,  although  score-keeping  is only  part  of  the  
     necessary resource.   The system must be capable,  not only  
     of  presenting  the  score  to the  learner,  but  also  of  
     allowing the author to cause action to be taken, based upon  
     the score.

           The number of correct or incorrect responses, and its  
     derivatives,  such  as  percentage correct over  a  certain  
     number  of items or the learner's percentile standing,  can  
     indicate only the crudest model of performance.   In  order  
     to frame an accurate gauge of linguistic performance,  many  
     more  intangible  factors must be assessed.   An  authoring  
     system  must  be  capable  of providing  a  wide  range  of  
     different  types of scores.   As in the case of  branching,  
     however,   the  subleties  of  score-keeping  are  directly  
     related   to   those  of  the   available   answer-analysis  

     3.12 Categorization by pedagogical factors

               It was shown in section 3.5 that most authorities  
     categorized activities according to a second gradient which  
     reflects pedagogical, rather than formal, features.  Almost  
     half   of  the  data  represent  activities  presented   to  
     exemplify  these pedagogically based typologies (see  table  
     3.3 for list).

           There  is less agreement among authorities as regards  
     pedagogical  classifications  than  was the  case  for  the  
     operational  categories discussed in section  3.6.    Table  
     3.14  lists  the  authors and the  variety  of  approaches,  
     categories, and  terms employed to differentiate activities  
     along  this gradient.    Even where the same or like  terms  
     are   used,   they  are  presented  often  with   differing  
     connotations.    As   a   general   pattern,   "meaningful/  
     contextualised"  activities are universally preferred  over  
     "mechanical/ manipulative/ meaningless/ non-contextualised/  
     non-communicative" activities". The latter are condemned as  
     useless or even counter-productive to the learning process.   
     There   is  much  less  agreement  as  to  what  the   term  
     "meaningful  activity"  actually  represents.  The  example  
     cited by one author as a "meaningful" drill matches another  
     author's example of a "meaningless" one.

           Where  more  than  two  categories  are   postulated,  
     "Communicative/  situational"  activities are  proposed  as  
     being  even better than simple "meaningful/ contextualised"  
     ones.     Table  3.15  shows  that  uniform  criteria   for  
     separating  these  two  groups are even more  difficult  to  
     find,  than  for  distinguishing "meaningful"  drills  from  
     "meaningless"  ones.      Each  author  uses  a  completely  
     different  scheme  for sub-classifying  activities  in  the  
     "communicative" sphere.

               It   is   difficult   to   isolate,    from   the   
     categorization schemes exemplified in tables 3.14 and 3.15,   
     a  uniform  set  of  factors by which  to  evaluate  drills  
     pedagogically.    Nevertheless,   for  the  teacher,   this  
     pedagogical  classification is in all probability the  most  
     important.   Teachers'  questions about  authoring  systems  
     most  often focus on their ability to create activities  of  
     the "meaningful" or "communicative" type.

               Such questions cannot be addressed without having  
     concrete  criteria  by which to judge an activity as  being  
     "meaningful" or "communicative" rather than  "meaningless".    
     These  criteria should reflect pedagogical  realities,  but  
     must also relate to possible implications,  in terms of the  
     use  of  an  authoring  system  to  produce  computer-based  
     activities   of  a  similar  nature.     Once  the  factors  
     contributing   to   an  activity's   "meaningfulness"   are  
     isolated,  it  should  be possible to examine  each  factor  

               Nine  such possible factors have been  postulated  
     and   are discussed below.   They are based only in part on  
     points  explicitly raised by the authors in  setting  forth  
     their typologies.   The primary basis for extraction of the  
     nine  factors  was an initial examination of  the  authors'  
     supporting   example  drills,   in  which  their   implicit  
     assumptions about pedagogical effectiveness become evident:  

           I.    Realistic interchange:    The stimulus-response  
     pairs  represent  sequences that might be  produced  during  
     normal language interaction (Stack, 1971, pp. 135, 137-138;  
     Dakin,  1973,  p. 62).     Also  included as positive  were  
     those  activities in which learner is working closely  with  
     realistic  interchanges,  even  though the actual  activity  
     does not simulate real language interchange.

           II.     Extended  context:     Subsequent  items  are  
     presented  within  a  global  context,   or  contribute  to  
     building  up  such  a  context  (Dakin,  1973,  p. 85).   A  
     secondary factor,  perhaps the more important,  is  whether  
     the   context   plays  any  part  in  determining   correct  

           III.  Truth value:   Responses are judged in relation  
     to  a  given  context or domain  on  non-grammatical,  non- 
     lexical  grounds,  such that the learner cannot  produce  a  
     correct  response  without an understanding of the  context  
     (Stack,   1971,  p. 134;  Dakin,  1971,  p. 8-9;  Robinett,  
     examples  3.48-3.50;  Paulston,  1972,  p. 134).   Possible  
     complications arise out of the relationship of this  factor  
     to  II (Extended Context) and VII (Lexical  Understanding).   
     The  former becomes clear as the factors are applied to the  
     data.   The latter must be defined:  Truth Value represents  
     more  than  simply responding correctly to a  lexical  cue,  
     such as a picture.   It  consitututes,  rather,  responding  
     correctly to a given situation.

          IV.    Information  gap:   There is  information  that  
     the learner does not have, but the interlocutor does.  This  
     factor  was  suspected of being related,  in some  way,  to  
     factor III (Truth Value).

           V.    Function  practice:   There  is  focus  on  the  
     functional aspect of the language being used, as opposed to  
     the  grammatical  or  lexical  aspect.       The  immediate  
     complication in defining this factor was that all realistic  
     interchanges  (I.) would clearly expose the learner to some  
     illustration  of  the  functional  use  of  language.    As  
     defined here,  the isolation of one functional aspect  must  
     be made much more explicit.

           VI.    More than one response:  More than one correct  
     response  to  the  stimulus is  possible  (Paulston,  1972,  
     p. 134;  Dakin, 1973, p. 85).  Extremely narrow cases, such  
     as  close synonyms or variants like "plane" and  "airplane"  
     are  not considered.

           VII.   Lexical  understanding:   The understanding of  
     the  meaning of a word,  phrase,  or idiom is  required  to  
     produce a correct response (Dakin,  1973,  p. 8-9).  In its  
     narrow  sense,  this  can  be defined as a  case  where  an  
     exercise  cannot  be completed if the text is  replaced  by  
     nonsense words (Paulston,  1972,  p. 135).   This criterion  
     alone  ceases  to be productive with more  advanced  drill- 
     types,  where  a  general understanding of the language  is  
     always required.   In such  cases,  "lexical understanding"  
     is  considered  a factor only where there is focus  on  the  
     meaning of a word or expression.

     constraints:   New, creative semantic elements (not present  
     in the overall context or in the stimulus) must be provided  
     by  the learner.  The learner's choice is  constrained,  to  
     some degree, however, by a given component of the activity.   
     Completely free creation is,  therefore,  not possible.  To  
     eliminate    confusion    with   factor     VII    (Lexical  
     Understanding),  this process must involve more than simply  
     the definition or re-phrasing of a word or expression.

           IX.    Elicitation of new, creative elements, without  
     constraint:   There  is   opportunity for  the  learner  to  
     include original (i.e. personally-chosen) semantic elements  
     in the response,  without constraining factors.  Obviously,  
     this  must only refer to creative additions to an otherwise  
     constrained  response.   Completely free  expression  would  
     fall outside our definition of CALL activity.

               The list of these nine factors is not exhaustive.   
     Other  important factors  are often cited in the context of  
     communicative drills,  such as motivation and relevance  of  
     the  situation  to the learner.   In the case  of  Robinett  
     (examples  3.49-3.50),  for example,  the relevance of  the  
     situation  to  the  learner  is the  sole  deciding  factor  
     between  a  meaningful  and  a  communicative  drill.   The  
     decision  was  taken to include only  those  factors  which  
     could  have  a forseeable effect on the computerisation  of  
     the drills examined.   

           Table 3.16 examines the relevant data in terms of the  
     nine  discrete factors postulated.    Activities were  then  
     grouped  according  to  individual  factors  (Tables   3.21  
     through  3.28) to observe any possible  inter-relationships  
     between the factors. 

               Table 3.21 shows all the cases in which there  is  
     a  realistic  interchange.   Many of the cases  (75%)  also  
     exhibit  an extended context.  Originally,  the expectation  
     was that all examples where the stimulus-response pair were  
     of   a  question-answer  nature  would   automatically   be  
     realistic  interchanges.   Those  examples  representing  a  
     question-answer  interchange are listed in table 3.20,  and  
     this  phenomenon  is shown to  be  generally  true.  Beile,  
     however,  shows with example 3.156,  that it is possible to  
     create a question-answer exchange that does not represent a  
     realisitic interchange.

               Table  3.22  shows all the cases in the  data  in  
     which   there  is  an  extended  context  and  is   further  
     subdivided,  according  to  the  role  of  the  context  in  
     determining  the correct answer.   Many of the cases  (70%)  
     also  exhibit a realistic interchange.  It seems clear that  
     extended  context  and  realistic interchange  tend  to  be  
     associated,  but still independent factors.  The fact  that  
     66% of the cases where context directly affects the correct  
     response   also  involve  the  truth-value  factor  is  not  
     surprising,  as these two factors would seem, in the nature  
     of things, to go together.  How, then, can the 33% of cases  
     providing the exception be accounted for?   3.71(Byrne) and  
     3.73.1(Byrne)  exhibit an information  gap,  which,  as  is  
     suggested below, might be considered an alternative case of  
     Truth  Value.    The  others call on the learner's  lexical  
     skill, which differs from Truth Value only according to the  
     arbitrary definition given above.

               Table  3.21  lists  those examples which  involve  
     Truth  Value  or Information Gap and separates them  as  to  
     whether  the  role of the learner is to answer  a  question  
     (Question-Answer),  pose a question  (Answer-Feedback),  or  
     neither (no question).  Truth Value and Information Gap and  
     listed  together  because it was felt that  the  difference  
     between  the  two  factors  was simply a  function  of  the  
     direction  of information flow (from the learner or to  the  
     learner).    It  is difficult to postulate a  general  rule  
     based on the small number of cases present in the data, but  
     the  trend seems to indicate that Information Gap and Truth  
     Value  can  be treated as different manifestations  of  the  
     same basic factor.  

          What  of  the  possible  relationship  between   Truth  
     Value/Information Gap, on the one hand, and the presence of  
     an   extended  context  affecting  the  choice  of  correct  
     response,  on the other?    Those cases which exhibited the  
     latter  factor,  but  did  not  exhibit  Truth  Value  were  
     discussed  above,  and  were shown not to  be  significant.     
     Truth  Value,  without  the  presence  of  a  corresponding  
     extended context affecting the choice of correct  response,  
     is  exhibited by 18% (3/16) of the examples listed in table  
     3.23.    The three exceptional ( 3.69(Byrne),  3.95(Dakin),  
     and  3.96(Dakin)) cases exhibit item-specific  rather  than  
     extended contexts.  It can be assumed,  then, that where an  
     extended  context  affects  the choice of  correct  answer,  
     Truth  Value or Information Gap will also be  factors,  but  
     not necesarily the reverse. 

               Table  3.24  presents the few cases in  the  data  
     where  there  is explicit practice  of language  functions.   
     In all cases,  there is also a realistic  interchange.   In  
     nearly  all cases,  there is also an extented context,  but  
     the one exception,3.56(Cook), indicates that this might not  
     always be the case.

           Table  3.25 examines all the cases in which there  is  
     more  than  one response.   Cases can be imagined  where  a  
     purely   morpho-syntactic  variation  would  lead  to   the  
     possibility of multiple correct responses. In all the cases  
     exhibited  in  the sample,  however,  some  combination  of  
     factors  VII  (Lexical skill),  VIII (Elicitation  of  new,  
     creative elements, within constraints), and IX (Elicitation  
     of  new  elements,  without constraints) is  also  evident,  
     indicating that the variation in correct answers must be of  
     a semantic nature.

              Table  3.26 groups all the examples where  Lexical  
     Understanding  is  a  factor  in  determining  the  correct  
     response.   In  63%  of the cases,  more than  one  correct  
     response  must also be accepted.    A possible significance  
     of this trend is explored,  below, in the discussion of the  
     difference    between   "meaningful"   and    "meaningless"  

               Table 3.27 shows all examples where creative  new  
     semantic elements are added,  within constraints.  As might  
     be expected, all such cases lead to the possibility of more  
     than  one  response.   With only one exception,  they  also  
     require   lexical skill.    Table 3.28 shows those examples  
     where some degree of free creative input on the part of the  
     learner was permitted.  Again, all must allow for more than  
     one correct response.


               It  is  necessary  to postulate  some  manner  of  
     distinguishing  between those activities  characterized  by  
     their   authors  as  being  "mechanical",   "manipulative",  
     "meaningless", "non-contextualised", or "non-communicative"  
     (hereafter referred to simply as "meaningless"  activities)  
     and   those  characterized  as  being  pedagogically   more  
     appropriate  (hereafter referred to simply as  "meaningful"  
     activities).   (The  difference  between  "meaningful"  and  
     "communicative"   activities  is  not  important  in   this  
     context.)   In some cases,  authors are in direct conflict.   
     Stevick  (example 3.93) classes a drill which  demonstrates  
     only a realistic interchange as a "meaningless" drill while  
     Stack  (1971,  p. 134)  explicitly defines  a  "meaningful"  
     drill as one that has,  at least,  a realistic interchange.   
     For Robinett (1978,  p. 208),  a meaningless drill  becomes  
     meaningful  when  there is reference to truth  value.   All  
     other factors are incidental.  

               It  was decided to explore how the  nine  factors  
     postulated,  which had their ultimate basis in the authors'  
     contrasts  of  "meaningless" and  "meaningful"  activities,  
     might contribute to a common,  working understanding of the  
     difference   between  the  two  types.     All  the  drills  
     identified  as  being  of  the  "meaningless"  variety  are  
     grouped together in table 3.19. It is clear that in all but  
     three  cases,  none  of  the nine  factors  considered  are  
     involved.    In the three exceptional cases only one factor  
     is evident,  either realistic interchange or lexical skill.   
     In table 3.18 those examples characterized as "meaningful",  
     but  showing  the influence of only a  single  factor,  are  
     shown.  The list is quite short, as only four such examples  
     are found in the data.   Interestingly,  the single factors  
     are  the  same  as  for  "meaningless"  drills,   realistic  
     interchange and lexical skill.

           Ninety-one  percent  of  the  examples  in  the  data  
     characterized  by  their  authors  as  being  "meaningful",  
     exhibit  the  presence of at least two of  the  pedagogical  
     factors.   Realistic Interchange, alone, is evident in both  
     "meaningless"  and "meaningful" examples.    It  was  shown  
     above  that  in nearly all cases where there was  Realistic  
     Interchange one could also expect to find Extended  Context  
     and  in  many  cases where Lexical Skill was  involved  one  
     could expect more than one response to be possible.   Beile  
     alone  seems  to  feel that the  basic  difference  between  
     "meaningless"  and  "meaningful"  is  the  presence  of   a  
     realistic interchange (examples 3.150 vs.  3.151-4).    For  
     Paulston   and  Stevick,   the  presence  of  a   realistic  
     interchange  is  not sufficient to make a drill  meaningful  
     (examples 3.51/52 vs. 3.53 and 3.93 vs. 3.94). 3.94).   

           It  is  also  interesting  to  note  that  the  drill  
     examples supplied by those authors who postulate a  single,  
     deciding  factor actually evidence the presence of multiple  
     factors (Stack, examples 3.160 vs 3.161; Robinett, examples  
     3.48 vs. 3.49).  

           It  seems  safe  to  say,   then,  that  "meaningful"  
     activities   should  exhibit  at  least  two  of  the  nine  
     pedagogical factor postulated above.    Should an  activity  
     exhibit  only  a  realistic interchange,  or the  need  for  
     lexical  skill,  it  is possible to rule that it  does  not  
     qualify as a "meaningful" activity.   There are no examples  
     in the data of any the other factors in isolation and so it  
     is  impossible to speculate whether any of them,  on  their  
     own, can render an activity "meaningful". 

     3.13  The relationship between pedagogical and  operational  

           The  operational classification developed in  section  
     3.7  was applied to the pedagogical sample to see if  there  
     were  any  discernible  relationships.    The  results  are  
     summarized  in table 3.17b.    The presentation and answer- 
     analysis factors,  developed and explored in 3.9 and  3.10,  
     were  also applied to the sample.   A detailed analysis  of  
     this application is shown in table 3.17,  and summarized in  
     table  3.17c.    Tables  3.20b through 3.28b  consider  the  
     operational  factors  in relation to  discrete  pedagogical  

           The  breakdown of the pedagogical sample in terms  of  
     operational categories is shows similar results to that  of  
     the  original operational sample upon which the  categories  
     are  based.   Most of the activities (82%) represent simple  
     operations and, of those, most (71%) could be classified as  
     insertions.   All the complex operations in evidence can be  
     described in terms of the simple operations.    It was  not  
     necessary  to postulate any new opertational types in order  
     to  accommodate  the  drills in  this  sample.  There  are,  
     however, operational types which are not in evidence in the  
     pedagogical  sample.    There  are no deletion  or  removal  
     operations  and  only  one  isolated  case  of  a  re-order  
     operation.     Nor are there any mixed  operations,  though  
     this  is understandable as most of the mixed operations  in  
     the operational sample involved removal.

           A  factor-by-factor examination (tables  3.20b-3.28b)  
     revealed   no  evident  pattern.   Drills  illustrating   a  
     particular  factor  seem  to be evenly  spread  across  all  
     operational types.   The only exception seems to be in  the  
     case of drills having more than one response, in which case  
     no insertions of Type A are represented.

           Table   3.17c   illustrates  the   presentation   and  
     response-analysis  factors  as they apply to  those  drills  
     which can be called "meaningful" by the criteria  discussed  
     in  section 3.12.  Some interesting global observations can  
     be   made.    It  might  be  anticipated  that   meaningful  
     activities  would  require  item-specific  response  models  
     (mmm)  because universal response models (MMM)  would  make  
     the  activities excessively rigid and mechanical.  In fact,  
     universal  response-models could be set up for 60%  (21/35)  
     of  those  exercises with explicit  response-models.    One  
     might  expect  meaningful activities to  exhibit  large  or  
     infinite   response-sets,   while  meaningless   activities  
     exhibited  small,  closed  response  sets.   In  fact,  44%  
     (20/45)  of the examples exhibit response sets which  could  
     be  termed  "very small" (vs) and 28% (13/45) more  exhibit  
     response sets which are still small enough to be manageable  
     using  standard  answer-analysis  techniques.  Finally,  it  
     might be expected that  meaningful drills would require the  
     acceptance   of  a  large  number  of  correct   responses.    
     Thirty-three percent (15/45) if the activities  only  allow  
     for  one correct answer and an additional 40% (18/45)  have  
     answer sets that could be characterized as "very small".

           Table  3.20b  examines the  operational  factors  for  
     those  drills which are question-answer  interchanges.   As  
     might  be expected,  all,  including the type B  insertions  
     (C:+M:) have separate, explicit cues.   These all have very  
     small  or  small  response sets and single  or  very  small  

           Table   3.21b  lists  the  drills  which  exhibit   a  
     realistic  interchange.    No  particular  presentation  or  
     answer-analysis  pattern is observable.   Answer-sets,  for  
     example,  range  from  a  single  correct  response  to  an  
     infinite set of correct responses.    A similar case can be  
     found  in  table  3.22b  (extended  context),   3.23b(truth  
     value/information    gap),     3.24b(function    practice),  
     3.25b(more  than  one  response),   3.26b(lexical   skill),  
     3.27b(creative   input,    within   constraints).        In  
     3.28b(creative input,  without constraints),   there are no  
     very small answer-sets, which is, of course, predictable.

           Those  drills  with very large or  infinite  response  
     sets  or  answer sets would  require  modification,  before  
     presentation  on the computer,  so as to limit the range of  
     response  possibilities  in  some  way.     This  could  be  
     accomplished in a natural fashion, as in 3.96(Dakin) where,  
     if  the  domain were restricted to musical  instruments  by  
     contextualising  the  activity  more  fully,   very   small  
     response- and  answer-sets  would result.    Where  natural  
     limitations cannot be introduced, it is necessary to resort  
     to multiple choice.  Indeed, Candlin has already taken such  
     a  step in 3.76 and 3.77,  where the large implicit  answer  
     set has been artificially reduced by providing a very small  
     explicit response set with one correct answer.

           Drills with large or infinite answer-sets that  could  
     not  be delivered well as multiple choice activities  would  
     be  difficult to accomplish on the computer without a  full  
     parsing-mechanism.   An example of this case is provided by  
     3.89(Candlin).   Such an activity borders on the production  
     stage   of  language  learning.    Though  structured   and  
     mediated, it is not inherently interactive.

Converted from the original Wordstar format and placed in Html package
Roger Kenner, 2006
[Return to Top Level]