A New Day in Plant Identification
Courses Taught

Introduction

We are at a turning point in how we use computers to teach plant identification and identify unknowns, but the revolution has yet to be codified into a coherent formula. Still, the pieces are in place to create a unified teaching and identification environment that can be used as easily by novices as it can by experts. The key to this claim is the use of images.

Good field biologists are visual experts. They are able to identify taxa at a glance; sometimes from only a fragment of the plant. They can do this because, through experience, they have learned to use a different part of their brain than novices. They have activated the Fusiform Face Area, which is associated not only with face perception but with other types of visual expertise (Bukach et al. 2006). The use of the FAA allows visual experts to see wholes (configurations), and to see them quickly. Novices see only parts (or primarily parts). The ability to identify a taxon at a glance depends on being able to see the whole at a glance.

Using these insights, innovative software for teaching visual expertise has been developed (Kirchoff 2007). This software is tightly tied to the cognitive psychology literature on visual expertise (Gauthier and Tarr 1997; Gauthier et al. 1998). The first application of this software has just been released by from Missouri Botanical Garden Press (Kirchoff 2008). The software is written in Java, and can be adapted for web delivery.

Again inspired by recent research in cognitive psychology, a completely new approach to computer-based keys has been developed. These keys are primarily visual; they use images instead of text, and thus are not dependent on the arcane terminology of plant taxonomy. The keying method involves (a) visual displays from which the user selects images similar to the unknown; (b) a Bayesian substructure that tracks the selections; (c) an adaptive similarity database that “learns” as program components are used; (d) a game-like element based on the work of Luis von Ahn when he was at Carnegie Mellon, which is used to create the initial similarity matrix. A summary of these ideas has been published (Kirchoff et al. 2008).

Why Is Plant Identification Important?

Student’s interest and ability to identify plants is declining at a time when there is a great need for these skills (Tilling 1987; Secretariat 1996; Terlizzi et al. 2003). This lack of interest is an especially serious problem for smaller institutions, which have become the primary training ground for young professionals entering the fields of conservation, management, and biotic surveys (Ertter 2000). The need for trained taxonomists has been emphasized by the Secretariat of the Convention on Biological Diversity: “Effective habitat conservation, bioprospecting and the sustainable use of biodiversity on a global basis all require taxonomic decisions and expertise on a scale not presently available” (Secretariat 1996). The establishment of NSF’s programs in Planetary Biodiversity Inventories (PBI) and Partnerships for Enhancing Expertise in Taxonomy (PEET) are indications of this urgency. The goals of these programs are to “accelerate the discovery and study of the world’s biodiversity” (PBI), and to “enhance taxonomic research and help prepare future generations of experts” (PEET). The PBI program was initially established in partnership with the All Species Foundation, whose goal it is to complete the inventory of all species on earth in the next 25 years (Anonymous 2004). For this goal to succeed, the world’s approximately 10,000 taxonomists will need an army of apprentices to assist in the initial stages of this work (Janzen 2003). Rapid training in plant identification will greatly assist this process.

Plant identification is also important for those working in the horticulture and forestry industries (Arteca 2006). Correct identification affects every aspect of how plants are used in the landscape: time of planting, time and method of pruning, fertilization schedule, and etc. The need for proper identifications extends not just to species, but to cultivar identification. Proper cultivar identification is a critical issue to the horticulturist. Plants are not always distributed under the correct name, so horticulturalists must learn proper identification to assure proper care. Difficulties arise when cultivars of the same species have similar appearances and characteristics. Although cultivar keys are available (Bailey 1949; Dirr 1998), they are difficult to use, and take a long time to learn. A busy horticulturalist rarely has time to key out a cultivar. They must know them by sight. Plant identification is a critical building block in every horticulture class.

How is Plant Identification Currently Taught?

There are two ways in which plant identification is commonly taught in plant systematics courses (Judd 2006). The first involves focusing on diagnostic characters. The instructor teaches students to search for and recognize the diagnostic characters that distinguish the taxa. Once the student has a good memory of the features for each group (usually families), he or she is taught to evaluate an unknown in the following way (technical characteristics in italics). “I’ve never seen this plant before, but I can tell that it is a member of the family Aquifoliaceae because it is a woody plant with alternate leaves, minute triangular, black stipules, its flowers are imperfect with the flat stigma arising directly from the globose ovary, and it has colorful drupes with several pits.” This method depends on a clear understanding and memory of the technical terms.

The second method consists of teaching the students to use a taxonomic key, and giving them extensive practice in keying out unknown plants. In using keys, students are forced to be observant, and must learn the technical terms upon which the keys depend. After keying out several members of the same family, a student begins to see family-level resemblances and eventually develops a mental concept of the family. He or she begins to see suites of common characters. In this way, knowledge of plant diversity at the family level is build up through knowledge of the local flora.

Both of these approaches depend on a student’s ability to learn the technical and often arcane terminology of plant identification. The learning curve for these terms is very steep. A partial list of the terms for leaf shape will demonstrate the difficulty: linear, oblong lanceolate, elliptic, oblanceolate, ovate, broadly elliptic, obovate, orbicular, reniform (Walters and Keil 1996). There are separate sets of terms for the shape of the leaf apex, of the leaf base, of the margin, for the texture of the leaf, its venation, its covering of hairs, not to mention the terms associated with the other parts of the plant (Walters and Keil 1996; Judd et al. 2002; Simpson 2006). Students who see a direct application for this knowledge in their careers are the ones who often do best in these courses.

Field Botany courses use a very different approach. The emphasis in these courses is on identification of the plants in the field. The instructor points out and identifies the plants each time they are seen, and usually mentions a few of the distinguishing characteristics that help identify the plant. While some technical terms may be used, the emphasis is on learning to recognize species by sight. Cut specimens are often brought into the lab for review, and a list of plants seen on each field trip is often provided to the students, as is a list of required plants.

A variant on Field Botany is a course on woody plant identification. These courses are most commonly taught as part of horticulture or forestry programs (Kahtz 2000). As in Field Botany, outdoor laboratories expose the students to the plants and provide an opportunity for review of previously learned species. Some conceptual material is generally included in these classes, in a lecture format.

Limitations of Current Teaching Methods

In addition to problems with learning an enormous technical vocabulary, there are several shortcomings of current methods of teaching plant identification. The most effective current methods depend on students being able to see living representative specimens. While seeing living plants is an excellent way of learning, the method has shortcomings. The specimens must either be available as cut specimens in a lab setting, or as living specimen in the field. Cut specimens require a great amount to time to collect, require a good deal of dedicated laboratory space for display, and must be replaced frequently. Plant identification tests based on living specimens are also laborious to set up and grade (Pokorny 1988). Outdoor laboratories are susceptible to inclement weather, can only be conducted during daylight hours, and are time consuming (Kahtz 2000; Anderson and Walker 2003). With increasing urbanization, travel time to good field localities can also eat up a considerable amount of the class time.

No matter how they are delivered, current methods only allow students to see of a limited amount of taxonomic variability. For instance, in studying the sunflower family the students may be exposed to three or four genera out of approximately 1,500 in the family. Increasing students’ exposure to variability is important, because like all concepts, taxonomic concepts capture information about variability as well as information about the prototype of the conceptual category (Wisniewski 2002).

Student’s study habits also cause traditional teaching methods to be less effective than they should be. Students often adopt passive study methods that are not effective in teaching visual recognition, and do not stimulate the brain areas associated with visual expertise (Rhodes et al. 2004; Bukach et al. 2006). The ability to identify taxa depends on the ability to form accurate visual concepts, yet instructors have no effective method of assessing whether a student’s study methods are effective at forming these concepts. Even the amount of time students spend studying seems to be inadequate. At a typical university, students are expected to spend 2-3 hours outside of class for each hour in class. Yet almost 80% of the students polled at UNC Greensboro in 2005 reported that they spent less than one hour outside of class for each class hour (Salinger 2005).

Some Results from Cognitive Psychology

Visual experts perceive and understand visual information differently than novices (Bransford et al. 2000). For instance, chess masters are able to recognize meaningful chess configurations, which allows them to consider sets of moves that novices would ignore (deGroot 1978). Experts in other domains display similar abilities (Bransford et al. 2000). Experts in plant identification are no exception. Good field botanists are able to recognize species from rapidly moving vehicles, sometimes even in difficult groups like grasses. Specialists in specific groups can recognize these plants by sight, even when identification keys are difficult to use, and other good botanists are unable to do so. When questioned about this ability they do not mention technical characters, but speak about reading the plant’s form (Baldwin 2006). Expert entomologists report similar abilities (Conner 2006).

Humans processes visual information in at least two distinct ways. Analytic processing involves selectively attending to distinct features within a stimulus, while holistic processing involves attending to the overall form of the stimulus, its Gestalt. Holistic processors have difficulty attending to specific features of a stimulus, even when specifically instructed to do so. Research suggests that unfamiliar stimuli are processed analytically, while stimuli with which one possesses considerable expertise are processed holistically (Diamond and Carey 1986; Gauthier and Tarr 1997; Gauthier et al. 1998; Gauthier et al. 1999; Gauthier and Logothetis 2000)

Holistic processing is associated with changes in brain activity in the Fusiform Face Area (FAA) and Occipital Face Area (OFA) (Gauthier and Tarr 1997; Gauthier et al. 1998; Baldwin 2006; Bukach et al. 2006). Despite their names, these areas are activated by visual expertise of objects other than faces (Gauthier and Logothetis 2000; Bukach et al. 2006). FAA activation has been shown to be stimulated more by tasks that require the active engagement of the viewer than those that only require passive participation (Rhodes et al. 2004). This finding links the visual processing literature to the literature on active learning (Perkins 1991; Bork 1995; Schroeder and Spannagel 2006).

An additional competency of experts is their ability to segment their perceptual field into meaningful parts. Being able to recognize and use the information contained in parts does not come automatically, but is dependent on prior experience with the classification of the objects (Schyns and Rodet 1997; Schyns et al. 1998). Take a subject and show him an ambiguous object, an object that can be divided into parts in a number of ways. Ask him to identify its parts. His answer will depend upon his prior experience with similar objects. If experience has shown him that it belongs to a group whose members have clearly defined parts, he will find these parts in the ambiguous object. A second subject, who knows the object in a different context, will find parts that are consistent with that context. The subjects find the parts that make the classifications work (Schyns and Rodet 1997; Schyns et al. 1998). If we want someone to quickly learn to see an objects’ parts, it is helpful to teach him the object in the context of a classification that implies these parts. Traditional teaching methods do not do this. Instead, they ask students to use domain-specific perceptual mechanisms that they do not possess.

Finally, visual experts are able to classify objects at a more specific level of abstraction than are novices. For instance, a novice may recognize a plant as a pine while an expert is able to recognize it as a specific species of pine, Pinus palustris for example. Research shows that training in species-level discrimination generalizes to new species of the genus better than does genus-level discrimination (Tanaka et al. 2005). While this is an empirical finding, we can also relate it to another area of cognitive psychology, concept formation. In order to learn to recognize a given taxon students must form a concept of that taxon. This concept allows them to recognize new exemplars as group members. It is likely that learning lower-level categorization generalizes better because concepts encode information not only about the prototype of the category, but about its variation (Wisniewski 2002). Learning lower-level categorizations requires students to encode more variability, and thus form more realistic concepts.

Putting it all Together: Teaching Visual Identification

 

Computer programs that draw on what we know about visual expertise and active learning can be developed and delivered over the web. Use of these programs can simplify the process of learning taxon identification. Although they will not replace traditional instruction methods, they have the potential to make these methods more effective. The following description draws on an early example of one of these programs (Kirchoff 2008).

Image Preview

Before the user can make begin to learn a new taxon, he or she must first be exposed to the range of variation of that taxon represented in the database. Image preview mechanisms, allow the user to gain this exposure, and to begin to associate the images with the taxon name.

Quiz and Test Design

Quiz and test routines are adapted from the visual processing literature (Gauthier and Tarr 1997; Gauthier et al. 1998; Tanaka et al. 2005). Four types of quizzes and three types of tests are incorporated into the program. The difference between the quiz and test modes is that the users do not receive feedback on their responses or get a second chance to respond in the test mode.

Image naming with prompt (Quiz mode only): The user sees an image of a plant with the taxon name superimposed, and responds by typing the name in a response box. If the user gets the name correct, they are given positive feedback. If they make an error, they are given a second chance. This routine familiarizes the user with the strange and often confusing Latin taxonomic names.

Image naming without prompt (both modes): As #1, but without a name prompt. If the user gets the name correct, they are given positive feedback. If they make an error, they are given a second chance. Spelling fidelity can be adjusted to allow some errors. This routine requires the most active participation from the user.

Image comparison (both modes): The user sees two plant images side by side and presses “Y” if they are from the same group (family, genus, or species depending on the user’s choice earlier in the program), and “N” if they are not. If the user gets the answer correct, he or she is given positive feedback. If they make an error, the images are displayed again and they get another chance.

Image verification (both modes): The user sees a plant image, the screen is cleared and, after a short delay, a taxon name appears. The user presses “Y” if the name is of the pictured plant, and “N” if not. If he or she gets the answer correct, they are given positive feedback. If they make as mistake, they are given a second chance.

The duration of image display is adjustable from 0.1 to 5 seconds in each routine. Short display times are easier for visual experts, while long display times accommodate the part-based perceptual mode of novices.

Image Sources and Standardization

 

Images can be drawn from on-line image databases such as the USDA Plants Database, CalPhotos: Plants (Biodiversity Sciences Technology Group 1995-2008), Morphbank (Morphbank Team 2004-2008), or Steve Baskauf’s BioImages site (Baskauf 2003-2009). The plant images on this last site are standardized, making them ideal for use in this type of project. For example, when leaf margins are shown in a separate image, the lower surface of a second leaf is visible in the background and the major veins of this leaf are visible. Baskauf’s images of woody plants were used to create Woody Plants of the Southeastern United States: A Field Course on CD (Kirchoff 2008).

Database

Within the program, a taxonomic mage database is linked to plant image files so that taxa can be selected for study at the family, genus or species level, and learned by scientific or common name. Inclusion of information on the part represented in each photograph would allow taxa to be learned based on a restricted set of characters.

Tracking User Compliance and Grading

Scripts can be created that control of the operation of the program so that an instructor can design custom study sessions to teach certain taxa. An output file can be used to track user compliance and, in classroom situations, grade a student’s progress.

Visual Keys

By a “visual key” a key that uses images with little or no reliance on terminology is meant. These keys will be easier to use than traditional or even matrix based keys (Walter and Winterton 2007) for both experts and novices. For experts, the use of images more closely matches the way that they recognize plants. For novices, the key avoids the often-considerable problems that novices have with terminology. Plants can be identified solely by visual means.

In the simplest type of visual key, images are displayed on the computer screen, and the user makes selections based on the similarity of the displayed images to the relevant parts of the plant he or she wants to identify. The program tracks these selections and computes Bayesian posterior probabilities for the likelihood of the unknown’s identity. These probabilities are used to assign the unknown plant to the most likely correct taxon.

Like conventional keys, the user progresses through the visual keys character by character. The difference is that the characters are photographs of natural plants parts, and are not be associated with terminology. For the purpose of these keys, a character is a collection of images of homologous plant parts (Kirchoff et al. 2007). For instance, a collection of leaf images represents one character; of bud images another. Because the characters are sets of images, they will be more familiar to novices than terminology-based characters. A shorter learning period will be required before the keys can be used effectively.

When an image belonging to a specific taxon is selected, its probability of being the unknown plant increases. Display and selection of images continues until the posterior probability of some taxon reaches a predefined level (ca. 95-98%). If this probability level is not reached with the first character, the program gives the user the option of moving to a second character, and so forth until the desired probability level is reached. At this point, the program displays one image from each character of the selected taxon and asks the user to confirm that it is the unknown. The user also has access to a picture of the whole plant, a written description of the taxon and, if possible, a summary drawing that shows the structure of parts not included in the key. If he or she rejects this choice, the program gives him two options. He can either start over, or see summary displays of other likely species.

The sequence of image display is determined by a by a similarity matrix. Entries in the matrix are pair-wise similarities for each taxon/character pair. The images that are the closest to the previously selected images are displayed on the next screen. The entries in the matrix are based on similarity assessments made by users while playing an on-line game, similar to those devised by Luis Von Ahn (von Ahn 2004, 2006). One of these games was incorporated into Google Image Labeler before it was discontinued. Because the database is stored on-line, it can be continuously updated with new data as the game is played. This will improve the quality of image delivery in the key.

Entry into the key requires an initial assessment of similarity, which is used to guide future image display. For large groups this accomplished through drawings that allow the discrimination of large numbers of taxa. For instance, if the user has access to leaves he or she would select this entry point and be presented with a diagram that illustrates the difference between simple and compound leaves. Further diagrams will assist in narrowing the type of leaf, until it is possible to display a screen of leaf images for selection. Because the database tracks real perceptual similarities, this and subsequent screens will contain images that users find similar, and that must be discriminated for proper taxon recognition.

A difficulty in database construction occurs because novices and experts use different perceptual modes. Novices and experts will make different similarity assessments, and will require different similarity matrices for image display. These matrices can initially be created by requiring those who play the on-line game to rate their experience with plant identification: novice verses expert. Two primary matrices can thus be constructed and used for novice and expert key users. Use of these matrices allows the keys to be tuned to the experience of the user. A matrix for intermediate users can also be created by averaging the entries in the primary matrices.

A Complete Plant Identification Learning Environment

Use of the visual training tool in conjunction with the visual key will create a very complete learning environment for plant identification. Novices can begin by learning to recognize common plants then use the key to practice their identification skills with the same plants. Learning sequences can be designed and recorded as scripts so that users progress from easy to difficult plants. In the process, the users will become visual experts in plant identification, and will be able to learn taxa that are more difficult. Since both the key and the visual learning tool are based on plant parts, the users will gain familiarity with plant construction and will be better able to learn the complex terminology of plant taxonomy. Additional modules can be developed to teach this terminology in a computer-based, active learning environment. Custom modules can be constructed and used for teaching cultivar identification, and to create certification examinations for technical fields that depend on plant identification.

References

* Anderson, N. O., and J. D. Walker. 2003. Effectiveness of Web-based versus live plant identification tests. Horttechnology 13:199-205.

* Anonymous. 2004. All Species Foundation. San Francisco, CA. [see http://en.wikipedia.org/wiki/All_Species_Foundation]

* Arteca, R. N. 2006. Introduction to Horticulture Science. Thomson Delmar Learning, Florence, KY

* Bailey, L. H. 1949. Manual of Cultivated Plants Most Commonly Grown in the Continental United States and Canada. MacMillan Publishing Company, New York, NY

* Baldwin, B. 2006. The tarweeds are a very difficult group, the keys do not work and other botanists doubt the species descriptions, but I can recognize them by sight. I can recognize the Gestalt of the plants. Curator of the Jepson Herbarium and Professor of Integrative Biology, Univeristy of California, Berkeley, CA, Berkeley, CA.

* Baskauf, S. J. 2003-2008. Bioimages. Vanderbilt University, Nashville, TN. [http://bioimages.vanderbilt.edu/]

* Biodiversity Sciences Technology Group. 1995-2008. CalPhotos: Plants. Berkeley, CA. pp. CalPhotos is a collection of 168,943 photos of plants, animals, fossils, people, and landscapes from around the world. A variety of organizations and individuals have contributed photographs to CalPhotos. Please be aware that these various contributors maintain copyright and follow the usage guidelines provided with each image. [http://calphotos.berkeley.edu/]

* Bork, A. 1995. Why has the computer failed in schools and universities? Journal of Science Education and Technology 4:97-102.

* Bransford, J. D., A. L. Brown, R. R. Cocking, M. S. Donovan, and J. W. Pellegrino. 2000. How People Learn: Brain, Mind, Experience, and School: Expanded Edition. National Research Council, Washington, DC. pp. 374.

* Bukach, C. M., I. Gauthier, and M. Tarr. 2006. Beyond faces and modularity: The power of an expertise framework. TRENDS in Cognitive Sciences 10:159-166.

* Conner, W. E. 2006. Gestalt entomology: A review of Stephen A. Marshall’s Insects:Their natural history and diversity. American Scientist 94. [http://www.americanscientist.org/template/BookReviewTypeDetail/assetid/53126]

* Degroot, A. D. 1978. Thought and choice in chess, 2nd edition. Mouton De Gruyter, The Hague, Netherlands

* Diamond, R., and S. Carey. 1986. Why faces are and are not special – an effect of expertise. Journal of Experimental Psychology-General 115:107-117.

* Dirr, M. A. 1998. Manual of Woody Landscape Plants: Their Identification, Ornamental Characteristics, Culture, Propagation and Uses. Stipes Publishing, Champaign, IL

* Ertter, B. 2000. Floristic Surprises in North America North of Mexico. Annals of the Missouri Botanical Garden 87:81-109. [http://ucjeps.berkeley.edu/floristic_surprises.html]

* Gauthier, I., M. Behrmann, and M. J. Tarr. 1999. Can face recognition really be dissociated from object recognition? Journal of Cognitive Neuroscience 11:349-370.

* Gauthier, I., and N. K. Logothetis. 2000. Is face recognition not so unique after all? Cognitive Neuropsychology 17:125-142.

* Gauthier, I., and M. J. Tarr. 1997. Becoming a ”greeble” expert: Exploring mechanisms for face recognition. Vision Research 37:1673-1682.

* Gauthier, I., P. Williams, M. J. Tarr, and J. Tanaka. 1998. Training “greeble” experts: a framework for studying expert object recognition processes. Vision Research 38:2401-2428.

* Janzen, D. 2003. How is all this work going to get done with a limited number of taxonomists? All Species Foundation, San Francisco, CA.

* Judd, W. S. 2006. Personal Communication: Summary of methods of teaching plant identification. Professor of Botany. University of Florida, Gainesville, FL, Gainesville, Fl.

* Judd, W. S., C. S. Campbell, E. A. Kellog, P. F. Stevens, and M. J. Donoghue. 2002. Plant Systematics: A phylogenetic approach, 2nd edition. Sinauer Associates, Sunderland, MA

* Kahtz, A. W. 2000. Can computer assisted instruction beused by students for woody plant identification. Horttechnology 10:381-384.

* Kirchoff, B. K. 2007. Image Quiz: Using principles of cognitive psychology to teach visual expertise In: Proceedings, UNC Teaching and Learning with Technology Conference. UNC TLT Collaborative, Raleigh, NC: 1-7.

* Kirchoff, B. K. 2008. Woody Plants of the Southeastern United States: A Field Course on CD. Missouri Botanical Garden Press, St. Louis, MO

* Kirchoff, B. K., D. L. Remington, Lixin Fu, F. Sadri. 2008. A New Type of Image-Based Key. Proceedings of the International Conference on BioMedical Engineering and Informatics, 2008. BMEI2008. 1: 825-829. DOI 10.1109/BMEI.2008.301

* Kirchoff, B. K., S. J. Richter, and D. L. Remington. 2007. Characters as groups: A new approach to morphological characters in phylogenetic analysis. Taxon 56:497-492.

* Morphbank Team. 2004-2008. Morphbank. Florida State University, School of Computational Science, Tallahassee, FL. [http://www.morphbank.net]

* Perkins, D. N. 1991. Technology meets constructivism: Do they make a marriage? 31:18-23.

* Plants Web Development Team. 2006. Plants Database. United States Department of Agriculture, Natural Resource Conservation Service, USDA National Plant Data Center. [http://plants.usda.gov/]

* Pokorny, F. A. 1988. Computer assisted plant identifiction tests. National Association of College Teachers of Agriculture Journal 32:28-30.

* Rhodes, G., G. Byatt, P. T. Michie, and A. Puce. 2004. Is the fusiform face area specialized for faces, individuation, or expert individuation? Journal of Cognitive Neuroscience 16:189-203.

* Salinger, W. 2005. Forum on academic quality. UNC Greensboro, Greensboro, NC.

* Schroeder, U., and C. Spannagel. 2006. Supporting the active learning process. International Journal on eLearning 5:245-264. [http://proquest.umi.com/pqdweb?did=1005284951&Fmt=7&clientId=15109&RQT=309&VName=PQD]

* Schyns, P., and L. Rodet. 1997. Categorization creates functional features. Journal of Experimental Psychology: Learning, Memory and Cognition 23:681-696. [http://cogprints.org/713/]

* Schyns, P. G., R. L. Goldstone, and J. Thibaut. 1998. The development of features in object concepts. Behavioral and Brain Sciences 21:1-54.

* Secretariat. 1996. Practical approaches for capacity building for taxonomy. Convention on Biological Diversity: Subsidiary body on scientific, technical and technological advice, New York. pp. 1-30. 

* Simpson, M. G. 2006. Plant Systematics. Elsevier, Amsterdam

* Tanaka, J. W., T. Curran, and D. L. Sheinberg. 2005. The training and transfer of real-world perceptual expertise. Psychological Science 16:145-151.

* Terlizzi, A., S. Bevilacqua, S. Fraschetti, and F. Boero. 2003. Taxonomic sufficiency and the increasing insufficiency of taxonomic expertise. Marine Pollution Bulletin 46:556-561.

* Tilling, S. M. 1987. Education and taxonomy – The role of the Field-Studies-Council and Aidgap. Biological Journal of the Linnean Society 32:87-96.

* Von Ahn, L. 2004. Labeling Images with a Computer Game In: ACM Conference on Human Factors in Computing Systems, CHI 2004: 319-326. [http://www.cs.cmu.edu/~biglou/ESP.pdf]

* Von Ahn, L. 2006. Games with a purpose. Computer 39:92-94. [http://www.cs.cmu.edu/~biglou/ieee-gwap.pdf]

* Walter, D. E., and S. Winterton. 2007. Keys and the crisis in taxonomy: Extinction or reinvention? Annual Review of Entomology 52:193-208.

* Walters, D. R., and D. J. Keil. 1996. Vascular Plant Taxonomy., 4th Updated Version edition. Kendall Hunt, Dubuque, IA

* Wisniewski, E. J. 2002. Concpets and categorization. In: D. Medin, ed. Stevens’ handbook of experimental psychology. vol 2. Memory and cognative processes., 2. pp.467-531.