Abstract
Based on a survey of the American Society for Horticultural Science (ASHS), membership need was identified for an online peer review system to validate innovation and recognize excellence in science-based teaching and extension scholarship for promotion and tenure purposes. This system would also provide a clearinghouse for instructional materials of merit for use in classrooms, laboratories, and outreach education, which fall outside the parameters of the three academic journals of ASHS. It was determined HortTechnology already provided a valued outlet for peer review of manuscript style teaching and extension scholarship; however, a need was identified for a mechanism to provide peer review of instructional materials which did not conform to a traditional manuscript format. Herein we describe the process that led to the development and launch of HortIM™, a new peer review system for teaching and extension instructional materials. An online peer review process for juried assessment of instructional materials such as articles, bulletins, case studies, fact sheets, instructional videos, teaching modules, and laboratory exercises was developed. A beta test of initial solicited materials in each category was piloted resulting in an initial database of these scholarly materials. This activity culminated in an initial opening of the system for submissions in Fall 2016. This article documents the development of HortIM™, including the submission and review process.
An enhanced appreciation and need for peer review of teaching and extension scholarship has become apparent in recent years in higher education (Bernstein et al., 2006; Fincher et al., 2000; Sachs and Parsell, 2013). Peer review of teaching has long been defined through the development of rigorous structured evaluation of submissions to peer reviewed journals and recognized as an important scholarly enterprise (Solomon, 2007). Over time, several well-respected outlets for peer review of horticulture-related research have developed, including three academic journals published by ASHS. The Journal of the American Society for Horticultural Science, and the annual Proceedings of the American Society for Horticultural Science from which it evolved in 1969, was the initial repository for peer reviewed horticultural research for the society. This was followed with the addition in 1966 of HortScience. HortTechnology was launched in 1991 to meet a need for publication of applied or translational research as well as to provide an outlet for peer review of manuscripts describing innovations in teaching and public outreach. However, a peer review system or a repository for horticultural teaching and outreach/extension instructional materials that did not conform to a traditional manuscript format has not existed.
During 2012, several ASHS Board of Directors (Board) members were asked to review promotion or tenure packets for peer institutions and the challenge of effective documentation of peer review of teaching artifacts was recognized and discussed. Traditionally, peer review of research efforts were readily documented through accomplished publication of peer reviewed research manuscripts and successful garnering of competitive funding for research. Student evaluation and peer observation of teaching provided a means to measure classroom or laboratory presentation and organizational effectiveness, but opportunities to document peer evaluation of innovation and quality of instructional materials were limited. Some general online peer review repositories for multimedia educational resources for learning and teaching were available, such as MERLOT II (California State University, 2016) or eXtension (U.S. Department of Agriculture, 2016); however, none were specific to horticulture. With instructor positions increasing as a proportion of faculty at institutions offering horticulture degrees (Arnold et al., 2014c; Reed et al., 2016), the need for avenues to document scholarly contribution for innovative, science-based, horticultural instructional materials not produced in traditional formats suitable to peer review as manuscripts was recognized. It was also hoped such a repository might broaden ASHS’ appeal beyond our traditional membership base.
Thus, a committee, chaired by then president-elect Michael A. Arnold, was appointed by ASHS president Mary H. Meyer and issued a “Committee Charge: To develop a peer reviewed set of broad, scholarly criteria and a system for reviewing teaching and extension materials. ASHS should manage the system, original materials should be available through ASHS, it should provide an income to ASHS, increase membership and most of all, advance the scholarship of teaching and extension materials in horticultural science.” The committee planned its initial meeting in Autumn 2013. The goal was to devise a framework for the system within 6 months with a pilot of the project ready for implementation in Aug. 2014. It became acutely apparent once discussions commenced that the committee needed to perform a needs assessment and solicit the advice of the Board.
Due diligence: the survey
At the committee’s request and the Board’s approval, W. Timothy Rhodus, Michael A. Arnold, and Michael W. Neff were tasked with developing the online survey and then soliciting the ASHS membership and other groups, such as extension agents and community college instructors, for input. Data from this survey were used to address a portion of the following questions that arose from the committee meetings and the Board discussions:
Who would be our clientele, both from a contributor and user perspective?
Would the service be aimed at the current membership or target potential new members?
Would there be a real demand for such a system?
Would the outcomes of a peer review system be helpful in the tenure and promotion process at various institutions?
What alternative outlets were already available?
What types of teaching and extension materials would we review?
What would be the structure of the review process?
At what stage of development would material be reviewed?
How would the materials be archived?
How would we recognize the material in such a manner that it would be acknowledged as peer reviewed materials?
How would we finance the undertaking?
Does it offer a potential new revenue stream for ASHS or would it be more of a member service?
At the very least, how long would it take to break even?
During Nov. 2013, a survey instrument was developed by the committee, with particular aid from W. Timothy Rhodus and Michael W. Neff. Specific procedures used to administer the survey are described in Arnold et al. (2014a). The survey was completed in early 2014, data were summarized, and results were published (Arnold et al., 2014a). These results were also presented to the membership in a special roundtable discussion format workshop at the 2014 ASHS Annual Conference, which was attended by more than 50 people. This discussion resulted in additional suggestions, which were incorporated into the development of the system (Arnold et al., 2014b).
Answers for many of the questions posed above were derived from the results of the survey and membership discussions. There appeared to be substantial interest in an online review system as indicated by the majority of the 500 survey participants and enthusiastic participation in the workshop. Interest was in the form of both a peer review of materials for documenting excellence in teaching and outreach scholarship, and a source for validated science-based instructional materials.
The clientele would be both contributors and users. This posed a bit of a challenge as in the survey we solicited information on the demand for and submission potential of more than 20 distinct instructional materials. Demand vs. supply did not coincide for all individual artifacts. It was decided that initial acceptances for review and archiving would be those materials that were well aligned with high levels of both demand and potential submissions. This included extension articles, bulletins, case studies, fact sheets, instructional videos, teaching modules, and laboratory exercises. Additional formats of materials accepted could be added as the system matured.
Another issue was the stage of development of the instructional materials at which peer review would take place. With typical scholarly journals, a proposed final draft of a manuscript is submitted to a journal, a peer review is conducted, and then often the manuscript proceeds through one or more rounds of revision before acceptance and subsequent publication. This would be problematic for many non-manuscript teaching materials, such as instructional videos, extension bulletins, or fact sheets, that are already finished products and would not be at stages of development where revisions could readily be made. This situation is analogous to artwork submitted to an art contest, reviewed by critics as a finished product, and found worthy or wanting. Thus, the committee recommended a juried review process should be undertaken using a set of prescribed criteria unique to each category of submission along with a list of required documents or artifacts to be submitted (Fig. 1). The committee then set out to develop a set of rubrics that awarded points for various characteristics. If the submitted material was found to be scientifically correct and reached a threshold of points, it would be deemed accepted, if not it would be declined for inclusion in the collection.
Screen capture of the reviewer evaluation of format for submitted material to HortIM™.
Citation: HortTechnology hortte 27, 2; 10.21273/HORTTECH03650-17
How do we archive the material and make it accessible to potential users? Some of the material would be copyrighted or offered for a fee, such as an extension bulletin, video, or workbook. These materials would need to be available from the entities producing them, so an image and link in the database would link to the designated site by the authors/producers of the material and an abstract consisting of a paragraph description of the instructional materials submitted by the author plus a paragraph describing the strengths and limitations written by the coeditors based on the reviewers’ comments and rubrics would serve as the portion housed on the HortIM™ site (ASHS, 2016).
After due deliberations, the ASHS Board of Directors instructed the committee to proceed with development of the system and approved the name suggested by W. Timothy Rhodus of HortIM™. A proposed purpose statement was developed: “The purpose of HortIM™ is to provide a juried review system to recognize excellence in innovative, science-based teaching and outreach materials that do not fall within publication parameters of the three ASHS academic journals. HortIM™ provides a valuable repository for online instructional materials that will be used in classrooms, laboratories, and outreach education by horticulture professionals and the general public.”
The development goals of the HortIM™ System were to
1) Inform authors about the project and describe procedures for uploading numerous types of instructional materials.
2) Facilitate the collection of item-specific metadata and uploading of original content.
3) Facilitate the online peer evaluation of submissions according to specific rubrics.
4) Display and manage approved submissions for public access.
5) Collect the administration data required to manage submissions, including reviewer name, e-mail, passcode and assignment date, acceptance status, and review summary.
The development team installed three open source applications on the ASHSmedia.org domain (ASHS, 2016) to serve as the backbone of HortIM™. The primary application, Omeka (Roy Rosenzweig Center for History and New Media, 2016), handles the public website and informs authors and the public about the project, handles author submissions, and displays accepted materials. Omeka is an open source web-publishing platform for the display of library, museum, archives, and scholarly collections and exhibitions (Roy Rosenzweig Center for History and New Media, 2016).
The peer review portion of HortIM™ is handled through LimeSurvey (LimeSurvey Project Team, 2015), a free and open source online survey application distributed under the GNU General Public License (Free Software Foundation, 2016). Secure and private access to the evaluation rubric is handled by assigning a unique passcode to each reviewer.
The shared administrative data are organized in Evernote (Evernote Corp., Redwood City, CA), a cross-platform app designed for note taking, organizing, and archiving information. A single Evernote notebook is shared among the project leader and editors and facilitates the submission management information needs.
Description of the HortIM™ workflow
This first step in the submission process for an author is to open HortIM™ (http://hortim.ashsmedia.org/) and browse the About, Submissions, and Submitting sections. Creating a guest account is required and explained in Upload Instructions (http://hortim.ashsmedia.org/upload_instructions). Authors will receive an e-mail confirmation with on-screen instructions. Once an item is uploaded, along with the appropriate metadata, a confirmation message is forwarded to the author and the Project Manager (PM). A quick examination of the required metadata and the item being submitted (or URL of an online item) is made by the PM and an e-mail response is sent to the author requesting payment for a review of their submission using the ASHS online store (http://hortim.ashsmedia.org/upload_instructions). Once payment has been made, the PM updates the Administrative database and creates a portable document format (PDF) copy of the HortIM™ layout for the new submission. Copies of the PDF file are forwarded to each of the coeditors.
At that point, the editor in charge of the submission solicits two reviewers. Reviewers are solicited from the reviewer database used for the other ASHS journals. Once reviewers agree to review a submission, they are provided the PDF copy of the submission, link to the peer review system and a custom token ID. Reviewers are asked to evaluate rubrics associated with the appropriateness of the format of the material (Fig. 1) as well as its content (Fig. 2). When the two reviews are completed, a general review statistics report is compiled by the PM and sent to the editor. Based on the reviews, the editor makes an acceptance decision and writes a summary of review comments to be added to the cover page representing the submission on the HortIM™ website. If the submission is declined, the submitter is notified and reviews are provided. If the possibility of revision exists, the submitter can revise and resubmit. Once a submission has been accepted, the cover page resides on the HortIM™ website along with access to the instructional material. No additional publication fee is currently charged for accepted submissions. Successful contributors are encouraged to advertise the fact the instructional material met favorably with peer review in HortIM™. A suitable graphic logo is in development that can be affixed to the teaching instrument to advertise its successful peer review in HortIM™.
Screen capture of the reviewer evaluation of content for material submitted for HortIM™.
Citation: HortTechnology hortte 27, 2; 10.21273/HORTTECH03650-17
One of the important tasks in creating HortIM™ was to develop a Terms of Service document for submitters and users of the system. The PM and the development team reviewed the standard Omeka policy (Roy Rosenzweig Center for History and New Media, 2016) and those policies created by other sites providing similar services. A final policy was developed and reviewed by ASHS legal counsel (http://hortim.ashsmedia.org/contribution/terms).
Browsing the HortIM™ website
Visitors to the HortIM™ website (ASHS, 2016) are presented with information for browsing or searching the collection of instructional materials, as well as instructions for uploading their own content. Content is organized according to the type of instructional material and grouped into a collection: Article, Bulletin, Case Study, Fact Sheet, Instructional Video, Laboratory Exercise, or Teaching module.
Browsing by Collections will allow one to cross various subject matter topics but still keep the relative format and audience the same. The site search tool allows one to enter key words for searching across all submissions.
Who would administer the system?
One of the key questions as the system took shape was who would administer it once the committee’s initial efforts were completed? Given his intimate involvement in its development, W. Timothy Rhodus was an obvious choice as the HortIM™ PM. One of the key people in defining the extension materials submissions and evaluation rubrics was Susan S. Barton, and she generously stepped forward to serve as a coeditor. Meanwhile David L. Hensley, a long-time ASHS member, was retiring from an administrative position at the University of Arkansas, but was interested in remaining involved with ASHS. He also volunteered to serve as a coeditor. The committee initially envisioned dividing submissions between a teaching- or extension-based coeditor, but it was unclear what the load would be for the two areas and there was a great deal of overlap, so the decision was to allocate submissions on an alternating basis between the two coeditors.
Beta testing and launch
Beta testing began in Nov. 2015 with solicited materials for the various categories from among the ASHS membership to test the workflow of the system. A positive outcome was that even among the solicited initial beta-tested materials some submissions were declined, lending credence that the peer review process would be one of rigor and not a rubber-stamped seal of approval. Results of initial beta tests and a proposed startup budget were submitted to the Board for consideration for the 2016 fiscal year. Tentative support for the first 3 years of the operation of HortIM™ was approved with W. Timothy Rhodus as PM, and Susan S. Barton and David L. Hensley as coeditors. Initially, the project leadership would answer directly to the Board through the Executive Director on an operational basis, with the potential for its own editorial board to be established as the project matured. Beta testing was completed in Spring 2016 and the initial example submissions populated in the searchable archive. Announcement of the launch of HortIM™ occurred during a workshop at the 2016 ASHS Annual Conference in Atlanta, GA. The HortIM™ website (ASHS, 2016) was opened for public submissions in Autumn 2016, ≈3 years after the initial inception. Subsequent marketing efforts will include featuring HortIM™ on the ASHS website (ASHS, 2017) and newsletter, press releases, e-mail notices to members, and informational print articles.
Conclusions
As the world’s premier professional society for horticultural science, ASHS is committed to “sharing the results of research, teaching, and extension activities with Horticulture colleagues and humankind the world over.” The society serves a diverse range of horticultural professionals, including researchers, teachers, and extension educators, by providing relevant research and significant education and training opportunities. ASHS’ three academic journals provide members with opportunities to share groundbreaking research, as well as promote the field’s best practices in training and education.
To recognize the essential work of horticulture practitioners, and to address gaps in accessibility and delivery of high-quality teaching and extension materials, the investigative committee designed an online juried review system called HortIM™. HortIM™ (ASHS, 2016) provides practitioners an online portal for submitting and accessing outreach and teaching materials that are user ready. HortIM™ is open to all commodity areas and all ASHS members. Through HortIM™, the public will gain access to an extensive online collection of exceptional resources attributed to ASHS members, thus helping to promote ASHS’ visibility and preserve the society’s standing as the world’s premier horticultural science organization.
Literature cited
American Society for Horticultural Science (ASHS) 2016 HortIM™. 3 Dec. 2016. <http://hortim.ashsmedia.org/>
American Society for Horticultural Science (ASHS) 2017 ASHS: American Society for Horticultural Science. 22 Jan. 2017. <http://www.ashs.org/>
Arnold, M.A., Azarenko, A., Bachman, G., Barton, S., Bosland, P., De Vos, N., Klett, J., McKenney, C., McMahon, P., Marini, R., Meyer, M., Neff, M., Ray, D., Rhodus, T. & Wilson, S. 2014a Peer review of non-journal article teaching or extension tools: Should we, shouldn’t we, could we? Amer. Soc. Hort. Sci. Nwsl. 30 8 1 5
Arnold, M.A., Azarenko, A., Bachman, G., Barton, S., Bosland, P., De Vos, N., Klett, J., McKenney, C., McMahon, P., Marini, R., Meyer, M., Neff, M., Ray, D., Rhodus, T. & Wilson, S. 2014b Peer review of non-journal article teaching or extension tools: Should we, shouldn’t we, could we? HortScience 49 S76 (abstr.)
Arnold, M.A., Lineberger, R.D., Davis, T.D., Reed, D.W. & McKinley, W.J. 2014c A survey of North American horticulture graduate programs: Demographics, policies, finances, and metrics HortTechnology 24 241 251
Bernstein, D., Burnett, A.N., Goodburn, A. & Savoy, P. 2006 Making teaching and learning visible: Course portfolios and the peer review of teaching. Anker Publ., Bolton, MA
California State University 2016 MERLOT II multimedia educational resource for learning and online teaching. 4 Dec. 2016. <https://www.merlot.org/merlot/index.htm>
Fincher, R.-M.E., Simpson, D.E., Mennin, S.P., Rosenfeld, G.C., Rothman, A., McGrew, M.C., Hansen, P.A., Mazmanian, P.E. & Turnbull, J.M. 2000 Scholarship in teaching: An imperative for the 21st century Acad. Med. 75 9 887 894
Free Software Foundation 2016 GNU general public license. 22 Jan. 2017. <https://www.gnu.org/licenses/gpl.html>
LimeSurvey Project Team 2015 LimeSurvey: An open source survey tool. 22 Jan. 2017. <http://www.limesurvey.org>
Reed, D.W., Arnold, M.A., Lineberger, R.D. & Davis, T.D. 2016 Challenges of higher education in the US – What will horticulture of the future look like? Acta Hort. 1126 27 205 217
Roy Rosenzweig Center for History and New Media 2016 Omeka.net. 22 Jan. 2017. <https://rrchnm.org/omeka-net/>
Sachs, J. & Parsell (eds.), M. 2013 Peer review of learning and teaching in higher education: International perspectives. Springer, New York, NY
Solomon, D.J. 2007 The role of peer review for scholarly journals in the information age J. Electron. Publ. 10 1 22 doi: 10.3998/3336451.0010.107
U.S. Department of Agriculture 2016 eXtension. 4 Dec. 2016. <https://extension.org/>