Abstract
Webinars have become an important aspect of many extension outreach programs, especially for large research projects that cover wide geographic regions. Evaluation of webinar impact is important to funders and stakeholders alike. Surveys to gather direct and indirect data can be used to evaluate webinars for their overall utility. The results can inform funders of program success and also serve as an indicator of the continued viability of webinar programming for extension audiences. The Northern Grapes Project administered a webinar series on topics related to grape (Vitis sp.) and wine production from 2012 through 2016. Thirty webinars were delivered to an audience of 3083, with nearly an additional 2400 views of recorded webinars. To estimate the value of the series, we modified two models, one that estimated viewer time investment and one that calculated the “green savings.” In addition, we surveyed webinar participants, asking them how much they would pay to watch a live webinar and access the archives, as well as how much money they had earned and/or saved from the information presented. Together, the models and survey indicated a webinar series value of nearly $3.4 million.
Webinars are a technology that has advanced greatly among researchers and extension workers in the past decade (Verma and Singh, 2010; Zoumenou et al., 2015). Various groups have championed its use as a way to conveniently disseminate research outcomes to stakeholders (Allred and Smallidge, 2010; Formiga et al., 2014; Hoke et al., 2018; Pulec et al., 2016; Rich et al., 2011; VanDerZanden, 2013; Zavell et al., 2017). Although the technology has been available for several years, challenges to successful implementation still exist. Recent upgrades in technology have spurred an increase in online webinar programs. Early adopters of webinar delivery have served as the vanguard for other extension efforts to use webinars as a viable education option (Allred and Smallidge, 2010; Formiga et al., 2014).
The advantages and challenges of hosting webinars has been covered extensively by others (Rich et al., 2011; Verma and Singh, 2010; Zoumenou et al., 2015). But one of the main benefits of using webinar technology is the reduction or elimination of costly travel expenses, especially when the webinars are designed to reach an audience throughout a large geographic region. Grapes are grown in every state, with most states having dozens, if not hundreds, of wineries as well (Wine America, 2017). The rate of growth in this industry is staggering and the need for educating new growers and wine makers is a high priority for those in viticulture extension and research (Martinson et al., 2016; Stafne et al., 2009).
Relatively recent advances in cultivar development have allowed high-quality interspecific hybrid wine grapes to be grown in regions where it was too cold and the season too short for the more common european grape [Vitis vinifera (Martinson et al., 2016)]. Many of the vineyard managers and winery owners in the northern states are new to viticulture and enology and, thus, are in great need of educational offerings to start them on a profitable path. A solution for this issue was a regional grant project called the Northern Grapes Project.
The Northern Grapes Project was a Coordinated Agricultural Project funded in 2011 by the U.S. Department of Agriculture (USDA) Specialty Crops Research Initiative (SCRI), addressing three SCRI focus areas of production (viticulture), processing (enology), and consumers and markets (winery business management and marketing). It focused on new, cold-hardy cultivars that initiated the development of a grape and wine industry in the upper-Midwest and Northeast United States, comprising more than 300 wineries, 3300 acres of grapes, and 1300 growers (Fig. 1). The goal of the Northern Grapes Project was to enhance and support the development of these wineries and vineyards. As part of this goal, webinars were presented once monthly from November through April and recordings were archived on the project website (Northern Grapes Project, 2016). Thirty webinars were delivered to a live audience of 3083, with nearly 2400 views of recorded webinars, from 2012 to 2016. Webinar registrants came from 47 states and Canada.
The Northern Grapes Project is a partnership among multidisciplinary research and extension personnel at 11 universities and 23 regional and state winery and grape grower associations in the states highlighted in black.
Citation: HortTechnology hortte 28, 4; 10.21273/HORTTECH04088-18
To assess the impact of the webinar series, participants were surveyed to gauge willingness to pay, generated income, savings, and other variables related to the information delivered by the webinar presenters. The results were used to help demonstrate project impact to grant sponsors and stakeholders.
Materials and methods
In Dec. 2014, a survey of 12 questions was sent to participants pertaining to their webinar experience to fully estimate the value of the webinar series (Table 1). The institutional review board determined a full review was not necessary and thus did not fall under their purview. Survey participants were asked how much they would pay to watch a live webinar and access the archived webinars and how much money they had earned and saved from information presented in the webinar series. The survey was initially sent to the Northern Grapes Project webinar series listserv, which contained 2009 e-mail addresses at the time; 179 responded. After the survey was initiated, 170 e-mail addresses were added, for a total of 2179, resulting in a final response rate of 8.2%. Follow-up reminders were given at the end of each individual webinar until the survey was concluded in Mar. 2015. Blank or nonnumeric answers were considered missing values.
The 12 questions pertaining to webinar experience that were sent to the members of Northern Grapes Project mailing list.
Responses were gathered via Qualtrics (Qualtrics, Provo, UT) and statistically analyzed in JMP (version 12.0; SAS Institute, Cary, NC) for one-way analysis of variance and Pearson product-moment correlations in the multivariate pairwise correlations procedure. Nonresponse error was handled as described by Miller and Smith (1983) and Lindner and Wingenbach (2002). Two methods were employed to address the issue of nonresponse error: comparison of early to late respondents (<30 vs. >30 d past initial survey) and “days to respond” as a regression variable (first response to survey until last response). “Days to respond” was treated as an independent variable to which major response variables where regressed (Lindner and Wingenbach, 2002).
Results and discussion
Survey responses
Follow-up reminders were given at the end of individual webinars, so it was possible that someone who only attended one webinar would have only received one additional notification to fill out the survey beyond the first e-mail. The mean number of live webinars attended was three, whereas it was two for recorded webinars. Thus, there were many who had the opportunity to be reminded to fill out the survey. Although only 179 responses were returned, methods to address possible nonresponse bias are available.
Nonresponses are a typical problem for surveys, especially those administered via e-mail or other electronic method. Response rates for these types of surveys have been declining since the 1980s (Shaheen, 2001). Several reasons could lead to nonresponse to the survey, including perceived lack of confidentiality (Stafne et al., 2009), confidence in knowledge to answer questions, lack of incentive (Fincham, 2008), poor understanding of how results would be useful, survey fatigue (Shaheen, 2001), and exceeded expectation for interaction (Allred and Smallidge, 2010). Although e-mail surveys have poor response rates, it does not mean there is evidence of bias (Rindfuss et al., 2015), and a representative population is just as, or more, important than response rate (Fincham, 2008).
Two common methods of assessing bias are comparing early to late respondents and using “days to respond” as a regression variable against the dependent variables (Lindner and Wingenbach, 2002; Miller and Smith, 1983). Nonsignificant results would indicate that all respondents answered in a similar manner, and thus, these results could be applied to the entire survey population including nonresponders. In the present study, both methods showed survey responses were not significantly different (data not shown).
Additional postwebinar surveys (data not shown) indicated viewers were satisfied with the series; 90% of the respondents said the logistics were satisfactory. Furthermore, 83% said their awareness and 80% said their knowledge of webinar subjects increased at a moderate or higher level. Ninety-five percent of respondents indicated they would employ at least a little of what they learned in their vineyard or winery operation. These high percentage responses show that subjects taught during the webinar series were at the appropriate level for the audience. Because most respondent’s vineyards and wineries are relatively new and only small-to-medium in size (average 11.5 vineyard acres and 4282 cases of wine), it was important to tailor the information toward the novice and intermediate level.
The number of live webinars attended was significantly and positively correlated to the value of information delivered (r = 0.309, P = 0.001), reported increased revenue (r = 0.311, P = 0.006), and reported savings (r = 0.455, P < 0.001). Those willing to pay more for live webinars would pay more for unlimited access to recorded webinars and those who highly valued the information delivered would be more likely to pay more for a live webinar. However, not all attendees were willing to pay. This may stem from the long tradition within extension of not charging for programs. Several respondents had difficulty in understanding how to value something that was offered for free.
The number of recorded webinars viewed was significantly and positively correlated with the perceived value of the information received from webinars (r = 0.243, P = 0.011). It was also correlated to increased revenue (r = 0.312, P = 0.005), and savings (r = 0.281, P = 0.007). Those willing to pay more for unlimited access to recorded webinars valued the information delivered more than those who would pay less. Respondents willing to pay for live and recorded webinars reported an average increase of $524 in revenue from the information. Revenue and savings were both positively correlated to the perceived value of information received (P < 0.001). It was evident that those who attended more webinars placed value on what they learned and were able to put it into practice.
There were no significant correlations between the number of cases of wine produced by a winery and any economic factor. This was also true for number of acres farmed. The acreage size of vineyards ranged from 0.25 to 275 acres, and wine case production ranged from 4 to 60,000 cases. Therefore, the information from the webinars were applicable to a broad range of operation sizes.
Valuation of the series
Direct impact measures.
Both grant sponsors and stakeholders now require evidence of impact and placing a dollar value on the webinar series allows us to better estimate the impact of the Northern Grapes Project as a whole. Although not always easy to gauge, estimates of economic data are readily understood by stakeholders, sponsors, and legislators. Data from survey questions were used to calculate the direct reported value of the webinar series. Because nonresponse analyses indicated no bias within the population, all answers were multiplied by 2179 (Table 2). Attendees answered both live and recorded webinars had significant value that lead to increased income and savings. Webinar topics were designed to be very applied, such that one attending a live or recorded webinar could put the knowledge into practice immediately.
Questions and responses to the economic portion of the Northern Grapes Project survey by members of the electronic mailing list to establish project valuation.
Indirect impact measures.
Two previously published models were used with modifications. Stafne and Fidelibus (2014) estimated the value of time invested in online extension products and Bardon et al. (2014) calculated the green savings (i.e., reduction of environmental impacts) of participating in webinars instead of driving to attend presentations in person.
To complete these calculations, the following data were used:
Participants in live webinars numbered 3083; 2397 estimated views of recorded webinars (divided the total minutes recordings were viewed by 60 min, the length of each webinar); 2179 membership of the webinar series listserv; 147 miles, the average distance webinar participants said (in separate postwebinar surveys) they would drive to see a similar presentation in person; and, $25.13, estimated average hourly wage of webinar participants, generated from the 2015 Wine Business Monthly Salary Survey Report (Fisher, 2015).
Viewer time investment.
where VTI = viewer time investment; WV = number of webinar views (live and recorded); AHW = $25.13, estimated average hourly wage of webinar participants (note: webinar run time was 60 min); and VTI = 5480 × $25.13 = $137,712.40.
Time is a critical element in understanding how stakeholders value a product. Attending a webinar becomes a form of payment because webinar visitors pay with their time to watch the presentation, a variation of time spent reading (McDonald and Cranor, 2008). Calculating the opportunity cost as viewer time investment partially assesses how webinar viewers value their time. It becomes a part of the overall understanding in how they are willing to invest in webinar sessions.
Green impact.
The equations in Bardon et al. (2014) all use distance. The authors calculated the distance from the participant’s location during the online event to the nearest extension office or university. Instead, we used 147 miles, as previously described. All equations were multiplied by the number of live and recorded webinar views. All other values are as described by Bardon et al. (2014).
Motor vehicle operating savings = d × m
d = distance (miles)
m = standard business mileage rate ($0.54)
Motor vehicle operating savings = 147 × $0.54 × 5480 = $435,002.40
Salary savings = d × (w/γ)
d = distance (miles)
w = average wage paid per hour ($25.13)
γ= average rate of travel (52 mph)
Salary savings = 147 × ($25.13/52) × 5480 = $389,302.36
Carbon dioxide (CO2) savings = [(d/ſe) × 19.4] × ($6/2204.6)
d = distance (miles)
ſe = average fuel economy (21.4 miles/gal)
19.4= pounds of CO2 when burned in a typical internal combustion engine
$6 = social cost of producing a metric ton of CO2
2204.6 = number of pounds in a tonne (i.e., metric ton)
CO2 savings = [(147/21.4) × 19.4] × ($6/2,204.6) × 5480 = $1987.50
Total green impact = $826,292.26
One major consideration for developing and delivering webinars is reducing or eliminating the need for costly travel (Allred and Smallidge, 2010; Hoke et al., 2018; Rich et al., 2011; Verma and Singh, 2010). Reduction of travel results in additional indirect benefits by reducing environmental costs. Assessing environmental costs is often overlooked, yet including these costs makes sense as part of the valuation process for the Northern Grapes webinar series.
Our survey had broad representation of location and business size. Impact is a transformative change, but reach and engagement are part of the process that leads to overall impact. Changes in attitude, environment, and revenue are all important factors in considering impact assessment. Summing both direct and indirect measures generated a total value of the Northern Grapes Project webinar series estimated at $3,381,452.63. This type of program impact analysis could be readily applicable to other projects that have difficulty in impact assessment.
Sometimes even successful projects do not achieve a lasting effect (James, 2017) but the Northern Grapes Project webinar series has continued past the initial funding and new webinars are still being produced. An additional measure of the value of the Northern Grapes Project outreach is the fact that 13 state-based grower organizations and commercial businesses contributed $14,750 to support continued outreach through the Northern Grapes webinar series after the federal SCRI funding ceased (Martinson and Koeberle, 2016).
To that end, the Northern Grapes Project webinar series proved to be a successful endeavor in terms of both knowledge and financial impact.
Conclusions
At nearly $3.4 million in value, the total calculated value of the webinar series represents 66% of the $5.1 million the Northern Grapes Project received from the USDA SCRI program. Given the project had numerous other outreach products that also have value, we believe this is a very good return on investment. Of note, the survey results indicated the webinar series enhanced participant economic bottom line by more than $1000 each (revenue + savings), and the perceived value of the series, in terms of how much participants would pay to access webinars, was more than $190,000 (live + recorded). Economic impact studies of the wine grape industry within the states of the Northern Grapes Project in 2011 (start of project) and 2015 (close of project) at the University of Minnesota showed a 34% increase in economic output over the course of the project. Furthermore, the growth of the cold-climate grape and wine industries during the years that the Northern Grapes Project was active added tremendous value to the communities in which they are located through increased employment and economic growth of the industry (summarized in Tuck, 2016).
Units
Literature cited
Allred, S.B. & Smallidge, P.J. 2010 An educational evaluation of web-based forestry education J. Ext. 48 6 6FEA2 12 June 2018. <https://www.joe.org/joe/2010december/pdf/JOE_v48_6a2.pdf>
Bardon, R.E., Taylor, E., Hubbard, W. & Gharis, L. 2014 Calculating the “green” impact of online extension programs J. Ext. 52 3 3IAW2 12 June 2018. <https://joe.org/joe/2014june/pdf/JOE_v52_3iw2.pdf >
Fincham, J.E. 2008 Response rates and responsiveness for surveys, standards, and the journal Amer. J. Pharm. Educ. 72 2 43
Fisher, C. 2015 2015 Salary survey Wine Business Monthly XXII 10 64 75 12 June 2018. <http://www.winebusiness.com/wbm/?go=getDigitalIssue&issueId=7920>
Formiga, A.K., Stone, A., Heleba, D., McQueen, J. & Coe, M. 2014 Evaluation of the eOrganic webinar program J. Ext. 52 4 4FEA5 12 June 2018. <https://joe.org/joe/2014august/pdf/JOE_v52_4a5.pdf>
Hoke, A.M., Francis, E.B., Hivner, E.A., Lipsett Simpson, A.J., Hogentogler, R.E. & Kraschnewski, J.L. 2018 Investigating the effectiveness of webinars in the adoption of proven school wellness strategies Health Educ. J. 77 249 257
James, J. 2017 When a virtual success is a real failure: Learning from the enabling change and innovation webinar series Rural Ext. Innovation Systems J. 13 176 180
Lindner, J.R. & Wingenbach, G.J. 2002 Communicating the handling of nonresponse error in journal of extension research in brief articles J. Ext. 40 6 6RIB1 12 June 2018. <https://www.joe.org/joe/2002december/rb1.php>
Martinson, T.E. & Koeberle, A. 2016 Donors contribute $14,750 to support the northern grapes project Northern Grapes News 5 4 1 12 June 2018. <http://northerngrapesproject.org/wp-content/uploads/2012/03/NG-News-Vol5-I4-Dec-2016.pdf>
Martinson, T.E., Mansfield, A.K., Luby, J.J., Gartner, W.C., Dharmadhikari, M. & Domoto, P. 2016 The northern grapes project: Integrating viticulture, enology, and marketing of new cold-hardy wine grape cultivars in the midwest and northeast United States Acta Hort. 1115 3 12
McDonald, A.M. & Cranor, L.F. 2008 The cost of reading privacy policies I/S 4 540 565
Miller, L.E. & Smith, K.L. 1983 Handling nonresponse issues J. Ext. 21 5 45 50 12 June 2018. <https://www.joe.org/joe/1983september/83-5-a7.pdf>
Northern Grapes Project 2016 Viticulture, enology, and marketing for cold-hardy grapes. 12 June 2018. <http://northerngrapesproject.org/>
Pulec, K.E., Skelly, C.D., Brady, C.M., Greene, E.A. & Anderson, K.P. 2016 Effectiveness of webinars as educational tools to address horse industry issues J. Ext. 54 3 3TOT8 12 June 2018. <https://www.joe.org/joe/2016june/pdf/JOE_v54_3tt8.pdf>
Rich, S.R., Komar, S., Schilling, B., Tomas, S.R., Carleo, J. & Colucci, S.J. 2011 Meeting extension programming needs with technology: A case study of agritourism webinars J. Ext. 49 6 6FEA4 12 June 2018. <https://www.joe.org/joe/2011december/pdf/JOE_v49_6a4.pdf>
Rindfuss, R.R., Choe, M.K., Tsuya, N.O., Bumpass, L.L. & Tamaki, E. 2015 Do low survey response rates bias results? Evidence from Japan Demogr. Res. 32 797 828
Shaheen, K.B. 2001 E-mail survey response rates: A review J. Computer-Mediated Commun. 6 2 JCMC621 12 June 2018. <https://onlinelibrary.wiley.com/doi/full/10.1111/j.1083-6101.2001.tb00117.x>
Stafne, E.T. & Fidelibus, M.W. 2014 Reader time investment as a partial impact measure of online extension content J. Ext. 52 2 2TOT1 12 June 2018. <https://www.joe.org/joe/2014april/pdf/JOE_v52_2tt1.pdf>
Stafne, E.T., McGlynn, W.G. & Mulder, P.G. Jr 2009 Post-course evaluation of a grape management short course J. Ext. 47 3 3RIB4 12 June 2018. <https://www.joe.org/joe/2009june/pdf/JOE_v47_3rb4.pdf>
Tuck, B. 2016 Lessons learned: Northern grapes baseline and economic impact study Northern Grapes News 5 4 2 3 12 June 2018. <http://northerngrapesproject.org/wp-content/uploads/2012/03/NG-News-Vol5-I4-Dec-2016.pdf>
Verma, A. & Singh, A. 2010 Webinar—Education through digital collaboration J. Emerg. Technol. Web Intell. 2 131 136
VanDerZanden, A.M. 2013 Asynchronous continuing education for Iowa’s green industry professionals HortTechnology 23 677 682
Wine America 2017 2017 Economic report on wine. 12 June 2018. <http://wineamerica.org/wp-content/themes/wineamerica/pdfs/impact/United-States-Report.pdf>
Zavell, A.E., Greenberg, J.N., Alam, M., Armbrecht, E.S. & Maher, I.A. 2017 A 30-minute, monthly, live, webinar-based journal club activity alters the self-reported behaviors of dermatologic surgeons Dermatol. Surg. 43 1144 1147
Zoumenou, V., Sigman-Grant, M., Coleman, G., Malekian, F., Zee, J.M.K., Fountain, B.J. & Marsh, A. 2015 Identifying best practices for an interactive webinar J. Fam. Consum. Sci. 107 62 69