Proving Value: Tools and Strategies for Evaluating the Effectiveness of a Faculty Community of Practice
Concurrent Session 6
Brief Abstract
Online communities of practice are an effective way to support faculty engaged in distance learning. Evaluating the effectiveness of a community of practice for faculty cannot be overlooked. Participants in this interactive session will explore research-based strategies and instruments for evaluation and engage with examples of real world implementation.
Presenters
Extended Abstract
Introduction
As colleges, universities, and other learning institutions explore teaching and learning through online environments, online communities of practice may provide solutions to organizational and professional development needs. Instructional designers and other support professionals are already designing, implementing, and maintaining electronic communities of practice as a means for supporting faculty engaged in distance learning. Evaluating the effectiveness of a community of practice for faculty development is an important step that must not be overlooked. Evaluating a community of practice requires a systematic plan that is specific to the community being evaluated. Evaluation must occur through all phases of community design from conception through implementation and continuously through maintenance (Schwier, Campbell, & Kenny, 2007). Participants in this session will interact with real examples and research-based tools/strategies to evaluate the domain, community, and practice of an online community of practice.
Virtual Faculty Communities of Practice.
Communities of practice can be designed to meet specific professional development needs for faculty teaching at a distance. Research has shown that faculty learning communities facilitate connections for isolated faculty members, providing real opportunities for professional growth (Cox, 2004). Virtual communities of practice may help improve online teaching practices while helping faculty feel more supported. Sherer, Shea, and Kristensen (2003) found that providing a virtual space for a faculty community provided participants the ability to network with experts, search a library of knowledge created by members, explore best practices, and participate in the community without the constraints of time.
Any virtual community of practice must incorporate technology that is easy to navigate and must not bombard users with information overload (Hemp, 2009; Wang & Lai, 2006). This is especially true for faculty already overwhelmed by the academic tenure process (Trowler & Knight, 2000). Designing a virtual community of practice for faculty professional development for faculty teaching a distance must incorporate time-saving measures. Eib and Miller (2006), while describing a blended, but mostly face-to-face faculty community of practice, encouraged the exploration of a faculty community of practice tailored especially for distance learning environments.
Evaluating a Virtual Faculty Community of Practice
Wenger, McDermott, and Snyder (2002) identified three elements that guide community development efforts for focusing efforts to the various areas for fostering a well-rounded community: domain, community, and practice. Domain refers to the shared repertoire of the community, community addresses the interaction and role definition of members, and practice is the knowledge building and sharing efforts required for a community of practice to thrive. When evaluating a community of practice, all three domains should be evaluated.
Evaluating the Domain
Since communities of practice are sometimes viewed as social gatherings or groups, rather than valuable resources within organizations, designed communities often fail because they are not viewed as contributing educational or workplace knowledge of the organization (Wenger, 2010). A virtual community of practice for faculty teaching at a distance can prove its worth to the institution through systematic evaluation. Institutional goals are generally wrapped around the improvement of instructional practices, student engagement, and other student centered areas. Ways to implement evaluative processes that show alignment will vary from institution but might include: student end of course surveys, faculty climate surveys, or volume of distance learning related research.
Evaluating the Community
Evaluation of a community of practice can be conducted by analyzing the level of trust displayed in the online interactions. If trust is not built in a community of practice it will show through the quality and quantity of interactions in the online forum (Palloff & Pratt, 1999). Wenger, Trayner and Pratt (2011) identified five ways to measure quantity of interactions: (1) attendance at meetings, (2) number and characteristics of active participants, (3) subscribers, (4) logs and website statistics, (5) participant lists from synchronous meetings. Alem and Kravis (2005) evaluated the success of a virtual community of practice based on active membership, lurkers, number of messages per participant, on topic discussions, level of trust and satisfaction, and average length of membership. Community of practice leaders can learn a great deal about the health of a community of practice by evaluating the quality and quantity of interactions (Ke & Hoadley, 2009; Wenger et al., 2002)
Member perceptions of a community of practice can be used as an evaluative tool. Verburg and Adriessen (2008) used community of practice member perceptions to evaluate the effectiveness of seven communities of practice. Wenger et al, (2011) argued that value of a community of practice may be measured by collecting participant perceptions of learning or improved performance. Though participant perceptions of learning or improvement does not always translate into real learning or improvement.
Evaluating the Practice
In order to ensure that all components of a community of practice are examined for effectiveness, evaluations should also focus on the technology employed to host a virtual community of practice. Teo, Chan, Wei, and Zhang (2003) suggested evaluating the usefulness and usability of the medium housing the community of practice. Additionally, researchers in the same study examined perceived usefulness, sense of belonging, perceived ease of use, intentions for use, and adaptivity to evaluate the effectiveness of the medium for a virtual community of practice (Teo et al, 2003). Getting feedback on the medium from community of practice members can help community leaders make decisions about adjustments to the medium or whether to change the medium altogether.
A community of practice can also be evaluated by analyzing the quality and quantity of knowledge that is created (Derry & DuRussel, 1999). Wenger et al. (2002) suggests that the knowledge repository that is created at the center of the community of is itself a tool for evaluation. Knowledge repository evaluators should look for intensity of discussions, challenges of assumptions, length of threads, the bringing of experiences of practice into the space, debates on important issues, feedback on quality of responses to queries, new knowledge construction, and any reification efforts (Wenger et al., 2011). Evaluation of the quality and quantity of knowledge sharing in a virtual community of practice for faculty teaching at a distance should include an analysis of existing artifacts, discussions, and any research projects associated with improving online teaching and learning practices.
Summary
Faculty development is an important part of institutional planning. Virtual communities of practice are emerging as one way for enhancing faculty professional development. The session will provide tips and strategies for incorporating evaluation of virtual faculty communities of practice, with examples of how evaluation strategies have been implemented at a large university. Participants in this session will explore research-based tools and instruments for evaluating a faculty community of practice and will work together to develop their own instruments.
References
Alem, L., & Kravis, S. (2005). Design and evaluation of an online learning community: A case study at CSIRO. SIGGROUP Bulletin, 25(1), 20–24. doi: http://doi.acm.org/10.1145/1067699.1067703
Cox, M. D. (2004). Introduction to faculty learning communities. New Directions for Teaching and Learning, (97), 5-23. doi: http:://dx.doi.org/10.1002/tl.129
Derry, S. J., & DuRussel, L. A. (1999, July). Assessing knowledge construction in online learning communities. Paper presented at the Annual Meeting of the International Society for Artificial Intelligence in Education, Lemans, France (ERIC Document Reproduction No. Ed 446897).
Eib, B. J., & Miller, P. (2006). Faculty development as community building – An approach to professional development that supports communities of practice for online teaching. The International Review of Research in Open and Distance Learning; 7(2). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/299
Hemp, P. (2009). Death by information overload. Harvard Business Review, 87(9), 83.
Ke, F., & Hoadley, C. (2009). Evaluating online learning communities. Educational Technology Research & Development, 57(4), 487-410.
Palloff, R., & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco: Jossey-Bass Publishers.
Schwier, R. A., Campbell, K., & Kenny, R. F. (2007). Instructional designers' perceptions of their agency: Tales of change and community. In M. Keppell (Ed.),
Instructional design: Case studies in communities of practice (pp. 1-18). doi:10.4018/978-1-59904322-7.ch001
Sherer, P. D., Shea, T. P., & Kristensen, E. (2003). Online communities of practice: A catalyst for faculty development. Innovative Higher Education, 27(3), 183.
Teo, H.-H., Chan, H.-C., Wei, K.-K., & Zhang, Z. (2003). Evaluating information accessibility and community adaptivity features for sustaining virtual learning communities. International Journal of Human-Computer Studies, 59(5), 671-697. doi:10.1016/S10715819(03)00087-9
Trowler, P., & Knight, P. (2000). Coming to know in higher education: Theorizing faculty entry to new work contexts. Higher Education Research and Development, 19(1), 27-42. doi: 10.1080/07294360050020453
Verburg, R. M., & Andriessen J. H. (2006). The assessment of communities of practice. Knowledge and Process Management, 13(1), 13-25. doi: 10.1002/kpm.2413
Wang, C.-C., & Lai, C.-Y. (2006). Knowledge contribution in the online virtual community: Capability and motivation. In J. Lang, F. Lin, & J. Wang (Eds.), Knowledge science, engineering and management, lecture notes in computer science (Vol. 4092, pp. 442-453). Springer Berlin/Heidelberg. Retrieved from http://dx.doi.org/10.1007/11811220_37
Wenger, E. (2010). Communities of practice and social learning systems: The career of a concept. In C. Blackmore, (Ed.), Social learning systems and communities of practice. Berlin: Springer Verlag and the Open University
Wenger, E., McDermott, R. A., & Snyder, W. (2002). Cultivating communities of practice: A guide to managing knowledge. Boston, Mass: Harvard Business School Press.
Wenger, E., Trayner, B., & de Laat, M. (2011). Promoting and assessing value creation in communities and networks: A conceptual framework. Rapport 18, Ruud de Moor Centrum, Open University of the Netherlands.