EvaluLand – Détails, épisodes et analyse

This episode featured Kim Leonard and Sheila Robinson discussing their careers in evaluation and survey design. They shared how they met on Twitter in 2012 and began collaborating, writing blog posts that eventually became their book "Designing Quality Survey Questions" published by Sage Publications. Sheila and Kim discussed their process for writing the book, starting with blog content and building it out over several years into a full manuscript. They also covered common issues they see in surveys and tips for developing effective survey questions.

Sheila Robinson: https://www.sheilabrobinson.com/ & https://www.linkedin.com/in/sheilabrobinson/
Kim Leonard: https://leonardrande.com/ & https://www.linkedin.com/in/kim-firth-leonard-1ba9447/

Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This month I chat with Brittany Dernberger about her evaluation background and practice.

Brittany Dernberger is a sociologist and gender inequality expert who has led research, evaluation, and organizational learning across academia, philanthropy, government, large international NGOs, and small nonprofits. Brittany currently leads global initiatives to measure systems-level change at CARE and co-chairs the American Evaluation Association Systems in Evaluation Topical Interest Group.

Learn more about Brittany’s work at http://brittanydernberger.com/. Access CARE’s completed Systems-Level Impact evaluations at https://careevaluations.org/evaluation/keywords/systems-level-impact/. The AEA Systems in Evaluation TIG paper on Principles for Systems Thinking in is available at https://www.betterevaluation.org/sites/default/files/SETIG-Principles-FINAL-DRAFT-2018-9-9.pdf.

Music by Matt Ingelson, http://www.mattingelsonmusic.com/

On this episode, I’m chatting with Dr. Tristi Nichols about her work conducting international evaluation and her journey into the international and evaluation spaces.

• Footprint evaluation
• OECD DAC Criteria for Evaluation Framework

List of websites to find international evaluation:

• https://www.ungm.org/ type “evaluation” in the title bar [for UN only]
• https://reliefweb.int/ or https://reliefweb.int/jobs
• https://www.evalcommunity.com/

Tristi Nichols
tnichols@manitouinc.com
914-414-8288
@tristiempo
https://www.linkedin.com/in/drtristinichols/

Music by Matt Ingelson, http://www.mattingelsonmusic.com/

On this episode, I’m chatting with Dr. Ayesha Boyce about the evaluation job market. Ayesha provides a ton of great insights and information about how to search for jobs, what to consider when looking for jobs, and tips for being a strong job candidate.

Ayesha was previously on the podcast discussing "teaching evaluation and supporting students and colleagues of color."

Ayesha Boyce
Email: ayesha.boyce@asu.edu
Twitter: @AyeshaBoyce
Website: https://education.asu.edu/about/people/ayesha-boyce

Ayesha Boyce is currently an associate professor in the Division of Educational Leadership and Innovation at Arizona State University. She also co-directs the STEM Program Evaluation Lab. Boyce’s scholarship focuses on attending to value stances and issues related to diversity, equity, inclusion, access, cultural responsiveness, and social justice within evaluation—especially multi-site, STEM, and contexts with historically marginalized populations. She also examines teaching, mentoring, and learning in evaluation. She has evaluated more than 55 programs funded by the National Science Foundation (NSF), US Department of Education, National Institutes of Health, and Spencer and Teagle foundations. Boyce is a 2019 American Evaluation Association Marcia Guttentag Promising New Evaluator Award recipient. In her teaching and mentorship, Boyce encourages students to develop a strong methodological foundation, conduct studies based on democratic principles, and promote equity, fairness, inclusivity, and diversity.

Music by Matt Ingelson, http://www.mattingelsonmusic.com/

On this episode, I’m chatting with Eric Persaud about evaluation at the National Institutes of Health, including his work in the Worker Training Program at the National Institute of Environmental Health Sciences. We also talked about his dissertation, how he got into the field of evaluation, and his experiences in conducting evaluation in general.

Eric Persaud received his doctorate in Public Health at the Department of Environmental and Occupational Health Sciences, State University of New York-Downstate Health Sciences University. He focuses on evaluating and researching training programs related to preparing workers for emergencies and disasters, and hazardous workplaces. He has been involved in evaluation and research associated with fentanyl and first responders, opioids and the workplace, and protecting workers from COVID-19.

You can reach Eric Persaud at Eric.Persaud@NIH.gov

You can learn more about the NIEHS Worker Training Program at https://www.niehs.nih.gov/careers/hazmat/about_wetp/index.cfm

• Biosafety and infectious disease occupational health training from the NIEHS Worker Training Program: A Historical look at capacity building that supported a COVID-19 response. Eric Persaud, Deborah Weinstock, Demia S. Wright. Journal of Emergency Management. 2022.
• Opioids and the Workplace Prevention and Response Awareness Training: Mixed Methods Follow-Up Evaluation - Eric Persaud, Aimee Afable, Laura A. Geer, Paul Landsbergis, 2021
• COVID-19 Biosafety Training and Infectious Disease Response Evaluation Report. NIEHS Worker Training Program. 2021.

Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This episode I chatted with Jan Noga about systemic design thinking. There’s a wealth of resources and information provided below!

Jan Noga
Jan.Noga@pathfinderevaluation.com
www.pathfinderevaluation.com

Jan Noga is an independent evaluation consultant based in Cincinnati, Ohio. She holds a bachelor’s degree from Stanford in developmental and counseling psychology with specialization in early and middle childhood and a master’s degree from the University of Cincinnati in instructional design and technology. Jan has worked in the non-profit and public sectors in human services and education for more than 30 years in roles spanning teaching, research, policy, and program planning and evaluation. As a program evaluator, Jan has planned and conducted both large and small-scale evaluations and provided organizational consulting and capacity building support to clients. She has also taught courses and workshops on such topics as systems thinking, systemic design thinking, research methods and techniques, program planning and development, and survey design and analysis. Jan has been a member of AEA since 2000 and was one of the founding members of the Systems in Evaluation TIG, serving as program chair and then TIG chair from 2004-2012. She is particularly interested in the use of systems approaches as a foundation for design, planning, implementation, and evaluation of change efforts in the human service and education arenas.

Hands on resources:

• Williams, Bob. 2020. Systemic evaluation design: A workbook. Available for download from https://bobwilliams.gumroad.com/
• Williams, Bob. 2021. Systems diagrams: A practical guide. Available for download from https://bobwilliams.gumroad.com/

Good for starting out

• Anderson, V. & Johnson, L. (1997). Systems thinking basics: From concepts to causal loops. Waltham, MA: Pegasus Communications.
• Meadows, D.H. (2008). Thinking in systems: A primer. White River Junction, VT: Chelsea Green Publishing.
• Ramage, M. & Shipp, K (2009). Systems Thinkers. New York: Springer.
• Sweeney, L.B. & Meadows, D. (2010). The systems thinking playbook. White River Junction, VT: Chelsea Green Publishing.
• Williams, B. & Hummelbrunner, R. (2011). Systems concepts in action: A practitioner’s toolkit. Stanford, CA: Stanford University Press.
• Williams, B. and Imam, I, eds. (2007). Systems concepts in evaluation: An expert anthology. Point Reyes, CA: EdgePress.
• Williams, B. and Van’t Hoft, S (2016). Wicked solutions: A systems approach to complex problems. Available at http://bit.ly/1SVoOH3

Good for more advanced reading:

• Bamberger, M, Vaessen, J., & Raimondo, E. (eds.) (2016) Dealing with complexity in development evaluation: A practical approach. Thousand Oaks, CA: Sage Publications.
• Cabrera, D., Colosi, L., & Lobdell, C. (2008) Systems thinking. Evaluation and Program Planning, 31(3), 299-310.
• Cabrera, D. & Cabrera, L (2015). Systems thinking made simple: New hope for solving wicked problems. Odyssean Publishing.
• Capra, F & Luisi, PL (2016). The systems view of life: A unifying vision (6th printing). New York: Cambridge University Press.
• Checkland, P. (1999). Systems thinking, systems practice. New York: John Wiley & Sons, Ltd. Cunliff, E., (2002) Connecting systems thinking to action, The Systems Thinker, 15(2), 6-7.
• Eoyang, G.H. & Holladay, R.J. (2013) Adaptive action: Leveraging uncertainty in your organization. Stanford: Stanford Business Books.
• Karach, R, (1997) How to see structure, The Systems Thinker, 8(4), 6-7.
• Patton M.Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press.
• Patton, M.Q., McKegg, K., & Wehipeihana, N., eds. (2015). Developmental evaluation exemplars: Principles in practice. New York: Guilford Press.
• Senge, P. (1990) The fifth discipline: The art and practice of the learning organization. New York: Doubleday.
• Stroh, DP (2015). Systems thinking for social change. White River Junction, VT: Chelsea Green Publishing.
• Ulrich, W & Reynolds, M (2010). Critical systems heuristics. In: Reynolds, Martin and Holwell, Sue eds. Systems approaches to managing change: A practical guide. London: Springer, pp. 243–292.
• von Bertalanffy, Ludwig. (1950). The theory of open systems in physics and biology. Science,
• 13, 23-29.
• von Bertalanffy, Ludwig. (1968). General systems theory. New York: George Braziller, Inc.
• Wolf-Branigin, M. (2013) Using complexity theory for research and evaluation. Oxford: Oxford University Press.

Some other resources:

• International Society for Systems Sciences
• https://aea365.org/blog/systemic-design-thinking-for-evaluation-of-social-innovations-a-pd-for-intermediate-and-advanced-evaluators-by-jan-noga/
• http://www.epreconsulting.com/SETIG%202018%20Principles.pdf
• https://systemic-design.org/
• https://modus.medium.com/what-the-is-systems-design-e005c1e9fef8
• https://rsdsymposium.org/
• Martin Reynolds Open University

Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This episode I chat with Martha Brown, PhD about trauma-informed evaluation, but more specifically about how we as evaluators can help organizations become more trauma-informed. We talk about trauma, the SAMHSA model of trauma-informed work, and much more!

Martha talked about trauma-informed evaluation also on the Glass Frog podcast with Debbie Gowensmith and on the Community Possibilities podcast. Other resources mentioned include the AEA365 series on trauma-informed evaluation and the Center for Victims of Torture.

For an introductory, self-paced course to learn more about trauma and what it means to be trauma-informed, visit https://www.rjaeconsulting.com/becoming-trauma-informed-how-to-apply-the-principles-of-trauma-informed-care-to-your-life-work-and-community- Suggested donation is $75-99 - special module for evaluators.

martha@rjaeconsulting.com

Martha Brown is president and founder of RJAE Consulting and an active member of the American Evaluation Association (AEA). She received her doctorate in Curriculum & Instruction from Florida Atlantic University in 2015 and is an accomplished author, evaluator, program designer, presenter, trainer, and teacher. Dr. Brown’s research has been published in numerous peer-reviewed journals and edited books. She also authored the best-selling book, Creating Restorative Schools: Setting Schools Up to Succeed, available from Living Justice Press. She has presented at numerous national and international conferences on the topics of educational policy reform, restorative justice, arts education, and culturally responsive evaluation.

Martha’s content expertise is in trauma informed organizations and practices, restorative justice, and arts evaluation. She brings a unique blend of creativity, caring, passion, and technical skills to her work. Martha utilizes a restorative approach to her work as an evaluator, always placing relationships at the center of her work.

Music by Matt Ingelson, http://www.mattingelsonmusic.com/

I chat with Libby Smith and Tiffany Tovey, two of the three hosts of the Radical Re(imagining) podcast, to chat about what felt radical about the 2021 AEA virtual conference.

We'd love to hear from you! What felt radical to you about the conference? Feel free to add your reflections on LinkedIn or Twitter. Be sure to use the #Eval21 hashtag, and tag us in your reflections!

Twitter accounts:

• @RadReImagining
• @work_with_libby
• @Tiffany7001
• @EvaluLand
• @danawanzer

Music by Matt Ingelson, http://www.mattingelsonmusic.com/

Jennifer Puma and I talk about project management in evaluation, including how she manages evaluation projects from start to finish and how she brings her project management training into her organizations and with the clients she works with. We discuss things like scoping projects, setting up ad hoc meetings, how project management aligns with various types of budgeting approaches, and so much more!

Also, be sure to check out the Glass Frog Podcast, which is another evaluation-related podcast that I highly recommend!

As a reformed management consultant, Jennifer leverages more than 15 years of consulting and project management experience to manage program evaluations and research projects at Glass Frog. Her functional specialty is in the development and practical application of theories of change and evaluation frameworks. Jennifer holds a B.A. degree from The College of New Jersey and a M.P.A. degree from The Maxwell School of Citizenship and Public Affairs at Syracuse University. She lives in Brooklyn, NY, with her husband and ridiculous rescue dog, Sammy.

Here is the John Oliver clip that Jen mentions as well: Nothing Good Happens in Excel. As Jen says, "When it comes to project management, we do some of our work in Excel and share with clients but are careful to check in, first, about their level of comfort with Excel. John Oliver's observation is funny because it's true."

Music by Matt Ingelson, http://www.mattingelsonmusic.com/

Jennifer Bisgard and I talk about project management in evaluation, including how she manages evaluation projects from start to finish and useful tools in her company's practice. Jennifer has a wealth of information in her over 20 years of experience leading evaluation projects, and I learned a lot about how to better do this work from her. Khulisa has a wonderful library of resources including blog posts about QASP, analyzing WhatsApp data, and using rubrics. Every Tuesday they publish #EvalTuesdayTip.

Ms Jennifer Bisgard co-founded Khulisa Management Services in 1993. An expert in monitoring and evaluation (M&E) and organisational development, she leads evaluations and capacity building assignments in the Education, Power, and Democracy and Governance sectors. Khulisa has about 100 staff, based in Johannesburg, Nairobi, Lusaka, Mbabane, and Bethesda.

Prior to establishing Khulisa, Jennifer was the Senior Education Specialist at USAID/Pretoria from 1988 to 1993.

Ms Bisgard has served on boards for the: African Evaluation Association (AfrEA), International Organization for Cooperation in Evaluation (IOCE) and South African Monitoring and Evaluation Association (SAMEA). She is a current board member of BetterEvaluation.

Jennifer co-authored a chapter of “Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned” published by Sage Publishers in August 2018. The book, edited by Kylie Hutchinson, includes a forward by Michael Quinn Paton.

She has a Master’s Degree in Social Change and Development from Johns Hopkins University.

She is an American but has been in South Africa for 33 years. She is married to Marc (a dual Dutch-South Africa citizen) and has three boys: Zuko (adopted, now 23); Dylan (18) and Thomas (14). The family is completed by Border Collie, Riley and Africanis, Milly.

Music by Matt Ingelson, http://www.mattingelsonmusic.com/