I chat with Robin Lin Miller about the book she wrote with George Ayala, "Breaking Barriers: Sexual and Gender Minority-Led Advocacy to End AIDS in Africa and the Caribbean." We discuss MPact, Project ACT, and the evaluation of the initiatives.
Robin Lin Miller, PhD is Professor of Psychology, director of doctoral training in ecological-community psychology, and associate director of training in program evaluation at Michigan State University. She served as lead evaluation specialist for Gay Men’s Health Crisis in the early years of the HIV epidemic and established its first Department of Evaluation. She specializes in evaluating community-led programs, including human rights and advocacy initiatives. Her evaluations – principally conducted in the United States, Africa, and Caribbean – focus on adolescent and Black gay and bisexual men, bisexual girls, transgender women, and male sex workers. She served as lead scientist on the American Psychological Association’s Task Force on Therapeutic Approaches to Sexual Orientation Distress, which is routinely cited in legislation banning conversion therapy practices. Awards include the 2023 Alva and Gunnar Myrdal Evaluation Practice Award from the American Evaluation Association for substantive cumulative contributions to the development of LGBTQ evaluation practice, and the 2022 Exemplary Project W. K. Kellogg Foundation Community Engagement Scholarship Award for evaluating human rights advocacy for LGBTQ people in Africa and the Caribbean. She a member of the Academy for Community Engagement Scholarship, and a fellow of the American Psychological Association and Society for Community Research and Action. She is author of more than 100 scholarly publications, including Breaking Barriers: Sexual and Gender Minority-led Advocacy to End AIDS in Africa and the Caribbean, published by Oxford University Press, co-authored by activist-scholar George Ayala. Past funders include AmFAR, the U.S. Centers for Disease Control and Prevention, National Institute of Mental Health, National Institute of Drug Abuse, the Eunice Kennedy Shriver National Institute of Child Health and Development, and the U.S. Department of State.
Contact information: mill1493@msu.edu
In this episode, I chat with the authors of The Garden of Evaluation Approaches about their innovative and helpful framework for thinking about evaluation theory and approaches. They each share their personal journeys into evaluation as well as into this work they have been collaborating on for years. We also discuss evaluation theory more broadly, including how to make it more accessible to new and emerging evaluators.
Bianca Montrosse-Moorhead is a Professor of Research Methods, Measurement, and Evaluation at the University of Connecticut, where she also directs the Partnership for Evaluation and Educational Research (PEER). As Co-Editor-in-Chief of New Directions for Evaluation and a fervent advocate for evaluation, Bianca has dedicated her career to bridging the space between evaluation theory and practice. Her work encompasses a broad spectrum of contributions, from evaluating various educational and social programs using diverse methodologies to enhancing the professional training of evaluators worldwide.
Daniela Schroeter serves as a Presidential Innovation Professor and Associate Professor at Western Michigan University. With a Ph.D. in Interdisciplinary Evaluation and over twenty years of research, Daniela has contributed substantially to developing evaluation theories and methodologies. Her global work has involved diverse sectors, focusing on enhancing the capacity and effectiveness of evaluations. Daniela also co-edits the Teaching & Learning of Evaluation section of the American Journal of Evaluation.
Lyssa Wilson Becho serves as a Principal Research Associate at The Evaluation Center of Western Michigan University. Garnering the 2024 Marcia Guttentag Promising New Evaluator Award, Lyssa has made a significant impact in advancing evaluation methodologies, focusing on culturally responsive practices and promoting equity within evaluation processes. Lyssa is a co-Executive Editor for the Journal of MultiDisciplinary Evaluation.
Collectively, their innovative work in evaluation spans continents, cementing their reputation as leaders in advancing the practical, theoretical, and methodological facets of the discipline.
Bianca Montrosse-Moorhead
Daniela Schröter
Lyssa Wilson Becho
Resources mentioned:
Project vita (biography or résumé) with links to all free and publicly available resources: https://tinyurl.com/EvalGardenVita
Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2024). The garden of evaluation approaches visualization. Journal of MultiDisciplinary Evaluation, 20(48), 49–58. https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/1029
Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2024). The garden of evaluation approaches. American Journal of Evaluation, 45(2), 166–185. https://doi.org/10.1177/10982140231216667
Bledsoe, K. L., & Graham, J. A. (2005). The use of multiple evaluation approaches in program evaluation. American Journal of Evaluation, 26(3), 302-319. https://doi.org/10.1177/1098214005278749
King, J. (2024) Garden of evaluation approaches with SROI and CBA https://linkedin.com/posts/julian-king-87a015a_cba-sroi-vfi-activity-7201338499776618497-J3-z/
Miller, R. L. (2010). Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31(3), 390–399. https://doi.org/10/fmzjxp
In this week’s episode, I talk with Dr. Jennifer Villalobos about evaluator education, the scholar-practitioner model, evaluation careers, and how the current administration is affecting evaluation.
Contact information: Jennifer P. Villalobos - jennifer.villalobos@cgu.edu
About Dr. Jennifer Villalobos: Jennifer P. Villalobos, PhD, is an Assistant Professor of Practice and Director of the Doctor of Evaluation Practice (D.Eval) program at Claremont Graduate University. As an evaluator, scholar-practitioner, and organizational psychologist, her work focuses on advancing socially responsive evaluation, helping organizations increase their DEI profile, positive psychological interventions, and the intersection of evaluation education and practice. Outside of work, Jennifer enjoys spending time with her husband and three kids, cheering them on in their many sports and theatrical events. She’s also a firm believer in the power of community—whether that means mentoring students, collaborating with peers, or just gathering people around good food and conversation.
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
I share my reflections on teaching Foundations of Evaluation after a major course revision, including how it's been going for the first four weeks.
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
This episode featured Kim Leonard and Sheila Robinson discussing their careers in evaluation and survey design. They shared how they met on Twitter in 2012 and began collaborating, writing blog posts that eventually became their book "Designing Quality Survey Questions" published by Sage Publications. Sheila and Kim discussed their process for writing the book, starting with blog content and building it out over several years into a full manuscript. They also covered common issues they see in surveys and tips for developing effective survey questions.
Sheila Robinson: https://www.sheilabrobinson.com/ & https://www.linkedin.com/in/sheilabrobinson/
Kim Leonard: https://leonardrande.com/ & https://www.linkedin.com/in/kim-firth-leonard-1ba9447/
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
This month I chat with Brittany Dernberger about her evaluation background and practice.
Brittany Dernberger is a sociologist and gender inequality expert who has led research, evaluation, and organizational learning across academia, philanthropy, government, large international NGOs, and small nonprofits. Brittany currently leads global initiatives to measure systems-level change at CARE and co-chairs the American Evaluation Association Systems in Evaluation Topical Interest Group.
Learn more about Brittany’s work at http://brittanydernberger.com/. Access CARE’s completed Systems-Level Impact evaluations at https://careevaluations.org/evaluation/keywords/systems-level-impact/. The AEA Systems in Evaluation TIG paper on Principles for Systems Thinking in is available at https://www.betterevaluation.org/sites/default/files/SETIG-Principles-FINAL-DRAFT-2018-9-9.pdf.
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
This month I talk with Allison Prieur about her experiences as a graduate student in evaluation and running a business.
Some resources mentioned or suggested for listeners:
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
In this episode, I chat with Corey Newhouse and Jessica Manta Meyer, both from Public Profit, about how to make evaluation capacity building sticky for the organizations we work with.
Corey got her start as a teacher at Summerbridge Cincinnati in the early 1990s and has been involved in educational equity and social justice movements ever since. She founded Public Profit to build a team that would seamlessly blend social science research methods, organizational change strategy, and a deep commitment to supporting changemakers.
As the Founder and Principal of Public Profit, Corey leads the team’s strategic direction, external relationships, and business development. In addition, Corey serves as an internal thought partner to project teams, assisting with the design of Public Profit’s engagements in evaluation, capacity building, and strategic program design. She is co-author of Public Profit’s Creative Ways to Solicit Stakeholder Feedback and Dabbling in the Data, and a contributor to Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned. She is a co-editor of the volume, Measure, Use, Improve! Data Use in Out-of-School Time.
Jessica began her career as a peer volunteer on a national youth talkline providing referrals and support to youth in crisis. Challenged by what she heard on the talkline, and by some of her own experiences with the health care system, she developed an interest in health education, equity and advocacy, which launched a career in the social services sector. Eventually, Jessica combined that with a lifetime love of numbers and inquiry, which led her to program evaluation.
Prior to joining Public Profit, Jessica had several years of experience creating, directing and evaluating a range of youth development, LGBT, and health programs. Her work has run the gamut of nonprofit and social service roles including nonprofit finance, human resources, development and agency-led evaluations.
Jessica directs many of the projects at Public Profit. She designs evaluation studies, develops project strategy, and manages implementation including all aspects of data collection, analysis, and reporting. An expert facilitator, Jessica also facilitates large stakeholder meetings and evaluation capacity building trainings, and provides evaluation coaching to clients and staff alike.
In this episode, I chat with Michael Moses about theory-based approaches for navigating complexity, adaptive management, participatory strategy, and actions to shift power.
Michael Moses
mmoses@encompassworld.com
LinkedIn
Michael Moses is a strategist, facilitator, and evaluator with over 12 years of experience working with public and private sector partners to achieve social impact. He advises foundations, companies, governments, and nonprofits in their efforts to design and implement strategies for advancing change, including by capturing and using data to navigate the complex systems in which they work. In doing so, he helps changemakers learn how to improve programs and organizations, adapt, and over time, strengthen their impact.
In this episode, I chat with Dr. Tatiana Elisa Bustos on community-based participatory research (CBPR). We talked about what it is, how it compares to research and other similar forms of inquiry, and how to get started doing CBPR.
Disclaimer: Views expressed here are personal and not reflective of the speaker's respective employers or agencies.
Dr. Tatiana Elisa Bustos
tbust002@gmail.com
@TElisa72
https://www.linkedin.com/in/tebustos/
Dr. Tatiana Elisa Bustos knows that community partner engagement is key to understanding social issues. She’ll share her experience applying community-based participatory research approaches.
Dr. Bustos innovates outside the box ways to do research that invite community participation, improving programs through implementation with a social justice lens. As a 1st generation college student and the daughter of Nicaraguan immigrants, equity is deeply important to her. She is an author and award-winning researcher. She leads professional development workshops on implementation science and community based participatory research.
She received her PhD in Community Psychology from Michigan State University, an MS in Psychology from Nova Southeastern University, and a BA in Psychology from Florida International University. Connect with her on LinkedIn.
Dr. Bustos also appeared on The Sci-Files on Impact 89FM and Beyond the Manuscript, the podcast of Progress in Community Health Partnerships.
Professional Organizations
Training Institutes
Toolkits
Journals
In this episode, I provide a brief overview of the #Eval22 conference, describe the schedule at a glance, and provide tips for getting the most out of the conference. I hope to see you in New Orleans! Say hi to me at the conference to get an EvaluLand nametag ribbon.
I talked with Dr. Tamara Hamai of Hamai Consulting about contracting, subcontracting, and independent consulting in evaluation. We discussed the minor differences between contracting and subcontracting, cleared up a common misconception of what subcontracting relationships are like, how to get into subcontracting, and tips for setting up contracts.
Tamara Hamai
Hamai Consulting
assistant@hamaiconsulting.com
https://sustainableimpact.co
Tamara Hamai, Ph.D., has dedicated her career to empowering organizations and rebuilding our global systems to encourage children’s holistic growth and well-being, from prenatal through the completion of higher education – especially those who are most vulnerable and facing the greatest challenges. In 2008, she founded Hamai Consulting as a platform to help organizations increase their impact, stability, and strength to make a bigger impact in children’s lives. Dr. Hamai’s work spans most aspects of child development, such as early childhood education, higher education, child welfare, parenting and discipline, trauma, and adverse childhood experiences. She has previously been featured by KTLA, KPIX CBS San Francisco, ABC 7 News, NBC Radio, American Psychological Association, Western Psychological Association, Institute for Violence, Abuse, and Trauma, American Evaluation Association, National Head Start Association, and several blogs. She is also a reviewer and on the Editorial Board for publications such as the Journal of Aggression, Maltreatment & Trauma, Journal of Child & Adolescent Trauma, Journal of Sexual Abuse, and the Journal of Child Custody.
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
This episode I chatted with Carrie Tanasichuk and Harry Daley about how they have been using theories of change as their main process for facilitating strategic planning with non-profits.
In the episode, they also mentioned:
*Carrie Tanasichuk *(Twitter DrCarrieTee & [carrie@sjfoundation.ca](carrie@sjfoundation.ca)) has 16 years of experience in evaluation. She is passionate about using her expertise to help organizations demonstrate impact, improve, and innovate. She has worked across sectors (non-profit, for-profit, government) in diverse areas, including poverty reduction, youth development, criminal justice, health promotion and screening, and technology. She has a Ph.D. in Applied Social Psychology from the University of Saskatchewan.
Harry Daley ([harry@sjfoundation.ca](harry@sjfoundation.ca)) was born and raised in Saint John, New Brunswick. He has dedicated his personal and professional life to working with organizations that focus on poverty reduction and amplifying the youth voice. He has used evaluation as a tool for designing experiential and participatory youth programming and believes evaluation is an integral aspect of program design and facilitation rather than something that lives outside the programs. Harry has a Bachelor of Philosophy in Interdisciplinary Leadership from the University of New Brunswick’s Renaissance College.
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
This episode I chatted with Bradlie Nabours about how he got into the field of evaluation and his experience applying for evaluation jobs. He also talks about his evaluation experience and provides great tips on applying for jobs in the evaluation space.
Bradlie is a perinatal health program evaluator at Johns Hopkins All Children’s Hospital in St.Petersburg, Fl. He works for a federal Healthy Start program focused on reducing adverse perinatal outcomes for black women and their families. He holds a Bachelors of Science in Public Health from Georgia Southern University and completed his Master’s in Public Health with a graduate certificate in Maternal and Child Health from the University of South Florida. He started his evaluation journey while in grad school where he worked as a graduate assistance on the evaluation of a non-profit family violence prevention program. While in grad school his research focused racial disparities in birth outcomes which led him to start his career at the intersection of his research interest and evaluation.
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
On this episode, I’m chatting with Dr. Tristi Nichols about her work conducting international evaluation and her journey into the international and evaluation spaces.
List of websites to find international evaluation:
Tristi Nichols
tnichols@manitouinc.com
914-414-8288
@tristiempo
https://www.linkedin.com/in/drtristinichols/
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
On this episode, I’m chatting with Dr. Ayesha Boyce about the evaluation job market. Ayesha provides a ton of great insights and information about how to search for jobs, what to consider when looking for jobs, and tips for being a strong job candidate.
Ayesha was previously on the podcast discussing "teaching evaluation and supporting students and colleagues of color."
Ayesha Boyce
Email: ayesha.boyce@asu.edu
Twitter: @AyeshaBoyce
Website: https://education.asu.edu/about/people/ayesha-boyce
Ayesha Boyce is currently an associate professor in the Division of Educational Leadership and Innovation at Arizona State University. She also co-directs the STEM Program Evaluation Lab. Boyce’s scholarship focuses on attending to value stances and issues related to diversity, equity, inclusion, access, cultural responsiveness, and social justice within evaluation—especially multi-site, STEM, and contexts with historically marginalized populations. She also examines teaching, mentoring, and learning in evaluation. She has evaluated more than 55 programs funded by the National Science Foundation (NSF), US Department of Education, National Institutes of Health, and Spencer and Teagle foundations. Boyce is a 2019 American Evaluation Association Marcia Guttentag Promising New Evaluator Award recipient. In her teaching and mentorship, Boyce encourages students to develop a strong methodological foundation, conduct studies based on democratic principles, and promote equity, fairness, inclusivity, and diversity.
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
On this episode, I’m chatting with Eric Persaud about evaluation at the National Institutes of Health, including his work in the Worker Training Program at the National Institute of Environmental Health Sciences. We also talked about his dissertation, how he got into the field of evaluation, and his experiences in conducting evaluation in general.
Eric Persaud received his doctorate in Public Health at the Department of Environmental and Occupational Health Sciences, State University of New York-Downstate Health Sciences University. He focuses on evaluating and researching training programs related to preparing workers for emergencies and disasters, and hazardous workplaces. He has been involved in evaluation and research associated with fentanyl and first responders, opioids and the workplace, and protecting workers from COVID-19.
You can reach Eric Persaud at Eric.Persaud@NIH.gov
You can learn more about the NIEHS Worker Training Program at https://www.niehs.nih.gov/careers/hazmat/about_wetp/index.cfm
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
This episode I chatted with Jan Noga about systemic design thinking. There’s a wealth of resources and information provided below!
Jan Noga
Jan.Noga@pathfinderevaluation.com
www.pathfinderevaluation.com
Jan Noga is an independent evaluation consultant based in Cincinnati, Ohio. She holds a bachelor’s degree from Stanford in developmental and counseling psychology with specialization in early and middle childhood and a master’s degree from the University of Cincinnati in instructional design and technology. Jan has worked in the non-profit and public sectors in human services and education for more than 30 years in roles spanning teaching, research, policy, and program planning and evaluation. As a program evaluator, Jan has planned and conducted both large and small-scale evaluations and provided organizational consulting and capacity building support to clients. She has also taught courses and workshops on such topics as systems thinking, systemic design thinking, research methods and techniques, program planning and development, and survey design and analysis. Jan has been a member of AEA since 2000 and was one of the founding members of the Systems in Evaluation TIG, serving as program chair and then TIG chair from 2004-2012. She is particularly interested in the use of systems approaches as a foundation for design, planning, implementation, and evaluation of change efforts in the human service and education arenas.
Hands on resources:
Good for starting out
Good for more advanced reading:
Some other resources:
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
This episode I chat with Martha Brown, PhD about trauma-informed evaluation, but more specifically about how we as evaluators can help organizations become more trauma-informed. We talk about trauma, the SAMHSA model of trauma-informed work, and much more!
Martha talked about trauma-informed evaluation also on the Glass Frog podcast with Debbie Gowensmith and on the Community Possibilities podcast. Other resources mentioned include the AEA365 series on trauma-informed evaluation and the Center for Victims of Torture.
For an introductory, self-paced course to learn more about trauma and what it means to be trauma-informed, visit https://www.rjaeconsulting.com/becoming-trauma-informed-how-to-apply-the-principles-of-trauma-informed-care-to-your-life-work-and-community- Suggested donation is $75-99 - special module for evaluators.
Martha Brown is president and founder of RJAE Consulting and an active member of the American Evaluation Association (AEA). She received her doctorate in Curriculum & Instruction from Florida Atlantic University in 2015 and is an accomplished author, evaluator, program designer, presenter, trainer, and teacher. Dr. Brown’s research has been published in numerous peer-reviewed journals and edited books. She also authored the best-selling book, Creating Restorative Schools: Setting Schools Up to Succeed, available from Living Justice Press. She has presented at numerous national and international conferences on the topics of educational policy reform, restorative justice, arts education, and culturally responsive evaluation.
Martha’s content expertise is in trauma informed organizations and practices, restorative justice, and arts evaluation. She brings a unique blend of creativity, caring, passion, and technical skills to her work. Martha utilizes a restorative approach to her work as an evaluator, always placing relationships at the center of her work.
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
I chat with Libby Smith and Tiffany Tovey, two of the three hosts of the Radical Re(imagining) podcast, to chat about what felt radical about the 2021 AEA virtual conference.
We'd love to hear from you! What felt radical to you about the conference? Feel free to add your reflections on LinkedIn or Twitter. Be sure to use the #Eval21 hashtag, and tag us in your reflections!
Twitter accounts:
Music by Matt Ingelson, http://www.mattingelsonmusic.com/
