The Test Community.Network

23rd April- Reliability, Validity, Fairness: with Mary Pitoniak

33 min · 21 de abr de 2026
portada del episodio 23rd April- Reliability, Validity, Fairness: with Mary Pitoniak

Descripción

In this week's episode, Tim Burnett chats with Mary Pitoniak, Assessment and Measurement Expert and President of Pitoniak Educational Measurement. They discuss: Why reliability, validity and fairness are the three pillars of good assessment — and how they connect, reinforce and constrain each other Whether personalised assessment powered by AI is really ready for prime time, especially for high-stakes tests like licensure and certification How AI is the "hot topic" at every conference right now, and why handing it to organisations without psychometric expertise is like "giving the keys to a Maserati to someone who barely knows how to drive" Mary brings nearly forty years of experience in assessment, including twenty-four years at ETS working on NAEP and auditing programmes to the ETS standards. She and her long-time collaborator Linda Cook were recently named the first women editors of Educational Measurement — now in its fifth edition — and won the National Council on Measurement in Education's Award for Exceptional Achievement in Educational Measurement. In a refreshingly plain-spoken conversation, Mary makes the technical accessible, explaining why standardisation has historically meant standardisation to the dominant culture, why the question isn't really "is the test valid?" but "are the interpretations of the test score for the proposed use appropriate?", and why the most exciting developments in AI and assessment still need to be anchored to the same enduring principles. Read the book: Educational Measurement (Fifth Edition) — Edited by Linda L. Cook and Mary J. Pitoniak (Oxford University Press, open access) https://testcommunity.network/content/educational-measurement-fifth-edition-edited-by-linda-l-cook-and-mary-j-pitoniak Connect with Mary Pitoniak on LinkedIn: https://www.linkedin.com/in/mary-pitoniak-3560a691/ Connect with Tim Burnett on LinkedIn: https://www.linkedin.com/in/tburnett/ If you would like to join Tim on a future episode or sponsor the Test Community Network then get in touch with Tim to schedule a call: https://calendly.com/educationtech/15min-teams-chat Events: https://www.testcommunity.network/upcoming-events Podcasts: https://www.testcommunity.network/podcasts Marketplace: https://www.testcommunity.network/marketplace Subscribe to the Test Community Network here: https://www.testcommunity.networkFull Transcript here: https://testcommunity.network/download/6NGB7H

Comentarios

0

Sé la primera persona en comentar

¡Regístrate ahora y forma parte de la comunidad de The Test Community.Network!

Prueba gratis

Empieza 7 días de prueba

$99 / mes después de la prueba. · Cancela cuando quieras.

  • Podcasts solo en Podimo
  • 20 horas de audiolibros al mes
  • Podcast gratuitos

Todos los episodios

88 episodios

episode 14th May - Global Assessment News artwork

14th May - Global Assessment News

Assessment & Testing Updates 1. Oxford seminar exploring how AI could transform examinations, personalised learning, authenticity, surveillance and inequality. https://testcommunity.network/a-new-examination-era-ai-and-the-transformation-of-educational-assessment [https://testcommunity.network/a-new-examination-era-ai-and-the-transformation-of-educational-assessment] 2. Surpass Conference 2026 heads to Manchester with sessions on AI, digital assessment, skills and the future of qualifications. https://testcommunity.network/surpass-conference-2026-speakers [https://testcommunity.network/surpass-conference-2026-speakers] 3. IACAT invites proposals for its 2026 conference in São Paulo, covering CAT, IRT, AI in psychometrics and modern assessment research. https://testcommunity.network/iacat-2026-call-for-proposals [https://testcommunity.network/iacat-2026-call-for-proposals] 4. Dr Luca Longo argues AI reveals deeper questions about what education and assessment are really measuring. https://testcommunity.network/e-atp-2026-conference-opening-keynote-cheating-or-revealing-what-ai-exposes-about-education [https://testcommunity.network/e-atp-2026-conference-opening-keynote-cheating-or-revealing-what-ai-exposes-about-education] 5. Integrity Advocate explains why human-reviewed ID verification creates stronger evidence than automated confidence scores alone. https://testcommunity.network/why-automated-identity-verification-isnt-enough-for-high-stakes-exams [https://testcommunity.network/why-automated-identity-verification-isnt-enough-for-high-stakes-exams] 6. A discussion on building academic integrity culture, Ireland’s education-first approach, AI, trust and transparency. https://testcommunity.network/the-opposite-of-cheating-the-opposite-of-cheating-podcast-season-2-episode-60-loretta-goff [https://testcommunity.network/the-opposite-of-cheating-the-opposite-of-cheating-podcast-season-2-episode-60-loretta-goff] 7. Charlie Burley shares why school wellbeing strategies fail and introduces the Six C’s framework for tackling teacher burnout. https://testcommunity.network/edufuturists-why-your-school-wellbeing-strategy-isnt-working-and-what-to-do-about-it-with-charlie-burley-343 [https://testcommunity.network/edufuturists-why-your-school-wellbeing-strategy-isnt-working-and-what-to-do-about-it-with-charlie-burley-343] 8. This episode examines why more students are enrolling in college but failing to complete their degrees. https://testcommunity.network/education-insight-what-20-years-of-data-reveals-about-college-success [https://testcommunity.network/education-insight-what-20-years-of-data-reveals-about-college-success] 9. UPESSC used AI-enabled surveillance and real-time monitoring for large-scale lecturer examinations in Uttar Pradesh. https://testcommunity.network/upessc-lecturer-exam-conducted-smoothly-under-ai-monitoring [https://testcommunity.network/upessc-lecturer-exam-conducted-smoothly-under-ai-monitoring] 10. Indian students and educators raise concerns about cognitive fatigue in the revised CUET UG 2026 exam format. https://testcommunity.network/students-worried-over-no-break-3-hour-cuet-ug-exam-format [https://testcommunity.network/students-worried-over-no-break-3-hour-cuet-ug-exam-format] 11. A look at how classroom.cloud supports classroom management, safeguarding, BYOD and AI-driven risk analysis in schools. https://testcommunity.network/the-edtech-podcast-325-unlocking-the-power-of-classroom-cloud-the-ultimate-edtech-solution-for-schools [https://testcommunity.network/the-edtech-podcast-325-unlocking-the-power-of-classroom-cloud-the-ultimate-edtech-solution-for-schools] 12. Dr Ghulam Nabi Shakir argues that AI-assisted assessment could improve fairness, transparency and efficiency in Pakistan’s exams. https://testcommunity.network/from-e-assessment-to-ai-assisted-assessment [https://testcommunity.network/from-e-assessment-to-ai-assisted-assessment] 13. Türkiye moves vocational qualification exams to digital e-exam centres, supporting national education and workforce transformation. https://testcommunity.network/vocational-qualification-exams-shift-to-e-exam-system-in-trkiye [https://testcommunity.network/vocational-qualification-exams-shift-to-e-exam-system-in-trkiye] 14. Two people were arrested in Kerala for alleged high-tech cheating using Bluetooth devices during a PSC exam. https://testcommunity.network/psc-exam-cheating-two-arrested-in-thiruvananthapuram-for-high-tech-malpractice [https://testcommunity.network/psc-exam-cheating-two-arrested-in-thiruvananthapuram-for-high-tech-malpractice] 15. Tim Burnett speaks with Lee Fawcett about cheap hardware, AI apps, online proctoring vulnerabilities and red-teaming assessments. https://testcommunity.network/test-community-network-7th-may-how-a-7-webcam-a-vibe-coded-ai-app-could-be-a-test-security-problem-with-guest-lee-fawcett [https://testcommunity.network/test-community-network-7th-may-how-a-7-webcam-a-vibe-coded-ai-app-could-be-a-test-security-problem-with-guest-lee-fawcett] 16.

13 de may de 20269 min
episode 7th May: How a £7 Webcam, + a Vibe-Coded AI App, Could Be a Test Security Problem with guest Lee Fawcett artwork

7th May: How a £7 Webcam, + a Vibe-Coded AI App, Could Be a Test Security Problem with guest Lee Fawcett

In this week's episode, Tim Burnett chats with Lee Fawcett, a counter-fraud and investigation specialist supporting the UK qualifications and assessment industry. They discuss: - How a £7.36 USB webcam and a vibe-coded cheating app scored 8 out of 10 on a GCSE algebra test — with almost no technical expertise required - Why online proctoring platforms still can't detect the full range of concealed hardware and AI-assisted cheating methods available today - The case for "red teaming" your assessments — thinking like a criminal rather than an exam owner to uncover vulnerabilities before bad actors do Tim starts by demonstrating a live proof-of-concept: a tiny USB camera taped to a keyboard, feeding exam questions to an AI app built with Lovable, which reads the screen every ten seconds and whispers the answers back through audio. Lee shares his reaction and his experience investigating cheats and criminals who exploit test security vulnerabilities. The pair discuss the limits of remote proctoring, the commercial cheating industry, and why organisations need to start by assessing the real-world value of what their qualification represents — not just the security of the delivery method. They also flag the Conference on Test Security coming to Cambridge in November as a must-attend for the community. Connect with Lee Fawcett on LinkedIn: https://www.linkedin.com/in/leefawcettuk/ Connect with Tim Burnett on LinkedIn: https://www.linkedin.com/in/tburnett/ If you would like to join Tim on a future episode or sponsor the Test Community Network then get in touch with Tim to schedule a call: https://calendly.com/educationtech/15min-teams-chat Events: https://www.testcommunity.network/upcoming-events Podcasts: https://www.testcommunity.network/podcasts Marketplace: https://www.testcommunity.network/marketplace Subscribe to the Test Community Network here: https://www.testcommunity.network

6 de may de 202622 min
episode 30th April: ReadSpeaker: Rethinking Accessibility in Assessment with Hugh McNeela artwork

30th April: ReadSpeaker: Rethinking Accessibility in Assessment with Hugh McNeela

In this week's episode, Tim Burnett chats with Hugh McNeela, Sales Account Manager for Corporate Learning (UK & Ireland) at ReadSpeaker. They discuss: - Why text-to-speech in assessments isn't just an accessibility tool, and how a Universal Design for Learning (UDL) approach benefits all candidates, including the 70% of people with neurodiversity challenges who never disclose their condition to employers - How ReadSpeaker integrates directly into assessment platforms, from LMS environments like Moodle and Canvas through to high-stakes exam delivery, without compromising the security or integrity of assessments - The real-world impact of TTS on learning outcomes, including how Penn Foster improved their 30-day course completion rates by 54% simply by adding text-to-speech to their content Hugh and Tim explore what happens when accessibility is built into the design of an assessment from the start, rather than bolted on as a reasonable adjustment. From reducing the logistical burden on teachers during exam days, to eliminating the stigma that stops learners from asking for help, ReadSpeaker's approach treats voice support as a core part of fair assessment — not an optional extra. The conversation also touches on why deterministic TTS rendering matters in high-stakes exams, and how ReadSpeaker handles secure environments, including compatibility with safe exam browsers. Connect with Hugh McNeela on LinkedIn: https://www.linkedin.com/in/hughmcneela/ Learn more about ReadSpeaker: https://www.readspeaker.com/?utmsource=testcommunitynetwork Connect with Tim Burnett on LinkedIn: https://www.linkedin.com/in/tburnett/ If you would like to join Tim on a future episode or sponsor the Test Community Network then get in touch with Tim to schedule a call: https://calendly.com/educationtech/15min-teams-chat Events: https://www.testcommunity.network/upcoming-events Podcasts: https://www.testcommunity.network/podcasts Marketplace: https://www.testcommunity.network/marketplace Subscribe to the Test Community Network here: https://www.testcommunity.network

30 de abr de 202624 min
episode 23rd April- Reliability, Validity, Fairness: with Mary Pitoniak artwork

23rd April- Reliability, Validity, Fairness: with Mary Pitoniak

In this week's episode, Tim Burnett chats with Mary Pitoniak, Assessment and Measurement Expert and President of Pitoniak Educational Measurement. They discuss: Why reliability, validity and fairness are the three pillars of good assessment — and how they connect, reinforce and constrain each other Whether personalised assessment powered by AI is really ready for prime time, especially for high-stakes tests like licensure and certification How AI is the "hot topic" at every conference right now, and why handing it to organisations without psychometric expertise is like "giving the keys to a Maserati to someone who barely knows how to drive" Mary brings nearly forty years of experience in assessment, including twenty-four years at ETS working on NAEP and auditing programmes to the ETS standards. She and her long-time collaborator Linda Cook were recently named the first women editors of Educational Measurement — now in its fifth edition — and won the National Council on Measurement in Education's Award for Exceptional Achievement in Educational Measurement. In a refreshingly plain-spoken conversation, Mary makes the technical accessible, explaining why standardisation has historically meant standardisation to the dominant culture, why the question isn't really "is the test valid?" but "are the interpretations of the test score for the proposed use appropriate?", and why the most exciting developments in AI and assessment still need to be anchored to the same enduring principles. Read the book: Educational Measurement (Fifth Edition) — Edited by Linda L. Cook and Mary J. Pitoniak (Oxford University Press, open access) https://testcommunity.network/content/educational-measurement-fifth-edition-edited-by-linda-l-cook-and-mary-j-pitoniak Connect with Mary Pitoniak on LinkedIn: https://www.linkedin.com/in/mary-pitoniak-3560a691/ Connect with Tim Burnett on LinkedIn: https://www.linkedin.com/in/tburnett/ If you would like to join Tim on a future episode or sponsor the Test Community Network then get in touch with Tim to schedule a call: https://calendly.com/educationtech/15min-teams-chat Events: https://www.testcommunity.network/upcoming-events Podcasts: https://www.testcommunity.network/podcasts Marketplace: https://www.testcommunity.network/marketplace Subscribe to the Test Community Network here: https://www.testcommunity.networkFull Transcript here: https://testcommunity.network/download/6NGB7H

21 de abr de 202633 min
episode Webinar: AI Superpowers - Vibe Coding, Cowork & AI Skills artwork

Webinar: AI Superpowers - Vibe Coding, Cowork & AI Skills

You don't need to be a developer to build software anymore — and this session proves it.In this hands-on webinar, Tim Burnett explores three game-changing AI developments that are reshaping how professionals work and build:🛠️ Vibe Coding — Learn how anyone can create functional apps and tools just by describing what they want in plain English. No programming experience required. We cover the real potential, the limitations, and why this trend is heading straight for the assessment industry.🤖 Claude Cowork — Anthropic's desktop agent that works alongside you to manage files, automate tasks, and handle complex workflows — all without writing a single line of code. Like having an AI colleague sitting right next to you.📋 AI Skills & Agent Workflows — Discover how reusable AI instruction files are turning assistants into specialist agents, and why this is a breakthrough for organisations looking to build scalable, repeatable AI workflows across their teams.Plus, Tim shares his regular Beyond Assessment AI Update — the latest AI tools, trends, and developments that matter most to the assessment community.Whether you're an assessment professional curious about AI's next leap, or someone who's always wanted to build tools but lacked the technical skills — this one's for you.🔔 Subscribe for more from the AI Superpowers series by the Test Community Network — the global platform for assessment professionals.

16 de abr de 202645 min