The Test Community.Network

23rd April- Reliability, Validity, Fairness: with Mary Pitoniak

33 min · 21. apr. 2026
episode 23rd April- Reliability, Validity, Fairness: with Mary Pitoniak cover

Beskrivelse

In this week's episode, Tim Burnett chats with Mary Pitoniak, Assessment and Measurement Expert and President of Pitoniak Educational Measurement. They discuss: Why reliability, validity and fairness are the three pillars of good assessment — and how they connect, reinforce and constrain each other Whether personalised assessment powered by AI is really ready for prime time, especially for high-stakes tests like licensure and certification How AI is the "hot topic" at every conference right now, and why handing it to organisations without psychometric expertise is like "giving the keys to a Maserati to someone who barely knows how to drive" Mary brings nearly forty years of experience in assessment, including twenty-four years at ETS working on NAEP and auditing programmes to the ETS standards. She and her long-time collaborator Linda Cook were recently named the first women editors of Educational Measurement — now in its fifth edition — and won the National Council on Measurement in Education's Award for Exceptional Achievement in Educational Measurement. In a refreshingly plain-spoken conversation, Mary makes the technical accessible, explaining why standardisation has historically meant standardisation to the dominant culture, why the question isn't really "is the test valid?" but "are the interpretations of the test score for the proposed use appropriate?", and why the most exciting developments in AI and assessment still need to be anchored to the same enduring principles. Read the book: Educational Measurement (Fifth Edition) — Edited by Linda L. Cook and Mary J. Pitoniak (Oxford University Press, open access) https://testcommunity.network/content/educational-measurement-fifth-edition-edited-by-linda-l-cook-and-mary-j-pitoniak Connect with Mary Pitoniak on LinkedIn: https://www.linkedin.com/in/mary-pitoniak-3560a691/ Connect with Tim Burnett on LinkedIn: https://www.linkedin.com/in/tburnett/ If you would like to join Tim on a future episode or sponsor the Test Community Network then get in touch with Tim to schedule a call: https://calendly.com/educationtech/15min-teams-chat Events: https://www.testcommunity.network/upcoming-events Podcasts: https://www.testcommunity.network/podcasts Marketplace: https://www.testcommunity.network/marketplace Subscribe to the Test Community Network here: https://www.testcommunity.networkFull Transcript here: https://testcommunity.network/download/6NGB7H

Kommentarer

0

Vær den første til at kommentere

Tilmeld dig nu og bliv en del af The Test Community.Network-fællesskabet!

Kom i gang

2 måneder kun 19 kr.

Derefter 99 kr. / måned · Opsig når som helst.

  • Podcasts kun på Podimo
  • 20 lydbogstimer pr. måned
  • Gratis podcasts

Alle episoder

89 episoder

episode 28th May - Global Assessment News cover

28th May - Global Assessment News

This week's collection brings together 18 items from across the assessment world. AI continues to dominate the conversation, with research warning that generative AI overuse may deskill graduates, a striking statistic showing student AI use in exams has risen to 92%, and several voices calling for a fundamental rethink of assessment design in response. Elsewhere, a teenage researcher's disclosure of critical vulnerabilities in India's CBSE marking portal raises urgent questions about exam system security, Pearson Edexcel has apologised for errors in a GCSE Computer Science paper, and there are calls for proposals from conferences in Kuala Lumpur and Sydney. On the professional side, the FAB Awards are open for entries, the e-Assessment Association is running its 2026 Online Proctoring Survey, and RCS England is recruiting a Director of Education. Podcasts this week cover serious games in credentialing, human-centred academic integrity, and whether AI literacy is substance or marketing. Whether you work in higher education, vocational assessment, professional certification or K-12, there is something here for you. The collection in full: https://testcommunity.network/collections/28th-may-2026 [https://testcommunity.network/collections/28th-may-2026]

27. maj 20267 min
episode 14th May - Global Assessment News cover

14th May - Global Assessment News

Assessment & Testing Updates 1. Oxford seminar exploring how AI could transform examinations, personalised learning, authenticity, surveillance and inequality. https://testcommunity.network/a-new-examination-era-ai-and-the-transformation-of-educational-assessment [https://testcommunity.network/a-new-examination-era-ai-and-the-transformation-of-educational-assessment] 2. Surpass Conference 2026 heads to Manchester with sessions on AI, digital assessment, skills and the future of qualifications. https://testcommunity.network/surpass-conference-2026-speakers [https://testcommunity.network/surpass-conference-2026-speakers] 3. IACAT invites proposals for its 2026 conference in São Paulo, covering CAT, IRT, AI in psychometrics and modern assessment research. https://testcommunity.network/iacat-2026-call-for-proposals [https://testcommunity.network/iacat-2026-call-for-proposals] 4. Dr Luca Longo argues AI reveals deeper questions about what education and assessment are really measuring. https://testcommunity.network/e-atp-2026-conference-opening-keynote-cheating-or-revealing-what-ai-exposes-about-education [https://testcommunity.network/e-atp-2026-conference-opening-keynote-cheating-or-revealing-what-ai-exposes-about-education] 5. Integrity Advocate explains why human-reviewed ID verification creates stronger evidence than automated confidence scores alone. https://testcommunity.network/why-automated-identity-verification-isnt-enough-for-high-stakes-exams [https://testcommunity.network/why-automated-identity-verification-isnt-enough-for-high-stakes-exams] 6. A discussion on building academic integrity culture, Ireland’s education-first approach, AI, trust and transparency. https://testcommunity.network/the-opposite-of-cheating-the-opposite-of-cheating-podcast-season-2-episode-60-loretta-goff [https://testcommunity.network/the-opposite-of-cheating-the-opposite-of-cheating-podcast-season-2-episode-60-loretta-goff] 7. Charlie Burley shares why school wellbeing strategies fail and introduces the Six C’s framework for tackling teacher burnout. https://testcommunity.network/edufuturists-why-your-school-wellbeing-strategy-isnt-working-and-what-to-do-about-it-with-charlie-burley-343 [https://testcommunity.network/edufuturists-why-your-school-wellbeing-strategy-isnt-working-and-what-to-do-about-it-with-charlie-burley-343] 8. This episode examines why more students are enrolling in college but failing to complete their degrees. https://testcommunity.network/education-insight-what-20-years-of-data-reveals-about-college-success [https://testcommunity.network/education-insight-what-20-years-of-data-reveals-about-college-success] 9. UPESSC used AI-enabled surveillance and real-time monitoring for large-scale lecturer examinations in Uttar Pradesh. https://testcommunity.network/upessc-lecturer-exam-conducted-smoothly-under-ai-monitoring [https://testcommunity.network/upessc-lecturer-exam-conducted-smoothly-under-ai-monitoring] 10. Indian students and educators raise concerns about cognitive fatigue in the revised CUET UG 2026 exam format. https://testcommunity.network/students-worried-over-no-break-3-hour-cuet-ug-exam-format [https://testcommunity.network/students-worried-over-no-break-3-hour-cuet-ug-exam-format] 11. A look at how classroom.cloud supports classroom management, safeguarding, BYOD and AI-driven risk analysis in schools. https://testcommunity.network/the-edtech-podcast-325-unlocking-the-power-of-classroom-cloud-the-ultimate-edtech-solution-for-schools [https://testcommunity.network/the-edtech-podcast-325-unlocking-the-power-of-classroom-cloud-the-ultimate-edtech-solution-for-schools] 12. Dr Ghulam Nabi Shakir argues that AI-assisted assessment could improve fairness, transparency and efficiency in Pakistan’s exams. https://testcommunity.network/from-e-assessment-to-ai-assisted-assessment [https://testcommunity.network/from-e-assessment-to-ai-assisted-assessment] 13. Türkiye moves vocational qualification exams to digital e-exam centres, supporting national education and workforce transformation. https://testcommunity.network/vocational-qualification-exams-shift-to-e-exam-system-in-trkiye [https://testcommunity.network/vocational-qualification-exams-shift-to-e-exam-system-in-trkiye] 14. Two people were arrested in Kerala for alleged high-tech cheating using Bluetooth devices during a PSC exam. https://testcommunity.network/psc-exam-cheating-two-arrested-in-thiruvananthapuram-for-high-tech-malpractice [https://testcommunity.network/psc-exam-cheating-two-arrested-in-thiruvananthapuram-for-high-tech-malpractice] 15. Tim Burnett speaks with Lee Fawcett about cheap hardware, AI apps, online proctoring vulnerabilities and red-teaming assessments. https://testcommunity.network/test-community-network-7th-may-how-a-7-webcam-a-vibe-coded-ai-app-could-be-a-test-security-problem-with-guest-lee-fawcett [https://testcommunity.network/test-community-network-7th-may-how-a-7-webcam-a-vibe-coded-ai-app-could-be-a-test-security-problem-with-guest-lee-fawcett] 16.

13. maj 20269 min
episode 7th May: How a £7 Webcam, + a Vibe-Coded AI App, Could Be a Test Security Problem with guest Lee Fawcett cover

7th May: How a £7 Webcam, + a Vibe-Coded AI App, Could Be a Test Security Problem with guest Lee Fawcett

In this week's episode, Tim Burnett chats with Lee Fawcett, a counter-fraud and investigation specialist supporting the UK qualifications and assessment industry. They discuss: - How a £7.36 USB webcam and a vibe-coded cheating app scored 8 out of 10 on a GCSE algebra test — with almost no technical expertise required - Why online proctoring platforms still can't detect the full range of concealed hardware and AI-assisted cheating methods available today - The case for "red teaming" your assessments — thinking like a criminal rather than an exam owner to uncover vulnerabilities before bad actors do Tim starts by demonstrating a live proof-of-concept: a tiny USB camera taped to a keyboard, feeding exam questions to an AI app built with Lovable, which reads the screen every ten seconds and whispers the answers back through audio. Lee shares his reaction and his experience investigating cheats and criminals who exploit test security vulnerabilities. The pair discuss the limits of remote proctoring, the commercial cheating industry, and why organisations need to start by assessing the real-world value of what their qualification represents — not just the security of the delivery method. They also flag the Conference on Test Security coming to Cambridge in November as a must-attend for the community. Connect with Lee Fawcett on LinkedIn: https://www.linkedin.com/in/leefawcettuk/ Connect with Tim Burnett on LinkedIn: https://www.linkedin.com/in/tburnett/ If you would like to join Tim on a future episode or sponsor the Test Community Network then get in touch with Tim to schedule a call: https://calendly.com/educationtech/15min-teams-chat Events: https://www.testcommunity.network/upcoming-events Podcasts: https://www.testcommunity.network/podcasts Marketplace: https://www.testcommunity.network/marketplace Subscribe to the Test Community Network here: https://www.testcommunity.network

6. maj 202622 min
episode 30th April: ReadSpeaker: Rethinking Accessibility in Assessment with Hugh McNeela cover

30th April: ReadSpeaker: Rethinking Accessibility in Assessment with Hugh McNeela

In this week's episode, Tim Burnett chats with Hugh McNeela, Sales Account Manager for Corporate Learning (UK & Ireland) at ReadSpeaker. They discuss: - Why text-to-speech in assessments isn't just an accessibility tool, and how a Universal Design for Learning (UDL) approach benefits all candidates, including the 70% of people with neurodiversity challenges who never disclose their condition to employers - How ReadSpeaker integrates directly into assessment platforms, from LMS environments like Moodle and Canvas through to high-stakes exam delivery, without compromising the security or integrity of assessments - The real-world impact of TTS on learning outcomes, including how Penn Foster improved their 30-day course completion rates by 54% simply by adding text-to-speech to their content Hugh and Tim explore what happens when accessibility is built into the design of an assessment from the start, rather than bolted on as a reasonable adjustment. From reducing the logistical burden on teachers during exam days, to eliminating the stigma that stops learners from asking for help, ReadSpeaker's approach treats voice support as a core part of fair assessment — not an optional extra. The conversation also touches on why deterministic TTS rendering matters in high-stakes exams, and how ReadSpeaker handles secure environments, including compatibility with safe exam browsers. Connect with Hugh McNeela on LinkedIn: https://www.linkedin.com/in/hughmcneela/ Learn more about ReadSpeaker: https://www.readspeaker.com/?utmsource=testcommunitynetwork Connect with Tim Burnett on LinkedIn: https://www.linkedin.com/in/tburnett/ If you would like to join Tim on a future episode or sponsor the Test Community Network then get in touch with Tim to schedule a call: https://calendly.com/educationtech/15min-teams-chat Events: https://www.testcommunity.network/upcoming-events Podcasts: https://www.testcommunity.network/podcasts Marketplace: https://www.testcommunity.network/marketplace Subscribe to the Test Community Network here: https://www.testcommunity.network

30. apr. 202624 min
episode 23rd April- Reliability, Validity, Fairness: with Mary Pitoniak cover

23rd April- Reliability, Validity, Fairness: with Mary Pitoniak

In this week's episode, Tim Burnett chats with Mary Pitoniak, Assessment and Measurement Expert and President of Pitoniak Educational Measurement. They discuss: Why reliability, validity and fairness are the three pillars of good assessment — and how they connect, reinforce and constrain each other Whether personalised assessment powered by AI is really ready for prime time, especially for high-stakes tests like licensure and certification How AI is the "hot topic" at every conference right now, and why handing it to organisations without psychometric expertise is like "giving the keys to a Maserati to someone who barely knows how to drive" Mary brings nearly forty years of experience in assessment, including twenty-four years at ETS working on NAEP and auditing programmes to the ETS standards. She and her long-time collaborator Linda Cook were recently named the first women editors of Educational Measurement — now in its fifth edition — and won the National Council on Measurement in Education's Award for Exceptional Achievement in Educational Measurement. In a refreshingly plain-spoken conversation, Mary makes the technical accessible, explaining why standardisation has historically meant standardisation to the dominant culture, why the question isn't really "is the test valid?" but "are the interpretations of the test score for the proposed use appropriate?", and why the most exciting developments in AI and assessment still need to be anchored to the same enduring principles. Read the book: Educational Measurement (Fifth Edition) — Edited by Linda L. Cook and Mary J. Pitoniak (Oxford University Press, open access) https://testcommunity.network/content/educational-measurement-fifth-edition-edited-by-linda-l-cook-and-mary-j-pitoniak Connect with Mary Pitoniak on LinkedIn: https://www.linkedin.com/in/mary-pitoniak-3560a691/ Connect with Tim Burnett on LinkedIn: https://www.linkedin.com/in/tburnett/ If you would like to join Tim on a future episode or sponsor the Test Community Network then get in touch with Tim to schedule a call: https://calendly.com/educationtech/15min-teams-chat Events: https://www.testcommunity.network/upcoming-events Podcasts: https://www.testcommunity.network/podcasts Marketplace: https://www.testcommunity.network/marketplace Subscribe to the Test Community Network here: https://www.testcommunity.networkFull Transcript here: https://testcommunity.network/download/6NGB7H

21. apr. 202633 min