Episode 28: Strategies to Tame Flaky Tests

Episode 28: Strategies to Tame Flaky Tests

In this episode of TMI, Chip and Tess dive deep into the frustrating world of flaky tests. These are the tests that pass sometimes and fail at other times, even when the code hasn't changed. We discuss reference research from Trunk, who analyzed over 20 million CI jobs and found that flaky tests are a widespread problem, even among large tech companies like Meta, Google, and Uber. We examine strategies for managing flaky tests, such as quarantining them. We also cover how to prevent flaky tests by ensuring isolation, using dynamic waits instead of hard-coded timeouts, and utilizing deterministic inputs. Finally, we discuss the importance of the testing pyramid, controlled test environments, contract testing, robust error handling, retry mechanisms, and clear communication. References: * https://www.testrail.com/blog/flaky-tests/ * https://testsigma.com/guides/flaky-test/ * https://trunk.io/blog/what-we-learned-from-analyzing-20-2-million-ci-jobs-in-trunk-flaky-tests-part-1 * https://trunk.io/blog/what-we-learned-from-analyzing-20-2-million-ci-jobs-in-trunk-flaky-tests-part-2

Episode 27: Mastering Agile Estimation and Planning

Explore the three pillars of Agile estimating and planning: embracing change, prioritizing value, and effective communication—principles that resonate beyond software development. * Learn why multitasking can hinder progress despite feeling productive. * Understand the pitfalls of setting initial estimates in stone. * Learn why providing estimation dates and team velocity as ranges -- rather than a definitive number -- helps manage expectations. * Discover how properly prioritizing features can deliver 80% of benefits with 20% of the effort. * We discuss using Story Points [https://www.geeksforgeeks.org/story-points-and-velocity-in-scum/?ref=header_outind] to estimate complexity, Ideal Days [https://daily.dev/blog/story-points-vs-ideal-days-which-agile-estimation-technique] for estimating the time it will take to completer a task, and tips for splitting large user stories. * Delve into planning poker [https://www.geeksforgeeks.org/wideband-delphi-and-planning-poker/?ref=header_outind] for estimation and strategies for backlog prioritization. Finally, we consider a sprint structure that includes dedicated time for technical debt. More Resources: * Agile Estimating and Planning [https://www.amazon.com/Agile-Estimating-Planning-Mike-Cohn/dp/0131479415/ref=sr_1_1?crid=2K45059MZFTKM&dib=eyJ2IjoiMSJ9.UDajWA4koqDEN7ag_i333T0xSc-CaEeCPc6irFcqVDp-oy2zkNtYYnv3Jw-sAnK4_uZkU5aNCxFbqf1uwXpmrj4DPS8_hPSoVV_sz2Chzd_3Po1fvG-HFIAk8-B8xZyyicMidq4gF16Drbk2lA6oGlTEmFBFKwQ6nT5GumaatE8MI3RDRdJM3QNApwGlgAfcWXAA77w5qta02thKcZ0ZuAkSHNcJ5uhFlrkINebothQ.OIRs2Xml4_um0ip2ClcNPQImD1M8IvOsgpQ0bsOC8Ao&dib_tag=se&keywords=Agile+Estimating+and+Planning&qid=1727971011&s=audible&sprefix=agile+estimating+and+planning%2Caudible%2C99&sr=1-1-catcorr]

Episode 25: Effective Agile Practices for Team Success

In this episode, explore the Agile philosophy in software development. Learn about using story points and “Ideal Days” [https://daily.dev/blog/story-points-vs-ideal-days-which-agile-estimation-technique] for estimation, and the importance of communication and trust in Agile teams. Discover how to choose optimal sprint durations, understand velocity, and use burn-down charts [https://www.geeksforgeeks.org/overview-of-burndown-chart-in-agile/?ref=header_outind] to spot risks. Dive into Scrum as a framework, including the role of the Scrum Master, [https://www.geeksforgeeks.org/understanding-scrum-master-and-its-roles/?ref=header_outind] sprint planning, daily meetings, and retrospectives. Understand continuous integration’s impact on quality and the significance of visualization with diagrams. Learn how Google fosters a testing culture and why automation doesn't replace human tester's, but rather enhances their productivity. F inally, get tips for distributed teams and effective backlog prioritization by the product owner. More Resources: * More Agile Testing: Learning Journeys for the Whole Team [https://www.amazon.com/s?k=More+Agile+Testing%3A+Learning+Journeys+for+the+Whole+Team&crid=1D9O0U0NLC53Y&sprefix=more+agile+testing+learning+journeys+for+the+whole+team%2Caps%2C275&ref=nb_sb_noss] * Agile Estimating and Planning [https://www.amazon.com/Agile-Estimating-Planning-Mike-Cohn/dp/0131479415/ref=sr_1_1?crid=2K45059MZFTKM&dib=eyJ2IjoiMSJ9.UDajWA4koqDEN7ag_i333T0xSc-CaEeCPc6irFcqVDp-oy2zkNtYYnv3Jw-sAnK4_uZkU5aNCxFbqf1uwXpmrj4DPS8_hPSoVV_sz2Chzd_3Po1fvG-HFIAk8-B8xZyyicMidq4gF16Drbk2lA6oGlTEmFBFKwQ6nT5GumaatE8MI3RDRdJM3QNApwGlgAfcWXAA77w5qta02thKcZ0ZuAkSHNcJ5uhFlrkINebothQ.OIRs2Xml4_um0ip2ClcNPQImD1M8IvOsgpQ0bsOC8Ao&dib_tag=se&keywords=Agile+Estimating+and+Planning&qid=1727971011&s=audible&sprefix=agile+estimating+and+planning%2Caudible%2C99&sr=1-1-catcorr] * Succeeding with Agile: Software Development Using Scrum [https://www.amazon.com/Succeeding-Agile-Software-Development-Using/dp/0321579364/ref=sr_1_1?crid=22HTU1JIAVQW9&dib=eyJ2IjoiMSJ9.v_HMHUSZIbp-gxHkhHLODGKyhBYiL3MuJBs7PlWESw1VXfuy05HDo3EMM8NItYNg3SEa5IjnYufYBUkNoduzD5HlQNBy4b7dgESNt_rowk5ssX7IyHjfSjhez8jPtHuw5yxeRuJpkQNJhismxYZbpGNTKgbjcZHrdjK36Xz11WjnRu444smJPsupeAKB3Nqg0Cz_jHkfrElIX8zTr8KQLonwqgJ2i64Z2JIjXRAjvdw.FPEY_5rnfM9RbUKE_3pNifw3nEG3cMvCOi-I8zgDNig&dib_tag=se&keywords=Succeeding+with+Agile&qid=1727971151&s=audible&sprefix=succeeding+with+agile%2Caudible%2C93&sr=1-1-catcorr] * How Google Tests Software [https://www.amazon.com/Google-Tests-Software-James-Whittaker/dp/0321803027/ref=sr_1_1?crid=1D0LRA0G4ZKSG&dib=eyJ2IjoiMSJ9.S7T6Yi-hNpgJN6urQiqeMAWbDbrbYNFSCUf86JbF6Z9FTnWlRWeVL_YtltMgqJ1c5PFOfsoItFBjI9s8dG5VphY2slA1rt8VxJesprsEBK3ehuKW8umS9wGZT1vXfJrHF0J3dQaiFZjGYrwsmev0gxDrhJb9imU6f5G8Siu1wazkBoYiyyi8HaKtdqUKTzjmvZ_vvnPKerM1IlzWdJsIR2F1yBXHdQUiHSakKe9LU4s.41hvQC1RVNePZI33AeAERl3qaHPQj9fM0lEemd8e8DY&dib_tag=se&keywords=how+google+tests+software&qid=1727727867&sprefix=how+google+tests+%2Caps%2C113&sr=8-1] * The Feiner Points of Leadership: The 50 Basic Laws that Will Make People Want to Perform Better for You [https://www.amazon.com/Feiner-Points-Leadership-People-Perform/dp/B0886FBBRP/ref=sr_1_1?crid=3J5EBEM7URDW2&dib=eyJ2IjoiMSJ9.7tbxXXnec8Nc1YHAXPktSfDLdnGJhgZL0iI48YZ68ojGjHj071QN20LucGBJIEps.Xs5HrbqeK5GciAYLpXk2-U5mhbr5v8PzfUPrf2kj3kY&dib_tag=se&keywords=The+Feiner+Points+of+Leadership&qid=1727896663&sprefix=the+feiner+points+of+leadership%2Caps%2C157&sr=8-1] * The A Word [https://angryweasel.com/TheAWord.pdf] * Exploratory Software Testing [https://www.amazon.com/Exploratory-Software-Testing-Tricks-Techniques-ebook/dp/B002RVDV3S/ref=sr_1_1?crid=1U4S0GDKJG3JD&dib=eyJ2IjoiMSJ9.aNcb37qLEz1YZ2CxT_J6Pbxiv17J5WFUht91Ui2F1eas2X23I3_9rysbcrCEZrtwXxP_ZCR8-D9kyp2UYIvU3tgiig5iD3khF-OyLIPyhV4xZHbKC4HJg0wZxKCKC_4VAuboAgrpXyL-o1RRqeTFlg4ctku9gHkPlgrjy5I7wYvkz70ExoY_9SglX8qwINA4NmslsnCf3l-NOvDOxgObF1OCw9IbKkBB6EjSD7mnFOU.4hVphVGh5Xgm1AiHc83Ujiwp3Q6k_TuGBJ5ThriOsug&dib_tag=se&keywords=exploratory+software+testing&qid=1727727736&sprefix=exploratory+softwa%2Caps%2C104&sr=8-1] * The One Thing: The Surprisingly Simple Truth Behind Extraordinary Result [https://www.amazon.com/Surprisingly-Extraordinary-Results-bestselling-Paperback/dp/B08L589JK3/ref=tmm_pap_swatch_0?_encoding=UTF8&dib_tag=se&dib=eyJ2IjoiMSJ9.SBAWI1oH8P9QAR8qxBL9jxrheDE4FT0KOGxrtsvIuQJ3KquYn5_dXIcG1Zu7fV4J8MnqHDfpXUFJULKV3PpGnJIlNGkdPvJ3jU-V7SWY9NefgJxBladVUCx8TMVm6A6rQg0QnJ4bFAhR-7FtvwNqCqz20Gs1oYse5FxxhACFe48WMblQAd-EocF4E3z_t-KZKbXB12O_Pc9-LknrKESbaUvdASBLGkbav31pcfPM2mA.11CTcwu87nxZvPsOadGnvlfk8xASIgqAe26DgXqawBo&qid=1727970271&sr=8-1]

Episode 24: Master Exploratory Testing Techniques

In this episode, we dive into exploratory testing, a dynamic technique where tests are designed and executed simultaneously, using insights from previous experiments to guide future ones. We explore various exploratory testing tours [https://test-able.blogspot.com/2013/11/web-testing-exploratory-tours.html], including: * The Guide Book Tour: Following the user manual step by step. * The Money Tour: Prioritizing features most essential to users. * The Saboteur Tour: Intentionally introducing problems like network errors and corrupted files. * The Anti-Social Tour: Doing the opposite of expected user actions. We also discuss testing in the small versus testing in the large, scenario testing [https://en.wikipedia.org/wiki/Scenario_testing#:~:text=Scenario%20testing%20is%20a%20software,which%20is%20easy%20to%20evaluate.], and designing tests using states and events. Finally, we share thoughts on the future of testing. Other Resources: * Exploratory Software Testing [https://www.amazon.com/Exploratory-Software-Testing-Tricks-Techniques-ebook/dp/B002RVDV3S/ref=sr_1_1?crid=1U4S0GDKJG3JD&dib=eyJ2IjoiMSJ9.aNcb37qLEz1YZ2CxT_J6Pbxiv17J5WFUht91Ui2F1eas2X23I3_9rysbcrCEZrtwXxP_ZCR8-D9kyp2UYIvU3tgiig5iD3khF-OyLIPyhV4xZHbKC4HJg0wZxKCKC_4VAuboAgrpXyL-o1RRqeTFlg4ctku9gHkPlgrjy5I7wYvkz70ExoY_9SglX8qwINA4NmslsnCf3l-NOvDOxgObF1OCw9IbKkBB6EjSD7mnFOU.4hVphVGh5Xgm1AiHc83Ujiwp3Q6k_TuGBJ5ThriOsug&dib_tag=se&keywords=exploratory+software+testing&qid=1727727736&sprefix=exploratory+softwa%2Caps%2C104&sr=8-1] * Explore it? Explore it! (YouTube Video) [https://www.youtube.com/watch?v=9FKY1Is0lgs]

Episode 23: Mastering Testing in Production

In this episode of the TMI Podcast, we dive into the intriguing world of Testing in Production and how it allows you to observe your software in real-world conditions. Join us as we discuss key techniques to minimize risks while testing in production: * Feature Flags: Deploy code to production servers without immediate exposure to users. Feature flags allow you to unveil new features only when you’re confident they will function seamlessly. * Shadowing: Safely test new code by sending a copy of live traffic without impacting the customer experience. * Staged Rollouts and Canarying: Gradually introduce new code to a small percentage of users, increasing exposure as confidence grows. * A/B Testing: Compare different versions of features to determine the most effective approach. We also discuss the innovative practice of chaos engineering, where intentionally causing disruptions builds system resilience and confidence. Discover why a corporate culture that embraces risk is essential for successful Testing in Production. Tune in for insights on how to leverage these strategies to enhance your software’s performance and reliability in the real world. More Resources: https://www.youtube.com/watch?v=n2iE0wUt3bc https://medium.com/life-at-apollo-division/testing-in-production-339dc58dd8cd https://copyconstruct.medium.com/testing-in-production-the-safe-way-18ca102d0ef1 https://copyconstruct.medium.com/testing-in-production-the-hard-parts-3f06cefaf592