Data Discovery: Dos and Don’ts

The Keys to Repeatable and Scalable Analysis

Have you ever put together some analysis only to have your stakeholders say, “This is great, can I get this every month?”  Or have you ever thought, “this report takes too much time to build every week, I wish there was a way to shorten the amount of time I spend putting this report together?”  If you have, this episode is for you.  Tune in to hear some tips that will help you create repeatable analysis. Timeline: 01:41 - The Goal of Analysis 04:54 - Minimizing Technical Debt 06:46 - Eliminate Manual Touches 09:28 - Consider How You Use Filters 11:08 - Combine Data Sources First 11:48 - Set Up a Framework or Template 13:02 - Use Consistent Naming Conventions 14:35 - Document Your Process 15:40 - The Last Record The Last Record: 1. The goal of any analysis is to provide actionable insights.  Be sure to keep this in mind as you are building any reports 2. Strive to minimize the technical debt and technical interest.  That’s the time needed to re-do portions of any analysis in order to do it properly, or the time needed to refresh the data on a regular basis. * Avoid manual touches - this could be data pulls, or even hand keying in information into a spreadsheet. * Put filters up front, and remove as much data as possible in the first filter, and each subsequent filter should remove less and less data. * Set up a framework so you only have to update the raw data, and not rebuild a summary or data visualization * Watch out for your naming conventions * Be consistent with how you name your fields throughout your workflow 3. Document your steps and process.  Your future self will thank you for saving pain and heartache trying to figure out why you created the process the way you did.

Data Discovery: Dos and Don’ts

In this episode I’ll discuss some keys to remember when examining a dataset for the first time. Timeline: 02:25 - Do #1 - Look at your data carefully 05:32 - Don’t #1 - Don’t try to bite off too much 07:32 - Do #2 - Look for patterns and trends 10:15 - Do #3 - Consider the data source 13:13 - Do #4 - Identify blind spots in your data 15:22 - Don’t #2 - Don’t include incomplete or missing data Survey of Data Workers: https://community.useready.com/whitepapers/idc-infobrief-state-of-data-science-and-analytics/?auto-trigger [https://community.useready.com/whitepapers/idc-infobrief-state-of-data-science-and-analytics/?auto-trigger] The Last Record: 1. Look at the data carefully * Do you have all the fields that you will need? * Is it a number field?  Is it text? 2. Look for the patterns and trends 3. Consider the source * Are there controls in your data? * Are you pulling from an outside source that may not always be available or updated? 4. Identify the blind spots in your data * Think about the questions you can answer using this data. * Are there any limitations?  If there are, you may need to supplement with additional data.

The Progression of Analytics

In this episode you’ll learn about the 4 main types of analytics and when to use each method in your own analysis. Previous Episode: https://datarockspodcast.buzzsprout.com/1276397/5866354-having-fun-with-data-sabermetrics-style [https://datarockspodcast.buzzsprout.com/1276397/5866354-having-fun-with-data-sabermetrics-style] Timeline: 01:26 - Guideposts in Analytics 03:21 - Descriptive Analytics 05:26 - Diagnostic Analytics 06:55 - Predictive Analytics 10:04 - Prescriptive Analytics 11:05 - A Deeper Dive 17:56 - Which Method Should You Use? 19:22 - The Last Record The Last Record: 1. The most important thing you can do as an analyst is help tell the story. 2. The progression of analytics contains four main methods.  Descriptive (what happened); Diagnostic (why did it happen); Predictive (what will happen); and Prescriptive (what actions can I take based on what will happen).  With each step in the progression, the analysis becomes more difficult, but the payoff is with greater value. 3. We talked about an example in transportation where I walked you thru some of the questions and types of analysis you may encounter in each of the four methods. 4. In order to know what method to use, you have to know what data you have available and what you are capable of providing.  It’s important to listen to your stakeholders and understand the problem that you’re trying to solve, so you can provide the best solution possible.

Having Fun With Data: Sabermetrics Style

Description: In this episode, learn one way that you can have fun with data.  We’ll discuss Sabermetrics, and how using data can enhance your enjoyment of the game of baseball. Timeline: 00:55 - Welcome to the Show 03:05 - Why I Love Baseball 04:36 - A Few Baseball Rules 07:58 - A Pivotal Moment in Baseball 12:00 - Basic Statistics 15:34 - More Contributions from Bill James 17:05 - Lasting Effects on the Game 22:40 - The Last Record Useful Links: Retrosheet.org [https://www.retrosheet.org/] Baseball-Reference.com [https://www.baseball-reference.com/] Fangraphs WAR [https://www.fangraphs.com/warleaders.aspx?season=2020&team=all] Baseball-Reference WAR [https://www.baseball-reference.com/leaders/WAR_career.shtml] The Last Record: 1. Data can be fun, and there are so many questions that you can answer using data.  In baseball that means questions like… * Do better hitters get fewer good pitches to hit? * Or what impact does the composition of the ball have on a hitter’s performance? 2. This leads to the 2nd key takeaway that data can enhance your understanding of something that you enjoy, even if it is only a hobby.  And sometimes, if you enjoy the data that much, it can even lead you to a career. 3. Data often has gaps.  Advanced statistics were created to try isolating the performance of the player being analyzed.  In your own analysis, it is important to be cognizant of the gaps that may be present.

How to Approach Problems Like an Analyst

In this episode you’ll learn 5 tips to help you solve problems more efficiently and with greater ease. Timeline: 00:54 - Intro 02:08 - Today’s Interesting Data Fact 03:06 - Ask Great Questions 07:58 - Embrace a “Could-Do” Mindset 11:04 - Work Backwards 14:54 - Break the Obstacle Down 18:06 - Take a Break 21:53 - The Last Record Today’s Interesting Data Fact: https://www.swnsdigital.com/2017/12/kids-ask-a-staggering-73-questions-every-day-half-of-which-mums-and-dads-struggle-to-answer/ [https://www.swnsdigital.com/2017/12/kids-ask-a-staggering-73-questions-every-day-half-of-which-mums-and-dads-struggle-to-answer/] The Advantage: https://www.amazon.com/Advantage-Organizational-Health-Everything-Business/dp/1491510803 [https://www.amazon.com/Advantage-Organizational-Health-Everything-Business/dp/1491510803] The Last Record: 1. Ask great questions - this helps you get to the root of the problem, and define what you’re solving. * Examples of great questions: * Can you describe the situation in more detail and why is this happening? * When and how often does this problem occur? * What is the impact of solving this problem? * What additional context should I be aware of? 2. Embrace “could do” mindset.  Don’t just stop at the obvious solution, or settle for the way it’s always been done, especially if there is a better way.  The key to coming up with the best solution is to consider all the available options and implement the one that is going to give you the greatest impact. 3. Work backward.  Starting at the end can help make the problem solving process easier.  Don’t be afraid to steal from your previous work or emulate others who have solved a similar problem.  One of the best ways to gain knowledge for future problems is to pick apart others’ solutions. 4. Break the problem down into more manageable pieces.  Don’t try to map out the entire solution.  Instead, focus on what is most important right now, and work towards getting quick wins to build momentum. 5. Don’t rush the incubation period.  Go for a walk, get a snack, or hydrate.  Keep your mind from thinking of biological needs, and free yourself up to focus on the problem.  Ensure that you have some quiet time to think and come up with the best solution possible.