How to Optimize for Crawling

Google Updates & Search Ranking Systems

Sources Cited: * Google Search gets Status Dashboard to provide updates during outages [https://9to5google.com/2022/12/14/google-search-status-dashboard/] * Google December 2022 Link Spam Update [https://www.searchenginejournal.com/google-december-2022-link-spam-update/474233/] * Google Launches Visual Guide To Search Elements [https://www.searchenginejournal.com/google-launches-visual-guide-to-search-elements/474082/] * Lily ray on google updates [https://twitter.com/lilyraynyc/status/1602692742605316096?s=12&t=D4U_ud9vQuRa0r1v60e9wA] * A guide to Google Search ranking systems [https://developers.google.com/search/docs/appearance/ranking-systems-guide]

Think Tank Ranking Preview

In a short episode, Cord and Einar preview the new and improved Tallest Tree Think Tank Ranking. The new rankings will be automated, updated monthly, and incorporate more data than ever. This includes: * Budget data from ProPublica [https://projects.propublica.org/nonprofits?gclid=Cj0KCQiA1sucBhDgARIsAFoytUuErOQAfUYrEyM2N84IvbzUyZHBaC1GmbRtZYGaUQ2T2sIUFqf4urUaAk5sEALw_wcB] * Traffic, traffic value, domain rating, and backlink data from Ahrefs [https://ahrefs.com/site-explorer] * Pages indexed  from Serp API [https://serpapi.com/] * Web software from Wappalyzer [https://www.wappalyzer.com/] We're now able to pull historical traffic data from Ahrefs, which will give us insights into the last 36 months of performance for each of the 300+ groups in our study. We thank Ahrefs, who reviewed and approved our use of this data, for their support of this project. We're also working with a new statistician, assistant professor of marketing at Clemson University, Lura Forcum. We're grateful for her insights on the data. A few trends popped up in data review: * About 12% of groups are still dealing with www vs non-www canonical issues * Many sites have pop-ups that likely violate Google's Page Experience guidelines * We found "spikey" traffic patterns for many groups, especially state-based think tanks, which may be explained by the location from which the traffic data was collected * We saw a few sites that relaunched w/o redirects in place, which resulted in their referring domains crashing and large losses in traffic * Many groups were affected by recent Google updates * Many groups saw growth in referring domains w/o growth in traffic

How to Optimize for Crawling

Cord & Einar discuss how to optimize your website for web crawlers. How do things like robots.txt, site speed, HTTP codes, URLs, links, redirects, XML sitemaps, PDFs vs web pages, robots meta tags, and canonical URLs all affect web crawlers? How can you optimize these aspects of your site to make sure all the content you want robots to discover can be crawled and indexed? Sources Cited: * Google Robots Testing Tool [https://www.google.com/webmasters/tools/robots-testing-tool] * Merkle robots.txt Validator [https://technicalseo.com/tools/robots-txt/] * PageSpeed Insights [https://pagespeed.web.dev/] * Lighthouse [https://developer.chrome.com/docs/lighthouse/overview/] * Pingdom Speed Test [https://tools.pingdom.com/] * Robots Meta Tags [https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag]

All About Links: JavaScript, HTTP/HTTPS, Link Rot, Ranking, and Link Networks

Cord & Einar discuss all things linking. Can JavaScript links pass PageRank? How to HTTP to HTTPS redirects work? Are old links irrelevant? Will links always be a ranking factor? What's a link graph? Sources Cited: * Search Engine Roundtable: Google Updates Help Documentation To Say Using JavaScript For Links Can Be Fine [https://www.seroundtable.com/google-javascript-for-links-can-be-crawlable-too-34385.html] * Google Search Central: Make your links crawlable [https://developers.google.com/search/docs/crawling-indexing/links-crawlable] * Search Engine Roundtable: Google Can Pass PageRank From HTTP To HTTPS URLs With Redirects [https://www.seroundtable.com/google-pagerank-http-to-https-redirects-34380.html] * Search Engine Roundtable: Google: Some Old Links May Be Irrelevant (AKA Link Rot) [https://www.seroundtable.com/google-old-links-irrelevant-34301.html] * Search Engine Roundtable: Google: Links Will Be Less Important As A Ranking Factor In The Future [https://www.seroundtable.com/google-links-less-important-ranking-34357.html?utm_source=%23SEOFOMO&utm_campaign=130887eadd-seofomo-newsletter-by-aleyda-nov-6&utm_medium=email&utm_term=0_408af963b1-130887eadd-1403632647] * Kevin Indig: https://twitter.com/kevin_indig/status/1590115045236965376 [https://twitter.com/kevin_indig/status/1590115045236965376] * SEMRush: Get a Bird’s Eye View of Any Website’s Link Network [https://www.semrush.com/blog/backlink-network-graph/] * Search Engine Roundtable: Google Search Console API Delay To Be Fixed In Days [https://www.seroundtable.com/google-search-console-api-delay-34388.html] * Search Engine Journal: Is Bounce Rate A Google Ranking Factor? [https://www.searchenginejournal.com/ranking-factors/bounce-rate-ranking-factor/#close]

How we do SEO audits

In this episode Cord & Einar discuss the ever-evolving Tallest Tree approach to SEO audits. Here's the outline Cord & Einar used on the show: TOOLS * Sitebulb: run both an HTML-only and JavaScript crawl * Ahrefs Crawler * Ahrefs Site Explorer * Ahrefs Chrome extension * Google Search Console * Chrome Inspector w/Lighthouse * Google’s Robots Tester [https://www.google.com/webmasters/tools/robots-testing-tool?utm_source=support.google.com/webmasters/&utm_medium=referral&utm_campaign=%206062598] * Inspect URL in GSC * Schema Validator * Merkle’s https://technicalseo.com/tools/ [https://technicalseo.com/tools/] * The Wayback Machine FIRST LOOK * Explore the site by hand, view source on whatever the half dozen most-used templates seem to be * Turn off JavaScript using Chrome extension and review the site again * Explore the backend and get some counts on posts, categories, authors, etc. * Look at plugins/modules * Review Sitebult/Ahrefs crawl results * Review Google Search Console The goal of the first look is form a basic hypothesis about how technical SEO and site structure can be improved. Take screenshots and start making slides right away. CHECKLIST * Review 192 items in 35 categories REPORTING * How a search engine works * Crawling * Indexing * Ranking * E-A-T * Measurement * Rank Tracking * GTM & GA4 for Traffic  & Conversions * Link Growth * Google News