AI Visibility Audit: Can LLMs Crawl Your Website? James Dooley Interviews Stephen Burns

James Dooley: AI visibility audit. Today I'm joined with Stephen Burns from Common Crawl and we're going to talk about how certain websites is actually blocking different large language models. So Stephen, first and foremost, why is it that you created an AI visibility audit yourself?

Stephen Burns: Well, two months ago, I wrote a, a PDF, a guide for AI visibility. I've been speaking on the SEO circuit about this and people were asking for my presentation. So, you know, presentations don't tell you anything because they're all just pictures. So, I wrote a guide on it and I, we published it on the Common Crawl Foundation's blog and it's hosted there. You can download this 18-page guide on how to, uh, make your site visible to AI. And then, so this tool that I built, uh, I, you know, it struck me. I go, well, why don't I build a tool as a follow-up so that you actually can do a check on your site and help these people, uh, help them check it to see if their site's visible to CCBot and other AI crawlers and if you're in, in the crawl. And I'll actually give you your... It gives your harmonic centrality score too. So if I share my screen, so I'll share the screen here now and I'll go onto the tool, which is here. So it's StephenBurns.ai. It's a subdomain, audit.stephenburns.ai. And what I've done is I've loaded in one of my websites, FatRank.com, and it's checking to see, is your site visible to AI? I've loaded it. And here it says verdict open. So, I'm presuming that means it's fully open and everything's working fine for the different LLMs. Is that correct?

James Dooley: Yes. You're doing a very good job, James. Cheers, Stephen. I, I'm glad you've told me that because I didn't want it to have a big red cross and a big block. Um, but let's, let's move down here then. Let's start having a look at this because here it says gate number one, permission. So, with regards to the AI visibility audit, what does gate number one, permission, actually mean?

Stephen Burns: Yeah, there's three gates. So, the first gate is permissions. Your robots.txt file, the directive that tells the crawlers if they can crawl your site or not. So, your site, uh, FatRank.com, uh, you have your, your robots set up correctly and says, "Yeah, please crawl me, CCBot." And then, and then gate number two says edge. So, what edge means? So yeah, your site is hosted and it may, it has a CDN and an edge that has filters built into it. And your... If those filters are set up to block bots, which, you know, 35% of the internet is blocked by a lot of these, these filters that, by these major CDN companies, uh, it will not get the... The crawler can't get past that. You see, an SEO or a brand marketer may think, "Oh, yeah, we're open to crawlers with the permission level at robots. We, come on in." But they're not actually checking their edge, which is even more important and hidden. So, you got to talk to your, you know, your admin who manages that to say, "Hey, are we open to these crawlers?" And this will tell you if you're open to those crawlers. And then gate number three says rendering. So, what does that mean in the AI audit? So CCBot gets through both gate, the first two gates, and then it, uh, loads your page. And if your page is fully rendered in JavaScript, uh, CCBot and the other AI crawlers don't render JavaScript. They read HTML and text. So yeah, if you're... In this case you're serving, uh, words that it can read and it likes that. If it was... This would be red if it was all JavaScript. You'd be basically... If you load all JavaScript on your homepage or your sites, you're invisible to AI. It's not going to get any information off your page.

James Dooley: And then with regards to the AI visibility audit that you've created now, it shows not only CCBot, which is Common Crawl bot, it's also showing GPTBot, ChatGPT User, ClaudeBot, Anthropic. So all these we're checking against all these to make certain that it's available and visible for AI. Is that correct?

Stephen Burns: Yeah, I added a little bonus section here to see if your, uh, site is viewable to the other crawlers. But there's a caveat here, right? You... The other crawl... A lot of these CDN settings, you, uh, may have, they may check the IP of the crawler. So every crawler has an IP address. This tool, we can't... You can't mimic an IP address. Uh, so this tool is showing the... The CCBot one's accurate because it's pulling it from the, the database that where it shows that it got a 200 right through. The other ones, uh, sometimes they'll let it in, uh, the user agent mimicking the, the crawler, it will let it in. That's good. But sometimes it says, "No, you're not the real one. You're not coming from that IP address." So it may not show that accuracy there. But it's good, a good check-up tool to have anyways.

James Dooley: Yeah, for sure. So anyone, anyone who's watching this and you're looking for greater AI visibility, obviously step number one is like technical SEO. It's getting the foundations being set up. This is, this is crucially important. You've got to be making certain that ChatGPT and Anthropic and Gemini and CCBot can all actually crawl and render your actual site. It's really important. So make sure you head over to StephenBurns.ai and go onto the audit section and go and check to see whether your actual website is available for more AI visibility. Stephen Burns, it's been an absolute pleasure. Thank you very much.

Stephen Burns: Thanks for having me.