A sitemap and robots.txt, two files that help Google
There are two files on your site that no visitor will ever see, yet which genuinely affect how it does in Google: the sitemap and robots.txt. They sound technical, but their role is simple to understand. Let us explain what they are and why it is worth taking care of them.
How Google gets to know your site
For your site to appear in the search results, Google first has to visit and read it, it does this using an automated program called a robot (or "bot"). The robot wanders around the site, clicks links and gathers information about the pages. The sitemap and robots.txt are two signposts that make its work easier, and the better the robot understands your site, the better it can show it to people.
The sitemap: a table of contents for Google
A sitemap is a file (usually sitemap.xml) containing a list of all the important pages of your site. It is a bit like a table of contents in a book, telling Google: "here are all the pages I have, take a look at them".
Why bother, if the robot clicks links anyway? Because a sitemap:
- helps Google not miss anything: especially on larger sites or when a page is poorly linked,
- speeds up discovery of new content: you added a post or a page, and the sitemap points to it right away,
- hints at what is important and when it was updated.
For a small business site it may not be a revolution, but it is a simple way to make sure Google sees everything it should.
robots.txt: instructions on where the robot may look
robots.txt is a file that tells robots which parts of the site they should (or should not) visit. It is like "no entry" and "this way please" signs for the automated programs. It is used to:
- not waste the robot's attention on irrelevant parts of the site (e.g. internal files, technical panels),
- point out where the sitemap is (robots.txt usually contains a link to it),
- keep order in what Google browses.
Important: robots.txt is a request "do not look here", not a security measure. It is not for hiding sensitive data, real protections (passwords, permissions) are for that. It is a tool for organising, not for protecting.
The most common, dangerous mistake
There is one costly mistake associated with robots.txt worth knowing about: accidentally blocking the whole site. It happens that during site development a block for robots is set (so the unfinished site does not reach Google) and forgotten after publication. The result: the finished site does not appear in the results, because it told itself "Google, do not enter". It is one of the first things worth checking if a site "will not" get indexed.
What to do about it
For most WordPress or Joomla sites the sitemap is generated automatically (often by an SEO plugin), and robots.txt just needs to be set once, correctly. It is only worth:
- making sure the sitemap exists and is submitted in Google Search Console,
- checking that robots.txt does not accidentally block the site from Google,
- keeping both files up to date after major changes.
Not sure whether Google sees your site correctly, or whether something is accidentally blocking it? Get in touch, we will check the sitemap, robots.txt and indexing.