The five steps below help you optimize the way your website is crawled and indexed to make your website much easier to find on the web. 1. Basics The file is a simple text file that gives the Google bot specific instructions on how the website should be crawled. For example, by excluding certain directories. These are often data-sensitive areas, such as login and customer accounts, that should not be indexed. When creating the file, it is important to ensure that the bot has access to all the resources needed to display your website correctly. For example, you should avoid blocking CSS or JavaScript files in the file. If you want to exclude a specific directory from scanning, use the following code in the file: Advertising Continue reading below User Agent: Ban: directory
The star is a placeholder (called a generic) and represents all other content associated with that directory. After creating the image masking service file, you need to save it to the root directory of the Point Use Google Search Console to test your file. Please note that this requires that you have registered the website in Search Console. tester 1.2 The XML sitemap Besides the file, there is another file that plays a key role for indexing: the XML sitemap . This is a machine-readable file listing all of your website URLs. This structured data is created as text and saved in XML format. This file also allows you to transmit other information in addition to the URLs, such as the date of the last update of the various URLs. Advertising Continue reading below After creating the XML file, add it to Google Search Console to notify Google of existing URLs. However, the XML sitemap only recommends URLs to Google and does not give the bot any instructions like in the file.
Google will therefore ignore the contents of the file when indexing the website. The XML sitemap is often mismanaged despite being very handy for indexing new and large websites as it informs Google of all existing subpages. For example, if you have new content on a web page that is not very well interlinked, use the sitemap to notify Google of this content. The structure of a simple XML sitemap without additional attributes looks like this: code1 There are different ways to create a sitemap. Some CMSs even come with the appropriate tools for automatic sitemap creation. You can also use one of the free programs available online. Once the sitemap is ready, save it to the root directory of your website: the sitemap or save it dynamically to save space on the server. Google recommends splitting the sitemap if you have more than 50,000 URLs. In this case, you need to use an index and create a sitemap sitemap.