robots.txt: manage crawling and protect private pages properly

1. Decide who should be able to view each page

When launching your Austrian business website, you may have a public services page alongside a collection of draft quotations. Write down two separate decisions: which material may anyone read, and which material should only agreed people be able to open? A search crawler is just one possible type of visitor. Check the draft in an ordinary browser while signed out too. If confidential documents are visible, the website needs an access-protection review. Adding an instruction to a public file will not fix that exposure. Before uploading anything sensitive, agree who will configure the private area and how you will verify that access works as intended.

2. Find the file at the correct web address

RFC 9309 places robots.txt at the top-level path of the relevant web service, for example https://www.example.org/robots.txt. It is not a DNS record. Record the full address shown in your browser before investigating. If you also have a preview address or another subdomain, ask the website service which file it serves there. Do not rely on an administration screenshot from an older project. Request the current file and note which published website it belongs to. This avoids changing settings for an unrelated test environment. You should be able to examine the returned text; a nicely designed error page does not establish what the file contains.

3. Review instructions against actual pages

The file can identify a crawler group with User-agent and describe matching paths with Allow or Disallow. Have each proposed instruction explained using a specific page from your own website. For example, should the addressed crawler still be able to fetch your public services page? Which instruction concerns the preview? Avoid importing an entire file from an unrelated shop. Before approving a change, save the previous contents and agree on a short list of addresses to check afterwards. Appearance in search results is a separate question: a single crawling instruction guarantees neither inclusion nor removal. Ask your website team to explain the intended outcome without promising a ranking.

4. Protect drafts at the website service

Private material needs effective access control at the website or file service. RFC 9110 describes HTTP authentication; ask the responsible operator which supported protection suits your project. Start by identifying who needs access, then use the service’s intended controls. Do not share permanent credentials through public pages. Check a document’s direct address as well as the page linking to it. Hiding a menu item is insufficient if the underlying file remains openly available. The robots.txt file is public itself, so it should not be treated as an inventory of secret folders or a place to store passwords. Use invented sample documents when agreeing how the protection will be tested.

5. Send each task to the right administrator

When an agency proposes a change, ask for the precise task and affected service. A robots.txt file normally belongs to website configuration. Changing nameservers does not automatically move that file or create a login system. Use the overview below to assign work; it describes separate tasks rather than granting permission to alter existing settings. Agree who will perform each action and who will check the result. If you manage the website yourself, record where the relevant setting lives and retain the last working version.

6. Use a short launch acceptance checklist

Before publishing, work through your own acceptance list, ideally with a second person. Use only your website and the test access you have agreed. A public page should work while signed out; a private area should require the access you specified. Also open a selected file using its direct address. Then compare the current robots.txt with the approved version. Record the date, full URLs and observed results. If something is wrong, have the responsible website service correct that specific issue first. Buying the domain again or making speculative DNS changes is not a useful first response to a page-access problem. Keep the completed checklist with the project handover.

• Open the public page while signed out.
• Check the private area and direct file link separately.
• Compare approved crawler instructions with the current file.
• Record the result and responsible contact.

7. Report a discrepancy with reproducible evidence

A report such as “the robot cannot get through” is too vague for a targeted investigation. Give your website team the affected URL, time and time zone, and exact diagnostic text. If available, include the HTTP status and returned content. RFC 9110 describes HTTP responses, but a status number alone does not replace checking your particular request. State whether you were signed in and what you expected to happen. After a correction, repeat the same agreed functional test first instead of waiting for a promised ranking. Keep confidential documents and credentials out of public issue reports. A small reproducible example is easier to investigate than a folder of unrelated screenshots.

8. Budget for the domain and website management separately

A standard new .at domain at atdomain.at currently costs €12.99 total for the first year. Renewal currently costs €15.99 total per additional year. We support DNS management with our default nameservers and custom nameservers; we do not sell hosting or email mailboxes. Maintaining robots.txt and protecting website content therefore belong in the plan you make with your website service. If you need help with the domain or DNS, we can assist on request within the controls we actually have. For a new project, first search for a suitable available name, then confirm the website features and access arrangements you need before ordering additional services.