Robots.txt
Robots.txt tells well-behaved crawlers which paths they may crawl. It is useful for reducing crawler noise, but it is not access control and does not guarantee removal from search results.
Noindex
Noindex tells crawlers that can access the page not to include it in search results. It works best when the crawler is allowed to fetch the page and see the directive.
Common mistakes
Blocking a page in robots.txt and also adding noindex can backfire because the crawler may not see the noindex directive. Do not use robots.txt to hide private information.
FAQ
Should private pages rely on robots.txt?
No. Use authentication or proper access control for private content.
Should filtered search URLs be noindex?
Usually yes, unless the filtered page is a deliberate SEO landing page.