Overview of "don't allow search engines to index my user profile"
Search engines visit sites across the internet with automated programs called "spiders" which read the contents of web pages and store them in their index. By default, search engines will follow every link they find. However, most major search engines adhere to the "robots.txt" standard which allows content providers to specify which pages should not be put in the index. When you select the "don't allow search engines to index my user profile" preference, the saidit code will insert a special tag on your user page that tells compliant spiders not to index that page.
Caveats
Only spiders that comply with the voluntary robots.txt standard will honor this feature.
The flag to disable indexing will only be detected the next time the spider crawls your user page.
Most search engines re-index reddit relatively frequently, but it is not by a long shot an instant update.
Your comments and submissions will still be searchable from other parts of the site.
For example, if you submit a comment on a post in /r/AskReddit, search spiders may still find that comment and show it when someone searches for your username.
revision by — view source