Robots.txt

[lpug]

Ive looked through the forum and could not see answer to my question, in the tools menu I configured my robots.txt file and when I select go it shows the prompt about it will rewrite file if it exists, however nothing happens i.e. there is no file generated. In each page under SEO i have also set my no follow links etc. Am I missing something obvoius?
The sitemap file did generate and ftp to my site.

Thanks in advance.

[Pablo]

Maybe this will be helpful?
https://www.wysiwygwebbuilder.com/robots_txt.html

[lpug]

Ok got it, its published when you publish a page or site. However I have individually setup which pages I want indexed and the robot file still shows allow all to be indexed. Even in my pages I have set them up in the SEO tab....is there anywhere else I should be looking at

User-agent: *

Allow: /

Allow: /images/

Sitemap: https://mysite.com/sitemap.xml

Maybe this will be helpful?
https://www.wysiwygwebbuilder.com/robots_txt.html

[Pablo]

If you need further assistance then please share a DEMO project
Also, describe exactly which settings you have used and what the expected result is.

See also:
https://www.wysiwygwebbuilder.com/forum ... 10&t=82134

[lpug]

Hi Pablo,

Ive created a new project and the Robot file is still not being generated as it should be. I have created a demp project with 4 pages, page 3 I have set as to not index or follow and yet the robot file is not correctly outputing.

https://1drv.ms/u/s!AkH9C6w0oL9tiw9Pzv6 ... L?e=Mn4bNH

[Pablo]

Everything seems to be correct.
What did you expect to happen?

Note that robots.txt only include folders rules
Rules for individual pages are set in the page itself via meta tags:

Code: Select all

<meta name="robots" content="noindex, nofollow">

[lpug]

The robots outputs this

User-agent: *

Allow: /

Allow: /images/

Shouldn't show that page3 is disallowed
Like this

User-agent: *

Allow: /

Disallow:/ page3

Allow: /images/

[Pablo]

Shouldn't show that page3 is disallowed
Like this

User-agent: *

Allow: /

Disallow:/ page3

Allow: /images/

No!
The robot rules for individual pages are set in the page itself via meta tags.
Otherwise the page may still be indexed if it was linked via another website.

[jerryco]

Otherwise the page may still be indexed if it was linked via another website.

I'm learning something here! Thanks Pablo!