If you want to exclude our crawler to not visit specific sections of your site, please add a separate paragraph to the robots.txt and specify the path you'd like to exclude:
Make sure that the robots.txt is in the correct location. It must be in the top directory, e.g. www.domain.com/robots.txt.
Placing the file in a subdirectory won't have any effect. Furthermore please note that the IP addresses used by the crawler change from time to time and that it may take up to a day for changes in robots.txt to propagate across all systems.