If you want to improve SEO and give those crawler robots directives on how to search your website, you can set it up through Google Search Console.
In Google Search Console, you can claim your website as a property, create / test / add robots.txt to it. You can also link your Search Console with Google Analytics.
In simplified terms, robots.txt is a text file that webmasters create to instruct search engine robots how to crawl pages on their websites. To learn more about Google Search Console and robots.txt, please click on links in references below.