Force bots to use compatibility mode

[alexstandiford]

Related, #6

By forcing bots to use compatibility mode instead of the web app, we can ensure that all pages will be crawled by bots such as Google, ensuring that we do not lose any SEO benefits that comes with our site.

I envision that this would work off of an allow-list of user agents, and the list would be filter-able. This would not only allow us to set up a default list of agents, but it would also allow us to add more agents to the list should the site need that.

[alexstandiford]

Possible list, pulled from Webnots

• Googlebot
• Googlebot-News
• Googlebot-Video
• Googlebot-Image
• Mediapartners-Google
• Mediapartners-Google
• AdsBot-Google
• AdsBot-Google-Mobile-Apps
• Bingbot
• MSNBot
• MSNBot-Media
• AdIdxBot
• BingPreview
• Desktop
• Mobile
• Business Search (Advertisements)
• Baidu Union
• Baidu Favorites
• Image Search
• News Search
• Video Search
• YandexBot
• YandexMobileBot
• Yandex
• YandexDirect
• YandexDirectDyn
• YandexMedia
• YandexImages
• YaDirectFetcher
• YandexBlogs
• YandexNews
• YandexPagechecker
• YandexMetrika
• YandexCalendar
• YandexScreenshotBot
• YandexFavicons
• YandexWebmaster
• YandexImageResizer
• YandexSitelinks
• YandexAntivirus
• YandexVertis
• Slurp
• DuckDuckBot
• ia_archiver
• aolbuild
• teoma
• Ecosia

[alexstandiford]

Might be worth using a package, like: https://packagist.org/packages/jaybizzle/crawler-detect

[alexstandiford]

in #6, use_compatibility_mode was made to be extend-able. This makes it possible to add this as a decision list item. I'm inclined to make this a separate composer package that is used by the boilerplate by default. This would make it possible to remove this if someone wanted to-do that for some reason.