Known Bots 6.0.3

This allows more accurate indication of the number of users vs robots online at any given time.

Requirements

This addon requires PHP 7.0.0 or higher and works on XenForo 2.x

Usage

When you look at Current Visitors, you'll see additional robots identified - also look at the "Robots" list on that page http://www.example.com/community/online/?type=robot





We also...

Click to expand...

ew false positives:

• cubot; j5
• baiduboxapp

new bots:

• 200pleasebot
• a8bot
• abilogicbot
• acoonbot
• adform robot
• arhpostbot
• atomseobot
• awariorendererbot
• badoobot
• bl.uk_ldfc_bot
• brobot
• charityengine bot
• charlotte
• cosmos
• coveobot
• crawlbot/1.0.0
• cxensebot
• facebot
• fandomopengraphbot
• freshpingbot
• fuelbot
• geedobot
• getlocalbot
• google-safety
• gpcsupbot
• grub-client
• gynxbot
• healrworld crawler...

Click to expand...

new bots discovered in June 2021

• btbot: BT Bot
• catchbot: Catchbot
• comodospider: Comodo SSL Spider
• deepnoc: deepnoc bot (network optimized crawling)
• dispenserbot: Dispenser Dab Solver Checklinks Bot
• epicbot...

Click to expand...

KnownBots v4 is a completely new build - bots are no longer hard coded, but updated via API calls and uses the XF code cache to store bot data

• raw bot data downloaded from API is stored in internal_data/knownbots.json
• new CLI tool for manually fetching bots from API (Cron task is also provided)
• new CLI tool for manually loading bots from knownbots.json
• new CLI tool for testing user agent matches

Click to expand...

This release includes additional sanity checks to prevent bad data returned from the API from breaking the forums.

If any of the data returned by the API is not in the exact format we expect, the entire download is discarded and no changes applied to the forum. An error message will be logged prompting further investigation.

After upgrading to 4.0.1, you should manually force a fetch of new API data by executing the following command from your forum root:

php cmd.php known-bots:fetch -f

Click to expand...

v5.0.0 is a major rewrite of the core functionality of this addon aimed at improving processing speed, bot detection sophistication and greatly enhancing our ability to identify new bots.

Note that the options have changed - so please check the options after upgrading. More information about each option is provided on the main addon page.

• major rewrite - no longer use "bot|spider|crawl" search strings and false-positive lists to identify possible bots, rely instead on search...

Click to expand...

• new CLI tool known-botsarse to parse web server log files and display detected bots
• new CLI tool known-bots:send to send newly detected user agents to the KnownBots API for analysis
• new CLI tool known-bots:check-token to validate that the API token successfully authenticates - and optionally have the system regenerate a new API token if it has expired
• knownbots@hampel.io email address is deprecated and will be removed soon - emails should no longer be sent...

Click to expand...

Minor bug fix - no need to update unless you are experiencing problems sending bot updates.

• when sending bots via email, include the bot list as an attachment rather than in the body
• new CLI tool to send bots via email directly, used for debugging bot sending issues

Click to expand...

This update contains an important bugfix to handle malformed UTF-8 in user agent strings.
This update will simply ignore any user agents with malformed UTF-8, avoiding errors when trying to send updates via the API. These user agents are invalid and so there is no point undertaking any further analysis - thus they are silently discarded.
This new version also contains an additional CLI tool for importing user agents from a text file, for testing purposes.

Click to expand...