Merge pull request #28 from bclindner/feature/rewrite

Re-write

.gitignore

@@ -1,5 +1,6 @@
 # sensitive data
 config.yml
+config.json
 cookies.pickle
 # venv
 bin/
@@ -10,5 +11,6 @@ lib64/
 pyvenv.cfg
 # log files
 geckodriver.log
+ivory.log
 # pycache
 *__pycache__*

Dockerfile

@@ -0,0 +1,11 @@
+FROM python:3.7.4
+
+WORKDIR /app
+
+COPY . /app
+
+RUN pip install -r requirements.txt
+
+RUN python -m pytest -s tests/
+
+CMD ["python", "."]

README.md

@@ -1,118 +1,196 @@
-# Ivory, a crappy Mastodon automoderator
+# Ivory
 
-Ivory is an automoderation system for Mastodon which logs in as a normal user to
-the Mastodon frontend, scans the reports page, and automatically deals with
-common moderation tasks, particularly spam. It does this by reading over each
-report it finds and running a number of configuration-declared *rules* which are
-checked on each report.
+Ivory is an automoderator and anti-spam tool for Mastodon admins. It automates
+handling certain trivial reports and new user requests using *rules* -
+configurable tests that check reports and pending users for bad usernames,
+malicious links, and more.
 
-Currently, Ivory is intended to function as a stopgap measure to curb spam while
-we await the actual moderation API, though I have intentionally designed things
-in a way that will allow me to convert it to use said API when (or if) it
-releases.
+## Installation Guide
 
-## Installation and Usage
+This installation guide assumes you know your way around a Linux terminal,
+have Python and Git installed, and maybe a little bit about common tech like
+Python and JSON formatting.
 
-First, install Geckodriver and make sure it's in your terminal's PATH. If you're
-running Linux, you may have a version in your distro's package manager.
+### Installing
 
-After doing that, snippet should work. You'll want Python 3 (preferably 3.7 or
-above) for this:
+
+In a Linux terminal, the following commands will clone and install Ivory to
+whichever folder you're in. Make sure you have Git and Python installed:
 
 ```bash
 git clone https://github.com/bclindner/ivory
 cd ivory
 python -m venv .
 source bin/activate
-pip install -r requirements.txt
+python -m pip install -r requirements.txt
 ```
 
-After that, create a config.yml in the project root as shown in the Configuring
-section below and then run:
+This repo also comes with a Dockerfile, so if you want to deploy with that, that
+works too:
 
 ```bash
-python .
+git clone https://github.com/bclindner/ivory
+cd ivory
+docker build -t ivory .
+docker run -v /srv/ivory/ivory_config.json:/app/config.json ivory
 ```
 
-You will be asked for a username and password, and optionally an OTP if your
-account is set up for that. This should only happen once; after that, cookies
-are stored in the project root as `cookies.pickle` and the app will log in with
-those. (If at some point Ivory stops signing in correctly, delete this file and
-try manually logging in again.)
-
-## Configuring
-
-Ivory is configured using a YAML file. An example configuration is below:
-```yaml
-# Time to wait in between checks (in seconds)
-wait_time: 600 # 10min; lower numbers shouldn't stress your servers out
-driver:
-  type: browser # browser is the only supported driver type at present
-  # Instance URL
-  instance_url: https://mastodon.technology
-# Array of rules for Ivory to judge with
-rules:
-  # This name is what Ivory mentions in the moderation notes when finishing a
-  # report.
-- name: "No womenarestupid.site spam links"
-  # This rule parses over links in every post attached to a report.
-  # Also supports text phrases in reported posts with the 'content' type.
-  type: link_content
-  blocked:
-  # This list supports regexes!
-  - womenarestupid.site
-  - dontmarry.com
-  punishment:
-    # The highest severity punishment in a single judgement is the one used when
-    # punishing the user.
-    severity: 1000
-    # Currently only suspend is supported.
-    type: suspend
-    # Not implemented, but the following are for local users.
-    delete_account_data: yes
-    local_suspend_message: "Your account has been suspended for spamming."
-- name: "No womenarestupid.site shorturls"
-  # This rule type resolves shorturls!
-  type: link_resolver
-  blocked:
-  - dontmarry.com
-  - womenarestupid.site
-  punishment:
-    severity: 1000
-    type: suspend
-    delete_account_data: yes
-    local_suspend_message: "Your account has been suspended for spamming."
-- name: "No inflammatory usernames"
-  type: username_content
-  blocked:
-  # You can do case insensitive searches using regex, too!
-  - (?i)heck
-  punishment:
-    severity: 1000
-    type: suspend
-    delete_account_data: yes
-    local_suspend_message: "Your account has been suspended for having an inflammatory username."
+### Configuration
+
+Before starting Ivory, you need to create a new application in the Preferences
+menu. Don't worry about setting redirect URIs or anything that isn't required -
+just make sure you enable all of the `admin:` scopes. Once you've created the
+application, you'll want to grab its access token to place in the configuration
+file (example below).
+
+*Be* ***EXTREMELY*** *careful with handling the access token this generates -
+this key has a lot of power and in the wrong hands, this could mean someone
+completely obliterating your instance.*
+
+Once you've done that, it's time to set up your config file. Configuring Ivory
+is done with JSON; a sample is below:
+
+```json
+{
+  "token": "<YOUR_ACCESS_TOKEN_HERE>",
+  "instanceURL": "<YOUR_INSTANCE_URL_HERE>",
+  "waitTime": 300,
+  "reports": {
+    "rules": [
+      {
+        "name": "No spam links",
+        "type": "link_resolver",
+        "blocked": ["evilspam\\.website", "dontmarry\\.com"],
+        "severity": 2,
+        "punishment": {
+          "type": "suspend",
+          "message": "Your account has been suspended for spamming."
+        }
+      },
+      {
+        "name": "No link-in-bio spammers",
+        "type": "bio_content",
+        "blocked": ["sexie.ru"],
+        "severity": 1,
+        "punishment": {
+          "type": "disable",
+          "message": "Your account has been disabled for spamming."
+        }
+      }
+    ]
+  },
+  "pendingAccounts": {
+    "rules": [
+      {
+        "name": "No <a> tags",
+        "_comment": "Because honestly, you're definitely a bot if you're putting <a> tags into the field",
+        "type": "message_content",
+        "blocked": ["<a href=\".*\">.*</a>"],
+        "severity": 1,
+        "punishment": {
+          "type": "reject"
+        }
+      },
+      {
+        "name": "StopForumSpam test",
+        "type": "stopforumspam",
+        "threshold": 95,
+        "severity": 1,
+        "punishment": {
+          "type": "reject"
+        }
+      }
+    ]
+  }
+}
 ```
 
-## Caveats
+A more [in-depth guide to Ivory configuration](https://github.com/bclindner/ivory/wiki/Getting-Started)
+and the list of [rules](https://github.com/bclindner/ivory/wiki/Rules) and
+[punishments](https://github.com/bclindner/ivory/wiki/Punishments)
+can be found on the wiki.
+
+Ideally you only have to change this once in a blue moon, but if you do, you can
+use the `"dryRun": true` option to prevent Ivory from taking action, so you can
+test some rules on recent live moderation queues.
 
-This code is using Selenium to drive its report handling system. Selenium can be
-finicky. Stuff can break.
+### Running
 
-Take care when writing your rules. Ivory doesn't care if you get them wrong, and
-Ivory will absolutely ban users with impunity if you do. Test them if you can.
-Support for dry runs will come available when I get to it.
+After you've set up a config file, run the following in a Linux terminal:
 
-## Maintainers
+```
+# if you're running in the same terminal session you installed from, you can
+# skip this next line:
+source bin/activate
+python .
+```
 
-This is currently solely maintained by me,
-[@bclindner@mastodon.technology](http://mastodon.technology/@bclindner).
+Hopefully, no errors will be thrown and Ivory will start up and begin its first
+moderation pass, reading the first page of active reports and pending users and
+applying your set rules. Ivory will handle these queues every 300 seconds,
+or 5 minutes. (This is controlled by the `waitTime` part of the above config
+file - if you wanted 10 minutes, you could set it to 600!)
 
+If you'd rather run it on some other schedule via a proper task scheduler like
+cron or a systemd .timer unit, you can use `python . oneshot` which will run
+Ivory only once. This sample cron line will run Ivory every 5 minutes and output
+to a log file:
+
+```cron
+*/5 * * * * cd /absolute/path/to/ivory; ./bin/python . oneshot >> ivory.log
+```
+
+## Extending (custom rules)
+
+You'll notice the `rules/` folder is a flat folder of Python scripts, one per
+Ivory rule. If you've got a little Python experience, you can easily create your
+own rules by just dropping in a new Python file and using one of the other files
+in the folder as a jumping-off point.
+
+The reports and pending accounts that Ivory rules receive are the same as what
+Mastodon.py provides for
+[reports](https://mastodonpy.readthedocs.io/en/stable/#report-dicts) and [admin
+accounts](https://mastodonpy.readthedocs.io/en/stable/#admin-account-dicts),
+respectively.
+
+**Don't forget to use `dryRun` in your config when testing your new rule!**
+
+Once you've finished writing up your custom rule, say as
+`rules/filename_of_your_rule.py`, you can address it by its filename in your
+config:
+
+```json
+...
+"pendingAccounts": {
+  "rules": [
+    {
+      "name": "An instance of my cool new rule",
+      "type": "filename_of_your_rule",
+      "custom_option": true,
+      "severity": 5,
+      "punishment": {
+        "type": "reject"
+      }
+    },
+  ]
+}
+...
+```
+
+If you come up with any useful rules and wouldn't mind writing a schema and some
+tests for it, making a pull request to include it in Ivory's main release would
+be highly appreciated! The more rules Ivory gets, the more tools are
+collectively available to other admins for dealing with spammers and other
+threats to the Fediverse at large.
 
 ## Bugs & Contributing
 
-Contributions welcome. I'm a JS dev, not a Python one, so I may well need
-them.
+If you have any issues with Ivory not working as expected, please file a GitHub
+issue.
 
-Got bugs or feature request? File them as a GitHub issue and I'll address them
-when I can. Same goes for PRs.
+Contributions are also welcome - send in pull requests if you have anything new
+to add.
+
+If you have any other questions, go ahead and [ping me on
+Mastodon](https://mastodon.technology/@bclindner) and I might be able to answer
+them.

__main__.py

@@ -1,4 +1,46 @@
+#!/usr/bin/env python3
+"""
+Main "executable" for Ivory.
+
+Running this file will start Ivory in watch mode, using the config provided in
+config.json.
+"""
+
+import logging
+import sys
+import json
+import argparse
 from ivory import Ivory
+from constants import DEFAULT_CONFIG_PATH, COMMAND_WATCH, COMMAND_ONESHOT
 
-if __name__ == '__main__':
-    Ivory().run()
+if __name__ == "__main__":
+    logger = logging.getLogger(__name__)
+    argparser = argparse.ArgumentParser(
+        description="A Mastodon automoderator.")
+    argparser.add_argument("--config",
+                           dest="configpath",
+                           help="Path to the configuration file (default is config.json)",
+                           default=DEFAULT_CONFIG_PATH)
+    argparser.add_argument('command',
+                           help="Command to run (oneshot to run once, watch to run on a loop). Runs in watch mode by default.",
+                           default=COMMAND_WATCH,
+                           nargs='?',
+                           choices=[COMMAND_WATCH, COMMAND_ONESHOT])
+    args = argparser.parse_args()
+    try:
+        # set up logging
+        logging.basicConfig(stream=sys.stdout)
+        with open(args.configpath) as config_file:
+            config = json.load(config_file)
+        logging.getLogger().setLevel(config.get('logLevel', logging.INFO))
+        # start up ivory in watch mode
+        if args.command == COMMAND_WATCH:
+            Ivory(config).watch()
+        elif args.command == COMMAND_ONESHOT:
+            Ivory(config).run()
+    except OSError as err:
+        logger.exception("failed to load config file")
+        exit(1)
+    except KeyboardInterrupt as err:
+        logger.info("interrupt signal detected, exiting")
+        exit(1)

constants.py

@@ -1 +1,23 @@
-VERSION = "0.2"
+"""
+Constants for Ivory.
+"""
+
+# Ivory version number
+VERSION = "v1.0.0"
+
+# Default amount of seconds to wait between report passes
+DEFAULT_WAIT_TIME = 300
+
+# Default configuration path to use when no path is manually specified
+DEFAULT_CONFIG_PATH = "config.json"
+
+# Punishment types
+PUNISH_WARN = "warn"
+PUNISH_REJECT = "reject"
+PUNISH_DISABLE = "disable"
+PUNISH_SILENCE = "silence"
+PUNISH_SUSPEND = "suspend"
+
+# Command types
+COMMAND_WATCH = "watch"
+COMMAND_ONESHOT = "oneshot"