Delete all Mesos-using code (+ misc. cleanups) (#215)

1. Clean-up makefile (removed all dead targets, made all targets PHONY)
2. Remove all unused plugins (deleted code is code we don't need to maintain)
3. Regenerate docs (target was not phony, so it would never regenerate docs unless run with --always-make)
4. Clean-up README.md (removed Mesos references as well as references to deleted plugins and whatnot)
5. Cleaned up dead tests (this included deleting all the itests since they were all Mesos-related)

Makefile

@@ -5,24 +5,27 @@ else
 	BUILD_ENV?=$(shell hostname -f)
 endif
 
+.PHONY: venv
 venv:
 	tox -e venv
 
+.PHONY: test
 test:
 	tox
 
+.PHONY: tox_%
 tox_%:
 	tox -e $*
 
-itest:
-	tox -e integration
-
+.PHONY: docs
 docs:
 	tox -e docs
 
+.PHONY: pypi
 pypi:
 	tox -e pypi
 
+.PHONY: clean
 clean:
 	rm -rf docs/build
 	find . -name '*.pyc' -delete

README.md

@@ -6,47 +6,9 @@ Interfaces and shared infrastructure for generic task processing (also known as
 
 ### Pre-requisites
 
-+ [Docker](https://www.docker.com/get-docker)
 + [Python 3.8](https://www.python.org/downloads/)
 + [Virtualenv](https://virtualenv.pypa.io/en/stable/installation/)
 
-### Running examples
-
-[hello-world.py](/examples/hello-world/py) is a very simple annotated example that launches a task to echo `hello world`. From the root of the repository, run:
-
-    docker-compose -f examples/cluster/docker-compose.yaml \
-      run playground examples/hello-world.py
-
-This will bring up a single master, single agent Mesos cluster using [Docker Compose](https://docs.docker.com/compose/) and launch a single task which will print "hello world" to the sandbox's stdout before terminating.
-
-Other examples available include:
-+ async.py
-Example of the [async](#async) task runner.
-
-+ dynamo_persistence.py
-Example that shows how task events may be persisted to [DynamoDB](https://aws.amazon.com/dynamodb) using the `stateful` plugin.
-
-+ file_persistence.py
-Example that shows how task events may be persisted to disk using the `stateful` plugin.
-
-+ promise.py
-Example that shows how the [promise/future](#Promise/Future) task runner (not yet implemented) may be used.
-
-+ subscription.py
-Example of the [subscription](#subscription) task runner.
-
-+ sync.py
-Brief example using the [sync](#sync) task runner.
-
-+ timeout.py
-Example that shows how to timeout a task execution using the `timeout` plugin.
-
-+ retry.py
-Example that shows how to retry a task on failure using the `retry` plugin.
-
-+ task_logging.py
-Example that shows how to fetch task logs from Mesos agents using the `logging` plugin.
-
 ### Running tests
 
 From the root of the repository, run:
@@ -65,30 +27,10 @@ From the root of the repository, run:
 
 ### /plugins
 
-Plugins can be chained to create a task execution pipeline with more than one property. Please refer to persistence/retry/timeout examples.
-
-#### mesos
-Implements all required interfaces to talk to Mesos deployment. This plugin uses [PyMesos](https://github.com/douban/pymesos) to communicate with Mesos.
-
-#### timeout
-Implements an executor to timeout task execution.
-
-#### retrying
-Implements an executor to retry task execution upon failure.
-
-#### logging
-Implements an executor to retrieve task logs from Mesos agents. Note that it has to be the immediate upstream executor of the mesos executor.
-
-##### Configuration options
-
-- authentication\_principal Mesos principal
-- credential\_secret\_file path to file containing Mesos secret
-- mesos\_address host:port to connect to Mesos cluster
-- event_translator a fucntion that maps Mesos-specific events to `Event` objects
-
-#### stateful
+Plugins can be chained to create a task execution pipeline with more than one property.
 
-TODO: documentation
+#### Kubernetes
+Implements all required interfaces to talk to Kubernetes. This plugin uses [kubernetes-client](https://github.com/kubernetes-client/python) to communicate with Kubernetes.
 
 ### /runners
 

docs/source/generated/task_processing.interfaces.event.rst

@@ -1,7 +1,7 @@
-task\_processing\.interfaces\.event module
-==========================================
+task\_processing.interfaces.event module
+========================================
 
 .. automodule:: task_processing.interfaces.event
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.interfaces.persistence.rst

@@ -0,0 +1,7 @@
+task\_processing.interfaces.persistence module
+==============================================
+
+.. automodule:: task_processing.interfaces.persistence
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.interfaces.rst

@@ -1,19 +1,21 @@
-task\_processing\.interfaces package
-====================================
+task\_processing.interfaces package
+===================================
 
 Submodules
 ----------
 
 .. toctree::
+   :maxdepth: 4
 
    task_processing.interfaces.event
+   task_processing.interfaces.persistence
    task_processing.interfaces.runner
    task_processing.interfaces.task_executor
 
 Module contents
 ---------------
 
 .. automodule:: task_processing.interfaces
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.interfaces.runner.rst

@@ -1,7 +1,7 @@
-task\_processing\.interfaces\.runner module
-===========================================
+task\_processing.interfaces.runner module
+=========================================
 
 .. automodule:: task_processing.interfaces.runner
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.interfaces.task_executor.rst

@@ -1,7 +1,7 @@
-task\_processing\.interfaces\.task\_executor module
-===================================================
+task\_processing.interfaces.task\_executor module
+=================================================
 
 .. automodule:: task_processing.interfaces.task_executor
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.metrics.rst

@@ -0,0 +1,7 @@
+task\_processing.metrics module
+===============================
+
+.. automodule:: task_processing.metrics
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.kube_client.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.kube\_client module
+=======================================================
+
+.. automodule:: task_processing.plugins.kubernetes.kube_client
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.kubernetes_pod_executor.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.kubernetes\_pod\_executor module
+====================================================================
+
+.. automodule:: task_processing.plugins.kubernetes.kubernetes_pod_executor
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.rst

@@ -0,0 +1,23 @@
+task\_processing.plugins.kubernetes package
+===========================================
+
+Submodules
+----------
+
+.. toctree::
+   :maxdepth: 4
+
+   task_processing.plugins.kubernetes.kube_client
+   task_processing.plugins.kubernetes.kubernetes_pod_executor
+   task_processing.plugins.kubernetes.task_config
+   task_processing.plugins.kubernetes.task_metadata
+   task_processing.plugins.kubernetes.types
+   task_processing.plugins.kubernetes.utils
+
+Module contents
+---------------
+
+.. automodule:: task_processing.plugins.kubernetes
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.task_config.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.task\_config module
+=======================================================
+
+.. automodule:: task_processing.plugins.kubernetes.task_config
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.task_metadata.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.task\_metadata module
+=========================================================
+
+.. automodule:: task_processing.plugins.kubernetes.task_metadata
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.types.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.types module
+================================================
+
+.. automodule:: task_processing.plugins.kubernetes.types
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.kubernetes.utils.rst

@@ -0,0 +1,7 @@
+task\_processing.plugins.kubernetes.utils module
+================================================
+
+.. automodule:: task_processing.plugins.kubernetes.utils
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.plugins.rst

@@ -1,17 +1,18 @@
-task\_processing\.plugins package
-=================================
+task\_processing.plugins package
+================================
 
 Subpackages
 -----------
 
 .. toctree::
+   :maxdepth: 4
 
-    task_processing.plugins.mesos
+   task_processing.plugins.kubernetes
 
 Module contents
 ---------------
 
 .. automodule:: task_processing.plugins
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.rst

@@ -5,15 +5,26 @@ Subpackages
 -----------
 
 .. toctree::
+   :maxdepth: 4
 
-    task_processing.interfaces
-    task_processing.plugins
-    task_processing.runners
+   task_processing.interfaces
+   task_processing.plugins
+   task_processing.runners
+
+Submodules
+----------
+
+.. toctree::
+   :maxdepth: 4
+
+   task_processing.metrics
+   task_processing.task_processor
+   task_processing.utils
 
 Module contents
 ---------------
 
 .. automodule:: task_processing
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.runners.async_runner.rst

@@ -0,0 +1,7 @@
+task\_processing.runners.async\_runner module
+=============================================
+
+.. automodule:: task_processing.runners.async_runner
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/task_processing.runners.promise.rst

@@ -1,7 +1,7 @@
-task\_processing\.runners\.promise module
-=========================================
+task\_processing.runners.promise module
+=======================================
 
 .. automodule:: task_processing.runners.promise
-    :members:
-    :undoc-members:
-    :show-inheritance:
+   :members:
+   :undoc-members:
+   :show-inheritance: