introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Research Object Composer tutorial\n",
    "\n",
    "* Version: 2019-06-28  (see https://github.com/researchobject/research-object-composer for latest)\n",
    "\n",
    "This is a [Jupyter Notebook](https://jupyter.org/) demonstrating how a client can use the [Research Object Composer](https://github.com/researchobject/research-object-composer) REST API.\n",
    "\n",
    "For requirements to run this notebook interactively, see the [README](https://github.com/ResearchObject/research-object-composer/blob/master/README.md). \n",
    "\n",
    "The [RO Composer API](https://researchobject.github.io/research-object-composer/api/) is documented using [Swagger OpenAPI](https://swagger.io/docs/specification/about/) 2.0, which means the REST API can be integrated into programming languages, however this notebook uses [Python](https://www.python.org/) to not hide too much of the HTTP details.\n",
    "\n",
    "To execute this notebook, select each cell in order, then click the **▶️Run** button above.\n",
    "\n",
    "## Python requirements\n",
    "\n",
    "For the below examples we'll use the Python library [requests](https://pypi.org/project/requests/) to show the HTTP  interactions. Below assumes a basic knowledge of [REST services](https://en.wikipedia.org/wiki/Representational_state_transfer).\n",
    "\n",
    "If the below `import` does not work, try on the command line where you started Jupyter Notebook: `pip install requests`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "true,false = (True,False) # for JSON example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RO Composer is meant to be installed on a local infrastructure or as a cloud service. The below uses a **demonstration** service hosted by The University of Manchester which is not supported and *may become unavailable* in the future.\n",
    "\n",
    "If you are testing the service locally using _Docker Compose_ (see [README](https://github.com/ResearchObject/research-object-composer/blob/master/README.md)) - change below to `http://localhost:8080` or use equivalent server name if you are hosting it as a cloud service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Billboard Document\n",
    "\n",
    "As a starting point we'll retrieve the **billboard resource** - a kind of programmatic homepage that tells us what we can do at the RO Composer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "host = \"http://openphacts.cs.man.ac.uk:8080/\"\n",
    "r = requests.get(host)\n",
    "r.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[HTTP status code](https://tools.ietf.org/html/rfc7231#section-6.3.1) `200` means **OK**, so let's see what is the _content type_ of the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'application/hal+json;charset=UTF-8'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.headers[\"Content-Type\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The API results from RO Composer are regular [JSON](https://tools.ietf.org/html/rfc7159) that also follow the _Hypertext Application Language_ ([HAL](http://stateless.co/hal_specification.html)) patterns for RESTful services. Let's look at the content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/'},\n",
       "  'profiles': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles'},\n",
       "  'researchObjects': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects'}}}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index = r.json()\n",
    "index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In HAL, the `_links` section contain links to related REST resources, in this case `self` refer back to the HTTP resource we just requested. Hyperlinks are given with `href`, almost like in HTML.\n",
    "\n",
    "This way of navigating REST resources mean we do not have to commit to fixed URI patterns, we will see later why this style of [Hypermedia as the Engine of Application State](https://restfulapi.net/hateoas/) is an important aspect of RO Composer.\n",
    "\n",
    "The other two links are `profiles` and `researchObjects`. Let's explore:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List known Research Objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_embedded': {'researchObjectSummaryList': [{'id': 3,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': 'https://sandbox.zenodo.org/api/records/313550',\n",
       "    'createdAt': '2019-06-26T02:46:05.512+0000',\n",
       "    'modifiedAt': '2019-06-26T02:46:35.785+0000',\n",
       "    'depositedAt': '2019-06-26T02:46:35.784+0000',\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/3'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       "     'deposition': {'href': 'https://sandbox.zenodo.org/api/records/313550'}}},\n",
       "   {'id': 4,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': 'https://doi.org/10.17632/y75h929bwk.1',\n",
       "    'createdAt': '2019-06-26T02:49:07.175+0000',\n",
       "    'modifiedAt': '2019-06-26T02:49:41.220+0000',\n",
       "    'depositedAt': '2019-06-26T02:49:41.218+0000',\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/4'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       "     'deposition': {'href': 'https://doi.org/10.17632/y75h929bwk.1'}}},\n",
       "   {'id': 9,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': None,\n",
       "    'createdAt': '2019-06-26T14:37:20.403+0000',\n",
       "    'modifiedAt': None,\n",
       "    'depositedAt': None,\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/9'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'}}},\n",
       "   {'id': 5,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': None,\n",
       "    'createdAt': '2019-06-26T11:29:03.002+0000',\n",
       "    'modifiedAt': '2019-06-26T11:30:41.577+0000',\n",
       "    'depositedAt': None,\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/5'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'}}},\n",
       "   {'id': 11,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': None,\n",
       "    'createdAt': '2019-06-26T14:43:38.147+0000',\n",
       "    'modifiedAt': None,\n",
       "    'depositedAt': None,\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/11'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'}}},\n",
       "   {'id': 12,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': None,\n",
       "    'createdAt': '2019-06-26T14:46:55.050+0000',\n",
       "    'modifiedAt': None,\n",
       "    'depositedAt': None,\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/12'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'}}},\n",
       "   {'id': 6,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': 'https://sandbox.zenodo.org/api/records/314431',\n",
       "    'createdAt': '2019-06-26T13:20:28.091+0000',\n",
       "    'modifiedAt': '2019-06-26T13:48:27.557+0000',\n",
       "    'depositedAt': '2019-06-26T13:48:27.555+0000',\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/6'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       "     'deposition': {'href': 'https://sandbox.zenodo.org/api/records/314431'}}},\n",
       "   {'id': 7,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': None,\n",
       "    'createdAt': '2019-06-26T14:15:28.335+0000',\n",
       "    'modifiedAt': None,\n",
       "    'depositedAt': None,\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/7'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'}}},\n",
       "   {'id': 8,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': None,\n",
       "    'createdAt': '2019-06-26T14:17:16.695+0000',\n",
       "    'modifiedAt': None,\n",
       "    'depositedAt': None,\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/8'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'}}},\n",
       "   {'id': 13,\n",
       "    'profileName': 'data_bundle',\n",
       "    'depositionUrl': None,\n",
       "    'createdAt': '2019-06-26T14:53:10.795+0000',\n",
       "    'modifiedAt': None,\n",
       "    'depositedAt': None,\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/13'},\n",
       "     'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'}}}]},\n",
       " '_links': {'first': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects?page=1&size=10'},\n",
       "  'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects?page=1&size=10'},\n",
       "  'next': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects?page=2&size=10'},\n",
       "  'last': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects?page=2&size=10'}},\n",
       " 'page': {'size': 10, 'totalElements': 14, 'totalPages': 2, 'number': 1}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links = index[\"_links\"]\n",
    "ro = links[\"researchObjects\"][\"href\"]\n",
    "researchObjects = requests.get(ro).json()\n",
    "researchObjects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top-level `/research_objects` resource gives us a *paged* listing of [Research Objects](http://www.researchobject.org/) known to the RO Composer. \n",
    "\n",
    "(**Tip**: If the `researchObjectSummaryList` listing is blank, come back to this section later after we have [constructed an RO](#Profiles).\n",
    "\n",
    "### Paging\n",
    "\n",
    "Here under `_links` we find `self` with the canonical URL for the *paged result* (including `?page=1&size=10`). If there are many research objects there will also be links like `next`, `prev`, `first` and `last` with the expected semantics.\n",
    "\n",
    "### Purpose of RO Composer\n",
    "\n",
    "Note that the RO Composer is **not** a registry of research objects, but it can list research objects _currently under construction_. \n",
    "\n",
    "The intended purpose of the composer is to be a temporary construction site that can be completed by multiple services (e.g. a data management system, a workflow system, a user interface). These clients will be jointly building a Research Object that can then be _validated_ according to a pre-defined schema, before the RO is _downloaded_ or _deposited_ into an archive like [Zenodo](http://zenodo.org/) or [Mendeley Data](https://data.mendeley.com/).\n",
    "\n",
    "_Clients_ of the RO Composer are envisioned to be either _applications_ (driven by a user interface) or _agents_ (engaged automatically from other events, e.g. a workflow run). \n",
    "\n",
    "From this we can think of the RO Composer is a kind of _micro service_ which responsibility is to help other services create and deposit research objects. It is important to point out that the RO Composer is not a required component, any software may instead generate research object themselves by following the [Reserach Object specifications](http://www.researchobject.org/specifications/).\n",
    "\n",
    "### Embedded previews\n",
    "\n",
    "In the above listing we saw only a subset of the research object properties, here embedded as a preview under the `_embedded` section.\n",
    "\n",
    "For each item in the `researchObjectSummaryList` we can find nested `_links` for the research object `self` or the `profile` it is intended to comply with. Let's just pick any RO to have a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/3'},\n",
       " 'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       " 'deposition': {'href': 'https://sandbox.zenodo.org/api/records/313550'}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ro = researchObjects[\"_embedded\"][\"researchObjectSummaryList\"][0]\n",
    "ro_links = ro[\"_links\"]\n",
    "ro_links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(_if this does not work, come back to this section later after [creating an RO](#Profiles)_)\n",
    "\n",
    "### Viewing a Research Object\n",
    "\n",
    "We'll have a brief look at an existing research object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 3,\n",
       " 'content': {'data': [{'url': 'https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes.tsv',\n",
       "    'length': 1982,\n",
       "    'filename': 'repository-sizes.tsv',\n",
       "    'checksums': [{'type': 'sha256',\n",
       "      'checksum': 'c2160e931a6ddb8cddb451190816196fc667c5f25020a89a356a69e75ec8dc0a'}]},\n",
       "   {'url': 'https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes-chart.png',\n",
       "    'length': 23803,\n",
       "    'filename': 'repository-sizes-chart.png',\n",
       "    'checksums': [{'type': 'sha256',\n",
       "      'checksum': 'e8bf79ca6fbe83aa0c34ec12705e34d70c348d53e0795504210e13982725300c'}]}],\n",
       "  '_metadata': {'title': 'A good start',\n",
       "   'version': 'E07F620AEBEBD2B2AC3FF332F8AC770F40ECD07A75EED96A9F3CE7BE2ADEEDBA',\n",
       "   'creators': [{'name': 'Alice W Land', 'orcid': '0000-0002-1825-0097'}],\n",
       "   'description': 'A test dataset of not much interest',\n",
       "   'upload_type': 'dataset',\n",
       "   'access_right': 'open',\n",
       "   'publication_date': '2019-06-26'}},\n",
       " 'createdAt': '2019-06-26T02:46:05.512+0000',\n",
       " 'modifiedAt': '2019-06-26T02:46:35.785+0000',\n",
       " 'depositedAt': '2019-06-26T02:46:35.784+0000',\n",
       " 'mutable': False,\n",
       " 'profileName': 'data_bundle',\n",
       " 'checksum': 'AE388A46A4C532E63E314ADACCCEBC4893228AB5B83250FFB76D10595AD42B96',\n",
       " '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/3'},\n",
       "  'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       "  'content': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/3/content'},\n",
       "  'deposition': {'href': 'https://sandbox.zenodo.org/api/records/313550'},\n",
       "  'fields': [{'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/3/content/data',\n",
       "    'name': 'data'},\n",
       "   {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/3/content/_metadata',\n",
       "    'name': '_metadata'}]}}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_ro = requests.get(ro_links[\"self\"][\"href\"]).json()\n",
    "full_ro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Puh! We see quite some additional fields this time. \n",
    "\n",
    "There are two parts to an RO while it is in the composer, most of the above are related to its construction phase (e.g. `createdAt`, `profileName`, `_links`) and are automatically filled in. \n",
    "\n",
    "The elements that are important for deposition are under `content` - these are the fields we will need to fill in using the RO Composer API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': [{'url': 'https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes.tsv',\n",
       "   'length': 1982,\n",
       "   'filename': 'repository-sizes.tsv',\n",
       "   'checksums': [{'type': 'sha256',\n",
       "     'checksum': 'c2160e931a6ddb8cddb451190816196fc667c5f25020a89a356a69e75ec8dc0a'}]},\n",
       "  {'url': 'https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes-chart.png',\n",
       "   'length': 23803,\n",
       "   'filename': 'repository-sizes-chart.png',\n",
       "   'checksums': [{'type': 'sha256',\n",
       "     'checksum': 'e8bf79ca6fbe83aa0c34ec12705e34d70c348d53e0795504210e13982725300c'}]}],\n",
       " '_metadata': {'title': 'A good start',\n",
       "  'version': 'E07F620AEBEBD2B2AC3FF332F8AC770F40ECD07A75EED96A9F3CE7BE2ADEEDBA',\n",
       "  'creators': [{'name': 'Alice W Land', 'orcid': '0000-0002-1825-0097'}],\n",
       "  'description': 'A test dataset of not much interest',\n",
       "  'upload_type': 'dataset',\n",
       "  'access_right': 'open',\n",
       "  'publication_date': '2019-06-26'}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_ro[\"content\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if you access the RO Composer using a browser (rather than say `curl`) you will get a debug web interface that use the same JSON API. See for instance http://openphacts.cs.man.ac.uk:8080/ to see an equivalent listing of  ROs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll see how we can **create** such a research object. But before we can get that far, we need to know what should go into `content`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Profiles\n",
    "\n",
    "Research Objects can be used for different purposes depending on domain- and application-specific expectation. \n",
    "\n",
    "[Profiles](http://www.researchobject.org/scopes/) help define the shape and form of a class of research objects, for instance a _dataset research object_ can contain a couple of vaguely related data files, while a _workflow-centric research object_ may keep more structured workflow definitions, example inputs, execution provenance, etc.\n",
    "\n",
    "Loosely, a profile defines an expectation of what **kind of resources** should be expected, and what **metadata** is required. In a way, a profile defines the general **purpose** of that type of Research Objects, and documents what assumptions a consumer can rely on when processing such Research Objects beyond \"some files\".\n",
    "\n",
    "The RO Composer supports creating Research Object for multiple **profiles**. Each profile is [defined internally](https://github.com/ResearchObject/research-object-composer/tree/master/src/main/resources/public/schemas) using [JSON Schema](https://json-schema.org/), for creating different kind of Research Objects. \n",
    "\n",
    "As a kind of simplification for how research objects are eventually serialized, the choice of _profile_ effectively determines which objects must appear under `content` when constructing that type of ROs.\n",
    "\n",
    "We can query the `/profiles` service to see which profiles are installed:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_embedded': {'researchObjectProfileList': [{'name': 'data_bundle',\n",
       "    'fields': ['data', '_metadata'],\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       "     'schema': {'href': 'http://openphacts.cs.man.ac.uk:8080/schemas/data_bundle.schema.json'},\n",
       "     'template': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle/template'},\n",
       "     'researchObjects': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle/research_objects'}}},\n",
       "   {'name': 'draft_task',\n",
       "    'fields': ['input', 'workflow', 'workflow_params', '_metadata'],\n",
       "    '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/draft_task'},\n",
       "     'schema': {'href': 'http://openphacts.cs.man.ac.uk:8080/schemas/draft_task.schema.json'},\n",
       "     'template': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/draft_task/template'},\n",
       "     'researchObjects': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/draft_task/research_objects'}}}]},\n",
       " '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles'}}}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = requests.get(links[\"profiles\"][\"href\"]).json()\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Profile links\n",
    "\n",
    "Similar to the listing of research objects we get an `_embedded` section of the `researchObjectProfileList`, as each profile itself is a separate REST resource with multiple related resources.\n",
    "\n",
    "Let's look at their `name` fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data_bundle', 'draft_task']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profiles = p[\"_embedded\"]['researchObjectProfileList']\n",
    "[profile[\"name\"] for profile in profiles]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this installation, the profile `data_bundle` is for Research Objects containing arbitrary datasets, while `draft_task` is for more specific ROs describing workflow executions. \n",
    "\n",
    "The intention is that if a client of the RO Composer knows a particular profile in advance, it can build a richer interface or provide additional information from underlying data sources; while it can also deal with the remaining profiles in a more generic way.\n",
    "\n",
    "\n",
    "We'll look at the `data_bundle` in detail. We see that the `content` we can create with this profile only expects the fields `data` and `_metadata`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data', '_metadata']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bundle_profile = next(p for p in profiles if p[\"name\"]==\"data_bundle\")\n",
    "bundle_profile[\"fields\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the particular profile the `_links` refers to several resources, `self`, `schema` and `researchObjects`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       " 'schema': {'href': 'http://openphacts.cs.man.ac.uk:8080/schemas/data_bundle.schema.json'},\n",
       " 'template': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle/template'},\n",
       " 'researchObjects': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle/research_objects'}}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links = bundle_profile[\"_links\"]\n",
    "links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON Schema\n",
    "\n",
    "If we want we can request the underlying [JSON Schema](https://json-schema.org/) to see details of these fields at `/schemas/{name}` - linked to from `schema` above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'href': 'http://openphacts.cs.man.ac.uk:8080/schemas/data_bundle.schema.json'}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema = links[\"schema\"]\n",
    "schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'application/json'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_response = requests.get(schema[\"href\"])\n",
    "schema_response.headers[\"Content-Type\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we are no longer navigating HAL resources at the RO Composer; this is the native representation of the JSON Schema, which of course is also defined in JSON, but using different kind of keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "schema_response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This schema will be used when _validating_ the Research Object `content` created under this profile. You may recognize that `properties` here list the fields `_metadata` and `data`, with further type definitions given by reference. \n",
    "\n",
    "This introduction does not go in detail on JSON Schema, to a large degree the role of the RO Composer is also to hide these implementation details and present a simplified picture by making individual REST resources for each field."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Future profile work \n",
    "\n",
    "The JSON Schemas are discovered during deployment and cannot be modified through the RO Composer APIs or without restarting the service.\n",
    "\n",
    "_Versioning_ and careful updating of schemas will be important for longer-term use, as older Research Objects may no longer be valid (e.g. adding a new required field).\n",
    "\n",
    "Documentation fields could be lifted from the JSON Schema definitions to be embedded in individual profile `field` listing and error messages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Research Object\n",
    "\n",
    "The REST resource that collect [Research Objects for the given profile]((https://researchobject.github.io/research-object-composer/api/#operation/listResearchObjectsForProfile)) is at `/profiles/{name}/research_objects` and is linked to from the `researchObjects` link:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle/research_objects'}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "researchObjects = links[\"researchObjects\"]\n",
    "researchObjects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As this resource is a _collection_ it supports RO creation using [POST](https://researchobject.github.io/research-object-composer/api/#operation/createResearchObject) to start building a research object of that type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [201]>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "created = requests.post(researchObjects[\"href\"])\n",
    "created"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In HTTP, **201 Created** means a new HTTP resource was made. We can find out _where_ from the `Location` header:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://openphacts.cs.man.ac.uk:8080/research_objects/17'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ro_uri = created.headers[\"Location\"]\n",
    "ro_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(If you previously got an empty listing of `/research_objects` now is a good time to check back on the [previous section](#List-known-Research-Objects) - this RO is definitely under construction as we have not provided any data yet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Completing the Research Object\n",
    "\n",
    "The response also includes a preview of the created Research Object resource (we don't need to `GET` it), where we'll find the same URI under the `self` link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 17,\n",
       " 'content': {'data': [],\n",
       "  '_metadata': {'title': None, 'description': None, 'creators': []}},\n",
       " 'createdAt': '2019-06-28T13:39:36.784+0000',\n",
       " 'modifiedAt': None,\n",
       " 'depositedAt': None,\n",
       " 'mutable': True,\n",
       " 'profileName': 'data_bundle',\n",
       " 'checksum': None,\n",
       " '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17'},\n",
       "  'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       "  'content': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content'},\n",
       "  'fields': [{'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content/data',\n",
       "    'name': 'data'},\n",
       "   {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content/_metadata',\n",
       "    'name': '_metadata'}]}}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "created.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember before we heard about the fields `data` and `_metadata`? We now see them under `content`, however they are partially populated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': [], '_metadata': {'title': None, 'description': None, 'creators': []}}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "created.json()[\"content\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Research Object links\n",
    "\n",
    "We have a corresponding REST resource to populate each field, which we find under `_links`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17'},\n",
       " 'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       " 'content': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content'},\n",
       " 'fields': [{'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content/data',\n",
       "   'name': 'data'},\n",
       "  {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content/_metadata',\n",
       "   'name': '_metadata'}]}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links = created.json()[\"_links\"]\n",
    "links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While each field might have additional properties (e.g. documentation), we are here only interested in their `name` and the corresponding `href` HTTP resource to fill the field. This Python code converts this to a `name`->`href` dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content/data',\n",
       " '_metadata': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content/_metadata'}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fields = dict((f[\"name\"], f[\"href\"] ) for f in links[\"fields\"])\n",
    "fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding data to a Research Object\n",
    "\n",
    "Let's fill `data` first. We saw in `content` it was a array  `[]`. RO Composer exposes this as a REST _collection_ where we can `POST` to add items. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [400]>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = requests.post(fields[\"data\"], json={})\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uups, **400 Bad Request**, perhaps `{}` was a bit minimal? Maybe we should have read that JSON Schema after all... Let's look at the returned **schema violations**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'pointerToViolation': '#',\n",
       " 'message': '#: 4 schema violations found',\n",
       " 'causingExceptions': [{'keyword': 'required',\n",
       "   'pointerToViolation': '#',\n",
       "   'message': '#: required key [length] not found',\n",
       "   'causingExceptions': [],\n",
       "   'schemaLocation': '/schemas/_base.schema.json#/definitions/RemoteItem'},\n",
       "  {'keyword': 'required',\n",
       "   'pointerToViolation': '#',\n",
       "   'message': '#: required key [filename] not found',\n",
       "   'causingExceptions': [],\n",
       "   'schemaLocation': '/schemas/_base.schema.json#/definitions/RemoteItem'},\n",
       "  {'keyword': 'required',\n",
       "   'pointerToViolation': '#',\n",
       "   'message': '#: required key [url] not found',\n",
       "   'causingExceptions': [],\n",
       "   'schemaLocation': '/schemas/_base.schema.json#/definitions/RemoteItem'},\n",
       "  {'keyword': 'required',\n",
       "   'pointerToViolation': '#',\n",
       "   'message': '#: required key [checksums] not found',\n",
       "   'causingExceptions': [],\n",
       "   'schemaLocation': '/schemas/_base.schema.json#/definitions/RemoteItem'}],\n",
       " 'schemaLocation': '/schemas/_base.schema.json#/definitions/RemoteItem'}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.json()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This detailed error message is coming straight from the JSON Schema validator. The RO Composer will not allow us to change the resource into an invalid state. \n",
    "\n",
    "Probably the most useful information is under `message`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['#: required key [length] not found',\n",
       " '#: required key [filename] not found',\n",
       " '#: required key [url] not found',\n",
       " '#: required key [checksums] not found']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[ex[\"message\"] for ex in data.json()[\"causingExceptions\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see here that we are missing 4 fields from the [RemoteItem](https://github.com/ResearchObject/research-object-composer/blob/master/src/main/resources/public/schemas/_base.schema.json#L4) type, `length`, `filename`, `url` and `checksums`; these properties are used in `data` to reference remote files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding remote resources\n",
    "\n",
    "For the purpose of this demonstration we'll create a [simple dataset](https://github.com/ResearchObject/ro-lite/tree/master/examples/simple-dataset-0.1.0/data) containing a [TSV file](https://github.com/ResearchObject/ro-lite/blob/master/examples/simple-dataset-0.1.0/data/repository-sizes.tsv) and a [PNG image](https://github.com/ResearchObject/ro-lite/blob/master/examples/simple-dataset-0.1.0/data/repository-sizes-chart.png). \n",
    "\n",
    "For simplicity the resources are hosted by GitHub and we have already calculated the `length` and `checksums`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tsv = { \"url\": \"https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes.tsv\",\n",
    "        \"length\": 1982,\n",
    "        \"filename\": \"repository-sizes.tsv\",\n",
    "        \"checksums\": [{\"type\": \"sha256\", \n",
    "                       \"checksum\": \"c2160e931a6ddb8cddb451190816196fc667c5f25020a89a356a69e75ec8dc0a\"}]\n",
    "      } \n",
    "png = { \"url\": \"https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes-chart.png\",\n",
    "        \"length\": 23803,\n",
    "        \"filename\": \"repository-sizes-chart.png\",\n",
    "        \"checksums\": [{\"type\": \"sha256\", \n",
    "                       \"checksum\": \"e8bfc12705e34d70c348d53e0795504210e13982725300c\"}]\n",
    "      } \n",
    "tsv_uploaded = requests.post(fields[\"data\"], json=tsv)\n",
    "tsv_uploaded.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "400"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_uploaded = requests.post(fields[\"data\"], json=png)\n",
    "csv_uploaded.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**200 OK** here means we complied with the JSON Schema, you may get an error if you get any of the keys wrong, or the checksum value is of the incorrect length (supported `checksums`: `md5`, `sha1`, `sha256`, `sha512`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Future RO creation work\n",
    "\n",
    "In this implementation, the RO Composer assumes the client already have stable http/https URLs for any files that are to be included in the RO. For instance, the [Seven Bridges Platform](https://www.sevenbridges.com/platform/) has an underlying data store used during workflow execution, where each file have a corresponding URL and a pre-calculated checksum.\n",
    "\n",
    "As web references such URLs could change or disappear, out of control of the RO Composer or the repository. While RO Composer can record the checksum at the time of recording, for long-term storage such files could be archived in an immutable data store, potentially also minting a [MinID](http://minid.bd2k.org/) to use as an indirect  reference.\n",
    "\n",
    "\n",
    "A firewalled, desktop or web client would not have the ability to expose files on the web, and would need a different mechanism for including local files. It was decided that the RO Composer itself should not be holding such (potentially large or incriminating) files, but that it could be adapted to facilitate a pass-thru upload to a directly stage at the destination repository (both Mendeley Data and Zenodo support such \"draft\" status).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting the Research Object\n",
    "\n",
    "Now let's reload the RO and see that we have populated `content` with the two items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 17,\n",
       " 'content': {'data': [{'url': 'https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes.tsv',\n",
       "    'length': 1982,\n",
       "    'filename': 'repository-sizes.tsv',\n",
       "    'checksums': [{'type': 'sha256',\n",
       "      'checksum': 'c2160e931a6ddb8cddb451190816196fc667c5f25020a89a356a69e75ec8dc0a'}]},\n",
       "   {'url': 'https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes-chart.png',\n",
       "    'length': 23803,\n",
       "    'filename': 'repository-sizes-chart.png',\n",
       "    'checksums': [{'type': 'sha256',\n",
       "      'checksum': 'e8bf79ca6fbe83aa0c34ec12705e34d70c348d53e0795504210e13982725300c'}]},\n",
       "   {'url': 'https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes.tsv',\n",
       "    'length': 1982,\n",
       "    'filename': 'repository-sizes.tsv',\n",
       "    'checksums': [{'type': 'sha256',\n",
       "      'checksum': 'c2160e931a6ddb8cddb451190816196fc667c5f25020a89a356a69e75ec8dc0a'}]}],\n",
       "  '_metadata': {'title': None, 'creators': [], 'description': None}},\n",
       " 'createdAt': '2019-06-28T13:39:36.784+0000',\n",
       " 'modifiedAt': '2019-06-28T13:45:15.359+0000',\n",
       " 'depositedAt': None,\n",
       " 'mutable': True,\n",
       " 'profileName': 'data_bundle',\n",
       " 'checksum': None,\n",
       " '_links': {'self': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17'},\n",
       "  'profile': {'href': 'http://openphacts.cs.man.ac.uk:8080/profiles/data_bundle'},\n",
       "  'content': {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content'},\n",
       "  'fields': [{'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content/data',\n",
       "    'name': 'data'},\n",
       "   {'href': 'http://openphacts.cs.man.ac.uk:8080/research_objects/17/content/_metadata',\n",
       "    'name': '_metadata'}]}}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ro = requests.get(ro_uri)\n",
    "ro.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a dataset\n",
    "\n",
    "A good question at this point is - can you add one dataset to another?\n",
    "\n",
    "In theory yes, but in practice we will see some considerations are needed. Let's say we have found Paul Groth's dataset [Evaluating Open Information Extraction on Scientific and Medical Text](https://doi.org/10.17632/6m5dyx4b58.2) and we want to include it in our Research Object, as we have evolved our dataset further. It has a nice DOI, authors, descriptions etc. and so is already playing along well.\n",
    "\n",
    "Immediately you may see a challenge - a data set is naturally a **set** - it can have many files. So which one do we mean? Let's say we used both `aggregated_results_anon.csv` which we see in Mendeley Data has a link of `https://data.mendeley.com/datasets/6m5dyx4b58/2/files/03de3c93-8a01-4cd2-bfe5-c5bfeaa4f492` - would this naturally fit into `data` above? No, that seems to give a HTML landing page - we don't want to add the HTML to the bag, we want the CSV file.  \n",
    "\n",
    "After hunting around you see `http://data.mendeley.com/datasets/6m5dyx4b58/2/files/03de3c93-8a01-4cd2-bfe5-c5bfeaa4f492/aggregated_results_anon.csv?dl=1` which we could use (and if you are familiar with Mendeley Data API you know you can also collect the MD5 checksum of the file.\n",
    "\n",
    "However now we have fully lost the DOI of the dataset, we have lost the author of the dataset, and we have lost any description that we could have seen on the landing page. If we are making a **research object** and not just a _file collection_, then we should not need to pick up files and re-fill metadata. \n",
    "\n",
    "So here is a future role for RO Composer to help with these tasks in **preserving metadata** - in a wonderful future world it could be able to connect to any DataCite repository and find out how to download their files, perhaps as a Research Object - without having to text-mine the HTML landing page of the DOI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filling in metadata\n",
    "\n",
    "\n",
    "Next we'll add sufficient *metadata* so that the RO can later be published and assigned a DOI. Let's look at what fields we already have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': None, 'creators': [], 'description': None}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata = requests.get(fields[\"_metadata\"]).json()\n",
    "metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The RO Composer has filled in some default values based on the schema. A `title` is a good start. As this is a single resource we can use `PUT` to replace it's value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'pointerToViolation': '#',\n",
       " 'message': '#: 2 schema violations found',\n",
       " 'causingExceptions': [{'keyword': 'minItems',\n",
       "   'pointerToViolation': '#/creators',\n",
       "   'message': '#/creators: expected minimum item count: 1, found: 0',\n",
       "   'causingExceptions': [],\n",
       "   'schemaLocation': '/schemas/_base.schema.json#/definitions/Metadata/properties/creators'},\n",
       "  {'keyword': 'type',\n",
       "   'pointerToViolation': '#/description',\n",
       "   'message': '#/description: expected type: String, found: Null',\n",
       "   'causingExceptions': [],\n",
       "   'schemaLocation': '/schemas/_base.schema.json#/definitions/Metadata/properties/description'}],\n",
       " 'schemaLocation': '/schemas/_base.schema.json#/definitions/Metadata'}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata[\"title\"] = \"A good start\"\n",
    "updated = requests.put(fields[\"_metadata\"], json=metadata)\n",
    "updated.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again the RO Composer will not allow us to push the RO into an invalid state; we see we also need `description` and at least one element of the `creators` aray.\n",
    "\n",
    "These attributes correspond to fields in the [DataCite schema](https://schema.datacite.org/), but in JSON instead of XML. From the name and error message we may guess that `creators` is an array, but we don't know it's fields yet. \n",
    "\n",
    "Let's fill in our `description` and for now try with `{}` as the creator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'keyword': 'required',\n",
       " 'pointerToViolation': '#/creators/0',\n",
       " 'message': '#/creators/0: required key [name] not found',\n",
       " 'causingExceptions': [],\n",
       " 'schemaLocation': '/schemas/data_bundle.schema.json#/definitions/Author'}"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata[\"description\"] =  \"A test dataset of not much interest\"\n",
    "metadata[\"creators\"] = [{}]\n",
    "updated = requests.put(fields[\"_metadata\"], json=metadata)\n",
    "updated.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a sense this highlights one of the advantages of this staged approach to filling the Research Object, you trigger any errors as the item is being set rather than buried deep inside a large schema validation report at a later stage. \n",
    "\n",
    "Under the special key `_metadata`, any fields from [DataCite schema](https://schema.datacite.org/) can be added, so we can for instance include `affiliation` or `orcid` - although not required we recommend always including [ORCID](http://orcid.org/) to uniquely identify the creator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "updated = requests.put(fields[\"_metadata\"], json=\n",
    "                        { \"title\": \"A good start\", \n",
    "                          \"description\": \"A test dataset of not much interest\",\n",
    "                          \"access_right\": \"open\",\n",
    "                          \"creators\": [{\"name\": \"Alice W Land\", \n",
    "                                        \"orcid\": \"https://orcid.org/0000-0002-1825-0097\"}] })\n",
    "updated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip:** You may try to deliberately break the `orcid` value above to show that schema validation can also be done on optional fields. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a RESTful resource it is safe to use `PUT` multiple times in case we change our mind, e.g. if the user was editing equivalent forms in the UI. [Other metadata fields](https://developers.zenodo.org/#representation) from Zenodo can also be added to `_metadata` to be passed on to the archive at deposit time, e.g. `keywords` or `related_identifiers`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Research Object\n",
    "\n",
    "Now we can **download** the Research Object as a [BagIt archive](https://tools.ietf.org/html/rfc8493) (RFC8493) from the `/research_objects/{id}/bag` resource (currently this REST resource is not listed under `_links`).\n",
    "\n",
    "As this is a binary (ZIP file) we'll use a slightly different Python method to save it to a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "with requests.post(ro_uri + \"/bag\", stream=True) as bag:\n",
    "    bag.raise_for_status()    \n",
    "    with open(\"bag.zip\", \"wb\") as zipfile:\n",
    "        shutil.copyfileobj(bag.raw, zipfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that RO Composer will not let you create the BagIt archive if its `content` is not valid, which is why we call `bag.raise_for_status()` to fail on any `400` errors (we don't want the error message written to the ZIP file).\n",
    "\n",
    "Let's check the content of the downloaded zip file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bag-info.txt',\n",
       " 'bagit.txt',\n",
       " 'data/content.json',\n",
       " 'fetch.txt',\n",
       " 'manifest-md5.txt',\n",
       " 'manifest-sha256.txt',\n",
       " 'manifest-sha512.txt',\n",
       " 'metadata/manifest.json',\n",
       " 'tagmanifest-md5.txt',\n",
       " 'tagmanifest-sha256.txt',\n",
       " 'tagmanifest-sha512.txt']"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import zipfile\n",
    "zip = zipfile.ZipFile('bag.zip')\n",
    "files = zip.namelist()\n",
    "files.sort() # list in alphabetical order\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These paths follows the [BagIt structure](https://tools.ietf.org/html/rfc8493#section-2) where the [Research Object manifest](https://github.com/ResearchObject/bagit-ro) is under `metadata/manifest.json` and the _payload_ is under `data`.\n",
    " \n",
    "You may notice that the two remote files we added are **not** present in the zip file, they are referenced from `fetch.txt` and `manifest-sha256.txt` - this means that even if large files are added to the Research Object, its download ZIP remains small (until the bag is _completed_ using BagIt tools like [BDBag](http://bd2k.ini.usc.edu/tools/bdbag/)).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes.tsv 1982 data/repository-sizes.tsv\n",
      "https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes-chart.png 23803 data/repository-sizes-chart.png\n",
      "https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes.tsv 1982 data/repository-sizes.tsv\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with zip.open(\"fetch.txt\") as f:\n",
    "    fetch = f.read()\n",
    "print(str(fetch, \"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, the checksums have been propagated to the BagIt manifest, so that the integrity of the completed bag archive can be _validated_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7ef2f8ec27011e43abe312f04d1a0a572a95b61d8258793e883ab1ec366d8d35  data/content.json\n",
      "e8bf79ca6fbe83aa0c34ec12705e34d70c348d53e0795504210e13982725300c  data/repository-sizes-chart.png\n",
      "c2160e931a6ddb8cddb451190816196fc667c5f25020a89a356a69e75ec8dc0a  data/repository-sizes.tsv\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with zip.open(\"manifest-sha256.txt\") as f:\n",
    "    manifest = f.read()\n",
    "print(str(manifest, \"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Research Object manifest\n",
    "\n",
    "Let's have a look at `metadata/manifest.json` - the manifest of the Research Object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'@context': ['https://w3id.org/bundle/context'],\n",
       " 'id': '../',\n",
       " 'manifest': ['manifest.json'],\n",
       " 'createdOn': '2019-06-28T13:51:17.925Z',\n",
       " 'aggregates': [{'uri': 'https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes.tsv',\n",
       "   'bundledAs': {'uri': 'urn:uuid:415d8a80-5b2a-4993-968b-bc90a90041b1',\n",
       "    'folder': '../data/',\n",
       "    'filename': 'repository-sizes.tsv'}},\n",
       "  {'uri': 'https://raw.githubusercontent.com/ResearchObject/research-object-composer/master/examples/repository-sizes-chart.png',\n",
       "   'bundledAs': {'uri': 'urn:uuid:fbd039d8-066b-4b85-a797-18ac90616bae',\n",
       "    'folder': '../data/',\n",
       "    'filename': 'repository-sizes-chart.png'}}]}"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "with zip.open(\"metadata/manifest.json\") as f:\n",
    "    manifest = json.load(f)\n",
    "manifest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the [Research Object manifest](https://github.com/ResearchObject/bagit-ro) we see the two external files have been aggregated and given local paths and identifiers. Future work will explore propagating additional metadata about individual files (e.g. _creator_) into the RO manifest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Future metadata work\n",
    "\n",
    "The RO Composer will currently propagate some of the `content` and `_metadata` properties to higher-level constructs in the archived Research Object. Future work could include:\n",
    "\n",
    "* Generating `datacite.xml` based on `_metadata` (as proposed by [RDA data packaging](https://docs.google.com/document/d/155lA2BcixTl-zwJHGfLkxsmg7WmQbBK00QWyP8QggkE/edit))\n",
    "* Align RO Composer with the community effort [RO-Crate](https://researchobject.github.io/ro-crate/) \n",
    "  * Mapping `_metadata` to http://schema.org/ terms to include in `manifest.json`\n",
    "  * Formalize RO-Crate requirements in RO Composer schemas  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publishing the Research Object\n",
    "\n",
    "Now we can **publish** the Research Object to [Zenodo](https://zenodo.org/) to assign a DOI (**Note**: the demo server is configured to use [https//sandbox.zenodo.org/](https//sandbox.zenodo.org/) which does not actually issue DOIs).\n",
    "\n",
    "To deposit the RO into the archive we `POST` to the `/research_object/{id}/deposit/zenodo` resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "published = requests.post(ro_uri + \"/deposit/zenodo\")\n",
    "published"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any errors in `metadata` above that caused issues in the [Zenodo API](https://developers.zenodo.org/#quickstart-upload) would have caused an error, but we got **200 OK**, so let's have a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://sandbox.zenodo.org/api/records/315263\n"
     ]
    }
   ],
   "source": [
    "print(published.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting the deposited RO\n",
    "\n",
    "In a browser that is logged in to [https://sandbox.zenodo.org/](https://sandbox.zenodo.org/) you can access the above API resource, and will find something similar to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{\"conceptdoi\":\"10.5072/zenodo.313547\",\"conceptrecid\":\"313547\",\"created\":\"2019-06-26T01:51:38.401353+00:00\",\"doi\":\"10.5072/zenodo.313548\",\"files\":[{\"bucket\":\"8c85959b-94ed-44c5-a4d7-15cbe16d9d33\",\"checksum\":\"md5:ee34651d2a8474aa0cb065776d509043\",\"key\":\"data_bundle-31.zip\",\"links\":{\"self\":\"https://sandbox.zenodo.org/api/files/8c85959b-94ed-44c5-a4d7-15cbe16d9d33/data_bundle-31.zip\"},\"size\":3765,\"type\":\"zip\"}],\"id\":313548,\"links\":{\"badge\":\"https://sandbox.zenodo.org/badge/doi/10.5072/zenodo.313548.svg\",\"bucket\":\"https://sandbox.zenodo.org/api/files/8c85959b-94ed-44c5-a4d7-15cbe16d9d33\",\"conceptbadge\":\"https://sandbox.zenodo.org/badge/doi/10.5072/zenodo.313547.svg\",\"conceptdoi\":\"https://doi.org/10.5072/zenodo.313547\",\"doi\":\"https://doi.org/10.5072/zenodo.313548\",\"html\":\"https://sandbox.zenodo.org/record/313548\",\"latest\":\"https://sandbox.zenodo.org/api/records/313548\",\"latest_html\":\"https://sandbox.zenodo.org/record/313548\",\"self\":\"https://sandbox.zenodo.org/api/records/313548\"},\"metadata\":{\"access_right\":\"open\",\"access_right_category\":\"success\",\"creators\":[{\"name\":\"Alice W Land\",\"orcid\":\"0000-0002-1825-0097\"}],\"description\":\"A test dataset of not much interest\",\"doi\":\"10.5072/zenodo.313548\",\"license\":{\"id\":\"CC0-1.0\"},\"publication_date\":\"2019-06-26\",\"related_identifiers\":[{\"identifier\":\"10.5072/zenodo.313547\",\"relation\":\"isVersionOf\",\"scheme\":\"doi\"}],\"relations\":{\"version\":[{\"count\":1,\"index\":0,\"is_last\":true,\"last_child\":{\"pid_type\":\"recid\",\"pid_value\":\"313548\"},\"parent\":{\"pid_type\":\"recid\",\"pid_value\":\"313547\"}}]},\"resource_type\":{\"title\":\"Dataset\",\"type\":\"dataset\"},\"title\":\"A good start\",\"version\":\"60174F2E1B21E7D7BF2C6B413C582AC7751B787519FDEB59890311C902C90F2E\"},\"owners\":[25426],\"revision\":1,\"stats\":{\"downloads\":0.0,\"unique_downloads\":0.0,\"unique_views\":0.0,\"version_downloads\":0.0,\"version_unique_downloads\":0.0,\"version_unique_views\":0.0,\"version_views\":0.0,\"version_volume\":0.0,\"views\":0.0,\"volume\":0.0},\"updated\":\"2019-06-26T01:51:38.555034+00:00\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `latest_html` key gives the more human-readable Zenodo record for browsers, e.g. [https://sandbox.zenodo.org/record/275055](https://sandbox.zenodo.org/record/275055), while `doi` gives the DOI (which would work on the production Zenodo)\n",
    "\n",
    "We recognize our `metadata` properties, which have been augmented to indicate a `dataset` and the `publication_date`.\n",
    "\n",
    "We can also see that the Research Object appears in the [most recent datasets](https://sandbox.zenodo.org/search?page=1&size=20&type=dataset&sort=mostrecent) on Zenodo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Depositing in other archives\n",
    "\n",
    "It is possible to change the [deposition configuration](https://github.com/ResearchObject/research-object-composer/blob/deposition/src/main/resources/depositor.properties) of the RO composer to support depositing to other archives, e.g. [Mendeley Data](https://data.mendeley.com/), although a corresponding [implementation](https://github.com/ResearchObject/research-object-composer/tree/deposition/src/main/java/uk/org/esciencelab/researchobjectservice/deposition) must be added to the code. \n",
    "\n",
    "Current depositors include Zenodo, Mendeley Data, and pure HTTP Post, but we are also planning a [SWORD](http://swordapp.org/) depositor to support multiple repositories. A remaining challenge here is how to unify the minimum metadata across repositories.\n",
    "\n",
    "### Depositing in Mendeley Data\n",
    "\n",
    "RO composer has experimental support to deposit in Mendeley Data, by POSTing to the `/research_object/{id}/deposit/mendeleydata` resource.\n",
    "\n",
    "However Mendeley Data currently requires a short-lived OAuth token, which can be tricky to demonstrate from a Jupyter Notebook. \n",
    "\n",
    "For demonstration purposes use Mendeley Data's [Show Me Access Tokens web page](https://mendeley-show-me-access-tokens.herokuapp.com/) -- note that you need to have a classical Mendeley Data acccount with local password registered on https://dev.mendeley.com/ (SSO login not supported).  Then modify the `access_token` part below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "access_token=\"MSwxNTYxNTIwODEwNzgxLDU1NTA2MTAzMSwxMDI4LGFsbCwsLDY0Yzg3Yzc4MzllYTIwNDBlMzRiNmVjMWIwYjkzYmQ1ODY1M2d4cnFiLDBmNTVmMzZkLTgyZWEtM2E0Yy1hYzVkLWY1NTBkYjE2OThhMSx3akdkazRsRlhqcENYNWpFNy1NcUYyeVhSZEk\"\n",
    "published = requests.post(ro_uri + \"/deposit/mendeleyData?accessToken=\" + access_token)\n",
    "print(published.text)\n",
    "published"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the token is expired you will get a largish _Internal Server Error_ above. Log in again at https://mendeley-show-me-access-tokens.herokuapp.com/\n",
    "\n",
    "If you managed to get the token working, the above should fail with `{\"field\":\"contributors[0].email\",\"message\":\"may not be empty\"}`.  This is an example of minimum metadata requirements varying between repositories. Further testing reveals we need to provide `email`, `first_name` and `last_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "updated = requests.put(fields[\"_metadata\"], json=\n",
    "                        { \"title\": \"A good start\", \n",
    "                          \"description\": \"A test dataset of not much interest\",\n",
    "                          \"creators\": [{\"name\": \"Alice W Land\", \n",
    "                                        \"email\": \"alice@example.com\",\n",
    "                                        \"first_name\": \"Alice\",\n",
    "                                        \"last_name\": \"Land\",\n",
    "                                        \"orcid\": \"https://orcid.org/0000-0002-1825-0097\"}] })\n",
    "published = requests.post(ro_uri + \"/deposit/mendeleyData?accessToken=\" + access_token)\n",
    "print(published.text)\n",
    "published"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in the demonstration RO Composer, the above DOI have **not** been registered yet, the Mendeley Data deposit is instead marked as a _draft_. Log in to https://data.mendeley.com/ to verify the draft has appeared and that the Research Object has been uploaded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example client: Debug interface\n",
    "\n",
    "As an example client we have developed a React.js web UI that uses the RO Composer UI.\n",
    "\n",
    "This can be accessed in the browser:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://openphacts.cs.man.ac.uk:8080/\n"
     ]
    }
   ],
   "source": [
    "print(host)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For existing ROs you can click **View** to see a light-weight rendering of the `content`. The deposited DOI is shown if available.\n",
    "\n",
    "For each profile you can explore the JSON Schemas, or click **New RO**.\n",
    "\n",
    "Note that the form for creating/editing RO is auto-generated based on the schema, so it may not be the most intuitive interface. This is meant more as a _debug_ interface as well demonstrating different levels of integration.\n",
    "\n",
    "In this approach the UI is more closely integrated with the profile's schema and uses a separate JavaScript implementation of JSON Schema to evaluate the field inputs on the go. \n",
    "\n",
    "A more specific implementation could have more customized UI for particular profiles, e.g. file selection without having to fill in `length`, `checksum` etc., or fill in person details from ORCID identifier alone using their API."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}