Merge pull request #28 from amosproj/Dev

Dev

.gitignore

@@ -1 +1,2 @@
-__pycache__
+__pycache__
+/venv/

README.md

@@ -5,7 +5,7 @@
 <!-- PROJECT SHIELDS -->
 <!--
 *** I'm using markdown "reference style" links for readability.
-*** Reference links are enclosed in brackets [ ] instead of parentheses ( ).
+*** Reference links are enclosed in bracketChooses [ ] instead of parentheses ( ).
 *** See the bottom of this document for the declaration of the reference variables
 *** for contributors-url, forks-url, etc. This is an optional, concise syntax you may use.
 *** https://www.markdownguide.org/basic-syntax/#reference-style-links
@@ -107,32 +107,72 @@
 <!-- GETTING STARTED -->
 ## Getting Started
 
-<!--To get a local copy up and running follow these simple steps.
+<!--To get a local copy up and running follow these simple steps. -->
 
 ### Prerequisites
 
-<!--This is an example of how to list things you need to use the software and how to install them.
-* npm
-  ```sh
-  npm install npm@latest -g
-  ```
+* git
+* docker: [follow instructions for your platform](https://docs.docker.com/get-docker/)
 
-<!--### Installation
+<!--### Installation -->
+### Installation
 
-<!--1. Clone the amos-ss2021-synthetic-file-system
+1. Clone the amos-ss2021-synthetic-file-system
    ```sh
    git clone https://github.com/amosproj/amos-ss2021-synthetic-file-system.git
    ```
-2. Install NPM packages
-   ```sh
-   npm install
-   ```
-
+2. The cloned repository contains two files needed to set up the docker containers (but this can also be done manually if you can't run them for some reason):
+    Automatic:
+        `.\init.sh   # Builds the docker container and downloads/sets up the metadatahub`
+    Manually:
+      Set up our docker and metadahub-docker:
+      ```sh
+      git clone https://github.com/amos-project2/metadata-hub
+      cd metadata-hub
+      docker pull amosproject2/metadatahub:latest
+      docker volume create --name metadatahub-database -d local
+      ```
+
+      Run the metadatahub container container:
+      ```sh
+      docker run \
+          -p 8080:8080 \
+          -v /home/data:/filesystem  \
+          -v metadatahub-database:/var/lib/postgresql/12/main \
+          amosproject2/metadatahub &>/dev/null & disown;
+      ```
+
+      Run our container:
+      ```sh
+      docker run -it --net="host" --cap-add=SYS_ADMIN --device=/dev/fuse --security-opt apparmor:unconfined --tty fuse_skeleton
+      ```
+
 
 
 <!-- USAGE EXAMPLES -->
 ## Usage
 
+1. The FUSE pulls its information from the running metadatahub webservice. This can be found at http://localhost:8080
+   To fill in some dummy data go to "http://localhost:8080/?p=treewalk-controller" and start an action that parses some directory
+    Notes:
+    * You should set a date way in the past because the container may be running with a different date 
+    * for some reason the treewalker sometimes does not parse directories when running a job (or I've been using it wrong).
+          What always works is just giving it the root directory: "/, True"
+    To verify that some files have been parsed go to http://localhost:8080/?p=graphiql-console and run the following query:
+    ```query{searchForFileMetadata {numberOfTotalFiles}} ```
+    If files were parsed correctly you should get a result with "numberOfTotalFiles" > 0.
+   
+2. Run the container via the ./run.sh script or manually. This should spawn a shell in the docker.
+    WARNING: The way the FUSE is configured it runs in the foreground and blocks the current shell so that we can see debug output. 
+    This obviously means that you can't use this shell to navigate the FUSE. This is why the container runs [tmux](https://github.com/tmux/tmux/wiki) (Terminal Multiplexer) which allows you to have multiple terminals at the same time.
+    To spawn a new terminal in tmux run "ctrl+b %". To navigate between the two terminals use "ctrl+b 'arrow_keys'".
+    An other option would be setting the FUSE to run in the background in `src/main.py:215`
+
+3. To mount the metadatahub as a FUSE run the mount script: `./mount.sh`
+4. Navigate to the directory mounted diractory: `cd /fuse_mount`
+5. Navigate the directory via `ls` and `cd`
+
+
 <!--Use this space to show useful examples of how a project can be used. Additional screenshots, code examples and demos work well in this space. You may also link to more resources.
 
 <!--_For more examples, please refer to the [Documentation](https://example.com)_

docker-compose.yml

@@ -0,0 +1,23 @@
+version: "3.7"
+
+services:
+
+    metadata-hub:
+        container_name: metadata-hub
+        image: amosproject2/metadatahub:latest
+        ports:
+            - 8080:8080
+        volumes:
+            - /home/data:/filesystem
+
+    synthetic-file-system:
+        build:
+            context: .
+            dockerfile: docker/Dockerfile
+        container_name: synthetic-file-system
+        image: fuse_skeleton
+        stdin_open: true
+        tty: true
+        network_mode: "host"
+        depends_on: 
+            - metadata-hub

docker/Dockerfile

@@ -1,18 +1,23 @@
 # syntax=docker/dockerfile:1
 FROM ubuntu:20.04
 
+RUN groupadd --gid 1000 sfs \
+  && useradd --uid 1000 --gid sfs --no-create-home --shell /bin/bash sfs
+
 RUN apt-get update && apt-get install -y \
     libfuse-dev \
     python3-pip \
     zsh \
     wget \
     curl \
+    tmux \
     && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /app
 
 COPY requirements.txt requirements.txt
-COPY ./main.py ./main.py
+COPY mount.sh mount.sh
+COPY ./src/ ./src/
 
 RUN pip3 install -r requirements.txt
 RUN mkdir -p /fuse_mount
@@ -24,6 +29,6 @@ RUN touch /test_root/test.txt /test_root/test2.text /test_root/test_sub_folder/t
 RUN echo "test text" >> /test_root/test.txt
 RUN echo "test2 text" >> /test_root/test_sub_folder/test2.txt
 
-ENTRYPOINT ["zsh"]
+USER sfs
 #["python3"]
 #CMD ["main.py", "/test_root", "/fuse_mount"]

mount.sh

@@ -0,0 +1,3 @@
+cd src
+python3 main.py /test_root /fuse_mount
+cd /fuse_mount

requirements.txt

@@ -1 +1,3 @@
 fusepy
+requests
+anytree

run.sh

@@ -6,4 +6,4 @@ docker run \
     -v metadatahub-database:/var/lib/postgresql/12/main \
     amosproject2/metadatahub &>/dev/null & disown;
 
-docker run -it --net="host" --cap-add=SYS_ADMIN --device=/dev/fuse --security-opt apparmor:unconfined --tty fuse_skeleton
+docker run -it --net="host" --cap-add=SYS_ADMIN  --device=/dev/fuse --security-opt apparmor:unconfined --tty fuse_skeleton

src/fuse_utils.py

@@ -0,0 +1,47 @@
+from anytree import *
+from send_mdh_request import *
+
+
+def build_tree_from_files(files: [File]):
+    """
+    build_tree_from_files Turns a list of FilePaths into a directory tree. The tree is built upon a "Root" node
+    :param files: a list of file paths e.g ["/metadatahub/crawler/test", "/metadatahub/docs/test", ...]
+    :return: The root Node (anytree.Node) of the resulting tree
+    """
+
+    root_node = Node("Root")
+
+    # prepare directory names for easier processing
+    file_paths = []
+    for file in files:
+        full_file_name = file.dir_path + "/" + file.name
+        file_paths.append(full_file_name.split("/"))
+
+    i = 1
+    num_files_to_parse = len(file_paths)
+    while True:
+        for file_path in file_paths:
+            if i >= len(file_path):
+                num_files_to_parse -= 1
+                continue
+            path_part = file_path[i]
+            prev_path = ""
+            for j in range(1, i):
+                prev_path += file_path[j] + "/"
+
+            prev_path = prev_path[:len(prev_path) - 1]  # remove trailing "/"
+
+            has_child = False
+            parent_finder = Resolver("name")
+            parent_node: Node
+            parent_node = parent_finder.get(root_node, prev_path)
+            for child in parent_node.children:
+                if child.name == path_part:
+                    has_child = True
+            if not has_child:
+                Node(path_part, parent_node)
+        if num_files_to_parse <= 0:
+            break
+        i += 1
+
+    return root_node