From 1f727b705408900a497ddfeb881f9c0df1fd7095 Mon Sep 17 00:00:00 2001 From: varkha-d-sharma <112053040+varkha-d-sharma@users.noreply.github.com> Date: Tue, 6 Feb 2024 08:07:48 +0530 Subject: [PATCH] Docs updation (#132) * Update cmf-server.md * Create ssh-setup.md * Update cmf-server.md * Update cmf-server.md * Update Getting Started with cmf.md * Update ssh-setup.md * Update minio-server.md * Update cmf-server.md * Update cmf-server.md * Update ssh-setup.md * Update ssh-setup.md --------- Co-authored-by: abhinavchobey <111754147+abhinavchobey@users.noreply.github.com> --- docs/cmf_client/Getting Started with cmf.md | 2 +- docs/cmf_client/minio-server.md | 8 +-- docs/cmf_client/ssh-setup.md | 25 ++++++++ docs/cmf_server/cmf-server.md | 63 +++++++++++++++++---- 4 files changed, 82 insertions(+), 16 deletions(-) create mode 100644 docs/cmf_client/ssh-setup.md diff --git a/docs/cmf_client/Getting Started with cmf.md b/docs/cmf_client/Getting Started with cmf.md index 0801ca51..40d73a69 100644 --- a/docs/cmf_client/Getting Started with cmf.md +++ b/docs/cmf_client/Getting Started with cmf.md @@ -3,7 +3,7 @@ Common metadata framework (cmf) has the following components: - **Metadata Library** exposes API’s to track the pipeline metadata. It also provides API’s to query the stored metadata. - **cmf-client** interacts with the server to pull or push metadata from or to the cmf-server. -- **cmf-server** interacts with all the remote clients and is responsible to merge the metadata transferred by the cmf-client and manage the consolidated metadata. +- **cmf-server with GUI** interacts with all the remote clients and is responsible to merge the metadata transferred by the cmf-client and manage the consolidated metadata. GUI renders metadata for simplified tracking. - **Central Artifact Repositories** hosts the code and data. ## Setup a cmf-client diff --git a/docs/cmf_client/minio-server.md b/docs/cmf_client/minio-server.md index bd38776a..e0c9269d 100644 --- a/docs/cmf_client/minio-server.md +++ b/docs/cmf_client/minio-server.md @@ -35,16 +35,16 @@ Follow the below mentioned steps to set up a MinIO server: 5. Build a MinIO server using a Docker container. `docker-compose.yml` available in `example-get-started` directory provides two services: `minio` and `aws-cli`. User will initialise the repository with bucket name, storage URL, and credentials to access MinIO. -6. Execute the following command to start the docker container. Following command requires root privileges. +6. Execute the following command to start the docker container. MYIP variable is the IP address of the machine on which you are executing the following command. Following command requires root privileges. ``` - docker-compose up + MYIP= XX.XX.XXX.XXX docker-compose up ``` or ``` - docker compose up + MYIP= XX.XX.XXX.XXX docker compose up ``` After executing the above command, following messages confirm that MinIO is up and running. - + > Also you can adjust `$MYIP` in `examples/example-get-started/docker-compose.yml` to reflect the server IP and run the `docker compose` command without specifying 7. Login into `remote.minio.endpointurl` (in the above example - http://localhost:9000) using access-key and secret-key mentioned in cmf configuration. 8. Following image is an example snapshot of the MinIO server with bucket named 'dvc-art'. diff --git a/docs/cmf_client/ssh-setup.md b/docs/cmf_client/ssh-setup.md new file mode 100644 index 00000000..52165d43 --- /dev/null +++ b/docs/cmf_client/ssh-setup.md @@ -0,0 +1,25 @@ +# SSH Remote Artifact Repo Setup +## Steps to set up a SSH Remote Repo +SSH (Secure Shell) remote storage refers to using the SSH protocol to securely access and manage files and data on a remote server or storage system over a network. SSH is a cryptographic network protocol that allows secure communication and data transfer between a local computer and a remote server. + +Proceed with the following steps to set up a SSH Remote Repository: + +1. Get started with initializing `project directory` with SSH repo. +2. Check whether cmf is initialized in your project directory with following command. + ``` + cmf init show + ``` + If cmf is not initialized, the following message will appear on the screen. + ``` + 'cmf' is not configured. + Execute the 'cmf init' command. + ``` + +3. Execute the following command to initialize the SSH remote storage as a CMF artifact repository. + ``` + cmf init sshremote --path ssh://127.0.0.1/home/user/ssh-storage --user XXXXX --port 22 --password example@123 --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://127.0.0.1:80 + ``` + > When running `cmf init sshremote`, please ensure that the specified IP address has the necessary permissions to allow access using the specified user ('XXXX'). If the IP address or user lacks the required permissions, the command will fail. + +4. Execute `cmf init show` to check the CMF configuration. +5. To address permissions issues related to SSH, it is recommended to initiate troubleshooting by examining the `/etc/ssh/sshd_config file`. This configuration file serves as the primary starting point for diagnosing and resolving SSH permission-related challenges. diff --git a/docs/cmf_server/cmf-server.md b/docs/cmf_server/cmf-server.md index 001d9949..9bf2f46b 100644 --- a/docs/cmf_server/cmf-server.md +++ b/docs/cmf_server/cmf-server.md @@ -8,19 +8,22 @@ They accept and return JSON-encoded request bodies and responses and return stan ### List of APIs -| Method | URL | Description | -|--------|------------------------------|----------------------------------------------| -| `Post` | `/mlmd_push` | Used to push Json Encoded data to cmf-server | -| `Get` | `/mlmd_pull/{pipeline_name}` | Retrieves a mlmd file from cmf-server | -| `Get` | `/display_executions` | Retrieves all executions from cmf-server | -| `Get` | `/display_artifacts` | Retrieves all artifacts from cmf-server | +| Method | URL | Description | +|--------|------------------------------|------------------------------------------------------------------------ | +| `Post` | `/mlmd_push` | Used to push Json Encoded data to cmf-server | +| `Get` | `/mlmd_pull/{pipeline_name}` | Retrieves a mlmd file from cmf-server | +| `Get` | `/display_executions` | Retrieves all executions from cmf-server | +| `Get` | `/display_artifacts/{pipeline_name}/{data_type}` | Retrieves all artifacts from cmf-server for resp datat type | +| `Get` | `/display_lineage/{lineage_type}/{pipeline_name}` | Creates lineage data from cmf-server | +| `Get` | `/display_pipelines` | Retrieves all pipelines present in mlmd file | + ### HTTP Response Status codes | Code | Title | Description | |-------| ------------------------- |--------------------------------------------------------------| | `200` | `OK` | mlmd is successfully pushed (e.g. when using `GET`, `POST`). | -| `400` | `Bad request` | When the cmf[env](cmf%2Fenv)-server is not available. | +| `400` | `Bad request` | When the cmf-server is not available. | | `500` | `Internal server error` | When an internal error has happened | @@ -34,6 +37,7 @@ There are two ways to start cmf server - 1. Install [Docker Engine](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) with [non root user](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) privileges. 2. Install [Docker Compose Plugin](https://docs.docker.com/compose/install/linux/). > In earlier versions of docker compose, `docker compose` was independent of docker. Hence, `docker-compose` was command. However, after introduction of Docker Compose Desktop V2, compose command become part of docker engine. The recommended way to install docker compose is installing a docker compose plugin on docker engine. For more information - [Docker Compose Reference](https://docs.docker.com/compose/reference/). +3. **Docker Proxy Settings** are needed for some of the server packages. Refer to the official Docker documentation for comprehensive instructions: [Configure the Docker Client for Proxy](https://docs.docker.com/network/proxy/#configure-the-docker-client). ## Using `docker compose` file > This is the recommended way as docker compose starts both ui-server and cmf-server in one go. @@ -51,7 +55,8 @@ There are two ways to start cmf server - server: image: server:latest volumes: - - /home//cmf-server/data:/cmf-server/data + - /home/xxxx/cmf-server/data:/cmf-server/data + - /home/xxxx/cmf-server/data/static:/cmf-server/data/static container_name: cmf-server build: .... @@ -79,6 +84,7 @@ There are two ways to start cmf server - ``` docker compose -f docker-compose-server.yml stop ``` +> It is neccessary to rebuild images for cmf-server and ui-server after `cmf version update` or after pulling latest cmf code from git. ## Using `docker run` command @@ -113,18 +119,53 @@ There are two ways to start cmf server - ``` docker run --name mycontainer -p 0.0.0.0:8080:80 -v /home/user/cmf-server/data/static:/cmf-server/data/static myimage ``` +6. After cmf-server container is up, start `ui-server`, Go to `cmf/ui` folder. + ``` + cd /cmf/ui + ``` +7. Execute the below-mentioned command to create a `ui-server` docker image. + ``` + Usage: docker build -t [image_name] -f ./Dockerfile ../ + ``` + Example: + ``` + docker build -t uiimage -f ./Dockerfile ../ + ``` +8. Launch a new docker container using the image with directory + 
+   Usage: docker run --name [container_name] -p 0.0.0.0:3000:80 [image_name]
+
+ Example: + ``` + docker run --name mycontainer -p 0.0.0.0:3000:80 uiimage + ``` + `Note` - If you face issue regarding `Libzbar-dev` as follows you need add proxy: + ![Screenshot (115)](https://github.com/varkha-d-sharma/cmf/assets/111754147/9830cbe9-bad8-404a-8abe-5470fc2303c4) + -6. To stop the docker container. + ``` + ~/.docker/config.json + { + proxies: { + "default": { + "httpProxy": "http://web-proxy.labs.xxxx.net:8080", + "httpsProxy": "http://web-proxy.labs.xxxx.net:8080", + "noProxy": ".labs.xxxx.net,127.0.0.0/8" + } + } + } + ``` +10. To stop the docker container. ``` docker stop [container_name] ``` -7. To delete the docker container. +11. To delete the docker container. ``` docker rm [container_name] ``` -8. To remove the docker image. +11. To remove the docker image. ``` docker image rm [image_name] ```