Sample notebook (#19)

* Sample notebook

Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com>

* Getting started notebook

Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com>

* Bump version

Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com>

* Improved conda build

Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com>

* Improved conda build

Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com>

---------

Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com>

.dockerignore

@@ -42,3 +42,4 @@ flake.lock
 
 .metals/
 
+workspace/

.github/workflows/pr.yml

@@ -50,13 +50,6 @@ jobs:
           python3 -m poetry config virtualenvs.create false
           python3 -m poetry install --no-cache
           poetry run flake8 chenpy --count --select=E9,F63,F7,F82 --show-source --statistics
-      - name: Test conda
-        if: runner.os == 'Linux'
-        run: |          
-          # Test conda build
-          $CONDA/bin/conda update -n base -c defaults conda
-          $CONDA/bin/conda install anaconda-client conda-build
-          $CONDA/bin/conda build --output-folder ./conda-out/ ./conda/
       - uses: actions/cache@v3
         with:
           path: |

.github/workflows/release.yml

@@ -58,7 +58,7 @@ jobs:
         run: |
           poetry publish --build --username $PYPI_USERNAME --password $PYPI_PASSWORD
           $CONDA/bin/conda update -n base -c defaults conda
-          $CONDA/bin/conda build --output-folder ./conda-out/ ./conda/
+          $CONDA/bin/conda build -c conda-forge --output-folder ./conda-out/ chenpy
           anaconda upload --label main -u appthreat ./conda-out/noarch/*.tar.bz2
         env:
           PYPI_USERNAME: ${{ secrets.PYPI_USERNAME }}

.gitignore

@@ -56,3 +56,4 @@ chen.zip
 .vscode/
 project/metals.sbt
 conda-out/
+notebooks/.ipynb_checkpoints/

README.md

@@ -17,18 +17,30 @@ Code Hierarchy Exploration Net (chen) is an advanced exploration toolkit for you
 
 chen container image has everything needed to get started.
 
-### Interactive console
+### Jupyter notebook with docker compose
 
-To start the interactive console, run `chennai` command.
+Use the docker compose from this repo to try chennai with Jupyter notebook.
 
 ```shell
-docker run --rm -v /tmp:/tmp -v $HOME:$HOME -v $(pwd):/app:rw -it ghcr.io/appthreat/chen chennai
+git clone https://github.com/AppThreat/chen
+cd chen
+docker compose up
 ```
 
-### Jupyter notebook server
+- Navigate to the link "http://127.0.0.1:9999/tree?token=chennai"
+- Click notebooks and then `getting-started.ipynb`
+
+Use the controls in Jupyter to interact with the cells. For a preview via github click [here](./notebooks/getting-started.ipynb)
+
+<img src="./docs/_media/chennai-jupyter1.png" alt="Jupyter console" width="512">
+<img src="./docs/_media/chennai-jupyter2.png" alt="Jupyter console" width="512">
+
+### Interactive console
+
+To start the interactive console, run `chennai` command.
 
 ```shell
-docker run --rm -v /tmp:/tmp -v $HOME:$HOME -v $(pwd):/app:rw -it ghcr.io/appthreat/chen jupyter notebook --ip 0.0.0.0 --port 9000 --no-browser --allow-root
+docker run --rm -v /tmp:/tmp -v $HOME:$HOME -v $(pwd):/app:rw -it ghcr.io/appthreat/chen chennai
 ```
 
 ### Chennai server mode
@@ -73,7 +85,7 @@ Once the download finishes, the command will display the download location along
 [21:53:36] INFO     To run chennai console, add the following environment variables to your .zshrc or .bashrc:
 export JAVA_OPTS="-Xmx16G"
 export JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF-8 -Djna.library.path=<lib dir>"
-export SCALAPY_PYTHON_LIBRARY=python3.10
+export SCALAPY_PYTHON_LIBRARY=python3.11
 export CHEN_HOME=/home/user/.local/share/chen
 export PATH=$PATH:/home/user/.local/share/chen/platform:/home/user/.local/share/chen/platform/bin:
 ```
@@ -168,10 +180,11 @@ chennai> help
 1 error found
 ```
 
-This error is mostly due to missing python .so (linux), .dll (windows) or .dylib (mac) file. Ensure the two environment variables below are set correctly.
+This error is mostly due to missing python .so (linux), .dll (windows) or .dylib (mac) file. Ensure the below environment variables below are set correctly.
 
 - SCALAPY_PYTHON_LIBRARY - Use values such as python3.10, python3.11 based on the version installed. On Windows, there are no dots. python311
 - JAVA_TOOL_OPTIONS - jna.library.path must be set to the python lib directory
+- SCALAPY_PYTHON_PROGRAMNAME - Path to Python executable in case of virtual environments (Usually not required)
 
 ## Origin of chen
 

build.sbt

@@ -1,6 +1,6 @@
 name                     := "chen"
 ThisBuild / organization := "io.appthreat"
-ThisBuild / version      := "0.0.18"
+ThisBuild / version      := "0.0.19"
 ThisBuild / scalaVersion := "3.3.1"
 
 val cpgVersion = "1.4.22"

chenpy/cli.py

@@ -183,8 +183,8 @@ def install_py_modules(pack="database"):
 conda install -n chenpy-local -c conda-forge networkx --solver=libmamba -y
 conda install -n chenpy-local -c pytorch pytorch torchtext cpuonly --solver=libmamba -y
 pip install pyg_lib -f https://data.pyg.org/whl/torch-2.1.0+cpu.html
-conda install -n chenpy-local -c conda-forge packageurl-python nbconvert jupyter_core jupyter_client notebook --solver=libmamba -y
-conda install -n chenpy-local -c conda-forge httpx websockets orjson rich appdirs psutil gitpython --solver=libmamba -y"""
+conda install -n chenpy-local -c conda-forge scipy numpy packageurl-python nbconvert jupyter_core jupyter_client notebook --solver=libmamba -y
+conda install -n chenpy-local -c conda-forge oras-py httpx websockets orjson rich appdirs psutil gitpython --solver=libmamba -y"""
             for line in conda_install_script.split("\n"):
                 if line.strip():
                     task = progress.add_task(line, start=False, total=100)

chenpy/client.py

@@ -45,11 +45,11 @@ class Connection:
     Connection object to hold following connections:
        - Websocket to chen server
        - http connection to chen server
-       - http connection to cpggen server
+       - http connection to atomgen server
     """
 
-    def __init__(self, cpggenclient, httpclient, websocket):
-        self.cpggenclient = cpggenclient
+    def __init__(self, atomgenclient, httpclient, websocket):
+        self.atomgenclient = atomgenclient
         self.httpclient = httpclient
         self.websocket = websocket
 
@@ -62,7 +62,7 @@ async def ping(self):
 
     async def close(self):
         """Close all connections"""
-        await self.cpggenclient.close()
+        await self.atomgenclient.close()
         await self.httpclient.close()
         await self.websocket.close()
 
@@ -72,37 +72,37 @@ async def __aexit__(self, exc_type, exc_value, exc_traceback):
 
 def get_sync(
     base_url="http://localhost:9000",
-    cpggen_url="http://localhost:7072",
+    atomgen_url="http://localhost:7072",
     username=None,
     password=None,
 ):
-    """Function to create a plain synchronous http connection to chen and cpggen server"""
+    """Function to create a plain synchronous http connection to chen and atomgen server"""
     auth = None
     if username and password:
         auth = httpx.BasicAuth(username, password)
     base_url = base_url.rstrip("/")
     client = httpx.Client(base_url=base_url, auth=auth, timeout=CLIENT_TIMEOUT)
-    cpggenclient = None
-    if cpggen_url:
-        cpggenclient = httpx.Client(base_url=cpggen_url, timeout=CLIENT_TIMEOUT)
-    return Connection(cpggenclient, client, None)
+    atomgenclient = None
+    if atomgen_url:
+        atomgenclient = httpx.Client(base_url=atomgen_url, timeout=CLIENT_TIMEOUT)
+    return Connection(atomgenclient, client, None)
 
 
 async def get(
-    base_url="http://localhost:9000",
-    cpggen_url="http://localhost:7072",
+    base_url="http://localhost:8080",
+    atomgen_url="http://localhost:7072",
     username=None,
     password=None,
 ):
-    """Function to create a connection to chen and cpggen server"""
+    """Function to create a connection to chen and atomgen server"""
     auth = None
     if username and password:
         auth = httpx.BasicAuth(username, password)
     base_url = base_url.rstrip("/")
     client = httpx.AsyncClient(base_url=base_url, auth=auth, timeout=CLIENT_TIMEOUT)
-    cpggenclient = None
-    if cpggen_url:
-        cpggenclient = httpx.AsyncClient(base_url=cpggen_url, timeout=CLIENT_TIMEOUT)
+    atomgenclient = None
+    if atomgen_url:
+        atomgenclient = httpx.AsyncClient(base_url=atomgen_url, timeout=CLIENT_TIMEOUT)
     ws_url = f"""{base_url.replace("http://", "ws://").replace("https://", "wss://")}/connect"""
     websocket = await websockets.connect(ws_url, ping_interval=None, ping_timeout=None)
     connected_msg = await websocket.recv()
@@ -112,7 +112,7 @@ async def get(
         )
     # Workaround to fix websockets.exceptions.ConnectionClosedError
     await asyncio.sleep(0)
-    return Connection(cpggenclient, client, websocket)
+    return Connection(atomgenclient, client, websocket)
 
 
 async def send(connection, message):
@@ -348,12 +348,12 @@ async def create_cpg(
     auto_build=True,
     skip_sbom=True,
 ):
-    """Create Atom using cpggen server"""
-    client = connection.cpggenclient
+    """Create Atom using atomgen server"""
+    client = connection.atomgenclient
     if not client:
         return {
             "error": "true",
-            "message": "No active connection to cpggen server. Pass the cpggen url to the client.get method.",
+            "message": "No active connection to atomgen server. Pass the atomgen url to the client.get method.",
         }, 500
     # Suppor for url
     url = ""

chenpy/detectors/common.py

@@ -432,3 +432,51 @@ async def export(
         if colorize
         else (res[0] if res and len(res) == 1 else res)
     )
+
+
+async def summary(connection):
+    return await client.q(connection, "summary(true)")
+
+
+async def files(connection, title="Files"):
+    return await client.q(connection, f'files(title="{title}", true)')
+
+
+def bool_to_str(b):
+    return "true" if b else "false"
+
+
+async def methods(connection, title="Methods", include_calls=False, tree=False):
+    return await client.q(
+        connection,
+        f'methods(title="{title}", includeCalls={bool_to_str(include_calls)}, tree={bool_to_str(tree)}, as_text=true)',
+    )
+
+
+async def annotations(connection, title="Annotations"):
+    return await client.q(connection, f'annotations(title="{title}", true)')
+
+
+async def imports(connection, title="Imports"):
+    return await client.q(connection, f'imports(title="{title}", true)')
+
+
+async def declarations(connection, title="Declarations"):
+    return await client.q(connection, f'declarations(title="{title}", as_text=true)')
+
+
+async def sensitive(
+    connection, title="Sensitive", pattern="(secret|password|token|key|admin|root)"
+):
+    return await client.q(
+        connection, f'sensitive(title="{title}", pattern="{pattern}", as_text=true)'
+    )
+
+
+async def show_similar(
+    connection, method_fullname, compare_pattern="", upper_bound=500, timeout=5
+):
+    return await client.q(
+        connection,
+        f'showSimilar(methodFullName="{method_fullname}", comparePattern="{compare_pattern}", upper_bound={upper_bound}, timeout={timeout}, as_text=true)',
+    )

chenpy/meta.yaml

@@ -0,0 +1,79 @@
+{% set version = "0.0.19" %}
+
+package:
+  name: chen
+  version: {{ version }}
+about:
+  home: https://github.com/appthreat/chen
+  license: Apache-2.0
+  license_file: LICENSE
+  summary: "Code Hierarchy Exploration Net (chen)"
+  dev_url: https://github.com/appthreat/chen
+dependencies:
+  - python>=3.8.1,<3.12
+  - conda-forge::httpx>=0.24.1,<0.25.0
+  - conda-forge::websockets>=11.0.2,<12.0.0
+  - conda-forge::orjson>=3.9.0,<4.0.0
+  - conda-forge::rich>=13.4.1,<14.0.0
+  - conda-forge::oras-py>=0.1.25
+  - conda-forge::appdirs>=1.4.4,<2.0.0
+  - conda-forge::psutil>=5.9.5,<6.0.0
+  - conda-forge::packageurl-python>=0.11.2,<0.12.0
+  - conda-forge::gitpython>=3.1.37,<4.0.0
+  - conda-forge::networkx>=3.1
+  - conda-forge::numpy>=1.26
+  - conda-forge::scipy>=1.11.3
+requirements:
+  build:
+    - python
+    - pip
+  host:
+    - pip
+    - python
+    - setuptools
+    - poetry
+  run:
+    - python
+    - httpx
+    - websockets
+    - orjson
+    - rich
+    - oras-py
+    - appdirs
+    - psutil
+    - packageurl-python
+    - gitpython
+    - networkx
+    - numpy
+    - scipy
+
+build:
+  noarch: python
+  script: |
+    cp ${RECIPE_DIR}/../pyproject.toml ${SRC_DIR}/
+    cp ${RECIPE_DIR}/../LICENSE ${SRC_DIR}/
+    cp ${RECIPE_DIR}/../README.md ${SRC_DIR}/
+    cp ${RECIPE_DIR}/../docker-compose.yml ${SRC_DIR}/
+    cp -rf ${RECIPE_DIR}/../docs ${SRC_DIR}/
+    cp -rf ${RECIPE_DIR}/../notebooks ${SRC_DIR}/
+    {{ PYTHON }} -m pip install --no-build-isolation --no-deps --ignore-installed .
+  entry_points:
+    - chen = chenpy.cli:main
+
+source:
+  path: .
+  folder: chenpy
+
+outputs:
+  - name: chen
+    files:
+      - chenpy
+      - docs
+      - notebooks
+      - README.md
+      - LICENSE
+      - docker-compose.yml
+
+extra:
+  maintainers:
+    - Team AppThreat <cloud@appthreat.com>

chenpy/utils.py

@@ -89,6 +89,9 @@ def t(result, title="", caption="", language="javascript"):
 
 
 def print_table(result, title="", caption="", language="javascript"):
+    if isinstance(result, Table):
+        console.print(result)
+        return
     """Function to print the result as a table"""
     table = Table(
         title=title,
@@ -455,11 +458,20 @@ def fix_json(sout):
                     sout = sout.split(': String = "')[-1]
             else:
                 sout = sout.split(': String = "')[-1][-1]
-        elif "tree: ListBuffer" in sout or " = ListBuffer(" in sout:
+        elif (
+            "tree: ListBuffer" in sout
+            or " = ListBuffer(" in sout
+            or ': String = """' in sout
+        ):
             sout = sout.split(": String = ")[-1]
             if '"""' in sout:
                 sout = sout.replace('"""', "")
             return sout
+        elif "me.shadaj.scalapy.py.Dynamic | Unit = " in sout:
+            sout = sout.split("me.shadaj.scalapy.py.Dynamic | Unit = ")[-1]
+            if '"""' in sout:
+                sout = sout.replace('"""', "")
+            return sout
         elif 'String = """[' in sout:
             tmpA = sout.split("\n")[1:-2]
             sout = "[ " + "\n".join(tmpA) + "]"
@@ -496,12 +508,13 @@ def parse_error(serr):
     if "No projects loaded" in serr:
         return """ERROR: Import code using import_code api. Usage: await workspace.import_code(connection, directory_name, app_name)"""
     if "No Atom loaded" in serr:
-        return """ERROR: Import cpg using import_cpg or create_cpg api."""
+        return """ERROR: Import atom using import_atom or import_code api."""
     return serr
 
 
 def read_image(file_path):
-    """Function to read image file safely optionally converting binary formats to base64 string. Useful to render images in notebooks"""
+    """Function to read image file safely optionally converting binary formats to base64 string. Useful to render
+    images in notebooks"""
     if os.path.exists(file_path):
         (mt, encoding) = mimetypes.guess_type(file_path, strict=True)
         if mt.startswith("image/svg"):