diff --git a/README.md b/README.md
index 627fadb0..172eb379 100644
--- a/README.md
+++ b/README.md
@@ -1,234 +1,431 @@
![](docs/screenshot.png)
-# DiscordChatExporter-frontend (DCEF)
+# DiscordChatExporter-frontend (DCEF / DCE-Frontend)
+
+View and search your [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) exports in a Discord like user interface. [Try a demo in your browser :)](https://dcef.slada.sk/)
-Browse your [Discord chat exports](https://github.com/Tyrrrz/DiscordChatExporter) in a single discord like user interface. [Try a demo in your browser :)](https://dcef.slada.sk/).
## Features
-- browse multiple servers, channels, threads and forum posts in a single web based user interface
-- search messages in a server with autocomplete
-- optimized to handle really large exports well
-- customizability - change font and many other things in settings
-- self hosting option - you can host DCEF on your own server and share it with your friends. Web interface is mobile friendly too
+- Familiar Discord-like interface
+- Search with autocomplete
+- Optimized to handle really large exports well
+- You own the data - use locally or self-host on a server
+- Runs on Linux or Windows
+
+> Want to chat? Join [Tyrrrz's discord server](https://discord.gg/2SUWKFnHSm). I usually hang out in `#dce-frontend` channel.
+
+NOTE: The exporter [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) is maintained by different maintainer than this viewer.
+
+
+## How to use
+
+### 1. Check prerequisites
+
+Windows:
+- Windows 10 or newer
+- x64 architecture
+- CPU with AVX support
+- Available (and not excluded) ports `21011`, `27017` and `58000`
+
+Linux:
+- x64 or arm64 architecture
+- Docker or Podman installed
+- CPU with AVX support
+
+Mac:
+- Not supported (yet)
+
+
+### 2. Export your data from Discord
+
+JSON exports are downloaded using [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter).
+
+I am using GUI version of DiscordChatExporter
+
+1. Download the latest GUI version of [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/releases) - most commonly used is `DiscordChatExporter.win-x64.zip` for 64-bit Windows
+
+2. If you want to export threads and forum posts, set `Show threads` option to `All` in the general settings of DiscordChatExporter
+
+![](docs/dce-show-threads.png)
+
+3. Obtain a Discord token by following [this guide](https://github.com/Tyrrrz/DiscordChatExporter/blob/master/.docs/Token-and-IDs.md)
+
+4. Enter your Discord token to DiscordChatExporter and press Enter
+
+5. If you have enabled `Show threads` option, load times are longer than expected, it will eventually load :)
+
+6. Select the channels you want to export (or press CTRL+A to select all channels)
+
+7. **Make sure that export format is set to `JSON` and `Format markdown` is disabled. Enable `Download assets`+`Reuse assets` option to download images, videos and other types of assets.**
+
+![](docs/dce-export-more.png)
+
+
+
+I am using CLI version of DiscordChatExporter
+
+1. Download the latest CLI version of [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/releases) - most commonly used is `DiscordChatExporter.Cli.win-x64.zip` for 64-bit Windows
+2. Extract it.
+3. Open a terminal in the extracted folder by pressing ALT+D to highlight the path in the file explorer, then type type `cmd` and press Enter to open a terminal in that folder
+3. Verify that DiscordChatExporter is working by running `DiscordChatExporter.Cli.exe --help`
+
+
+Export all accessible channels, threads and forum posts in a server:
+
+```bash
+DiscordChatExporter.Cli.exe exportguild --token DISCORD_TOKEN -g SERVER_ID --media --reuse-media --markdown false --format Json --include-threads All --output OUTPUT_FOLDER_PATH
+```
+
+
+Export private messages:
+
+```bash
+DiscordChatExporter.Cli.exe exportdm --token DISCORD_TOKEN --media --reuse-media --markdown false --format Json --output OUTPUT_FOLDER_PATH
+```
+
+> TL;DR - always use `--media --reuse-media --markdown false --format Json` command line options for your exports to work with DCEF.
+
+
+TIP: if you use Linux and want to automatically export your servers or DMs incrementally, you can try my [slatinsky/DiscordChatExporter-incrementalBackup](https://github.com/slatinsky/DiscordChatExporter-incrementalBackup) wrapper. You declare what you want to export in a config file and the wrapper will automatically export new messages from servers and DMs.
+
+
+
+
+### 3. Install DCEF
+
-> Want to chat? Join [Tyrrrz's discord server](https://discord.gg/2SUWKFnHSm). I usually hang out in `#dce-frontend` channel
+I am using Windows
-## Quick start (Windows)
+Only x64 architecture is supported. Won't work on Windows 7. Tested on Windows 10.
-1. Export your data from Discord using [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/tree/master/.docs). Your export should be in JSON format, disable `format markdown` option and enable `download assets` and `reuse assets` options. If exporting using a CLI - use `--media --reuse-media --markdown false --format Json` command line options. Include threads and forum posts by adding `--include-threads All` option. More information in [How to export data from Discord to view it in DCEF?](#supported-exports)
-2. Download the latest release from [releases page](https://github.com/slatinsky/DiscordChatExporter-frontend/releases)
+1. Download the latest release from [releases page](https://github.com/slatinsky/DiscordChatExporter-frontend/releases).
3. Extract the archive
-4. Move all your [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) exports to `/exports/` folder ([supported exports](#supported-exports)).
-5. Run `dcef.exe`
+4. Move your exports downloaded in the first chapter to `exports/` folder.
+5. Apply a registry tweak located in `registry_tweaks/change_260_character_path_limit_to_32767.reg` and restart your computer. This tweak will increase the path limit to 32767 characters.
+
+BUG: images and other assets won't load if the path to the assets [exceeds 260 characters](https://trac.nginx.org/nginx/ticket/1598) even with the registry tweak applied. FIX - Place your DCEF folder to a shorter path.
+
+BETA BUILDS: Want to try new features? [Beta builds](https://github.com/slatinsky/DiscordChatExporter-frontend/actions/workflows/windows-build.yml). Beta builds are automatically built from the latest commit on `main` branch. (github account is needed to download them)
+
+
+
-## Quick start (Linux amd64 / arm64)
-Docker version is the best way to [host the viewer on a server](docs/Server-hosting.md) for others to use. Works also on raspberry pi 5 (thanks [ritiek](https://github.com/slatinsky/DiscordChatExporter-frontend/pull/54) for pull request). To run on raspberry pi 4b, you need to build your own image using provided `Dockerfile.rpi4b`.
+I am using Linux
-1. Export your data using [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/blob/master/.docs/Getting-started.md#using-the-cli). Use `--media --reuse-media --markdown false --format Json` command line options.
+Both `amd64` and `arm64` architectures are supported. Also your CPU is required to support AVX instructions (some VPS providers don't support it). Works for Raspberry Pi 5, but not on Raspberry Pi 4b (does not support AVX).
-2. Pull the image from docker hub
+If you use `podman` instead of `docker`, replace `docker` with `podman` in the commands.
+
+1. Pull the image from [docker hub](https://hub.docker.com/r/slada/dcef)
```bash
docker pull slada/dcef:main
```
-3. Navigate to a folder with your exports
+2. Navigate to a folder with your exports downloaded in the first chapter
```bash
cd /path/to/your/exports
```
-4. Run the container
+3. Run the container. This command will mount your current working directory to exports folder inside the container
+
```bash
-docker run --restart=always --volume "$(pwd):/dcef/exports" --volume dcef_cache:/dcef/cache --name dcef -p 21011:21011 -it slada/dcef:main
+docker run --restart=always --volume "$(pwd):/dcef/exports" --volume dcef_cache:/dcef/cache --name dcef -p 127.0.0.1:21011:21011 -it slada/dcef:main
```
-5. Open `http://127.0.0.1:21011/` in your browser
-
-**Note:** Apple M1 is not supported. Pull requests are welcome :)
+The app will autostart with your computer if you are using `docker`. You need to create a systemd service to autostart the app if you use `podman`.
-## Edit server configuration
-
-If you want to select which discord servers are shown in the viewer or which users are hidden, you need to edit server configuration. Changes in this configuration are applied immediately - you don't need to restart DCEF. This configuration is enforced server-side on all users of the viewer.
+
-### Windows
-1. run `dcef.exe` and wait for the user interface to appear
-2. run `configurator.exe`
-3. follow the instructions. Select actions by typing number and pressing enter.
+[for developers] I am to run the development version directly from the source code (Windows)
-### Linux
+Install python3 and nodejs. Make sure that your `node --version` is at least `v18.17.1` and your `py --version` is at least `Python 3.11.0`. Make sure [Windows Terminal](https://github.com/microsoft/terminal) is installed (`wt --version`). Windows terminal is installed by default on Windows 11.
+Install `nodemon` globally, which is used to automatically reload programs on change, by running `npm install -g nodemon`. Verify that it is installed correctly by running `nodemon --version`.
-`dcef` container needs to be running. Then run:
+Then place your exports into `src/exports/` folder. Create that folder if it doesn't exist.
+Then run in `cmd` the batch script `src/RUN_DEV.bat`:
```bash
-docker exec -it dcef /usr/bin/python3.11 /dcef/configurator.py
+cd src
+RUN_DEV.bat
```
-## Other ways to run DCEF
+This script will install Python dependencies in virtual environment (stored in `src/_temp` folder) and will install frontend nodejs dependencies into `src/dcef/frontend/node_modules/`. Then it will open windows terminal with split panes with all the services needed to develop the app.
+
+- frontend - automatically reloads on change. If you add a new dependency into `src/frontend/package.json`, close the watcher, run `npm install` and rerun `RUN_DEV.bat`
+- fastapi - automatically reloads on change. Delete `src/_temp/fastapi/venv/` folder if you change a dependencies. Then rerun `RUN_DEV.bat`.
+- nginx - needs to be restarted each time a config is updated - `taskkill /f /im dcefnginx.exe` and rerun `RUN_DEV.bat` again
+- mongodb - if you need to clear the database, `taskkill /f /im dcefmongod.exe`, delete `src/_temp/mongodb/` folder and rerun `RUN_DEV.bat` again (preprocess will recreate the database from the scratch)
+- preprocess - automatically reloads on change using nodemon. Delete `src/_temp/preprocess/venv/` folder if you change a dependencies. Then rerun `RUN_DEV.bat`.
-Windows beta builds
+
+
+[for developers] I want to compile my own Windows binaries from the source code
+
+There are two ways to compile the Windows binaries:
-If you want to try out the latest features, you can use [beta builds](https://github.com/slatinsky/DiscordChatExporter-frontend/actions/workflows/windows-build.yml). They are automatically built from the latest commit on `main` branch. You need to be logged in to Github account to download them.
+1. Fork the project and run the github actions (located in `.github/workflows/windows-build.yml`) yourself. Then download the built binaries from the Github website.
+2. Or follow the steps in previous chapter about setting up development environment. Then run `BUILD_RELEASE.bat` script located in `src/` directory. This script will compile the binaries and place them into `release/` folder. Test the binaries by running `release/dcef.exe`
+
+Also you can download build for each commit into main branch from [my github actions](https://github.com/slatinsky/DiscordChatExporter-frontend/actions/workflows/windows-build.yml). You need to be logged-in to Github account to download them
-Build docker image from source code on x86_64 architecture
+[for developers] I want to build and run a docker image from the source code
+
+Prerequisites: You need docker and git installed.
-You need docker and git installed. Then run:
+Clone the source code:
```bash
git clone https://github.com/slatinsky/DiscordChatExporter-frontend
cd DiscordChatExporter-frontend
+```
+
+Pull new changes from the git repository:
+
+```bash
+git pull
+docker rm dcef --force
docker build -t dcef .
```
-Then use the same instructions as for the Linux docker version, but replace in commands `slada/dcef:main` with `dcef`.
+
+
+Run it:
+
+```bash
+cd release/exports/
+docker run --restart=always --volume "$(pwd):/dcef/exports" --volume dcef_cache:/dcef/cache --name dcef -p 127.0.0.1:21011:21011 -d dcef
+cd ../..
+```
+
+
+TIP: In case you encounter any problems with outdated exports, you can try to rebuild the database by deleting `dcef_cache` volume and then rerunning the container:
+
+```bash
+docker volume rm dcef_cache
+```
+
+
+
+
+
+### 4. Run DCEF
+
+I am using Windows
+
+1. Run `dcef.exe`
+2. Wait for the program to process your exports. This will take several minutes
+3. After preprocessing is finished, a window with graphical interface will appear. The same interface will also be available in your browser at [127.0.0.1:21011](http://127.0.0.1:21011/)
+
+
+
+
+I am using Linux
+
+
+Open [127.0.0.1:21011](http://127.0.0.1:21011/) in your browser to see graphical interface. It may take a while for all your exports to be visible. Check `docker logs dcef` to see the progress of the preprocessing.
+
+TIP: change `-p 127.0.0.1:21011:21011` to `-p 21011:21011` in `docker run` command to make the viewer available from other devices in your network.
-Build docker image from source code on Raspberry pi 4b
-You need docker and git installed. Then run:
+### 5. Secure DCEF
+
+I am using Linux
+
+
+The client-server architecture allows you to host the viewer on public servers. It is recommended to put the server behind another reverse proxy, such as nginx, that will handle SSL termination and authentication.
+
+**Restrict open ports using firewall:**
+
+Create firewall rules to open only TCP ports 22 (SSH) and 80 (HTTP) and enable firewall. If you want to use SSL, open port 443 (HTTPS) too.
```bash
-git clone https://github.com/slatinsky/DiscordChatExporter-frontend
-cd DiscordChatExporter-frontend
-docker build -t dcef -f Dockerfile.rpi4b
+ufw allow 22/tcp
+ufw allow 80/tcp
+ufw allow 443/tcp
+ufw enable
+```
+
+
+**Check if DCEF port is bound only to loopback interface (localhost `127.0.0.1`):**
+
+```bash
+docker run ... -p 127.0.0.1:21011:21011 ...
+```
+
+**Secure the viewer with basic authentication:**
+
+Create `.htpasswd` file:
+```bash
+sudo apt install apache2-utils
+htpasswd -c /etc/nginx/.htpasswd
+
+```
+
+**Change nginx site config to require authentication** (add lines `auth_basic` and `auth_basic_user_file`):
+
+```conf
+server {
+ listen 80;
+
+ location / {
+ proxy_pass http://localhost:21011/;
+ auth_basic "Auth only";
+ auth_basic_user_file /etc/nginx/.htpasswd;
+ }
+}
+```
+
+**Restart nginx:**
+
+```bash
+sudo systemctl restart nginx
```
-Then use the same instructions as for the Linux docker version, but replace in commands `slada/dcef:main` with `dcef`.
+
+
+NOTE: this is just a basic configuration and your password **is sent in plaintext**. If you own a domain, add SSL to encrypt the connection, use port 80 just to redirect to port 443. You can use free certificates from [Let's Encrypt](https://letsencrypt.org/) by configuring [Certbot](https://certbot.eff.org/) to automate the process. Find a more detailed guide on the internet.
-## Upgrade guide
-Upgrade windows binary release
-Want to upgrade from previous version? Follow these steps:
+## Upgrade DCEF to a new version
-1. Download the latest release from [releases page](https://github.com/slatinsky/DiscordChatExporter-frontend/releases).
-2. Delete everything (except `exports` folder) in your discordchatexporter-frontend folder.
-3. Move everything (except `exports` folder) from the new release to your discordchatexporter-frontend folder.
+I am using Windows
+
+1. Download the latest release zip from [releases page](https://github.com/slatinsky/DiscordChatExporter-frontend/releases) - name of the zip should be `DiscordChatExporter-frontend-vX.Y.Z-win.zip` where `X.Y.Z` is the version number.
+2. Delete everything (except `exports/` folder) in your DiscordChatExporter-frontend folder.
+3. Move everything (except `exports/` folder) from the new release zip to your DiscordChatExporter-frontend folder.
+
+Tip: some upgrades don't require the database to be rebuilt. If you want to speed up the upgrade process, don't delete `_temp/mongodb/` folder. If the upgrade requires the database to be rebuilt, it will be handled automatically.
-Upgrade docker image
+I am using Linux
+
+Pull the new image from the [docker hub](https://hub.docker.com/r/slada/dcef) and restart the container.
```bash
cd path/to/your/exports/
docker rm dcef --force
docker image rm slada/dcef:main
docker pull slada/dcef:main
-docker run --restart=always --volume "$(pwd):/dcef/exports" --volume dcef_cache:/dcef/cache --rm --name dcef -p 21011:21011 -it slada/dcef:main
+docker run --restart=always --volume "$(pwd):/dcef/exports" --volume dcef_cache:/dcef/cache --rm --name dcef -p 127.0.0.1:21011:21011 -it slada/dcef:main
```
-## Uninstall
-Windows (binary release)
+## Downgrade DCEF to an older version
-DCEF does not create any files outside of its folder, so you can just delete the folder to uninstall it.
+I am using Windows
-Move your `exports` folder somewhere else if you want to keep your exports.
+1. Download older release zip from [releases page](https://github.com/slatinsky/DiscordChatExporter-frontend/releases) - name of the zip should be `DiscordChatExporter-frontend-vX.Y.Z-win.zip` where `X.Y.Z` is the version number.
+2. Delete everything (except `exports/` folder) in your DiscordChatExporter-frontend folder.
+3. **Delete `_temp/` folder**, which contains the database and other temporary files. This will force DCEF to rebuild the database from scratch and fill fix issues with the possibly incompatible database format.
+4. Move everything (except `exports/` folder) from the release zip to your DiscordChatExporter-frontend folder.
-Linux (docker)
+I am using Linux
+
+[Docker hub](https://hub.docker.com/r/slada/dcef) contains only the latest image built from the latest commit in main branch. if you want to use older version, you need to build the image from the source code.
-1. kill and delete the container
+Clone the git repository:
```bash
-docker rm dcef --force
+git clone https://github.com/slatinsky/DiscordChatExporter-frontend
+cd DiscordChatExporter-frontend
```
-2. remove the volume
+Switch to the commit you want to build:
```bash
-docker volume rm dcef_cache
+git checkout COMMIT_HASH
```
-3. remove the image
+Remove the old image and the old database (`dcef_cache` volume):
```bash
-docker image rm slada/dcef:main
+docker rm dcef --force
+docker volume rm dcef_cache
```
-
+Build the image:
+```bash
+docker build -t dcef .
+```
+Run the image:
-
-## How to export data from Discord to view it in DCEF?
+```bash
+cd path/to/your/exports/
+docker run --restart=always --volume "$(pwd):/dcef/exports" --volume dcef_cache:/dcef/cache --name dcef -p 127.0.0.1:21011:21011 -d dcef
+```
-JSON exports are created using [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter).
+
-> - JSON export format (`--format Json`) is required
-> - Skipping markdown prerendering (`--markdown false`) is highly recommended, but not required
-> - Downloading assets (`--media --reuse-media`) is highly recommended, but not required
-> - Including threads (`--include-threads All`) is recommended for a full server export
+## Uninstall DCEF
-Incremental export using DiscordChatExporter-incrementalBackup wrapper
+I am using Windows
-I have created simple wrapper for DiscordChatExporter to simplify the process of incremental export for this frontend viewer. You declare what you want to export in a config file and the wrapper will automatically export only new messages since the last export.
+This is a portable app. Just delete the folder with the app. But keep your exports from the `exports/` folder :).
-For more information and instructions how to use, see [slatinsky/DiscordChatExporter-incrementalBackup](https://github.com/slatinsky/DiscordChatExporter-incrementalBackup)
+Optionally also revert the registry tweak to decrease the path limit back to 260 characters. Run `registry_tweaks/restore_260_character_path_limit.reg` and restart your computer. But this is really not necessary.
-Full export using GUI version of DiscordChatExporter
-
-Make sure that export format is set to `JSON` and `Format markdown` is disabled. Optionally, you should also enable `Download assets`+`Reuse assets` option to download images, videos and other types of assets.
-
-Recently `Show threads` option was added to the general settings of DiscordChatExporter - if you want to export threads. Note that fetching threads is slow and it may take a while to show them in the GUI.
-
-![](docs/dce-export-more.png)
-
-Note: Exporting using CLI version of DiscordChatExporter is actually easier if you want to mass export everything.
-
-
+I am using Linux
-Full server export using CLI version of DiscordChatExporter
+1. Kill and delete the container
-Export all accessible channels, threads and forum posts in a server:
```bash
-DiscordChatExporter.Cli.exe exportguild --token DISCORD_TOKEN -g SERVER_ID --media --reuse-media --markdown false --format Json --include-threads All --output OUTPUT_FOLDER_PATH
+docker rm dcef --force
```
+2. Remove the volume
-
+```bash
+docker volume rm dcef_cache
+```
-Export private messages using CLI version of DiscordChatExporter
+3. Remove the image
-Export all dms (sadly, exporting dms can't be done without selfboting):
```bash
-DiscordChatExporter.Cli.exe exportdm --token DISCORD_TOKEN --media --reuse-media --markdown false --format Json --output OUTPUT_FOLDER_PATH
+docker image rm slada/dcef:main
```
+## Troubleshooting
+I am using Windows
-## FAQ
-
-No servers show up / troubleshooting steps
+Check the logs See logs `logs/dcef.log`.
-See logs (Windows `dcef/logs.txt`, Linux `docker logs dcef`) for more info.
+Please check, if the top of the log contains line `windows-runner: OK: All required ports are available.`. DCEF needs ports `21011`, `27017`, `58000` to be available. If you have any of these ports occupied, DCEF won't start or won't work properly.
-[Windows only] Please check, if the top of the log contains line `windows-runner: OK: All required ports are available.`. DCEF needs ports `21011`, `21013`, `27017`, `58000` to be available. If you have any of these ports occupied, DCEF won't start or won't work properly.
+Check if one of those ports is not excluded by running `netsh int ipv4 show excludedportrange protocol=tcp`
-[All platforms] Find line `found X json channel exports` - if this number is 0, you don't have any valid exports in `/exports/` folder.
+Find line `found X json channel exports` - if this number is 0, you don't have any valid exports in `/exports/` folder. Did you export in `json` format instead of the default `html` format? Did you apply the registry tweak and restarted your computer?
-[All platforms] Find if there is line `preprocess done` (at the end of lines `processing `) - if you see this line, preprocessing step finished successfully. If you don't see this line, please check if there is any stacktrace in the logs. Stacktrace looks like this:
+Find if there is line `preprocess done` (at the end of lines `processing `) - if you see this line, preprocessing step finished successfully. If you don't see this line, please check if there is any stacktrace in the logs. Stacktrace looks like this:
```
Traceback (most recent call last):
@@ -241,30 +438,55 @@ Exception: Example stack trace exception
Please [report this issue](https://github.com/slatinsky/DiscordChatExporter-frontend/issues/new) with the stacktrace attached.
-[Windows] If no other solution works for you, run Docker version of DCEF.
+The viewer is tested against standard install of Windows 10, so there may be issues if you have some non-standard Windows setup. [advanced users] If no other solution works for you, run Docker version of DCEF inside WSL2 and follow linux instructions.
+
+
+
+
+I am using Linux
+
+Check the logs
+
+```bash
+docker logs dcef
+```
+
+
+Find line `found X json channel exports` - if this number is 0, you don't have any valid exports in `/dcef/exports/` folder (inside the container). Did you mount the folder correctly? To debug this issue, get a shell inside the container and list the folder
+
+```bash
+docker exec -it dcef bash
+cd /dcef/exports/
+ls
+```
+
+Export should be visible in that `/dcef/exports/` folder
-Some assets won't diplay on Windows / Windows path length limit
-Files in `/exports/` folder may exceed Windows path length limit of 260 characters. If you have any issues with loading your assets you can choose one of the following solutions:
-- move DCEF to a folder with shorter path
-- or run `registry_tweaks/change_260_character_path_limit_to_32767.reg` to increase the limit to 32767 characters (requires admin privileges) and restart your computer. To revert this change, run `registry_tweaks/restore_260_character_path_limit.reg` and restart your computer.
+## FAQ
+
+Some images and other assets won't display on Windows
+
+ISSUE: Files in `/exports/` folder may exceed Windows path length limit of 260 characters. [An issue with nginx server](https://trac.nginx.org/nginx/ticket/1598) is preventing assets from being served correctly.
+
+FIX: Move DCEF to a folder with shorter path, e.g. `C:\Users\User\Downloads\DiscordChatExporter-frontend-vX.Y.Z-win\` -> `C:\DCEF\`
-Assets won't display after moving them / how to clear cache
+Messages are out of sync with the exports folder
-After you put your export to `/exports/` folder, don't remove them. DCEF keeps track of assets and if you remove or move them, they won't show up in DCEF, because the old path would become invalid.
+After you put your export to `exports/` folder, don't remove and don't move them. DCEF keeps track of files and other assets in this folder and if you remove or move them, they won't show up correctly in DCEF, because the old paths would become invalid.
-**TL;DR - only adding new files to `/exports/` folder is supported. If you want to remove or move files, you need to clear cache afterwards.**
+**TL;DR - only adding new files to `exports/` folder is supported. If you want to remove or move files, you need to clear the database afterwards.**
-Clearing cache on Windows:
+Clearing database on Windows:
- close DCEF
-- delete `dcef/backend/mongodb/db` folder
+- delete `_temp/mongodb/` folder
- start DCEF again
-Clearing cache on Linux:
+Clearing database on Linux:
- kill DCEF container
- remove `dcef_cache` volume (`docker volume rm dcef_cache`)
- start DCEF container again
@@ -272,20 +494,18 @@ Clearing cache on Linux:
-DCEF hangs and prints `Slow SessionWorkflow loop` to the console / long preprocessing time
+Long preprocessing time
-`Slow SessionWorkflow loop` messages are completely normal - if you see them, you know that data is pushed to mongodb database and the process is not stuck. Just be patient and wait for the process to finish. If you have a lot of exports, it may take a while.
+DCEF is not just an simple viewer. It needs to preprocess the data into a database, so you can search through the messages. This process can take a long time for large exports. The next time you run DCEF, the startup will be fast (it will only process new exports).
-DCEF is not just an simple viewer. This process enriches your exports with additional data and stores them in a database for search and other features to work.
-
-Impatient? Navigate to `http://127.0.0.1:21011/` in your browser to see already processed exports.
+Impatient? Navigate to `http://127.0.0.1:21011/` in your browser to see already processed exports :)
-DCEF won't run on M1 mac
+DCEF won't run on Arm mac
-[This pull request](https://github.com/slatinsky/DiscordChatExporter-frontend/pull/30) may help you
+Mac is not supported, but [this pull request](https://github.com/slatinsky/DiscordChatExporter-frontend/pull/30) may help you
@@ -297,33 +517,19 @@ The windows release is exactly the same as the ones built on Github's servers by
Docker image is built on Github's servers github actions [github actions](https://github.com/slatinsky/DiscordChatExporter-frontend/blob/main/.github/workflows/docker-image.yml) too. This action directly uploads the image to docker hub.
-But the project uses a lot of dependencies - it is susceptible to supply chain attack such as dependency hijacking. If you find anything suspicious in dependencies used, please let me know.
-
-Tinfoil hat on? Replace `nginx.exe`, `mongod.exe`, `msvcp140.dll` and `vcruntime140_1.dll` with your own trusted copy. Then compile your own version from source code. The easiest way to compile is to run github action on your own fork.
+Tinfoil hat on? Replace `nginx.exe`, `mongod.exe`, `msvcp140.dll` and `vcruntime140_1.dll` with your own trusted copy. Then compile your own version from source code.
[Related discussion #13](https://github.com/slatinsky/DiscordChatExporter-frontend/discussions/13)
-
-## For developers
-
-[Architecture choices](docs/Architecture.md)
-
-[Setting up a development environment on Windows](docs/Development-env.md)
-
-[Compile the Windows release from source code](docs/Compile.md)
-
-[Supporting other exporters](docs/Supporting-other-exporters.md)
-
-
## Thanks
+
- [Tyrrrz/DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) - for a great export tool
-- [brussell98/discord-markdown](https://github.com/brussell98/discord-markdown) - for discord markdown rendering library
-- [themattman/mongodb-raspberrypi-binaries](https://github.com/themattman/mongodb-raspberrypi-binaries) - unofficial mongodb builds for Raspberry Pi
-And for other technologies used in this project - sveltekit, docker, nodejs, nvm, pyinstaller, nginx, mongodb
+And for other great technologies used in this project - SvelteKit, Node.js, PyInstaller, nginx, MongoDB, Docker
+
## Related projects
@@ -332,14 +538,18 @@ And for other technologies used in this project - sveltekit, docker, nodejs, nvm
- [mlomb/chat-analytics](https://github.com/mlomb/chat-analytics) - analytics for your Discord chats
- [slatinsky/DiscordChatExporter-incrementalBackup](https://github.com/slatinsky/DiscordChatExporter-incrementalBackup) - incrementally export your Discord chats for DiscordChatExporter-frontend
+
## License
+
GNU GENERAL PUBLIC LICENSE. See [LICENSE](LICENSE) for more details.
This product contains software provided by NGINX and its contributors.
DiscordChatExporter-frontend is not affiliated with Discord. Discord is a registered trademark of Discord Inc.
+
## Contributing
+
Feel free to open issues and pull requests.
Short guide, how to contribute
@@ -353,4 +563,4 @@ Feel free to open issues and pull requests.
-If you find this project useful, give it a star ⭐. Thank you!
\ No newline at end of file
+If you find this project useful, give it a star ⭐. Thank you!
diff --git a/docs/Architecture.md b/docs/Architecture.md
index 4aa2b38d..2705f628 100644
--- a/docs/Architecture.md
+++ b/docs/Architecture.md
@@ -1,11 +1,10 @@
-# Architecture choices
+# Architecture
The project is divided into two main parts: the server and the client.
## Used port numbers
- 21011 - nginx (reverse proxy)
-- 21013 - http-server (static files)
- 27017 - mongodb (database)
- 58000 - fastapi (backend api)
@@ -25,7 +24,7 @@ Specifically, the process does the following:
- downloads ggsans font from discord cdn.
-### mongodb
+### Mongodb
Database used to store data. The database is divided into multiple collections:
- assets - precomputed assets with local paths and dimensions
@@ -36,23 +35,18 @@ Database used to store data. The database is divided into multiple collections:
- jsons - stores sha256 hashes of processed files. This way, the preprocessor can skip already processed files.
- messages - stores all messages, including authors, emojis, attachments, etc.
-### fastapi
+Collections are sharded by guilds.
-Middleman between the client and the mongo database. Provides JSON api for the client (search, guild list, channel list, and messages).
+### Fastapi
-Search and channel endpoints return message ids. Those ids are then fetched by the client when needed.
+Middleman between the client and the mongo database. Provides JSON api for the frontend client (search, guild list, channel list, and messages).
-### http-server
-http-server is used to server static files for the frontend. Nginx could be used instead, but sometimes paths exceed the maximum path length of 260 characters on Windows and nginx fails to serve the files.
-
-http-server is used as a workaround for that bug in nginx. But it is also needed to apply registry patch to increase the maximum path length (use `change_260_character_path_limit_to_32767.reg`).
-
-### windows runner (Windows only)
+### Windows runner (Windows only)
Windows runner is the main entry point of the program on Windows. This script is compiled into `dcef.exe` in the release.
-- writes logs from other services to `logs.txt` file for easier debugging.
+- writes logs from other services to `logs/dcef.log` file for easier debugging.
- checks if all required ports are free on startup
@@ -62,17 +56,10 @@ Windows runner is the main entry point of the program on Windows. This script is
- cleans up all services on window close
-### nginx
-
-Nginx combines multiple services into one. See `nginx-prod.conf`.
-
-1. First is server frontend files needed for the client to load (paths `/_app/`, `/css/`, `/js/`, `/fonts/`, `/`).
-
-2. Then it proxies static files from http-server from port 21013 (path `/input/`).
-3. Then it serves static files created by preprocess (path `/data/`) - now used only for ggsans font.
+### Nginx
-4. Finally, it proxies api requests from fastapi from port 58000 (path `/api/`).
+Nginx combines fastapi backend and static frontend into one. See `nginx-prod.conf`.
## Client (Frontend)
diff --git a/docs/Compile.md b/docs/Compile.md
deleted file mode 100644
index fd847283..00000000
--- a/docs/Compile.md
+++ /dev/null
@@ -1,23 +0,0 @@
-# Compile the Windows release from source code
-
-## Easy way to compile
-
-Fork the repository and enable GitHub Actions. Then, go to the Actions tab and click on the latest workflow. Let it run and download the artifacts from the bottom of the page.
-
-## Manual compilation (Windows)
-
-
-### Prerequisites
-
-Follow the instructions in [Development environment](./Development-env.md) to install all the dependencies.
-
-Then install additional dependencyto compile http-server:
-
-```bash
-npm install -g pkg
-```
-
-
-### Compile
-
-Run `BUILD_RELEASE.bat`. The result will be the contents of the `release` folder.
diff --git a/docs/Development-env.md b/docs/Development-env.md
deleted file mode 100644
index 2b7c22b0..00000000
--- a/docs/Development-env.md
+++ /dev/null
@@ -1,42 +0,0 @@
-# Setting up a development environment on Windows
-
-## Prerequisites
-- Python 3.11
-- Node.js 18.17.1
-- pyinstaller 5.5 (installed globally from `pip`)
-- nodemon (installed globally from `npm`)
-- wt (windows terminal)
-
-## Install dependencies
-
-Install frontend dependencies:
-```bash
-cd frontend
-npm install
-cd ..
-```
-
-Install backend dependencies:
-```bash
-cd backend/preprocess
-py -m pip install -r requirements.txt
-cd ../..
-```
-
-```bash
-cd backend/fastapi
-py -m pip install -r requirements.txt
-pip install uvicorn==0.20.0
-cd ../..
-```
-
-```bash
-cd backend/windows-runner
-py -m pip install -r requirements.txt
-cd ../..
-```
-
-## Start the development script
-
-run `DEV.bat`
-
diff --git a/docs/Server-hosting.md b/docs/Server-hosting.md
deleted file mode 100644
index 72e45f2e..00000000
--- a/docs/Server-hosting.md
+++ /dev/null
@@ -1,45 +0,0 @@
-# Hosting on a Linux server
-
-## Disclaimer
-The client-server architecture allows you to host the viewer on public servers. But that doesn't mean that you should do it. Please be polite to other users and don't share any sensitive messages with broader audience (or the whole internet), than it was intended to be shared with before.
-
-Intended use cases:
-- old private discord server was hacked and new one was created. You want to share a backup with the same people, who were in the old server.
-- or you host your own backup on your own server, so you can access it from anywhere.
-- ... use common sense
-
-## Protecting the viewer with password authentication
-
-It is recomended to put the server behind another reverse proxy, such as nginx. The reverse proxy should be configured to require authentication (for example using [basic auth](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/)).
-
-Create firewall rules to open only TCP ports 22 (SSH) and 80 (HTTP) and enable firewall:
-```
-ufw allow 22/tcp
-ufw allow 80/tcp
-ufw enable
-```
-
-Bind port 21011 from docker container only to loopback (localhost `127.0.0.1`):
-```bash
-docker run ... -p 127.0.0.1:21011:21011 -it dcef
-```
-
-Create .htpasswd file:
-```bash
-sudo apt install apache2-utils
-htpasswd -c /etc/nginx/.htpasswd
-
-```
-
-Change nginx site config to require authentication (add lines `auth_basic` and `auth_basic_user_file`):
-```
-server {
- listen 80;
-
- location / {
- proxy_pass http://localhost:21011/;
- auth_basic "Auth only";
- auth_basic_user_file /etc/nginx/.htpasswd;
- }
-}
-```
\ No newline at end of file
diff --git a/docs/Supporting-other-exporters.md b/docs/Supporting-other-exporters.md
deleted file mode 100644
index eb77911f..00000000
--- a/docs/Supporting-other-exporters.md
+++ /dev/null
@@ -1,8 +0,0 @@
-# Supporting other exporters
-
-Any exporter exporting in the same format as [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) does, should work with this viewer. One of them is [Roachbones/discordless](https://github.com/Roachbones/discordless) - real time man-in-the-middle exporter.
-
-
-## For developers
-
-Are you creating a new discord exporter and want to view the exported data with this viewer? Just convert the data from your format to the same format as DiscordChatExporter uses :).
\ No newline at end of file
diff --git a/docs/dce-show-threads.png b/docs/dce-show-threads.png
new file mode 100644
index 00000000..a3c5166b
Binary files /dev/null and b/docs/dce-show-threads.png differ
diff --git a/docs/screenshot.png b/docs/screenshot.png
index be0a832e..a085e5d9 100644
Binary files a/docs/screenshot.png and b/docs/screenshot.png differ