Migration from Hugo to Mkdocs Material is ready

[opa-jan]

Initially, I had my website created with Hugo, using the Blist theme. At first, I thought it was nice because of all the visuals, but later on, I began to dislike it. I was looking for a more professional look and feel. When I found mkdocs-material, I was immediately enthusiastic. It had a professional appearance and offered a lot of plugins to customize the site.

Compare the websites:

Before:

After:

[jcyrss]

Yeah, I plan to migrate from Hugo also. Huge claim itself the fastest static web generator. But I found the learning curve is deep. Python for sure would be slower than go-native executable, but I suppose MKdocs Material's feature of sub projects would mitigate it well.

[kamilkrzyskow]

Huge claim itself the fastest static web generator

The question is do you need the fastest? How big will be your documentation and how often will it be updated?
Perhaps someone will make MkDocs work with Mojo at some point in time and the speed will increase considerably, but until then we have to make do.

To decrease rebuild times when using mkdocs serve you can use the --dirty flag to see rendered changes quicker, though it mostly applies to changes to content / assets, modifying hooks or the structure requires a full rebuild.

I did a simple performance benchmark with the Community version of mkdocs-material. With my setup and with their mkdocs.yml the documentation builds in ~9 seconds. Turning off the minify plugin for local builds would increase performance, perhaps it's worth changing the default as it would cut ~3 seconds of my time 😅

Benchmark (sorted by time)

On Windows 10 with Ryzen 3600

INFO    -  Documentation built in 9.22 seconds
BUILD_STAGE_TIMES =
[('build_pages', 4.90557),
 ('populate_pages', 3.82674),
 ('build_theme_templates', 0.16944),
 ('get_files', 0.10926),
 ('copy_static_files', 0.09138),
 ('on_config', 0.01901),
 ('on_nav', 0.01503),
 ('on_files', 0.01334),
 ('on_post_build', 0.00443),
 ('get_navigation', 0.00418),
 ('on_pre_build', 4e-05),
 ('on_env', 2e-05),
 ('build_extra_templates', 0.0)]
ALL_EVENT_TIMES =
[('post_page', 2.81611),
 ('page_context', 1.51964),
 ('page_markdown', 0.06419),
 ('config', 0.01901),
 ('post_template', 0.01622),
 ('page_content', 0.01554),
 ('nav', 0.01502),
 ('files', 0.01334),
 ('post_build', 0.00442),
 ('shutdown', 0.00144),
 ('pre_page', 4e-05),
 ('page_read_source', 4e-05),
 ('pre_build', 3e-05),
 ('env', 2e-05),
 ('pre_template', 0.0),
 ('template_context', 0.0)]
PLUGIN_TIMES =
[('mkdocs_minify_plugin/plugin.py', 2.83047),
 ('search/plugin.py', 0.92443),
 ('blog/plugin.py', 0.66887),
 ('hooks/shortcodes.py', 0.02797),
 ('hooks/translations.py', 0.0215)]
PLUGIN_EVENT_TIMES =
[('post_page--mkdocs_minify_plugin/plugin.py', 2.81431),
 ('page_context--search/plugin.py', 0.90238),
 ('page_context--blog/plugin.py', 0.61359),
 ('page_markdown--hooks/shortcodes.py', 0.02797),
 ('page_markdown--hooks/translations.py', 0.0215),
 ('config--search/plugin.py', 0.01767),
 ('post_template--mkdocs_minify_plugin/plugin.py', 0.01616),
 ('nav--blog/plugin.py', 0.015),
 ('page_content--blog/plugin.py', 0.01356),
 ('files--blog/plugin.py', 0.01331),
 ('page_markdown--blog/plugin.py', 0.0107),
 ('post_build--search/plugin.py', 0.00437),
 ('shutdown--blog/plugin.py', 0.00142),
 ('config--blog/plugin.py', 0.00128),
 ('pre_build--mkdocs_minify_plugin/plugin.py', 0.0),
 ('env--blog/plugin.py', 0.0),
 ('post_build--mkdocs_minify_plugin/plugin.py', 0.0)]
sum(BUILD_STAGE_TIMES) = 9.15844
sum(ALL_EVENT_TIMES) = 4.48506

On Windows 10 with Ryzen 3600

From the benchmark we can see that most of the time is spent populating the pages, which is the step that processes the Markdown with the markdown_extensions like Pygments that create code highlights, and building the pages, which is the step where the Jinja2 environment processes the theme templates. So the Material theme's and MkDocs' performance relies on the performance of their upstream dependencies.

The speed of the web site generation is not relevant for me. That's handled by the pipeline. I don't care if that takes 30 seconds or 30 minutes.

This could be a dangerous flow of logic. Topics like power consumption and CO2 emission get more and more attention nowadays. Considering that everything goes into cloud computing and AI it's worth thinking about such stuff. However, I just comment about it, I'm not an eco-activist, mind you. I like my O(N^N) solutions the best, because I can understand them the most, but I know they're not fit for production on scale.

I am still looking for an mkdocs plugin for image zooming

If by "zooming" you mean a pop-out window that will put focus on the clicked image, then there is https://blueswen.github.io/mkdocs-glightbox/#demo
Though, it modifies the HTML of the page on the on_post_page event, so with the theme of this comment, it's not optimal, it wastes performance.

[opa-jan]

I had a look at glightbox. That is not exactly what I need.

Regarding the performance question, I have already stated that I don't care. When I commit a change to my repository, the build pipeline works its magic. I use MkDocs Material because of the quality of the end result, regardless of whether it is built in a blazing fast 3 seconds or a more sluggish 5 minutes.

[squidfunk]

@kamilkrzyskow how did you generate the timing data? Very interested in that one. I've done some CPU profiling in Python and found that quite useful, where I also discovered that rendering of navigation takes a very large fraction of time in larger sites, but this looks very handy for a quick test what causes a build to be slow.

[kamilkrzyskow]

@squidfunk The benchmark is from a hacky monkey patch hook, which I'm slowly extending for personal use as I need it, and I started using it quite recently. Actually the nice prints and sorted by time results got added for this presentation 😺

performance_debug.zip

The hook

"""Simple MkDocs benchamrk.

Developed with version 1.5.3

MIT Licence - 2024 - Kamil Krzyśków (HRY)
"""

import inspect
import logging
import time
from pathlib import Path
from pprint import pprint
from typing import Literal
from urllib.parse import urljoin

from mkdocs import plugins, utils
from mkdocs.commands import build
from mkdocs.commands.build import (
    _build_extra_template,
    _build_page,
    _build_theme_template,
    _populate_page,
    log,
    site_directory_contains_stale_files,
)
from mkdocs.config.defaults import MkDocsConfig
from mkdocs.exceptions import Abort, BuildError
from mkdocs.livereload import LiveReloadServer
from mkdocs.plugins import PluginCollection
from mkdocs.structure.files import InclusionLevel, _set_exclusions, get_files
from mkdocs.structure.nav import get_navigation
from mkdocs.structure.pages import Page

FLAG = False

ALL_EVENT_TIMES = {}
PLUGIN_TIMES = {}
PLUGIN_EVENT_TIMES = {}
BUILD_STAGE_TIMES = {}


@plugins.event_priority(100)
def on_startup(*, command: Literal["build", "gh-deploy", "serve"], dirty: bool) -> None:
    print("Performance debug on_startup")
    global FLAG

    if FLAG is False:
        PluginCollection.run_event = run_event_wrapper(PluginCollection.run_event)
        build.build = build_wrapper(build.build)
        FLAG = True


def build_wrapper(func):

    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        passed = time.perf_counter() - start
        print(f"Build took {passed} seconds")
        return result

    def wrapper2(
        config: MkDocsConfig, live_server: LiveReloadServer | None = None, dirty: bool = False
    ) -> None:
        """Perform a full site build."""

        logger = logging.getLogger("mkdocs")

        # Add CountHandler for strict mode
        warning_counter = utils.CountHandler()
        warning_counter.setLevel(logging.WARNING)
        if config.strict:
            logging.getLogger("mkdocs").addHandler(warning_counter)

        inclusion = InclusionLevel.all if live_server else InclusionLevel.is_included

        try:
            start_global = time.monotonic()

            # Run `config` plugin events.
            start = time.perf_counter()
            config = config.plugins.on_config(config)
            BUILD_STAGE_TIMES["on_config"] = time.perf_counter() - start

            # Run `pre_build` plugin events.
            start = time.perf_counter()
            config.plugins.on_pre_build(config=config)
            BUILD_STAGE_TIMES[f"on_pre_build"] = time.perf_counter() - start

            if not dirty:
                log.info("Cleaning site directory")
                utils.clean_directory(config.site_dir)
            else:  # pragma: no cover
                # Warn user about problems that may occur with --dirty option
                log.warning(
                    "A 'dirty' build is being performed, this will likely lead to inaccurate navigation and other"
                    " links within your site. This option is designed for site development purposes only."
                )

            if not live_server:  # pragma: no cover
                log.info(f"Building documentation to directory: {config.site_dir}")
                if dirty and site_directory_contains_stale_files(config.site_dir):
                    log.info("The directory contains stale files. Use --clean to remove them.")

            # First gather all data from all files/pages to ensure all data is consistent across all pages.

            start = time.perf_counter()
            files = get_files(config)
            env = config.theme.get_env()
            files.add_files_from_theme(env, config)
            BUILD_STAGE_TIMES["get_files"] = time.perf_counter() - start

            # Run `files` plugin events.
            start = time.perf_counter()
            files = config.plugins.on_files(files, config=config)
            BUILD_STAGE_TIMES["on_files"] = time.perf_counter() - start
            # If plugins have added files but haven't set their inclusion level, calculate it again.
            _set_exclusions(files._files, config)

            start = time.perf_counter()
            nav = get_navigation(files, config)
            BUILD_STAGE_TIMES["get_navigation"] = time.perf_counter() - start

            # Run `nav` plugin events.
            start = time.perf_counter()
            nav = config.plugins.on_nav(nav, config=config, files=files)
            BUILD_STAGE_TIMES["on_nav"] = time.perf_counter() - start

            log.debug("Reading markdown pages.")
            excluded = []
            start = time.perf_counter()
            for file in files.documentation_pages(inclusion=inclusion):
                log.debug(f"Reading: {file.src_uri}")
                if file.page is None and file.inclusion.is_excluded():
                    if live_server:
                        excluded.append(urljoin(live_server.url, file.url))
                    Page(None, file, config)
                assert file.page is not None
                _populate_page(file.page, config, files, dirty)
            BUILD_STAGE_TIMES["populate_pages"] = time.perf_counter() - start
            if excluded:
                log.info(
                    "The following pages are being built only for the preview "
                    "but will be excluded from `mkdocs build` per `exclude_docs`:\n  - "
                    + "\n  - ".join(excluded)
                )

            # Run `env` plugin events.
            start = time.perf_counter()
            env = config.plugins.on_env(env, config=config, files=files)
            BUILD_STAGE_TIMES["on_env"] = time.perf_counter() - start

            # Start writing files to site_dir now that all data is gathered. Note that order matters. Files
            # with lower precedence get written first so that files with higher precedence can overwrite them.

            log.debug("Copying static assets.")
            start = time.perf_counter()
            files.copy_static_files(dirty=dirty, inclusion=inclusion)
            BUILD_STAGE_TIMES["copy_static_files"] = time.perf_counter() - start

            start = time.perf_counter()
            for template in config.theme.static_templates:
                _build_theme_template(template, env, files, config, nav)
            BUILD_STAGE_TIMES["build_theme_templates"] = time.perf_counter() - start

            start = time.perf_counter()
            for template in config.extra_templates:
                _build_extra_template(template, files, config, nav)
            BUILD_STAGE_TIMES["build_extra_templates"] = time.perf_counter() - start

            log.debug("Building markdown pages.")
            start = time.perf_counter()
            doc_files = files.documentation_pages(inclusion=inclusion)
            for file in doc_files:
                assert file.page is not None
                _build_page(
                    file.page,
                    config,
                    doc_files,
                    nav,
                    env,
                    dirty,
                    excluded=file.inclusion.is_excluded(),
                )
            BUILD_STAGE_TIMES["build_pages"] = time.perf_counter() - start

            # Run `post_build` plugin events.
            start = time.perf_counter()
            config.plugins.on_post_build(config=config)
            BUILD_STAGE_TIMES["on_post_build"] = time.perf_counter() - start

            counts = warning_counter.get_counts()
            if counts:
                msg = ", ".join(f"{v} {k.lower()}s" for k, v in counts)
                raise Abort(f"Aborted with {msg} in strict mode!")

            log.info(f"Documentation built in {time.monotonic() - start_global:.2f} seconds")

        except Exception as e:
            # Run `build_error` plugin events.
            config.plugins.on_build_error(error=e)
            if isinstance(e, BuildError):
                log.error(str(e))
                raise Abort("Aborted with a BuildError!")
            raise

        finally:
            logger.removeHandler(warning_counter)

    return wrapper2


def run_event_wrapper(func):

    def wrapper(self, name: str, item=None, **kwargs):
        start = time.perf_counter()
        pass_item = item is not None
        events = self.events[name]
        for method in events:
            file = "/".join(Path(inspect.getfile(method)).as_posix().split("/")[-2:])
            start_method = time.perf_counter()
            if pass_item:
                result = method(item, **kwargs)
            else:
                result = method(**kwargs)
            # keep item if method returned `None`
            if result is not None:
                item = result
            passed_method = time.perf_counter() - start_method
            PLUGIN_TIMES[file] = PLUGIN_TIMES.get(file, 0) + passed_method
            file_event = name+"--"+file
            PLUGIN_EVENT_TIMES[file_event] = PLUGIN_EVENT_TIMES.get(file_event, 0) + passed_method
        passed = time.perf_counter() - start
        ALL_EVENT_TIMES[name] = ALL_EVENT_TIMES.get(name, 0) + passed
        if name == "shutdown":
            for key, value in BUILD_STAGE_TIMES.items():
                BUILD_STAGE_TIMES[key] = round(value, 5)
            for key, value in ALL_EVENT_TIMES.items():
                ALL_EVENT_TIMES[key] = round(value, 5)
            for key, value in PLUGIN_TIMES.items():
                PLUGIN_TIMES[key] = round(value, 5)
            for key, value in PLUGIN_EVENT_TIMES.items():
                PLUGIN_EVENT_TIMES[key] = round(value, 5)
            print("BUILD_STAGE_TIMES = ")
            pprint(sorted(BUILD_STAGE_TIMES.items(), key=lambda x: x[1], reverse=True), sort_dicts=False)
            print("ALL_EVENT_TIMES = ")
            pprint(sorted(ALL_EVENT_TIMES.items(), key=lambda x: x[1], reverse=True), sort_dicts=False)
            print("PLUGIN_TIMES = ")
            pprint(sorted(PLUGIN_TIMES.items(), key=lambda x: x[1], reverse=True), sort_dicts=False)
            print("PLUGIN_EVENT_TIMES = ")
            pprint(sorted(PLUGIN_EVENT_TIMES.items(), key=lambda x: x[1], reverse=True), sort_dicts=False)
            print("sum(BUILD_STAGE_TIMES) =", sum(BUILD_STAGE_TIMES.values()))
            print("sum(ALL_EVENT_TIMES) =", sum(ALL_EVENT_TIMES.values()))

        return item

    return wrapper

---

I've made a gist with a better hook, please read:

• Build speed : mkdocs vs Hugo #6802 (reply in thread)

---

It overrides the internal MkDocs build and plugin event handling, so once MkDocs updates those functions, the hook might stop working altogether or create other issues. However, I don't mind that too much.

Before the hook, I would often change the site-packages to benchmark it by hand. I found the Pygments bug this way:

• [Bug] Lexer lookup doesn't cache custom plugin lexers pygments/pygments#2479
• Hook fix: https://github.com/Gothic-Modding-Community/gmc/blob/abcaa7127432c348136671144f32c0945e3abd07/overrides/.hooks/all_plugin_patch.py#L141-L154

However, reinstalling python packages would delete my changes, so I finally decided to benchmark with a hook.

A more "professional" approach, I guess, would be to use the cProfile module like so:

python -m cProfile -s module -m mkdocs build > cProfile.log

Or instead of saving the result to file it could be piped to another script

The -s module could be also changed to -s cumtime, as per the docs.
And later the results could be filtered with regex and aggregated into a similar form as I did above.
The only issues, I see now is that cProfile adds overhead and make the build even slower, and it uses the base file name of the files, so there will be a lot of plugin.py files in the results for the regex to filter.

Some months back I've made a log manager hook as well:

• Enable users to change log level mkdocs/mkdocs#2876 (comment)
  But to be honest I didn't use it that much, especially since I didn't work with MkDocs since June to September. However, you might be interested in it ✌️

[squidfunk]

Thanks for sharing! It'll be of great use for me ☺️ And yes, cProfile makes things slower, but it's great to see what's making the build slow relatively. It was also how I learned about navigation rendering being the main bottleneck.