Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

espanso-wayland: service start failure #249364

Open
bashfulrobot opened this issue Aug 15, 2023 · 46 comments · May be fixed by #328890
Open

espanso-wayland: service start failure #249364

bashfulrobot opened this issue Aug 15, 2023 · 46 comments · May be fixed by #328890
Labels
0.kind: bug Something is broken

Comments

@bashfulrobot
Copy link

Describe the bug

A clear and concise description of what the bug is.

The service will not start properly. Resulting in Espanso not working. Running journalctl --user -u espanso -e I can see the following:

Aug 15 10:14:09 rembot systemd[3238]: espanso.service: Scheduled restart job, restart counter is at 37388.
Aug 15 10:14:09 rembot systemd[3238]: Stopped espanso.
Aug 15 10:14:09 rembot systemd[3238]: Started espanso.
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [INFO] reading configs from: "/home/dust>
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [INFO] reading packages from: "/home/dus>
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [INFO] using runtime dir: "/home/dustin/>
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [INFO] system info: NixOS v23.11 - kerne>
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [WARN] keyboard layout watcher couldn't >
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [INFO] watching for changes in path: "/h>
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [INFO] espanso version: 2.1.8
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [INFO] spawning the worker process...
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [INFO] binded to IPC unix socket: /home/>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [INFO] reading configs from: "/home/dust>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [INFO] reading packages from: "/home/dus>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [INFO] using runtime dir: "/home/dustin/>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [INFO] system info: NixOS v23.11 - kerne>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [INFO] binded to IPC unix socket: /home/>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [INFO] using WaylandAppInfoProvider
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [WARN] EVDEV backend is being used, but >
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [WARN]   Although you CAN run espanso EV>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [INFO] monitoring the status of the daem>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [WARN]   to security reasons. Espanso su>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [WARN]   area by only leveraging on the >
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [WARN]   /dev/input/* devices to detect >
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [WARN]   initial setup is completed.
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [WARN] unable to determine keyboard layo>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [INFO] using EVDEVSource
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [ERROR] Unable to open EVDEV devices, th>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [ERROR] You can either add the current u>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [ERROR] thread 'engine thread' panicked >
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [ERROR] Unable to block the LinuxEventLo>
Aug 15 10:14:09 rembot .espanso-wrapped[1387426]: 10:14:09 [worker(1387426)] [ERROR] thread 'main' panicked at 'unabl>
Aug 15 10:14:09 rembot .espanso-wrapped[1387420]: 10:14:09 [daemon(1387420)] [ERROR] received unexpected exit code fr>
Aug 15 10:14:09 rembot .espanso-wrapped[1387413]: thread 'main' panicked at 'failed to launch daemon: unexpected erro>
Aug 15 10:14:09 rembot .espanso-wrapped[1387413]: note: run with `RUST_BACKTRACE=1` environment variable to display a>
Aug 15 10:14:09 rembot systemd[3238]: espanso.service: Main process exited, code=exited, status=101/n/a
Aug 15 10:14:09 rembot systemd[3238]: espanso.service: Failed with result 'exit-code'.

The Espanso

Steps To Reproduce

Steps to reproduce the behavior:

  1. Install espanso-wayland via environment.systemPackages = with pkgs; [
  2. Enable the service with services.espanso.enable = true;
  3. Reboot, or restart the service with systemctl --user restart espanso
  4. Check the status of the service with systemctl --user status espanso
  5. you will see something like:
espanso.service - espanso
     Loaded: loaded (/home/dustin/.config/systemd/user/espanso.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Tue 2023-08-15 10:18:57 PDT; 611ms ago
    Process: 1391746 ExecStart=/nix/store/wnizz2x246m9vyqnlp1nmx77z80mniqm-espanso-2.1.8/bin/.espanso-wrapped launcher (code=exited, status=101)
   Main PID: 1391746 (code=exited, status=101)
        CPU: 148m
  1. Check journalctl with journalctl --user -u espanso -e and you will see the above error.

Expected behavior

  • I expected the service to start successfully, and for Espanso to respond to particular keystrokes.

Additional context

This is expected, but:

λ espanso start
unable to start service: timed out
Hint: sometimes this happens because another Espanso process is left running for some reason.
      Please try running 'espanso restart' or manually killing all Espanso processes, then try again.

Notify maintainers

@kimat
@theHedgehog0

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

λ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.44, NixOS, 23.11 (Tapir), 23.11.20230810.ce5e4a6`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.15.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Thank you.

@bashfulrobot bashfulrobot added the 0.kind: bug Something is broken label Aug 15, 2023
@bashfulrobot
Copy link
Author

bashfulrobot commented Aug 15, 2023

@bobvanderlinden I am only tagging you as if I remember correctly, the wayland support/update to the 2.x series was added by you. If not, please ignore. 👍

Thank you.

@AlecsFerra
Copy link
Contributor

I think this is related to this espanso/espanso#1638

@bobvanderlinden
Copy link
Member

I'm guessing the espanso service needs a wrapper. Wireshark does the following for its dumpcap capabilities:

https://github.com/NixOS/nixpkgs/blob/caac0eb6bdcad0b32cb2522e03e4002c8975c62e/nixos/modules/programs/wireshark.nix#L34-40

Instead of dumpcap, espanso needs cap_dac_override.

Maybe alternatively it might be better for systemd to set the capability. It seems https://www.freedesktop.org/software/systemd/man/systemd.exec.html#CapabilityBoundingSet= should be able to do so.

@bobvanderlinden
Copy link
Member

I see I forgot to add that I'm not running wayland. I tried espanso's wayland support under xorg, but I'm not sure about the full wayland integration. I'm probably not the right person to look into this.

@ManUtopiK
Copy link

Any update on this ? I've got the same issue.

@hatch01
Copy link
Contributor

hatch01 commented Nov 28, 2023

Any news on this issue?

@bashfulrobot
Copy link
Author

Is there any process to mark the package as "broken" until the maintainers can look at it?

@bobvanderlinden
Copy link
Member

The package itself doesn't seem to be broken. Last time I tried to run espanso-wayland in Weston within Xorg. That seemed to work.
What seems to be broken is the systemd .service file, not the package itself.

It is possible to fail when services.espanso.enable and one of the Wayland compositors are enabled. Then the configuration will fail to build. Would that be a better approach?

(It would still be great if someone with a Wayland setup could look into the .service file/configuration 😅)

@bashfulrobot
Copy link
Author

Well, what I meant by "broken" is that the software itself does not work in its current state.

I had a look for the service file, and they seem to have:

λ cat -p /home/dustin/.config/systemd/user/default.target.wants/espanso.service

[Unit]
Description=espanso

[Service]
ExecStart=/nix/store/kflc5d4nli6kmyllrdjmn7900ib1jsfq-espanso-2.1.8/bin/.espanso-wrapped launcher
Restart=on-failure
RestartSec=3

[Install]
WantedBy=default.target

I also ran the wrapper direct and received:

14:21:47 [daemon(57987)] [INFO] reading configs from: "/home/dustin/.config/espanso"
14:21:47 [daemon(57987)] [INFO] reading packages from: "/home/dustin/.config/espanso/match/packages"
14:21:47 [daemon(57987)] [INFO] using runtime dir: "/home/dustin/.cache/espanso"
14:21:47 [daemon(57987)] [INFO] system info: NixOS v23.11 - kernel: 6.1.62
14:21:47 [daemon(57987)] [WARN] keyboard layout watcher couldn't determine active layout.
14:21:47 [daemon(57987)] [INFO] watching for changes in path: "/home/dustin/.config/espanso"
14:21:47 [daemon(57987)] [INFO] espanso version: 2.1.8
14:21:47 [daemon(57987)] [INFO] spawning the worker process...
14:21:47 [daemon(57987)] [INFO] binded to IPC unix socket: /home/dustin/.cache/espanso/espansodaemonv2.sock
14:21:47 [worker(57993)] [INFO] reading configs from: "/home/dustin/.config/espanso"
14:21:47 [worker(57993)] [INFO] reading packages from: "/home/dustin/.config/espanso/match/packages"
14:21:47 [worker(57993)] [INFO] using runtime dir: "/home/dustin/.cache/espanso"
14:21:47 [worker(57993)] [INFO] system info: NixOS v23.11 - kernel: 6.1.62
14:21:47 [worker(57993)] [INFO] binded to IPC unix socket: /home/dustin/.cache/espanso/espansoworkerv2.sock
14:21:47 [worker(57993)] [INFO] using WaylandAppInfoProvider
14:21:47 [worker(57993)] [WARN] EVDEV backend is being used, but without enabling linux capabilities.
14:21:47 [worker(57993)] [INFO] monitoring the status of the daemon process
14:21:47 [worker(57993)] [WARN]   Although you CAN run espanso EVDEV backend as root, it's not recommended due
14:21:47 [worker(57993)] [WARN]   to security reasons. Espanso supports linux capabilities to limit the attack surface
14:21:47 [worker(57993)] [WARN]   area by only leveraging on the CAP_DAC_OVERRIDE capability (needed to work with
14:21:47 [worker(57993)] [WARN]   /dev/input/* devices to detect and inject text) and disabling it as soon as the
14:21:47 [worker(57993)] [WARN]   initial setup is completed.
14:21:47 [worker(57993)] [WARN] unable to determine keyboard layout automatically, please explicitly specify it in the configuration.
14:21:47 [worker(57993)] [INFO] using EVDEVSource
14:21:47 [worker(57993)] [ERROR] Unable to open EVDEV devices, this usually has to do with permissions.
14:21:47 [worker(57993)] [ERROR] You can either add the current user to the 'input' group or run espanso as root
14:21:47 [worker(57993)] [ERROR] thread 'engine thread' panicked at 'failed to initialize detector module: detection source initialization failed': espanso/src/cli/worker/engine/mod.rs:139
14:21:47 [worker(57993)] [ERROR] Unable to block the LinuxEventLoop: receiving on an empty and disconnected channel
14:21:47 [worker(57993)] [ERROR] thread 'main' panicked at 'unable to run main eventloop: receiving on an empty and disconnected channel': espanso/src/cli/worker/mod.rs:160
14:21:47 [daemon(57987)] [ERROR] received unexpected exit code from worker 101, exiting
thread 'main' panicked at 'failed to launch daemon: unexpected error, 'espanso daemon' returned a non-zero exit code.', espanso/src/cli/launcher/mod.rs:213:45
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Thank you.

@bobvanderlinden
Copy link
Member

Ah sorry, I'm using Espanso integration from home-manager, which seems to be working fine. I'm not sure whether Espanso needs to be run as root (on system level) or as part of the user. I thought the latter.

Maybe the Espanso service in NixOS needs to be removed eventually?

@bashfulrobot
Copy link
Author

ok, I was not running it with home-manager. Do you have a repo with your config? Happy to move it over and test.

@bobvanderlinden
Copy link
Member

@GimmeDataNow
Copy link

Any updates on this?

@SkyWriter
Copy link

So the core problem is that Espanso binary doesn't get proper capabilities. In the world of ordinary UNIX'es it's just a setpcap away. In Nix I tried the following to no avail:

  • Changing the derivation to do setpcap as a part of the postInstall. I got a read-only filesystem error.
  • Using system level security.wrappers and pointing my user level systemd ExecStart to those wrappers. For some reason Espanso failed to acknowledge those capabilities and start. Patching Espanso to forcefully try to acquire them resulted in the permission denied error. This may be related to those several wrappers around Espanso being executed. 🤷‍♂️

Becoming tired of it I've resorted to a hack...

TLDR 🥳

Here's what I've put into my Home Manager config:

  services.espanso = {
    enable = true;
    package = unstable.espanso-wayland;
    # ...
  };
  
  systemd.user.services.espanso.Service.ExecStart =
    lib.mkForce "/home/sky/.config/espanso/espanso daemon";

In order for this to work I had to perform the following as well:

  • Copy espanso & .espanso-wrapped to ~/.config/espanso
  • Change espanso so that it starts the newly copied .espanso-wrapped file
  • sudo setcap "cap_dac_override+p" ./.espanso-wrapped

This is obviosly a hack, and will most likely break and require me to fix those Espanso binaries with mostly every update. However, this is the problem of the future Homer (tm).

@pitkling
Copy link
Member

My draft PR #328890 may resolve this issue. It uses the security.wrappers framework mentioned by @SkyWriter above together with an injection that ensures that forks that espanso creates of itself pickup the capability-enabled wrapper.

@GimmeDataNow
Copy link

@pitkling
Sorry to bother, but would you please tell me how to use this in my flake.nix without forking the repo?
I am still very new to nixos and can't figure out how to do this myself.

@pitkling
Copy link
Member

pitkling commented Aug 1, 2024

@GimmeDataNow Sorry for the delayed answer, I was a bit busy. The original draft PR was probably difficult to use without using my fork directly. But I recently updated the PR, it is now just a simple NixOS module. One should be able to copy the directory nixos/modules/services/desktops/espanso-capdacoverride of my PR. If you store that directory in the same directory as your flake.nix, you probably should be able to get it working via something like the following in your flake.nix (untested!):

…
    nixpkgs.lib.nixosSystem {
      …
      modules = [
        …
        ./espanso-capdacoverride

        ({ config, ... }: {
          nixpkgs.overlays = [
            (final: prev: {
              _espanso-orig = prev.espanso;
              espanso = config.programs.espanso-capdacoverride.packageOverriden;
            })
          ];

          programs.espanso-capdacoverride = {
            enable = true;
            package = pkgs._espanso-orig;
          };
        })
        …
      ]
      …
    }
…

With this, other modules can simply use pkgs.espanso to reference the correctly wrapped espanso.

@GimmeDataNow
Copy link

Thank you very much! It is working :).

A quick note:
If you set programs.espanso-capdacoverride.package = pkgs.espanso-wayland then it will error and complain about infinite recursion. I circumvented this by simply using programs.espanso-capdacoverride.package = unstable.espanso-wayland where unstable = import <nixos-unstable> { config = { allowUnfree = true; nixpkgs.config.allowBroken = true; }; };

This is probably due to an error on my part, but this resolves it regardless

@pitkling
Copy link
Member

pitkling commented Aug 2, 2024

Glad it works :).

The infinite recursion seems strange. Were you using the overlay I posted? In that case, you indeed have to also use programs.espanso-capdacoverride.package = pkgs._espanso-orig from my post (or something similar adapted to espanso-wayland) to avoid the infinite recursion (I think). If you were not using the overlay I'm not sure why its complained about the infinite recursion; in that case I'll take a look later (I'm using it myself actually very similar to you with an unstable espanso-wayland package)

@bashfulrobot
Copy link
Author

@GimmeDataNow Sorry for the delayed answer, I was a bit busy. The original draft PR was probably difficult to use without using my fork directly. But I recently updated the PR, it is now just a simple NixOS module. One should be able to copy the directory nixos/modules/services/desktops/espanso-capdacoverride of my PR. If you store that directory in the same directory as your flake.nix, you probably should be able to get it working via something like the following in your flake.nix (untested!):


…

    nixpkgs.lib.nixosSystem {

      …

      modules = [

        …

        ./espanso-capdacoverride



        ({ config, ... }: {

          nixpkgs.overlays = [

            (final: prev: {

              _espanso-orig = prev.espanso;

              espanso = config.programs.espanso-capdacoverride.packageOverriden;

            })

          ];



          programs.espanso-capdacoverride = {

            enable = true;

            package = pkgs._espanso-orig;

          };

        })

        …

      ]

      …

    }

…

With this, other modules can simply use pkgs.espanso to reference the correctly wrapped espanso.

That's amazing! I assume it's just waiting for merge, etc....

@pitkling
Copy link
Member

pitkling commented Aug 5, 2024

[…] That's amazing! I assume it's just waiting for merge, etc....

We're not quite there yet w.r.t. the merge. I discussed with @n8henrie and there are some valid concerns whether this is the best/correct way to enable user-space espanso, even though it seems the best solution we have so far. If it were to be merged, it still needs some cleanup.

Anyway, @robryk was so nice to also take a look, since the underlying problem that prevents to simply use security.wrappers might be something that maybe would be better handled directly by security.wrappers. This might lead to a better long-term solution, but it takes some time to assess feasibility.

In any case, if the draft PR is not merged, I might just provide a flake with the module, so at least it will become easier to use this until we have a proper solution.

@n8henrie
Copy link
Contributor

@pitkling I think the idea of getting this built into security.wrappers would make a lot of sense -- if using security.wrappers, I can't imagine why one wouldn't expect forked processes to inherit capabilities. I think you've done a great job of explaining the issue in your PR -- would you mind opening an issue for discussion with a few more senior nix devs to see what they think?

@n8henrie
Copy link
Contributor

Hmm, actually is this already supposed to work? e355020

@pitkling
Copy link
Member

@pitkling I think the idea of getting this built into security.wrappers would make a lot of sense -- if using security.wrappers, I can't imagine why one wouldn't expect forked processes to inherit capabilities. I think you've done a great job of explaining the issue in your PR -- would you mind opening an issue for discussion with a few more senior nix devs to see what they think?

@n8henrie I would also say so, although integrating this into security.wrappers might be somewhat subtle and not so simple. I played around a bit and there seem to be some obstacles to get this to work reliably. So I guess discussing this in a separate issue with the corresponding devs seems reasonable. I only first want to wait and see if @robryk confirms my guess for the cause of this in my draft PR #328890 (and maybe sees an immediate solution).

Hmm, actually is this already supposed to work? e355020

Kind of, although this does not work in the case of espanso. If I understand the implementation of security.wrappers correctly, the patched line that commit references raises the correct capabilities into the ambient set (to ensure that capabilities are inherited by forks). However, the problem is that espanso drops its capabilities (and with them, the capabilities in the ambient set!) and relies on file capabilities when forking itself (via /proc/self/exe) to regain the correct capabilities. Unfortunately, at the time of the fork proc/self/exe points to the actual binary (without the file capabilities).

@n8henrie
Copy link
Contributor

That's right, thanks for refreshing my memory. (BTW your technical writing is excellent -- very approachable and readable! In spite of your humility, I think you understand this much better than me.)

#328890 (comment)

That said, I'm using Linux only occasionally

Just out of curiosity, looks like you may be primarily on darwin? (guessing -- one of your popular repos appears to be for Hammerspoon). One of my few NixOS machines that uses a graphical desktop is my dual-boot Asahi Macbook, on which I'm primarily running macOS -- reluctance to close down all my open tabs and windows to reboot into NixOS has definitely slowed my participation here.

A few things I'll tinker on:

Alternatively, maybe a PR to espanso to drop capabilities post-fork (instead of pre-fork) could be considered?

@pitkling
Copy link
Member

[…]

That said, I'm using Linux only occasionally

Just out of curiosity, looks like you may be primarily on darwin? (guessing -- one of your popular repos appears to be for Hammerspoon). One of my few NixOS machines that uses a graphical desktop is my dual-boot Asahi Macbook, on which I'm primarily running macOS -- reluctance to close down all my open tabs and windows to reboot into NixOS has definitely slowed my participation here.

No worries, we all have to do enough other things in real life; and I can definitely relate wrt the tabs 😆. But yes, I'm primarily on Darwin, using Linux mostly for hobby projects like tinkering with some Raspberry Pies.

A few things I'll tinker on:

  • Quickly refreshing my memory on the espanso codebase, does espanso service start --unmanaged do much more than fork off espanso daemon?

I think so after looking into the code. It seems --unmanagedcalls fork_daemon which forks a call to spawn_launcher to initiate espanso launcher. Then espanso launcher eventually (after possibly showing some GUI elements) calls daemon::launch_daemon which finally initiates espanso daemon.

  • If not, perhaps running espanso daemon directly might avoid the fork and keep the capabilities? […]

I don't think that'll work. espanso daemon itself forks espanso worker, which seems to be the only subcommand requiring the capabilities.

Alternatively, maybe a PR to espanso to drop capabilities post-fork (instead of pre-fork) could be considered?

Hm, this might be difficult. It seems that espanso relies quite a bit on the interplay between the different subcommands, and the worker subcommand is called quite late. If I understand right, currently any subcommand except the worker will drop its capabilities almost immediately.

Also, it seems actually a valid design choice for an app to rely on file capabilities together with the capabilities in the ambient set to reduce the potential attack area. So even if one could change espanso to avoid it, the problem might surface in other places. It would be good to have some way to deal with it. If not possible in general by security.wrappers, then at least with something like the PR's wrapper library (or another, better solution)

@n8henrie
Copy link
Contributor

n8henrie commented Sep 3, 2024

@pitkling -- I have espanso running (including forms and the UI) under KDE Plasma / Wayland using just the security.wrappers (copied yours verbatim from #328890 (comment)) and running espanso worker directly (instead of espanso daemon or launcher).

Can you try and see if you can confirm?

image

@n8henrie
Copy link
Contributor

n8henrie commented Sep 4, 2024

Actually, I didn't copy verbatim -- I used capabilities = "cap_dac_override+p"; (which should be the minimum).

n8henrie added a commit to n8henrie/nixpkgs that referenced this issue Sep 4, 2024
On Wayland, Espanso depends on `cap_dac_override+p` for the EVDEV
backend. Specifically, this capability is required by the `worker`
thread, which is forked from the main espanso process when run by the
usual means (`espanso start` or `espanso daemon`).

Espanso (responsibly) drops capabilities as soon as possible, prior
to forking the worker process. Unfortunately, `security.wrappers` sets
the capabilities in such a way that the forked process does not pick
up these capabilities (due to `/proc/self/exe` pointing to the original
espanso binary, which does *not* have these capabilities).

By running `worker` directly from the capability-enabled wrapper,
the worker thread is able to run without issue, and Espanso runs as
expected on wayland.

- NixOS#249364
- NixOS#328890
- https://espanso.org/docs/install/linux

- fixes NixOS#249364
@pitkling
Copy link
Member

pitkling commented Sep 4, 2024

@pitkling -- I have espanso running (including forms and the UI) under KDE Plasma / Wayland using just the security.wrappers (copied yours verbatim from #328890 (comment)) and running espanso worker directly (instead of espanso daemon or launcher).

Can you try and see if you can confirm? […]

Indeed, launching the worker directly with the security wrapper should avoid all the problems I described before. I tried that myself when I first looked into this issue and basic functionality was there. However, I did not do any extensive testing.

I see two potential problems with this approach:

  1. It seems to me (and I might be wrong here) that the espanso worker subcommand is an internal command and not intended for the end user. E.g., it is not documented under espanso --help. This means that the espanso team could remove/change/substitute/rename/… that subcommand any time, possibly breaking things without us noticing right away.
  2. Maybe more severe: If I read the code correctly, running espanso worker manually will skip miscellaneous monitoring code. E.g., it seems this code or this code would never run. I don't know how important that code is, but it seems to do stuff like automatically restarting the worker if the configuration changed and such.

So even if it seems to work, we might easily be breaking some stuff without us noticing right away. Even if it works now, it might break future features if some internals change.

@n8henrie
Copy link
Contributor

n8henrie commented Sep 4, 2024

It seems to me (and I might be wrong here) that the espanso worker subcommand is an internal command

You're likely correct, I only know about it from months worth of digging to figure out #247162 / espanso/espanso#1946

This means that the espanso team could remove/change/substitute/rename/… that subcommand any time

True, but the nix input won't change until we change it. I've been trying to help the espanso team here and there; there's been talk about a big-ish refactor of the GUI to remove the wx dependency, but I haven't heard of any immediate plans for a major refactoring of the backend. (EDIT: espanso roadmap)

automatically restarting the worker if the configuration changed

Good point, I need to double check this. Requiring manual reload to pick up config changes would be a bit of a bummer (since they've already done the work to make this smoother).

The systemd unit should be automatically restarting on failure, which might obviate the need for some of their checks.

Even if it works now, it might break future features if some internals change.

That's of course possible, though I'm not sure what approach could prevent upstream changes from breaking the always-unconventional nix way.


I also wonder why they're using ffi to get the executable path: https://github.com/espanso/espanso/blob/fd9a7dd4f2afccaeb61623286b1230c0df0c7033/espanso-info/src/x11/native.cpp#L138

as opposed to current_exe

Actually, nevermind: https://github.com/espanso/espanso/blob/fd9a7dd4f2afccaeb61623286b1230c0df0c7033/espanso/src/path/linux.rs#L71

Looks like we might be able to spoof the executable path with APPIMAGE: https://github.com/espanso/espanso/blob/fd9a7dd4f2afccaeb61623286b1230c0df0c7033/espanso/src/cli/service/linux.rs#L276 (😆)

@KenMacD
Copy link
Contributor

KenMacD commented Sep 5, 2024

In case it helps anyone, this is my current config for espanso that's been working for me:

  # Allow input access of espanso
  security.wrappers.espanso = {
    source = "${pkgs.espanso-wayland}/bin/espanso";
    capabilities = "cap_dac_override+p";
    owner = "root";
    group = "root";
  };

  services.espanso.enable = true;
  systemd.user.services.espanso = {
    serviceConfig.ExecStart = lib.mkForce "/run/wrappers/bin/espanso worker";
    # Commands needed for expansions:
    path = with pkgs; [
      bash
      dig
      gopass
    ];
  };

@n8henrie
Copy link
Contributor

n8henrie commented Sep 5, 2024

@KenMacD thanks for chiming in -- @pitkling brings up some reasonable concerns, in my mind most relevant for whether this affects the current functionality of espanso.

  • how long have you been running espanso like this? Just since my post above or previously as well?
  • have you been an espanso user on non-NixOS systems? If so, have you found any differences in how it functions with this setup?
  • have you noticed whether or not espanso can pick up changes to its config with this configuration?

@KenMacD
Copy link
Contributor

KenMacD commented Sep 5, 2024

  • how long have you been running espanso like this? Just since my post above or previously as well?

Umm… I've had it running on and off for most of the last year, iirc. In the past I did run in to some issues around espanso showing a rainbow screen, so I had to locally bump the version for that. When it's not working in NixOS as a service I've just keep a espanso worker running in a terminal.

  • have you been an espanso user on non-NixOS systems? If so, have you found any differences in how it functions with this setup?

Sorry, only NixOS

  • have you noticed whether or not espanso can pick up changes to its config with this configuration?

No, it doesn't currently pick up change to the config files. I don't change them very often though. It also doesn't pick up when I put in another keyboard. In both cases I just systemctl --user restart espanso.

Not saying my solution is the best solution or anything. Just posting it as a workaround for people. One thing that would be nice to have in the module would be have an easy way to add things to the path. It took me a bit to figure out why my type: shell ones weren't working. (Or at the very least adding bash to the package)

@pitkling
Copy link
Member

pitkling commented Sep 8, 2024

[…] In any case, if the draft PR is not merged, I might just provide a flake with the module, so at least it will become easier to use this until we have a proper solution.

I finally got around to create a flake that exports the module from my PR #328890. So while we're waiting for one of the pending PRs #328890/#339594 (or a different one) to be finalized and merged, it should now be possible to get Espanso to work under Wayland simply by:

  • importing the module nixosModule.espanso-capdacoverride from my flake github:pitkling/nixpkgs/espanso-fix-capabilities-export
  • activating Espanso via its NixOS or Home Manager module like so:
    services.espanso = {
        enable = true;
        package = pkgs.espanso-wayland;
      };

Some notes:

  • You can import the NixOS module via something like the following in the flake.nix of your NixOS configuration:
    {
      inputs = {espanso-fix.url = "github:pitkling/nixpkgs/espanso-fix-capabilities-export";};
    
      outputs = { self, nixpkgs, espanso-fix, ... }: {nixosConfigurations.<yourMachine> = nixpkgs.lib.nixosSystem {modules = [espanso-fix.nixosModules.espanso-capdacoverride];};};
    }
  • This exported module version is enabled by default. Use programs.espanso.capdacoverride.enable = false to disable it.
  • By default, the module wraps pkgs.espanso-wayland and overlays it. You can also use another base package. E.g., if you pulled in espanso-wayland from nixpkgs-unstable under pkgs.unstable.espanso-wayland), you can set:
    programs.espanso.capdacoverride.package = pkgs.unstable.espanso-wayland
  • See nixos/espanso: fix wayland problems due to missing capabilities #328890 for details what this does and how it works.

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/espanso-daemon-problem/35309/27

@sachinchaudhary1808
Copy link

why do i get attribute wayland missing this config ?

  services.espanso = {
    enable = true;
    package = pkgs.espanso-wayland;
  };

@pitkling
Copy link
Member

why do i get attribute wayland missing this config ?

  services.espanso = {
    enable = true;
    package = pkgs.espanso-wayland;
  };

Are you on NixOS 24.05? If so, then it is because the services.espanso.wayland option got removed but a check for it remained. That is fixed in unstable by PR #316519. (Interestingly, in unstable the services.espanso.wayland option is back but not used anymore, because of a minor race condition; it will be removed for good in an upcoming commit).

@sachinchaudhary1808
Copy link

why do i get attribute wayland missing this config ?

  services.espanso = {
    enable = true;
    package = pkgs.espanso-wayland;
  };

Are you on NixOS 24.05? If so, then it is because the services.espanso.wayland option got removed but a check for it remained. That is fixed in unstable by PR #316519. (Interestingly, in unstable the services.espanso.wayland option is back but not used anymore, because of a minor race condition; it will be removed for good in an upcoming commit).

okay

@pbek
Copy link
Contributor

pbek commented Oct 2, 2024

Espanso works for me under Wayland with integrated GPUs from Intel or AMD (didn't need espanso-fix) under KDE Plasma 6 on unstable nixpkgs.

With an NVIDIA GPU it panics with:

[ERROR] thread 'detect thread' panicked at 'called `Result::unwrap()` on an `Err` value: NoCompositor': espanso-detect/src/evdev/sync/wayland.rs:42

espanso-fix didn't help with my NVIDIA problem.

@pitkling
Copy link
Member

pitkling commented Oct 3, 2024

Espanso works for me under Wayland with integrated GPUs from Intel or AMD (didn't need espanso-fix) under KDE Plasma 6 on unstable nixpkgs.

With an NVIDIA GPU it panics with:

[ERROR] thread 'detect thread' panicked at 'called `Result::unwrap()` on an `Err` value: NoCompositor': espanso-detect/src/evdev/sync/wayland.rs:42

espanso-fix didn't help with my NVIDIA problem.

@pbek Hmm, strange that the GPU seems to make a difference here. My laptop (Dell Latitude 7440) comes with an integrated Intel GPU and does experience the Wayland problems described in this thread. Are you sure there is no other difference between the systems that could cause the problems? Or maybe there is even a problem with the GPU detection in general on the NVIDIA system? What does espanso log show on both systems?

@pbek
Copy link
Contributor

pbek commented Oct 3, 2024

This the log on an Intel GPU Laptop:

08:03:32 [daemon(2161)] [INFO] reading configs from: "/home/omega/.config/espanso"
08:03:32 [daemon(2161)] [INFO] reading packages from: "/home/omega/.config/espanso/match/packages"
08:03:32 [daemon(2161)] [INFO] using runtime dir: "/home/omega/.cache/espanso"
08:03:32 [daemon(2161)] [INFO] system info: NixOS v24.11 - kernel: 6.6.52
08:03:32 [daemon(2161)] [WARN] keyboard layout watcher couldn't determine active layout.
08:03:32 [daemon(2161)] [INFO] watching for changes in path: "/home/omega/.config/espanso"
08:03:32 [daemon(2161)] [INFO] espanso version: 2.2.1
08:03:32 [daemon(2161)] [INFO] spawning the worker process...
08:03:32 [daemon(2161)] [INFO] binded to IPC unix socket: /home/omega/.cache/espanso/espansodaemonv2.sock
08:03:32 [worker(2175)] [INFO] reading configs from: "/home/omega/.config/espanso"
08:03:32 [worker(2175)] [INFO] reading packages from: "/home/omega/.config/espanso/match/packages"
08:03:32 [worker(2175)] [INFO] using runtime dir: "/home/omega/.cache/espanso"
08:03:32 [worker(2175)] [INFO] system info: NixOS v24.11 - kernel: 6.6.52
08:03:32 [worker(2175)] [INFO] binded to IPC unix socket: /home/omega/.cache/espanso/espansoworkerv2.sock
08:03:32 [worker(2175)] [INFO] using WaylandAppInfoProvider
08:03:32 [worker(2175)] [WARN] EVDEV backend is being used, but without enabling linux capabilities.
08:03:32 [worker(2175)] [WARN]   Although you CAN run espanso EVDEV backend as root, it's not recommended due
08:03:32 [worker(2175)] [INFO] monitoring the status of the daemon process
08:03:32 [worker(2175)] [WARN]   to security reasons. Espanso supports linux capabilities to limit the attack surface
08:03:32 [worker(2175)] [WARN]   area by only leveraging on the CAP_DAC_OVERRIDE capability (needed to work with
08:03:32 [worker(2175)] [WARN]   /dev/input/* devices to detect and inject text) and disabling it as soon as the
08:03:32 [worker(2175)] [WARN]   initial setup is completed.
08:03:32 [worker(2175)] [WARN] unable to determine keyboard layout automatically, please explicitly specify it in the configuration.
08:03:32 [worker(2175)] [INFO] using EVDEVSource
08:03:32 [worker(2175)] [INFO] Querying modifier status...
08:03:33 [worker(2175)] [WARN] unable to determine keyboard layout automatically, please explicitly specify it in the configuration.
08:03:33 [worker(2175)] [INFO] using EVDEVInjector
08:03:33 [worker(2175)] [INFO] using WaylandFallbackClipboard
08:03:33 [worker(2175)] [WARN] Can't read from device /dev/input/event20, this error usually means the device has been disconnected, removing from epoll.
08:03:35 [worker(2175)] [ERROR] Unable to show notification: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
09:05:09 [daemon(2161)] [INFO] configuration change detected, restarting worker process...
09:05:09 [worker(2175)] [INFO] engine eventloop has terminated, propagating exit event...
09:05:09 [worker(2175)] [INFO] waiting for engine exit mode...
09:05:09 [worker(2175)] [INFO] exiting worker process...
09:05:09 [daemon(2161)] [INFO] spawning the worker process...
09:05:09 [worker(67671)] [INFO] reading configs from: "/home/omega/.config/espanso"
09:05:09 [worker(67671)] [INFO] reading packages from: "/home/omega/.config/espanso/match/packages"
09:05:09 [worker(67671)] [INFO] using runtime dir: "/home/omega/.cache/espanso"
09:05:09 [worker(67671)] [INFO] system info: NixOS v24.11 - kernel: 6.6.52
09:05:09 [worker(67671)] [INFO] binded to IPC unix socket: /home/omega/.cache/espanso/espansoworkerv2.sock
09:05:09 [worker(67671)] [INFO] using WaylandAppInfoProvider
09:05:09 [worker(67671)] [WARN] EVDEV backend is being used, but without enabling linux capabilities.
09:05:09 [worker(67671)] [INFO] monitoring the status of the daemon process
09:05:09 [worker(67671)] [WARN]   Although you CAN run espanso EVDEV backend as root, it's not recommended due
09:05:09 [worker(67671)] [WARN]   to security reasons. Espanso supports linux capabilities to limit the attack surface
09:05:09 [worker(67671)] [WARN]   area by only leveraging on the CAP_DAC_OVERRIDE capability (needed to work with
09:05:09 [worker(67671)] [WARN]   /dev/input/* devices to detect and inject text) and disabling it as soon as the
09:05:09 [worker(67671)] [WARN]   initial setup is completed.
09:05:09 [worker(67671)] [WARN] unable to determine keyboard layout automatically, please explicitly specify it in the configuration.
09:05:09 [worker(67671)] [INFO] using EVDEVSource
09:05:10 [worker(67671)] [INFO] Querying modifier status...

On my main working machine, I switched back to X11, because not having Espanso is not an option. 😁
So I currently can't do an espanso log there.

They use almost the same nixcfg (see above links).

@pitkling
Copy link
Member

pitkling commented Oct 3, 2024

@pbek: I had a quick look at your config. I think I know at least why it is working on some of your systems under Wayland for you without the fix from this thread. You already have another workaround for the Wayland issue in your configuration. The espanso-fix-solution from my comment basically replaces the workaround from your configuration. Instead of the more lax permissions for /dev/uinput (which is perfectly fine for personally use on a private system) it gives the espanso binary the corresponding linux capabilities; see also the warning related to EVDEV backend is being used, but without enabling linux capabilities in your log.

Concerning your problem with the NVIDIA GPU: That's probably a different problem. I'd suggest to open a new issue and, if possible, somehow get a more complete log file for the failing machine via espanso log and/or journalctl --user -u espanso (not sure whether this is more verbose or not). Also dmesg might be interesting, maybe something's not right with the NVIDIA driver?

@pbek
Copy link
Contributor

pbek commented Oct 3, 2024

@pitkling, thank you! I opened #346221.

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/home-managers-espanso-module-does-not-create-config-directory/51170/14

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-espanso-on-wayland-nixos/55268/2

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/4910

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug Something is broken
Projects
None yet