Wayland Roadmap

[kwhat]

It looks like Wayland is now the default option on several distributions and supporting it is becoming more of a priority. There are still a number of features missing from the Wayland ecosystem and fragmentation is going to be a real problem for any software trying to support multiple compositors. Getting this library to work with Wayland is going to require a combination of runtime dependency loading, XDG Portals and cutting features that aren't available. I have created this bug to track the progress of the outstanding issues. Please add any information or ideas you may have to this ticket.

• hook_create_screen_info(unsigned char *count);

Support Added

Support was added using wl_output_interface provided by Wayland. This provides enough information to satisfy this requirement.

Gnome/GTK Support

gdbus call -e -d org.gnome.Mutter.DisplayConfig -o /org/gnome/Mutter/DisplayConfig -m org.gnome.Mutter.DisplayConfig.GetCurrentState

We need to use something like glib g_variant_get() https://github.com/fzwoch/obs-gnome-screencast/blob/master/gnome-mutter-screencast.c#L90 to pare this information.

Plasma/QT Support

Unknown

X11

Continue to use XRandr / Xinerama but change the code to use dlsym and convert from a compile time dependency to a runtime dependency.

Other Compositor

Unknown, we may need to cut this feature unless we can use XDG Portal to acquire monitor info.

---

• hook_get_auto_repeat_rate();

Gnome/GTK Support

Unknown

Plasma/QT Support

Unknown

X11

Continue to use XkbGetAutoRepeatRate / XF86MiscGetKbdSettings but change the code to use dlsym and convert from a compile time dependency to a runtime dependency.

Other Compositor

Unknown. As it currently stands, this feature will likely be removed from all platforms.

---

• hook_get_auto_repeat_delay();

Gnome/GTK Support

Unknown

Plasma/QT Support

Unknown

X11

Continue to use XkbGetAutoRepeatRate / XF86MiscGetKbdSettings but change the code to use dlsym and convert from a compile time dependency to a runtime dependency.

Other Compositor

Unknown. As it currently stands, this feature will likely be removed from all platforms.

---

• hook_get_pointer_acceleration_multiplier();

Gnome/GTK Support

Unknown

Plasma/QT Support

Unknown

X11

Continue to use XGetPointerControl but change the code to use dlsym and convert from a compile time dependency to a runtime dependency.

Other Compositor

Unknown. As it currently stands, this feature will likely be removed from all platforms.

---

• hook_get_pointer_acceleration_threshold();

Gnome/GTK Support

Unknown

Plasma/QT Support

Unknown

X11

Continue to use XGetPointerControl but change the code to use dlsym and convert from a compile time dependency to a runtime dependency.

Other Compositor

Unknown. As it currently stands, this feature will likely be removed from all platforms.

---

• hook_get_pointer_sensitivity();

Gnome/GTK Support

Unknown

Plasma/QT Support

Unknown

X11

Continue to use XGetPointerControl but change the code to use dlsym and convert from a compile time dependency to a runtime dependency.

Other Compositor

Unknown. As it currently stands, this feature will likely be removed from all platforms.

---

• hook_get_multi_click_time();

Gnome/GTK Support

Bus Name: org.freedesktop.portal.Desktop
Object Path: /org/freedesktop/portal/desktop
Interface: org.freedesktop.portal.Settings

Read("org.gnome.settings-daemon.peripherals.mouse", "double-click")

Plasma/QT Support

Unknown

X11

Continue to use XtGetMultiClickTime / XGetDefault but change the code to use dlsym and convert from a compile time dependency to a runtime dependency.

Other Compositor

Unknown. This is kind of a critical feature for tracking click count for mouse events. If we cant find a solution to this problem, we will have to drop the click count tracking from events.

[kwhat]

This project is probably dead and Wayland killed it. There is no feasible why to support non-retaliative mouse movement in a compositor agnostic way. I naively though that if I could obtain the current pointer position I could calculate the subsequent position based on the relative movement. Unfortunately, that isn't possible due to acceleration and scaling applied at the compositor level. The only way to continue development would be to target each compositor individually which would require porting this library to C++ causing even more issues for everyone using it. You might be asking yourself: "But what about XQueryPointer?" After all, Wayland supports most of the X11 API right? Well, it doesn't work properly in my tests. Take the following basic C program and build it with gcc -lX11 test.c

#include <X11/Xlib.h>
#include <stdio.h>

int main(void) {
    Display *disp = XOpenDisplay(NULL);
    Window screen = DefaultRootWindow(disp);  // to To access the home screen 
    int x, y, winX, winY;
    unsigned int mask;
    Window child;
    XQueryPointer(win, screen, &screen, &child, &x, &y, &winX, &winY, &mask);
    printf("\nx: %d, y: %d\n", x, y);
    XCloseDisplay(win);
    return 0;
}

It doesn't matter where I move the mouse, I always get x: 2560, y: 547 as my output. Why 2560 and 547? I have no idea but the same program works correctly under X11. If we ignore that problem and just go with relative mouse movement, there really isn't much difference between where this library ends up and the existing and better supported HID libraries that are already out there. Unless a project like xdg-desktop-portal can come up with a solution, I see no way forward with this project or its child projects. Wayland was a nice idea created by a bunch of academics who have spent more than a decade solving a problem no one had.

[raphaelmenges]

I have no idea about the details, but the mouce crate seems to have a Wayland compatible mouse hook: https://github.com/emrebicer/mouce/blob/master/src/nix/uinput.rs

[kwhat]

@raphaelmenges This comment basically sums up the problem.

// For some reason, absolute mouse move events are not working on uinput
// (as I understand those events are intended for touch events)
//
// As a work around solution; first set the mouse to top left, then
// call relative move function to simulate an absolute move event

The reason this doesn't work is because uinput has no idea what the resolution or scaling of the monitor is. The "work around solution" only works if the scaling of the monitor is at 100% and the mouse sensitivity / acceleration is 1:1. This is the same problem that I've been battling with. I can determine the current mouse position and monitor sizes and ordination, however, if I just move the pointer +/- x or y pixels based on uinput we eventually get out of sync. How many screen pixels +217 x, +445 y ends up being from 0, 0 is not necessarily 217, 445. I could very easily be 434, 890 if the acceleration is 2x.

Wayland seems like it was very much designed to kill access to information programs like this library depend on, all in the name of security! They set out to prevent key logging and still failed to do so. They feel that raw input (HID) should be more than enough information for any program requiring this sort of info and that programs should only be aware of the windows they own. It is very much of a Java style sandbox for the entire graphical environment. You can track mouse movements across a window you draw with the correct scaling and acceleration, but as soon as you leave the bounds of that window, the input is no longer available. I tried to use this concept to draw a window that covers the entire screen for a frame to acquire the absolute position and then use the uinput relative movement to track the mouse in a similar way as they code you posted, and it works to get the position but fails as the mouse moves if I play with scaling or acceleration. Because both of these are controlled by the compositor. It is up to Plasma, Gnome or one of the myriad of other compositors to provide that information and none of them do at this time. Even if they did it would be up to the compositor to provide the API, which all but guarantees there will be N+1 ways of pulling it in with no overlap which likely makes the fragmentation problem for this library insurmountable.

Let me ask all of you a question: What functionality does this library provide that cannot be accomplished with raw HID? On the surface, it seems like the only thing it provides over HID are absolute mouse coordinates and language support for key typed events. Why not just use HID? If I was to sit down and create this library again, I would probably just use HID instead as it fulfills all of the original requirements and several portable libraries already exist.

[raphaelmenges]

Thank you very much for the explanation @kwhat! I often read comments that Wayland can makes things more complicated or even impossible and it is interesting yet sad to read a well laid out example like this.

[ZakharEl]

Using the evdev library should make this work with both X11 and Wayland. Heck, you'll be able to use this without a desktop environment of any kind on Linux with evdev. Rdev for the rust library uses evdev with its grab function.

[stupidratcreature1000]

I hate x11

[Corben78]

The OBS plugin input-overlay is using your library. Unfortunately there is an issue with multiple HIDs on Gnome + X11 (see: https://gitlab.gnome.org/GNOME/mutter/-/issues/398#note_2423890) which is not present on Wayland.
As the purpose is to show keypresses and maybe mouse clicks, it would be helpful enough if libuiohook would work on Wayland without mouse movement events (see: univrsal/input-overlay#464 (comment))
Is there a chance libuiohook could be updated so it works on Wayland for input-overlay?

[Descolada]

This project is probably dead and Wayland killed it.

I do not believe that Wayland will kill it. Having a cross-platform solution to hook and synthesize input is very powerful and useful, and even if not possible in Wayland then Windows, MacOS and X11 are all still good targets. In my opinion the best course of action at the moment is to simply wait and focus on other platforms and X11, and ignore Wayland. I'll try not to rant on the topic of Wayland restrictedness because it's been done so many times already elsewhere, but hopefully in time they will see the error of their ways and implement the basic tools needed to make the features of this library possible. If they don't then they will ignore the needs of those who require accessibility tools (eg visual impairment), users who desire greater control over their environment (eg complex desktop automation), and who knows who else. Personally I wouldn't put any effort into implementing Wayland support until the basic tools needed for this library are implemented by them in a standardized manner, and a manner which wouldn't require using different implementations for different compositors.

[kwhat]

Using the evdev library should make this work with both X11 and Wayland.

I have an evdev implementation working, the major issue is try to determine the absolute pointer position for mouse coordinates. Currently, the only way I can return mouse position is relative movent. Ex: -1, 0 or +1. Figuring out where the mouse is relative to the desktop requires a lot more work. I can determine where the mouse is by basically painting a full screen window on each monitor and then figuring out where the mouse is on those windows, but doing that on every mouse input isn't really a viable option. I was initially thinking that I could use the window overlay method once on startup and then calculate the mouse position in real time, but each compositor can setup different scaling or acceleration and then the coordinates I am tracking wont match what is actually being rendered on the screen.

If this ever gets resolved, it will likely be though a common set of exposed d-bus properties that can be leveraged with some XDG Portal interface. This would require Plasma, Gnome and all the other players expose some API to get this information and something else like XDG to persist it to d-bus so everyone else can write code around these settings. Last time I looked, QT didn't have an API for accessing any of that info and GTK only had info for double click interval in the best of circumstances.

I need to look around and see what I did with all this proof-of-concept code. I think it is sitting in some VM I have parked on my workstation.

[TolikPylypchuk]

@kwhat have you considered using libinput instead of libevdev for input monitoring? libinput provides a higher level of abstraction over libevdev and it does provide absolute mouse coordinates. It is also available in basically all Linux distributions and variants, as far as I know.

I've been looking into Wayland support for SharpHook, a .NET wrapper of libuiohook (albeit my own fork as I've added some additional stuff there), and libinput seems promising. I'm also working on some proof-of-concept code to see whether I can come up with something.

Here's what a global hook based on libinput looks like:

void handle_events(struct libinput *li) {
    libinput_dispatch(li);

    struct libinput_event *event;

    while ((event = libinput_get_event(li)) != NULL) {
        enum libinput_event_type event_type = libinput_event_get_type(event);

        if (event_type == LIBINPUT_EVENT_POINTER_AXIS) { // This event type is deprecated and should be skipped
            libinput_event_destroy(event);
            continue;
        }

        switch (event_type) {
            case LIBINPUT_EVENT_POINTER_MOTION_ABSOLUTE:
                struct libinput_event_pointer *pointer_event = libinput_event_get_pointer_event(event);
                double x = libinput_event_pointer_get_absolute_x(pointer_event);
                double y = libinput_event_pointer_get_absolute_y(pointer_event);

                printf("Mouse moved to (%i, %i)\n", (int)x, (int)y);
                break;

            // Other cases
        }

        libinput_event_destroy(event);
        libinput_dispatch(li);
    }
}

As you can see in the handle_event function, libinput provides the LIBINPUT_EVENT_POINTER_MOTION_ABSOLUTE event type, and the libinput_event_pointer_get_absolute_{x,y} functions. Now, the problem with these functions is that they always return values in the range from 0 to 65535 which is of little use to us. But there are also the libinput_event_pointer_get_absolute_{x,y}_transformed functions which accept the width/height of the screens and normalize the absolute pointer position to that range.

OK, now we need to know the total width and height of all screens in order to get the normalized pointer position. On Wayland, we can use the core protocol to enumerate all screens and get their sizes and positions (using wl_output). This approach alone won't work since the core protocol doesn't support fractional scaling. What will work though is also enumerating the screens using the xdg-output-unstable protocol. Yes, this protocol is unstable, but apparently, most protocols in Wayland are unstable, and this protocol seems to be widely implemented. Having all screens, their sizes, and locations will allow us to calculate the total width and height of the screen surface and normalize the mouse coordinates for the libinput_event_pointer_get_absolute_{x,y}_transformed functions.

I've attached a small project which contains the libinput global hook and Wayland screen enumeration. I haven't implemented the total screen size calculation yet - that's what I'm planning on doing next.

sandbox.tar.gz

You can build and run this project with the following commands:

cmake -B build
cmake --build build
sudo -E XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR ./build/sandbox

libinput requires additional privileges, so sudo is used, but root doesn't have a Wayland session by default, so we need to add XDG_RUNTIME_DIR to the command. I've also tried running this program under a user that's added to the input group and it works as well.

[TolikPylypchuk]

Regarding input event simulation, I think libevdev can be used as it provides abstractions over /dev/uinput.

There is also libei which seems to be the standard way to emulate events in Wayland. At least, most Wayland compositors seem to support it. This approach depends on the compositor though, as it may disallow emulating specific events, or it can simply choose to ignore them.

I think that it's OK to use /dev/uinput which requires additional privileges since we already require additional privileges for the global hook. Like with the input hook, /dev/uinput doesn't necessarily require root. We can instead add a udev rule that adds access to /dev/uinput to a specified group:

sudo touch /etc/udev/rules.d/99-some-group.rule
sudo echo "KERNEL=="uinput", MODE="0660", GROUP="some-group" > /etc/udev/rules.d/99-custom-uinput.rule
sudo udevadm control --reload-rules

Adding this rule does require root, but it's a one-time action.