State of Linux (Cont) #107
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This is a continuation of #1
The current top priorities for our Linux build are:
There are also some features that haven't been ported over yet:
Nice to have:
(List last updated on 13-01-2023)
When building from source we use vcpkg for managing dependencies on all platforms and our officially supported and recommended compiler is Clang 12 or higher (and MSVC on Windows). With the
-DENABLE_VCPKG=OFFoption you can use system libraries instead of vcpkg but we generally discourage it as we prefer having builds with the same exact library configuration across all platforms for simplicity of debugging. We also won't officially support distributing Cemu as a distro package. Of course package maintainers are free to package it, but our focus is to streamline the troubleshooting and bug reporting process and it's way easier if there aren't dozens of different builds.Lastly, when making a new Linux-specific issue try to name it
[Linux] <Bug description>.ext4 has case-folding support. Would it be possible to make it a requirement for Cemu? Users would have to store stuff accessed by the Wii U in a separate volume with this feature enabled though, which could be hard to do.
That's a very bad idea. This requires, as you said, a seprate volume. Users might want to use a case-sensitive filesystem, so they need a own partition/drive just for Cemu.
It is also not very end user friendly to show a Message to new Linux Users, who want to play their Games: "Welcome to Cemu on Linux! Linux is a user friendly OS and not only a nerd OS as everyone says. Now please reformat your drive and create a partition with a case-insensitive filesystem to use Cemu."
Especially Steam Deck Users will have fun.
Yeah I was just pointing out that the thing exists. It would be cool if Cemu supported it though, as I doubt that any workaround for case-folding would be more performant than an in-kernel implementation.
Steam Deck wouldn't have this issue afaik since it's using a case-insensitive filesystem. But yeah, really only Wine does this to my knowledge and they do heavy caching using a virtual filesystem to prevent it from hurting performance.
To have a bit more context: what data used by Cemu needs casefolding support? Only the mlc01 folder? Only the games one? Both? Something else?
Case-insensitivity is needed for all files that map to the virtual Wii U file system in some way. Which is mlc and game folder files. So basically a majority of our file access code done by many different HLE modules
This might be too hacky and im not too much of a programmer but cant all the files and directories which will be mapped to the wii u filesystem just be all uppercased or all lowercased
That wouldn't work because the case must be kept intact. Anything else is not accurate emulation and some games might rely on case-sensitive checks internally
Edit - I was wrong
Another possibility though - https://www.collabora.com/news-and-blog/blog/2020/08/27/using-the-linux-kernel-case-insensitive-feature-in-ext4/
How do other emulators cope with this?
It seems that y'all are talking about the Wii U's case-insensitive FS, but I'll throw this issue I'm having here since I think it's related to the main discussion:
About the online play feature(s), has anyone managed to get online actually working on Linux? I'm using the pre-compiled binary for Ubuntu on Arch, and after symlinking the missing dependencies the games I play work fine but when I try to go online by copying
mlc01/(recursively, of course),otp.binandseeprom.binit just throws this error in the account settings:My host machine is running Arch. Online works just fine on a Windows 10 VM (it says that the account is valid on Windows), so it's definitely not me following the instructions incorrectly. Does anyone know what could be causing this issue?
You can create an image file and format it as ext4
To do it properly someone would need to write a Wii/U filesystem driver just like this one for FATX...
https://github.com/mborgerson/fatx
... which uses a userspace FUSE based library and can seamlessly handle mounting Xbox & Xbox 360 images directly.
I'm on latest ArchLinux and compiled Cemu with gcc and g++ and make. Few issues:
Tried it on my NTFS drive and ext4 both have the same issue
This may work for the mlc01 folder, but not for the games one. The latter has to be easily accessible to the user. Still, better than requiring a different partition :)
Mh, I believe that the issue here is the opposite; you don't need to mount a Wii U filesystem on the host, but you have to read the host's filesystem on the (emulated) Wii U
Wine handles the case mismatch problem by recursively iterating directories and comparing file/folder names manually. With lots of caching it can be made fast.
@imsl0wer
Are you using Vulkan or OpenGL as the graphics API?
There's no functional difference between the two, once a filesystem is mounted on the host it is accessible by both the host and the emulator. So you create a 32GB image, format it for the WiiU, mount it with FUSE and its good. Emulator can read it as a native volume and the user can drag/drop files as needed.
Edit Still not exactly ideal though since you could only ever hold 32GBs worth of games at one time and would need to swap them out as needed. Also any kind of image solution would break the GUI game list.
Nope, they're currently stubbed on Linux, like a few other things. One of the things listed by Exzap. Task is still up for grabs for anyone who wants :p.
@Crementif I managed to get Cemu to say that my account is valid by compiling from source. It was a problem with the binary in the releases. I'll update if I encounter any issues, but the problem I mentioned is fixed for now.
I don't know what does Wine implement below the hood, but, maybe, a small SQLite db can be used to cache the entire tree?
For Unix, Redis would be a better choice than SQLite.
Using Vulkan, OpenGL does the same thing
XDG Base Directory Specification
The files that Cemu writes and reads from should not be next to the binary. It should be in the XDG directories.
graphicsPacksshould go in XDG_DATA_HOME: e.g.~/.local/share/cemu/graphicsPacks, as shouldmemorySearcherandmlc01. Controller profiles and custom game profiles should also go here.log.txtshould be written here as well or don't do that and write to syslog.settings.xmlshould go in XDG_CONFIG_HOME.keys.txtshoudl also be placed here.shaderCacheshould go in XDG_CACHE_HOME.Why would there be any need to persistently store such a cache? I can't think of any good reason for that.
Nor would there be any need for a database engine to manage it.
To me they look more like configuration files than general data, so why not
~/.config/cemu/?Mh, shouldn't logs go to
~/.cache/cemu/?Yeah, cache is more for what programs do internally; if the files it generates are usable/useful for the end user it shouldn't be stored in cache generally.
Good news! I had a quick chat with Krisman, the guy who implemented casefolding (case-insensitive) support for ext4, and he said the following:
All things considered, I think that using an ext4 image managed by Cemu (as also mentioned by @JPabloLassala) might be the best solution so far. Opportunistic use of the casefold feature on the host's filesystem might not be helpful in practice, because as far as I know there are no distros that ship with this feature enabled by default, and enabling it requires unmounting the filesystem (that could be hard/impossible to do for users that store everything in a single partition, like me).
Using an image would likely mean that the mlc01 and games folder won't be normally accessible by the user, but I guess Cemu could transparently mount and unmuont the image when launched and closed.
IMO using fs built-in casefolding when available and using libcasefold as a fallback sounds good. It will be much more user-friendly than images, because of images not being always mounted or being too small when moving new games in. Also it sounds like libcasefold would benefit from having a big project like Cemu using it.
In theory yes, but I fear that in practice it would lead to poor performance, as kernel casefolding is not that widely available by default. I may be wrong though, as the only way to know is to implement the feature :)
True. Both ext4 and qcow2 should support live growing though, so moving new games in shouldn't be an issue.
Also true, and I'd really like to see libcasefold improve too.
I don't think any solution would really suffer from performance issues, even when translating each path manually (and not caching). The only thing that might suffer is loading times and that's not really critical (correct me if I'm wrong).
By the way, not saying that this isn't an important issue to solve, but is there even a game which regresses from not having case-insensitive support?
Yes I think those should go there instead.
Not sure. There is not a good spot for logs in the XDG specification. It makes me think they want everything writing to syslog (preferably journal from systemd).
Also I forgot to mention, the CMake files have no
install()targets. It should be standardinclude(GNUInstallDirs)and putting the main bin and stuff inbin/(in the repo; should be in/usr/share/cemu) anddist/linuxinto the correct locations.Never used this before, but XDG says logs go in
XDG_STATE_HOME, defaulting to~/.local/state.I have been also been playing on a case-sensitive filesystem and I've been waiting to see the "case sensitivity induced error" so I can know what one would look like, so I would second this. I'd like to know a reproducible scenario in which playing from
mlc01stored in a case-sensitive filesystem results in a symptomatic bug, so that I can test whether any given potential solution is correctly applied and actually working. I hope it doesn't require a specific game to observe, otherwise people who don't own that game won't be able to test the solutions... or we could just do it like WINE does and have a solution that does not have any of those restrictions. Remember, we always want to be as user-friendly as possible.
Case-insensitivity can be emulated transparently by doing the path lookup manually. I'll try to explain the algorithm:
Assuming we want to look up
./content/data/foo.binbut the host filesystem stores it under./CONTENT/DATA/FOO.BIN. The algorithm starts at the root of the path and recursively checks each sub-directory for the actual case of the next lower child. The steps would go something like this:./in the host file system and case-insensitive compare all file and folder names againstcontent. This gives youCONTENT../CONTENT/fordata. This gives usDATA../CONTENT/DATA/forfoo.bin. This gives usFOO.BIN../CONTENT/DATA/FOO.BINwhich we then use to access the file.The actual algorithm is quite simple (it's really only one step but repeated recursively). It would likely be fast enough for our needs even without any kind of caching of previously looked up paths. And if not, the priority is to get it to work and then we can always make it faster afterwards. If nobody else steps up to the task I will implement this in the future, but I'm currently swamped with other tasks
Pick any game without an update, go to it's content folder (it must be in extracted format). Rename a bunch of folders and files to have random upper and lowercase letters. This will break the game on case-sensitive file systems. But I doubt you even have to go that far, most games don't have coherent case to begin with and probably crash on boot.
System logs should go in /var/logs, user logs should go in /home/$USER/.local/share/appname.
Since XDG is about interoperability between different DEs it doesn't have a spec for logs, it would be redundant. Unix already has well established log locations.
Thats news to me and honestly, I've never encountered an app in the wild that uses this part of the spec (on my system right now that folder doesn't even exist). Heck even X dumps its logs into /home/$USER/.local/share/Xorg.
Still I guess it does make sense, logs are indeed state files.
Edit - Using the ~ alias is frowned upon when coding paths into an application, if the user somehow runs as root you will end up writing to the /root folder and this can cause issues for the entire environment. Always use the full path with the $USER variable, this way if the user somehow runs as root and avoids the fact the root environment doesn't have any XDG vars set the operation will still fail since /home/root doesn't exist.
Yeah I know this is kinda nitpicky but remember, users are almost always idiots and if it can happen its likely someone will do it.
Yeah, this doesn't look complex at all, but how it this going to handle non-ascii paths? This article contains some helpful information about how the casefold-for-ext4 team decided to handle cases like the aforementioned one. It doesn't look trivial.
Since we don't want to use images (that I understand how they could be confusing to users), what's the benefit in implementing case-insensitivity manually instead of using libcasefold? Krisman has also said that he would love to start working on it again if someone would actually use it.
Using libcasefold instead of manually emulating case-insensitive path lookups would have the following benefits:
All Wii U paths are ASCII.
Custom solutions are always fine-tuned for the specific problem they are trying to solve which has all sorts of benefits, usually it's better performance and much greater flexibility. I know that outside of high performance software, it's common to reuse as much as possible, but we don't really do this in Cemu. There is a little bit of reinventing the wheel here and there and usually the end result is better for it. That said, I'm not against dependencies in general but the problem needs to be sizeable enough to justify introducing another dependency. And this isn't. My personal opinion of course.
It's obvious that we would not do translation when it's not necessary.
Good, I wasn't sure about it.
I'm not against custom stuff either, if it makes sense to do this for Cemu than it is indeed appropriate. I was just trying to make sure that this was the case :)
Perfect.
I finished implementing XDG https://github.com/cemu-project/Cemu/pull/126
I'm hitting these assertions from GX2_Texture when going in game in BOTW
cemu_assert_debug(texUnit < Latte::GPU_LIMITS::NUM_TEXTURES_PER_STAGE);with texUnit == 18
Are these assertions valid? Continuing past them in the debugger gets in game without visible issues.
When building did you set
-DPUBLIC_RELEASE=ON?That would disable these assertions, but IIRC I hit some other crash with that on that I didn't look into.Nevermind, that does work. Might have just ran into something intermittent before.
Why would things in
/rootbe a problem? That's the home directory for therootuser, so it's exactly the right place to put things.Don't ever hardcode
/home/$USER, that's going to break for anyone who has a different home directory set in/etc/passwd.I also cannot get Star Fox Zero to start. Just gets stuck at the startup/logo screen.
Cemu crashes on loading games in Ubuntu too, though at least it compiles under certain circumstances. It will take some time before games actually run given that the linux side of the codebase is still quite new and there are plenty of issues remaining to resolve (hence this report),
With
cf598e38c1we now emulate case-insensitive file/directory lookup on case-sensitive file systems. The fix is baked into our core file access helper (FileStream) and thus covers almost all operations. The only thing remaining is to refactor legacy code that directly accesses files (via fstream or fopen) to use FileStream instead.amazing progress, honestly. Only thing left is the flatpak/appimage (atleast only thing left that causes the majority of the linux issues)
Wayland support, without going through XWayland (in which no games boot), isn't implemented yet either.
Edit: nevermind looks like it works through XWayland for others, proper Wayland support not implemented though
What do you mean? It's working for me through xwayland. (on sway)
Wayland support != XWayland
That was not my point. I read Latrolage's post as saying games don't work in xwayland, which doesn't seem to be true.
Indeed if games boot on XWayland
Titles are starting to work in testing, at least on Ubuntu-based systems, and Wayland or not, I'm impressed that they're already working so well. It's not just that rendering seems as smooth and GPU-efficient as on Windows, but also secondary features such as rumble and motion (on a DS5 pad) that I never expected to be functional so soon given the intra-platform library differences.
I don't want to derail the intent of this thread, but I think it speaks to the quality of the codebase from Exzap and the original team that so few changes had to be made to it, relatively speaking, to approach performance and feature parity so soon after source release. This isn't a common thing among projects that take this path.
Okay, back on point: in short, the "State of Linux" is already quite good, and continued testing will help nail it all down.
Will there be a arch Linux PKGBuild or releases soon? The only one available is Cemu on the AUR but that uses wine. This would also help Linux user who haven’t compiled before and are used to the AUR.
We'll officially release easy ways of accessing Cemu on Linux, using appimage and/or flatpaks, when Cemu 2.0 is in a good state of releasing.
I haven't tested Cemu on Linux in a bit, but afaik it still suffers from:
On a few of the issues you mentioned:
Crashing: There's an occasional need to delete the shader cache to avoid crashing. I haven't been able to trace the cause. Also, some games (e.g. DK TF) crash consistently on loading even with a fresh shader cache, as reported.
Motion controls: as a workaround to build Cemu on Ubuntu platforms, sdl2 can be removed from vcpkg.json, i.e. use system sdl2 instead (2.0.20 on Ubuntu 22.04, for example). However, motion works in this case.
I've already compiled since the first days, with errors, also now it compiles fine. Just keep eyes on dependencies, then you can do it.
Just want to give a clarification about something I (unintentionally) might've spread, but Exzap informed me about it and said there's not really a reason for there to be a performance regression in this 2.0 version. After benchmarking things myself between Cemu 1.26.2f and Cemu 2.0 on Windows I was fairly surprised to see that I was getting the same FPS (in BotW at least).
Though, it's important to note that from Cemu 1.27.1 onwards accurate barriers is enabled by default (which you can find in Cemu's toolbar under Debug) which might also be one of the reasons why people thought their games were running worse. You can toggle the option in-game to see the FPS difference, but it fixes quite a few effects which is why it's now by default enabled.
This does mean that Linux builds might have some performance issues if the wine version is indeed running better (which from what I've heard is the case, but I could be wrong). If anyone could test the performance difference on their system if they build it from source using the latest build methods in the build.md file, that'd be awesome. Make sure that the settings all match too obviously.
I can confirm that using system sdl2 fixes motion controls (DualSense PS5 Controller) on Ubuntu-based systems.
#286
./Cemu: error while loading shared libraries: libsepol.so.1: cannot open shared object file: No such file or directory
None of the libsepol packages I could find on Ubuntu did anything to fix the issue.
GCC worked compiling under PopOS, clang did not, tried using clang 12 as well.
Il giorno ven 23 set 2022 alle 19:04:57 -07:00:00, Dominic
@.***> ha scritto:
That's probably because Cemu is built on an older Ubuntu version that
links to libsepol1, while starting from Ubuntu 22.04 only libsepol2
is shipped. You can either install the libsepol1 package from an older
Ubuntu release (but you should't, as it can mess with your system
package manager), or you can re-compile Cemu from source.
You can work around the libsepol.so.1 by sym linking (https://github.com/cemu-project/Cemu/issues/1). ABI must somehow work out to be the same as I was able to get it working on ubuntu 22.04.
I've ran into cemu crashing for different reasons when manually compiling with GCC and Clang.
I still have issues with configuring controllers and no audio with the released tarball.
Please don't, it will create all kinds of bugs. The soname was bumped because the two ABIs are incompatible. Just install the older version, or download the library and put it in a directory pointed by the
LD_LIBRARY_PATHenvironment variable.If I may:
First, a note: I'm not at all familiar with Cemu's code structure so I don't know how feasible this would be.
Also, this may be too slow for emulation purposes, but still, here's my possible contribution.
I'm working on an open source project (Julius) which had the same issue. The project is a reimplementation of the original Caesar III game for Windows, and it also had the same case sensitivity problem when porting to other platforms.
In our case, the solution was to:
As mentioned, this can be slow as there's a lot of string comparisons going on. However, the results can be cached, and since Cemu needs to track file accesses from the games during gameplay anyway, the cache can always be kept updated.
Hopefully this is helpful!
I built this on Linux using the documentation, but when I go into
Options -> General settingsit crashes with this output:Before that it also kept crashing when I was trying to choose file directories on first Cemu startup.
From that error it seems like there is a issue with your system theme
The theme thing sounds like a benign warning to me.
I'd rather investigate the Intel driver. Does it happen when you override the mesa driver to llvmpipe/lavapipe via
MESA_LOADER_DRIVER_OVERRIDE? (Obviously won't be playable but if those don't crash we know it's the drivers' fault.)I don't get those messages about Intel anymore for some reason. I also ran
MESA_LOADER_DRIVER_OVERRIDE=llvmpipeas well asMESA_LOADER_DRIVER_OVERRIDE=lavapipebut I still have the same issue.However, I do see more output that I didn't post before. I'm not sure if I just didn't see it or what, but here it is:
(Cemu_release:4806): Gtk-CRITICAL **: 13:41:30.547: gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbarThe above line appears above the other output I posted.
The latest performance improvements (#370) brought performance in BotW up to parity with the Windows version, at least on my Steam Deck. Great to see!
Got v2.0.9 opening on Ubuntu 22.04.1 by copying libsepol.so.1 and libffi.so.7 to /usr/lib/x86_64-linux-gnu (used sudo find / -name to just copy them from snap).
Then general settings would crash it, along with trying to open botw. So went into settings.xml and changed, under Graphic tag, to the following:
So, looks like only OpenGL wants to run for me. I get some vulkan segmentation error when on api 1.
Notice the 3 in device - not all 0's.
Trying to mess with the settings in the UI reverts them to all 0's and causes crashes again.
Sound doesn't seem to want to work.
PS5 controller works great.
Save files and shader cache copied from Windows works for botw.
Using an AMD Ryzen 7 PRO 4750U.
Looking really good so far! Just a few snags.
AMD FSR upscale support?
@Docmine17 That has nothing to do with Linux support.
Using Wayland and Nvidia I got cemu and botw to work partially. They start up and run great, however:
log
Please open up a separate issue and provide more details like operation system, how you compiled it/whether you used the appimage etc.
I also had no sound at first and was confused. I needed to select an audio device in General Settings -> Audio -> TV. It doesn't seem to default to the first/default one on setup.
Yep, that's also what I tried, but I get a crash whenever I try to open general settings. More info in issue #527 (there is no reaction there yet tho)
Is it possible to edit the settings without the UI? The sound is honestly the only thing preventing me to start playing
I've been using the AppImages of Cemu (2.0.17, 2.0.18 and 2.0.19) for a few weeks now, no crashes and everything works great and fast (at least for BoTW), except that the app doesn't seem to respect the system dark theme on KDE.
This is what I've got in mine
/home/bunkbail/.config/Cemu/settings.xmlI updated the original post to match the current state of development.
Compiling from source in Ubuntu 22.04 , following the instructions on the site, gives me this error
cmake -S . -B build -DCMAKE_BUILD_TYPE=release -DCMAKE_C_COMPILER=/usr/bin/clang-12 -DCMAKE_CXX_COMPILER=/usr/bin/clang++-12 -G Ninja -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja
CMake Error at /usr/share/cmake-3.22/Modules/CMakeDetermineSystem.cmake:130 (message):
Could not find toolchain file:
/home/user/Cemu_2.0-25/Cemu-main/dependencies/vcpkg/scripts/buildsystems/vcpkg.cmake
Call Stack (most recent call first):
CMakeLists.txt:31 (project)
Trying to use the ebuild in
tatsh-overlayon gentoo; Am getting the following:meanwhile:
with
wxGTK 3.2.1being installed as a prereq for the ebuild.Any ideas?
You should file an issue on my repo if you are having an issue with the ebuilds.
Cemu is covered in the readme: https://github.com/Tatsh/tatsh-overlay#cemu
There is now a Flatpak version available!
I REJOICE FOR FREEDOM!
Any way to merge or transition from appimage to flatpak? Like configs and such?
You can copy the Cemu directory from
~/.local/shareto~/.var/info.cemu.Cemu/dataand from~/.configto~/.var/info.cemu.Cemu/configChecking in tried to play Lego dimensions and couldn't get the portal to work looks like that's because hid support isn't in the Linux version yet
I think the flatpak is stuck in stable and they don't have a beta branch on flatpak
In case it helps anyone else who couldn't get the audio devices to populate:
pactl list short sourcesand choose a sink id starting withalsa_output<Audio>tag, add the sink id to<TVDevice></TVDevice>Is there any tracking issue or PR on the missing HID support?
Just tested Lego dimensions on arch using the AUR package and can confirm that the the postal works and detects figures
Hey all, one suggestion for the Cemu linux builds is to change the directory for shadercache. Currently, they are stored in
~/.cache/Cemu/shaderCachewhich seems sensible considering the name, but people who clear their cache frequently will also end up deleting this folder, causing them to have to recompile all shaders.That's the point of the common cache folder, though? Removing that entire folder will affect many other applications: it's a convenient way to clear up some space, splitting that out into separate custom places per app is a step backwards.
Exactly, why would a user delete the cache folder and not expect everything to have to be re-cached again?
Maybe the solution would be to not blindly delete the entire cache folder to try to save space 🙃
I completely understand where you're coming from with the precompiled shader cache, but I think for the transferable cache these are not usually supposed to be removed. They would take a while to regenerate if you were to play again.