newlib porting challenges

Discussion in 'PS2 Homebrew' started by Maximus32, Oct 29, 2019.

  1. 11
    48
    12
    Maximus32

    Maximus32 Developer

    Joined:
    Sep 10, 2019
    Messages:
    11
    Likes Received:
    48
    Trophy Points:
    12
    Gender:
    Male
    Introduction
    For a long time there have been 2 libc libraries for ps2 development: newlib and ps2sdk/ee/libc. They have lots of duplicate functionality and duplicate header files, but with different implementations. The newlib one is more standard (posix) compliant, but the ps2 port has always been incomplete and broken. While the ps2sdk one is not broken, but it's not feature complete and not very posix compliant.

    To fix this issue I decided to complete the newlib port. This seems to be the best way forward, since having a good libc library also means we can compile software not written for the ps2. Normally this would require "porting" to get these programs working with all ps2 specific header files and functions. But with a good libc library, a lot of porting is no longer needed. As a good example of this I've been working together with fjtrujy to get more emulators (cores) running in the ps2 port of RetroArch. A previously difficult to compile core with lots of errors (snes9x) now compiles AND RUNS without any porting.

    Currently the port is not only feature complete, but also more complete than both libc libraries together ever have been. With an accurate nanosleep function. A time function (GMT/UTC only for now). And open/read/write/close automatically mapping to fioOpen/fileXioOpen/etc... Also all directory browsing with opendit/readdir/closedir works, as well as requisting file information using stat.

    The good thing is standard applications work very good, making possible many other programs on the ps2. The downside is that ps2 applications are made to work with non-standard headers and functions, so some ps2 specific projects need to be 'ported' to posix. And another problem I run into is bugs, lots of bugs. Not just in the newlib port itself, but also in other ps2 projects. These bugs where not causing crashes before, but with the newlib port they somehow are. For instance freeing the same piece of memory twice, or freeing a static piece of memory didn't cause a crash before, but it does now! I've already fixed a few of these issues, but I can't get OPL to function properly becouse of it. Delaying the introduction of the newlib port, since I think OPL compatibility is important.
    I'm hoping you can help in improving the newlib port, and fix the problems with OPL.

    For more info on the OPL issue, I've created an issue here:
    https://gitlab.com/ps2max/ps2sdk/issues/6

    To get started with the newlib port you'll need to use my development environment, which is loosly based on the development environment used for developing android. Once you get to know it you'll love it. Here goes...

    Getting the sources and compiling for the first time
    Extra requirements for when you already have a ps2dev environment:
    • packages: repo, cmake
    • remove all your $PS2DEV,$PS2SDK,etc,etc... environment variables
    • preferably remove or rename your /usr/local/ps2dev folder, just to be sure you're not still using it
    Using my build system you can have multiple toolchains running on the same system. That's why you need to remove all your old environment variables to $PS2DEV,$PS2SDK,etc,etc. In this short tutorial I'll explain how to get the normal toolchain (with some minor additions). And then the newlib toolchain. Both toolchains can then be used at the same time.

    First we need to get the old toolchain up and running:

    mkdir ps2dev
    cd ps2dev
    repo init -u https://gitlab.com/ps2max/ps2dev-repo -b ps2max
    repo sync -j4
    . envsetup.sh
    ./build-all.sh


    NOTE1: add "--depth=1" to the "repo init" command if you're not interested in the git history. This will make cloning all git repositories significantly faster.
    NOTE2: add "-c --no-tags" to the repo sync command to get only the current branch without tags. This will make cloning all git repositories significantly faster.
    NOTE3: The ". envsetup.sh" sets all environment variables in the current terminal session. You should ALWAYS execute this command first after opening a new terminal. This is also what makes it possible to use multiple toolchains on the same pc. Those familiar to android development will see a lot of similarities :).
    NOTE4: If you watched closely you'll have seen some project being built with cmake (the green lines with the percentages). These git projects build without any changes to the sources. Not even a Makefile is added or changed to compile these libraries.

    When ready this should result in not only the toolchain, but also all libraries, tools and some ps2 applications like OPL. Let's see if OPL works:

    cd apps/open-ps2-loader
    make clean all # already performed by ./build-all.sh but lets do this anyway
    make run # will send opl.elf to your ps2 using ps2client
    make sim # will try to start opl.elf using PCSX2


    Now that we have the basics working we can continue to the newlib port. Open a second terminal session:
    mkdir ps2dev-newlib

    cd ps2dev-newlib
    repo init -u https://gitlab.com/ps2max/ps2dev-repo -b move-libc-to-newlib
    repo sync -j4
    . envsetup.sh
    ./build-all.sh


    You'll now have all my latest newlib work compiled. Try again if OPL works in the same way as described above. HINT: do not use a HIRES resolution, and do not load any artwork (jpg).

    Working with repo and the toolchain(s)
    Updating to the latest changes is simple, assuming you don't have any local changes:
    repo sync # will fetch all the latest sources
    ./build-all.sh # will rebuild everything, so only use when needed


    Making changes and using the power of repo has the benefit of still being able to "repo sync" after making changes:
    repo start my-fantastic-changes --all # create a branch in ALL git repositories, yes, do it!
    # make some changes... in whatever git project...
    git add X
    git commit -m "Fantastic!"


    Now to see in what projects you made your changes:
    repo branch # view ALL branches in ALL projects YOU made
    repo status # view ALL uncommitted changes in ALL projects
    repo info -o # view the commits you made in ALL projects ahead of upstream, I love this command


    If you want to sync your branch to whatever changes are made 'upstream':
    repo sync # will fetch all the latest sources, and !REBASE! your work on top of it
    ./build-all.sh # will rebuild everything, so only use when needed


    Good luck testing the newlib toolchain. If there's anything I can improve let me know. If you find any bugs you can report them in this thread or https://gitlab.com/ps2max/ps2sdk/issues. If you fix something you can also send a pull request https://gitlab.com/ps2max. If I don't respond, leave me a message here, becouse I'm not sure if all notification settings are setup correcly on gitlab ;).
     
    Last edited by a moderator: Oct 29, 2019
    DeViL303, ntodek, littlebalup and 9 others like this.
  2. 3
    10
    32
    fjtrujy

    fjtrujy Developer

    Joined:
    Sep 29, 2018
    Messages:
    3
    Likes Received:
    10
    Trophy Points:
    32
    Gender:
    Male
    Thanks, @Maximus32 for your great collaboration improving our PS2 DEV environment & experience.

    As you mentioned before, it is amazing now how easy is to compile POSIX software compliant, as for instance RetroArch.

    Just to give you an overview, if I would start today, to fully port Retroarch from the scratch, it just could take a couple of hours, and not a few weeks as it was before.

    I will try today to play around OPL and see if I can help to find unexpected behaviours.

    Once again, thanks!
     
    DeViL303, Krah, jcorrea and 4 others like this.
  3. 11
    48
    12
    Maximus32

    Maximus32 Developer

    Joined:
    Sep 10, 2019
    Messages:
    11
    Likes Received:
    48
    Trophy Points:
    12
    Gender:
    Male
    I just rebased both open-ps2-loader branches on top of the latest ifcaro/master. Just so we can be working with the latest version on this, and becouse I forgot to push my latest open-ps2-loader changes ;).

    Since it's rebased, the normal "git pull" of "repo sync" commands will produce errors if you've already checked out the sources. If you've already synced all sources, you can update using:
    cd apps/open-ps2-loader
    git fetch ps2max
    git reset --hard ps2max/move-libc-to-newlib # if you're in the newlib repo
    git reset --hard ps2max/master-ps2max # if you're in the normal repo


    For the next couple of weeks I hope I won't need to rebase again so we can keep development simple (it's already complicated enough as it is).
     
    DeViL303, Algol, Krah and 2 others like this.
  4. 11
    48
    12
    Maximus32

    Maximus32 Developer

    Joined:
    Sep 10, 2019
    Messages:
    11
    Likes Received:
    48
    Trophy Points:
    12
    Gender:
    Male
    The bug in OPL, where loading a jpeg file would crash OPL, has been fixed by @fjtrujy in ps2sdk-ports/libjpeg. Dynamically allocated memory was not cleared, causing a pointer to be !=NULL. Freeing this pointer caused the crash.

    This is not the first dynamic memory bug that's been fixed in the newlib port. Somehow it seems the old ps2sdk malloc/free was much more resistent against programming bugs. So on the one hand it's a good thing these bugs are now visible and fixed. But on the other hand it's also causing a lot of headaches...

    On to the next bug...
     
    TnA, Algol, jolek and 1 other person like this.
  5. 16
    49
    37
    wisi

    wisi Member

    Joined:
    Aug 26, 2018
    Messages:
    16
    Likes Received:
    49
    Trophy Points:
    37
    Gender:
    Male
    Good work!

    I once wrote a few function wrappers that check malloc memory overwriting (of the memory just before and after the buffer) and multiple calls to free() and also can list all currently allocated memory so it gives some basic statistics and can this way check if anything was not freed. This proved extremely helpful to me, as several program I wrote had multiple mistakes of this kind, so I think this might be of some use to you. There probably are better tools for that out there, but I haven't looked.

    The overwriting check is done by placing four identical structures (each 0x10 in size) around the data - two before the start and two after the end. The second set is inverted, which makes it easier to see if overwriting really did happen.
    The code makes free() very slow, because its wrapper checks all allocations for errors, but this has the positive side of finding bugs earlier.
    It also prints both the name of the caller and the PC of the call, though for printing the name, the caller code must be recompiled (the call is made into a macro).
    The code is attached. I have only tested it on PC, and I have not had any problems with it.
     

    Attached Files:

    DeViL303, Maximus32, TnA and 2 others like this.
  6. 148
    251
    122
    uyjulian

    uyjulian Developer

    Joined:
    May 27, 2017
    Messages:
    148
    Likes Received:
    251
    Trophy Points:
    122
    Gender:
    Male
    A "lazy" way to fix double free would be to set the pointer to nullpo after freeing memory, then adding a nullpo check after each free.

    However, this would not fix memory leaks by double allocating.
     
    DeViL303, Algol, jolek and 1 other person like this.
  7. 1,060
    547
    222
    TnA

    TnA Senior Member

    Joined:
    Jul 1, 2018
    Messages:
    1,060
    Likes Received:
    547
    Trophy Points:
    222
    Gender:
    Male
    Location:
    Germany --> Saxony
    ...and it would also not find 'faulty functions' or those which have no boundary-checking (strcpy() and what not...) and other 'tests'!

    However... I think @wisi's wrapper could be useful once in a while (+ function-checking could be added).
     
    DeViL303 likes this.
  8. 11
    48
    12
    Maximus32

    Maximus32 Developer

    Joined:
    Sep 10, 2019
    Messages:
    11
    Likes Received:
    48
    Trophy Points:
    12
    Gender:
    Male
    @wisi, thanks. I was looking for something like this a couple of weeks ago. So basically I can replace all calls to malloc with mallocStatAAlloc, and all calls to free with mallocStatAFree.

    I only have to add something for calloc and memalign. Then check all libraries opl uses, and opl itself.
     
    uyjulian, TnA, DeViL303 and 1 other person like this.
  9. 16
    49
    37
    wisi

    wisi Member

    Joined:
    Aug 26, 2018
    Messages:
    16
    Likes Received:
    49
    Trophy Points:
    37
    Gender:
    Male
    Yes. Though malloc has to be a macro, if you want to see the caller's name too. If that can't be done for some reason, then you can give a NULL or empty string for the name, but it has to be static if it is an empty string.

    I have no idea how __builtin_return_address(0); works - most likely it reads $ra at the time, but if the function was called with a 'j' instruction, it will get the last jal.

    The code near the very end:

    #define malloc(x) mallocAndChkIn((x), (char*)__func__)
    void *mallocAndChkIn(int sz, char *callerFuncName) {
    void *cp = __builtin_return_address(0);
    void *bf = mallocStatAAlloc(sz, cp, callerFuncName, 0); // arg 4 = 0: alloc without clearing
    return bf;
    }

    int free(void *bf) {
    mallocStatAFree(bf);
    return 0;
    }


    mallocStatAFree() doesn't free NULL buffers, so if you want to detect them too, just add a printf or some other function to warn if the given buffer is NULL there.

    After the program completes (or at any point in its run) you can call mallocStatAPrtAllocs(); and it will print all allocations that weren't freed and the amount of memory allocated (note that it prints them in reverse order - the most recent ones - first) . It also does the same check mallocStatAFree() does (mallocStatACheck) so it will also give info about any buffer getting overwritten before its beginning or after its end. Of course there is no way to detect overwrites outside the checked areas (0x20 before and after the buffer) so it it is some array that is not linearly filled, writing outside the buffer can go undetected.

    The error reported when a buffer was freed more than once or somehow got missing from the linked-list is printf("\n %s: Entry found NOT one time = %d ", __func__, found); - and it prints the number of times that entry was found. So if you free some buffer that wasn't allocated or was already freed, this will print 0, and if something went very wrong with the linked-list it might print >1 (though that can probably never happen).

    The functions require 0x4-alignment of the buffer, and I am not sure if the core *alloc function used ever returns something with lower alignment (probably not).
     
    akuhak, uyjulian, TnA and 4 others like this.
  10. 11
    48
    12
    Maximus32

    Maximus32 Developer

    Joined:
    Sep 10, 2019
    Messages:
    11
    Likes Received:
    48
    Trophy Points:
    12
    Gender:
    Male
    Update for those not watching the progress on gitlab and github:
    - A jpeg bug has been fixed by @fjtrujy
    - rmdir function was missing and has been added by @fjtrujy
    - I've just added a docker build for the toolchain
    - Latest newlib based OPL is now automatically built by docker image and can be downloaded from... errr still not allowed to post links
     
    TnA, Algol, DeViL303 and 1 other person like this.
  11. 11
    48
    12
    Maximus32

    Maximus32 Developer

    Joined:
    Sep 10, 2019
    Messages:
    11
    Likes Received:
    48
    Trophy Points:
    12
    Gender:
    Male
  12. 1,441
    1,115
    347
    jolek

    jolek Senior Member

    Joined:
    Dec 29, 2017
    Messages:
    1,441
    Likes Received:
    1,115
    Trophy Points:
    347
    Gender:
    Male
    I've few errors with this new 1417 test version.
    1. OPL will not launch when 720p or 1080i res has been set previously in settings.
    2. I'm getting BSOD when I'll try to set 720p in settings.
    3. I'm getting strange video glitches when I'll set 1080i in settings
    4. I don't know why, but sometimes OPL will freeze when I want to save settings:
      [​IMG]
      I do not fully remember what I've set\change in settings.
      However settings suppose to be saved on HDD, didn't tried MC so far.

    Tested on SCPH-50004.
     
    Last edited: Nov 10, 2019 at 8:02 AM
    TnA likes this.
  13. 11
    48
    12
    Maximus32

    Maximus32 Developer

    Joined:
    Sep 10, 2019
    Messages:
    11
    Likes Received:
    48
    Trophy Points:
    12
    Gender:
    Male
    Hi @jolek .

    For 1. 2., and 3.: There is still 1 big bug left to squash in the newlib port: HIRES in OPL. Link to issue on gitlab: https://gitlab.com/ps2max/ps2sdk/issues/6.

    For 4.: is this while using a HIRES resolution? If not then perhaps it's another bug.

    If there's any bugs other than HIRES I'd be really interested.
     
    akuhak, TnA and Algol like this.

Share This Page