libntfs_ext library compilation with cellsdk

deank said:
Yes, that's exactly what is needed.

@bguerville: I didn't use libc.c from wMM. I added to webMAN's libc.c the external imports (but not as #defines).
Click to expand...
Thanks Deank, i wasn't sure if i'm wrong or not.

The only counterpart by doing that is that this entry in ntfs_log_debug will not be created when needed. That's not a big deal i think.
Code: 
#ifdef HAVE_MBSINIT

    /* Make sure we are back in the initial state. */
    if (!mbsinit(&mbstate)) {
        ntfs_log_debug("Eeek. mbstate not in initial state!\n");
        errno = EILSEQ;
        goto err_out;
    }
#endif

But this could lead to some issues ?
Code: 
    /* Now write the NULL character. */
    mbs[o] = '\0';
    if (*outs != mbs)
        *outs = mbs;
    return o;
err_out:
    if (mbs != *outs) {
        int eo = errno;
        free(mbs);
        errno = eo;
    }
 
deank said:
@freddy38510 http://deanbg.com/libntfs_prx.zip

When

HAVE_MBSINIT

is not defined the program uses other functions. There is no need to change any other files. That's why I implemented 2 functions in libc.c along with the exports.

@bguerville - There is nothing wrong with the library, but it can't be used for prx modules - hence the separation.
Click to expand...
Yes i noticed that in libc.c

But like i'm not very confident with "C" i had some doubts, and you removed them. Thanks you again :)

Without HAVE_MBSINIT the library can be used for both prx modules and elf, right ?

The "only" things to do for prx modules is to compile libc.c in the project, linking liballocator_export_stub.a, libstdc_export_stub.a and the library of course, then include ntfs.h to main.c
Is that correct ?

Edit: I didn't see your edit yet about CACHE_DEFAULT_PAGE_COUNT and ntfsLock/ntfsUnlock
I will investigate on this to have just one library for elf and prx.
 
Yes and no.

1) For PRX thread locks must be removed (in ntfsLock/Unlock), cache memory is just 64KB
2) For ELF the original libntfs_ext with your changes from few days ago is OK.

That requires two versions of the library for static linking. I'd suggest not to use the libntfs_prx for elf-apps.
 
@freddy38510
We should add a folder to the repo for all this libntfs_ext prx along with all the information deank just provided, I assume that you are already on top of it given the questions?
Maybe some extra info should be added to the main Readme as well..
 
bguerville said:
@freddy38510
We should add a folder to the repo for all this libntfs_ext prx along with all the information deank just provided, I assume that you are already on top of it given the questions?
Maybe some extra info should be added to the main Readme as well..
Click to expand...
Yes i'm on it. I think that we should provide both compiled libraries in the repo and add a #ifdef PRX in the makefile to let people compile the library for prx or for elf. Plus, add these informations in the main Readme you're right.

What do you think about that ?

Edit: for CACHE_DEFAULT_PAGE_COUNT we should only provide the information in Readme. I'll finish that later during the day.
 
Last edited:
Anything above 64/96 will cause insufficient memory when used in vsh PRX (depending on the memory usage of the prx itself). Yesterday I tested malloc()/free() in realtime to see what is going on... At startup it is possible to allocate ~1200kb, but later after everything loads max alloc is ~ 384KB (which is about 96 cache pages). I also tested the performance with the cache_default_page_count set to 1 through 16 and didn't notice great performance degradation. Still 16 pages is a good value, because it is exactly 64KB (which is exactly one sys_memory unit).
 
deank said:
Hmm... there is no problem with thread-locks... In the port to cellsdk there was a problem with these two functions.

I'll look into it.
Click to expand...
I merged commits of Andoma https://github.com/andoma/libntfs_e...9dd9bcb#diff-9109036e85147ceaa548255fa56db6a0 , maybe that's related to the problem.

I updated the repo, you can view the last two commits here : https://github.com/freddy38510/libntfs_ext/commit/4697e324eaaaa524d5f5a9894707858ab950a44b and here https://github.com/freddy38510/libntfs_ext/commit/b04542e3c180a370cef69a53e4682623f4618f10

Edit: sys_lwmutex_lock(&vd->lock, 0); was replaced by sys_mutex_lock(vd->lock, 1000000); if NTFS_USE_LWMUTEX is not defined. Same for sys_lwmutex_unlock()
 
Yesterday when I noticed the problem I fixed it (used sys_lwmutex*). That was the original problem when I posted webMAN for tests and it was not browsing folders.

Another thing... I think there is some problem with the "types"... For "file_to_sectors" to work I had to change "size_t" with "off64_t" for 2 functions in ntfsfile.c. I don't know what the problem is, but it looks like somewhere there is a mistake/error and instead of 0x7FFFFFFF the typedef is set to 0x7FFFFF.
 
deank said:
Yesterday when I noticed the problem I fixed it (used sys_lwmutex*). That was the original problem when I posted webMAN for tests and it was not browsing folders.

Another thing... I think there is some problem with the "types"... For "file_to_sectors" to work I had to change "size_t" with "off64_t" for 2 functions in ntfsfile.c. I don't know what the problem is, but it looks like somewhere there is a mistake/error and instead of 0x7FFFFFFF the typedef is set to 0x7FFFFF.
Click to expand...
Maximum value for unsigned long is 0xFFFFFFFF
Maximum value for signed long is 0x7FFFFFFF
Maximum value for unsigned long long is 0xFFFFFFFF FFFFFFFF
Maximum value for signed long long is 0x7FFFFFFF FFFFFFFF

In PSL1GHT size_t is defined with long unsigned int
In cellsdk size_t is defined with unsigned long long
off64_t is signed long long

This problem was present with the original library by Estwald ?

To come back to thread-locks, if we use sys_lwmutex* for prx, can we use it too for elf ?

Edit: Incompatible pointer is because sys_mutex_t lock; needs to be replaced by sys_lwmutex_t lock;
 
Last edited:
No, the problem is not present when the library is compiled with psl1ght (same source) so it is something with the type definitions.

About mutexes - since the PS3 doesn't natively support ntfs drives and never tries to operate with the external usb hdd I believe it is better to use sys_lwmutexes - they don't issue syscalls and don't interrupt the kernel when there are no thread-race conditions. I'd prefer to use them for PRX..
 
deank said:
No, the problem is not present when the library is compiled with psl1ght (same source) so it is something with the type definitions.
Click to expand...
I think i found where the definition is "wrong".
For cellsdk size_t is defined in sys/sys_types.h like that: typedef __CSTD uint32_t size_t;
uint32_t is defined in sys/integertypes.h like that: typedef unsigned int uint32_t;

uint32_t should be unsigned long int
That's why size_t is defined with unsigned int instead of unsigned long int

Edit:
deank said:
About mutexes - since the PS3 doesn't natively support ntfs drives and never tries to operate with the external usb hdd I believe it is better to use sys_lwmutexes - they don't issue syscalls and don't interrupt the kernel when there are no thread-race conditions. I'd prefer to use them for PRX..
Click to expand...
So is it better to use sys_lwmutexes for elf too ?
 
Last edited:
It is up to the developer.

***

Btw I decreased the size of the library (libntfs_prx.a) and thus the size of the resulting prx/sprx by 25% by disabling ntfs_log messages in logging.h/volume.c.

libntfs_prx.a = 618 KB (632 848 bytes) - was 782 KB (801 512 bytes)

webman.sprx = 158 KB (162 476 bytes) - was 183 KB (188 261 bytes)

webman.prx = 356 KB (365 144 bytes) - was 427 KB (438 144 bytes)

Basically it saves 64KB of memory, which can be used for other purposes.

***

After compiling both with different compiler-optimisation-options I got down to:

libntfs_prx.a = 514 KB (526 832 bytes)

webman.sprx = 157 KB (161 392 bytes)

webman.prx = 353 KB (361 728 bytes)

Still 30KB remaining in the current 64KB code segment for further additions before the required memory grows with another 64KB (when the prx gets above 384KB).
 
Last edited:
deank said:
It is up to the developer.

***

Btw I decreased the size of the library (libntfs_prx.a) and thus the size of the resulting prx/sprx by 25% by disabling ntfs_log messages in logging.h/volume.c.

libntfs_prx.a = 618 KB (632 848 bytes) - was 782 KB (801 512 bytes)

webman.sprx = 158 KB (162 476 bytes) - was 183 KB (188 261 bytes)

webman.prx = 356 KB (365 144 bytes) - was 427 KB (438 144 bytes)

Basically it saves 64KB of memory, which can be used for other purposes.
Click to expand...
Well done Deank.

Why specifically volume.c ? There is a lot of ntfs_log messages in others files too.

I think we should disabling all ntfs_log messages by default. And enable it when we want to debug by compiling a libntfs_*_debug.a
It should be easy to do for devs with a simple BUILD_TYPE = release/debug in Makefile.

What do you think about that ? I'm not really sure about disabling *all* ntfs_log messages
 
In volume.c to remove the static const char messages:

static const char *invalid_ntfs_msg =
...
static const char *access_denied_msg =

just to disable the "unused warnings".

And in logging.h:

#define ntfs_log_critical(FORMAT, ARGS...) // ntfs_log_redirect...
...
#define ntfs_log_warning(FORMAT, ARGS...) // ntfs_log_redirect..

just comment out the ntfs_log_redirect in these 8 defines. It disables all standard log messages. Debug messages are another thing, but they are not enabled unless you set debug_mode, so no need to touch them.

You can take a look at the library file (.a) and you can see all the strings in it. After log messages are disabled they are not present in the final library, thus reducing the size and the size of the apps that use the library.
 
I compared the memory usage of webMAN with NTFS and without.

After webMAN loads there are 2012KB free memory (64KB more, compared to 1948KB in the previous NTFS build with the debug strings).

Compared to older builds with no NTFS (free memory 2204KB) - it means that NTFS implementation requires only 192KB of additional memory for the code segment, compared to the non-ntfs PRX builds :) which is actually great.
 
Last edited:
I updated the repo with all changes suggested by Deank.
https://github.com/freddy38510/libntfs_ext

It needs some documentation for developers. They can compile standard libntfs.a or libntfs_prx.a by typing "make standard" or "make prx".
"make" command compiles both.

In each makefiles (libntfs.mk and libntfs_prx.mk) devs can set BUILD_TYPE to "release" or "debug".

I disabled log messages only for release build.
 
You must log in or register to reply here.

Similar threads

Back
Top