libntfs_ext library compilation with cellsdk

deank said:
Actually these ntfs functions cannot fully replace cellFs functions when used for async I/O (which is used for faster copying of data). Also if libntfs_ext is to be used in a module (prx) it is best not to have redundant functions (because cellFs functions are already loaded by the prx).
Click to expand...
Thanks for this very interesting answer.
About redundant functions, the libntfs_ext functions are supposed to execute its "own code" or simply called cellFs functions when needed. It's like an "if" condition as we could see them in many psl1ght projects with ntfs support.
Code: 
if(is_ntfs){ (libntfs_ext functions called) }

else {  (cellFs functions called) }
So, how could it be redundant ? I'm just trying to understand.

About async I/O, the only cellFs functions that are asynchronous are "cellFsAioInit", "cellFsAioFinish", "cellFsAioRead", "cellFsAioCancel" and "cellFsAioWrite". This means that others cellFs functions are sequentials, is that right ?

I just looked at cellsdk libfs sample, and now i understand how async I/O is working.
 
Last edited:
My comment was mainly about libntfs usage in prx modules because of the limited memory space. By "redundant" I mean exactly the checks and calls to the cellFs functions... Looking at pure asm level you get more code/bytes for nothing. I don't know how else to explain it. For example restoring the #if 0 commented sections you gain few KB in the library and few KB in the output prx. Few KB here and there and we get to meaningful values. Of course I'm being picky at the moment counting bytes, but sometimes these things matter.

Also when differentiating the functions one is able to compile an app (or module) with or without the ntfs support.

And great job with your work - I really appreciate it!

p.s. You said "in many psl1ght projects" - but here we talk about native ps3-sdk and some psl1ght workarounds while functional could be avoided or replaced by the native ps3 functions (which are better understood by the compiler and linker).
 
Something came to mind and will probably make it harder to implement libntfs as kernel module (prx). It should compile WITHOUT inclusion of the libm_stub (and probably without libc_stub). The reason is that the functions of these libraries are not available in kernel mode and cannot be used (alloc/free...). If you take a look at webMAN you'll see that all memory allocations are done with sys_memory_allocate and some (needed) libc functions are present locally in libc.c. None of this affects game-apps, but although the prx compiles fine it will simply crash. For .prx to work it should compile without "-lc_stub -lm_stub".

late edit:

There are two approaches to the malloc/free issue:

1) Use the kernel functions - cons: this will make the implementation firmware dependent
2) Create our own memory manager - con: this is complicated

These 2 functions (malloc/free) are most important:

Code: 
malloc (calloc / realloc / memalign / strdup)
free

These are much simpler to add:

Code: 
memmove

ceil / floor

mbrtowc
mbsinit
mbsrtowcs
wcrtomb

time

strerror
_Geterrno
 
Last edited:
deank said:
Something came to mind and will probably make it harder to implement libntfs as kernel module (prx). It should compile WITHOUT inclusion of the libm_stub (and probably without libc_stub). The reason is that the functions of these libraries are not available in kernel mode and cannot be used (alloc/free...). If you take a look at webMAN you'll see that all memory allocations are done with sys_memory_allocate and some (needed) libc functions are present locally in libc.c. None of this affects game-apps, but although the prx compiles fine it will simply crash. For .prx to work it should compile without "-lc_stub -lm_stub".

late edit:

There are two approaches to the malloc/free issue:

1) Use the kernel functions - cons: this will make the implementation firmware dependent
2) Create our own memory manager - con: this is complicated

These 2 functions (malloc/free) are most important:

Code: 
malloc (calloc / realloc / memalign / strdup)
free

These are much simpler to add:

Code: 
memmove

ceil / floor

mbrtowc
mbsinit
mbsrtowcs
wcrtomb

time

strerror
_Geterrno
Click to expand...
Lv2 sc 15 have access to kernel function execution fyi :D
 
Last edited:
@aldostools
vsh stub-libs + header are RE and created by me. Headers from other devs I use, like the headers from mysis
always contain the note about my changes to respect the creators work. :)

Code: 
// Mysis game_plugin.h v0.1
// 3141card (C style version)
#ifndef __GAME_PLUGIN_H__
#define __GAME_PLUGIN_H__
...

I know its shit but my own stuff never contains my nick. (only hidden, e.g. a packet ID or something ;))
BTW, many of e.g. the stdc and alloc stuff is based on master kakaroto's old IDA stuff.

@deank
the point, I never completed e.g. lib c stub, the libs only contains the stub code for exports but not lib constans. Few exports will not work by this reason.

I remember me that therifboy had a solution for use e.g. stdc full into a vsh plugin, but sry its years ago and I forgot it. It was something with renaming the lib name into the vsh plugin with hexeditor. In this way even not stub code is needed. Maybe someone remember...
 
:) Hopefully someone will remember. I compiled libntfs as part of webMAN (decided not to add it as external library) and included few of the libm and libc exports directly in the code, otherwise the ntfs binaries won't link (doesn't like the #define method of replacing the functions) - or may be it is too late and I miss something.

Anyway, almost everything links now exept two functions (in unistr.c):

undefined reference to `mbtowc'
undefined reference to `wctomb'

Looks good... :)

p.s. Just for a test I bypassed these two functions and webMAN compiled to prx/sprx :) without the use of libm/libc, but using the export_stubs! :)
 
Last edited:
deank said:
If someone wants to test:
http://deanbg.com/webftp_server_test.zip

0) Do not connect any ntfs usb devices
1) Check if it works at all as regular webMAN
2) Check if FTP works (it should show in welcome message [NTFS:x]
Click to expand...

Code: 
[00:50:41] [L] Connecting to 192.168.137.40 -> IP=192.168.137.40 PORT=21
[00:50:41] [L] Connected to 192.168.137.40
[00:50:41] [L] 220-VSH ftpd
[00:50:41] [L] 220 webMAN ftpd 1.45n[NTFS:0]
[00:50:41] [L] USER anonymous
[00:50:41] [L] 331 OK
[00:50:41] [L] PASS (hidden)
[00:50:41] [L] 230 OK
[00:50:41] [L] SYST
[00:50:41] [L] 215 UNIX Type: L8
[00:50:41] [L] FEAT
[00:50:41] [L] 211-Ext:
[00:50:41] [L]  SIZE
[00:50:41] [L]  MDTM
[00:50:41] [L]  PORT
[00:50:41] [L]  CDUP
[00:50:41] [L]  ABOR
[00:50:41] [L]  REST STREAM
[00:50:41] [L]  PASV
[00:50:41] [L]  LIST
[00:50:41] [L]  MLSD
[00:50:41] [L]  MLST type*;size*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;
[00:50:41] [L] 211 End
[00:50:41] [L] PWD
[00:50:41] [L] 257 "/"
[00:50:41] [L] PASV
[00:50:41] [L] 227 Entering Passive Mode (192,168,137,40,172,246)
[00:50:41] [L] Opening data connection IP: 192.168.137.40 PORT: 44278
[00:50:41] [L] MLSD
[00:50:41] [L] 150 OK
[00:50:41] [L] 226 OK
[00:50:41] [L] List Complete: 557 bytes in 0.07 seconds (0.5 KB/s)

3) Close your ftp client and attach ntfs usb drive
4) Check if FTP works (if you see [NTFS:1] it means that one ntfs volume is found)
5) End of test
Click to expand...

I still keep getting NTFS:0 instead of NTFS:1..

Tried two NTFS devices [8GB NTFS memory stick, and 2TB hybrid HDD that has both FAT32 and NTFS partition]
 
Well at least it doesn't crash when it calls ntfsmountall. :) Do you see the drives blinking? Can you test with normal ntfs hdd (no fat32, no thumb drives)? Also connect them and then start the PS3.
 
deank said:
Well at least it doesn't crash when it calls ntfsmountall. :) Do you see the drives blinking? Can you test with normal ntfs hdd (no fat32, no thumb drives)? Also connect them and then start the PS3.
Click to expand...

HDD blinks and the 2nd port seems working

Code: 
[01:31:36] [L] 220-VSH ftpd
[01:31:37] [L] 220 webMAN ftpd 1.45n[NTFS:1]
[01:31:37] [L] USER anonymous
[01:31:37] [L] 331 OK
[01:31:37] [L] PASS (hidden)
[01:31:37] [L] 230 OK
[01:31:37] [L] SYST
[01:31:37] [L] 215 UNIX Type: L8
[01:31:37] [L] FEAT
[01:31:37] [L] 211-Ext:
[01:31:37] [L]  SIZE
[01:31:37] [L]  MDTM
[01:31:37] [L]  PORT
[01:31:37] [L]  CDUP
[01:31:37] [L]  ABOR
[01:31:37] [L]  REST STREAM
[01:31:37] [L]  PASV
[01:31:37] [L]  LIST
[01:31:37] [L]  MLSD
[01:31:37] [L]  MLST type*;size*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;
[01:31:37] [L] 211 End
[01:31:37] [L] PWD
[01:31:37] [L] 257 "/"
[01:31:37] [L] CWD /
[01:31:37] [L] 250 OK
[01:31:37] [L] PWD
[01:31:37] [L] 257 "/"
[01:31:37] [L] PASV
[01:31:37] [L] 227 Entering Passive Mode (192,168,137,40,128,177)
[01:31:37] [L] Opening data connection IP: 192.168.137.40 PORT: 32945
[01:31:37] [L] MLSD
[01:31:37] [L] 150 OK
[01:31:37] [L] 226 OK
[01:31:37] [L] List Complete: 653 bytes in 0.06 seconds (0.6 KB/s)

Tried the same HDD [FAT32/NTFS hybrid] I currently don't have any single NTFS HDD yet
 
Ah, that looks nice. [NTFS:1] means ntfsMountall returned 1 :) and it seems it is working (also it should mean that malloc/free functions are ok). I'll try to add some folder listing to see if that works too.
 
deank said:
:) Hopefully someone will remember. I compiled libntfs as part of webMAN (decided not to add it as external library) and included few of the libm and libc exports directly in the code, otherwise the ntfs binaries won't link (doesn't like the #define method of replacing the functions) - or may be it is too late and I miss something.

Anyway, almost everything links now exept two functions (in unistr.c):

undefined reference to `mbtowc'
undefined reference to `wctomb'

Looks good... :)

p.s. Just for a test I bypassed these two functions and webMAN compiled to prx/sprx :) without the use of libm/libc, but using the export_stubs! :)
Click to expand...
I read all the above posts, and i'm happy to see how this project is moving so far with you guys.

I'll remove all #define in cellos_lv2.h and replace psl1ght functions by appropriate cellsdk functions in my repo.

If you don't mind, could you tell us how you resolve these two undefined references ?
Also, do you think i should include the necessary libm and libc exports in the library code (ntfs header file) or it should be only included in prx project ?

Edit: After looking to unistr.c, i think i know how you did it.
 
Last edited:
deank said:
Ah, that looks nice. [NTFS:1] means ntfsMountall returned 1 :) and it seems it is working (also it should mean that malloc/free functions are ok). I'll try to add some folder listing to see if that works too.
Click to expand...
Here is my log with only one ntfs partition on hdd. MountAll returned 1 also but no listing.
edf05fe125e65e5f4b8ded493e0d243f.jpg

Code: 
2017-01-18 10:44:32 7560 1 Status: Connecting to 192.168.0.45:21... 2017-01-18 10:44:32 7560 1 Status: Connection established, waiting for welcome message... 2017-01-18 10:44:32 7560 1 Response: 220-VSH ftpd 2017-01-18 10:44:32 7560 1 Response: 220 webMAN ftpd 1.45n[NTFS:1] 2017-01-18 10:44:32 7560 1 Command: AUTH TLS 2017-01-18 10:44:32 7560 1 Response: 530 Error 2017-01-18 10:44:32 7560 1 Command: AUTH SSL 2017-01-18 10:44:32 7560 1 Response: 530 Error 2017-01-18 10:44:32 7560 1 Status: Insecure server, it does not support FTP over TLS. 2017-01-18 10:44:32 7560 1 Command: USER anonymous 2017-01-18 10:44:32 7560 1 Response: 331 OK 2017-01-18 10:44:32 7560 1 Command: PASS ************** 2017-01-18 10:44:32 7560 1 Response: 230 OK 2017-01-18 10:44:32 7560 1 Command: SYST 2017-01-18 10:44:32 7560 1 Response: 215 UNIX Type: L8 2017-01-18 10:44:32 7560 1 Command: FEAT 2017-01-18 10:44:32 7560 1 Response: 211-Ext: 2017-01-18 10:44:32 7560 1 Response: SIZE 2017-01-18 10:44:32 7560 1 Response: MDTM 2017-01-18 10:44:32 7560 1 Response: PORT 2017-01-18 10:44:32 7560 1 Response: CDUP 2017-01-18 10:44:32 7560 1 Response: ABOR 2017-01-18 10:44:32 7560 1 Response: REST STREAM 2017-01-18 10:44:32 7560 1 Response: PASV 2017-01-18 10:44:32 7560 1 Response: LIST 2017-01-18 10:44:32 7560 1 Response: MLSD 2017-01-18 10:44:32 7560 1 Response: MLST type*;size*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*; 2017-01-18 10:44:32 7560 1 Response: 211 End 2017-01-18 10:44:32 7560 1 Status: Server does not support non-ASCII characters. 2017-01-18 10:44:32 7560 1 Status: Logged in 2017-01-18 10:44:32 7560 1 Status: Retrieving directory listing... 2017-01-18 10:44:32 7560 1 Command: PWD 2017-01-18 10:44:32 7560 1 Response: 257 "/" 2017-01-18 10:44:32 7560 1 Command: TYPE I 2017-01-18 10:44:32 7560 1 Response: 200 TYPE OK 2017-01-18 10:44:32 7560 1 Command: PASV 2017-01-18 10:44:32 7560 1 Response: 227 Entering Passive Mode (192,168,0,45,132,191) 2017-01-18 10:44:32 7560 1 Command: MLSD 2017-01-18 10:44:32 7560 1 Response: 150 OK 2017-01-18 10:44:32 7560 1 Response: 226 OK 2017-01-18 10:44:32 7560 1 Status: Directory listing of "/" successful
 
I had no time to work on these two (mbtowc/wctomb) but I guess they will need full implementation. The exports (libm/libc) are need only for the prx version (unless we use the hack mentioned by @3141card).

Once I'm ready with webMAN's FTP access to NTFS I'll upload the source.

I also had to change some options in ntfs's config.h and in mem_allocate.h.

@bguerville: There is no folder/device listing in this test version - I'm trying with very small steps and the idea was to see that the volume is mounted and the plugin doesn't crash. ;)
 
deank said:
I had no time to work on these two (mbtowc/wctomb) but I guess they will need full implementation. The exports (libm/libc) are need only for the prx version (unless we use the hack mentioned by @3141card).

Once I'm ready with webMAN's FTP access to NTFS I'll upload the source.

I also had to change some options in ntfs's config.h and in mem_allocate.h.
Click to expand...
In wMM, I got stuck with trying to use libc's mbstowcs & wcslen for the exact same reason last year when first trying to support download_plugin calls. It took me 3 days to figure it all out without having to rewrite a full implementation of the functions I needed or using yet another lib that would implement them, the funny thing is that I found the exact same thread that @3141card mentioned above back then. When I read it, I realised that Aldo had already placed all the vsh exports files in the wMM project for this very reason. Lol

Back then I tried to find in the sdk s#ny's recommendations on the best way to deal with this basic situation in prx code writing.. Unfortunately I was not so lucky as to find a single word about this, I either missed it or there is no mention of it.

Edit:
@deank
Gotcha. It's a success so far then. Lol
I had already performed this test with a series of new web commands about 10 days ago. I can already tell you WMM works perfectly fine with just a few simple ntfs functions. I tested these functions ok with wMM 1.45.08 Full. (only 1 hdd with 1 partition so no worries with memory issues for these tests)
ntfsMount
ntfsMountAll
ntfsUnmount
ntfsGetVolumeName
ps3ntfs_diropen
ps3ntfs_dirclose
ps3ntfs_dirnext
ps3ntfs_read
ps3ntfs_write
 
Last edited:
You must log in or register to reply here.

Similar threads

Back
Top