Back to HEN-3.0.3

[Algol]

Merci beaucoup à toi (@bguerville) et à tous les DEVs ici présents qui améliorent le système de nos PS3 compatibles et non-compatibles au CFW, merci encore une fois.

Thank you very much to you (@bguerville) and to all the DEVs here present who improve the system of our compatible and non-compatible PS3s at CFW, thank you once again

 

[aldostools]

It's already the case, I had to introduce a new syscall 8 subcode for map_path already, so devs can specify flags as an extra argument and get proper standard error codes return while the old syscalls argument syntax and return type remain "untouched" for legacy reasons.

However that doesn't help for the priority system, we must use FIFO to process the mappings list otherwise legacy homebrews will have issues.

In practice we could keep 2 separate lists operating under different priority order, the old syscalls operating on a FIFO list while the new syscall would operate a list under another priority order, but how could we manage both systems at the same time?
Each time a path is processed by the kernel, the open_path hook searches the mappings list for redirections, it would have to process the FIFO list first, then the other list and finally it would have to reconcile the redirection results somehow, assuming we can find a way to reconcile them in a coherent manner that would guarantee legacy support (I would not hold my breath about that), it would be messy at best..

I don't like it but quite frankly I cannot see a decent way out of this, I am open to suggestions if someone can, that is why I wished to submit this problem to scene contributors before releasing the rewrite.

IMHO The map path must be seen as a public list class (add, update, delete, enumerate).

1- The map path list internally can be a chained list or a regular vector (as it is currently)
2- The add operation by default should add the path to the end by default (as it is currently)
3- A new priority flag could be added to add the path to the beginning (or to a specific place in the list if you wish)
4- The remap order can stay being a FIFO (since the priority order is defined during the addition)

If the original path exists in the list, the operation replaces the target path (as it is currently)
If the target path is NULL, the operation deletes the map path from the list (as it is currently)

The additional flags needed are:
1- Permanent map path (prevent deletion)
2- Force priority (add map path to beginning)
3- Shadow mapping (search in original path, if the item isn't found in the target path)

The flags 1 and 3 are currently implemented using /./ or // as prefix in the target path.

So basically the search priority is what is not managed currently and would be solved with a flag to force adding the path to the beginning of the list.

 

[bguerville]

IMHO The map path must be seen as a public list class (add, update, delete, enumerate).

1- The map path list internally can be a chained list or a regular vector (as it is currently)
2- The add operation by default should add the path to the end by default (as it is currently)
3- A new priority flag could be added to add the path to the beginning (or to a specific place in the list if you wish)
4- The remap order can stay being a FIFO (since the priority order is defined during the addition)

If the original path exists in the list, the operation replaces the target path (as it is currently)
If the target path is NULL, the operation deletes the map path from the list (as it is currently)

The additional flags needed are:
1- Permanent map path (prevent deletion)
2- Force priority (add map path to beginning)
3- Shadow mapping (search in original path, if the item isn't found in the target path)

The flags 1 and 3 are currently implemented using /./ or // as prefix in the target path.

So basically the search priority is what is not managed currently and would be solved with a flag to force adding the path to the beginning of the list.

A vector?
The original implementation uses a fixed size array and my rewrite uses a dynamically allocated linked list.

Following your recommendations, no changes would be needed in the rewrite except for the function of the priority flag, currently all mappings with the flag get iterated through for matching first, it will be easy to remove this loop from the findMapping function and instead, in the addMapping, position the newly added mapping to the first place in the list.
I keep wondering how devs will know whether that first position (or any index position really) would be the ideal place for a mapping short of examining the entire list of existing mappings first but tbph I don't particularly care what system we use, I won't be the one using remappings in my apps anyway, I MERELY wanted you guys to let me know if you have any suggestions for this before we release the source and update HEN with it, so I can finalize the code..

 

[aldostools]

A vector?
The original implementation uses a fixed size array and my rewrite uses a dynamically allocated linked list.

LOL a vector is an array of one dimension.

I guess you understood my sentence as a vector data type in C++.

Following your recommendations, no changes would be needed in the rewrite except for the function of the priority flag, currently all mappings with the flag get iterated through for matching first, it will be easy to remove this loop from the findMapping function and instead, in the addMapping, position the newly added mapping to the first place in the list.
.

Yes. The priority flag needs to be implemented only in the addMapping, It would add the new path to the beginning, instead of last position (default behavior).

However, making a deeper analysis, maybe a priority flag is not even needed. I'll explain it below with an example.

I keep wondering how devs will know whether that first position (or any index position really) would be the ideal place for a mapping short of examining the entire list of existing mappings first but tbph I don't particularly care what system we use, I won't be the one using remappings in my apps anyway, I MERELY wanted you guys to let me know if you have any suggestions for this before we release the source and update HEN with it, so I can finalize the code..

In theory, if the priority flag is used, the specified path would have precedence over any other existing map path.

I say "in theory" because that's true only at the moment of add the path. If other parent paths are added later with priority flag, they will void the other paths. This gets worse if multiple homebrews are doing the same.

e.g.
sys_map_path("/app_home/PS3_GAME", "/dev_hdd0/GAMES/XXX/PS3_GAME", MAP_PRIORITY); // priority is irrelevant here
sys_map_path("/app_home", "/dev_hdd0/GAMES/XXX/PS3_GAME/USRDIR", MAP_PRIORITY); // priority forces this to be the first

The first remap will never reached, because /app_home is processed first. /app_home/PS3_GAME would be searched in /dev_hdd0/GAMES/XXX/PS3_GAME/USRDIR/PS3_GAME

This could be improved, if the findMapping would search the whole list trying to find the longer matching path, instead of stop in the first map path like it is currently.

So, /app_home/PS3_GAME should not stop in the first remap (/app_home). It have to continue checking the list to see if there is a longer path that also matches. In the example: /app_home/PS3_GAME

 

[bguerville]

Yes. The priority flag needs to be implemented only in the addMapping, It would add the new path to the beginning, instead of last position (default behavior).

However, making a deeper analysis, maybe a priority flag is not even needed. I'll explain it below with an example.



In theory, if the priority flag is used, the specified path would have precedence over any other existing map path.

I say "in theory" because that's true only at the moment of add the path. If other parent paths are added later with priority flag, they will void the other paths. This gets worse if multiple homebrews are doing the same.

e.g.
sys_map_path("/app_home/PS3_GAME", "/dev_hdd0/GAMES/XXX/PS3_GAME", MAP_PRIORITY); // priority is irrelevant here
sys_map_path("/app_home", "/dev_hdd0/GAMES/XXX/PS3_GAME/USRDIR", MAP_PRIORITY); // priority forces this to be the first

The first remap will never reached, because /app_home is processed first. /app_home/PS3_GAME would be searched in /dev_hdd0/GAMES/XXX/PS3_GAME/USRDIR/PS3_GAME

This could be improved, if the findMapping would search the whole list trying to find the longer matching path, instead of stop in the first map path like it is currently.

So, /app_home/PS3_GAME should not stop in the first remap (/app_home). It have to continue checking the list to see if there is a longer path that also matches. In the example: /app_home/PS3_GAME

That's exactly right, Aldo, hence the doubts I voiced about the proposed flag implementation for addMapping.
And the situation would get more and more complex as devs add more mappings in their apps.
I am not even sure the findMapping improvement would entirely solve this conceptual issue tbph or what impact that could have on the various legacy brews using the feature.

It's like I said earlier, maybe a simpler way would be for devs to backup the list, rebuild it entirely, replace it then restore the backup to leave the listing as it was on exit.
But that may not be appropriate for mappings made in a different context from a standard start/exit elf process, for instance mappings made through VSH modules that may not remain resident in memory.

One way or another, we would need to decide what to do about this asap, if we do anything that is, the more we delay, the later the update release.

 

[aldostools]

That's exactly right, Aldo, hence the doubts I voiced about the proposed flag implementation for addMapping.
And the situation would get more and more complex as devs add more mappings in their apps.
I am not even sure the findMapping improvement would entirely solve this conceptual issue tbph or what impact that could have on the various legacy brews using the feature.

It's like I said earlier, maybe a simpler way would be for devs to backup the list, rebuild it entirely, replace it then restore the backup to leave the listing as it was on exit.
But that may not be appropriate for mappings made in a different context from a standard start/exit elf process, for instance mappings made through VSH modules that may not remain resident in memory.

One way or another, we would need to decide what to do about this asap, if we do anything that is, the more we delay, the later the update release.

I was in the same dilemma as yours and my decision was to keep the status quo.

 

[bguerville]

I was in the same dilemma as yours and my decision was to keep the status quo.

I don't like it but in the absence of a better alternative, I am afraid I might do just that.

 

[DeViL303]

This is not really related but as you are discussing Cobra development. Would it be possible for the open_path debug info that we see in socat to be expanded a little. I think it would be really useful for lots of things but it really applies to the work I am doing on Playstation Home (250,000+ files). Basically when it says open_path in the output, we have no way to know if the file was found or not. I think it would be really useful if it could add a note to the output line when the file is not found like "NOT FOUND", if that is possible and not too hard hopefully it could be considered.

EDIT: Although it would be handy to have it in real time, I guess this probably not so important for Home. I can save the socat output, then use that list of files to do a check on the folder.

 

[esc0rtd3w]

This is not really related but as you are discussing Cobra development. Would it be possible for the open_path debug info that we see in socat to be expanded a little. I think it would be really useful for lots of things but it really applies to the work I am doing on Playstation Home (250,000+ files). Basically when it says open_path in the output, we have no way to know if the file was found or not. I think it would be really useful if it could add a note to the output line when the file is not found like "NOT FOUND", if that is possible and not too hard hopefully it could be considered.

EDIT: Although it would be handy to have it in real time, I guess this probably not so important for Home. I can save the socat output, then use that list of files to do a check on the folder.

its pretty easy to add some extra DPRINTF output for file access, generally. We have commented out a lot of it, because the payload size was too big after adding many changes. Having too much output gets confusing, so a lot of times, its best to just uncomment or add the infos you need at the time, and rebuild HEN. For release we generally just keep the standard and expected ones. I think what you are asking should be fairly straight forward, although @bguerville would know more specific things than myself concerning the open_path_hook.

 

[bguerville]

This is not really related but as you are discussing Cobra development. Would it be possible for the open_path debug info that we see in socat to be expanded a little. I think it would be really useful for lots of things but it really applies to the work I am doing on Playstation Home (250,000+ files). Basically when it says open_path in the output, we have no way to know if the file was found or not. I think it would be really useful if it could add a note to the output line when the file is not found like "NOT FOUND", if that is possible and not too hard hopefully it could be considered.

EDIT: Although it would be handy to have it in real time, I guess this probably not so important for Home. I can save the socat output, then use that list of files to do a check on the folder.

The hook doesn't print that info because it doesn't have it. It can print out mapping redirection info but not whether the path actually exists.
To be able to print that, the open_path hook would need to stat every path that comes its way, however if I am not mistaken open_path is part of a stat call (like a sub function if you prefer) therefore a stat call would make the hook call itself over and again in an infinite loop.
So afaik we would need to implement something to stop the hook from triggering itself to add a feature like that.

Also a stat in each hook call means a mutex lock, stat, mutex unlock, along with DPRINTFs, there would be a performance penalty.

 

[aldostools]

This is not really related but as you are discussing Cobra development. Would it be possible for the open_path debug info that we see in socat to be expanded a little. I think it would be really useful for lots of things but it really applies to the work I am doing on Playstation Home (250,000+ files). Basically when it says open_path in the output, we have no way to know if the file was found or not. I think it would be really useful if it could add a note to the output line when the file is not found like "NOT FOUND", if that is possible and not too hard hopefully it could be considered.

EDIT: Although it would be handy to have it in real time, I guess this probably not so important for Home. I can save the socat output, then use that list of files to do a check on the folder.

That information is feasible to show, however to determine if the path exists or not it would be required to call cellFsStat() which does a recursive call to open_path.

Currently MAMBA checks if the file exists only if the destination path starts // but the debug message is not different.
If the destinatioin path starts with // is treated as a shadow path. i.e. if the file is not found in the destination path, it will use the original path.

 

[DeViL303]

OK, Thanks for the replies everyone, it sounds a little complicated and I don't like the sound of the performance hit.

I will try create a bat that can take the output log from socat and use that to verify if a file exists in a folder.

 

[esc0rtd3w]

OK, Thanks for the replies everyone, it sounds a little complicated and I don't like the sound of the performance hit.

I will try create a bat that can take the output log from socat and use that to verify if a file exists in a folder.

if you just want a test build for those file accesses, i can build you one with that output. Let me know

 

[DeViL303]

if you just want a test build for those file accesses, i can build you one with that output. Let me know

Thanks a mill, That would be great if you could do it for Rebug 4.84.2 REX? as that is what I use mostly. THIS is what I am doing, you might be interested too. I can share the latest pkg with you soon if you wanted.

 

[esc0rtd3w]

Thanks a mill, That would be great if you could do it for Rebug 4.84.2 REX? as that is what I use mostly.

sure. It should run fine in CEX mode on 4.84. By default it builds 4.84-4.89, like normal. Ill post it in the next couple days.

THIS is what I am doing, you might be interested too. I can share the latest pkg with you soon if you wanted.

thats really cool. You put quite a bit of time into that so far. I may check that out a bit at some point. It is a really interesting project to research and test.

EDIT:
The 4.84 DEX bin should also work and i can actually post that too, as i have not had time to test that one yet.

 

[DeViL303]

Ill post it in the next couple days.

Cheers. That would be great.

thats really cool. You put quite a bit of time into that so far.

I really didn't put much time in at all, that's the thing. It all came down to this small (but big) issue with PS3 file system being case sensitive compared to dev environment on windows not being case sensitive, once I figured that out and made every file and folder uppercase, everything started working locally without needing to be ran on DEX from PC. See here. Kinda funny when you think about it.

 

[bguerville]

OK, Thanks for the replies everyone, it sounds a little complicated and I don't like the sound of the performance hit.

I will try create a bat that can take the output log from socat and use that to verify if a file exists in a folder.

It's not exactly complicated, the recursivity is just an obstacle which afaik could be overcome in at least 2 or 3 ways I can think of, so it's really a matter of finding a way that is both thread safe and with a perf penalty that remains acceptable for the system and your testing needs.
Regarding the perf penalty, in theory it shouldn't be an issue in most cases but may become apparent when using certain features stat-ing or open-ing lots of files, like in a file manager for instance.
I may try a couple of options with Jason tonight if he's available, see if we can help you with this.

 

[DeViL303]

I just realized HEN has way more information in the debug output than the cobra payload I have been using for Rebug 4.84.2. HEN seems to be getting a lot of the messages i was normally only seeing in the DEV console of the app. Is there some newer Rebug payloads with better debug output?

I normally only get "open_path...." with Rebug on socat.

 

[aldostools]

I just realized HEN has way more information in the debug output than the cobra payload I have been using for Rebug 4.84.2. HEN seems to be getting a lot of the messages i was normally only seeing in the DEV console of the app. Is there some newer Rebug payloads with better debug output?

I normally only get "open_path...." with Rebug on socat.

View attachment 38099

You can try to get debug info also with MAMBA. It may have a lot of debug messages commented like in Cobra to reduce the amount of network traffic. Also too much debug messages make difficult to read the important messages.

I can help you compiling HEN or Cobra if you need them too.
Although MAMBA usually is safer to implement than Cobra.

If you need help adding debug messages, just let me know.

 

[esc0rtd3w]

@DeViL303 Here are test bins for you to try with the stat now in open_path_hook.

@bguerville gets all the credit, as he wrote the code, and it seems pretty stable actually, but i have not tried large FTP transfers or file manager stuff yet, just playing games normally and launching basic homebrews was fine.