WebMAN-MOD pkg handling commands

Discussion in 'webMAN MOD' started by bguerville, Apr 11, 2016.

  1. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    Important:

    Users who want to use these new Web commands can now do so simply by installing the new webMAN-MOD release 1.43.31 from @aldostools.

    http://www.github.com/aldostools/webMAN-MOD/releases/tag/1.43.31

    Any other mirror still in use in this thread will contain experimental builds & are aimed at testers only.

    Also the syntax has changed with time so make sure you refer to the latest syntax when you use the commands.

    =============================================

    Hi everyone.
    I am currently working on new wMM commands to download & to install pkg files.
    This is an ongoing project which hopefully could lead to full pkg handling automation via wMM.

    To be clear, I am taking little credit for this. The core of the features comes from the research done by mysis & co on the PS3 system plugins so if these features are found reliable enough to be included in wMM, thank them because without their research, I would not have been able to implement these cool commands..

    I would be grateful if some of you guys would try the commands using the sprx linked below & report whatever problems you may have experienced when using the commands .
    The error handling for these features is to be improved but should be sufficient for testing.

    Here is the syntax to be used:

    1. Download

    ps3-ip-address/pkg.ps3$downloadfrom=

    Replace by a valid url, use a full url with http or https
    The downloaded files can be found by default in /dev_hdd0/tmp/downloader/ for the moment. Choosing the path to store the downloads to will come soon...
    Background downloading works also.

    There is no check regarding the downloaded file type so the command can be used to download files other than pkg. I am leaving it like this for now so the feature can easily be tested with other files but ultimately a separate command for non pkg file downloading will most likely be created.

    2. Install

    ps3-ip-address/pkg.ps3$install=

    Replace by the path of the pkg you want to install.

    The pkg commands can be used like any other command in wMM including with a custom combo using
    GET /pkg.ps3$......

    Download: (removed)
    There are 2 editions for testing just now (you can ask me for another if you want).

    The webftp_server.sprx Full Edition with pkg handling & most other features enabled.

    The other one, webftp_server_cobraonly.sprx, is a Cobra/no PS3MAPI (non Rebug) lighter version of wMM with pkg handling but fewer other features.

    Edit:
    This download urls remain the same but the sprx files get updated as I go along so keep checking for updated versions. This past hour I fixed the couple of errors I had found yesterday & started to work on other improvements.

    Many thanks in advance to all testers.

    Sent from my GT-I9305 using Tapatalk
     
    Last edited: Jul 18, 2016
  2. 666
    88
    38
    No0bZiLLa

    No0bZiLLa Moderator Developer

    Joined:
    Oct 1, 2014
    Messages:
    666
    Likes Received:
    88
    Trophy Points:
    38
    no ps3 sadly. looks neat though.
     
  3. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    Well I hope it can be real in the end.

    Aldo warned me of problems with this thing and I absolutely trust his judgement so it might not be such a straight road.
    But so far it all seems to work fine for me on my Rebug 4.78.2. I tested each feature 2 dozen times today & I froze the xmb once. I wasn't able to reproduce the freeze either.

    As I said the error handling needs improved for sure. I have noticed a couple of problems to fix already. So honestly it's not finished
    .
    Btw the pop-up messages are for debug info & are not there to stay, or at least not in that state.

    I wish I had more time to take care of most of the testing myself but it's not the case. I am sure some guys here will be happy to give me a hand in trying to get this thing working properly.

    Sent from my GT-I9305 using Tapatalk
     
    Last edited: Apr 12, 2016
  4. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    Here is an update to this project of mine. Progress is slow due to lack of time...as always.
    However this update takes the shape of a complete webMAN-MOD updater pkg this time.
    I updated my project to the 1.43.29 to include 4.80 CFW support.

    Mirror 1

    As far as I know the pkg download command is ok now. It seems to work well even though I don't really like the way I currently use to make it so...but still it works..

    In order to choose the path in which to store the downloaded file, I have created a new optional parameter: "downloadto".
    If the parameter is omitted or the path is wrong, wMM will try successively and in that order to store the file in
    /dev_hdd0/packages
    /dev_hdd0/tmp/downloader
    I wanted to include /dev_hdd0 in that list but the system download plugin defaults to
    /dev_hdd0/tmp/downloader automatically if given the internall HDD root path...

    Now the pkg download command syntax is:
    hxxp://<ip-console>/pkg.ps3$downloadfrom=<pkg-file-url>&downloadto=<storage-path>
    (Don't include the <> characters!)

    The pkg install command I am unsure of, I experienced one single unexplained XMB freeze during my latest tests but I could not reproduce it even by following the same steps again...So I dunno...
    Its syntax remains:
    hxxp://<ip-console>/pkg.ps3$install=<pkg-path>
    (Don't include the <> characters!)

    Testing was limited. Done only on one slim console and one CFW (Rebug 4.80.1 CEX mode).

    I expect that some of you will have issues I have not experienced yet...
    Your feedback would be most welcome.:tranquillity:


    Edit:
    Download link deleted.
    The file & the information in this post are both outdated.
    Please read on for the new release & change of syntax.
     
    Last edited: Jul 13, 2016
    lord3490, Zar and kozarovv like this.
  5. 3,833
    1,404
    123
    atreyu187

    atreyu187 PSTV Fanboy Extraordinaire Moderator Developer

    Joined:
    Sep 29, 2014
    Messages:
    3,833
    Likes Received:
    1,404
    Trophy Points:
    123
    Gender:
    Male
    Occupation:
    Lead Veterinary Tech
    Location:
    East Coast USA
    Any chance of seeing the new .30 used as a base? aldos new naming has really helped me with all ym kids LEGO games and what not but regardless this is a really neat feature!! I will guive it a go after I get my system back in shape after I re-installed a new MFW last night.
     
  6. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    Well I was going to merge with the .30 source but aldostools is about to release .31 so I thought I would wait for it...

    Sent with Tapatalk
     
  7. 1,642
    1,612
    123
    aldostools

    aldostools Developer

    Joined:
    Oct 30, 2014
    Messages:
    1,642
    Likes Received:
    1,612
    Trophy Points:
    123
    Location:
    /dev_hdd0/tmp/wmtmp
    Home Page:
    The latest test build .31 was canceled, due the bug was caused by ps3netsrv. So the fix in webMAN was not needed and would make the scanning of content stored on ps3netsvr less efficient.

    If you want to publish your source code as it is, I can merge it with the project (with credits to you & mysis of course).

    I tested the pkg installer, and it seems to be working properly.
    However, the I get "Web Command Syntax error" with the download command. I tested your full edition.

    When I install a pkg, there is cosmetic bug: a "</" after the return button.

    I'd prefer the following syntax:
    /install.ps3<path-of-pkg> (e.g. 10.0.0.8/install.ps3/dev_hdd0/packages/movian.pkg)
    /install.ps3?url=<url-of-pkg> (this command would download and install the pkg)

    /download.ps3?url=<url-of-file>
    /download.ps3?to=<path>&url=<url-of-file>

    The reason of having "?to=" first is to reduce issues with URIs containing parameters.
    If "?to=" is omitted, I think it's better to create the folder, instead of try to use multiple possible paths.
     
    STLcardsWS and atreyu187 like this.
  8. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    I forgot to say that at the moment, I have restricted the possible file type to be downloaded to only .pkg. However the command is capable to download other files.. Eventually I could authorise all file types to be downloaded. Therefore a download.ps3 command makes sense...
    No problem about the syntax, I will make the modifications you are asking for.

    Also the command does try to create the folder first if the path is wrong or omitted & only tries to use the next path in line if folder creation fails!


    Strange about this error with the download command, I will check this as soon as I get home.....

    Also I will pm you the source code after making the changes & ensuring the download command works.
    At the same time you can give me your thoughts about the workaround used to avoid the XMB freezes with file download to see if you can think of alternatives...



    Sent with Tapatalk
     
    Last edited: Jul 11, 2016
    STLcardsWS and aldostools like this.
  9. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    Sorry I made a typo error with the delimiter syntax I posted earlier. I edited the post.

    The delimiter to use with the download command is '$' NOT '?' .

    For example:
    hxxp://<ip-address-console>/pkg.ps3$downloadfrom=<pkg-url>&downloadto=<storage-path>

    Sent with Tapatalk
     
    0_obeWAN and aldostools like this.
  10. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    @aldostools

    I would like to check whether the passed url exists before launching the download itself.
    What do you think of this type of check?

    CellHttpClientId client = 0; CellHttpTransId trans = 0; CellHttpUri uri;
    uri.scheme="<httpmode>"; uri.hostname="<hostname>"; uri.username="";
    uri.password="";
    uri.path="<urlpath>";
    uri.port=<port>;
    int code = 0;
    int ret;
    ret = cellHttpCreateClient(&client);
    if(ret<0)
    {
    sprintf(msg_durl,"An error occurred when checking the Url. Could not create client. \nReturn value: %d\n",ret);
    show_msg(msg_durl);
    goto end_download_process;
    }
    ret = cellHttpCreateTransaction(&trans, client, CELL_HTTP_METHOD_GET, &uri);
    if(ret<0)
    {
    sprintf(msg_durl,"An error occurred when checking the Url. Could not create transaction. \nReturn value: %d\n",ret);
    show_msg(msg_durl);
    goto end_download_process;
    }
    ret = cellHttpSendRequest(trans, NULL, 0, NULL);
    if(ret<0)
    {
    sprintf(msg_durl,"An error occurred when checking the Url. Could not send request to client. \nReturn value: %d\n",ret);
    show_msg(msg_durl);
    goto end_download_process;
    }
    ret = cellHttpResponseGetStatusCode(trans, &code);
    if(ret<0)
    {
    sprintf(msg_durl,"URL is invalid.\nReturn value: %d\n Error code: %d\n",ret,code);
    show_msg(msg_durl);
    goto end_download_process;
    }

    Can you think of a better idea?

    Sent with Tapatalk
     
    Last edited: Jul 11, 2016
  11. 1,642
    1,612
    123
    aldostools

    aldostools Developer

    Joined:
    Oct 30, 2014
    Messages:
    1,642
    Likes Received:
    1,612
    Trophy Points:
    123
    Location:
    /dev_hdd0/tmp/wmtmp
    Home Page:
    I haven't tested your code, but it looks good. I would check also that the response code is not 4xx or 5xx (client or server error).

    IRISMAN uses a similar code in updates.c, but it only checks for errors 404 and 403 (not found & forbidden).
    I suggest you that view the functions download_file or download_update (which also supports SSL) to get some ideas.

    For a complete list of http status codes check this link

    BTW the error messages can be smaller ;)
     
    Last edited: Jul 11, 2016
    bguerville likes this.
  12. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    Great. SSL support would be nicer.

    Yes, the error messages should be changed. They were written quickly & aimed at myself rather than users in general anyway.

    Update:
    I just updated the pkg file 5mn ago as I found a small typo error in the source.

    Everyone please download the pkg again to make sure you have the latest version....

    Mirror 1

    Edit:

    Deleted link. Outdated.
     
    Last edited: Jul 13, 2016
    STLcardsWS, MixeryMaxe and aldostools like this.
  13. 4,244
    1,991
    123
    kozarovv

    kozarovv Super Moderator

    Joined:
    Nov 8, 2014
    Messages:
    4,244
    Likes Received:
    1,991
    Trophy Points:
    123
    Home Page:
  14. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    I updated the mirror with a fixed pkg.
    I realise only now that I uploaded the wrong test build last night.., yeah I know it was careless & sloppy but for my defence it was 4am! :tranquillity:
    As it turns out the download command was buggy in that build.


    Sorry guys..
    Feel free to download this fixed version.

    For information, it's not possible to use /dev_hdd0 as a download path, if you do the system download plugin defaults to /dev_hdd0/tmp/downloader.
    Except the root of the internal hdd, other paths I tried on /dev_hdd0 can be used without problems.

    The URL check code (with SSL support) is also ready. It should be included in the next build.

    There is one more tiny change for me to make for the syntax to be identical to what Aldo advised but for now it is as follows:

    Download Command
    hxxp://<ps3-ip-address>/download.ps3?url=<file-url>&to=<storage-path>
    the parameter "&to" is optional, if unused the default directory /dev_hdd0/packages will be chosen and in case of a problem it falls back to /dev_hdd0/tmp/downloader as a last resort.

    Install Command
    hxxp://<ps3-ip-address>/install.ps3<pkg-file-path>


    Download Link: Mirror 1
     
    Last edited: Jul 18, 2016
    lord3490, aldostools and kozarovv like this.
  15. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    Here is another update.
    You can use the link from post #15.

    This time the syntax overhaul is complete.
    More bug fixes as well.
    Next release will include URL check before download.


    Final syntax of /download.ps3:
    hxxp://<ps3-ip-address>/download.ps3?to=<storage-path>&url=<file-url> using the optional "to" parameter.
    hxxp://<ps3-ip-address>/download.ps3?url=<file-url> which defaults to /dev_hdd0/packages for storage path.
    The /install.ps3 command is unchanged:
    hxxp://<ps3-ip-address>/install.ps3<pkg-file-path>

    Enjoy...

    @aldostools
    The source is most likely coming with the next release! :cheerful:
     
    0_obeWAN, lord3490 and aldostools like this.
  16. 1,642
    1,612
    123
    aldostools

    aldostools Developer

    Joined:
    Oct 30, 2014
    Messages:
    1,642
    Likes Received:
    1,612
    Trophy Points:
    123
    Location:
    /dev_hdd0/tmp/wmtmp
    Home Page:
    Bug report...
    [​IMG]

    BTW it would be nice if you add the changes in games_xml.h from 1.43.30 (the other changes are space formatting and update of version).

    Also you could add this code to file_manager.h (add it before the 2nd #ifdef COPY_PS3)
    Code:
    #ifdef INSTALL_PKG
    else if( !strcmp(ext, ".pkg") || !strcmp(ext, ".PKG") )
        sprintf(fsize, "<a href=\"/install.ps3%s\">%'llu %s</a>", templn, sz, sf);
    #endif
    
    This code would let install any pkg remotely just clicking on the size column of the pkg listed on the web version of the file manager.

    A shorter form of the message "Installation of /dev_hdd0/packages/movian.pkg is under way" could be: "Installing /dev_hdd0/packages/movian.pkg"
     
    Last edited: Jul 17, 2016
    bguerville likes this.
  17. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    No problem Aldo. Thanks for the bug report & cool suggestions.
    I will look into it straight away.

    I was wondering whether you would have something to suggest regarding a feature I haven't been able to implement so far.
    Obviously I tried to find a way for a pkg to install directly after download however I have no idea how to catch the download plugin unload.
    Am pretty sure one of the download_plugin.sprx functions allows to do this but the documentation mysis left us for this plugin is largely incomplete & I have not been able to identify which one. I tried to select best candidates according to passed parameters but without luck.
    Any suggestions?

    FYI I am currently uploading the source to my github account, it will make things easier for us overall, especially if I keep working on other new commands.



    Sent with Tapatalk
     
    aldostools likes this.
  18. 1,642
    1,612
    123
    aldostools

    aldostools Developer

    Joined:
    Oct 30, 2014
    Messages:
    1,642
    Likes Received:
    1,612
    Trophy Points:
    123
    Location:
    /dev_hdd0/tmp/wmtmp
    Home Page:
    webMAN MOD build 1.43.31 (I changed your source code a bit)
    https://github.com/aldostools/webMAN-MOD/releases/tag/1.43.31
     
    STLcardsWS, Badboy and bguerville like this.
  19. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    Update


    I updated the project to 1.43.30.
    Fixed the the pkg extension check issue in the install command, it was a simple typo using strchr instead of strrchr therefore returning the first occurence of the '.' character instead of the last one.....
    The rest seems.ok.

    You can download the pkg from this mirror.

    Mirror

    @aldostools
    The change in file_manager.h works great but I had to add to modify the 2nd COPY_PS3 if statement with
    #ifndef PKG_HANDLER
    || !strcmp(ext, ".pkg")
    #endif
    otherwise the copy link took precedence.
    I reused strcmp as that was already in the if statement but this extension check could actually use strcasecmp instead, wouldn't it be more appropriate?

    Also the cosmetic bug "</" at the bottom of the webpage occurs in http_response but I don't really understand why. I didn't change this function at all. When I check the different values used by the function, I see no discrepancies.
    So I dumped the header variable to a log file just before it is sent with the ssend command to see what is being sent.
    I get this:

    HTTP/1.1 200 OK
    X-PS3-Info: [/install.ps3/dev_hdd0/packages/webMAN_MOD_1.43.26_Updater.pkg]
    Content-Type: text/html;charset=UTF-8
    Cache-Control: no-cache
    Content-Length: 242

    <body bgcolor="#101010" text="#c0c0c0"><font face="Courier New">webMAN MOD 1.43.30 MOD PKG Edition<hr><h2>Installing /dev_hdd0/packages/webMAN_MOD_1.43.26_Updater.pkg<hr> <input type="button" value=" &#9664; " onclick="location.href='/';"></font></body></html>

    Now if I open this log with a browser there is no bug....

    So I tried other web commands like http://192.168.0.15/remap.ps3/dev_usb000&to=/dev_hdd0/packages & I get the same cosmetic bug, the header variable being sent to the browser is:

    <body bgcolor="#101010" text="#c0c0c0"><font face="Courier New">webMAN MOD 1.43.30 MOD PKG Edition<hr><h2>Remap: <a href="/dev_usb000">/dev_usb000</a><br>To: <a href="/dev_hdd0/packages">/dev_hdd0/packages</a><p>Unmap: <a href="/unmap.ps3/dev_usb000">/dev_usb000</a><hr> <input type="button" value=" &#9664; " onclick="location.href='/';"></font></body></html>

    If I open this log with a browser, again, there is no bug...

    WTH...???


    It's late, time to put my head down.
    I will look at it again tomorrow.
     
    Last edited: Jul 18, 2016
  20. 4,604
    2,572
    123
    bguerville

    bguerville Moderator Developer

    Joined:
    Feb 25, 2015
    Messages:
    4,604
    Likes Received:
    2,572
    Trophy Points:
    123
    Location:
    Earth
    @aldostools
    I just realised that you posted a link to 1.43.31.
    I looked quickly at what you did, it looks perfect to me!
    No worries about changing the code, of course, on the contrary I was going to ask you to change it as you saw fit anyway.
    Nice one....:adoration:

    Gotta sleep for a couple of hours before work.
    Will look at your release in more details later...
     
    aldostools likes this.

Share This Page