Author Topic: The .hpprgm format  (Read 5016 times)

0 Members and 1 Guest are viewing this topic.

Offline Hasse

  • LV0 Newcomer (Next: 5)
  • Posts: 3
  • Rating: +0/-0
    • View Profile
The .hpprgm format
« on: August 05, 2014, 04:41:36 am »
I spent some time reverse engineering the .hpprgm format (the format of the user programs of the Prime).
MAIN HEADER:
0x0000-0x0003:
    size of the header, excludes itself
    (so the next header begins at size+4)
0x0004-0x0005:
    Amount of variables in table
0x0006-0x0007:
    Amount of something? (Perhaps views or something?)
0x0008-0x0009:
    Amount of exported functions in table
0x000A-0x000F:
    unneeded?
    Conn. kit generates
    7F 01 00 00 00 00
    but all zeros seems to work too.
0x0010-0x----:
    Exported item table.
    Entry format is as follows:
    Type of item:
        30 00 for variable,
        31 00 for exported function
    Name of item:
        UTF-16, until 00 00 00 00
    Then the next entry follows.

VARIABLE VALUES:
(There are as many blocks of this type as you have exported variables and the
 blocks are in the same order as the exported variables)
0x0000-0x0003:
    size of the value, excludes itself
0x0004-0x0005:
    01 00 for detecting that this is a list
    02 00 for single value entry
IF single value entry:
    0x0006-0x0007:
        type:
            10 01 for base-10 integer or float
            11 20 for base-16 integer
            12 02 for string
        IF base-10 integer or float:
            0x0008-0x000B:
                exponent
                signed little endian 32-bit integer
            0x000C-0x0013:
                mantissa
                little endian weird stuff. Hexadecimal to be interpreted as decimal... WTF?
                00 00 00 00 00 00 00 25 01 is supposed to be 1.25 in decimal,
                00 00 00 00 00 00 00 28 06 -> 6.28 and so on
               
            The value is mantissa*10^exponent
        IF base-16 integer:
            0x0008-0x000B:
                02 00 00 00 (why?)
            0x000C-0x0013
                55 63 62 00 00 00 00 00 becomes #626355
                25 06 00 00 00 00 00 00 becomes #625
        IF string:
            0x0008-0x0009:
                Length of string in characters, excludes the tailing 00 00
            0x000A-:
                string itself, ends in 00 00
IF list:
    0x0006-0x0007:
        ??? (Tends to be 16 00, 16 01 or 16 02)
    0x0008-0x0009:
        32-bit LE
        Amount of members in list, let's call this N
    0x000A-0x000B:
        ??? (Probably reserved for something, 7F 01 or 00 00)
    N 4-byte values:
        ???
    Actual list of values follows, they are in reverse order compared to what
    they are in the source.
    An entry in the list follows this formula:
        Stuff gets clever and recursive, every entry in here is handled like a
        "VARIABLE VALUE" itself minus the size integer at the beginning

CODE HEADER:
0x0000-0x0003:
    size of the header, excludes itself
0x0004-:
    Code in UTF-16 LE until 00 00


Sorry if this is common knowledge already, I did try to search the forums (no results for hpprgm).  I did find this: http://tiplanet.org/hpwiki/HP_Prime/File_Format#User_BASIC_programs, but it seems that the information there is true only for some very simpe cases. I have no wiki editing skills so I can't update that.

I hope someone else find this useful and that someone continues the reverse engineering of the format (if that stuff truly is bytecode, it could be very useful). My own motivation is has dropped due to other, more interesting projects unrelated to calculators, which is why I stopped working on this.

EDIT: updated, that stuff is not bytecode, they are the encoded objects like Tim Wessman said.
« Last Edit: August 06, 2014, 06:34:44 am by Hasse »

Offline Adriweb

  • Editor
  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1708
  • Rating: +229/-17
    • View Profile
    • TI-Planet.org
Re: The .hpprgm format
« Reply #1 on: August 05, 2014, 07:24:05 am »
Hi,

Thanks for the info - the link you gave has some info about the program format as seen by [MOHPC] member Eried - maybe it was only on simple programs, indeed, I don't know. However, it may be enough for file transfers. One would have to check with complex programs, though, with his program (see https://github.com/eried/PrimeComm/blob/master/PrimeLib/PrimeUsbFile.cs )

If you still have a tiny bit of motivation, you're welcome to register on the wiki you linked and try editing on the sandbox for example (you can try anything there, it won't matter).

If you decide to do that and edit/contribute to the file format page, let us know (just reply here telling so), otherwise we can wait for a bit (maybe people will have some suggestions) and then edit it ourselves from what you've posted here :)
« Last Edit: August 05, 2014, 07:35:51 am by Adriweb »
My calculator programs
TI-Planet.org co-admin.
TI-Nspire Lua programming : Tutorials  |  API Documentation

Offline xlibman

  • Now active at https://codewalr.us
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55812
  • Rating: +3149/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • CodeWalrus
Re: The .hpprgm format
« Reply #2 on: August 05, 2014, 11:37:38 am »
Yeah I remember someone there made a tool to send such program and I think optimize them on MoHPC board, mentionned in the post above, although I don't remember if it could edit files. Also, when opening such file in Wordpad, I noticed that we could still somewhat dechiper some of the code, so the format is probably not too hard to dechiper.



But more info about the format is definitively nice if you have any to share, so thank for your post :)

Welcome to the forums by the way! :)
« Last Edit: August 05, 2014, 11:39:39 am by DJ Omnimaga »
In case you are wondering where I went, I left Omni back in 2015 to form CodeWalrus due to various reasons explained back then, but I stopped calc dev in 2016 and am now mostly active on the CW Discord server at https://discord.gg/cuZcfcF


Bandcamp|Reverbnation|Facebook|Youtube|Twitter
Retired Omnimaga admin (2001-11) and editor (2012-14)

Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3820
  • Rating: +80/-8
    • View Profile
Re: The .hpprgm format
« Reply #3 on: August 05, 2014, 11:56:10 am »
Lol that reminds me of HP40G programs. I'd some times print out the file to type it on calc.
send it

Offline timwessman

  • LV3 Member (Next: 100)
  • ***
  • Posts: 94
  • Rating: +32/-0
    • View Profile
Re: The .hpprgm format
« Reply #4 on: August 05, 2014, 06:40:25 pm »
Hello,

I'll copy in a post I made over on the MoHPC.

Quote
A gentle reminder to people about the hpprgm extension and others in use in the calculator - these are not plain text files!
 
 I've seen several people now post files labeled with an hpprgm extension when in fact they are plain text. Please remember that hpprgm files are in fact BINARY files and not meant for editing with a text editor. While you can open an exe with a text editor, making changes will cause problems and most likely crashes. The same WILL happen if you are manually tweaking hpprgm files.
 
 When the format changes as new features are added, your programs will confuse people as they try to load them and they fail. The fact that you can currently open them and they seem to show plain text primarily does not mean you should go ahead and edit things.
 
 You have been cautioned and please remember this! For the sake of people in the future getting totally perplexed as to why things aren't working, I'd ask you to keep your plain text source files as source, and only have the hpprgm extension on files that were generated by HP software (or some other specialty program).

We don't have any issues with giving out information about the files, but you can be pretty certain the format will be changing as things get added. The program file format is far from ready to lock everything down permanently yet. You pretty much have everything of interest right now correct. There is not any arm functions or calls saved in the file. I'd recommend just keeping out of the object table at the moment as those will be the most likely to change. The source area is probably pretty safe.

The basic structure/process for saving is this:

Code: [Select]
Save Total Size // save descriptor info
 Save All Vars/Exports //saves the names for use in accessing things
 Save the source //as described

I haven't checked for certain, but your "optional" headers might just be the HP Objs encoded. Reals, lists, etc. That is probably what you are seeing I would guess.
« Last Edit: August 05, 2014, 06:47:41 pm by timwessman »
TW

Although I work for the HP calculator group, the comments and opinions I post here are my own.

Offline timwessman

  • LV3 Member (Next: 100)
  • ***
  • Posts: 94
  • Rating: +32/-0
    • View Profile
Re: The .hpprgm format
« Reply #5 on: August 05, 2014, 06:46:36 pm »
*dup* How do I delete my dups? Possible?
« Last Edit: August 05, 2014, 06:48:58 pm by timwessman »
TW

Although I work for the HP calculator group, the comments and opinions I post here are my own.

Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3820
  • Rating: +80/-8
    • View Profile
Re: The .hpprgm format
« Reply #6 on: August 05, 2014, 06:53:06 pm »
You can no longer do it yourself since the upgrade to smf 2.0. Just wait for a mod to do it.
send it

Offline xlibman

  • Now active at https://codewalr.us
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55812
  • Rating: +3149/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • CodeWalrus
Re: The .hpprgm format
« Reply #7 on: August 05, 2014, 07:41:29 pm »
We don't have any issues with giving out information about the files, but you can be pretty certain the format will be changing as things get added. The program file format is far from ready to lock everything down permanently yet. You pretty much have everything of interest right now correct. There is not any arm functions or calls saved in the file. I'd recommend just keeping out of the object table at the moment as those will be the most likely to change. The source area is probably pretty safe.
Does it mean that older .hpprgm files will eventually no longer send in newer OS versions?
In case you are wondering where I went, I left Omni back in 2015 to form CodeWalrus due to various reasons explained back then, but I stopped calc dev in 2016 and am now mostly active on the CW Discord server at https://discord.gg/cuZcfcF


Bandcamp|Reverbnation|Facebook|Youtube|Twitter
Retired Omnimaga admin (2001-11) and editor (2012-14)

Offline cyrille

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 8
  • Rating: +0/-0
    • View Profile
Re: The .hpprgm format
« Reply #8 on: August 06, 2014, 01:28:02 am »
Hello,

note that a lot of the data in the header is optional, you can send a program without it's variables or any of the exported data info, put the source nevertheless and, once on the calculator, open, add a space and exit to compile... that makes it much easier if what you want to do is to edit programs on the PC...

note, prety much all the saves data in the calc follows a format of the type:
[header] [blob size   blob of data]*

in a program, there is no fixed size header, and the source is always the last entry in the file. This should make it easy to get to it without having to understand the format...

cyrille

Offline Hasse

  • LV0 Newcomer (Next: 5)
  • Posts: 3
  • Rating: +0/-0
    • View Profile
Re: The .hpprgm format
« Reply #9 on: August 06, 2014, 02:19:08 am »
I originally wanted to make a C to HP PPL translator (obviously without pointer support), since I like C more. I wanted the "compiler" to output valid .hpprgm files which is why I tried to reverse engineer the format. However, I know it takes a long time to get something usable and the difference between the languages isn't great enough to make the project worthwhile.

The "optional headers" were only in some bigger .hpprgms like the hangman game over at MoHPC. I probably should have tried to make large programs myself and modify them to see how the header changes, but I was too lazy  :-[ . None of the small programs I wrote myself contained those headers.

Interestingly the hangman game has all zeroes where connection kit generates 7F 01 00 00 00 00. Could this be caused by the program being written on an older connection kit?
 

Offline xlibman

  • Now active at https://codewalr.us
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55812
  • Rating: +3149/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • CodeWalrus
Re: The .hpprgm format
« Reply #10 on: August 06, 2014, 02:27:09 am »
That said, it's also possible to send a modified firmware file to the calculator, so someone could just modify the official file with a command to launch C/ASM programs then release an IPS patch version of the mod, then you would be set. :P
In case you are wondering where I went, I left Omni back in 2015 to form CodeWalrus due to various reasons explained back then, but I stopped calc dev in 2016 and am now mostly active on the CW Discord server at https://discord.gg/cuZcfcF


Bandcamp|Reverbnation|Facebook|Youtube|Twitter
Retired Omnimaga admin (2001-11) and editor (2012-14)

Offline Adriweb

  • Editor
  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1708
  • Rating: +229/-17
    • View Profile
    • TI-Planet.org
Re: The .hpprgm format
« Reply #11 on: August 06, 2014, 04:35:00 am »
Anyway, the format is known enough for several third-party tools (Eried's program, Critor's mViewer GX Generator, for example...) to generate valid files, so there's that. If the unsupported parts are actually not necessary and not long-term safe, well, we may not need to take time to understand them, it wouldn't be useful :)
My calculator programs
TI-Planet.org co-admin.
TI-Nspire Lua programming : Tutorials  |  API Documentation

Offline Hasse

  • LV0 Newcomer (Next: 5)
  • Posts: 3
  • Rating: +0/-0
    • View Profile
Re: The .hpprgm format
« Reply #12 on: August 06, 2014, 06:40:17 am »
Sorry, I was being a little melodramatic yesterday. Reverse engineering is like a puzzle, and not completely understanding the file is like giving up for me. I have problems with giving up. I continued the work and updated the post accordingly.  Turns out there is no bytecode and Tim Wessman was correct, they are encoded values.

Regarding the usefulness of all of this, I agree that this is probably worthless but it is a fun puzzle nevertheless.  :thumbsup:

Offline xlibman

  • Now active at https://codewalr.us
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55812
  • Rating: +3149/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • CodeWalrus
Re: The .hpprgm format
« Reply #13 on: August 06, 2014, 12:32:12 pm »
Indeed. There are a bunch of things that aren't necessarily useful on the TI-84+, but it didn't stop anyone from documenting the entire calculator hardware and software, so it wouldn't hurt if the same was done with HP calcs. :)
In case you are wondering where I went, I left Omni back in 2015 to form CodeWalrus due to various reasons explained back then, but I stopped calc dev in 2016 and am now mostly active on the CW Discord server at https://discord.gg/cuZcfcF


Bandcamp|Reverbnation|Facebook|Youtube|Twitter
Retired Omnimaga admin (2001-11) and editor (2012-14)

Offline cyrille

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 8
  • Rating: +0/-0
    • View Profile
Re: The .hpprgm format
« Reply #14 on: August 07, 2014, 02:19:28 am »
Hello,

well, I don't remember the specifics from the top of my head, but basicaly:
- First blob: header with number of exported function, variables (exported and not) and views.
This first blob also contains the list of strings with the names of these exported (or not) objects.

- n blobs: the content/value of the program global variables. This is where it is complicated as a varaible can contain expressions/functions and the like, even CAS objects... not that the format is complciated, it's just has a lot... from numbers, to matrices, lists, complex...

- 1 blob: source code.

cyrille