Author Topic: gxp file format  (Read 3033 times)

0 Members and 1 Guest are viewing this topic.

Offline z80man

  • Casio Traitor
  • Project Author
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 977
  • Rating: +85/-3
    • View Profile
gxp file format
« on: July 08, 2011, 08:19:17 pm »
For the Walnut shell all machine executable programs will use the gxp format which is named after the popular 8xp extension used on the 83+. The packet design was originally designed by Qwerty.55 for use in a Prizm shell and has it's advantages in a variable size header that can be modified in the future with ease. The layout is in packets with each holding a different category. All are optional except for the first one which holds important information such as the name of the program. Other data such as the icon and advanced start up options are available in different packets. This is the current packet layout for the first version of Walnut. It is not set in stone and also keep in mind that the Walnut SDK will fill out these fields.

Code: (gxp header) [Select]

//all packets must be long word aligned

packet 0: required
word: identifier 0x0000
word: SDK version
long: Walnut identifier "wnut"
long: file offset to main
long: checksum of entire file excluding this
long: size of entire file
long: size of main
char: null terminated program name
char: null terminated description
//end of packet 0

packet 1: start-up info
word: identifier 0x0001
word: execution mode (0x0000: from flash in virtual memory, 0x0001: from ram in virtual memory, 0x0002: from flash in the p1 sector, 0x0003: from ram in the p1 sector, 0x0004: from flash in the p2 sector, 0x0005: from ram in the p2 sector)
long: requested origin (should be set to 0xffffffff in most cases for the default origin based off the previous mode choice)
long: value to be written to the FRQCR  0xffffffff for don't modify
//end of packet 1

packet 2: author info
word: identifier 0x0002
word: blank for alignment
long: author's 4 initial identifier eg "z80m"
char: null terminated author name
char: author logo compression mode (0x00: no image, 0x01: monochrome, 0x02: 8 color, 0x03: 256 color using Walnut's palatte, 0x04: 256 color with included palatte, 0x05: 16 bit bitmap, 0x06: 16 bit dxt)
image data: all author icons are 16x16 pixels
char: null terminated author name
//end of packet 2

packet 3: program icon
word: identifier 0x0003
word: height
word: width
char: icon compression mode (0x00: no image, 0x01: monochrome, 0x02: 8 color, 0x03: 256 color using Walnut's palatte, 0x04: 256 color with included palatte, 0x05: 16 bit bitmap, 0x06: 16 bit dxt)
image data:
//end of packet 3

List of stuff I need to do before September:
1. Finish the Emulator of the Casio Prizm (in active development)
2. Finish the the SH3 asm IDE/assembler/linker program (in active development)
3. Create a partial Java virtual machine  for the Prizm (not started)
4. Create Axe for the Prizm with an Axe legacy mode (in planning phase)
5. Develop a large set of C and asm libraries for the Prizm (some progress)
6. Create an emulator of the 83+ for the Prizm (not started)
7. Create a well polished game that showcases the ability of the Casio Prizm (not started)

Offline JosJuice

  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1341
  • Rating: +66/-14
    • View Profile
Re: gxp file format
« Reply #1 on: July 09, 2011, 10:15:40 am »
Based on what you've written in this post as well as the reply to Kerm's post on Cemetech, it seems like you're losing quite a bit of the flexibility that Qwerty.55's format was going to have. You have identifiers for which shell the program is for, and even what version of the SDK it was made with... These don't really have a purpose, unless you plan to treat programs that are for different shells/made with different versions of the SDK in separate ways. This is bad, because then old versions of shells won't be able to read those new files!

On the 8x calcs, programs from old shells work in new shells, but not vice-versa. Sometimes someone might want to use an older shell because they like it better, but they won't be able to do that. If we design this format in a good way, we might be able to avoid that problem completely on the Prizm. Programs should be made for all Prizm shells - not a specific shell on the Prizm. The format that Qwerty.55 suggested allowed new shells to add new features in files by adding new packages, while still allowing the files to work correctly with older shells, unless the packet was "critical", as indicated by a list.

There are also a few things with the packets that you currently have that I would like to be changed. (It's a good idea to split your packets a bit so that you don't have everything in one packet!)

1. Unlike Qwerty.55's format, the size of the packets don't seem to be stored anywhere. How is the shell supposed to know where they end in cases where the size of the packet isn't obvious?
2. Like I said before, I don't see the point in storing the SDK version and the program's shell.
3. The description of the program shouldn't be in the critical packet, because it's not necessary to include a description for every program.
4. The checksum should also be optional. Don't place it in the critical packet.
5. It's better to indicate main() by giving it a packet ID. "file offset to main" shouldn't be necessary.
6. Does "size of main" have a purpose?
7. Why do we need to include the first four characters of the author's name?
8. Please don't require programmers to include an author icon if they don't include an author name. The icon should be a separate packet.
9. Why is the author's name stored twice, once before the icon and once after?

Ashbad

  • Guest
Re: gxp file format
« Reply #2 on: July 09, 2011, 10:33:21 am »
^ some of those points I was gonna bring up last night, and JosJuice brought up some I haven't thought of.  Those are good points, and I stand by josJuices's critiquing.  Also though, why would there be shorts holding number identifiers for packets?   I think that they're pointless once you implement offsets to different packets later.  Unless I'm missing something :P. But if they are to stay, can you try to store them in either 2 bits for space or at least a char instead of a word (it would save only 4-7 bytes depending on the size stored, but hey that's 4-7 bytes more that can be used by either program data or RAM usage or whatever :3 )

Other than that, awesome :)
« Last Edit: July 09, 2011, 10:34:10 am by Ashbad »

Offline z80man

  • Casio Traitor
  • Project Author
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 977
  • Rating: +85/-3
    • View Profile
Re: gxp file format
« Reply #3 on: July 09, 2011, 06:20:23 pm »
Based on what you've written in this post as well as the reply to Kerm's post on Cemetech, it seems like you're losing quite a bit of the flexibility that Qwerty.55's format was going to have. You have identifiers for which shell the program is for, and even what version of the SDK it was made with... These don't really have a purpose, unless you plan to treat programs that are for different shells/made with different versions of the SDK in separate ways. This is bad, because then old versions of shells won't be able to read those new files!

On the 8x calcs, programs from old shells work in new shells, but not vice-versa. Sometimes someone might want to use an older shell because they like it better, but they won't be able to do that. If we design this format in a good way, we might be able to avoid that problem completely on the Prizm. Programs should be made for all Prizm shells - not a specific shell on the Prizm. The format that Qwerty.55 suggested allowed new shells to add new features in files by adding new packages, while still allowing the files to work correctly with older shells, unless the packet was "critical", as indicated by a list.

There are also a few things with the packets that you currently have that I would like to be changed. (It's a good idea to split your packets a bit so that you don't have everything in one packet!)

1. Unlike Qwerty.55's format, the size of the packets don't seem to be stored anywhere. How is the shell supposed to know where they end in cases where the size of the packet isn't obvious?
2. Like I said before, I don't see the point in storing the SDK version and the program's shell.
3. The description of the program shouldn't be in the critical packet, because it's not necessary to include a description for every program.
4. The checksum should also be optional. Don't place it in the critical packet.
5. It's better to indicate main() by giving it a packet ID. "file offset to main" shouldn't be necessary.
6. Does "size of main" have a purpose?
7. Why do we need to include the first four characters of the author's name?
8. Please don't require programmers to include an author icon if they don't include an author name. The icon should be a separate packet.
9. Why is the author's name stored twice, once before the icon and once after?
Okay I'll try to answer these questions as best as I can

1. Thinking about it now, to allow for newer programs to run on older shells the size of a packet should be the second word included that way Walnut can skip past unfamiliar packets. Otherwise Walnut was going to rely on knowing the size of the packet based off the SDK version but that can now be fixed.
2. The SDK version number allows for Walnut to know which version of the packet layout is being used. Having an SDK version number higher than the shell number shouldn't prevent a program from running but may cause some errors in displaying data. The way each packet is arranged shouldn't change once Walnut is released so that is why I want to get all the issues out now by making this public. Also for the "wnut" that is just an identifier that the shell uses so it knows that the file was designed for Walnut and uses the Walnut packet layout. It is just like how java .class files start with 0xcafebabe as an identifier.
3. That can be moved easily to another packet but even if the developer doesn't want to include a description that field will only take 1 byte if blank. The description is like a subtext that can be displayed with the title.
4. The reason for the checksum being there was that if there was a binary error in the header or executable then there is a possibility that the calculator could crash or even brick. For example if the FRQCR field was corrupted :P Don't forget that the SDK will calculate that on it's own when a project is built similar to what mkg3a already does.
5. A packet for main is also doable. The reason for having the offset was so that it could be quickly located and wouldn't have to be searched for.
6. Size of main just means the size of the executable. This is very important as it affects what Walnut will write to the MMU when the program begins.
7. The first 4 letters could also be more like their initials. This data will never be displayed to the user but is so that Walnut has the ability to organize programs by author if the user wishes to do so.
8. They don't have to include an icon. By setting the icon compression mode to 0 it means that there is no image and Walnut will not expect one.
9. Whoops ;)

List of stuff I need to do before September:
1. Finish the Emulator of the Casio Prizm (in active development)
2. Finish the the SH3 asm IDE/assembler/linker program (in active development)
3. Create a partial Java virtual machine  for the Prizm (not started)
4. Create Axe for the Prizm with an Axe legacy mode (in planning phase)
5. Develop a large set of C and asm libraries for the Prizm (some progress)
6. Create an emulator of the 83+ for the Prizm (not started)
7. Create a well polished game that showcases the ability of the Casio Prizm (not started)

Offline JosJuice

  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1341
  • Rating: +66/-14
    • View Profile
Re: gxp file format
« Reply #4 on: July 10, 2011, 03:12:44 am »
1. Thinking about it now, to allow for newer programs to run on older shells the size of a packet should be the second word included that way Walnut can skip past unfamiliar packets. Otherwise Walnut was going to rely on knowing the size of the packet based off the SDK version but that can now be fixed.
Good.
2. The SDK version number allows for Walnut to know which version of the packet layout is being used. Having an SDK version number higher than the shell number shouldn't prevent a program from running but may cause some errors in displaying data. The way each packet is arranged shouldn't change once Walnut is released so that is why I want to get all the issues out now by making this public.
I don't see how it would be necessary to know the version if the packet layout if the packets aren't going to change...
Also for the "wnut" that is just an identifier that the shell uses so it knows that the file was designed for Walnut and uses the Walnut packet layout. It is just like how java .class files start with 0xcafebabe as an identifier.
So if future shells want to have backwards compatibility with Walnut, would they need to include the wnut string? Wouldn't it be better to use something that doesn't refer to the name of this shell?
3. That can be moved easily to another packet but even if the developer doesn't want to include a description that field will only take 1 byte if blank. The description is like a subtext that can be displayed with the title.
Moving it to another packet makes the layout a bit more flexible. If everything is going to be put in the same packet, the advantage with the packet layout is not being used.
4. The reason for the checksum being there was that if there was a binary error in the header or executable then there is a possibility that the calculator could crash or even brick. For example if the FRQCR field was corrupted :P Don't forget that the SDK will calculate that on it's own when a project is built similar to what mkg3a already does.
But what if someone quickly wants to hack together a program without using the SDK? The file format is supposed to remove the limitations of the .g3a format, and the checksum is one of them.
5. A packet for main is also doable. The reason for having the offset was so that it could be quickly located and wouldn't have to be searched for.
6. Size of main just means the size of the executable. This is very important as it affects what Walnut will write to the MMU when the program begins.
This makes sense. I'm fine with having it the way it is now.
7. The first 4 letters could also be more like their initials. This data will never be displayed to the user but is so that Walnut has the ability to organize programs by author if the user wishes to do so.
Why can't authors be sorted based on their full name?
8. They don't have to include an icon. By setting the icon compression mode to 0 it means that there is no image and Walnut will not expect one.
Just like with 3., I think that splitting things up into multiple packets makes things more flexible.

Offline AngelFish

  • Is this my custom title?
  • Administrator
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3240
  • Rating: +269/-27
  • I'm a Fishbot
    • View Profile
Re: gxp file format
« Reply #5 on: July 10, 2011, 03:30:12 am »
@Josjuice:

2: That's a feature of most chunk based file formats. It's necessary if there are changes to the packet structure in the future.

Quote
So if future shells want to have backwards compatibility with Walnut, would they need to include the wnut string? Wouldn't it be better to use something that doesn't refer to the name of this shell?

It's just an arbitrary identification string. All future shells would be required to have it, but eliminating it would only make the shell type difficult to distinguish.

4: The file generator will take care of the details. The problem with the G3A CRC is that we don't know the generator polynomial. z80man's checksum doesn't require that, I believe.

7: What if the author's name is "Adolph Blaine Charles David Earl Frederick Gerald Hubert Irvin John Kenneth Lloyd Martin Nero Oliver Paul Quincy Randolph Sherman Thomas Uncas Victor William Xerxes Yancy Wolfeschlegelsteinhausenbergerdorffwelchevoralternwarengewissenschaftschafe rswessenschafewarenwohlgepflegeundsorgfaltigkeitbeschutzenvonangreifeudurch ihrraubgierigfeindewelchevoralternzwolftausendjahresvorandieerscheinenersch einenvanderersteerdemenschderraumschiffgebrauchlichtalsseinursprungvonkraft gestartseinlangefahrthinzwischensternaitigraumaufdersuchenachdiesternwelche gehabtbewohnbarplanetenkreisedrehensichundwohinderneurassevonverstandigmens chlichkeitkonntefortpflanzenundsicherfeuenanlebenslanglichfreudeundruhemitn icheinfurchtvorangreifenvonandererintelligentgeschopfsvonhinzwischenternart Zeus igraum Senior?" Fixed length names are smaller and infinitely easier to parse.

(real name)
∂²Ψ    -(2m(V(x)-E)Ψ
---  = -------------
∂x²        ℏ²Ψ

Offline JosJuice

  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1341
  • Rating: +66/-14
    • View Profile
Re: gxp file format
« Reply #6 on: July 10, 2011, 03:36:36 am »
4: The file generator will take care of the details. The problem with the G3A CRC is that we don't know the generator polynomial. z80man's checksum doesn't require that, I believe.
The checksum isn't going to cause any problems as long as we're using the SDK's file generator. However, someone will probably want to do things in a different way later on, and then it's good if the checksum isn't required. Of course, it should still be possible to have a checksum if the programmer wants to.
7: What if the author's name is "Adolph Blaine Charles David Earl Frederick Gerald Hubert Irvin John Kenneth Lloyd Martin Nero Oliver Paul Quincy Randolph Sherman Thomas Uncas Victor William Xerxes Yancy Wolfeschlegelsteinhausenbergerdorffwelchevoralternwarengewissenschaftschafe rswessenschafewarenwohlgepflegeundsorgfaltigkeitbeschutzenvonangreifeudurch ihrraubgierigfeindewelchevoralternzwolftausendjahresvorandieerscheinenersch einenvanderersteerdemenschderraumschiffgebrauchlichtalsseinursprungvonkraft gestartseinlangefahrthinzwischensternaitigraumaufdersuchenachdiesternwelche gehabtbewohnbarplanetenkreisedrehensichundwohinderneurassevonverstandigmens chlichkeitkonntefortpflanzenundsicherfeuenanlebenslanglichfreudeundruhemitn icheinfurchtvorangreifenvonandererintelligentgeschopfsvonhinzwischenternart Zeus igraum Senior?" Fixed length names are smaller and infinitely easier to parse.

(real name)
Isn't it possible to just read the first four characters of that name?

Offline AngelFish

  • Is this my custom title?
  • Administrator
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3240
  • Rating: +269/-27
  • I'm a Fishbot
    • View Profile
Re: gxp file format
« Reply #7 on: July 10, 2011, 03:39:47 am »
How do you know where the name ends, though? If the length isn't fixed, you have to either trawl through the entire name to find the next packet or store the size of the name in the packet. It'd also take up a lot more space than is necessary, since the user would only ever see the first four letters if they're the only things read. It's much better just to set a hard limit of 4 characters than to allow arbitrary length and deal with potential buffer overflows.
∂²Ψ    -(2m(V(x)-E)Ψ
---  = -------------
∂x²        ℏ²Ψ

Offline JosJuice

  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1341
  • Rating: +66/-14
    • View Profile
Re: gxp file format
« Reply #8 on: July 10, 2011, 03:43:25 am »
How do you know where the name ends, though? If the length isn't fixed, you have to either trawl through the entire name to find the next packet or store the size of the name in the packet.
I thought we were going to store the size of packets in the packets?
It'd also take up a lot more space than is necessary, since the user would only ever see the first four letters if they're the only things read. It's much better just to set a hard limit of 4 characters than to allow arbitrary length and deal with potential buffer overflows.
But shouldn't the user be able to see the full name even if it isn't used for sorting? If we're going to have a limit on the full name, it should be something long (like 256).

Ashbad

  • Guest
Re: gxp file format
« Reply #9 on: July 10, 2011, 09:11:52 am »
My input: 4 letters is horribly short -- why not 12-16 as a length, null terminated earlier if needed?

Offline z80man

  • Casio Traitor
  • Project Author
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 977
  • Rating: +85/-3
    • View Profile
Re: gxp file format
« Reply #10 on: July 10, 2011, 04:16:06 pm »
Think of the identifier as this. On Omnimaga my profile number is 1315 which SMF uses internally to organize all of my posts and account info. But to myself and every other user I'm seen as z80man. A similar concept is being followed by Walnut. Internally Walnut will use "z80m" for me or I could also set is as my initials such as "jtm ". The user though will rarely ever see those details but will instead see my full written out name as "z80man" which is also included in the Walnut packet. Because the name is null terminated I could set it to Adolph Blaine Charles... without any issues as long as there is also a 4 character identifier associated with that name.

List of stuff I need to do before September:
1. Finish the Emulator of the Casio Prizm (in active development)
2. Finish the the SH3 asm IDE/assembler/linker program (in active development)
3. Create a partial Java virtual machine  for the Prizm (not started)
4. Create Axe for the Prizm with an Axe legacy mode (in planning phase)
5. Develop a large set of C and asm libraries for the Prizm (some progress)
6. Create an emulator of the 83+ for the Prizm (not started)
7. Create a well polished game that showcases the ability of the Casio Prizm (not started)

Offline JosJuice

  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1341
  • Rating: +66/-14
    • View Profile
Re: gxp file format
« Reply #11 on: July 10, 2011, 04:21:38 pm »
I understand what you mean, but I can't see the purpose of doing so... On Omnimaga, you need an unique ID to make sure that someone else doesn't log on to your account and posts stuff using your user name. This issue doesn't really exist when it comes to shells, so why isn't the full name enough?