Omnimaga
Calculator Community => TI Calculators => Axe => Topic started by: BlackCode on August 24, 2013, 01:20:49 am
-
Hello everyone, first post and all! I just have some questions about the "best" way to store map data. I've been playing around with Axe, and am starting to work on a map, but I'm not sure how to store data in a way that fits my needs. I need to be able to change the map while the program is running. Ideally, I also need a large space to store my data and/or a way of efficient compression (consider that maps will likely be highly detailed). Any and all help is appreciated, even if it just consists of linking me to a resource I've missed.
-
You need an appvar. I will explain how to create one later since I am on my phone at the moment and typing is a pita.
Welcome to omnimaga by the way!
-
Yeah, appvars are what you are looking for. Here (http://ourl.ca/8294) is the tutorial that taught me everything about them. Besides, it doesn't only teach about appvars. Be sure to give the author a +1.
And welcome to Omnimaga, you can introduce yourself here (http://www.omnimaga.org/index.php?board=10.0).
-
And as for world maps: it really depends on the application. If your world consists of screens of tiles you should organize your data in a different manner than if your world consists of a large tile map. As for compression: If you only use 16 different tiles you can store two tiles in every byte instead of just one per byte. If you want to go even further you can replace all zero tiles with a zero followed by the number of zero tiles until a new kind of tile occurs. This technique is only suited if there is just a single tilemap in your appvar or if you indexed where all tilemaps start.
Example for storing 2 tiles in one byte:
.Not actual sourcecode:
.tile1 = 7
.tile2 = 3
.tile1 << 4 | tile2 = $73
.Axe syntax:
[73]Example for compressing zero tiles:
[15 00 26 00 00 00 00 00 00 00 00 24 A5]
.becomes
[15 00 01 26 00 08 24 A5]
.Downside to this technique is that single zero's will take up more space.
.Now to decompress:
.if the byte = 0
.load counter with next byte that defines the amount of zero's that should be inserted
.do
. insert 0 tile
. counter decrease
.while counter is not 0
.when the counter runs out go to the next tile and repeat the process until you aquired enough tiles.
-
do the changes you want to make to the map have to be written back and saved for the next time the program is run? if so, that's going to make compression stuff difficult. if they don't, you can just break the map up into compressed chunks and only decompress them when that section of the map needs to be loaded. if they do, though, you might be better off storing the changes applied to the map separately so that they can be changed dynamically without having to recompress the entire chunk. having to both decompress map sections and apply a diff can slow things down quite a bit, though, so it becomes a bit of a balancing act between saving space and saving time.
-
Another efficient compression method is the one deeph and chickendude showed me. Basically, it is RLE, but with a special treatment for isolated tiles. The only downside is that it only works if you have less than 128 tiles.
So, when you have 8 instances of a 4, you just put a 8 followed by a 4 in your data, and you just used 2 bytes for 8 bytes of data, so you saved space.
But when you have 1 instance of a 4, putting a 1 followed by a for would use 2 bytes for 1 byte of data. So instead of that, you'll store 4+128 (basically a 4 but with an extra bit at the "beginning").
And now, to uncompress, you just have to look at the first bit:
-if it's 0, treat the two bytes like RLE compressed data
-if it's 1, then put one instance of the byte (and don't forget to retrieve 128 from it).
-
Thanks for all the responses! I've checked out those tutorials, and appvars do sound like what I need, however I still have a few concerns. Because I would need to be able to write to the appvar at anytime, it would have to be unarchived throughout the program, so how much space would I actually have in an appvar and/or is there any way around this that would allow me to write to an archived appvar?
-
writing directly to archive is both tricky and a terrible idea, as it risks bricking your calculator. also, like i said above trying to write back a huge appvar every time your program is exited will be annoying for the user (taking a long time, causing lots of garbage collects, and, eventually, wearing out the flash). to get around both this issue and the issue of having too little ram during runtime, you basically have one of two options:
if you expect your user to only modify a few tiles here and there, the best method is to keep separate diff files for sections of your map, like i described above.
if you expect basically every tile on the entire map to be overwritten, you might want to break your map up into smaller chunks and write back only those that have been modified, decreasing the amount of writing necessary a bit.
-
Shmibs, could you perhaps elaborate on the differences between those two methods? I'm not seeing how they would vary. Additionally, wouldn't chunking the map mean consistent slow downs throughout the game as certain chunks were unarchived and archived. And it would compound the issue of wearing out flash (which I've seen as only happening after over 100,000 writes, something I don't forsee happening if it only writes to flash once, at the end of each program).
-
With the first method he is describing, I believe he means that you basically keep a list of changes in something like a save file. This way your tilemap data can remain archived, but when you copy (I mean copy, not unarchive) it to RAM, you check the list of changes to see if any of the tiles have to be modified. So for example, if Map00 has the (3,3) tile modified to a 6, you might include in the list a sequence of bytes: 00030306. When the player enters Map00, you copy the appvar data to RAM, then you read the list of changes to see that (3,3) needs to be changed to a 6.
-
And it would compound the issue of wearing out flash (which I've seen as only happening after over 100,000 writes, something I don't forsee happening if it only writes to flash once, at the end of each program).
Don't forget that other programs have to use that same flash. Every write you do is one less write for other people. That said, writing once per run is perfectly reasonable. Using flash as if it were RAM is not, which is the point I think shmibs was trying to make.
-
Ah, thank you Xeda, that clears it up a bit. If I were to do that though, I would still have to either archive the list of changes, or change the base map and archive that, which doesn't really seem efficient to me. So, it seems, to me, that my best bet is using a single appvar to hold all the data, unarchiving it at the beginning, and archiving it at the end of the program.
And AngelFish, I am well aware of that, which is why I was suprised when shmibs suggested using multiple appvars as a better method.
-
i suggested multiple because breaking up your map into smaller pieces allows you to only write back those which have had changes made to them.
-
But that would vastly increase the number of archive writes, would it not? A fairly severe problem (after a while).
-
The life of flash memory decreases with archive sector erases, not archive writes. Each sector is 64KB large. If you consistently unarchived and archived variables, each 64KB written to archive would result in one erase of the swap sector (probably the first sector likely to fail) at the next garbage collect. It is very roughly estimated that sectors may be likely to fail starting around 100,000 erases, so a total of about 6GB (yes, gigabytes) would need to be archived in the calculator's lifetime to start worrying about the flash memory failing.
-
Ahh yes, thanks for the reminder Runner. I forgot it was the erases, not the writes. Regardless, each archive writes to a new place, correct? So chunking the map and archiving/unarchiving multiple times per session would fill up the memory faster than just archiving once at the end, which would result in more flash clears. Of course, whether anybody should actually worry about this is questionable.
-
If the map is small enough that it can reasonably fit entirely in RAM, there's no reason not to do it that way. Splitting the map into chunks is mainly useful (necessary, really) when the map is larger than can fit into RAM.
-
How much exactly can fit into ram? I've seen 4,000 bytes of free ram, but is this the only place in which map data can be stored? I would assume more is available...
-
Usually, when the user archives everything, there are more than 20 KB of RAM on the calc.
-If you compile your code as an app, it will be run from Flash and you'll have those 20 KB all for your appvar.
-If you compile your code as a program, it will be run from RAM (even if you "run it from archive" with a shell, in fact it will be copied to RAM) and the RAM available for your appvar will be the 20KB-size_of_the_program.
-
Ok, that makes sense. Two questions in that case: 1. Is there any way to detect if the user has unarchived programs? 2. Is there a rule of thumb for how long unarchiving/archiving an appvar (based on its size) will take?
-
1. Is there any way to detect if the user has unarchived programs?
Yes, through the VAT, but that is not really needed.
What you want is basically to know if you have enough RAM for your program to run, right ?
Then use the same GetCalc command that is used in the tutorial. You'll see that it returns when you try to create an appvar if it could not be created (because of lack of RAM). So just do your code like that (where I use random names for variables):
GetCalc("appvNAME",Size)→P
!If
Disp "Not Enough RAM"
Return
End
<then the rest of the program>
And you can be more precise in your "Not Enough RAM" message, telling the user to archive variables, or the exact amount of RAM needed...
-
you can use the bcall _ErrNotEnoughMem to check if enough free ram is available (usable in axe via inline asm). apparently 9815h also contains the total amount of ram free, so you could check there too.
also, like several other people have said already, there is no reason for you to be unarchiving anything ever during your program's execution. all that "unarchiving" does, anyways, is copy the variables to RAM and then delete them from the archive, and there's no reason for them to be deleted.
-
Alright, thanks for the help with handling available space. But Shmibs I must be misunderstanding something...at somepoint the map data would need to be unarchived in order to alter it would it not?
-
Alright, thanks for the help with handling available space. But Shmibs I must be misunderstanding something...at somepoint the map data would need to be unarchived in order to alter it would it not?
You do need to place map data in RAM to alter it, but you don't have to both copy it to RAM and delete it from flash (which is what unarchiving does) – simply copying it to RAM is enough.
-
^exactly. axe allows you to read variables directly from the archive using what it calls files. basically, you assign a var to a file (which use the Y1-Y9 function variables) and then refer to it as you would any other pointer. that's why i was talking about using several smaller map chunks rather than one giant map; if you split your 20kb map into 4kb chunks and then the user only modifies two of those chunks, you only need to write back those 8kb upon exiting rather than the entire 20kb map. again, depending on what sort of game you're making, there are lots of more efficient ways to reduce the necessary amount of writing, so it would be nice to know more specifically what sort of game you want to make (particularly things like how often you expect tiles to be changing and how many different values they can take on etc.).
-
Alright, I get what you mean with copying. As far as more information, I'll give you what I can. I would expect most users to heavily modify tiles in a fairly concentrated area, and only do light changes in the rest of the map. However, there is no way to predict where these heavy changes will take place, and it is entirely possible, although unlikely, they will heavily modify everything after a long time. As for number of tiles, I would guesstimate under 64ish, with only about 16-20 being extremely common throughout the world.
Edit: I've been thinking some more, and I feel like chunking the data into smallish appvars is the best route. My map would consist of multiple appvars, of which, three would be loaded into ram at a single time: the chunk the player is at, and a chunk to either side. If the player modified any of the tiles in a chunk, the new data would be saved to the appvar, and that appvar would be flagged as modified. When the player changed chunks, the one chunk that will no longer be loaded is checked to see if it was modified, if it was, it is archived, if not, it is simply discarded. If anybody has any feedback on this (completely theoretical) method, please share.
Note: I considered using a list of changes that took place, but quickly ran into too many obstacles. For example: handling multiple changes to the same tile and preventing the list of changes from getting excessively large.