Author Topic: [TUTORIAL] How to save data in a program  (Read 3003 times)

0 Members and 1 Guest are viewing this topic.

Offline Deep Toaster

  • So much to do, so much time, so little motivation
  • Administrator
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 8206
  • Rating: +758/-15
    • View Profile
    • ClrHome
[TUTORIAL] How to save data in a program
« on: June 27, 2012, 08:14:42 pm »
Well, parserp asked me for this weeks ago, but apparently I forgot until he posted the question again today :P

What is SMC?

You may have noticed that some assembly games (such as Phoenix) can save a user's settings, high scores, and even paused game state without ever seeming to create appvars or other variables to hold that data. Instead, they store the information in the program itself.

This is called self-modifying code, or SMC for short. With SMC it's actually possible to create a program that modifies its own code while it's running, but that's usually a tricky thing to do, and especially in a compiled language like Axe where the programmer doesn't have full control over the machine code produced. However, data in Axe is easy to control. At the low level Axe is on, everyone's dealing with individual bytes and pointers, and since they're your bytes and pointers, you know (or should know) what they mean.

It's just data

Data inside a program is no different from data in another variable. We could keep a two-byte high score at GDB0:



Remember, any strings or hex data or data in a Data( command are put inside the program, and whenever they're "stored" to a static variable (such as GDB0, Pic0, and Str0, the static variable becomes a pointer like any other pointer (such as L1). So it's easy enough to manipulate data in a program. All that's left is to figure out a way to make the changes stay until the next time the program is run.

But that's where things get messy, because how the user runs your program makes a big difference in whether or not changes made with SMC will remain. The simplest case is when the program is in RAM and the user is running it from a shell, such as Ion, MirageOS, or DoorsCS. In this case, there's only one copy of the program, and whatever changes you make to it (or it makes to itself) are permanent. If the program was in archive, it gets trickier. Because programs are never run directly from archive, the shell makes a temporary copy of it in RAM and runs that. Some shells, such as MirageOS and DoorsCS, can "write back" the program into archive—basically deleting the original program and putting the RAM copy, along with whatever changes you made, into the archive. If your user is using a shell that doesn't support writeback (or if they have it disabled, such as when the option is set in MirageOS), the changes are lost when the temporary RAM copy is deleted. Note that all this depends on how the user has his calculator set up; you can't force someone else's shell to write back your program to archive, in other words. (Well, you probably could, but you probably shouldn't. It gets messier still.)

The simple case

For people using shells, this will do:



The first time the program is run, it will display the current high score as 0. If it's run from a shell, it'll show 65535 every time after that.

Unfortunately, not everyone uses a shell to run programs. This isn't a problem for you if you compile your program in MirageOS or DoorsCS format, since those programs can't be run from the homescreen, but if you want to be nice and compile your program as Ion (so that everyone can run it, with or without a shell), you get to deal with the quirks of the Asm( command. That's because for whatever reason, the TI-OS makes a temporary copy of the program even though it's in RAM before running it. That causes a few problems. First of all, it means it's impossible to run a program with Asm( when the amount of free RAM is smaller than the size of the program (try it yourself). Second, it means if you want to use SMC, you'll have to deal with two copies of the program, one of which could be anywhere in RAM.

Complications

You can use GetCalc( to look for that "other copy," find the position of the two bytes of GDB0 in that variable, and store the high score there. But GetCalc( needs a string containing the variable name in order to work. You could assume it's the name that you gave the executable (in this case, prgmAWESOMEG). As soon as your user renames your program, though, you're out of luck. And in today's calculator world of shells and other assembly utilities, renaming programs is a breeze.

Fortunately, when a program is run from the homescreen with Asm(, the full name of the program is stored at a specific spot in memory. This particular "spot in memory" is called OP1, located at E8478, and because it's used for a wide variety of things, from variable management to floating-point math, it can quickly be overwritten. However, if you don't create, (un)archive, or delete any variables in your program, that "other copy" will stay in the same place. So if you save its location at the very beginning, you'll be able to work with it later on:



That's fine. Now we know that Q points to the "other copy" of the program. But how do we know where in the program we should store the high score? If you store it in the wrong place, you could easily overwrite code in the program. That would be very bad.

What we do know is that at least in the running copy of the program, the high score is stored at an address called GDB0. We also know that the start of the program that's currently running is at address E9D93, because that's where a program is always moved before it's executed. So we could subtract E9D93 from GDB0, and we'd get the offset into the program where we're storing our high score!

If you don't understand how that works, imagine you're trying to give someone directions to your house. You know what block it's on, but not how far it is from the end of the block. Yet (for whatever reason) you know the exact GPS coordinates of your house, down to the meter, and you also know the exact GPS coordinates at the end of the block, down to the meter. All you have to do is subtract one from the other, and you know how far your house is from the end of the block.

It's the same idea here—you know the absolute location of the high score (it's GDB1). You also know the absolute location of the start of the program (it's E9D93). All you have to do is subtract, and now you know where the high scores are, relative to the start of the program.

Now imagine the entire block your house is on just got blown off the face of the earth and lands in Siberia. The search team knows exactly where the block landed, and now you need to tell them which house you're in so they can rescue you. (In our case, the block landed at the address saved in Q.) Add this to the end of your program:



More complications

The program should now work fine for people using Asm(, but it can cause problems for people using a shell because shells don't put the name of the program in E8478. We need to check for that, or else our little routine could write two bytes anywhere in RAM when a user runs the program from a shell. (Why can't we all just get along?)

One thing we could do is use a magic number. Basically, we put a known value at a known position in the program, and before we save to {Q+GDB0-E9D93}, we look at the corresponding position relative to the address held by Q. If it's not the same value, then Q doesn't point to the program we think it points to, so we don't try to save the high score there.

At the beginning of your program, add this line:



I chose the hex string D1ECA510 because the bytes it holds are very rare in code, so there's a lower chance that it just so happens to appear somewhere else. You could use a different value; in fact, you probably should, or else we'd all be using the same magic number and it wouldn't be so uncommon anymore. Just try to pick something random. (A string of zeroes is a very, very poor choice.)

Now change the last line of your program to this:



(Remember, the bytes are flipped because the processor is little-endian—that is, two-byte values are stored and read with the least-significant byte in front. You don't have to worry too much about it.)

The final program



Spoiler For The final program, optimized:

To summarize, here's a table to organize all the possible cases:

Program in RAM
Program in archive
Shell with writeback
Modifications savedModifications saved
Shell without writeback
Modifications savedModifications discarded
Asm(
Save modifications manuallyN/A
« Last Edit: August 29, 2012, 05:36:30 pm by Deep Thought »




Offline parserp

  • Hero Extraordinaire
  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1455
  • Rating: +88/-7
  • The King Has Returned
    • View Profile
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #1 on: June 27, 2012, 08:18:52 pm »
:w00t: ^9001
« Last Edit: June 27, 2012, 11:05:10 pm by parserp »
ticalc.org | Cemetech | TI-Freakware | casiocalc.org

Spoiler For inactive:
Serving an LDS (Mormon) mission from Dec 2014 - Dec 2016

Offline blue_bear_94

  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 801
  • Rating: +25/-35
  • Touhou Enthusiast / Former Troll / 68k Programmer
    • View Profile
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #2 on: June 28, 2012, 09:47:07 am »
Actually, the copy executed is located at, IIRC, $9D93, so one copy stays in the same place.
Due to dissatisfaction, I will be inactive on Omnimaga until further notice. (?? THP hasn't been much success and there's also the CE. I might possibly be here for a while.)
If you want to implore me to come back, or otherwise contact me, I can be found on GitHub (bluebear94), Twitter (@melranosF_), Reddit (/u/Fluffy8x), or e-mail (if you know my address). As a last resort, send me a PM on Cemetech (bluebear94) or join Touhou Prono (don't be fooled by the name). I've also enabled notifications for PMs on Omnimaga, but I don't advise using that since I might be banned.
Elvyna (Sunrise) 4 5%
TI-84+SE User (2.30 2.55 MP 2.43)
TI-89 Titanium User (3.10)
Casio Prizm User? (1.02)
Bag  東方ぷろの

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #3 on: June 28, 2012, 10:12:01 am »
isn't that what he said?

Offline blue_bear_94

  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 801
  • Rating: +25/-35
  • Touhou Enthusiast / Former Troll / 68k Programmer
    • View Profile
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #4 on: June 28, 2012, 10:16:13 am »
No, he said that both of the copies could be anywhere in the RAM.
Due to dissatisfaction, I will be inactive on Omnimaga until further notice. (?? THP hasn't been much success and there's also the CE. I might possibly be here for a while.)
If you want to implore me to come back, or otherwise contact me, I can be found on GitHub (bluebear94), Twitter (@melranosF_), Reddit (/u/Fluffy8x), or e-mail (if you know my address). As a last resort, send me a PM on Cemetech (bluebear94) or join Touhou Prono (don't be fooled by the name). I've also enabled notifications for PMs on Omnimaga, but I don't advise using that since I might be banned.
Elvyna (Sunrise) 4 5%
TI-84+SE User (2.30 2.55 MP 2.43)
TI-89 Titanium User (3.10)
Casio Prizm User? (1.02)
Bag  東方ぷろの

Offline Hayleia

  • Programming Absol
  • Coder Of Tomorrow
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3367
  • Rating: +393/-7
    • View Profile
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #5 on: June 28, 2012, 10:38:24 am »
No, he said that both of the copies could be anywhere in the RAM.
you'll have to deal with two copies of the program, one of which could be anywhere in RAM.
I own: 83+ ; 84+SE ; 76.fr ; CX CAS ; Prizm ; 84+CSE
Sorry if I answer with something that seems unrelated, English is not my primary language and I might not have understood well. Sorry if I make English mistakes too.

click here to know where you got your last +1s

Offline blue_bear_94

  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 801
  • Rating: +25/-35
  • Touhou Enthusiast / Former Troll / 68k Programmer
    • View Profile
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #6 on: June 28, 2012, 10:40:59 am »
I stand corrected.
I still don't see why we shouldn't be able to run programs from the archive. Is it because running an archived subprogram from another archived program will cause the existing copy at $9D93 to be overwritten? Or is it different for TI-Basic?

EDIT: I also believe that if you try to run an Ion program without a shell, then it quits immediately.
« Last Edit: June 29, 2012, 12:44:54 pm by blue_bear_94 »
Due to dissatisfaction, I will be inactive on Omnimaga until further notice. (?? THP hasn't been much success and there's also the CE. I might possibly be here for a while.)
If you want to implore me to come back, or otherwise contact me, I can be found on GitHub (bluebear94), Twitter (@melranosF_), Reddit (/u/Fluffy8x), or e-mail (if you know my address). As a last resort, send me a PM on Cemetech (bluebear94) or join Touhou Prono (don't be fooled by the name). I've also enabled notifications for PMs on Omnimaga, but I don't advise using that since I might be banned.
Elvyna (Sunrise) 4 5%
TI-84+SE User (2.30 2.55 MP 2.43)
TI-89 Titanium User (3.10)
Casio Prizm User? (1.02)
Bag  東方ぷろの

Offline aeTIos

  • Nonbinary computing specialist
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3913
  • Rating: +184/-32
    • View Profile
    • wank.party
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #7 on: June 29, 2012, 06:25:09 pm »
blue_bear: No, in fact if it's an Axe program it runs fine since it doesnt call any ion specific routines.
I'm not a nerd but I pretend:

Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55838
  • Rating: +3151/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #8 on: June 29, 2012, 11:13:25 pm »
I stand corrected.
I still don't see why we shouldn't be able to run programs from the archive. Is it because running an archived subprogram from another archived program will cause the existing copy at $9D93 to be overwritten? Or is it different for TI-Basic?

EDIT: I also believe that if you try to run an Ion program without a shell, then it quits immediately.
It depends which ones. I know Galaxian and Ztetris can be ran from the homescreen with Asm() too, despite being Ion programs. The calc may act weirdly after exiting Ztetris, though.
In case you are wondering where I went, I left Omni back in 2015 to form CodeWalrus due to various reasons explained back then, but I stopped calc dev in 2016 and am now mostly active on the CW Discord server at https://discord.gg/cuZcfcF



Official Website |T-Shirt store | Reverbnation | Facebook | Youtube | Twitter | Spotify

Offline Deep Toaster

  • So much to do, so much time, so little motivation
  • Administrator
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 8206
  • Rating: +758/-15
    • View Profile
    • ClrHome
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #9 on: June 29, 2012, 11:53:10 pm »
I stand corrected.
I still don't see why we shouldn't be able to run programs from the archive. Is it because running an archived subprogram from another archived program will cause the existing copy at $9D93 to be overwritten? Or is it different for TI-Basic?

EDIT: I also believe that if you try to run an Ion program without a shell, then it quits immediately.
It depends which ones. I know Galaxian and Ztetris can be ran from the homescreen with Asm() too, despite being Ion programs. The calc may act weirdly after exiting Ztetris, though.
There are two Ion header formats, depending on whether the program actually uses the Ion library or not. If the library is used, of course the program should quit when run from the homescreen (or it would almost definitely crash when it tries to run one Ion routine). If the library isn't used, there's no reason not to let people run the program with Asm(, and since Axe doesn't use any library routines, the Ion header Axe uses is the "universal" one, the format that can be used in just about every shell as well as from Asm(.

The Ztetris bug doesn't have to with Ion though. It's just a small oversight where the program doesn't clear the text buffer before quitting, so the data is used there doesn't go away until you clear the screen yourself.

And if you're wondering, the reason I haven't finished this tutorial is because I'm afraid I might have done something really bad. I'm not sure yet (I'll need to check on Saturday), but the routine I used might be corrupting a single byte in RAM each time you get a high score while running the program from archive from a shell that doesn't support writeback. (If you look on the table in the original post, you'll see that I marked it as "modifications discarded," meaning persistent data is pretty much impossible in that case, but that doesn't excuse me from being careless :/)

It's actually not hard at all to update the routines I used to get rid of this bug, but I don't have time to do anything until probably tomorrow night. And yes, everything I said in the original post is valid (as far as I know)—it's just that I implemented it in a lazy and sort of stupid way. Sorry.




Offline Deep Toaster

  • So much to do, so much time, so little motivation
  • Administrator
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 8206
  • Rating: +758/-15
    • View Profile
    • ClrHome
Re: [INDEV] [TUTORIAL] How to save data in a program
« Reply #10 on: August 29, 2012, 05:36:04 pm »
...two months later...

Yeah, I'm finally done :P

EDIT: Added to Tutorials.
« Last Edit: August 30, 2012, 12:19:24 pm by Deep Thought »