Omnimaga

General Discussion => Technology and Development => Computer Projects and Ideas => Topic started by: NecroBumpist on August 19, 2011, 07:58:46 pm

Title: Maximum Over Drive System - Lua Assembly Toolkit
Post by: NecroBumpist on August 19, 2011, 07:58:46 pm
Welp, I am in need of more posts before I can join the IRC apparently, so I will introduce you all to a project I'm working on :)

Since Lua is currently the fastest programming language for up-to-date NSpire calculators, I thought I would share this relevant project.
This really is for those who want to get the absolute most out of Lua. It's very impractical to write an entire program from Lua assembly, since the compiler does a decent job for most non-time-critical functions.

Lua is actually a semi-compiled language (at least that's what I'm going to call it).
It is compiled to bytecode, and than interpreted dynamically on a virtual machine.

You can execute Lua bytecode directly in Lua with the loadstring() function.

Code: (Example 1) [Select]
loadstring("\27\76\117\97\81\0\1\4\4\4\8\0\56\0\0\0\64\67\58\47\85\115\101\114\115\47\74\97\115\111\110\47\68\111\99\117\109\101\110\116\115\47\80\114\111\103\114\97\109\109\105\110\103\47\76\117\97\47\77\79\68\83\47\116\101\115\116\46\108\117\97\0\13\0\0\0\13\0\0\0\0\0\0\2\1\0\0\0\30\0\128\0\0\0\0\0\0\0\0\0\1\0\0\0\13\0\0\0\0\0\0\0\0\0\0\0")();

Now I'd go over the bytecode format, but you can learn all of that here (http://luaforge.net/docman/view.php/83/98/ANoFrillsIntroToLua51VMInstructions.pdf)
This document also includes all of the information on all of the instructions to the Lua VM.

Now, onto the project.
(In retrospect, the name I chose was rather silly, but meh, it applies)

Basically, it includes the following:

You can find the repository here: https://github.com/NecroBumpist/MODS (https://github.com/NecroBumpist/MODS)

So, time for "Hello World!" in Lua assembly
Code: [Select]
local mods = require("MODS.Current.main");
local bytecode = mods.Assemble(
[[
; Hello_World.lasm
.options 0, 0, 0, 2        ; a directive, which sets some properties, the '2' meaning MaxStackSize

getglobal    0,    <'print'>        ; <> denotes a constant, i've yet to bother with a directive for doing constants
loadk        1,    <'Hello World!'> ; The LuaVM is register based, we put the function in one register, then arguments above it
call         0,    2,    1          ; now we call register 0, with (2-1) parameters, accepting (1-1) values back
return       0,    0                ; and now we return
]]);

loadstring(bytecode)() -- turns the bytecode into a function, then we call that function

>Hello World!


So,  I'd like to hear what some of you Lua users who are looking for speed improvements think :)
I plan on updating the repository soon to include:

In ten or twenty minutes the repository should be much cleaner ;)

Comments, questions, or concerns ? Or Lua Assembly general thread.
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: XVicarious on August 19, 2011, 09:44:01 pm
Wow. That looks pretty cool, I know no Lua at all though but still :P
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: DJ Omnimaga on August 19, 2011, 10:48:41 pm
Hmm Interesting. I unfortunately no longer am into programming but I'M glad to see more programming tools coming out. For which platforms will this be?
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: Juju on August 19, 2011, 11:26:00 pm
Looks real nice. So that means another programming language for the NSpire? (And maybe anything that runs Lua?) Cool.
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: pianoman on August 19, 2011, 11:31:09 pm
Wow, looks pretty nice :)
Can't wait for it to be finished :D
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: NecroBumpist on August 20, 2011, 12:38:32 am
Hmm Interesting. I unfortunately no longer am into programming but I'M glad to see more programming tools coming out. For which platforms will this be?
I've yet to run the assembler/linker on calc (the code is horendous (no planning), and it would be really slow for large files.

But I'm mainly posting it because you can utilize whatever code you assemble on any platform that runs Lua.
So basically the NSpire series :)

Looks real nice. So that means another programming language for the NSpire? (And maybe anything that runs Lua?) Cool.
I'm not sure I would call it an entire new language, since the code it produces is ran normally in Lua. It's just a more minimalistic version of Lua.
So if you do your planning, you could definetly organize your code in way that would be better than the standard Lua compiler.
How much better, I do not know.

It's similar to the coding in C vs. ASM for x86. The best C compilers are better than average ASM coders.
But then again, the Lua compiler really does no optimzations at all. Writing important functions in Lua ASM might be of benefit, but writing simple or non-time-critical functions would likely be a waste of time.

I plan to do an example with a SHA256 algorithm tomorrow, one in normal Lua, another specifically in Lua ASM.
(I'm going to use an external library, so it won't run on calc, but it will show how Lua ASM can be helpful)
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: DJ Omnimaga on August 20, 2011, 02:08:39 am
Ah ok thanks for the info.
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: Jim Bauwens on August 20, 2011, 05:20:24 am
Very interesting!
I have yet to try if it actually works on the nspire, but if it does,that would be wonderfull :)
I've got lots of routines that could use a speed boost :)

Thanks for your work :)
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: pianoman on August 20, 2011, 07:55:21 pm
How exactly would we include this stuff in the actual script, though?
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: NecroBumpist on August 20, 2011, 08:16:21 pm
Update to v2.2
Now includes NASM like macros:

Code: (Example LASM) [Select]
.options 0, 0, 0, 200 ; yay for large stacks!

.macro print(s, msg){
    getglobal   s,  <'print'>
    loadk       s + 1,  msg
    call        s,  2,  1
}

print   0, <"Omnimaga :)">      ; this line will be replaced with everything in the print macro (this is good for inlining functions and stuff)
return  0,  0

>Omnimaga :)

It took forever to get this shit working, mainly because of my terrible code layout :P

Anyway, v2.3 will include cool stuff like automatically tracking the top of the stack so you don't have to use .options in every prototype, and LOTS of linker stage checks. This will prevent (well, help you figure out) the common "invalid bytecode" warnings loadstring() returns if you try to load bad code.

More can be read in the README in the repository (https://github.com/NecroBumpist/MODS).


How exactly would we include this stuff in the actual script, though?

You'd use the loadstring() function to convert the code Assemble() returns into a proper Lua function, which you can easily call and pass arguments to from your normal scripts.

Code: (Use this code to generate bytecode) [Select]
local mods = require("current");
local code = mods.Assemble([[
; put cool Lua ASM code here
]]);

print(code:gsub(".", function(a) return "\\" .. a:byte() end)) -- turn the code into an escape sequence
Code: (Use this code in your production code, or what you put on the calc) [Select]
local code = "" -- copy and paste the output of the previous script here
local func = loadstring(code);
func()
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: Adriweb on August 21, 2011, 07:29:57 am
Really nice indeed.

Looking forward to be able to test that project in more complex situations, once I understand how to program in this asm-looking language :P
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: NecroBumpist on August 21, 2011, 10:41:59 am
Really nice indeed.

Looking forward to be able to test that project in more complex situations, once I understand how to program in this asm-looking language :P

Thanks :)
I definetely do more basic documentation on the features of the assembler.

After that, I've included a copy of ANoFrillsIntroToLua51VMInstructions.pdf in the \docs\ folder(the de facto document for learning Lua bytecode, and ultimately Lua assembly)
So that should  be a great reference to all those who want to learn the specifics of a particular instruction.

I'll get to work on explaining how to use everything now  ;D
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: Levak on August 21, 2011, 11:49:02 am
Err,
Either I'm stupid (and this is not impossible), or I don't understand how you can excecute this function in the Lua TI-Nspire API :

Code: [Select]
require("MODS.Current.main");
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: NecroBumpist on August 21, 2011, 11:56:46 am
Yes, that wouldn't run, TI removed file interactions. The examples are really meant to be ran on a desktop.

I would not advocate using this on your calculator (as I've said before, it's slow),
but I'll steal ocLua's concept so you can, just for fun :)

Edit: in the \bin\ directory there is now a "ocLASM.tns" file. It works exactly like ocLua (many many thanks to ExtendeD :))
Though I still don't recommend actually typing LASM on an NSpire, as it is really tedious.
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: ExtendeD on August 22, 2011, 10:36:51 am
Interesting project :)
I'd love to see how TI-Nspire programs will benefit from it.
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: pianoman on August 22, 2011, 11:22:20 am
I'm sorry about all of the questions, but I'm still just a bit confused D:
Which file/folder would we make the example script you showed above in?
Then, once we do that, which script do we run to convert to bytecode?
And, finally, (yes, I know this will sound stupid) how do we run Lua scripts like that on the computer?
Thank you very much!

EDIT: Any chance you could make some sort of an API/less confusing and nspire-oriented tutorial? :w00t:
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: NecroBumpist on August 22, 2011, 06:56:42 pm
pianoman, I'm dearly sorry. I really have been neglecting properly documenting everything. I'll get working on that after I answer your questions :P

Which file/folder would we make the example script you showed above in?
Then, once we do that, which script do we run to convert to bytecode?
You could run it in the \src\ folder I believe.
The script you would use to convert it is the "current.lua" one.

So you would first have to require() that, (sort of like #include)

Code: (Lua) [Select]
-- let's assume this script is in the \src\ folder
local mods = require("current"); -- this runs current.lua, which returns a table of the major API functions

local bytecode = mods.Assemble([[

; put cool bytecode here

]]); -- this passes the multi-line string argument to Assemble(), which turns it into bytecode
-- The easiest way to utilize bytecode in a script is in an escape sequence like so: \104\101\108\108\111  (this is the ASCII sequence "hello")
print(bytecode:gsub(".", function(a) return "\\" .. a:byte() end)) -- this is the bytecode in the above format, which you can pass as a string argument to loadstring()

And, finally,  how do we run Lua scripts like that on the computer?
You would need the Lua executables:


Any chance you could make some sort of an API/less confusing and nspire-oriented tutorial?

Yes :)
I will also add command line support so you can do the following:
Code: (cmd.exe) [Select]
>lua current.lua input.lasm -o output.lua
Which will create a .lua file that will loadstring() an ASCII sequence like I demonstrated above :)

Also, thanks ExtendeD  ;D
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: pianoman on August 22, 2011, 09:41:49 pm
Nonono, NecroBumpist. You of all people have nothing to apologize for. If anything, I should be apologizing for asking so many questions and asking for so much so soon.
Thank you very much for your swift reply, and I can't wait to use all the cool things LASM can help us with :)
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: Ashbad on August 29, 2011, 09:40:24 am
This reminds me of the Sharp Assembly Suite that Harold made on cemetech a while back ( http://www.cemetech.net/forum/viewtopic.php?t=6355&postdays=0&postorder=asc&start=0 )

While it is different because it's intended for only one platform, some of the same questions Kllrnohj and others brought up remain, but concerning this "assembler" instead: assembling on the fly (which the does) will be quite slow, when usually Lua code when written well is going to be very optimized.  Won't the speed loss pretty much make this rather trivial, unless you re-use a certain Lua Bytecode module many many times (and the module actually holds optimized code)?  Would it not be better to build a separate "assembler" not built into a script using this technique that compiles some Lua IR into byte code that could be copy-pasta'd into a loadstring() call?

Just my 2 cents
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: Jim Bauwens on August 30, 2011, 03:03:18 am
The fun part on loadstring is that it return a function, which you will be able to use during the whole script. So if you do everything in the beginning (when you open the script), it should be fine :)
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: Ashbad on August 30, 2011, 11:21:28 am
The fun part on loadstring is that it return a function, which you will be able to use during the whole script. So if you do everything in the beginning (when you open the script), it should be fine :)

And you use that function a lot ;)
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: NecroBumpist on October 06, 2011, 12:36:37 am
Spoiler For For anyone knew to MODS, a quick summary:
The Lua language compiles to bytecode before it is ran.
Lua Assembly (LASM) is the format of this bytecode.

This thread was about my LASM assembler, named MODS (Maximum OverDrive System), but I wish to improve this.

Example LASM code:
Code: [Select]
.options 0, 0, 0, 2

getglobal 0, <"print">
loadk  1, <"hello world">
call 0, 2, 1
return 0, 1

I'm interested in reviving this project, but I need long time general assembly programmer's feedback.

I want to do two things with this project:
The 2.x line will be just about finished after I add more debugging tools such as a proper disassembler, which will output a file that can be properly parsed by MODS 2.x
However, I really dislike the way I had to implement A LOT of things, so I want to start from scratch.
To do this will require more planning, so I'm going to need to know what's actually useful in an assembler, so this is where I need help.

I've tried my hand in assembly before, briefly with x86, and I used NASM, which is where I've gotten some inspiration, but because of the limited time I used it for, I'm probably missing some things.

I incorporated NASM like macros into v2, which was a huge improvement, but because they were done so late in development, I did a pretty bad job of it, so they are more limited than I would of liked. This is one of the primary reasons I want to rewrite MODS.

Anyway, assemblers! Come one, come all! Please, offer your advice to me!
What sort of features do you use/like in your assemblers ?

Examples:
As I said, since I didn't get far into x86 assembly, I need help with ideas for MODS 3.
Title: Re: Maximum Over Drive System - Lua Assembly Toolkit
Post by: Jim Bauwens on October 06, 2011, 01:44:48 am
As much as I would like to help, I cant, as I don't know asm of any kind.
Good luck anyway!