Author Topic: Ndless bFLT loader  (Read 9921 times)

0 Members and 1 Guest are viewing this topic.

Offline tangrs

  • LV5 Advanced (Next: 300)
  • *****
  • Posts: 228
  • Rating: +97/-0
    • View Profile
    • tangrs blog
Ndless bFLT loader
« on: April 02, 2012, 01:33:00 am »
This topic will be for technical discussion for the bFLT loader.

Source code and API documentation

Let me start off this discussion.

I propose that, once bFLT becomes stable enough, it be integrated into the core Ndless code and have the SDK updated to only produce bFLT executables. However it seems that if bFLT were to be integrated into Ndless, we'll need to change some parts of the Ndless SDK itself to work in it's favor. I hope we can quickly deprecate the current way of loading binaries.

First, because bFLT stores sections in the order of .text, .data, .bss, we need to update the Ndless ldscript to reflect that. Otherwise, we run into overlapping problems. More information can be found at the bottom of the README. This is the first and easiest step.

Next: Because we do all the relocating, the startup files become redundant - even a hindrance when we deal with shared libraries. The only thing the startup files should do is branch to main since the bFLT loader will handle everything else.


Also, we need some sort of way to work out the assignment of library IDs. bFLT shared libraries work by IDs. Each library is given a unique ID and it is up to the user to ensure it is unique. Unfortunately, the number of IDs are limited and only 254 are actually available for use (255 and 0 are reserved).

I'll leave it at that for now.

@edit: is there a way of implementing _syscallvar_savedlr without relying on a linker script?
@edit2: I'm also having trouble producing a bFLT executable that doesn't rely on nspire-gcc or nspire-ld. Anyone have luck with this?
I'm having trouble writing a ldscript that will force the GOT to be at the beginning of the data section.
Got it.


Update:

The bFLT loader has a working SDK ready for developing and testing! You can now produce bFLT binaries without affecting your normal Ndless SDK!
« Last Edit: April 02, 2012, 09:18:13 am by tangrs »

Offline alberthrocks

  • Moderator
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 876
  • Rating: +103/-10
    • View Profile
Re: Ndless bFLT loader
« Reply #1 on: April 02, 2012, 09:04:07 am »
This is great! :D I was thinking about porting bFLT a while ago, but lacked skills and time to do it. This will definitely advance Nspire development quite a bit! :D

Also, do you think there's a way to change that limit?
Reading this, it says that the limit is "due to the way [he] encoded library addresses...can be changed but it requires quite a bit more effort".

I think the ID system should be changed if possible... it would be kinda weird to start assigning IDs, or having some kind of system manage IDs and such. I think having some kind of Java-style naming system, like "com.alberthrocks.myawesomelibrary", would be better. (And then call this bFLT2, Nspire edition :P) Just my 2 cents. ;)

Again, great work! :D This will open the platform to C++ code, and make porting ten-fold easier! :D

EDIT: Actually, bFLT has a version of 4.0 - I'm assuming you're using that spec, right? Also, might want to call the version 5.0 instead... :P
« Last Edit: April 02, 2012, 09:13:55 am by alberthrocks »
Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/


Proud member of ClrHome!

Miss my old signature? Here it is!
Spoiler For Signature:
Alternate "New" IRC post notification bot (Newy) down? Go here to reset it! http://withg.org/albert/cpuhero/

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/

Activity remains limited due to busyness from school et al. Sorry! :( Feel free to PM, email, or if you know me well enough, FB me if you have a question/concern. :)

Don't expect me to be online 24/7 until summer. Contact me via FB if you feel it's urgent.


Proud member of ClrHome!

Spoiler For "My Projects! :D":
Projects:

Computer/Web/IRC Projects:
C______c: 0% done (Doing planning and trying to not forget it :P)
A_____m: 40% done (Need to develop a sophisticated process queue, and a pretty web GUI)
AtomBot v3.0: 0% done (Planning stage, may do a litmus test of developer wants in the future)
IdeaFrenzy: 0% done (Planning and trying to not forget it :P)
wxWabbitemu: 40% done (NEED MOAR FEATURES :P)

Calculator Projects:
M__ C_____ (an A____ _____ clone): 0% done (Need to figure out physics and Axe)
C2I: 0% done (planning, checking the demand for it, and dreaming :P)

Offline tangrs

  • LV5 Advanced (Next: 300)
  • *****
  • Posts: 228
  • Rating: +97/-0
    • View Profile
    • tangrs blog
Re: Ndless bFLT loader
« Reply #2 on: April 02, 2012, 09:17:05 am »
This is great! :D I was thinking about porting bFLT a while ago, but lacked skills and time to do it. This will definitely advance Nspire development quite a bit! :D

Also, do you think there's a way to change that limit?
Reading this, it says that the limit is "due to the way [he] encoded library addresses...can be changed but it requires quite a bit more effort".

I think the ID system should be changed if possible... it would be kinda weird to start assigning IDs, or having some kind of system manage IDs and such. I think having some kind of Java-style naming system, like "com.alberthrocks.myawesomelibrary", would be better. (And then call this bFLT2, Nspire edition :P) Just my 2 cents. ;)

Again, great work! :D This will open the platform to C++ code, and make porting ten-fold easier! :D

There may be a way to change the ID limit but it requires modifying the linking tools and potentially breaking application support with other bFLT loaders (is this really an issue though?).

Shared libraries are implemented by having the 'ID' of the library in the high byte of the offset record in the relocations.

For example, if a offset record read 0x030000b4, it means "look for and load a library with ID of 3 and replace this offset with (the absolute address of that library image + 0xb4). That means the maximum ID number is 255 and the lowest 0 - giving us a total ID space of 256. The first and last are reserved so that only gives us 254 to work with. When you look at it this way, it becomes difficult to modify it to have more IDs because it's more of a fundamental issue.

You're right, I like the reverse domain notation too but it seems unlikely. I'm sure we can work out something though XD.

@edit: Yep, referring to version 4 XD
« Last Edit: April 02, 2012, 09:23:02 am by tangrs »

Offline Jim Bauwens

  • Lua! Nspire! Linux!
  • Editor
  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1881
  • Rating: +206/-7
  • Linux!
    • View Profile
    • nothing...
Re: Ndless bFLT loader
« Reply #3 on: April 02, 2012, 10:17:48 am »
Great work tangrs!

I think this (and the other loader stuff), deserve a news here :)

Offline ExtendeD

  • Project Author
  • LV8 Addict (Next: 1000)
  • *
  • Posts: 825
  • Rating: +167/-2
    • View Profile
Re: Ndless bFLT loader
« Reply #4 on: April 02, 2012, 02:00:02 pm »
Excellent! I'll test this as soon as possible.
Ndless.me with the finest TI-Nspire programs

Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55851
  • Rating: +3151/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: Ndless bFLT loader
« Reply #5 on: April 02, 2012, 05:18:16 pm »
Great to see more development tools. Sadly I cannot understand the use of this, since I'm pretty much illiterate regarding TI-Nspire ASM and C programming, but hopefully this helps a lot.

And yeah I think this and the Elf loader or whatever it is both deserve a news, although somebody else with more knowledge on the subject might have to write it so I can move it afterward.
In case you are wondering where I went, I left Omni back in 2015 to form CodeWalrus due to various reasons explained back then, but I stopped calc dev in 2016 and am now mostly active on the CW Discord server at https://discord.gg/cuZcfcF



Official Website |T-Shirt store | Reverbnation | Facebook | Youtube | Twitter | Spotify

Offline hoffa

  • LV6 Super Member (Next: 500)
  • ******
  • Posts: 322
  • Rating: +131/-13
    • View Profile
Re: Ndless bFLT loader
« Reply #6 on: April 02, 2012, 05:37:54 pm »
At least when it comes to nSDL, having a working and stable loader would be absolutely great. It would right away open a lot of doors and remove the burden of tedious debugging. Newlib would work, giving access to a working libc and making porting a lot easier (math library and whatnot). SDL_image and other libs would work with quasi no changes and nSDL code would be considerably cleaner. Good job tangrs, this'll be a great plus.

Offline tangrs

  • LV5 Advanced (Next: 300)
  • *****
  • Posts: 228
  • Rating: +97/-0
    • View Profile
    • tangrs blog
Re: Ndless bFLT loader
« Reply #7 on: April 02, 2012, 10:53:58 pm »
Update:

Houston, we have a problem!

I've been working on getting shared libraries working and I've modified the toolchain slightly to produce shared libraries and have them link (almost!) correctly to binaries. I can also grab the address of the function of the shared library at runtime. Basically, the bFLT loader support for shared libraries is done.

But there's a problem. I noticed that the address the binary actually branches to doesn't match the address of the function. Whatever it's branching to, it's also crashing the calculator.

Turns out GCC inserts some code verneers (or bridge, if you like) that is supposed to "glue" the main binary to the shared library. This is bad for us because the address the code verneer directs us to is not relocated.

Here's a nice colorful diagram:



Basically, calling a shared library function looks like this right now:

Code: [Select]
library_call() -> gcc code verneer -> unrelocated address = crash

We want GCC to skip making a code verneer so it looks more like this:

Code: [Select]
library_call() -> library function = win

Theoretically, we could work around this by defining and using a macro like this:

Code: [Select]
#define DIRECT_LIB_CALL(x, args...) do { \
        typeof(x) * volatile tmp = x; \
        tmp(#args); \
    } while (0)

DIRECT_LIB_CALL(library_call);

That should bypass the code veneer but of course, this isn't very elegant nor portable.

All the code is in the shared-library-support branch of the main project.

If anyone can help me look for an option to disable the making of a code veneer (also known as code glue, code bridge etc...) when linking with shared libraries, that would be great.


Never mind, discovered a workaround (although quite annoying)
« Last Edit: April 03, 2012, 09:14:26 am by tangrs »

Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
Re: Ndless bFLT loader
« Reply #8 on: April 03, 2012, 01:24:38 am »
Looks like the PLT code ( http://stackoverflow.com/questions/5469274/what-does-plt-mean-here and others)...
Indeed, either we don't want that code at all, or the Ndless loader needs to be able to resolve this kind of stuff :(

EDIT: pages such as http://www.acsu.buffalo.edu/~charngda/elf.html or http://fossies.org/unix/misc/glibc-2.15.tar.gz:a/glibc-2.15/FAQ might contain some useful information.
« Last Edit: April 03, 2012, 02:46:24 am by Lionel Debroux »
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline tangrs

  • LV5 Advanced (Next: 300)
  • *****
  • Posts: 228
  • Rating: +97/-0
    • View Profile
    • tangrs blog
Re: Ndless bFLT loader
« Reply #9 on: April 03, 2012, 09:19:37 am »
Update!

Surprise!

Shared library support is now stable!

Now, we need to work out a solution with the library ID problem. How do we allocate 254 IDs to the many people wishing to develop a library? Perhaps reserve IDs 3-10 for Ndless, IDs 10-250 for allocating to projects that are big enough and IDs 250-254 for private use? What do you guys think?

Apart from that, all that the bFLT loader needs now is a little polishing and integration into Ndless :)

(Don't you just love the look of all those "PASS"es? :P)

« Last Edit: April 03, 2012, 09:23:38 am by tangrs »

Offline Jim Bauwens

  • Lua! Nspire! Linux!
  • Editor
  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1881
  • Rating: +206/-7
  • Linux!
    • View Profile
    • nothing...
Re: Ndless bFLT loader
« Reply #10 on: April 03, 2012, 11:23:03 am »
Very nice :D

Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55851
  • Rating: +3151/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: Ndless bFLT loader
« Reply #11 on: April 03, 2012, 11:54:15 pm »
At least when it comes to nSDL, having a working and stable loader would be absolutely great. It would right away open a lot of doors and remove the burden of tedious debugging. Newlib would work, giving access to a working libc and making porting a lot easier (math library and whatnot). SDL_image and other libs would work with quasi no changes and nSDL code would be considerably cleaner. Good job tangrs, this'll be a great plus.

I assume nSDL is kinda like shells on the 84+ (MirageOS, Ion, etc), coming with routines to make development easier?
In case you are wondering where I went, I left Omni back in 2015 to form CodeWalrus due to various reasons explained back then, but I stopped calc dev in 2016 and am now mostly active on the CW Discord server at https://discord.gg/cuZcfcF



Official Website |T-Shirt store | Reverbnation | Facebook | Youtube | Twitter | Spotify

Offline tangrs

  • LV5 Advanced (Next: 300)
  • *****
  • Posts: 228
  • Rating: +97/-0
    • View Profile
    • tangrs blog
Re: Ndless bFLT loader
« Reply #12 on: April 04, 2012, 03:31:38 am »
Update: Committed some experimental Ndless patches for bFLT to the repo. The patches integrate bFLT loading directly into Ndless itself.
« Last Edit: April 04, 2012, 03:32:03 am by tangrs »

Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
Re: Ndless bFLT loader
« Reply #13 on: April 04, 2012, 03:39:51 am »
Great, this will speed up testing and integration further :)

BTW: you should update your local copy, and the patch, with Ndless 3.1 beta SVN r568 (or whatever the latest current version is). fixprint, for instance, requires r568 or later.
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline tangrs

  • LV5 Advanced (Next: 300)
  • *****
  • Posts: 228
  • Rating: +97/-0
    • View Profile
    • tangrs blog
Re: Ndless bFLT loader
« Reply #14 on: April 04, 2012, 03:52:05 am »
I think I'm on latest revision (r573 was it?). I think there was a typo in the file name. Let me fix it.