Ndless bFLT loader

[tangrs]

This topic will be for technical discussion for the bFLT loader.

Source code and API documentation

Let me start off this discussion.

I propose that, once bFLT becomes stable enough, it be integrated into the core Ndless code and have the SDK updated to only produce bFLT executables. However it seems that if bFLT were to be integrated into Ndless, we'll need to change some parts of the Ndless SDK itself to work in it's favor. I hope we can quickly deprecate the current way of loading binaries.

First, because bFLT stores sections in the order of .text, .data, .bss, we need to update the Ndless ldscript to reflect that. Otherwise, we run into overlapping problems. More information can be found at the bottom of the README. This is the first and easiest step.

Next: Because we do all the relocating, the startup files become redundant - even a hindrance when we deal with shared libraries. The only thing the startup files should do is branch to main since the bFLT loader will handle everything else.

Also, we need some sort of way to work out the assignment of library IDs. bFLT shared libraries work by IDs. Each library is given a unique ID and it is up to the user to ensure it is unique. Unfortunately, the number of IDs are limited and only 254 are actually available for use (255 and 0 are reserved).

I'll leave it at that for now.

@edit: is there a way of implementing _syscallvar_savedlr without relying on a linker script?
@edit2: I'm also having trouble producing a bFLT executable that doesn't rely on nspire-gcc or nspire-ld. Anyone have luck with this?
I'm having trouble writing a ldscript that will force the GOT to be at the beginning of the data section. Got it.


Update:

The bFLT loader has a working SDK ready for developing and testing! You can now produce bFLT binaries without affecting your normal Ndless SDK!

[alberthrocks]

This is great! I was thinking about porting bFLT a while ago, but lacked skills and time to do it. This will definitely advance Nspire development quite a bit!

Also, do you think there's a way to change that limit?
Reading this, it says that the limit is "due to the way [he] encoded library addresses...can be changed but it requires quite a bit more effort".

I think the ID system should be changed if possible... it would be kinda weird to start assigning IDs, or having some kind of system manage IDs and such. I think having some kind of Java-style naming system, like "com.alberthrocks.myawesomelibrary", would be better. (And then call this bFLT2, Nspire edition ) Just my 2 cents.

Again, great work! This will open the platform to C++ code, and make porting ten-fold easier!

EDIT: Actually, bFLT has a version of 4.0 - I'm assuming you're using that spec, right? Also, might want to call the version 5.0 instead...

[tangrs]

This is great! I was thinking about porting bFLT a while ago, but lacked skills and time to do it. This will definitely advance Nspire development quite a bit!

Also, do you think there's a way to change that limit?
Reading this, it says that the limit is "due to the way [he] encoded library addresses...can be changed but it requires quite a bit more effort".

I think the ID system should be changed if possible... it would be kinda weird to start assigning IDs, or having some kind of system manage IDs and such. I think having some kind of Java-style naming system, like "com.alberthrocks.myawesomelibrary", would be better. (And then call this bFLT2, Nspire edition ) Just my 2 cents.

Again, great work! This will open the platform to C++ code, and make porting ten-fold easier!

There may be a way to change the ID limit but it requires modifying the linking tools and potentially breaking application support with other bFLT loaders (is this really an issue though?).

Shared libraries are implemented by having the 'ID' of the library in the high byte of the offset record in the relocations.

For example, if a offset record read 0x030000b4, it means "look for and load a library with ID of 3 and replace this offset with (the absolute address of that library image + 0xb4). That means the maximum ID number is 255 and the lowest 0 - giving us a total ID space of 256. The first and last are reserved so that only gives us 254 to work with. When you look at it this way, it becomes difficult to modify it to have more IDs because it's more of a fundamental issue.

You're right, I like the reverse domain notation too but it seems unlikely. I'm sure we can work out something though .

@edit: Yep, referring to version 4

[Jim Bauwens]

Great work tangrs!

I think this (and the other loader stuff), deserve a news here

[ExtendeD]

Excellent! I'll test this as soon as possible.

[DJ Omnimaga]

Great to see more development tools. Sadly I cannot understand the use of this, since I'm pretty much illiterate regarding TI-Nspire ASM and C programming, but hopefully this helps a lot.

And yeah I think this and the Elf loader or whatever it is both deserve a news, although somebody else with more knowledge on the subject might have to write it so I can move it afterward.

[hoffa]

At least when it comes to nSDL, having a working and stable loader would be absolutely great. It would right away open a lot of doors and remove the burden of tedious debugging. Newlib would work, giving access to a working libc and making porting a lot easier (math library and whatnot). SDL_image and other libs would work with quasi no changes and nSDL code would be considerably cleaner. Good job tangrs, this'll be a great plus.

[tangrs]

Update:

Houston, we have a problem!

I've been working on getting shared libraries working and I've modified the toolchain slightly to produce shared libraries and have them link (almost!) correctly to binaries. I can also grab the address of the function of the shared library at runtime. Basically, the bFLT loader support for shared libraries is done.

But there's a problem. I noticed that the address the binary actually branches to doesn't match the address of the function. Whatever it's branching to, it's also crashing the calculator.

Turns out GCC inserts some code verneers (or bridge, if you like) that is supposed to "glue" the main binary to the shared library. This is bad for us because the address the code verneer directs us to is not relocated.

Here's a nice colorful diagram:



Basically, calling a shared library function looks like this right now:

Code: Select All | Copy To Clipboard

1 2	library_call() -> gcc code verneer -> unrelocated address = crash

We want GCC to skip making a code verneer so it looks more like this:

Code: Select All | Copy To Clipboard

1 2	library_call() -> library function = win

Theoretically, we could work around this by defining and using a macro like this:

Code: Select All | Copy To Clipboard

1 2 3 4 5 6 7

#define DIRECT_LIB_CALL(x, args...) do { \         typeof(x) * volatile tmp = x; \         tmp(#args); \     } while (0) DIRECT_LIB_CALL(library_call);

That should bypass the code veneer but of course, this isn't very elegant nor portable.

All the code is in the shared-library-support branch of the main project.

If anyone can help me look for an option to disable the making of a code veneer (also known as code glue, code bridge etc...) when linking with shared libraries, that would be great.

Never mind, discovered a workaround (although quite annoying)

[Lionel Debroux]

Looks like the PLT code ( http://stackoverflow.com/questions/5469274/what-does-plt-mean-here and others)...
Indeed, either we don't want that code at all, or the Ndless loader needs to be able to resolve this kind of stuff

EDIT: pages such as http://www.acsu.buffalo.edu/~charngda/elf.html or http://fossies.org/unix/misc/glibc-2.15.tar.gz:a/glibc-2.15/FAQ might contain some useful information.

[tangrs]

Update!

Surprise!

Shared library support is now stable!

Now, we need to work out a solution with the library ID problem. How do we allocate 254 IDs to the many people wishing to develop a library? Perhaps reserve IDs 3-10 for Ndless, IDs 10-250 for allocating to projects that are big enough and IDs 250-254 for private use? What do you guys think?

Apart from that, all that the bFLT loader needs now is a little polishing and integration into Ndless

(Don't you just love the look of all those "PASS"es? )

[Jim Bauwens]

Very nice

[DJ Omnimaga]

At least when it comes to nSDL, having a working and stable loader would be absolutely great. It would right away open a lot of doors and remove the burden of tedious debugging. Newlib would work, giving access to a working libc and making porting a lot easier (math library and whatnot). SDL_image and other libs would work with quasi no changes and nSDL code would be considerably cleaner. Good job tangrs, this'll be a great plus.

I assume nSDL is kinda like shells on the 84+ (MirageOS, Ion, etc), coming with routines to make development easier?

[tangrs]

Update: Committed some experimental Ndless patches for bFLT to the repo. The patches integrate bFLT loading directly into Ndless itself.

[Lionel Debroux]

Great, this will speed up testing and integration further

BTW: you should update your local copy, and the patch, with Ndless 3.1 beta SVN r568 (or whatever the latest current version is). fixprint, for instance, requires r568 or later.

[tangrs]

I think I'm on latest revision (r573 was it?). I think there was a typo in the file name. Let me fix it.