Omnimaga
Omnimaga => Our Projects => Ndless => Topic started by: tangrs on December 15, 2011, 04:41:52 am
-
The ELF loader has reached a stage where most people can use it to load basic stuff.
When you run the program, it installs a hook. After that, you can load ELF files as if you'd load a normal Ndless binary.
More instructions on how to compile the loader and how to develop applications that take advantage of it are described in the README (https://github.com/tangrs/ndless-elfloader/blob/master/README.md).
You can get the source code on Github (https://github.com/tangrs/ndless-elfloader).
Happy deving!
I'd also like to thank everyone who contributed to Ndless. I did borrow some of Ndless's code for use in this loader.
Original post:
I'm not sure if it's relevant here, but I'm developing an opensource ELF loader for Ndless.
I'm writing this because the current Ndless way of loading binaries doesn't work for code that relies on static initialization of pointers.
I.e. code like this doesn't work:
void foo() {
//blah
}
int main() {
static void (*var)() = foo; //Since it's static, the address will be inserted at link time (which is 0x8000+offset on my machine)
var(); //The GOT based relocation code in Ndless currently does not update the static variable function pointer.
return 0; //Crash
}
Unfortunately, there's no way to fix this because Ndless binaries are converted into a memory image before running. That means the relocation code doesn't know what to update in the .data section of the memory because it doesn't know what symbols exist and need updating and where they're located.
Loading from an ELF file works because the symbol definitions are there and the relocation code knows where to find the bits that need patching. That's why I wrote this ELF loader.
Anyway, the core code is there, just needs a lot of polishing up.
If anyone wants to help, the source code is available on Github https://github.com/tangrs/ndless-elfloader
I'm hoping eventually, it will be integrated into the program loader on Ndless.
Thanks for your time,
Apologies if this is the wrong forum to post.
-
Interesting :)
I don't however know much about this subject to give a good response :P
But thanks for sharing :)
(on a side note, did you get my pm of a few day's ago?)
-
Interesting :)
I don't however know much about this subject to give a good response :P
But thanks for sharing :)
(on a side note, did you get my pm of a few day's ago?)
Yeah, I did. I couldn't respond because I'm a new user and can't send messages yet or something.
-
Interesting indeed :)
The flip side of the coin of the power of ELF is its complexity, and the weight of a full-featured launcher. In the past, there were discussions about using BFLT, which is a much simpler format, easily derived from ELF, and fulfills the purpose you're describing.
-
BFLT does look interesting. I'll look into it.
Out of curiosity, why was BFLT never used?
Edit: You're right. ELF is pretty complex. Out of the week I spent on the project, 4-5 days were spent just reading and re-reading manuals and example code XD
-
Out of curiosity, why was BFLT never used?
Well, I think it has been much more a matter of finding enough free time, than a matter of feasibility :)
-
Out of curiosity, why was BFLT never used?
Well, I think it has been much more a matter of finding enough free time, than a matter of feasibility :)
Ahhh, I see. That makes sense.
-
Interesting tangrs.
But integrating to Ndless an ELF loader seems to be a bit overkilled just to support static initializers.
It's true that this would be annoying for initialization of arrays, but there's a simple workaround for this with nl_relocdata() (http://hackspire.unsads.com/wiki/index.php/Ndless_features_and_limitations#Global_variables).
-
Interesting tangrs.
But integrating to Ndless an ELF loader seems to be a bit overkilled just to support static initializers.
It's true that this would be annoying for initialization of arrays, but there's a simple workaround for this with nl_relocdata() (http://hackspire.unsads.com/wiki/index.php/Ndless_features_and_limitations#Global_variables).
Oh, I know for c there are workarounds easily but I'm also working on getting C++ code working on Ndless and vtables and the lot are a lot more difficult to relocate using workarounds. That was what the ELF loader was originally for haha.
Though I do agree it might be a bit of an overkill especially considering this is an embedded platform.
Thanks for the insights guys, you guys pointed out some things I overlooked :)
-
I see. But you can probably keep your ELF loader in user space until you reach a stable C++ toolchain, I then may integrate both if you want.
-
I think an ELF loader would be amazing :D It would allow C++ support, and (possibly) dynamic libraries! :D (Although I don't think dynamic libraries would be good on the Nspire - a bundled, static package is more portable - it still will be useful for some other aspects and the thought of it being possible is nice :)
On a somewhat related note, is the Newlib issue on the wiki (http://hackspire.unsads.com/wiki/index.php/Ndless_features_and_limitations#Newlib) still relevant? I see that we are including newlib in the API building...
-
I see. But you can probably keep your ELF loader in user space until you reach a stable C++ toolchain, I then may integrate both if you want.
Thanks, that would be great!
I think an ELF loader would be amazing :D It would allow C++ support, and (possibly) dynamic libraries! :D (Although I don't think dynamic libraries would be good on the Nspire - a bundled, static package is more portable - it still will be useful for some other aspects and the thought of it being possible is nice :)
On a somewhat related note, is the Newlib issue on the wiki (http://hackspire.unsads.com/wiki/index.php/Ndless_features_and_limitations#Newlib) still relevant? I see that we are including newlib in the API building...
Haha, dynamic libraries would be pretty interesting but would get pretty damn messy XD
-
On a somewhat related note, is the Newlib issue on the wiki (http://hackspire.unsads.com/wiki/index.php/Ndless_features_and_limitations#Newlib) still relevant? I see that we are including newlib in the API building...
Yes It's still relevant.
-
tangrs, I see on your blog (http://blog.tangrs.id.au/?p=712) that you actually have a C++ build chain that works more or less. Do C++ programs now runs with your ELF loader?
What are the "required functions" not implemented by Ndless that exception handling needs?
-
tangrs, I see on your blog (http://blog.tangrs.id.au/?p=712) that you actually have a C++ build chain that works more or less. Do C++ programs now runs with your ELF loader?
What are the "required functions" not implemented by Ndless that exception handling needs?
Yes, the ELF loader loads most C++ programs. There's still more testing to do though but it works for the basic test programs I wrote.
The C++ buildchain is really a hit and miss kind of thing. I can't remember what kind of things the C++ compiler needed to handle exceptions - I remember that the linker was complaining about missing symbols related to C++ exceptions. I'll look into what exactly is needed for exceptions though.
Edit: Probably something like <a href="http://wiki.osdev.org/C%2B%2B_Exception_Support">this</a> would work.
-
Just a quick update on this
The ELF loader is at a point where most users can use it. I've removed the debugging console and polished it up a bit. Most users can compile it and use it for loading their ELF files now (instructions on how to build your programs to work on this is on the readme).
It's implemented as a hook onto the normal program loader so you can open ELF binaries just as easily as you do opening normal Ndless binaries.
I'd be interested to hear your feedback and any bug reports!
Thanks!
-
Hmm interesting stuff, although I have no clue what is ELF at all, so I'M gonna have to do some research on the subject. I'm glad to see new Ndless-related stuff in the works again. :)
-
Nice tangrs :)
For my understanding, before I find more time to look into this :
- What are the remaining issues?
- How big the loader is, and how much bigger does ELF programs are compared to the current programs?
- How far would the loader be from being able to link programs with dynamic libraries (and is this something really want looking back on the TI-68k experience)?
-
ELF is the executable file format used on Linux and most UNIX-based OSes, right?
-
Right, juju :)
dynamic libraries (and is this something really want looking back on the TI-68k experience)?
Indeed, +1. Even a simple versioning scheme can cure a number of DLL-related woes.
-
Nice tangrs :)
For my understanding, before I find more time to look into this :
- What are the remaining issues?
- How big the loader is, and how much bigger does ELF programs are compared to the current programs?
- How far would the loader be from being able to link programs with dynamic libraries (and is this something really want looking back on the TI-68k experience)?
Most of the issues remaining are pretty much error checking and bug hunting and fixing.
The loader is roughly 53KB.
Depending on your linker options and program, ELF files are generally many times larger than normal binaries but it's hard to know exactly how much. To give an idea, you normally won't find a ELF file less than 30ish KB. I've seen a 480B program with an equivalent ~50KB ELF file. Weird enough, I've also seen a 53KB program with a 60KB equivalent ELF file.
I've had a thought and I think it's probably better it's made into an optional extra in Ndless since the loader contains a lot of bulk and most users won't be needing ELF files. Thoughts?
I haven't really looked into loading dynamic libraries so I'm unsure of it's complexities right now. It does seem a little messy though (i.e. as mentioned, we could easily run into an equivalent of DLL-hell). I'll take a look into it and see if I can work something out.
-
In my previous post, I forgot to post something along the lines of "nice job". Please accept my apologies ;)
I've seen a 480B program with an equivalent ~50KB ELF file.
Uh, does this one contain an awful amount of debugging information, or is it just the toolchain going insane ?
The complexity of ELF is the reason why several of us had thought of the much simpler BFLT :)
DLL hell can be alleviated by using versioning (in a correct fashion).
-
Nice to see much progress on it.
I only understand half of what is said, but I can see its something good :D
-
In my previous post, I forgot to post something along the lines of "nice job". Please accept my apologies ;)
I've seen a 480B program with an equivalent ~50KB ELF file.
Uh, does this one contain an awful amount of debugging information, or is it just the toolchain going insane ?
The complexity of ELF is the reason why several of us had thought of the much simpler BFLT :)
DLL hell can be alleviated by using versioning (in a correct fashion).
Thanks, it's appreciated :)
I don't know what's causing that massive ELF file. I'll need to investigate what's hogging all the file size. It does seem an awful lot LOL
I'll look into BFLT - maybe I should channel my energies into writing a BFLT loader instead XD
Nice to see much progress on it.
I only understand half of what is said, but I can see its something good :D
Thanks, but it's nothing really special or anything most users would even want to use XD
@ExtendeD, loading shared libraries seem to be a little more complex than I originally thought. It's most likely doable but I'm just wondering about the performance implications of it. I'll have a crack at it some time.
-
OK, good luck with this.
-
Any progress on this? ELF support would be very interesting to see on the Nspire! :D
-
Any progress on this? ELF support would be very interesting to see on the Nspire! :D
The basic loader works on the older 2.1 models but I haven't had a look on how to get it to work on Ndless 3.1 (and take advantage of it's built in file association system). I'll get around to it soon.
-
Good, I'm waiting for it :)
-
Glad to see you will support Ndless 3.1 :)
-
Did you manage to cut down on the size of the massive ELF files ? :)
-
I had another thought on how to tackle the problem of relocation and instead of loading ELF files, I made the programs relocate at runtime.
See this topic for more details: http://ourl.ca/15714
This new way of loading will (naturally) solve the issue of massive ELF files.