Omnimaga

General Discussion => Technology and Development => Computer Projects and Ideas => Topic started by: AaroneusTheGreat on December 27, 2011, 11:20:57 pm

Title: My Beowulf Cluster
Post by: AaroneusTheGreat on December 27, 2011, 11:20:57 pm
I've started a little project in my home, it's a Debian 6 (squeeze) based Beowulf cluster! From donated and unused desktops I've made what is more or less just a small LAN network right now with partial Beowulf setup implemented. I'm still working on the OpenMPI stuff. As it is right now, I've got them all running Stanford's Folding at Home, getting mucho work units done 24/7.

If you would like to assist me and know anything about clustering or Linux in general, I'm still kind of a noob at this and could use the help. I may post questions here for that reason, if anyone can help me, a bunch of calc programmers will probably have the answer if I can't find it on my own!

Thanks in advance to anyone who helps me with the project.

(http://a1.sphotos.ak.fbcdn.net/hphotos-ak-ash4/390338_10150439832707686_536897685_8739771_109823101_n.jpg)
Title: Re: My Beowulf Cluster
Post by: Juju on December 27, 2011, 11:23:40 pm
That looks pretty interesting.

Also welcome back!
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on December 27, 2011, 11:25:02 pm
Thanks! It's good to be back! By the way if anyone wants to join the project and my team feel free to do so:

http://folding.stanford.edu/

And the team number is 213973

Anyone can join my team, just insert the number in the configuration dialog.
Title: Re: My Beowulf Cluster
Post by: annoyingcalc on December 28, 2011, 12:06:57 am
Welcome back!
Title: Re: My Beowulf Cluster
Post by: TIfanx1999 on December 28, 2011, 07:48:50 am
Oh wow, it's been a while. I hope you're doing well. :) I don't know anything about the linux netwoking stuff, but hey w/e. What are you setting it up for? Also, I see lots of Dell computers. :P
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on December 28, 2011, 10:00:26 am
Thanks to everyone for the warm welcome back!

I'm setting it up to run the Folding@Home project for the most part, since I won't be using it myself all the time. Aside from that, when I am using it, I'm planning on writing a 3d rendering engine for OpenMPI and trying out some fun stuff there. It'll have to be in C or C++ so it can be portable to other Beowulf clusters.

OpenMPI is a Message Passing Interface, it sends parts of a large task to the units, and all the units crunch the numbers, then the nodes give their parts back to the head unit through the network. It's basically parallel processing, but on a larger scale than inside a multicore processor. Each node has it's own ram, gpu, etc, so there's a lot more to work with than a multicore cpu.

here's the wiki on what a cluster is and can do:

http://en.wikipedia.org/wiki/Beowulf_cluster

I am not well versed enough to tell you everything they are capable of or what the kind of power is in one of these things. I'm trying to figure out how to benchmark my system at the moment, and finish the OpenMPI setup. It's quite tricky when you don't know enough about Linux or networking in general. :p
Title: Re: My Beowulf Cluster
Post by: annoyingcalc on December 28, 2011, 11:10:42 am
I heard you met DJ_Omnimaga once
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on December 28, 2011, 12:07:14 pm
Sho nuff did. We met up when I was in Quebec studying french. We only got to hang out for a night. Dinner and drinks with my study abroad group. And of course a bit of nerding out with the calculators and such. It was awesome. I'm trying to get back up there at some point, for one thing I love the province of Quebec. I would actually live there if I could/didn't have so many ties and connections down here that I need to keep up. And for two, I want to be able to meet up with DJ again and possibly Juju if he can make it out to that area when I'm there. The three of us chatted about that last night. I may be making a considerable sum of money with a website job for a company and some install work for dental offices soon, so if that pans out you'll be hearing about me making another, probably more leisure based, trip up there.
Title: Re: My Beowulf Cluster
Post by: annoyingcalc on December 28, 2011, 12:34:34 pm
Wow you are so lucky!
Title: Re: My Beowulf Cluster
Post by: TIfanx1999 on December 28, 2011, 03:55:21 pm
@ATG: Alright, sounds pretty cool. =)
Title: Re: My Beowulf Cluster
Post by: Juju on December 28, 2011, 04:42:54 pm
Yeah, that would be pretty cool if the three of us meets together.
Title: Re: My Beowulf Cluster
Post by: Illumio on December 28, 2011, 04:45:43 pm
So many in one space! :O
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on December 28, 2011, 09:37:02 pm
@Illumio:
Yesh. :D it may get more diabolical soon. It's an odd cluster as it is, but I may be adding a netbook to it soon! O.o As it is my laptop is how I log into the cluster to do work on it. I can access each node with SSH and do installs and everything while laying in bed lololol. I set myself up for laziness. Gets more done sometimes, because I don't have to get up and scoot off the bed to work on the cluster. That was such a hassle! haha.

In all reality it was though, because I only have one kvm with two port sets on it, so I could only switch two units back and forth at the terminal easily, that's why there were so many keyboards. I did a little more work on this today and verified that my work units are getting worked on correctly, turns out 3 of the units were experiencing problems. My new stats should show up in a week or two when the units have finished enough work and the servers at stanford update.

Here's the stats link for the cluster on stanford's servers.

http://fah-web.stanford.edu/cgi-bin/main.py?qtype=teampage&teamnum=213973

This data will change every day. I have a desktop (my dad's) in the floor below me running a client on his GPU. It gets work units done MUCH faster than any of the units in this cluster. I'm not even sure if my cluster as a whole will outperform it honestly. These units are pretty old. I also haven't figured out a way to benchmark the system yet.

I'll let you all know what the final word is about processing power when the system is finally benchmarked though.

@annoyingcalc: Yes it was quite fortuitous to be able to do those things. I had to do a ton of work to get where I was though, french classes every year since the 6th grade without a gap but once in my freshman year of highschool. I'm not currently taking any french classes but I've been studying forever. I only ended up going because I just so happened to have made a good deal of money building a website, and I had run out of french classes at my community college. So I took the last thing they offered in french, which was the study abroad.

@Juju: We'll see if I can make it happen, but rest assured I'm trying my best! :)

It's really good to talk to you all again, I've missed you guys. It's really cool that you all were so quick to give me a warm welcome back. That's one of the things I love about this community. You don't get this kind of thing everywhere on the internet. Lots of things are exclusive and require a lot of rites of passage to get into. The calculator community has proven to be the most open programming community I've run into. I really hope that never changes.
Title: Re: My Beowulf Cluster
Post by: DJ Omnimaga on December 29, 2011, 01:03:57 am
Heya it's great to see you back Aaron. This project seems interesting. I can't help much though since I'm not very literrate about computer stuff. However good luck :)

Also I hope we can meet again at one point. One good spot would be during my Summer vacations where I am guaranteed to have at least two Sundays and Saturdays off. It's usually during the last two weeks of July or the first two of August of each Summer. I would most likely be able to help a bit financially as well. I usually see Juju once every month or so depending of our schedules and it would be nice if we could all see each others together. :D

Anyway glad to see you return. I hope you stay around for a while even if you don't do calc stuff anymore. Hopefully you might find some familiar faces too here as well. (Art_of_Camelot, calc84maniac and Iambian, for example)
Title: Re: My Beowulf Cluster
Post by: Illumio on December 31, 2011, 02:14:16 am
I see.  Though I can only imagine the amount of cords everywhere.  It's a sea that will never end.
Title: Re: My Beowulf Cluster
Post by: DJ Omnimaga on December 31, 2011, 02:44:55 am
Yeah that must be a lot of cords. You haven't seen behind my TV, though, with all the gaming consoles I got. O.O About 50% of them aren't even plugged in anymore due to running out of slots (idk if it's the right word for where you plug stuff in the wall)
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on December 31, 2011, 04:07:15 pm
There really are a lot of cords.

(http://a8.sphotos.ak.fbcdn.net/hphotos-ak-snc7/396414_10150453933852686_536897685_8808225_1111112815_n.jpg)

I added an 8th unit to the cluster today. Cost me $24.99 for the tower, and $9.99 for a harddrive for it. It was at a local Goodwill Computer Works store, and was marked for parts only, totally fine, just needed a harddrive and an OS. I set it up today and it's currently running just like the other units. It took me a few hours to do, and a lot of typing, but it works just fine.   :)

(http://a7.sphotos.ak.fbcdn.net/hphotos-ak-snc7/400790_10150453927387686_536897685_8808209_803070645_n.jpg)

The current computer count under my username for Folding@Home is now up to 10 computers, including my laptop and the desktop my dad uses downstairs, he asked me to add it to the project the other day. It's running the GPU client on an ATI Radeon 5700 graphics card. The thing flies through work units like nothing I've seen before, so I'm getting lots of stuff done.

Interesting side effect, I had to turn the heaters down in my apartment and crack the windows because the cluster is generating so much heat that it warms up my place. O.o

I'm curious to see what the cord bundle behind your tv looks like DJ_O. lol it's probably more diabolical than the one behind my cluster. Game systems generally require more cables than computers do to run them. My systems are running headless, so there's no need for keyboards or mice anymore, so that slimmed down the cord nightmare a bit.

Anyways I'm starting to ramble, just wanted to update you all on what's going on with the nerdiness.
Title: Re: My Beowulf Cluster
Post by: chattahippie on December 31, 2011, 09:03:59 pm
This is really cool! :D
Do you know the flops of the cluster?
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on January 05, 2012, 09:20:33 am
Not at all yet. lol. I haven't figured out a way to benchmark it yet. I've got the hostlist and whatnot setup, and installed OpenMPI on several of the machines, and I'm getting closer to getting it running as a true beowulf, but I'm still having problems getting the machines to talk to each other through the MPI stuff. I keep getting strange errors, I think it's because I don't know which flags to pass the mpiexe program (aside from the most basic things), which runs any MPI program, so I'm probably just invoking it incorrectly, or don't have the right services up on my nodes when the server unit tries to talk to them.

Cluster update:

(reposted from facebook)

I discovered some issues, wouldn't you know it? Nothing in the computer science field seems to be straight forward sometimes. I found out that several of my units have missed their deadlines that folding at home gives for when work units need to be finished, due to my startup scripts. /facepalm

So that said, I've just gone ahead and disabled them for now, and started the programs on all 8 machines individually, some of them got new units, the others seem to be attempting to finish some units, which may be pointless at this point, they might miss the deadlines too, which is okay, they will get new units soon. The deadlines coming up are tomorrow and the next day. We'll see what happens. At this point it's kind of a race against time for some of my units lol.

All of the systems have been switched over (with minor difficulty) to a completely command line based Linux, so as to save resources for the project, which is command line anyways. Initially I used the GUI to kind of ease myself into the more difficult aspects of Linux, but now it's kind of useless, so I just got rid of it, which turned out to be much easier than anticipated. I did have to fix several things that got broken when I removed the packages, but everything seems to be okay right now. I've had no issues getting the clients running or anything like that since.

Many thanks go out to Noah Putman ... yet again... saved my butt when I couldn't figure it out on my own. The guy's a whiz with Linux.

The units that did finish work got a lot done recently though. There have been 53 work units so far that have been completed. The cluster has also jumped into the middle 20th percentile in the rankings now, that's pretty impressive for such old machines.

(The server and the first four units went online in mid December, so it hasn't been very long, but it also hasn't been that long either, a couple of weeks, so if you do the math, that's a few work units a day, which is pretty good for having so many issues so far)
Title: Re: My Beowulf Cluster
Post by: flyingfisch on January 05, 2012, 09:24:54 am
How hard is it to set something like this up? I would like to try it.... maybe :P
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on January 05, 2012, 10:40:59 am
You know it's really not as hard as I expected. I've had some help from the drummer in my band. That's who I mentioned in the previous post. He's a Linux guru, has been using UNIX based systems since the late 90's or so I think. Around the time I got into programming more or less. So between the two of us, it's been pretty straightforward. With the help of some of the guys on here, I'm sure you could set one up in a matter of days. It only took one day of pretty inconsistent work to get my server and the first 3 nodes online. After that, each node only took about an hour to get completely setup.

The trick is to get them running the right way. If you want to run something like I am, you really don't need an MPI or any SMP technology (http://en.wikipedia.org/wiki/Message_passing_interface if you're curious where to start reading about that). I could run the whole system here as simply a small LAN network and it would be all I needed for running a dedicated Folding at Home farm. That said, I want to use this thing for other projects of my own, so I'm setting up OpenMPI which has a mildly steep learning curve when transitioning from Window$ to Linux to MPI clustering with Linux. I have focused on calculators so much that I've kind of had to catch up with the larger units. I decided to take this on because I knew I would need to learn some things to get into the research department at the college I'm attending.
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on January 07, 2012, 11:28:30 pm
Update: I'm quickly approaching the 100 work unit completed! As it is, while I'm writing this post, I've completed 85 work units total. I'm averaging about 4 units per 24 hours. They complete at different times, due to the size differences, but that figure is relative to when I started work with 4 running nodes, that weren't acting right. In the last week or so, I've seen about 6 to 7 units done every 24 hours. The cluster is really chugging along.

Speaking of how hard it's working. I've had to turn my air conditioner on in the middle of winter. I came home from a walk, and my house was about 80 degrees Fahrenheit, or about 26.6 degrees Celsius for those of you who live in the metric world.
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on January 09, 2012, 03:16:49 am
UPDATE:

Very exciting news! As of 2012-01-08 23:09:00 My computers have finished 100 work units! Yay for milestones! 15 work units were finished since my last post! I'm blown away! Pretty awesome to see the cluster working so well considering the age of the machines! I discovered one of my units is 12 years old! Wow. The compaq on the far right in the picture is the one I'm speaking of. 700mhz AMD duron processor. Not quite anywhere as old as my TI-99/4a but quite the dinosaur in it's own right. The TI-99/4a is older than me by about 7 years, if you can feature that. 1981 was the year, when the pterodactyls flew.
Title: Re: My Beowulf Cluster
Post by: TIfanx1999 on January 09, 2012, 11:23:05 am
O.O 1981 isnt *THAT* old! It's vintage I tell you, vintage!!! :P
Title: Re: My Beowulf Cluster
Post by: flyingfisch on January 09, 2012, 11:27:30 am
Sorry if this is a dumb question, but what is a work unit?
Title: Re: My Beowulf Cluster
Post by: Spyro543 on January 09, 2012, 07:34:35 pm
In folding@home, a "work unit" is just a way of measuring how much of a protein the computer has calculated so far.
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on January 10, 2012, 05:15:59 am
Speaking of work units, Willrandship's unit has just finished it's first unit and been awarded 600 points! Congrats Will on your first unit for the team!

If you would like to see a video of why we are doing this, here's a good example of the kind of work this program is doing. Below is a successfully simulated protein fold. Proteins fold up in the body for several reasons, it creates active surfaces on the molecule that allow the individual units to form complex structures for one thing, like the cillia on your muscle cells, which allow you to move, or the cell walls of your blood cells.

Protein folding is intensely important in human beings because, when it goes wrong,  it can cause a myriad of problems. Mad cow disease is a protein misfolding disease for instance, and as it's been discovered recently, it appears that Alzheimer's is as well. Several of my units are in fact working on the Alzheimer's Amyloid Protein sequences right now.  

By the way, I would like to try to recruit as many people as possible that I know to help the project. Anyone here is more than welcome to join my team!

TEAM NUMBER: 213973

http://folding.stanford.edu/English/Download (http://folding.stanford.edu/English/Download)

That is the download link for the client, and here's the installation instruction page:

http://folding.stanford.edu/English/Guide (http://folding.stanford.edu/English/Guide)

As previously mentioned, here's that nifty video:


Title: Re: My Beowulf Cluster
Post by: flyingfisch on January 10, 2012, 12:29:51 pm
do you have to have a powerful computer to participate in folding@home?
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on January 10, 2012, 09:49:38 pm
Not in the slightest. One of my machines, the very old 700 megahertz Duron processor unit, is probably much slower than your computer, unless you haven't bought a computer in 12 years, it has only finished about 2 work units, but it's contributing none the less. The minimum requirements for a system is a Pentium 3 at 450 megahertz.

The only thing I would tell you to watch out for is while setting up your client, take into account the amount of ram that you've got, that's the major thing. If your computer has 256 megabytes of ram or less, then you're going to want to set your work units to normal or small sized units, and just check on the progress every day or two to see if it's coming along quickly enough. If it looks as if the progress won't make the deadline, set the units to a size step down, and wait for the deadline to make the system get a new unit.

Chances are you'll have no problems though. The guides on the website, and the forum they've got setup are more than helpful, they make it rather easy to get started.

Here's the FAQ page, I've referred to this many times and found what I was looking for when it came to answers about what to do and/or what it does etc. Very useful info.

http://folding.stanford.edu/English/FAQ-main#ntoc1 (http://folding.stanford.edu/English/FAQ-main#ntoc1)
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on January 17, 2012, 10:25:45 am
My team has reached another milestone! As of today, we have folded over 200 work units!
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on February 01, 2012, 04:01:49 pm
Today the team reached yet another milestone! We've now folded over 400 work units! At the rate we're going, we fold on average 13 to 14 work units a day. My friend GraveKeeper just joined the team recently, running an instance of folding on his server machine and his laptop. I want to see if the team can make the top 500 in the project. we are currently 11549 out of 209056 which puts us in the 94th percentile in the world. Which is pretty impressive on it's own, but I want to see if we can get the team into the competitive ranks at the top. For one if the team makes news anywhere it could spark more interest in the calculator community, and two, it's very important work.

If you've been following this project and you've got a computer that runs quite a bit, like a desktop that you leave on all the time, or a laptop that you use quite a bit, then please join the team! I can walk you through the installation of the program, and rest assured, it can be paused when you need your full processor time back.

To give you an example of the impressive performance of this project's client program, I run two different instances of the client on my laptop, and I can still play SKYRIM on high quality graphics settings. This is on a laptop that doesn't have a dedicated graphics card, it's an integrated chip on the processor.

That means that this client is only using the waiting cycles of my processor, not slowing down my performance at all. I have no problems with performance because of the client. No excuses guys!!! :p Let's make history! Send me a personal message if you need help with installing anything.

The team number is 213973

You need to input this team number and a username in the configuration of the client in order to be counted as part of the team. 

Title: Re: My Beowulf Cluster
Post by: DJ Omnimaga on February 14, 2012, 02:43:44 pm
Nice Aaron. Also I'm really wonder... at which max res can you play Crysis before the game slow down too much? I'M curious for example if you could play with 6 monitors together (2 rows by 3 columns) at 1920x1080 each and still get 60 fps? :P
Title: Re: My Beowulf Cluster
Post by: Juju on February 14, 2012, 08:43:07 pm
I wonder if you can even cluster Crysis to run on several computers at the same time so it'll run flawlessly.
Title: Re: My Beowulf Cluster
Post by: DJ Omnimaga on February 16, 2012, 12:06:42 am
That might be pushing things a bit too far now, though. It's not a calculator, after all. :P
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on September 21, 2012, 03:01:51 pm
The Crysis thing may be doable but it would probably require a lot of work to get it to run as an SMP application as many games and such are timed based on real-time clock oscillators, this is due to modern machines being much faster than is necessary to run most games. SMP relies on parallel, mostly precomputed data sets. So when something is event based, pretty much all that can be done in the background through SMP would be things like rendering non event-based graphical data.

UPDATE:

The team is about to break 2,000,000 points!!

We also now outrank some prestigious universities and some major groups in the world.

Current ranking: 2,713 / 215586

We now out rank:

Oregon Institute of Technology
Auburn University
Unixheads folding group (major unix enthusiast group)
Hardin-Simmons University
Robert Morris University ACM
Wright State University

The most amazing thing of all WE OUTRANK WIKIPEDIA!!!!

http://folding.extremeoverclocking.com/team_list.php?s&t=34485#34485
Title: Re: My Beowulf Cluster
Post by: AaroneusTheGreat on September 21, 2012, 08:16:57 pm
Sweet! Thanks! Remember to join team 213973 !